Introducing the Benchmark Explorer

Last week we released the Revolab ASR Benchmark, a dataset of 1,899 annotated audio samples across twelve Malaysian speech domains. That was the data. Now we're releasing the Benchmark Explorer: a tool that lets you hear the audio, see every model's output, and understand where each one breaks down.

The Explorer covers the full benchmark: 820 public samples visible to everyone, plus 1,079 private samples that test models on conditions they weren't tuned for. The public leaderboard tells one story. The combined results tell another.

Why we built it

A WER number in a table doesn't tell you much. Scribe v2 scores 4.8% on the public set overall. But on singing clips it hits 11.2%. Aisyah 1.0 Pro scores 5.8% overall, with the lowest insertion rate of any model (0.62%), meaning it almost never hallucinates words that weren't said. Different models, different failure modes, and a single number hides all of it.

We kept running into questions a leaderboard couldn't answer:

• What exactly did Whisper hear on that street interview clip? Can I listen?
• When Qwen3-ASR substitutes "perniagaan" with "penyegaan", is that a one-off or a pattern?
• This model looks good on clean audio but falls apart on noise. How bad exactly?
• Are there samples where every model fails?

The Explorer answers these directly. Click any sample. Hear the audio. See the ground truth. Compare every model's output with word-level diffs showing what was substituted, inserted, or deleted. No aggregation, no averaging.

The Explorer's sample view. Left: filterable sample list sorted by WER. Right: native audio playback, human-verified ground truth in green, and model cards with word-level diffs. Orange = substitutions, blue = insertions. Each card shows edit operation counts. Click play to hear the actual audio from the benchmark.

A benchmark that only publishes aggregate scores is a leaderboard. A benchmark that lets you hear the audio and see the errors is something you can build on.

We evaluated our own models (Aisyah) with the same transparency as everyone else's. The Explorer shows where they fail too. If our numbers are going to mean anything, they need to be auditable by anyone who clicks.

The current ranking

Fourteen models evaluated across twelve speech domains. The table below shows scores on the 820 public samples. The full benchmark includes an additional 1,079 private samples, and the combined ranking is where things get interesting.

On the public set (820 samples), Scribe v2 leads by a thin margin. On the full 1,899-sample benchmark, Aisyah 1.0 Pro takes the top spot at 5.79% WER. The private set reshuffles the ranking: several models see a sizeable jump in WER moving from public to private, with Universal 2 and 3.5 Pro rising from ~15% to ~22%. Every model in this table was evaluated on both splits.

What we found when we looked closer

Models hallucinate in different ways

WER doesn't distinguish between a substitution (wrong word) and an insertion (word that was never said). The insertion column above is the hallucination rate. The range is wide:

On a parliament clip, Whisper inserts entire phrases that weren't spoken. Gemini 2.5 Pro stays silent on words it's unsure about. Both might have similar WER. One of them is safe to deploy on phone calls.

Word repetition — the same word said once, transcribed multiple times

WER doesn't penalise repetition heavily. But the transcription is clearly broken.

"macam" said once in the reference, repeated 2–3 times by Qwen3-ASR-1.7B and Universal 2. The WER penalty is small because the correct word is technically present. But anyone reading the transcription can see it's broken.

Four models, four different failure modes on the sample above. Scribe mishears "individu" as "individual". Gemini drops a "Depends". Aisyah duplicates it. Nova 3 returns nothing. Every model is wrong. The Explorer lets you hear this audio and see all four outputs on one screen.

Each model has its own blind spots

Word confusion analysis (which words a model consistently substitutes) shows that errors aren't random. They're specific to each model.

• Qwen3-ASR-0.6B (official) substitutes "kanak" with "kerana" (8 times), "perkhidmatan" with "perhimpatan" (6 times), "im" with "am" (6 times). Code-switching errors.
• Qwen3-ASR-1.7B (official) replaces the English "i" with "saya" (10 times) and "aku" (5 times). It normalises informal English into formal Malay. Linguistically reasonable, factually wrong.
• Scribe v2 and Ilmu ASR v4.2 have different confusion patterns entirely, tied to their own training data.

If you're improving a model, the Explorer tells you what to fix: the "kanak" → "kerana" confusion, not "lower WER generally."

Every model deletes the same words

Across the benchmark, some words are invisible to all ASR systems:

• "ah" was deleted 3,486 times across all models
• "um" was deleted 346 times
• "di" (preposition) was deleted 174 times
• "nya" (possessive) was deleted 110 times
• "ada" (verb "to have") was deleted 115 times

These aren't filler. "di", "nya", and "ada" are grammatically meaningful. Their systematic deletion means models produce incomplete transcriptions. Click any high-deletion sample in the Explorer and you can see exactly which words vanished.

Sometimes models just produce silence

It's not just one model. Nova 3 returned blank transcriptions on 319 out of 1,899 samples — that's 16.8% of the entire benchmark. Aisyah 1.0 Flash produced 41 blanks (mostly singing). Even Scribe v2, Universal 2, and Universal 3.5 Pro each had at least one silent failure on public data.

That's not a WER problem. If you're deploying in a call centre and more than 1 in 10 calls produce nothing, that's a system failure — regardless of what the leaderboard says.

A singing sample from the benchmark. Gemini 2.5 Pro gets most words right but misses the subject "Kau". Aisyah 1.0 Flash returns nothing. Nova 3 returns nothing too — and it does this on 319 samples across every category. These silent failures are invisible in aggregate WER but critical in production.

Noise breaks some models completely

We tagged every sample with an estimated noise level (clean: SNR above 25 dB, moderate: 15–25 dB, noisy: below 15 dB). Some models degrade gracefully. Others collapse.

Nova 3 ranks near the bottom of the leaderboard already, and noise makes it worse: from 31.9% WER on clean audio to 41.0% on noisy audio, a 9.1-point drop. Other models see smaller but still real jumps under noise. If you're deploying in a noisy environment (factory floor, poor call-centre lines, outdoor fieldwork), the "best" model on clean audio isn't necessarily the best for your use case — check the noise breakdown for the one you're considering.

Some samples defeat every model

The Explorer identifies samples where every model scores WER above 50%. Extreme background noise, overlapping speakers, rapid code-switching mid-sentence, sung speech. These define the current boundary of ASR on Malay. Knowing where that boundary is matters as much as knowing who's on top.

A street interview about Malaysian movies. Every model recognises "cerita-cerita gangster" but mangles the specific film titles. "KL Gangster" (a well-known Malaysian film) becomes "Kapten Gangster", "Korean gangster", "cat gangster", and "kek ada gangster". Click play to hear the actual audio.

What you get

• Every score is auditable. Click any WER number and hear the audio that produced it.
• Error patterns, not just error rates. Word confusion pairs, deletion hotspots, hallucination behaviour.
• Noise-aware comparison. See how each model degrades without running your own evaluation.

When someone releases a benchmark, the first reaction is usually scepticism. Are the numbers real? Are the samples cherry-picked? Are their own models conveniently on top? We get it. We'd be sceptical too.

That's exactly why we built the Explorer. Every score is a link. Click it and you hear the audio, see the ground truth, and read every model's output. Nothing is hidden behind an API or a summary table. You can check. You can judge for yourself. So can anyone else.

We're not asking you to trust our numbers. We're giving you the tool to verify them.

The code is on GitHub. The data is on HuggingFace. No sign-up, no API keys, no paywall.

Links:

• Try the Explorer: Open Benchmark Explorer
• Benchmark dataset: Revolab/ASR-Benchmark-Public on HuggingFace
• Code repository: github.com/Revolab-Sdn-Bhd/revolab-asr-benchmark
• Dataset announcement: Introducing the Revolab ASR Benchmark