Introducing the Malaysian Speech Benchmark Dataset

Most global speech recognition benchmarks have no category for a Malaysian call centre, because nobody built one. Vendors publish a word error rate that is technically true for the conditions it was measured on, and those conditions are almost always clean, single-language, well-recorded speech. They are not a Bahasa Malaysia and English conversation running over a compressed 8 kHz phone line with a call centre floor in the background.

We hit that gap directly while building voice AI for Malaysian customers. Models that looked state of the art on paper fell apart on real calls, and no public benchmark would have warned us. So we built one. Today we are releasing the Revolab ASR Benchmark, a human-annotated evaluation dataset for Malaysian Malay speech,

What the dataset is

The benchmark is a human-reviewed evaluation set for Malaysian Malay speech recognition, published as two companion datasets on HuggingFace. The public split is there so anyone can download real Malaysian audio and run their own evaluation. The private split stays with us, and produces the numbers on our leaderboard, which we are launching shortly.

Every row in both datasets has the same shape:

{
  "audio": <waveform>,
  "text": "Yes, my unit number is 2105.",
  "normalized_text": "Yes, my unit number is two one zero five.",
  "category": "telephony",
  "canonical": {"yes": ["ya", "yah", "yeah", "yep"]}
}

Audio, a human-verified transcript, a normalised version of that transcript, the domain it came from, and a map of word variants that should score as correct. Nothing hidden, including on the private side. Anyone with access can see exactly how a score was produced, even without being able to download the audio.

How we built it

Two of the twelve domains, FLEURS and Common Voice, come from existing multilingual corpora. Telephony and short inputs were each built differently for reasons specific to them. The remaining eight do not exist anywhere as ready-made Malaysian Malay datasets, so we built them domain by domain:

1. Pick the target domain first. We decided upfront which conditions needed stress-testing, then went looking for audio that fit, not the other way around.
2. Collect audio for that domain: publicly available Malaysian-language audio that matched it, with parliamentary proceedings for parliament, broadcast media for news, podcast recordings for podcast.
3. Sample randomly. Short clips drawn at random rather than hand-picked for clean audio. A benchmark built from cherry-picked recordings measures nothing useful.
4. Label against a written guideline. Every clip transcribed by a human following a shared style guide covering spelling, code-switched words, abbreviations and punctuation. Anything the annotator flagged was rechecked by a second reviewer.
5. Split public and private. Each labelled domain was divided into a public portion and a larger held-back portion that scores the leaderboard.

The two domains that broke the pipeline

Telephony could not follow that process, and the reason is privacy. The private split's telephony samples are real production call centre recordings, the actual calls our voice AI work is built around, which is exactly why they cannot be released. For the public split we had people role-play the same kind of support conversations, then ran those recordings through the same telephony compression as the real calls. Same acoustic conditions, same domain, not a single real customer conversation in the public release.

Short inputs is different again. Rather than collecting single words from existing recordings, we generated it with text to speech: the one-breath account confirmations, digits and command-style utterances a real caller produces. Because the audio is generated from the text, the reference is known by construction, with no manual transcription step.

Why these twelve domains

None were picked at random. Each one stresses a different failure mode we have seen in production speech systems.

Languages are Bahasa Malaysia and English, including the code-switching between them that is completely ordinary in everyday Malaysian speech. Audio is 16 kHz in the public split and 24 kHz in the private split.

Two transcripts per row, and why

Each row carries the transcript twice. text is what the annotator wrote verbatim, with numerals and currency in their natural written form. normalized_text renders the same utterance the way it was actually spoken.

This matters most in telephony, where callers read out account numbers, IC numbers and reference codes digit by digit:

A model that transcribes "102105" as "one zero two one zero five" scores correctly against the normalised reference instead of being penalised for not guessing the exact numeral formatting an annotator happened to type. Both fields ship in every row, so whoever is scoring picks the comparison that suits their use case.

Canonical variants

Every row also carries a canonical field: a map from each reference word to the other spellings that mean the same thing and should score as correct. A caller who says "okay" might be transcribed "ok", "oke" or "okey", and all of them are the same word.

This comes up constantly in Malaysian speech. English loanwords with several accepted spellings ("hello" and "helo"), Malaysian versus Indonesian pairs ("kerana" and "karena"), colloquial contractions ("ni" and "ini"), and filler words transcribed inconsistently ("ah", "aa", "err") all appear in the canonical map. Without it, a model that hears the word correctly but prefers a different accepted spelling is penalised for something that is not an error.

Why the split is deliberately lopsided

We put more samples in the private split than the public one. Leaderboard integrity matters more to us than public convenience: a larger hidden evaluation set gives a more reliable estimate of real-world performance and makes overfitting to the test set substantially harder. The public split exists so the community is not left guessing. It is real, representative audio, just not the exact set we score against.

Limitations

We would rather say this plainly than let someone discover it the hard way.

• Covers Malaysian Malay and English code-switching only. No other regional languages yet.
• Accent coverage reflects what we could source from public audio, not an exhaustive survey of every Malaysian dialect.
• 4.08 hours is enough to evaluate against, not to train a model from scratch.
• Not a training corpus. The private split's licence explicitly forbids it, precisely so it stays trustworthy as an evaluation set.
• Twelve domains is broad, not exhaustive. It will not cover every deployment scenario.
• Short inputs is text to speech, not real recorded speech. Useful for testing hallucination on brief utterances, not a substitute for how short utterances actually sound.
• Public telephony is role-played. We matched the acoustic conditions closely, but the private split’s real call centre recordings remain the more faithful measure.

What we release

The Revolab ASR Benchmark is published as an annotation layer. Every transcript, normalised transcript, canonical variant map and domain label was produced by Revolab annotators working against a written guideline, with flagged samples rechecked by a second reviewer. That layer is released under CC-BY-4.0.

For the collected domains, the underlying recordings remain subject to their original terms, and the CC-BY-4.0 grant covers the annotations. Rights-holder enquiries can be sent to support@revolab.ai and we will action requests promptly.

Provenance varies by domain, so it is worth being specific. FLEURS (Google Research, CC-BY-4.0) and Common Voice (Mozilla Foundation) carry their own upstream terms, and those terms travel with the data. Two domains are Revolab-created rather than collected: the public telephony split is role-played by people we engaged, and short inputs is generated with text to speech.

What we hope happens next

We do not think of this as a finished artifact. Our hope is that it becomes something other teams working on speech AI in Southeast Asia can lean on, a shared and honest reference for what Malaysian speech actually sounds like, instead of everyone separately discovering the same gap the hard way.

We would like to see both datasets grow, with more domains, more hours, and more of the messy, code-switched, imperfect audio that global benchmarks leave out. And we would like to see the same public and private approach applied to the other languages of this region, because the underlying problem is not unique to Malaysia.

Good benchmarks shape research. By making Malaysian speech measurable under realistic conditions, we hope future systems get optimised for the conversations people actually have, not the recordings that happen to be easiest to recognise.

Sources and attribution

Eight of the twelve domains were collected, sampled and manually labelled by Revolab for this benchmark. Telephony and short inputs were built as described above. The remaining two are existing corpora we are grateful to build on: FLEURS from Google Research (CC-BY-4.0), included so results stay comparable with the wider literature, and Common Voice from the Mozilla Foundation (CC0), whose volunteer recordings contribute speaker diversity our own collected domains do not capture. Full attribution and licensing detail is on each dataset card.