How We Built Speech Recognition for Malaysia

Some of the world's best speech recognition models can't reliably catch a customer's account number over a Malaysian phone line. The customer was clear and the audio was clean. Malaysian phone calls just sound nothing like what those models were trained on.

We ran into this building RevoCall, our voice agent for Malaysian customer calls: conversations that code-switch between Bahasa Malaysia and English, pivoting to English for a technical term and back again mid-sentence, over a compressed phone line with a call-centre floor humming behind it. Everything the agent does next, pulling up the account, taking the action, rests on that transcript being right.

So we built our own model for it, and the Revolab Speech API on top. The theory took about a week to understand. The rest of the year went to things no course mentioned.

First, just listen

Three clips from our benchmark, nothing unusual by our standards. Play them before you read the captions, and see whether you'd trust a transcript of any of them.

The first one is 8kHz, compressed by the phone network, which throws away most of the frequency range separating a "4" from a "7". If you've only ever tested on clean read speech, that clip is a different problem class than the one you've been solving.

How speech to text works: the one idea worth knowing

Here's the problem every speech model has to solve before it can do anything useful.

Take a three-second recording of someone saying "apple", chopped into 300 slices of ten milliseconds each. Which slice holds the "a"? Where does the "l" start?

You can't say. Neither can a human annotator, not reliably. The transcript tells you what was said and nothing about when. And you can't pay your way out of it either, labelling every sound boundary by hand costs more than the dataset is worth.

So the trick is to stop guessing the alignment. Train the model by asking whether any valid alignment produces the right transcript, and reward all of them.

This has a name: Connectionist Temporal Classification, or CTC, introduced by Graves et al. in 2006. It sums the probability over every alignment that produces the target transcript instead of committing to one, and maximises that total. It's why large-scale ASR became practical instead of staying stuck behind frame-by-frame human labelling, and some version of it sits underneath most non-autoregressive speech models trained today.

That's the piece of theory we'd insist a developer understand. Everything else you can look up when you need it.

One more, because it explains a whole class of errors you'll see early on. No amount of better hearing fixes a homophone. On a real call, "check" and "Czech" are acoustically identical, and only what came before the word separates them. That's why systems pair the audio model with a text-only model at decoding time.

Three things no course told us

1. Read your errors, not your word error rate

Two models can post the same WER and be broken in completely different ways. One drops a word here and there across every clip. The other is perfect on nineteen clips out of twenty and catastrophically wrong on the last one. Same number, different problem, different fix. The average hides which one you have.

So we stopped reading the score and started grouping errors by shape. Some patterns turned out to be systematic enough that we wrote detectors for them and now run them over every model we evaluate, ours included:

• Reduplication merges: Malay marks plurals by repeating the word, orang-orang, cerita-cerita. Models trained mostly on English collapse the pair back into one word, every time.
• Filler deletions: the ah, aa, haa that carry turn-taking on a call get quietly dropped.
• Repetition hallucination: the model gets stuck and emits the same word several times over.
• Script mismatch: the model answers Malay audio in an entirely different script.
• Empty output: the model returns nothing at all, and the whole clip counts as deletions.

None of these show up in an aggregate score, and each one points somewhere different. Reduplication merging is a tokenizer and training-data problem. Filler deletion is a labelling-convention problem. Repetition is a decoding problem. If all you have is "WER went up 1.4%", you have no idea which of those to go and fix.

If you can't point at the clip, you don't know what changed.

2. Some errors end the call

In a normal conversation, one wrong word barely matters. On a support call, the words carrying the actual information are exactly the ones a general model has seen least: account numbers, IC numbers, addresses, Malaysian names spelled out letter by letter.

Get one digit wrong and nothing after it is recoverable. The wrong account gets pulled up, and every step of the call is built on top of that. Meanwhile the overall score barely moves, because it's one word out of forty.

WER weights every word equally. Your customer does not.

So we read digit strings and names separately from the running text. A model that handles conversation well and gets shaky on account numbers is not a model we can put on a live call, however good its headline number looks.

3. Short audio is where models invent things

We expected a badly matched model to produce wrong words. We did not expect it to stop transcribing altogether.

Here is Gemini 3.6 Flash on a 1.2-second telephony clip of a customer saying "Okay.":

Forty-one inserted words against a one-word reference, which scores as a WER of 4,200%. The model didn't mishear the audio. It concluded there wasn't any, and replied as a chatbot instead.

It isn't a one-off either. The same shape of reply comes back on "22", on "tak payah", on "wait 5 minute", all of them under a second. A model is never given a cheap way to say "nothing" or "I'm not sure", so on input this short it reaches for the most probable text it knows, which for a chat-tuned model is an apology asking you to upload a file. Abstaining wasn't in its training data, so it isn't in its behaviour.

Short turns aren't an edge case on a support call. "Ya." "Okay." "Tak boleh." A digit repeated back to confirm. They're short precisely because they're routine, and they're most of a call. If your eval set is mostly full sentences, this failure is invisible to you and very visible to your customer.

What we shipped: the Aisyah speech to text API

All of this went into the model behind RevoCall, our voice agent for Malaysian customer calls. The same model is available directly as the Revolab Speech API: speech-to-text, plus text-to-speech for the other half of the call, built for compressed telephony audio rather than podcast audio.

Aisyah 1.0 Pro does not win everywhere. On the 820 public clips alone, Scribe v2 edges ahead (4.79% against 4.89%). Gemini 2.5 Pro is better on news and parliament recordings, and both Gemini 2.5 Pro and Scribe v2 beat it on street interviews, singing, and the open Common Voice and FLEURS sets. Where it leads is the audio a support call is made of: short turns and compressed telephony.

curl -X POST https://api.revolab.ai/v1/stt \
  -H "Authorization: Bearer $REVOLAB_API_KEY" \
  -F "file=@call.wav" \
  -F "model=aisyah-1.0-pro" \
  -F "language=ms"

{
  "text": "Boleh saya semak baki akaun, nombor satu kosong empat tiga dua.",
  "language": "ms",
  "duration_s": 3.8,
  "confidence": 0.96,
  "latency_ms": 430
}

If you want to check any of this rather than take our word for it, the benchmark dataset covers how we built the evaluation set, and the Malaysian ASR leaderboard lets you play any clip and read what every model we tested returned for it.

Next up: Bahasa Indonesia and Malaysian Chinese. Same gap in both.

If you're building speech for a language the big labs treat as an afterthought, start by opening the clips you fail on and listening to them. We should have started there.

Try it:

• API docs: docs.revolab.ai
• Malaysian ASR leaderboard: open the leaderboard
• Website: revolab.ai

Frequently asked questions