A Breakdown of Failure Modes in Autoregressive TTS
Orpheus loses its place on identical words, and the voice either repeats or skips with it. We traced the failure to three attention heads that keep its place in the text.
We kept hitting one bug in production. Our text normaliser sends OTPs and account numbers to the model as words, and the model reads "your verification code is five five five five five five" with one five too many, or one too few.
We tested it on Orpheus 3B, an open TTS model built on Llama that writes 7 audio tokens for every 85 ms of speech. Same digit repeated k times against k different digits, 32 clips each, plus a blind sample we counted by ear:
| "Five" repeated | Read exactly right | By ear |
|---|---|---|
| 4x | 100% | 4 of 4 |
| 5x | 94% | 4 of 4 |
| 6x | 50% | 2 of 4 |
| 8x | 6% | 2 of 5 |
| 8–12 different digits | 88–100% | 2 of 2 |
Listen
Different digits of the same length come out fine, so the repeats are what break it. We wanted to know what the model does inside when it loses count.
Three heads keep the model's place
To read aloud, the model has to know which word it is on. Orpheus has 672 attention heads, and while it writes audio, three of them in layer 7 follow the text: head 16 looks at the word being spoken (within one word 99% of the time), and heads 4 and 5 look one and two words ahead. We call them the reading heads.
Switching them off on 14 sentences Orpheus normally reads perfectly:
| Switched off | Clips that failed |
|---|---|
| The 3 reading heads | 99.6% |
| 3 random heads in layer 7 | 0.9% |
The voice keeps going, fluent and confident, and stops following the text.
Listen
Repeats and skips are the heads slipping
In a normal run the reading heads step through the sentence one word at a time. In "red lorry, yellow lorry" they reach the last "yellow lorry" and stay there while the voice keeps saying it. In "had had had had had had a better effect on the teacher" they sweep past all six "had"s in about a second, and the voice skips with them.
Normal run: one step per word
Repetition: heads stay on the last "yellow lorry"
Skipping: heads sweep past the six "had"s
Press play and watch where the heads are as the voice speaks:
To check that the voice follows the heads, and not the other way round, we moved them ourselves. Once they reached "present" in "our team will present the quarterly results to the board next Thursday afternoon", we biased their attention to stay on "present", or to jump to "Thursday":
| What we did (16 clips each) | "Present" repeated | Words skipped |
|---|---|---|
| Nothing | 0% | 0% |
| Held the reading heads on "present" | 100% | Most of the sentence |
| Pushed them to "Thursday" | 0% | 27% of the words in between |
| Held or pushed 3 random heads | 0% | 0% |
Listen
The voice goes where the heads go. That is also why every fix we tried for repeats caused skips: blocking the heads from looking back cut repeats from 40% to 26% and raised skips from 24% to 58%.
On a run of fives the heads give up
With different digits the reading heads walk one step per digit. With identical fives they cross the whole code in under a second and sit on the last five while the voice is still counting.
Control: 8 different digits, read correctly
Correct: 8 fives written, 8 said
Too many: 8 fives written, 10 heard
Skip: 6 fives written, 5 said
On a code of 8 fives the heads are parked on the last word for 58% of the time the voice spends on the code, against 35% for 8 different digits. Every five looks the same to them, so for the last few fives the text can't tell the voice where it is, and it counts from the audio it has already made. That works up to about five.
Nothing we tried at inference time made the count more exact:
| What we tried | Codes read exactly right |
|---|---|
| Nothing | 41% |
| Steering the hidden state away from repetition | 39–40% |
| Sharper reading heads | 35–41% |
| An alignment guard like Chatterbox's | No change |
| Repetition penalty off (at 6 fives) | 53% vs 50% |
What we would change
The model probably rarely saw long runs of the same word in training, so we would fix it in the data: codes with runs of the same digit at every length from 4 to 12, training transcripts that go through the same normaliser as inference, and repeated runs checked by ear.
Orpheus loses its place on identical words, and the voice either repeats or skips with it. Malay reduplication (kanak-kanak, rama-rama), laughter and lists repeat words too, so they belong in a TTS test set next to the digit runs.
You can rerun the main tests in our Colab notebook on a free GPU in about 25 minutes.
Hear Malay-first text to speech
Nada voices switch between Malay and English in the middle of a sentence. Compare them blind in the TTS Arena, or build with the Malaysian text to speech API.
Get an API key →