What generating TTS for six languages taught me

Notes from building Lingato's audio · 23 August 2026

Every word in Lingato is pre-generated speech, served as a static clip. I wanted one consistent, tested voice per language rather than whatever speech-synthesis voice a visitor's device happens to ship — which vary wildly, and on some phones simply don't exist for Polish or Norwegian at all.

That turned into about a month of work, most of it spent discovering that my intuitions about generated speech were wrong. These are the four that cost the most time. They are all variations on one theme: the checks you would naturally write do not detect the failures that actually happen.

1. A bad clip is not a broken clip — it is a confident, well-formed, wrong word

The model I use for most languages is autoregressive and was trained on sentences, so an isolated word does not reliably tell it to stop. My first guards were the obvious ones: reject a clip that runs too long for its text, reject one that clips or peaks wrong.

Those guards work, and they catch nothing important. They cannot see a clip that is exactly the right length, perfectly clean, and simply says a different word. Polish wschód came back as "Cool". English arm came back as "Aum".

This failure is invisible to everything downstream. The file exists, so the missing-clips list is empty. Nothing errored, so no log mentions it. The review queue stays quiet. The only thing that detects it is a human listening to every clip, which does not scale past a few hundred words.

The fix was to put an ASR gate in the loop: generate, transcribe the result with Whisper, compare it to the text it was supposed to say, throw away the failures and regenerate. Measured on the live deck:

I re-ran the comparison on CPU rather than GPU to rule out a hardware artefact. Identical. The good batches already in my deck had not come from a better model or better prompting — they were the same voice, gated.

The first properly gated run failed all seven Spanish words in the batch, and deserved to:

2. The gate is confidently wrong in ways specific to each language

Having been saved by the ASR gate, I trusted it, which was the next mistake. An automated judge has its own per-language failure modes, and they do not announce themselves either.

Norwegian is now excluded from the gate entirely. Whisper transcribes Norwegian phonetically — Dypt vann comes back as "Dipptvamm" — so it fails clips that are perfectly good. Two gate-driven runs churned 576 clips that I then confirmed by ear were fine. The gate was not improving that language, it was corroding it.

Polish is full of sub-second words, and Whisper hallucinates whole phrases on those. The word ty was transcribed as "Dzień dobry" — a greeting that shares nothing with it. On very short clips a persistent failure is more often a hallucination than a bad clip, so those now ship and get flagged for a human instead of being regenerated forever.

A similarity ratio is blind to length. A clip that says the right word and then keeps talking scores well. English courier → "Korea, back to the dull." scored 0.50 and shipped. That is now its own rule: a transcript more than about twice the length of the target is a bad clip whatever its ratio.

And an empty transcript has to be an explicit failure. It means silence — and it otherwise matches everything, because an empty string is trivially similar to nothing.

Model size mattered more than expected, too. Whisper base called roughly half its Norwegian failures wrongly; small is the floor for the smaller languages.

3. Different languages need different engines, for reasons that are not about quality

I assumed engine choice was a quality ranking — find the best one, use it everywhere. It is not. It is a question of which linguistic feature the model is structurally capable of representing.

Russian: stress is phonemic

In Russian, stress placement changes which word you have said. The multilingual model I use elsewhere has no lexical stress mechanism at all — not a bad one, none — and measured around 14% wrong against the ASR gate. No amount of tuning fixes a missing feature, so Russian runs on a different engine with an explicit accentuation step in front of it.

Polish: the same argument, reversed

Polish stress is rigidly rule-based: penultimate syllable, with a small set of known exceptions. One of the engines available to me resolves phonemes through espeak-ng before the model runs, which means it cannot get Polish stress wrong. Provably correct, by construction. The neural model guesses.

Polish was also my worst language by rejection rate — roughly one clip in seven rejected by ear — so re-voicing it with the phoneme-grounded engine was the obvious move. I did it for 106 flagged clips. It worked exactly as designed, and the stress was flawless.

Then I ran a blind five-way comparison: three phoneme-grounded voices, the stock neural voice, and the neural model zero-shot cloning my own voice from the 527 recordings I had made as training data.

This is the result I am still least comfortable with, and the one I would most like to be argued out of. The measurably-correct option lost to the subjectively-pleasant one, on my own ear, using exactly the methodology this whole page is about distrusting.

4. A missing clip is not silence — it is a different voice

English is served from a variant deck built by cloning a public-domain LibriVox reader, with per-word fallback to the base deck when the variant lacks a clip. I had reasoned about that fallback as a graceful degradation: worst case, a word is missing and the browser's own speech synthesis handles it.

That is not what happens. The base deck has a clip, and the base deck's built-in English speaker is American. So every English word added after the variant was built arrived silently in the wrong accent, mid-deck, next to words in the right one.

More than half the English texts were affected. Every automated check reported green throughout: the clip existed, so nothing was missing, and the ASR gate scored English 96% — because the gate transcribes words and is completely deaf to who is speaking. The score was true and entirely irrelevant.

The fix had to be structural rather than a new check. The reference recording is now applied automatically per language, and a missing reference file is a hard error. The engine can no longer quietly fall back to its default speaker, because a wrong voice is invisible to every other guard in the system.

The only lever for a mispronunciation is spelling

There is no way to tell a neural TTS model "these letters are Norwegian". Words that look English get read as English: fundamental came back with an English /y/ — "Fynda mental" — in four takes out of four.

The workaround is to respell what you feed the engine while the displayed word and the file's identity stay correct. The learner sees fundamental; the model is fed fondamental.

It has to be verified one word at a time, because it is just as easy to make things worse. About half the respellings I tried on Norwegian were reverted after measurement — hår → "haar" moved the vowel far enough to produce har, a different word entirely. English desert is a heteronym and kept coming back as the dessert; six candidate respellings at three takes each, and "dezzert" was the only one Whisper heard correctly every time.

A useful signal fell out of this: a bad roll varies, a misread spelling does not. If a word fails differently each take, generate again. If it fails identically every take, the spelling has beaten the model and more takes will not help.

What I would tell myself at the start

• Your guards detect malformed output. Almost none of the real failures are malformed.
• Add an automated listener early — then find out how it is wrong per language, because it will be.
• An eval that scores well can be measuring something you do not care about. English scored 96% while more than half of it was in the wrong voice.
• Absence is not the dangerous state. A confident, well-formed substitute is.
• Pick the engine by which feature the language needs, not by which sounds best in a demo.

Lingato is a free vocabulary trainer for English, Spanish, Italian, Polish, Norwegian and Russian — no signup required to try it, no ads, no tracking beyond an anonymous page counter. If you speak any of these languages natively and something sounds wrong, I would genuinely like to hear about it.