Speech-to-text can't tell you if an AI voice said a name right

We make short, AI-voiced lessons. Before a video ships, every sentence goes through automatic checks, and then a person listens. The automatic checks kept failing in one place: names.

This post explains why a transcript can't catch a mispronounced name, and the three-part check we built instead. All numbers are our own measurements. The code is open source and linked at the end.

The problem: a transcript hears what it expects

The usual way to check a voice-over is to transcribe it and compare the transcript with the script. That works for missing or swapped words. For names it fails in both directions.

• A wrong read can pass. A speech-to-text model is trained to write likely text. When a voice gets a name almost right, the model often writes the correct spelling anyway, and the mistake disappears from the transcript.
• A right read can fail. When the model doesn't know a name, it spells it some other way, even when the voice said it perfectly.

Our old check did exactly this. On 32 test reads of 8 hard names, it passed only 8 of the 22 correct reads. It did fail all 10 wrong reads, but only because it failed almost every name. It couldn't tell a right read from a wrong one.

Three ears instead of one

Every take now goes through three listeners, each built differently:

1. Words. A speech-to-text model transcribes the take, and we compare the transcript with the script. This catches missing, extra or wrong words.
2. Letters, with no language model. A second recognizer writes what it hears letter by letter, with no idea which words are likely. It catches the slips the first one smooths over, small dropped words ("from zero [to] two hundred") and takes that cut off early.
3. Sounds. A third model writes the phones it hears, in IPA. For every hard word we keep a researched pronunciation, also in IPA, and compare the two sound by sound.

Here is one real failing take. The name should sound like /ʃ ɪ v ɔ n/. The voice said /j ɑ b ə n/. Only the third ear sees that difference. The old check gave this take the same verdict as a correct read.

Comparing sounds fairly

A plain phone error rate counts every difference the same. That is too blunt: some differences are an accent, others make a different word.

So the comparison weighs each substitution by phonetic features (close sounds cost less than distant ones) and tolerates accent-level variation. The pass/fail rule is still strict: one clearly wrong sound fails the word. A false alarm costs one re-take; a misread name that ships costs more.

The target matters as much as the check

A sound-by-sound check is only as good as the pronunciation it compares against. So each hard word gets a pronunciation with a source, such as an encyclopedia's IPA, a dictionary entry or a native recording. It goes into a small lexicon file, together with the respelling the TTS engine reads.

The workflow becomes: find the risky words in the script, research each one with a source, write it in the TTS engine's syntax, record, then check every take.

Results

On our own takes. 8 hard names, each read by 2 voices, once from a researched respelling and once from the raw spelling: 32 reads, 22 right and 10 wrong.

• Correct reads that pass: 8 of 22 before, 22 of 22 after
• Wrong reads caught: 10 of 10, before and after

One honest caveat: these 32 takes are where we set the thresholds, so treat this as a fit, not a test.

On a held-out benchmark. There is no public benchmark for TTS name pronunciation. The closest public test of the core step (did this word sound like its target phones?) is a dataset of 2,500 recordings, 15,967 words, read by non-native English speakers and scored word by word by five experts. We ran the sound check on it with no training on the dataset.

• Spotting a word with an incorrect phone: ROC-AUC 0.843, against 0.788 for a plain phone error rate
• On words of two or more syllables, where names live: 0.858, against 0.838
• Strict by design: on learners' speech it flags 27% of words and catches 78% of the words the experts marked wrong. TTS voices make far fewer accent-level slips than learners do.

Speed. Everything runs on a laptop CPU, with about 3.5 GB of models. With the models kept loaded, a short take is checked in 3 to 4 seconds.

What it doesn't do yet

• The name results come from our own takes, because no public benchmark for TTS names exists yet.
• The sound check needs a researched pronunciation. A word that isn't in the lexicon only gets the word and letter checks.
• It doesn't replace a person. We run three checks that fail in different ways, keep a record of every failure, and someone still listens before anything ships.

The code

It's open source under the MIT licence, runs locally and sends no data anywhere. It works as a command-line tool, or as a skill an AI coding agent can follow:

https://github.com/logitsolutionsai/say-it-right

This post was written with AI assistance. The measurements are ours, and the scripts that produce them are in the repository.