How do you know a dubbed video is good? I built an open-source tool to check

If you use AI dubbing, you've probably done this: a dub comes back, and the only way to know whether it's any good is to watch all of it. That works for one video. It stops working at twenty videos in five languages.

So I built DubEval, a free, open-source desktop app that compares a dubbed video with its original and tells you whether the dub is ready to ship.

Reviewing a dub. The film is Tears of Steel (© Blender Foundation, CC BY 3.0).

What it does

You give it the original and the dub, from any service or from voice actors. You get:

• A verdict: Good, Needs review or Poor. Every row says what was measured and how it's graded, so there's no single black-box score.
• Every problem with a time: long silences, loudness jumps, noise, a stretch left in the original language, lines that run long, mistranslations, changed numbers. Click one and it plays.
• A recommendation: give it 2 to 8 dubs of one video and it ranks them with a fixed, printed rule, then says which one to deliver and why.

How it works

The checks run on your computer, and your videos never leave it:

• Same video, anything missing? Picture fingerprints, about 10 frames a second.
• Where is someone speaking? The Silero voice detector.
• What did they say, and when? Whisper, with a time for every word.
• Does the voice sound damaged or different? DNSMOS for quality, WeSpeaker for similarity.
• Loudness, silence, noise, sound in sync: plain signal measurements.
• Does the translation mean the same? Your own Claude, Gemini or ChatGPT key, which sends only the transcripts, or a local model. The local model is much weaker: it found 4 of 23 planted translation mistakes, and Gemini found all 23.

How I test it

A QA tool nobody has checked isn't worth much, so I test it two ways.

Planted problems. I take real dubs and put known problems in them: a silence, the original voice back in, a loudness jump, distortion, noise, a cut, the sound 1 second late. Then I count. It catches 102 of 112, and 18 of 24 on dubs it was never tuned on. It also raised 42 false alarms across those copies.

A person's grades. One reviewer graded 20 real dubs before seeing DubEval's results. It agreed on 12 to 15 of 20, and most of that gap comes from which AI model reviewed the translation.

That's a small test set with one grader, so use the verdict to decide where to look first, and still watch the dub.

Four things that surprised me

1. A well-known speech quality model was useless on real dubs. The first one I tried, UTMOS, rated clean synthetic speech well but scored every real recording around 1.2 to 1.5 out of 5, originals included. It couldn't tell a good dub from a bad one. I switched to DNSMOS and score the dub against the original at the same moment, because background music lowers both tracks alike.

2. Turning the volume down looked like a worse voice. One test copy is only 3 dB quieter, so anything flagged on it is wrong by definition. The voice score moved by up to 1.14 out of 5 from volume alone. Scoring the dub at the original's volume cut that to 0.51.

3. Numbers don't translate digit by digit. My numbers check would have flagged "July 4" → "7월 4일" because of the 7, and "2 million" → "200만" because of the 200. Now it flags only a number replaced by another, and compares significant digits. It still caught all 6 numbers I planted, but it misses a number wrong only by its zeros, like 5 and 50.

4. The same AI model doesn't always give the same answer. On identical transcripts with the same model, the verdict flipped on 1 of 6 dubs: one line scored 5 out of 10 in one review and 1 in the other. So you can lock a run to one model, and a comparison warns when different models reviewed the dubs.

What it can't do yet

• The installer is Windows only and isn't code-signed, so SmartScreen warns. macOS and Linux run from source.
• Speech recognition can't read some languages. On Uzbek, the line-level checks don't run, and DubEval says so instead of guessing.
• It misses hum, buzz and crackle, and noise shorter than 1.5 seconds.
• Lip sync is shown for information only.

Try it

• Download for Windows
• Source on GitHub (AGPL-3.0)

It needs no account. Dubs where it gets things wrong would help me most, so please open an issue.