Auto Subtitles Are Drafts: Why 99% Accuracy Isn’t the Finish Line

In one test clip, the transcript looked almost perfect. Then I saw gp where the speaker had actually said HP. It was one token in a long transcript, and that was exactly the problem: nothing in the editor made it look more dangerous than the clean words around it.

Disclosure: AI helped me edit and structure this article. The gp / HP mistake came from my own build, and I checked the technical details against the code and the working editor.

I ran into this while building a subtitle editor. The ASR system already returned word-level timing and confidence values, but a polished block of text made every word look equally trustworthy. The model exposed uncertainty; the interface hid it.

That led me to a narrower engineering conclusion:

Auto subtitles are drafts. An accuracy score describes a model result; it does not define a finished review workflow.

Why one accuracy percentage is not a finish line

Speech-to-text systems are often evaluated with word error rate, or WER. In its simplest form:

WER = (substitutions + deletions + insertions) / reference words

That is useful for comparing transcripts against a known reference. Trouble starts when a model-level metric is turned into a product-level promise.

Suppose a 100-word transcript contains one wrong word. Its word accuracy may look excellent. But the product impact depends on the word:

• Changing “and” to “an” may be harmless.
• Changing a person’s name damages trust.
• Changing 15 to 50 changes the meaning.
• Changing HP to gp made my test caption look careless.
• Dropping “not” reverses the sentence.

WER counts errors. It does not price their consequences.

Subtitle quality also includes things that a transcript-only score does not fully describe:

• whether words appear at the right time;
• whether cue boundaries follow the sentence;
• whether a line is readable before it disappears;
• whether punctuation helps or hurts comprehension;
• whether the user knows what remains unchecked.

A system can therefore have a strong aggregate score and still produce a bad editing experience.

Confidence is metadata, not a workflow

Word-level confidence is more actionable than a single document score. It can help answer a useful question: where should a reviewer look first?

But a confidence value is still model output, not automatically a calibrated probability. Research on ASR word confidence proposes explicit score calibration so values can be compared across models. A high-confidence token can still be wrong. A missing confidence value should not silently become “high confidence.”

When an ASR response enters the app, I require every word to have text, timing, and a finite confidence value. I then turn each word into an editor word that also records where it came from:

type EditorWord = {
  id: string;
  text: string;
  startMs: number;
  endMs: number;
  confidence?: number;
  source: 'asr' | 'manual';
};

The optional confidence field is deliberate at the editor layer. ASR words arrive with a real score; a manually created word should not inherit a score the model never produced.

The two-step contract matters more than the TypeScript: the ASR response must contain valid evidence, and the editor must preserve it while recording whether a word came from ASR or a manual edit. If the response is malformed, the app should reject it at the input instead of inventing certainty later.

Even with that contract, rendering every confidence value is not a review workflow. A transcript with hundreds of colored words simply replaces one haystack with another.

Turn uncertainty into a finite review queue

I found it more useful to separate three things:

1. The transcript is the editable content.
2. Review targets are the subset the system asks a human to inspect.
3. Resolved targets record progress through that finite set.

A public version of the state only needs two lists:

type ReviewState = {
  candidateIds: string[];
  reviewedIds: string[];
};

function pendingIds(state: ReviewState): string[] {
  const reviewed = new Set(state.reviewedIds);
  return state.candidateIds.filter((id) => !reviewed.has(id));
}

candidateIds defines the finite review job. reviewedIds records the work the user has already done. Pending work is simply the difference between them.

There is no universal recipe for choosing review targets. In my editor, tokens are ranked by confidence and time, and the queue is deliberately bounded. If every below-threshold word were surfaced without a cap, noisy audio could create a review experience almost as expensive as rereading the whole transcript.

I do not think there is a universal confidence threshold. A sensible policy depends on the ASR system, language, audio, domain vocabulary, and the cost of a miss. The threshold should be measured against your own material, not copied from a code sample.

The resulting pipeline is straightforward:

word-level ASR result
-> validate the ASR result
-> build timed subtitle cues
-> rank review candidates
-> create a finite review queue
-> accept or edit each target
-> export

The value is that uncertainty becomes a job the user can finish. Instead of asking the user to “check everything,” the interface offers a bounded task with visible progress.

Define “All clear” precisely

The phrase “All clear” is easy to overstate. It should not mean:

• the transcript is objectively perfect;
• the model achieved a guaranteed accuracy level;
• no unflagged error exists;
• the subtitles satisfy legal, medical, or accessibility review requirements.

In this workflow, it means something narrower and testable:

Every item in the current review queue has been accepted or edited.

The user can accept a flagged word when it is already correct, or edit the cue when it is not. Accepting moves that word into the checked list. Editing changes the cue and drops target IDs that no longer point to its current words. Re-transcription creates a new transcript, so it must create a new review list too.

That last rule matters. Review state is derived from a particular transcript. If the transcript changes but the old “resolved” state survives, the UI can display completion for words the user has never seen.

“All clear” is therefore a workflow state, not a quality certificate.

Failure modes worth designing for

A high-confidence mistake

Names, acronyms, numbers, and domain terms can be wrong even when the model is confident. A confidence queue reduces search cost; it does not remove the need for a final watch when the content matters.

Missing or malformed confidence

Treating absent confidence as 1.0 turns missing evidence into certainty. In this implementation, the ASR response is rejected. Another system could define an explicit fallback, but it must not treat missing confidence as high confidence.

Stale review decisions

After cue editing or range re-transcription, old token IDs may no longer exist. Prune both pending and resolved IDs against the current transcript before showing completion.

Too many warnings

A review queue that flags half the transcript is not a shortcut. Bound the queue, expose why an item was selected, and provide a way to review the full text when the audio is genuinely difficult.

Source confidence leaking into translated text

If the editor has original and translated subtitle tracks, do not paint source confidence onto translated text. The uncertainty belongs to the source ASR result. A translated track can link back to unresolved source words without pretending those English token scores describe the translation.

Where this approach fits—and where it does not

A finite uncertainty queue works well for creator videos, interviews, courses, and other workflows where a person can quickly confirm a small set of likely problems.

It is not enough by itself for high-risk transcription. Legal, medical, safety, and formal accessibility workflows may require full human review, speaker verification, domain-specific checks, audit trails, and standards beyond a general-purpose creator tool.

The right claim is not “human review is no longer needed.” It is “the system can make human review smaller and more explicit.”

A builder checklist

If you are turning ASR output into an editable product, I would check these before calling the workflow complete:

• Preserve word text, timing, confidence, and source.
• Reject malformed ASR responses when they enter the app.
• Keep transcript content separate from review progress.
• Rank uncertainty instead of painting the whole transcript.
• Bound the queue so that review remains finite.
• Let users accept a correct token without editing it.
• Invalidate stale review state after editing or re-transcription.
• Keep source-word confidence out of translated tracks.
• Define exactly what your completion label means.
• Never present queue completion as guaranteed transcript accuracy.

I built this review queue into SubtitleGenerator after seeing how easy it was to miss one wrong word in an otherwise clean transcript. Uncertain source words become a finite queue, and “All clear” appears only when that queue is empty.

How do you handle ASR uncertainty in your own products: a confidence threshold, a full transcript review, or a separate review state?