Your Transcript Has a Sentence Nobody Said During the Silence

You built the support tool
in an afternoon.

Every call recorded,
every recording sent through
an open speech recognition model,
every transcript searchable.

The team loved it.

Then a supervisor
read a transcript back to a customer,
during a complaint,
and the customer said,
I never said that.

She was right.

The recording had
eleven seconds of silence
while she looked for her account number.

The transcript had a sentence there.

Polite.

Fluent.

Invented.

Here is the mechanism.

A model like Whisper
does not detect speech
and then write it down.

It is trained to produce
the text that most likely goes
with a piece of audio.

It learned from a huge amount
of audio paired with text
taken from the web,
including subtitles
where the quiet end of a video
was paired with a credit line
or a thank you for watching.

So when the audio is silence,
hold music,
or a keyboard,
the model still writes something.

It is doing what it was trained to do.

Researchers who studied this
found invented phrases
in roughly one transcript in a hundred,
and some of them
said things that were never close
to what was spoken.

Nothing in the text
tells you which parts are real.

The invented sentence
has the same punctuation
as the true ones.

So stop trusting the text alone.

Run voice activity detection first,
and only send the parts
where somebody is actually talking.

Keep the per segment numbers
the model gives you,
how likely it thinks
there was no speech,
how confident it was in the words,
and flag the segments that look weak.

Keep the audio,
linked to the timestamps,
so anyone reading a line
can play the line.

And never let a transcript
be quoted as a record
without that link back.

A transcript feels like evidence
because it looks like writing.

It is a guess
that the model wrote down
in the voice of the person
it was listening to.

– Serguey Asael Shinder