I built a video translator: the technical choices beyond the translation model

I built a video translator for spoken product demos, tutorials and conversations. Getting a translated sentence back was only one part of the product. The more useful engineering questions were about what happened around it: whether the original sound should survive, how users could correct a model number, and which files they needed when they left the site.

This post covers those product-facing choices and their trade-offs. It includes a small, reproducible FFmpeg exercise for checking subtitle delivery. It does not describe our provider setup or backend implementation, and the exercise is independent of the production pipeline.

The public workflow is straightforward: upload a recording, choose a target language, review the words and export. Each of those steps hides a decision that affects how much control the user keeps.

Product workflow illustration; it is not a screenshot or evidence of processing speed.

Start with the artifact the user needs

A translated MP4 is convenient for someone who wants to share a clip immediately. It is less convenient for someone who needs to correct one sentence tomorrow, add captions to a course platform or hand the text to a reviewer.

That is why the product offers separate SRT and VTT subtitles and plain-text transcripts with full-video exports. The video is one deliverable; the text is useful working material. A transcript can support a language review, while a timed track can be reused in an editor or player.

Those files are not interchangeable. A TXT file has no playback timing. A subtitle file does not automatically include enough information to reconstruct every decision made in an editor. Keep the exact source video alongside the text when preparing a release. If someone trims the first five seconds from a clip, its old timestamps no longer refer to the same moments.

The practical choice is to preserve editable outputs even when the main interface emphasizes an MP4 download. The cost is a slightly larger set of files to explain. The benefit is that users are not forced to restart a translation just to move their work elsewhere.

SRT for interchange, WebVTT for the browser

Supporting two timed-text formats may look redundant until the destination changes. An editing workflow may expect SRT, while an HTML video element uses a WebVTT track.

WebVTT has its own syntax and timing conventions. Renaming an SRT extension to .vtt does not perform that conversion. Use a converter and inspect its output. For an ordinary reviewed subtitle file, this is enough to demonstrate the handoff locally:

ffmpeg -hide_banner -loglevel error -n \
  -i reviewed.srt reviewed.vtt

A same-origin browser example is small:

<video controls preload="metadata" width="640">
  <source src="source.mp4" type="video/mp4">
  <track default kind="subtitles" src="reviewed.vtt"
         srclang="en" label="English">
</video>

The HTML track reference explains the language and label attributes. Its distinction between subtitles and captions also matters: translated dialogue alone does not necessarily describe meaningful sound effects or other audio information needed for accessibility.

For debugging, inspect the actual VTT network request. Confirm the response body, text/vtt content type and the selected subtitle track. Cross-origin hosting adds CORS configuration to the problem; it is worth removing that variable from the first local test.

Make audio replacement an explicit choice

“Translate this video” leaves an important question unanswered: what should happen to the original soundtrack?

The product exposes two modes. Subtitles only keeps the original audio. Subtitles with voiceover replaces that track with synthetic speech. In the latter mode, the original music and ambient sound are replaced too. That behavior needs to be visible before someone uploads a carefully mixed video.

Preserving a soundtrack while changing only speech is a different requirement. It can involve separated stems, source separation or a production edit with material supplied by the creator. A single mixed track does not arrive as neatly labeled dialogue and music. This product does not promise background-music separation.

There is another distinction between choosing a synthetic voice and cloning a speaker. The interface provides voice choices; it does not reproduce the original person's voice. For conversations, users can review detected speaker labels and assign voices themselves. Labels should help organize dialogue, without implying that the system knows someone's identity or gender.

Making those boundaries explicit narrows what the product promises. It also gives someone an early reason to choose subtitles only, or take the job to an editing workflow with separate audio stems.

Put correction before the final export

Product demonstrations contain awkward material for automatic transcription: model identifiers, measurements and short phrases that make little sense outside a particular catalog. A fluent translation can hide a transcription error rather than fix it.

The review step shows source and translated text together. Users can correct the wording before the final export and inspect speaker assignments for dialogue. This adds an interruption to a one-click flow, but the interruption is useful when a wrong number changes the meaning of a demonstration.

Consider an invented example: a buyer requests 500 units, a supplier states a 200-unit minimum, and production takes 15 days after sample approval. Merely checking that all three numbers survived is insufficient. The translation must retain who said what and the condition that starts the production period.

For a review interface, text editing and voice selection are different jobs. Someone might accept the wording but dislike a pronunciation. Someone else might keep the voice and correct a product code. Treating every complaint as “generate again” makes it difficult to understand what changed.

A useful review begins with a small list of critical terms, units and conditions. Then listen to the final audio. Reading an accurate-looking subtitle cannot prove that its spoken version is understandable.

Separate a quick preview from a full job

A short preview gives a user evidence about their own recording before they commit to a full export. Our public preview covers up to ten seconds at reduced resolution. It is not a sample of every speaker or every difficult sentence in a long video.

That distinction influences the interface. A preview should explain which excerpt was selected and what it can establish. It can reveal an obviously wrong voice or an unsuitable source recording. It cannot certify the translation of a section that it never processed.

For longer work, I prefer a visible job state over a page that appears frozen until a download arrives. The generic design problem is that uploading, preparing text, waiting for review and rendering an export are different activities. A single spinner cannot explain which of them needs attention.

This is a design discussion, not a description of our queue technology. Any implementation needs to answer what happens after a refresh, how a user recognizes an accepted job and whether a retry creates duplicate work. A timeout in the browser is not proof that the server stopped. Check the existing job before submitting the same conversion again.

I would also keep processing estimates separate from measured results. One fast demonstration does not establish a typical latency for longer recordings or a busy queue. Show uncertainty rather than turning a favorable sample into a promise.

Check the exported file independently

The final check should operate on the deliverable, not only on the application state that produced it. A successful rendering request does not prove that the correct streams were included or that the closing subtitle fits inside the video.

Here is a reproducible exercise that avoids real customer footage. It creates a 12-second solid-color video and test tone. The tone is not speech, and this exercise does not evaluate translation quality:

ffmpeg -hide_banner -loglevel error -n \
  -f lavfi -i color=c=0x123047:s=640x360:r=25:d=12 \
  -f lavfi -i sine=frequency=440:sample_rate=48000:duration=12 \
  -c:v libx264 -pix_fmt yuv420p -c:a aac source.mp4

Use an FFmpeg build with libx264. Save these invented cues as reviewed.srt:

1
00:00:00,500 --> 00:00:02,800
Open the export menu.

2
00:00:03,000 --> 00:00:05,500
Choose the subtitle language.

3
00:00:06,000 --> 00:00:09,500
Check the last sentence before publishing.

For a destination that accepts an optional subtitle stream inside MP4, mux the text explicitly:

ffmpeg -hide_banner -loglevel error -n \
  -i source.mp4 -i reviewed.srt \
  -map 0:v:0 -map '0:a:0?' -map 1:s:0 \
  -c:v copy -c:a copy -c:s mov_text \
  -metadata:s:s:0 language=eng \
  -movflags +faststart translated.mp4

The FFmpeg stream-selection documentation describes how mapping chooses streams independently of codec handling. This command keeps the first video stream, the first audio stream if present, and the supplied subtitle track. It does not preserve every track in an arbitrary input.

Video and audio are copied; subtitle text is encoded as mov_text. This is appropriate for the controlled H.264/AAC fixture. Copying arbitrary source codecs into MP4 is not guaranteed to work. The eng metadata describes the example's English text, not an instruction to translate it.

The -n flag prevents accidental overwrites. If you rerun the exercise, use a new output name. I also avoid adding -shortest automatically: subtitle cues can end before the picture, so output termination should be a deliberate decision.

Probe the result:

ffprobe -v error \
  -show_entries stream=index,codec_type,codec_name:format=duration \
  -of json translated.mp4

FFprobe exposes both stream and container information. The local run used FFmpeg 8.0 and produced h264, aac and mov_text streams with a 12.000-second duration. Extracting the subtitle stream back to SRT retained all three texts and their timestamps. The separate VTT conversion also completed.

This verifies a file-handling example, not the production backend. I used a small local checker to exercise malformed timestamps, non-positive durations, out-of-order cues, overlaps and cues ending beyond the media. Those checks caught the intended failures. Overlap itself can be legitimate; a flag should trigger review rather than silently delete dialogue.

Choose delivery based on the destination

A selectable subtitle track is useful only if the receiving player keeps it. Some upload destinations discard embedded subtitle streams. Burned-in captions avoid that dependency, but they become part of the picture and require a new video export to change.

That is a delivery choice to make before publication. Ask whether viewers need a language menu, whether the platform accepts a sidecar subtitle file and whether the same source will support several languages later. A permanently captioned social clip and an editable training library have different requirements.

Timing must be checked against the exact version being delivered. A dub can divide sentences differently from the source language. Different cue boundaries alone do not prove drift; listen to each language against its own subtitle track.

What this leaves for the reviewer

Can a valid subtitle file still be unreadable? Yes. Timing syntax does not measure line length, contrast, font size or reading speed. Test the actual player at the intended screen size.

Does a synthetic voice remove the need for review? No. Listen for pronunciation, missing words and speaker changes. Confirm which audio mode was used before checking for missing background sound.

Does the short preview prove a long video will be correct? It only covers its excerpt. Review the remaining dialogue and watch the complete final output before publishing.

The choices I care about most are the ones a user can inspect and correct: wording, audio behavior and useful export files. A reviewer should be able to identify a wrong line, fix it and check the new export without guessing which part changed.

Project: Video Translator is the tool I am building. The local exercise above is independent of its service and does not require an account.

AI disclosure: AI helped draft this article and prepare the local verification exercise. The topic, product context and disclosure boundaries were directed by the project owner. Public feature descriptions were checked against the live site; the FFmpeg commands were run locally. No production backend implementation is included.