How I Run Whisper and Export Captioned MP4s Entirely in the Browser

Getting a transcript is only the first handoff in a caption editor. After that, the words have to survive several clocks, a live preview, a style system, and a video encoder.

I’m building Karaoke Captions, a React and TypeScript editor that transcribes and renders video locally. In an earlier post I gave an overview of the product. This time I want to walk through the implementation choices that make a word-by-word caption become a finished MP4.

The interesting parts are the boundaries: compressed media to PCM, model output to editable timing, preview time to export time, and decoded frames to an MP4 container.

The pipeline I keep separate from the interface

The local workflow has six main stages:

File / Blob
  → decode audio to 16 kHz mono Float32 PCM
  → transcribe in a Web Worker
  → normalise word timestamps and build phrases
  → edit text, timing and style
  → draw captions over decoded video frames
  → encode and mux the finished MP4

React owns the editing interface. The document holds the words and style choices. The transcription worker handles model work. The rendering code decides what a caption looks like at a given time.

That separation lets me revise a font or a sentence without running speech recognition again. It also gives the preview and export a shared description of the video they’re supposed to produce.

1. Get the audio contract right before asking Whisper anything

A video file isn’t the audio representation the speech pipeline expects.

My transcription path uses 16 kHz, mono, Float32 PCM. The input might arrive with a different sample rate, multiple channels, or an audio codec that needs another decoding path.

I start with Web Audio decoding, downmix the channels, and resample. There’s also a streaming decoding fallback for inputs the first path can’t handle and for memory-constrained cases.

The memory side is easy to underestimate. At 16,000 samples per second and four bytes per Float32 sample, ten minutes of the final mono PCM is about 38.4 MB in decimal units. That’s just one representation. The original video, a higher-rate decoded buffer, the model, and intermediate allocations can coexist.

So I avoid copying a large ArrayBuffer simply to pass it into the next step, and I release references when the decoder no longer needs them. A small compressed file doesn’t imply a small decoded working set.

2. A worker needs a useful protocol, not just a completion event

I run Whisper through Transformers.js in a Web Worker. The main thread receives progress, partial text, the selected execution device, completed words, or an error.

Those are separate states because they describe separate work. Downloading a model isn’t the same as transcribing a clip. A worker that can only say “started” and “finished” gives the interface very little to explain.

I also check for a usable GPU adapter rather than treating the existence of navigator.gpu as proof that inference will run. My implementation has a WebAssembly path and retries in a fresh worker if WebGPU session initialisation fails.

That last detail is specific to the runtime behaviour my app has to handle. I don’t want a failed acceleration attempt to be the end of the user’s captioning session.

For longer audio, I segment the PCM around pauses, transcribe those segments, and add the segment’s offset back to each word timestamp. Otherwise, words from a later segment would be timed as though they occurred near the start of the video.

3. Give each clock a name

Model timestamps arrive in seconds. Phrase boundaries in my renderer use milliseconds. Remotion also has a frame clock.

The conversions are small enough to look harmless:

const secToMs = (seconds: number) => Math.round(seconds * 1000);

const msToFrame = (milliseconds: number, fps: number) =>
  Math.round((milliseconds / 1000) * fps);

The important choice is where they live. I keep the conversion boundary explicit rather than scattering multiplications throughout the caption components.

In the current export path, a decoded sample’s timestamp is in seconds. The drawing call converts it to the millisecond time the caption renderer expects. When working directly with raw WebCodecs objects, check their documented timestamp units too; the surrounding abstraction matters.

I also normalise the words before grouping them: trim empty text, reject non-finite times, keep them chronological, and constrain implausible durations. A timestamp is model output, not a promise that the next stage can skip validation.

4. Phrase grouping is part of rendering correctness

Grouping every three words is a reasonable starting point. Speech also contains punctuation and pauses.

My grouping logic breaks at punctuation and longer pauses, bridges short gaps, and clears captions during longer gaps. More importantly, it clamps a phrase’s end against the next phrase’s start.

The intended display interval is half-open:

startMs <= currentTimeMs < endMs

That gives adjacent captions an unambiguous handoff. If one path draws every matching phrase and another path chooses the first match, overlapping intervals can make preview and export disagree. Fixing the interval data removes that ambiguity before either renderer sees it.

This is also why SRT import needs a clear timing model. A normal SRT cue supplies a caption window, not the exact delivery of every word. Imported caption times are preserved; word highlights are estimated inside them and remain editable.

5. Share the drawing code between preview and export

I don’t want to maintain a browser-CSS version of a style and a separately interpreted export version.

For the Pro looks, the preview, style thumbnails, and exported frames use the same canvas rendering engine. The renderer receives the style, phrase data, time, and dimensions, then draws that state.

That makes a paused preview frame and an exported frame comparable. It also encourages deterministic animation: derive the look from the supplied timeline time, rather than Date.now() or a new random value on every draw.

The gallery is another view of the rendering system, rather than a separate set of illustrative mockups.

Fonts are another dependency of the frame. If text is measured before the chosen font is ready, a later font load can change line breaks or alignment. I wait for the required fonts before relying on their layout.

6. Export decoded frames in sequence

The current MP4 path uses Mediabunny around browser media APIs. It reads the file, decodes video frames in order, draws the source frame and caption overlay onto a canvas, and encodes H.264 into an MP4 output.

The drawing callback is conceptually:

sample timestamp
  → source video frame, with rotation and fit applied
  → motion graphics at that time
  → caption style and active word at that time
  → optional watermark
  → encoded output

Reading frames in sequence avoids asking a hidden video element to seek independently for every output frame. I retain a seeking fallback for inputs the conversion path can’t handle, but it isn’t the preferred route.

Audio deserves its own decision. In the conversion path, a compatible audio track can be copied into the output rather than unnecessarily decoded and encoded again. Container compatibility and the selected path determine what’s possible.

WebCodecs provides low-level audio and video codec interfaces. It doesn’t turn encoded chunks into a ready-to-download MP4 by itself; the container and muxing work still has to happen.

What the user should see at these boundaries

A creator doesn’t need to learn any of this to caption a clip. They do need the interface to explain what the app is doing and what the downloaded file contains.

That’s why I keep transcript review, Fix timing, style selection, and output format visible in the workflow.

An MP4 includes the visible caption design. SRT and VTT contain timed text, so they’re useful for another editor or a separate subtitle track rather than carrying the animation.

If you want to try the implementation as a product, Karaoke Captions has free, watermark-free Classic and Lyrics exports with no account. Extra Pro styles can be tried in the editor; their free MP4 exports have a small watermark. Desktop Chrome or Edge is the workflow I recommend for MP4 output.

The part I keep coming back to is the handoff. A speech model, a preview, and an encoder can each work in isolation while disagreeing about the same second of video. Naming the units, sharing the renderer, and validating the timeline make those pieces behave like one editor.