Why my local Whisper captions never write straight into the project

I build CutVoke, an open-source local video editor that Claude Code / Codex can drive over MCP while you keep a normal timeline open in the app. The most-asked-for agent task is boring: "cut the silences and add captions." Captions turned out to be the place where I had to be most careful about who is allowed to change the project, and when.

Here is the design I ended up with, in case you're wiring speech recognition (or any slow, fallible model) into an app that both a human and an agent edit.

1. Recognition returns a proposal, not an edit

Automatic captions use faster-whisper locally (optional cutvoke[asr] extra; the model downloads once, and you can pre-download it for offline machines). Nothing is uploaded.

The important part: a recognition run never touches the project. It produces a draft. In the UI you fix words and timings, delete bad candidates, then click "Add to project", and all accepted lines land as one undoable command.

Agents get the same split. caption.transcribe only returns the proposal. caption.bulkAdd commits many subtitle lines at once. So an agent can transcribe, look at the text, drop the junk, and commit, and the human can undo the whole batch in one step.

(There is also an older caption.autoSegment, which only splits on silence and inserts placeholder text. It isn't speech recognition, and the docs say so explicitly so agents don't confuse the two.)

2. Bind the proposal to the source it was made from

Speech recognition is slow. In the time it runs, someone can trim the clip, move it, swap the source file or change its speed. If you then commit the old timestamps, every caption is off.

So each proposal carries a source signature. It's checked when you resume a job and again at commit time. If the clip or file changed, the draft is rejected and you're told to re-run recognition, instead of silently writing misaligned subtitles. It's the same idea as the expectedRevision check on normal timeline edits: stale writes come back as conflicts, they don't land.

3. Jobs survive closing the app

Proposals and proofreading drafts live in a job ledger in the same SQLite database as the project, not in browser memory. Draft edits autosave about 350 ms after you stop typing. Close the app, reopen the project, and the draft is still there.

A few edge cases that bit me:

• Cancel must win. Cancellation is persisted, and a late model result can't overwrite a cancelled job.
• Interrupted jobs don't auto-commit. If the service dies mid-run, the job shows "waiting to resume". Resuming re-runs from the start of the clip with the same model and language, re-checks the source first, and still doesn't write captions by itself.
• Older saves can't clobber newer ones. If an earlier save request arrives late, it's ignored.
• Bounded queue. At most two queued/running jobs, and the ledger keeps running jobs plus the latest 24 others from the past 7 days.

4. Small details that matter for real footage

• Device auto: GPU float16 if CTranslate2 sees CUDA, falling back to CPU int8 if the GPU model fails to load. CUTVOKE_ASR_DEVICE=cpu|cuda forces it.
• For Chinese, output is normalized to Simplified with OpenCC tw2s over the whole word sequence, so per-word timestamps stay intact and single characters keep their context.
• Accepted captions stay editable on their own track, export to SRT, and get burned into MP4 exports.
• Recognition still needs a human pass, especially names, dialects and music-heavy audio. That's exactly why it's a proposal.

What the agent flow looks like

With CutVoke added to Claude Code (setup guide), a prompt like:

"Find the sentence where I say 'pricing' in the transcript, remove it, and export a new MP4 next to the original."

runs against the same timeline you have open, and anything it got wrong is a clip or a caption you can fix by hand, not a re-roll.

The general lesson

If a model is slow and sometimes wrong, give its output a staging area: a proposal bound to the exact input it saw, committed as one undoable step, rejected if the world moved underneath it. That made captions the least scary thing to hand an agent instead of the most.

Repo (MIT, Windows/macOS/Linux): https://github.com/frankShan001/CutVoke. Local ASR doc: docs/user/LOCAL_ASR.md. I'd love to hear how others handle stale model output in collaborative apps.