Vibe directing: letting a sentence edit a video timeline without trampling your hand edits
Disclosure up front: we build ClickPop, an AI video studio, and this post is about a design problem we hit while building its editor. The lessons apply to anyone putting an agent and a human in the same document. "Vibe
Disclosure up front: we build ClickPop, an AI video studio, and this post is about a design problem we hit while building its editor. The lessons apply to anyone putting an agent and a human in the same document.
"Vibe directing" is our shorthand for a simple idea: you say what should change ("tighter open", "warmer grade", "captions only on the hook") and the edit happens on the timeline in front of you. No hunting for the right panel. On our homepage it is shown as a recorded session where every cursor on screen is an agent: one cuts to the beat, one grades, one writes the titles, one scores it.
That demo looks like magic. The engineering underneath is mostly about boring, careful state handling. Here is what we learned.
1. Plan in text, render later
The most expensive mistake in AI video is rendering before the plan is right. So the agent's first output is not video. It reads the brief, asks a few quick questions (format, length, music, voice-over, on-screen text; skip any and it uses a default), then produces a script and a storyboard. It shows you the price, and only starts when you confirm.
Changing a storyboard panel costs almost nothing. Changing a rendered shot costs a re-roll. If you are building anything similar, make the cheap representation the one users argue with.
A practical version of this for your own work, even without any tool:
Brief: 30s vertical promo, upbeat, captions on
Beats: 1 hook (0-3s) / 2 problem / 3 product / 4 proof / 5 CTA
Per beat: what we see, camera, on-screen text, sound cue
Fix that table before you generate a single frame.
2. The timeline is the source of truth, not the chat
Early on it was tempting to treat the conversation as the state: "the user said make it warmer, so regenerate everything warmer". That breaks fast. Chat is ambiguous and it has no memory of what the user did by hand.
So every Vibe Direct video opens as one editable project timeline: tracks, clips, text layers, grade, music. Each layer records who made it, the agent or the creator. Instructions from the chat become operations on that structure: trim, move, split, text, effects, grade, music. Those are the same ops the editor's own buttons use, and they show up in your timeline and can be undone like any other edit.
If you are designing an agent-editable document, this is the single most useful rule we have: the agent should speak the editor's operation language, not a private one. Then undo, history and collaboration come for free.
3. Respect what the human touched
Here is the failure that bit us. A user trims the hook by hand, then asks the agent to "make the ending punchier". A naive agent re-plans the whole cut and the hand trim silently disappears.
The fix has two parts:
- Provenance per layer. Because every layer knows whether a person edited it, the agent's ops leave hand-edited layers as they are.
- A diff before acting. Before a new pass, the agent reads what changed since its last run: the creator's manual edits, or the difference between two versions. It plans around them instead of over them.
In general terms: before an agent writes, it should read the diff since it last wrote. Most "the AI undid my work" complaints are really missing-diff bugs.
4. Small, scoped requests beat global re-rolls
"Make it better" invites a full rebuild. "Swap the hook", "redo shot 4", "remove the filler words", "captions on" are scoped, and scoped requests are cheap, predictable and reversible. In ClickPop you can change any shot on the storyboard before filming, redo a single shot without touching the rest, and after the cut, trim, swap the hook, or change captions, music or voice in the editor.
For a single shot you want to restyle, the same principle holds at the clip level: describe the change ("make it night", "turn the jacket red") and the edit keeps the clip's motion and timing, changing only what you asked for.
When you write directions, aim for one verb and one target per sentence. You will get fewer surprises from any tool, human editors included.
5. Check the output against the promise
Agents are good at saying they did something. So the last step of a pass is a check: compare the export against what was promised. Is it the right length? Is there sound? Are the captions there? Is the frame right? It is a short list, and it catches the embarrassing misses before a person has to.
If you build pipelines like this, write the acceptance checks from the brief, not from the agent's own summary of what it did.
6. Hand the wheel back
Vibe directing does not replace an editor. It gets you to a first cut fast, then gets out of the way. The Pro Editor framing is literal: watch the director agent cut, then take over the timeline yourself. Anything you do by hand is respected on the next agent pass.
That handoff, in both directions, is the whole product. A sentence is a great way to say "warmer, faster, captions on". A mouse is still the best way to nudge a cut by four frames.
A workflow you can steal
- Write a five-beat brief (hook, problem, product, proof, CTA).
- Get a storyboard first and argue with it while it is cheap.
- Approve the price, then render.
- Make scoped changes in plain language, one verb and one target at a time.
- Do the fine trims by hand.
- Check length, sound, captions and framing before you post. Label it as AI-generated.
That is how we use ClickPop day to day. If you want to try vibe directing, the storyboard is free to look at before you pay for any rendering.
We would love to hear from anyone building agent-editable documents of any kind: how do you handle provenance and hand edits? Tell us in the comments.
The ClickPop team (we build ClickPop)
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.