Building MVFlow: Turning Music into an Editable AI Video Workflow

I’m building MVFlow.ai, an AI music video studio that connects a music track, creative direction, editable storyboards, generated shots, and a finished video.

The workflow starts with audio and a description of the visual world you want to create. You can add reference images for characters, locations, or style, review a timed storyboard, generate different shot versions, and choose which versions appear in the final composition.

There is also a singing-video workflow that starts with a character photo and a vocal track.

For this post, I want to share the engineering behind the music video workflow. Several of its most interesting problems happen between model calls: validating plans, preserving timing, recovering interrupted tasks, and assembling media.

The stack

The project uses:

• Next.js 16, React 19, and TypeScript for the application.
• PostgreSQL and Drizzle ORM for persisted project and workflow state.
• Cloud Tasks for queued work.
• A Python media worker designed for Cloud Run.
• FFmpeg for media processing and composition.
• Cloudflare R2 for media storage.

The application manages creative decisions and project state, while the media worker handles operations such as cropping audio and composing the final video.

A useful way to describe the pipeline is:

Audio + creative direction + optional references
                       ↓
             Sections and shot timing
                       ↓
          Creative plan generation and review
                       ↓
           Reference images and shot videos
                       ↓
              Selected shot versions
                       ↓
              Final video composition

Each arrow represents a boundary where inputs need validation and progress needs to survive failures.

A storyboard needs an executable contract

An AI-generated storyboard can sound convincing while containing references or timing that the application cannot execute.

MVFlow establishes section and shot timing before generating the complete creative plan. A separate review stage receives the source context and the generated plan, then returns a complete corrected plan.

Both stages produce structured JSON. Local validation checks the structure, and a compiler checks relationships within the plan: section and shot IDs must match the canonical timeline, referenced assets must exist, and image dependencies must satisfy the resource rules.

That distinction matters. Valid JSON tells us that a response can be parsed. Semantic validation tells us whether the application can use it.

The review stage adds another opportunity to improve the plan, but deterministic checks still decide whether the result satisfies the execution contract.

Checkpointing saves useful work

The generation result is saved before review begins.

If review fails, the application can reuse that draft when its checkpoint still matches the current inputs. The fingerprint includes source context, references, the canonical plan, model configuration, and prompt version.

This makes retries more focused. A temporary failure during review does not require discarding an already validated generation result.

It also creates an explicit invalidation rule: changing the inputs or relevant configuration prevents an old checkpoint from being reused.

Timing is a data integrity problem

A music video has to cover a continuous audio timeline. A small gap or overlap between shots can become visible in the final output.

The project stores timeline boundaries on a hundredth-of-a-second grid and checks continuity before composition. The core check looks like this:

const expectedStart =
  index === 0 ? 0 : shots[index - 1].end;

if (Math.abs(shot.start - expectedStart) > tolerance) {
  throw new Error(
    'Shot timeline must be continuous before composition'
  );
}

Composition also verifies that the selected shots cover the complete source audio and that explicitly selected versions are available.

These checks give the media worker a coherent timeline to assemble. They also make errors easier to explain before an expensive render starts.

Recovery needs durable ownership

Video generation can outlive a browser session or an individual HTTP request.

The recovery design uses persisted project state, queued execution, and database leases. A lease identifies the worker currently responsible for a recovery pass. Release operations check the ownership token so an older worker cannot accidentally release a newer worker’s lease.

Provider calls happen outside long-running database transactions.

The browser reads project snapshots, while recovery execution handles unfinished stages. Keeping those responsibilities separate makes a status request predictable and gives background work its own execution path.

There is another useful distinction inside the media worker: processing can succeed even when delivering its completion callback fails.

The worker persists its terminal result before sending the callback. When delivery is retried and a cached result is available, it can resend that result without rendering the video again.

Composition has a conservative fast path

FFmpeg composition can sometimes reuse encoded video streams through stream copy, avoiding another encoding pass.

That path needs more validation than matching width and height. The worker checks compatibility details such as frame rate, codec headers, timestamps, and color metadata. Clips requiring trimming, resizing, padding, or other normalization fall back to encoding.

This keeps the optimization conditional on the media actually being compatible.

The worker also records phase timings. Downloading, preparing clips, concatenating, adding audio, and uploading are separate costs, so measuring them individually helps identify where time is going.

Building MVFlow has made the boundaries between these stages especially interesting to me. A useful creative tool needs editable decisions, validated inputs, durable progress, and recoverable results throughout the workflow.

You can explore it at MVFlow.ai. I’d love feedback from developers building asynchronous AI or media applications: how do you decide which intermediate results are worth checkpointing?