I stopped rendering my Tiny Desk AI video shot by shot. Here's what replaced it

The "Outstanding" Tiny Desk AI meme puts you and your friends into a remix of Charlie Wilson's 2023 Tiny Desk performance. I built a generator for it, Tiny Desk AI (tinydeskai.run), and my first pipeline rendered each shot separately. It looked like the obvious design, but it had real problems. This post covers why I replaced it with one render of the whole video, and how the output still lands on the song.

Disclosure: this is my product. Fan-made parody, not affiliated with NPR or Charlie Wilson.

The template

The video is a fixed edit of four shots with five seats: lead vocals, keys, two backing singers and sax.

That adds up to about 18 seconds at 720p, 9:16. We generated two reference videos with fictional performers, a puppy band and a people band, so nothing depends on the original footage.

Version 1: one render per shot

The first version split the work up. Each shot had its own reference clip and its own description of every seat, and only shots containing a swapped seat were rendered. A lead-only video needed one render; the whole band needed four.

On paper this was cheap and controlled. In practice it had two problems:

1. A seat could get missed. Per-shot renders could leave someone out.
2. The outfits were always the band's. The prompt told the model to ignore the clothes in the photos, and a paying customer flagged the result.

Version 2: one render, whole video

Now each video is a single Kling 4.0 reference-to-video render through the Kie API. The band video goes in as @video_1, and each photo goes in as @image_N:

{
  prompt: buildWholePrompt(band.id, roles),
  image_urls: roles.map((role) => photos.get(role)!),
  video_urls: [SEATS[band.id].videoUrl],
  keep_original_sound: true,
  duration: 18,
  resolution: '720p',
  aspect_ratio: '9:16',
}

Each photo now replaces one seat, clothes included, and the prompt asks for the same clothing in every shot.

The prompt

This is the prompt for a video with keys and sax replaced in the puppy band:

Edit @video_1 and keep it exactly the same: every camera cut and its timing,
framing, camera movement, room, shelves, lighting, instruments, microphones,
body positions, gestures and mouth movement.
Replace these performers, one for one, staying in the exact position of the
performer replaced: the keyboard player: the beagle in the white cap seated in
front at the red keyboard becomes the person in @image_1; the saxophone player:
the corgi in the navy jacket with a white hood who plays the saxophone becomes
the person in @image_2.
Each replaced performer becomes a complete realistic human with the face, hair,
skin tone, build and the clothing and headwear shown in their own photo, with
the same clothing in every shot. If a photo shows only part of the outfit,
complete the rest with matching, plausible clothing.
Do not copy phones, photo backgrounds or anything else from the photos.
Everyone keeps a headset microphone and over-ear headphones as in the video.
Every other performer stays exactly as in @video_1. Never swap positions
between performers and never show one person twice. No added text, watermark
or new people.

What some of those lines are for:

• "one for one, staying in the exact position" keeps each person in their chosen seat.
• "never show one person twice" keeps one photo from appearing on two performers.
• "Do not copy phones, photo backgrounds" is there because people upload selfies.
• "complete the rest" covers tight selfies that show little of the outfit.

Getting back on the beat

The catch with one long render is timing. The model keeps the cuts, but they drift, by up to about a second. Since the song is fixed, a late cut means the big laugh misses its moment.

So a stitcher running ffmpeg detects the cuts in the render and fits each shot back to its slot:

const ratio = (edges[i + 1] - edges[i]) / target;
const speed =
  ratio > MAX_FASTER ? 1 : ratio < MAX_SLOWER ? MAX_SLOWER : ratio;

• A small difference is retimed, within 0.87x to 1.15x.
• A shot far too long keeps its speed and loses its tail.
• A shot far too short slows to the limit and holds its last frame.
• If a cut can't be found within 1.5 seconds of where it should be, the render plays as it is, cut to the total length, rather than risk a bad edit.

The output comes in two versions: one with the song, and a silent copy for people who add the sound on TikTok or Reels.

What I took from this

Splitting a video job into small pieces felt like good engineering, but it caused the problems above. Now the model gets the whole clip, and ordinary ffmpeg code fixes the timing afterwards. Results still vary between renders, and a photo that shows little of a person leaves more for the model to invent.

Some product decisions follow from the pipeline: one price for one to five people ($9.99 per video, no subscription), credits returned automatically when a render fails, and no free personal preview, since every preview would be a full paid render. Samples are free to watch.

Try it or read more

• Generator: https://tinydeskai.run/
• A real one-photo result next to its input: https://tinydeskai.run/examples/one-photo-lead
• The shot-by-shot template: https://tinydeskai.run/outstanding-tiny-desk-ai-template

Questions about the pipeline are welcome in the comments.