Turning a video clip into a game sprite sheet in the browser

I make 2D characters by generating a short animation video and cutting it into a sprite sheet. For a while that meant ffmpeg, a Python script and a lot of fiddling with transparency. I moved the whole thing into a browser page so it runs on the user's machine and the video never leaves it. This post is what I learned along the way: reading frames from a <video>, removing a flat backdrop from compressed frames, and keeping the character's feet in place.

A character reference turned into an animated sprite (AI-generated with SpriteCanvas)

All code below is plain JavaScript with no dependencies.

1. Reading frames: seek, wait, draw

There is no "give me frame 37" API on HTMLVideoElement. What works everywhere is: set currentTime, wait for seeked, draw the video to a canvas, read the pixels.

// Call after the video fired 'loadeddata', so the first frame is decoded.
async function readFrames(video, times, width, height) {
  const canvas = document.createElement('canvas');
  canvas.width = width;
  canvas.height = height;
  const ctx = canvas.getContext('2d', { willReadFrequently: true });
  const frames = [];
  for (const t of times) {
    if (Math.abs(video.currentTime - t) > 0.001) {
      const seeked = new Promise((resolve) => video.addEventListener('seeked', resolve, { once: true }));
      video.currentTime = t;
      await seeked;
    }
    ctx.clearRect(0, 0, width, height);
    ctx.drawImage(video, 0, 0, width, height);
    frames.push(ctx.getImageData(0, 0, width, height).data);
  }
  return frames;
}

Three things bit me:

• Don't seek to the very end. A seek to exactly video.duration often shows no frame. I keep every sample time below duration - 0.01.
• Round the sample times. start + i / fps accumulates float error; I round to microseconds so 0.1 + 0.2 style noise doesn't skip or duplicate a frame.
• Background tabs stall. Chrome holds back video loading and decoding in a hidden tab, so a long read just stops when the user switches tabs. Before every seek I wait until the tab is visible again:

function whenVisible() {
  if (!document.hidden) return Promise.resolve();
  return new Promise((resolve) => {
    const shown = () => {
      if (document.hidden) return;
      document.removeEventListener('visibilitychange', shown);
      resolve();
    };
    document.addEventListener('visibilitychange', shown);
  });
}

I also cap the read at 512 frames and check frames × width × height × 4 bytes up front, because a 10-second 1080p clip at 30 fps is about 2.5 GB of RGBA.

requestVideoFrameCallback and WebCodecs are faster, but the first follows playback and drops frames when the page is busy, and the second needs its own demuxer. Seeking is the one path that behaves the same in Chrome, Firefox and Safari, and for a few hundred frames it is fast enough.

2. Finding the backdrop

Generated clips and green-screen recordings have a flat backdrop, so I look at the four corners. For PNG sheets I require the corners to match within 12 per channel. Video frames carry compression noise, so there I allow 24 and use the average of the corners as the key colour.

function detectBackdrop(rgba, w, h) {
  const corners = [0, (w - 1) * 4, (h - 1) * w * 4, ((h - 1) * w + w - 1) * 4];
  const first = corners[0];
  const close = (p) => [0, 1, 2].every((c) => Math.abs(rgba[p + c] - rgba[first + c]) <= 24);
  if (!corners.every(close)) return null;
  return [0, 1, 2].map((c) => Math.round(corners.reduce((sum, p) => sum + rgba[p + c], 0) / 4));
}

3. Removing it: flood fill vs. "everywhere"

For a still sprite sheet I flood-fill from the edges, so a backdrop-coloured pixel inside the character (a green gem on a green screen) survives. For video I clear every matching pixel, like a chroma key, because the gaps between arms and legs are not connected to the border in every frame, and a flood fill leaves them filled.

The hard part is the edge. A scaled, compressed video leaves a one-pixel ring of blended colour around the character: half character, half magenta. Clearing by a tolerance either leaves a pink halo or eats into the character.

What worked: for each remaining pixel that touches a cleared pixel, compare it with the pixel just inside it, the neighbour on the opposite side from the cleared one. If it sits more than halfway from that inner pixel towards the backdrop colour, it is fringe and goes. When the inner pixel is cleared too, the pixel is part of a one-pixel line, so I keep it.

// How far p is from the inner pixel q towards the backdrop colour: 0 at q, 1 at the backdrop.
function towardBackdrop(rgba, p, q, key) {
  let dot = 0, len = 0;
  for (let c = 0; c < 3; c++) {
    const axis = key[c] - rgba[q + c];
    dot += (rgba[p + c] - rgba[q + c]) * axis;
    len += axis * axis;
  }
  return len ? dot / len : 1;
}

This is a projection onto the line between the sprite colour and the key colour, so it works for any backdrop colour and any character colour, without a per-video tolerance.

4. Trimming without losing the motion

Each video frame is mostly empty, so I crop. The trap: cropping each frame to its own bounding box makes the character jump around, because a sword swing makes the box wider in some frames. Crop every frame by the union of all the boxes instead; relative motion stays intact and the sheet is still small.

const union = boxes.reduce((u, b) => !b ? u : !u ? b : {
  x0: Math.min(u.x0, b.x0), y0: Math.min(u.y0, b.y0),
  x1: Math.max(u.x1, b.x1), y1: Math.max(u.y1, b.y1),
}, null);

There is an optional second step for clips where the camera drifts: move each frame so the bottom centre of its content lands where the first frame's does. I leave it off by default, because for a jump the feet are supposed to leave the ground.

5. The atlas

The output is one PNG in a grid plus a JSON Hash atlas, which Phaser and PixiJS load as is. Two fields matter beyond the frame rectangles:

"frames": {
  "hero-attack-03": {
    "frame": { "x": 768, "y": 0, "w": 256, "h": 256 },
    "pivot": { "x": 0.5, "y": 0.875 },
    "duration": 83
  }
}

• pivot is the foot point, so every animation of a character stands on the same line in the engine.
• duration is per frame in milliseconds. Phaser keeps unknown frame fields in customData and ignores duration for playback, so pass it yourself:

const tex = this.textures.get('hero-attack');
this.anims.create({
  key: 'attack',
  frames: tex.getFrameNames().sort().map((frame) => ({
    key: 'hero-attack', frame, duration: tex.get(frame).customData?.duration,
  })),
});

(getFrameNames() has no guaranteed order; zero-padded names make .sort() enough. This is for Phaser 3.80 and later, where a frame's duration replaces the base frame time; up to 3.70 it is added on top.)

What I'd do differently

• Ask for a flat, unusual backdrop colour (magenta) when generating the clip. Green screen fights green cloaks.
• Ask for the last frame to match the first. Loops stop popping, and you don't have to hunt for a loop point.
• A "breathing" idle that is just the sprite scaled up and down looks fake in the engine. Generate a real idle motion (chest, shoulders, cloth) instead. I shipped the scaled kind first and had to redo it.

The tool from this post is free and runs in the browser: video to sprite sheet. I built it as part of SpriteCanvas, which also generates the character and the animation clips; the sample art here was made with it (AI-generated).