Dev.to WebDev 🛠 Dev 👁 0 📖 2 min read

Rendering a photo-and-music video in the browser with Canvas and MediaRecorder

Most tools that turn a song into a video upload the file to a server, render it with ffmpeg, and send back an MP4. That round trip is 10 to 60 seconds of waiting, a privacy question, and a bandwidth bill. None of it is n

Most tools that turn a song into a video upload the file to a server, render it with ffmpeg, and send back an MP4. That round trip is 10 to 60 seconds of waiting, a privacy question, and a bandwidth bill. None of it is necessary any more: a modern browser can do the whole job locally.

Here is the pipeline that works, and the three places it breaks.

The first step is decoding the audio instead of playing it. AudioContext.decodeAudioData() gives you a Float32Array you can analyse: RMS envelope, peak position, bar boundaries. Do not use an audio element and record its output, because you get the latency of the media pipeline baked into your file and the sync drifts.

The second step is drawing frames on a canvas and capturing the canvas. canvas.captureStream(fps) returns a MediaStream of whatever you draw. Draw the photo with a slow scale, redraw at 30 fps with requestAnimationFrame, and the browser encodes it for you.

The third step is where most implementations go wrong: joining the audio. Instead of merging two streams after the fact, route the decoded buffer through a MediaStreamAudioDestinationNode and add its track to the recorder stream. Video and audio are then encoded by the same clock, so they cannot drift apart.

Three things bite. video/mp4 is not supported everywhere, so check MediaRecorder.isTypeSupported() and fall back to video/webm;codecs=vp9,opus. Recorders stop at the first dropped frame on a background tab, so keep the tab focused or draw on an OffscreenCanvas inside a worker. And duration is set by the audio, not the animation: stop the recorder on the audio track ended event, never on a timer.

The whole thing is about 200 lines and no server. If you want to see the result rather than the code, here is the same workflow without writing any of it: three routes from a folder of photos and one song to a finished video, including how to time the photo changes to the beat.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.