Detecting BPM and musical key in the browser (and encoding MP4 with WebCodecs)

We run AIMusicVideoGen, an AI music video generator for independent artists. The main product needs a server, but along the way we built three small free tools that run entirely in the browser: a BPM and key finder, an audio-to-video converter and a Spotify Canvas maker. No upload, no backend, no account.

This post walks through how they work, because most of it is plain Web Audio, one hand-written FFT and WebCodecs, and it was less work than we expected.

1. Decoding the file

Everything starts with decodeAudioData, which takes MP3, WAV, M4A and friends and hands back an AudioBuffer of raw float samples. We cap uploads at 80 MB so a long WAV doesn't eat the tab's memory.

For analysis we don't need stereo or 44.1 kHz. An OfflineAudioContext does the downmix and resampling for free:

async function toMono(buffer, sampleRate, maxSeconds) {
  const seconds = Math.min(buffer.duration, maxSeconds);
  const offline = new OfflineAudioContext(1, Math.ceil(seconds * sampleRate), sampleRate);
  const source = offline.createBufferSource();
  source.buffer = buffer;
  source.connect(offline.destination);
  source.start();
  const rendered = await offline.startRendering();
  return rendered.getChannelData(0);
}

One channel means the browser mixes stereo down for you, and a lower sample rate means far less data to push through the FFT later.

2. Tempo: don't reinvent it

For BPM we use the small web-audio-beat-detector package and pass it a tempo range:

const { bpm } = await guess(buffer, { minTempo: 70, maxTempo: 180 });

The range matters more than the algorithm. Beat trackers love to report half or double time: a 140 BPM trap beat comes back as 70, a 75 BPM ballad as 150. Clamping to 70 to 180 covers most pop, hip-hop and EDM and removes most of those octave errors. For songs that still come out wrong, the BPM finder also has a tap-tempo button, because a human tapping along is still the most reliable fallback.

3. Musical key: chromagram + key profiles

Key detection was the interesting part. The classic recipe:

1. Downmix to mono at 11,025 Hz and take at most the first 4 minutes.
2. Slide a Hann window of 8192 samples along it, hop 4096.
3. FFT each frame and fold every bin between C2 and C7 into one of the 12 pitch classes. That gives a 12-number "chromagram" per frame.
4. Normalise each frame by its own peak, so a loud chorus doesn't outvote the verses, and add the frames up.
5. Correlate the result against the Krumhansl-Kessler major and minor profiles, rotated to all 12 tonics. The best of the 24 scores wins.

At 11,025 Hz an 8192-point FFT gives about 1.35 Hz per bin, which is enough to tell neighbouring semitones apart even around C2 (65 Hz), where they are only about 4 Hz apart.

We wrote a plain iterative radix-2 FFT (about 30 lines) instead of pulling in a DSP library. The scoring step looks like this:

for (let tonic = 0; tonic < 12; tonic++) {
  const rotated = chroma.map((_, i) => chroma[(i + tonic) % 12]);
  scores.push({ tonic, mode: 'major', score: correlation(rotated, MAJOR_PROFILE) });
  scores.push({ tonic, mode: 'minor', score: correlation(rotated, MINOR_PROFILE) });
}
scores.sort((a, b) => b.score - a.score);

Two details made it feel nicer to use:

• Don't freeze the tab. A 4-minute song is a few hundred frames. Every 40 frames we report progress and await new Promise(r => setTimeout(r, 0)) so the progress bar actually repaints. A Web Worker would be cleaner; this was good enough.
• Show the relative key and the Camelot code. Major and minor relatives (C major / A minor) share the same notes and are the most common confusion, so we show both. DJs mostly want the Camelot code (8B, 5A...) for harmonic mixing, which is just a 12-entry lookup.

4. Finding the chorus without any ML

Our music video creator lets people make a short clip from a full song, so it suggests where to cut. The heuristic is almost embarrassingly simple: compute RMS energy in half-second blocks, then slide a window of the clip length and pick the loudest stretch. In most pop songs that lands on the chorus or the drop. It's wrong sometimes, so the user can always drag the selection, but as a default it's right far more often than "start at 0:00".

5. Encoding MP4 in the browser with WebCodecs

The two video tools need to produce real MP4 files, not screen recordings.

• The audio to video converter draws a cover image, optionally with a spectrum visualiser, frame by frame, then muxes the frames with the original audio.
• The Spotify Canvas maker turns a photo or clip into a silent 1080x1920 loop of 3, 5 or 8 seconds, the format Spotify asks for.

MediaRecorder on a canvas stream works, but it records in real time, often gives you WebM rather than MP4, and you don't control exact frame timing. WebCodecs' VideoEncoder encodes as fast as the machine allows, and we use the Mediabunny library for muxing so we don't have to write MP4 boxes by hand. The flow is:

1. Draw each frame onto an OffscreenCanvas on a fixed timeline: 30 fps when something moves, but only 2 fps for a still cover with no visualiser, which makes encoding a full song far quicker.
2. Hand the canvas to the encoder source with an explicit timestamp.
3. Finalise and download the blob.

The spectrum for each video frame comes from the same FFT code as the key detector, so the visualiser is in sync by construction rather than by luck.

Things we learned the hard way:

• Feature-detect VideoEncoder. Some browsers, especially in-app browsers on phones, don't have it. We show a clear "use Chrome or Edge on a computer" message instead of a silent failure.
• Seamless loops need symmetric motion. For still photos the Canvas maker does a slow 8% zoom in and back out over the loop, so the last frame matches the first. A one-way zoom produces a visible jump every time Spotify restarts the loop.
• Keyframe every second keeps scrubbing smooth and costs little at these lengths.

Wrap-up

None of this needs a server: decode, analyse and encode all happen on the user's machine, which keeps the tools free for us to run and private for the people using them. If you're building something similar, the parts worth stealing are the OfflineAudioContext downmix, the tempo range clamp and WebCodecs over MediaRecorder.

Happy to answer questions about any of it in the comments.