This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Step Up! is a web app that tells you whether your salsa steps land on the beat. You prop your phone up, it plays the music, and it watches you dance through the camera. When you finish, you get a score out of 100 and one tip for next time.

The first thing it taught me was about myself. In every session I recorded, my steps landed one count late: I was stepping on 2, 3, 4 instead of 1, 2, 3. I'd been dancing like that without noticing, and nobody at home was going to tell me.

That's the problem it solves. In a class, a teacher can tell you you're off the beat. At home, practising alone between classes, you can't hear it in your own feet, so it's easy to practise the same mistake for weeks.

Who would pay for it: Salsa learners who take a weekly class and practise at home. A class costs around £10 to £15, so a few pounds a month for something that checks your homework is easy to justify. The next step up is dance schools: a teacher could set a tempo as homework and see each student's scores, which is worth a monthly studio licence.

What it does:

• Plays the music. A built-in rhythm (cowbell, clave and conga), a salsa band with piano and bass, or one of three real songs, at 140 to 200 BPM.
• Works hands-free. After you tap once to turn the camera on, you never touch the phone again. Raise both hands above your head to open the hands-up GUI. Your wrists show as yellow rings, and holding one over a button for 1.5s presses it.
• Scores four things. Tempo (are you the right speed?), Pause length (do you hold on 4 and 8?), Timing (does your basic start on the 1?) and On the beat (does each step land within 80ms of the beat, or are you dragging or rushing?).
• Shows your typical basic. The average of every basic you danced, drawn as six dots, green, amber or red. Tap a score to see it on those dots.
• Explains every score. The full analysis goes through the same four scores, and every box ends with one thing to do. It compares your 123 with your 567, shows the pause on the 4 and the 8 separately, and draws every basic you danced.
• Keeps your history. Every session is saved in your browser on your own device, with a "Your score over time" chart. Tap a dot to open that session.

My Results after a session: right speed and pause, but still one beat late.

Demo

Try it: https://luke-ms-devlin.github.io/salsa-sync/

It runs in your browser with nothing to install. To try it:

1. Prop your phone or laptop 2.5–3m away, low down and tilted up, with your whole body in frame and the light on.
2. Tap the camera button, then raise both hands above your head to open the hands-up GUI.
3. Choose Full sync first (stand still, then march to 16 clicks), then Start.
4. Dance the basic for 30s and read your score.

Code

The whole app is one index.html file: no build step, no server, no accounts. It's MIT licensed.

How I Built It

The open-source AI at the core is Google's MediaPipe Pose Landmarker, running the pose_landmarker_lite model in the browser with MediaPipe Tasks Vision. It finds 33 points on your body in every camera frame, on your own phone's GPU. I use the hips, knees and ankles. Everything else is plain JavaScript and the Web Audio API.

Getting from "33 dots" to "your step on 5 was 120ms late" took four problems.

1. Two clocks. The music runs on the audio clock, the camera on the video clock, and they don't agree. If they're 50ms apart, every step looks 50ms off, and that's the size of the mistakes I'm trying to measure. Each camera frame gets its timestamp from requestVideoFrameCallback, and each beat is moved onto the same clock with AudioContext.getOutputTimestamp(). To check it, I marched on the spot to a click: the detector found 78 of 80 steps, with a median of 6ms after the beat.

2. What counts as a step. Salsa steps are small. The steps on 2 and 6 are barely a weight change. For each foot I measure how high the ankle is above where it was resting, divided by leg length so it doesn't matter how far you stand from the camera. A 3s "Motion sync" at the start (just stand still) measures how much the points wobble when you're not moving, and sets how high a foot has to lift to count. The step is timed when the foot comes back down to half its peak height, because a step counts when your weight lands, not when your foot is highest.

3. Bluetooth. Bluetooth headphones and speakers play sound around 150–200ms late, and the browser doesn't know. That would make every step look late, even a perfect one. So there's an Audio sync: you march to 16 clicks, the app takes the median gap, and allows for it from then on. On a new device, one Full sync button on the hands-up GUI runs Motion sync and then Audio sync, and it's highlighted yellow with "do this first" until you've done it.

4. Scores that agree with each other. My first results screen showed percentages and charts, and the numbers sometimes contradicted each other. Now everything comes from your typical basic, the average position of each of the six steps. The words, points and colour of each score come from that one basic, so a green tile never says "Too short". The tip is the first score that fails, so a dancer who's a count late is told "Start on the 1" before anything else. The summary is the quick answer; the full analysis underneath shows where it came from, like a short pause on the 4 hiding behind a long one on the 8.

Things I learned the hard way:

• A menu you can't touch has to stay calm. You're 3m from your phone, so the hands-up GUI used to put a big red "Step back. I can't see your feet." on screen and say it out loud. Now "Can't see you" and "Can't see your feet" are a small silent notice, and the buttons stay a button-height up from the bottom of the screen.
• The model needs your face. I tried filming just my legs from a low camera, and it stopped finding me. MediaPipe's detector uses your face to find where your body is. The fix is a low camera tilted up, head to feet in frame.
• Light changes everything. One late-night session scored nothing. The webcam had halved its frame rate in the dim room (16fps down to about 8fps), so short steps fell between frames. With the main light on it was fine.

For music, the built-in rhythm and band are generated note by note with Web Audio. The three songs are Creative Commons tracks from the Free Music Archive. A live band drifts slightly in tempo, so I found the time of every beat in each song with a beat tracker, and the app scores you against those real beat times.

Why Does Open Innovation Matter?

Privacy. This app films people dancing in their living rooms. With MediaPipe running in the browser, the video never leaves the phone. There's nothing to upload, so there's nothing to leak. I don't think many people would dance in front of a camera that streams to a server.

Cost. A closed vision API charges per frame. At 15–30 frames a second, a 30s session is up to 900 calls, and a keen learner does several a day. Local inference costs nothing per session, which is the only reason a cheap or free version is possible at all.

Speed and offline use. There's no network round trip, so each frame is processed as it arrives, and once the page and model have loaded, a session doesn't need the internet.

I could see how it works. MediaPipe's model and documentation are open, which is how I found out why it lost me when I filmed only my legs: the detector is built on a face detector. With a black-box API I'd have been guessing.

Open all the way down. The music is Creative Commons, and the app is MIT. A dance teacher who wants a different tempo, a different style, or their own counts can fork it.

My Agent Session

I built Step Up! over three days with Claude Code as my coding assistant. I decided what the app should do and how it should look, and tested every version by dancing in front of my phone and laptop. Claude wrote the code and analysed the session data I exported, for example spotting that I dance one count late and that low light was halving my camera's frame rate. Every change came as a pull request with a preview link, and I tried it out before merging it myself: 46 pull requests in total.

Credits

• Body tracking: MediaPipe by Google (Apache 2.0).
• Songs from the Free Music Archive: "Cupido" and "Vamos A Tocar Sonero" by Frank Guerrero y Su Grupo Aché (CC BY-NC-ND 3.0, unchanged), and "La Mujer de Antonio" by SONGO 21 (CC BY-NC-SA 3.0, tempo-changed). Both are non-commercial licences, so a paid version would need licensed music.
• Beat tracking for the songs based on Ellis (2007), "Beat Tracking by Dynamic Programming".