Building Riffly: from a hummed melody to a playable guitar idea

I keep losing small musical ideas.

I will hum something, think "that might work on guitar", then forget enough of it before I have worked out the notes. I built Riffly to test whether a browser could shorten that gap: record a few seconds, turn the audio into a playable guitar idea, listen to the result, then save it for later.

That sounds like a transcription problem. It is really three different problems:

1. detect musical events in messy audio;
2. map those events onto a physical guitar fretboard;
3. run the model safely on a small public server.

The beta now handles microphone capture and short uploads, produces a few tab variants, synthesizes playback in the browser, and keeps real results in a private saved-tab library. It also has visible confidence and demo labels because the output is not an official transcription.

The pipeline

The useful boundary is between generic pitch detection and guitar-specific interpretation.

microphone or upload
        |
        v
bounded transcription job
        |
        v
Basic Pitch: notes + onsets
        |
        v
key / tempo / confidence estimates
        |
        v
fret-position optimiser
        |
        +--> beginner riff
        +--> fuller hook
        +--> power-style variation
        |
        v
Web Audio playback + share/save

Riffly uses Spotify's open-source Basic Pitch library for automatic music transcription. Basic Pitch gives me note events with timing and pitch information. It does not know whether MIDI note 64 should become the open high E string, the fifth fret on the B string, or another valid position.

That choice matters. A sequence can be technically correct and still feel awful to play.

Turning pitches into fret positions

For every detected pitch, Riffly first enumerates the valid string and fret positions in standard tuning. A simplified version looks like this:

STRINGS = [64, 59, 55, 50, 45, 40]  # high E to low E

def possible_positions(pitch):
    positions = []
    for string, open_pitch in enumerate(STRINGS):
        fret = pitch - open_pitch
        if 0 <= fret <= 22:
            positions.append((string, fret))
    return positions

Picking the first valid position would produce a tab, but not necessarily a useful one. Riffly scores transitions between consecutive notes. Large fret jumps cost more. String changes cost something too. Open strings get slightly different treatment.

A dynamic-programming pass then finds a lower-cost path through all of the available choices.

def transition_cost(previous, current):
    previous_string, previous_fret = previous
    current_string, current_fret = current

    fret_jump = abs(current_fret - previous_fret)
    string_jump = abs(current_string - previous_string)

    if fret_jump > 5:
        return fret_jump * 4

    return fret_jump + 0.5 * string_jump

This is deliberately a heuristic. Human fingering decisions depend on hand position, articulation, the next phrase and personal preference. The optimiser gives the product a better default without pretending it has solved guitar technique.

The strongest next feature would be correction. A guitarist should be able to click a note and move it to another valid string and fret. That would improve the result and create useful feedback about which positions players actually prefer.

One transcription, several useful answers

Raw model output can look more authoritative than it deserves. I wanted the interface to make room for interpretation instead.

Riffly creates a reduced beginner riff by dropping notes that arrive too close together. The fuller hook keeps more of the detected sequence. A power-style version changes the shape again. The complexity control lets the user compare those outputs rather than accepting one wall of tab as truth.

The browser plays the generated ASCII tab with the Web Audio API. Playback is not a realistic guitar amp simulation, but it answers an immediate question: does this generated phrase resemble the idea I recorded?

The interface calls the result a "playable guitar idea" for a reason. Distortion, chords, room noise and other instruments can all confuse transcription. Even clean note detection does not guarantee natural fingering. Confidence belongs in the product, not hidden in a debug response.

Why the API returns 202

The first version could have run Basic Pitch inside the upload request. That is simple until two or three people submit audio at the same time and inference work starts fighting the web server for memory.

The public beta uses a bounded job contract instead:

POST /api/transcribe

202 Accepted
{
  "jobId": "opaque-id",
  "jobToken": "one-time-secret",
  "pollAfterMs": 1000
}

The API stores the upload outside the web root, creates a durable job, and returns quickly. One worker claims one job at a time. The browser polls an authenticated status endpoint and shows queued, processing, completed or failed states.

The important part is not the queue itself. It is the bounds around it:

• 20 MiB maximum upload;
• 30 seconds maximum decoded audio;
• a small per-client active-job limit;
• global job and staged-byte caps;
• explicit 429 and 503 capacity responses;
• source-audio deletion after completion or failure.

The job token is random, while the database keeps only its hash. Accepted jobs survive an API restart, and a worker lease allows abandoned processing jobs to recover without starting parallel inference.

I also kept one rule in the frontend: once the server accepts a real upload, the client must not quietly replace a failure with the demo fixture. A capacity error should look like a capacity error. Fake success would make the product feel smoother while destroying trust.

Saving the result without saving the recording

Riffly uses passwordless email links for accounts. A user can save a bounded transcription snapshot containing the title, tab variants, chord events, confidence and useful metadata.

The original recording is not part of that saved object. Demo results are rejected as well. The library exists so somebody can return to a guitar idea, not so the server can accumulate an indefinite audio archive.

This separation also keeps the account model simpler. Authentication owns users, magic links and sessions. The saved-tab library owns validated transcription snapshots. The transcription queue owns temporary audio and short-lived processing state.

The part I have not validated yet

Riffly is a product proof of concept, not a validated transcription engine.

The next quality gate needs a small set of legally obtained guitar clips with known notes and timings. I want to measure pitch, timing and fingering usability separately. Full mixes and isolated guitar should be separate test groups because they fail in different ways.

I am also more interested in behaviour than a polished demo metric. Do people replay the generated tab? Do they choose the beginner version? Do they correct the fingering? Do they save the result and return to it later?

A technically impressive transcription that nobody wants to play is not useful.

Try it

The beta is live at riffly.happymonkey.ai.

Short, clear clips give the model a better chance than a dense full mix. Treat the output as a starting point. If you play guitar, I would be interested in whether the detected phrase resembles what you recorded and whether the chosen string and fret positions feel natural.

The original HappyMonkey article, screenshots and product context are available at Riffly: hum a melody and turn it into a playable guitar idea.