Building a Privacy-Friendly AI Music Tool That Runs Entirely in the Browser

When people think about AI-powered web applications, they usually imagine a familiar architecture:

The user uploads a file, a backend sends it to an AI service, and the server returns the result.

That approach works, but it isn't the only option.

Modern browsers can perform surprisingly complex machine learning tasks locally.

For certain workloads, this makes it possible to build useful AI applications without sending a user's private file to a processing server.

Music chord recognition is an interesting example.

Let's look at the architecture, technical trade-offs, and user experience challenges involved in running audio analysis directly in the browser.

1. Why Run AI Locally?

Consider an application that detects chord progressions from uploaded audio.

A conventional implementation might look like this:

User selects audio
        ↓
Upload to server
        ↓
Server preprocesses audio
        ↓
AI model runs
        ↓
Return chord predictions
        ↓
Display results

This introduces several challenges.

Large audio uploads consume bandwidth.

Server-side inference requires compute resources.

Users may be uncomfortable uploading private recordings.

And as usage increases, inference costs can become difficult to predict.

An alternative is to move the audio processing and model inference into the browser.

User selects audio
        ↓
Browser decodes audio
        ↓
Extract audio features locally
        ↓
Run downloaded AI model
        ↓
Post-process predictions
        ↓
Display chord timeline

The file doesn't need to leave the user's device for processing.

This changes both the privacy model and the infrastructure economics.

2. Browsers Can Decode Audio Locally

The Web Audio API provides browser-native audio processing capabilities.

For example, you can decode a user-selected audio file into an AudioBuffer:

async function decodeAudio(file) {
  const context = new AudioContext();

  try {
    const bytes = await file.arrayBuffer();

    const audioBuffer =
      await context.decodeAudioData(bytes);

    console.log({
      duration: audioBuffer.duration,
      sampleRate: audioBuffer.sampleRate,
      channels: audioBuffer.numberOfChannels
    });

    return audioBuffer;
  } finally {
    await context.close();
  }
}

This example assumes the selected file uses an audio format supported by the browser.

It doesn't upload the file.

Instead, the browser decodes it locally into audio samples that can be used for further processing.

But decoding is just the first step.

The next challenge is converting the audio into information that a machine learning model can understand.

3. Turning Sound into Musical Features

A raw audio waveform contains changes in sound pressure over time.

A chord recognition system needs to estimate harmonic relationships from those samples.

One common technique is the Constant-Q Transform (CQT).

Unlike a conventional FFT with equally spaced frequency bins, CQT uses logarithmically spaced frequency bins.

This is useful for many musical tasks because musical pitch relationships are logarithmic.

For example, moving up one octave doubles the frequency.

A feature extraction stage can transform the audio into a time-frequency representation:

Audio samples
      ↓
Resampling / preprocessing
      ↓
CQT feature extraction
      ↓
Time-frequency matrix
      ↓
Chord recognition model

The exact preprocessing steps must match what the model expects.

A mismatch in sample rate, normalization, frequency bins, or tensor layout can seriously reduce prediction quality.

This is an important engineering lesson:

A good model is only one part of a good inference pipeline.

4. Running ONNX Models in a Browser

ONNX provides a portable format for representing machine learning models.

With ONNX Runtime Web, a compatible model can execute in the browser using supported backends such as WebAssembly.

A simplified inference flow might look like this:

import * as ort from "onnxruntime-web";

// The model URL and tensor shapes are examples.
// Real applications must match the model's inputs.

const session =
  await ort.InferenceSession.create(
    "/models/chord-model.onnx",
    {
      executionProviders: ["wasm"]
    }
  );

// Input preparation depends on the model.
// The feature matrix must be converted
// into the expected tensor representation.

The difficult part isn't simply loading the ONNX file.

You also need to handle:

• Model download and initialization
• Audio preprocessing
• Tensor creation
• Memory usage
• Inference execution
• Chord label decoding
• Prediction smoothing
• Timeline generation

And because this work runs on the user's device, performance characteristics vary.

A powerful desktop may handle a model comfortably, while a low-memory mobile device may struggle.

5. Designing the First Model Download

Local AI introduces a user experience problem that traditional web applications don't always have.

The model needs to be downloaded before inference.

Depending on its size and the user's internet connection, this may take noticeable time.

A good interface should make the process understandable.

For example:

[1] Audio selected
      ↓
[2] Model download required
      ↓
[3] Downloading model
      ↓
[4] Initializing AI runtime
      ↓
[5] Preparing audio features
      ↓
[6] Analyzing chords
      ↓
[7] Displaying results

These stages communicate different kinds of work.

A download progress indicator should represent actual network progress when measurable.

An analysis progress indicator should not pretend to provide exact percentages unless the application can calculate them.

And if the model can be cached, the interface should explain that subsequent use may not require downloading the same resources again.

Good feedback matters because users don't know what the browser is doing behind the scenes.

6. A Real Example: ChordFold

One example of a browser-based approach is ChordFold, an AI chord finder for analyzing music recordings.

It uses a BTC-based chord recognition model with ONNX Runtime.

Users can select MP3, WAV, or other browser-supported audio files within the site's size and duration limits.

The audio is decoded and analyzed locally on their device, rather than being uploaded to a remote inference server.

The interface allows users to:

• Explore estimated chord progressions
• Follow chord changes over time
• Replay sections of the recording
• View supported guitar chord diagrams
• Inspect corresponding piano notes

This is particularly interesting because the output isn't just a static prediction.

The chord labels are connected to the original audio, creating a practical tool for listening and musical exploration.

7. Why Chord Recognition Isn't Perfect

Automatic chord recognition is a difficult problem even with deep learning.

An audio recording might contain several instruments playing simultaneously.

Guitar distortion introduces additional harmonics.

The melody may contain notes outside the underlying chord.

Extended chords and inversions create further ambiguity.

For example, a simple progression:

C → G → Am → F

might be relatively straightforward in a clean piano recording.

But a dense arrangement with vocals, bass, electric guitars, and effects can produce much more ambiguous features.

There's also a timing problem.

The model may identify the correct chord but place the transition slightly too early or late.

This is why a useful interface should let people inspect the results rather than treating the model's predictions as unquestionable facts.

For music learning, a partially correct chord timeline can still be valuable.

It gives users a starting point for listening, checking, and practicing.

8. The Trade-Offs of Local Inference

Browser-based AI isn't automatically better for every application.

It has meaningful advantages:

Privacy

Selected user files can remain on the device.

Predictable server inference costs

The server doesn't need to process every uploaded recording.

Lower dependency on remote inference availability

Once the application and model resources are loaded, inference can happen locally.

But the disadvantages matter too.

Initial download

Models may require a significant download.

Device limitations

Memory and CPU performance vary dramatically.

Browser compatibility

Not every browser supports the same acceleration backends.

Resource management

Large audio files and intermediate feature arrays can use substantial memory.

Model updates

Changing the model requires managing new resources and potentially invalidating cached versions.

A robust product must handle these trade-offs carefully.

9. When Should You Consider This Architecture?

Local inference makes sense when:

• The model fits reasonably within client-device limits.
• Data privacy is important.
• Inference doesn't require extremely large hardware resources.
• The application can tolerate device-dependent performance.
• You want to avoid a server inference request for every user action.

Audio analysis, image classification, certain text-processing tasks, and other specialized workloads may fit this pattern.

However, large generative models or applications requiring powerful GPUs may still benefit from server-side infrastructure.

The correct choice depends on the workload, user devices, and product requirements.

Final Thoughts

Browser-based AI has become a practical architecture for more than simple demonstrations.

With the Web Audio API, machine learning runtimes, and efficient preprocessing, web applications can perform meaningful analysis locally.

A music chord finder is a useful example because it combines several engineering challenges:

• Audio decoding
• Musical feature extraction
• Model inference
• Temporal prediction processing
• Interactive visualization
• Privacy and memory management

The result can be an application that doesn't require users to upload personal recordings to a processing server.

If you're interested in seeing this approach from the user's perspective, you can explore ChordFold's browser-based chord finder.

It demonstrates how local AI can turn an ordinary audio file into an interactive musical learning tool.

For developers, the larger opportunity is clear: not every AI feature needs a cloud inference endpoint.

Sometimes, the browser itself is enough.