I fixed an unbounded audio queue in a live translation server before it could take the process down

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

The project is a live speech translation app.

A browser client captures microphone audio, converts it to 16 kHz PCM, and streams it to a NestJS server over Socket.IO. The server opens one WebSocket session to a local ML service, forwards the audio frames, and relays translated segments back to listeners.

The important part for this bug: audio starts arriving from the browser before the ML WebSocket is guaranteed to be open.

That is normal for realtime systems. The dangerous part was what happened when the ML service was unavailable, slow to handshake, or stuck behind a bad connection.

Bug Fix or Performance Improvement

The MlSession class buffered audio frames while waiting for the ML WebSocket to open:

sendAudio(pcm: Buffer) {
  if (this.closed) return;
  if (this.open) this.ws.send(pcm);
  else this.queue.push(pcm);
}

That looks harmless until you remember what the data is: raw PCM audio.

At 16 kHz, mono, 16-bit PCM, the stream is roughly 32 KB per second per active speaker. If the ML WebSocket never opens, every audio event keeps appending another Buffer to queue. A stalled ML service could quietly turn one active audio source into an unbounded memory growth path.

The server did not have to be under heavy traffic for this to matter. One broadcaster with a stuck ML backend was enough to create an ever-growing in-memory queue.

Code

I added an explicit cap for queued audio before the ML connection opens, tracked queued bytes, reset the counter once the connection opens, and released queued buffers on close/error.

const MAX_QUEUED_AUDIO_BYTES = Number(
  process.env.ML_MAX_QUEUED_AUDIO_BYTES || 512 * 1024,
);

The send path now fails fast instead of buffering forever:

sendAudio(pcm: Buffer) {
  if (this.closed) return;
  if (this.open) this.ws.send(pcm);
  else {
    if (this.queuedBytes + pcm.byteLength > MAX_QUEUED_AUDIO_BYTES) {
      this.finish(new Error("ML websocket audio queue overflow before connection opened"));
      try {
        this.ws.close();
      } catch {}
      return;
    }
    this.queue.push(pcm);
    this.queuedBytes += pcm.byteLength;
  }
}

When the socket opens, queued audio is flushed and the byte counter goes back to zero:

this.ws.on("open", () => {
  this.ws.send(JSON.stringify({ type: "config", ...config }));
  this.open = true;
  for (const buf of this.queue) this.ws.send(buf);
  this.queue = [];
  this.queuedBytes = 0;
});

Close/error paths also clear the queue:

close() {
  this.closed = true;
  this.queue = [];
  this.queuedBytes = 0;
  try {
    this.ws.close();
  } catch {}
}

private finish(err?: Error) {
  if (this.closed) return;
  this.closed = true;
  this.queue = [];
  this.queuedBytes = 0;
  this.onClose(err);
}

My Improvements

The fix changes the failure mode.

Before:

• ML service unavailable or stuck
• browser keeps streaming audio
• server queues every PCM frame
• memory grows until the process is restarted or killed

After:

• ML service unavailable or stuck
• server accepts only a bounded amount of pending audio
• session closes with a specific overflow error
• existing gateway cleanup runs and listeners get the normal ended signal
• memory is released immediately

I kept the default cap at 512 KB, which is around 16 seconds of 16 kHz mono PCM audio. That is enough for normal connection delay but small enough to prevent a single stalled session from becoming a memory leak.

The value is configurable:

ML_MAX_QUEUED_AUDIO_BYTES=1048576

That lets production tune the buffer based on expected network conditions without changing code.

Verification

The normal server build still passes:

npm run build

Output:

> [email protected] build
> tsc -p tsconfig.json

I also ran a smoke test that accepts the TCP connection but intentionally never completes the WebSocket handshake. Then I sent audio chunks into MlSession with a small queue limit.

Expected result: the session should close with the overflow error instead of keeping the buffers forever.

Observed result:

{"closed":true,"errMessage":"ML websocket audio queue overflow before connection opened"}

That is the behavior I wanted: when the ML backend is not actually ready to receive audio, the realtime server fails the session explicitly instead of becoming a quiet memory sink.

What I learned

Realtime systems make small buffers feel safe.

This was not a dramatic bug in the happy path. The app worked when the ML WebSocket opened quickly. The bug lived in the waiting room between "the browser is already sending audio" and "the ML service is ready."

That is exactly where production bugs like to hide: not in the main algorithm, but in the glue code that assumes the next component will be ready soon.

The fix was not a bigger abstraction. It was a boundary:

audio may wait briefly, but it may not wait forever.

That one rule turns an unbounded failure into a controlled one.