The VAD that saves STT billed minutes also silences tab audio

Streaming STT bills whatever audio actually leaves the browser. On microphone capture, a small RMS gate on the main thread drops silent chunks before they reach the socket. Point the same threshold at tab audio or a system loopback and the whole signal looks like silence, so the transcript dies. The gate therefore runs only for a mic. As soon as a system stream is mixed in, it is skipped, and the noise goes to the vendor.

The capture path is the usual browser one: getUserMedia or getDisplayMedia, an AudioWorklet that emits PCM16 mono at 16 kHz, then small chunks over a WebSocket. The gate sits on the main thread and looks at one number per chunk, the RMS of the float samples before they are packed to PCM16.

The threshold is a noise floor, not a constant

A fixed RMS cutoff is what I tried first. Quiet speech and anyone sitting a bit far from the laptop fell through it. The gate is clamp(noiseFloor * 1.8, 0.0015, 0.02), with the floor as an EMA that only moves on chunks already classified as silence. 0.0015 is low enough that a quiet room does not demand a shout. 0.02 stops a loud room from ratcheting the threshold until normal speech fails it.

The first chunk seeds the floor

Leave noiseFloor at 0 and the gate opens at the absolute minimum, 0.0015. The first chunk in a room that is not dead quiet crosses that, the state machine enters speech, and the EMA never runs during speech. The floor stays 0. The gate latches open for the rest of the session.

if (this.noiseFloor === 0 && rms > 0) {
  this.noiseFloor = rms;
}

That runs before classification. The first audible chunk becomes the floor. Later silence chunks pull the EMA from there.

Pre-roll and trailing

Speech onset is late: RMS has to clear the gate, so the first syllables are already in chunks that looked like silence. A ring buffer keeps the last 5 silent chunks, about half a second, and flushes them when the state moves from silence to speech.

The other hole is the pause inside a sentence. RMS dips under the gate on a breath or a comma. Cutting there clips the tail. After speech, the gate stays open for 20 more chunks, about 2 seconds (trailing), then returns to silence. The chunk that ends trailing is not sent. It feeds the noise-floor EMA instead.

Tab audio is line level

Every constant above assumes a close mic. Tab audio and OS loopback are line-level. The same cutoff classifies that signal as silence, the socket goes quiet, and the transcript goes quiet with it.

Any system stream in the mix bypasses the gate. Mic plus system counts as system. Every chunk is sent, and the speech flag stays on, so silence and room noise are billed. I kept that cost. A muted transcript was the failure I was trying to avoid. The gate stays on only when the capture is the microphone by itself.

// Line-level (tab / system), not close-mic. Mic thresholds would mute the stream.
if (disableVAD) {
  emit();
  return;
}

hasSystemAudio is just systemStream !== null. No second threshold, no attempt to run this VAD on tab audio.

This runs in LiveSuggest, a browser assistant that streams the captured audio to STT while the conversation is still going. On a mic-only session the gate is what stops silence from being shipped. The moment a tab joins the mix, that same gate is the thing I turn off.