ZONOS3-Mini’s 80 ms TTFA and 15 Hz Codec — Dual-Track WASAPI Still Owns the Overlay Latency Budget

On 12 November 2026 Zyphra released ZONOS3-Mini, an Apache-2.0 sparse MoE TTS model with roughly 800 M total parameters and only about 100 M active per token. It uses a custom 15 Hz dual-codec (ZAC-1) that packs 44.1 kHz audio into 10 codebooks of 4 096 entries (one semantic from Wav2Vec2-BERT plus nine acoustic), then renders frames with a causal multi-resolution iSTFT head. Reported numbers: 80 ms median time-to-first-audio on a single H100, under 150 ms on an M3 Air, under 570 ms in-browser via WebGPU on a Pixel 9, and peak throughput around 457× real-time. It ships 27 default voices across 43 languages and supports zero-shot cloning from a ≤60 s reference or a 192-d speaker embedding.

Those numbers are real and useful for anyone trying to keep a full speech stack on-device. They do not move the first millisecond of a live Windows overlay that has to hear both sides of a call, turn the remote party’s audio into text, and optionally paint a short translated reply or suggestion before the next sentence finishes.

The capture and render tax still dominates

A 100 M active TTS that can start emitting audio in under 100 ms is almost free once the text is ready. Getting clean, timed samples into the STT and then getting the synthesized reply out without glitches is not:

1. Endpoint choice — USB headset, Bluetooth HFP/A2DP, built-in, or exclusive-mode WASAPI.
2. Dual-track acquisition: a dedicated capture client for the local mic plus a render-loopback client so the remote party’s audio stays distinct.
3. Windows Audio Engine (audiodg.exe + audioeng.dll) in shared mode. Mixing, communications enhancements and vendor APOs routinely add 10–40+ ms and become non-deterministic under load.
4. Consumer thread: lock-free SPSC ring copy, format conversion, resampling, VAD.
5. Streaming STT (local GGUF or remote) plus optional local model polish for translation or short replies.
6. Local TTS (ZONOS3-Mini via ggml/llama.cpp or a smaller model) writing into a render client.
7. Overlay paint on a dedicated thread that never blocks the audio callback.

Bluetooth HFP alone can inject another 50–150 ms of buffering that no tokenizer or codec speed recovers. Even if the STT and TTS both run locally and finish in well under a frame, the overlay still waits for the Windows path that delivered the samples and the path that will play the reply.

A minimal shared-mode capture callback that stays under 1 ms and does no allocation looks like this:

// WASAPI shared-mode capture — callback must be allocation-free
static HRESULT STDMETHODCALLTYPE CaptureCallback(IAudioCaptureClient *pCapture) {
    UINT32 packetLength = 0;
    BYTE *pData = NULL;
    DWORD flags = 0;
    HRESULT hr = pCapture->GetBuffer(&pData, &packetLength, &flags, NULL, NULL);
    if (FAILED(hr) || !packetLength) return hr;

    // Writer side of a pre-allocated power-of-two SPSC ring (float32 frames)
    ring_push_float32(capture_ring, (const float *)pData, packetLength);
    pCapture->ReleaseBuffer(packetLength);
    return S_OK;
}

The consumer owns resampling (libsamplerate or a fixed polyphase), VAD, and the write to the model or WebSocket. Keep the audio thread pure; any lock or malloc here shows up as glitches or higher DPC latency.

Dual-track still matters more than TTS speed

A meeting bot that joins the call receives mixed audio after the conferencing platform has already done its own mixing and jitter buffering. A local overlay that needs speaker-separated, low-latency audio for an interview or support call does not have that luxury. You need:

• Mic track: local speech (candidate or agent).
• Loopback track: remote party (interviewer or customer).

If you keep them separate, you can send the remote-side track to STT with any agent text as context, and the mic track to a different stream or the same model with different biasing. Local models (faster-whisper, Whisper.cpp, or the smaller Nemotron/Granite speech checkpoints) avoid the network hop entirely. When the reply is short enough, you can feed it straight into a local TTS such as ZONOS3-Mini running through llama.cpp’s new /v1/audio/speech endpoint or a custom ggml runner.

Trade-offs that actually show up in the wild

• Shared-mode WASAPI is convenient and works with every headset; exclusive mode or a virtual cable drops the engine tax but breaks system sounds and requires careful format negotiation.
• Bluetooth HFP gives you the mic and the remote audio on the same endpoint at the cost of 50–150 ms and mono 8/16 kHz. A2DP is better for playback but usually does not expose a clean loopback.
• Local GGUF STT + local TTS eliminates the round-trip but still has to share the CPU/GPU with the rest of the desktop. On a laptop under thermal limit the first token can still miss the 200 ms window that feels live.
• Overlay rendering must stay off the audio callback. A single UI thread that also does D2D or DirectComposition is fine; anything that blocks the message pump is not.

ZONOS3-Mini (and the llama.cpp 1.0.0 /v1/audio/speech surface that landed the same week) makes the generation side cheap enough that the remaining work is almost entirely in the Windows audio graph and the dual-track acquisition. That is the floor that still sets whether a language overlay feels instantaneous or just another delayed caption.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.