Deepgram Nova-3 + Cartesia Sonic 3 + GPT-4o: Where 750ms of Voice AI TTFB Actually Hides
Voice-AI conversations die the moment the agent takes too long to say the first word. Not the full answer. The first byte of audio. The industry calls this TTFB (time to first byte), or TTFA (time to first audio). I mea
Voice-AI conversations die the moment the agent takes too long to say the first word. Not the full answer. The first byte of audio.
The industry calls this TTFB (time to first byte), or TTFA (time to first audio). I measure it from the moment the user stops speaking to the moment the first audio frame plays. Everything else, including total answer length and follow-up quality, happens after the user has already decided whether the agent feels alive.
Below is a ~750ms streaming budget on a Deepgram Nova-3 + GPT-4o + Cartesia Sonic 3 stack, broken into what each hop actually costs. And where almost every vendor-quoted number stops matching the number you will measure in production.
1,200ms to 750ms is the easy part
Batch pipelines wait for the full response before TTS starts:
[User stops speaking] β STT β LLM (full answer) β TTS (full) β [play]
^ >1,200ms
Streaming pipelines hand sentence 1 to TTS the moment the LLM emits it, while sentences 2 and 3 are still generating:
[User stops speaking] β STT β LLM (sentence 1) β TTS (sentence 1) β [play]
LLM (sentence 2) β TTS (sentence 2) β [continue]
LLM (sentence 3) β TTS (sentence 3) β [continue]
On the stack above, the streaming budget breaks down roughly like this:
| Hop | Time | Notes |
|---|---|---|
| STT (Deepgram Nova-3 or Flux) | 350ms | includes end-of-turn detection on Flux |
| LLM first sentence (GPT-4o) | 150ms | first-token latency, best-case |
| TTS first chunk (Cartesia Sonic 3) | 170ms | first audio frame |
| Network round-trips | 80ms | user β server + server β vendor |
| Total (streaming TTFB) | ~750ms |
The 1,200ms to 750ms move pays for itself: no code change beyond "do not wait for the full response." The hard part is everything below 750ms.
STT: Nova-3 is 300ms, Flux adds end-of-turn for about 260ms
Deepgram's public numbers, cross-referenced with Softcery's 2026 comparison:
- Deepgram Nova-3: streaming latency under 300ms, 36 languages. General ASR; does not detect end-of-turn.
-
Deepgram Flux: around 260ms end-of-turn, English only. Flux folds ASR together with turn detection into a single model and emits
StartOfTurn,EagerEndOfTurn,EndOfTurnevents (Deepgram launch post).
The combination matters. Nova-3 transcribes, but you still need a separate endpointing step to know when the user is done. Flux folds endpointing in, which cuts a 400-500ms round-trip, but costs you multi-language support.
For reference across other vendors: OpenAI gpt-4o-transcribe is around 320ms with 100+ languages. AssemblyAI Universal-3 Pro Streaming is under 600ms. ElevenLabs Scribe v2 Realtime is around 150ms with 90 languages.
Each of these is a vendor-quoted number. None of them is what you will measure from Tokyo over LTE to a US-East endpoint.
LLM: GPT-4o claims 200-400ms, measured 719-1,608ms
This is the hop where the vendor-vs-real gap gets ugly.
Deepgram's own blog post listing reasons they chose GPT-4o for voice agents cites a first-token latency of 200-400ms. The same post's measured table, on a MacBook M1 Pro over Wi-Fi, reports GPT-4o first-token times of 719ms to 1,608ms. That is 2x to 5x the vendor's number, on a laptop, on consumer Wi-Fi.
A separate Dev.to benchmark from CloudX tested 30+ stacks and found GPT-4.1-nano landing at 0.30-0.40 seconds on 2nd-turn-onward, where the KV cache is warm. First turn is always worse.
What the numbers teach: first-token latency is dominated by TTFT variance, which is dominated by model size, GPU type, quantization, input length, KV cache hit rate, inference backend load, and cold starts. A table of "model X equals N ms" is strictly less useful than one measurement from your own region to your own endpoint.
Running a small LLM locally removes the LLM-side network round-trip entirely. On a tuned setup, that is where sub-500ms TTFB actually lives.
TTS: Cartesia Sonic 3 says sub-90ms, measured median is 166-190ms
Cartesia's two models get swapped in half the voice-AI blog posts:
- Cartesia Sonic 3: sub-90ms (vendor claim, measurement definition undisclosed), 40+ languages, emotion support.
- Cartesia Sonic Turbo: 40ms (vendor claim, speed-specialized model), 9 languages.
The widely-circulated 40ms number is Sonic Turbo, not Sonic 3. Real Sonic 3 measurements:
- Cartesia's own launch disclosure: median around 190ms.
- Vapi's Humanness Index (50-run test, round-trip included): median 166ms.
Both numbers are roughly 2x the vendor-quoted sub-90ms. Neither 166 nor 190 is "wrong." They measure different things: vendor model-inference vs client-side round-trip.
ElevenLabs Flash v2.5 quotes around 75ms, with their own docs noting that 75ms refers to model inference only, and that end-to-end latency will vary with your location and endpoint type (ElevenLabs latency docs).
Research on streaming TTS reaches sub-50ms TTFA (arXiv 2508.04721). But the same paper's own end-to-end telecommunications voice agent, with the LLM in the loop, measured 678ms average TTFA. The TTS was fast. The stack still was not.
The 3-level optimization ladder
Treating these numbers as a budget, the ceiling moves in three steps.
Level 1, 500ms target: fast TTS + sentence-level streaming + streaming STT. This is the "do not wait for the full response" move above; gets you from 1,200ms+ to roughly 500-750ms.
Level 2, 300ms target: Level 1 plus Flux-style end-of-turn detection, speculative LLM invocation on partial ASR, a smaller LLM, and a TTS whose measured (not quoted) first byte is under 100ms. The emphasis is on measured.
Level 3, 200ms target: edge inference, a unified STT+LLM+TTS model, hardware tuning. At this point you stop composing vendor APIs and start building.
The 300ms figure is not arbitrary. It is the UX cliff below which a voice agent starts feeling like a human conversation partner, as opposed to a very fast IVR. Everything above 500ms feels like an echo.
Three rules before you trust any TTFB number in a blog post
- Was the network included? ElevenLabs says the quiet part out loud (75ms is model inference only). Most vendor numbers do not.
- Which model? Cartesia's 40ms is Sonic Turbo; the Sonic 3 you probably want is 2x that.
- Who measured it? Vendor, third-party, or user-run on their own hardware? Each of the three routinely disagrees by a factor of 2.
Build your own measurement. Pick the stack above, or three alternatives on each row of the table, point at your actual region, record a 50-run distribution. The p95 is the number you should budget against. The vendor's median is marketing.
The full chapter this article compresses, including the TTS vendor-vs-measurement table and the 3-level ladder in more detail, is in The 300ms Threshold on Kindle. The book is built around voice-AI UX cliffs at 300, 500, and 800ms, and how to budget inside them.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.