Why WebSockets Break Telephony Voice AI: Building a 60ms Barge-In Loop on Raw 8kHz RTP
Author: Wang Guo (@wangguode) Source Lab: https://github.com/wangguode/ai-agent-voice/tree/main/01-voice-agent-rtp-bargein Stack: FreeSWITCH 8kHz Raw RTP / Pipecat Full-Duplex / SenseVoice / SOXR VHQ Resampling
Author: Wang Guo (@wangguode)
Source Lab: https://github.com/wangguode/ai-agent-voice/tree/main/01-voice-agent-rtp-bargein
Stack: FreeSWITCH 8kHz Raw RTP / Pipecat Full-Duplex / SenseVoice / SOXR VHQ Resampling
1. The Telephony Gap: Browser Demos vs. PSTN Reality
Most Voice AI tutorials work great over a clean laptop microphone in a quiet room using browser WebSockets. The moment you connect that same pipeline to real enterprise telephony (PSTN / SIP trunks / FreeSWITCH), three brutal physical realities hit at once:
-
The 8kHz vs. 16kHz Sample-Rate Wall: Traditional PSTN/SIP networks stream
8kHzG.711 (PCMU/PCMA) orL16audio, while modern acoustic models (SenseVoice, Whisper, Silero VAD) expect16kHzPCM. NaΓ―ve linear interpolation introduces aliasing artifacts that tank ASR accuracy and cause false VAD triggers. - TCP Head-of-Line Blocking: Tunneling continuous 20ms telephony audio frames over TCP WebSockets creates packet-queue stalls under cellular jitterβturning a 50ms interruption into a 700ms cross-talk collision.
- Echo & Unflushed Outbound Ring Buffers: Even when your backend decides to stop speaking, an unflushed 20msβ200ms outbound RTP playback queue keeps talking over the caller.
2. The 60ms Enterprise RTP Barge-In Architecture
In our open-source lab (01-voice-agent-rtp-bargein), we bypassed generic JSON-over-WebSocket wrappers and engineered a deterministic telephony loop:
[PSTN / SIP Caller (8kHz RTP, 20ms frames)]
β
βΌ (UDP Raw RTP)
[FreeSWITCH Media Gateway]
β
ββββΊ [SOXR VHQ Upsampler (8kHz β 16kHz)] βββΊ [SenseVoice / Acoustic VAD (~18ms)]
β β
β βΌ (Barge-In Triggered!)
ββββ [256-Frame Outbound Ring Buffer] βββββ [0.16ms clearAudio() Hard Flush]
Key Architectural Primitives
- 20ms Raw RTP Frame Alignment: Audio arrives in strict 20ms packets (160 samples at 8kHz). Processing directly on frame boundaries avoids arbitrary chunk buffering delays.
-
SOXR VHQ Zero-Artifact Resampling: Using SoX Resampler (
SOXR_VHQ) for8kHz <-> 16kHzconversion preserves vocal formants and prevents high-frequency aliasing from poisoning the local VAD classifier. -
256-Frame Outbound Ring Buffer + 0.16ms
clearAudio(): Rather than letting TTS audio accumulate in an uncontrolled OS socket buffer, outbound TTS frames sit in a fixed 256-frame circular buffer. The instant acoustic barge-in fires,clearAudio()resets the read/write pointers in0.16msβproducing immediate silence on the caller's phone.
3. Why Decoupling Interruption from Transcription Matters
A common mistake in Voice AI is waiting for the LLM or full-sentence STT transcript to confirm "the user is interrupting". Transcription requires context windows (often 200msβ400ms).
By splitting the pipeline into two asymmetric loops:
-
Fast Loop (~50β60ms): Acoustic VAD + energy/formant gate β triggers immediate
clearAudio()buffer flush. - Semantic Loop (~200β400ms): Full ASR transcript β feeds the LLM context & updates turn state (discarding false backchannels like "uh-huh" or "yeah" without permanently derailing the agent's thought).
4. Explore the Open-Source Lab
All reference code, architecture diagrams, and runnable barge-in experiments are open-sourced:
- RTP Barge-In Lab: https://github.com/wangguode/ai-agent-voice/tree/main/01-voice-agent-rtp-bargein
- Web & macOS Companion (HardCall 52ms): https://github.com/wangguode/hardcall
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.