Building a local voice agent pipeline: VAD faster-whisper streaming TTS — no cloud APIs
Cloud voice APIs are easy until you count the latency stack: STT round-trip + LLM round-trip + TTS round-trip, each adding 300–800ms. For real-time voice agents that adds up to a conversation that feels like a walkie-tal
Cloud voice APIs are easy until you count the latency stack: STT round-trip + LLM round-trip + TTS round-trip, each adding 300–800ms. For real-time voice agents that adds up to a conversation that feels like a walkie-talkie. I built a fully local pipeline — the same architecture inside my $20 voice-agent kit — that keeps everything on one machine.
The pipeline
mic → webrtcvad (speech detect) → faster-whisper (STT)
→ sentence-buffered LLM stream → Piper TTS → speaker
Four stages, each chosen for a specific reason:
1. VAD first, always. webrtcvad gates the audio before it hits the STT — without it you're transcribing silence and burning CPU. Speech detection at frame level, not utterance level.
2. faster-whisper over openai-whisper. CTranslate2 backend — roughly 4x faster than vanilla whisper at the same accuracy on CPU. For real-time you need the transcript before the speaker finishes the next sentence.
3. Sentence-buffered LLM streaming. The classic mistake: wait for the full LLM response, then TTS the whole thing. Instead, stream tokens and start TTS at the first complete sentence — TTS overlaps with LLM generation, hiding most of the latency.
4. Piper for TTS. Local, fast, sounds decent. Cloud TTS sounds better but adds the round-trip you're trying to avoid.
The honest numbers
I can't benchmark audio quality here (no audio hardware on this box — flagged honestly in the kit docs), but the architecture is sound: the latency budget is dominated by STT+LLM, and both run locally where network isn't the bottleneck.
What I can verify: python3 -m py_compile passes on every module, the CLI flags parse correctly, and the pipeline degrades gracefully when audio devices are absent.
When local voice makes sense
- Privacy: medical, legal, anything where audio can't leave the building
- Cost: at volume, cloud voice APIs are real money per minute
- Latency: sub-second response matters for conversational UX
- Offline: field work, airgapped environments
When it doesn't: you need broadcast-quality voices, or your hardware can't run whisper-sized models at usable speed.
The kit — full pipeline source, CLI flags, integration notes — is in the store:
Store (pay in BTC, no account, instant download): https://dev.to/nexusfoundry/nexus-store-live-link-auto-updated-1918 (auto-updated current store URL)
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.