One voice for all my coding agents
I usually have more than one coding agent running in separate terminal tabs (Pi and Claude Code). I don't like watching spinners or switching windows every minute to check if a task finished or if an agent is waiting for
I usually have more than one coding agent running in separate terminal tabs (Pi and Claude Code). I don't like watching spinners or switching windows every minute to check if a task finished or if an agent is waiting for input. It's much more comfortable if they just tell me out loud and I can reply by voice.
I made that work locally inside Pi: hold Space to dictate with faster-whisper, and a Piper's Spanish voice (es_AR "Daniela") reading short updates back.
That broke as soon as two sessions ran in parallel. Each session spawned its own voice worker, so when two agents finished together, both spoke on top of each other and I couldn't understand either of them.
So I pulled the voice layer out into a single local daemon that owns the speaker, the microphone, and one FIFO queue.
How I set it up
One Python process listens on 127.0.0.1:47830. Pi talks to it over HTTP and SSE, Claude Code connects through a small stdio MCP server, and any terminal can control it from a CLI. Whichever client starts first spawns the daemon in the background.
Instead of reading full replies aloud, agents use an "announce, then ask" flow through five MCP tools (avisar_usuario, voz_estado, voz_contar, voz_saltar, voz_descartar):
- An agent calls
avisar_usuario(aviso, detalle?)with a one-line headline and optional detail. - Daniela reads the headline and leaves the detail pending.
- From the terminal (or MCP) I can ask her to read the detail (
/contame), leave it for later, drop it unheard (/descartar), or cut whatever is playing right now (/saltar). - Holding Space to dictate pauses the queue so playback doesn't talk over me.
In a test with two processes announcing at the exact same instant, the first spoke from 1.1 s to 9.0 s and the second waited and ran from 9.0 s to 15.8 s.
Spanish voice with English terms
In Socra, I ended up dropping Spanish and sticking to English because the Spanish TTS mispronounced technical words.
For this daemon I kept Piper's es_AR-daniela-high voice. Before synthesis, speech.py strips code blocks, URLs, and commit hashes, matches words against a local glossary (pull request, commit, branch, backend, UI), phonemizes those matches with eSpeak's en-us phonemizer, and inlines them using Piper's [[ phonemes ]] syntax. I also prepend 120 ms of silence to every WAV because Windows clips the first milliseconds of short audio clips.
What broke along the way
A switched-off Bluetooth headset broke dictation
Unplugging my wired headset made dictation fail with PaErrorCode -9999. Windows still lists a paired Bluetooth headset (JBL WAVE100TWS over WDM-KS) even when it's turned off; my device picker preferred anything matching "headset" and had no fallback. On top of that, PortAudio only enumerates devices once per process, so a long-lived worker never noticed hardware changes. While inspecting the list I also noticed my Bluetooth-paired phone showed up as a Hands-Free HF Audio input that matched the same keywords.
I changed the recorder to re-enumerate devices on each recording, exclude the phone, and try candidates in order (headset, default mic, anything else). Falling back to the laptop mic now takes ~0.2 s.
Lowercasing acronyms and a bad benchmark
Transcription got worse over time. One bug I found right away was my post-processor applying the casing of an all-lowercase speech glossary to Whisper's output, turning UI into ui.
Dictation also runs Whisper base on CPU while another project of mine runs large-v3-turbo on GPU. I tried testing models by feeding Daniela's TTS output into Whisper, which turned out to be useless: Spanish TTS with spliced English phonemes trips up every model. Instead of guessing, the daemon now saves the last 20 real clips with device info so I can replay them across models and decide with real audio.
Localhost wasn't enough for a service with a mic
At first I only rejected requests carrying an Origin header or missing a custom X-Daniela-Client header. A security review caught the hole: with DNS rebinding, a web page becomes same-origin with 127.0.0.1, and browsers don't send Origin on same-origin GET requests. A page could have read pending notices or subscribed to the SSE stream.
I locked it down to:
- Reject any
Hostother than127.0.0.1:<port>orlocalhost:<port>, and rejectSec-Fetch-*headers. - Never broadcast dictated text over SSE.
- Write a fresh 32-byte random token on startup to
%LOCALAPPDATA%\daniela\token. Every endpoint exceptGET /hellorequiresAuthorization: Bearer <token>. - Before sending a token or text, clients call
GET /hello?nonce=<hex>and verifyHMAC-SHA256(token, nonce)in constant time, so another local process can't squat the port and steal the token.
One Windows networking quirk made the 403 test flaky: replying 403 and closing the socket without reading the request body makes Windows reset the connection, so the client sometimes saw a reset instead of 403. Draining the body (capped at 1 MB) before rejecting fixed it.
Where it stands
- Windows-only for now. Runs daily with Pi and Claude Code (
claude mcp add daniela ...); there's a Codex snippet in the README, not wired up yet. - Dictation still uses
faster-whisperbaseon CPU while I collect real clips to run the model comparison. - Still local in my workspace; I haven't published the repo yet.
AI disclaimer: I built this daemon with AI coding agents (Claude Code and Pi) under my direction and review, and AI helped me draft this post from my project notes. I edited it and every claim comes from the repository and its logs.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.