Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 5 min read

How I built spoken controls for my desktop AI companion: 7 phrases mean stop, and none of them cancels your task

Say "stop" to a voice assistant and it should stop talking. It should not cancel the job it was telling you about, approve the thing waiting for approval, or decide you meant the shop down the road. Ankita AI, the open-s

Say "stop" to a voice assistant and it should stop talking. It should not cancel the job it was telling you about, approve the thing waiting for approval, or decide you meant the shop down the road. Ankita AI, the open-source desktop AI assistant by Krish Verma, now has spoken controls that run without a single planner call β€” and the whole design is one rule: a control word only counts when it is the whole utterance.

Ankita lives at ankita-sable.vercel.app, and the code is open source at github.com/akyourowngames/A.N.K.I.T.A. This is the story of the smallest feature that could still ruin your day if it misfires.

The problem: "stop" is a word, not a command

Voice controls sit in the worst possible place: between a microphone and a to-do list. The assistant is reading a progress report aloud, and you say "stop". Three things could go wrong, and two of them are silent. The speech stops but the task keeps running with nobody watching it. Or the word "stop" inside a longer sentence β€” "stop by the store on the way home" β€” gets treated as an interruption and the report dies mid-sentence. Or "pause Spotify" gets swallowed as a control command instead of reaching the planner that actually knows what Spotify is.

The usual fix is a substring match with a confidence threshold. I did not want a threshold. A threshold is a guess with a number on it, and a wrong guess here either kills work or ignores the user. So the controls are exact phrases, matched against the whole utterance, in a table. Nothing fuzzy, nothing scored.

The mechanism: a table of exact phrases

Everything lives in desktop/shared/voice.mjs, in three phrase sets. The matching is deliberately boring:

  • LOCAL_CONTROL_PHRASES β€” 18 phrases across five actions: resume ("continue", "continue speaking", "continue the report", "resume speaking", "resume the report"), repeat ("repeat that", "repeat the report", "say that again"), skip ("skip the report", "skip this report", "skip that report"), stop speaking ("stop speaking", "stop talking", "be quiet"), and pause ("pause listening", "pause the microphone", "mute the microphone", "mute your microphone").
  • VOICE_STOP_PHRASES β€” 7 whole-utterance stop phrases: "stop", "stop it", "stop that", "cancel", "cancel that", "never mind", "nevermind". The comment above the set says it plainly: a command that merely contains one of these words ("stop by the store") is not a stop, so matching is exact, not a search.
  • VOICE_END_PHRASES β€” 9 phrases that end the voice session: "exit", "quit", "end", "end voice", "exit voice", "leave", "leave voice", "goodbye", "bye".

voiceCommand() checks the end phrases first, then the stop phrases, then the local-control table. The wake word is a single lowercase token, ankita β€” the only phrase that wakes standby listening. The phrase maps are exact vocabulary, not substring patterns, by design.

Mixed requests never reach the table at all. "Pause Spotify" stays a normal planner request, because it names a target the controls do not own. A bare "continue" with nothing to continue also stays with the planner instead of guessing. The controls only fire when the utterance is one of theirs, in full.

The safety line: speech controls never touch tasks

The sharpest line in the code is a comment on VOICE_SPEECH_ACTIONS: "Separate speech control from task cancellation." Stopping speech is not stopping work, and the code treats them as different verbs in different layers.

desktop/electron/voice-actions.mjs validates the action payload's shape and bounds, and the audio owner checks the session, the interruption turn, and that the report actually exists before anything happens. Speech controls never cancel a task and never approve pending work β€” that is stated as a rule in the implementation, not as a hope. A temporary microphone pause keeps the unfinished report around for the Resume microphone button, and suspended speech stays selectable ahead of other queued reports, so pausing mid-report does not lose your place.

The panel got the same treatment: separate Stop speaking, Continue, Repeat, and Skip report controls, a truthful speaker-fallback and headphone-interruption message instead of a generic error, and an explicit hint when the microphone is paused. If the mic is paused, it cannot hear "resume" β€” and the UI says so rather than pretending otherwise.

What "local" actually means

"Local" here means zero planner calls. Status reads the real voice jobs and results without asking the model anything; pause, stop, continue, repeat, and skip all run before delegation. A control that needs a model round trip is not a control, it is a request with a fast path.

But I want to be honest about the word. Spoken controls still wait for the transcription service, and any new spoken status still needs speech synthesis. "Local" describes the decision, not the audio path. The finding doc records that plainly, and its confidence score for the whole change is 92 out of 100 β€” docked for exactly the things that were not tested: no physical microphone or headset trials, no packaged-desktop check, no full-repository suite run.

Verification: 124 tests, 0 work cancellations

The numbers that made me trust it: 124 out of 124 focused regression tests pass across the voice command, state, engine, UI, report-control, and speech-queue suites. 3 out of 3 renderer and service round trips pass, using real React components, injected microphone and speech synthesis, disk task receipts, and the actual speech queue β€” and the controls add zero planner calls. One spoken stop-and-continue integration test records exactly one original planner request while stop, continue, skip, and repeat operate on its real report, with zero work cancellations. Typecheck exits 0, and the whitespace gate passes.

The regression story is the part I like most. Before the implementation, the focused checks were 1 pass and 3 fails: the local matcher did not exist, and a status question made a second planner request. After it, 4 out of 4. A queue-focus regression first failed because a queued report hid the Continue button; it passed once suspended reports were prioritised. The tests caught real behaviour, not just typos.

Honest limits

Speakers cannot hear interruptions during playback, so "stop" while audio is playing is a race the user loses. A paused microphone cannot hear "resume" β€” the button is the only way back. The vocabulary is English only; anything unmatched, or in another language, falls through to the normal planner. And the 124 tests are focused regressions, not a full-suite run on a packaged desktop.

Tell me I'm wrong

Exact-phrase matching is the laziest safe design I could ship: it understands "stop speaking" and nothing else. A substring matcher would understand "could you stop talking please" β€” and would also kill a report when you say "stop by the store on the way home". I chose the table because a missed control is a shrug and a wrong control is lost work. But maybe I have the asymmetry backwards: is a voice assistant that only understands seven ways to say "stop" actually controllable, or is it just a polite way of not listening?

β€” Krish Verma, creator of Ankita AI

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.