I Built a Fully-Local Voice AI Assistant with a 3D Avatar — Here's What I Learned

For the past few months I've been building a personal AI assistant called Xiaoqingting ("Little Dragonfly"). Not a chat wrapper — a full desktop assistant: voice in, voice out, tools that actually touch my machine, multiple agents with different jobs, and a 3D anime avatar that talks to me.

Everything important runs locally. The LLM is the only cloud component.

It's open source: github.com/lujianfu2008/xiaoqingting

This post is the architecture tour and the honest lessons list — including the bugs that cost me full nights.

Architecture: Two Processes and a Contract

┌─────────────────┐   REST + WebSocket   ┌─────────────────┐
│  @blm/desktop   │ ◄──────────────────► │   @blm/server   │
│                 │                      │                 │
│ Electron client │                      │  Fastify + SDK  │
│ voice / chat /  │                      │  agents / auth  │
│ tools / browser │                      │  gateway / keys │
└─────────────────┘                      └─────────────────┘

Two packages, one monorepo:

• @blm/desktop — the Electron client. Voice capture, conversation UI, the three-vrm avatar, and a browser extension for voice dictation on any web page.
• @blm/server — Fastify server wrapping the Claude Agent SDK. It owns agents, skills, tool approval, multi-user auth (JWT), and an LLM gateway with per-user quotas.

Every message between them goes through a shared protocol package (@blm/protocol) validated with Zod. If I could give past-me one piece of advice, it's this: define the protocol as a package on day one. Every refactor since has been cheap precisely because both sides fail loudly and at compile time when the contract drifts.

Lesson 1: Local-First Is an Architecture Decision, Not a Feature

Early on I had to decide where ASR, TTS, and wake-word detection live. The lazy answer is "call a cloud API for everything." I chose local:

• ASR: sherpa-onnx Paraformer (streaming, int8, ~79 MB model) — fully offline, Chinese recognition works well
• TTS: Windows SAPI locally, streamed in chunks so the avatar starts speaking before the full audio is ready
• Wake word: a small keyword-spotting model, with wake words generated at runtime from pinyin so homophones work automatically

Was it worth the extra work? Absolutely, for three reasons: latency (no network round-trip before the avatar starts talking), privacy (my voice never leaves the machine), and cost (voice calls to cloud APIs are surprisingly expensive at conversational volume).

The one thing I'd redo: I underestimated model file logistics. 79 MB binaries don't belong in git, so the repo ships without the model and the user downloads it from sherpa-onnx releases. Write that setup step clearly in your README or your issues tab will fill up.

Lesson 2: Multi-Agent Doesn't Need a Framework

The assistant is actually several agents: assistant (chat + voice), coder (gets shell/file tools), writer (long-form, no destructive tools). Different system prompts, different tool allowlists, same SDK underneath.

I never needed LangGraph-style graphs for this. The whole thing is:

const agents = {
  assistant: { tools: [...safeTools], prompt: PERSONA },
  coder:     { tools: [...devTools],   prompt: CODER_PERSONA },
  writer:    { tools: [...readTools],  prompt: WRITER_PERSONA },
};

What actually made it feel like "multiple agents" was tool gating per agent — the coder can run commands, the writer can't touch anything destructive. If you're building multi-agent today, spend your complexity budget on permission boundaries, not orchestration graphs.

Related: the server's default posture for tools is deny unless allowlisted, and the engine never receives secrets from the server environment. An assistant with shell access is a remote-code-execution vector wearing a cute avatar. Treat its tools like you'd treat an API's public endpoints.

Lesson 3: Long Conversations Will Stall, So Watch Them

This one cost me a night. Occasionally an agent turn would simply produce nothing — the SDK had burned its budget and the CLI process went silent without emitting an error. The user stared at a spinner forever.

The fix is a turn watchdog: when a turn starts, set a deadline. If no message arrives by the deadline, kill the engine process hard and tell the user what actually happened ("this turn's compute budget ran out") instead of showing a zombie spinner.

Two details that matter:

• The watchdog deadline does not extend when the user impatiently clicks "are you there" — a restarted timer is a lie
• After aborting, force-tear down the socket after 10 seconds; graceful close can hang on a dead process

I wrote up the other half of this reliability story — zombie WebSocket connections and heartbeat design — in my previous post.

Lesson 4: The Avatar Will Betray You (three-vrm Edition)

The 3D avatar is a three-vrm model in a WebGL scene. The trap: VRM 0 models face away from the camera by convention, and everyone's first fix is vrm.rotation.y = Math.PI. Mine promptly turned its back to the camera the first time an animation clip overrode the transform.

The lesson generalizes: when a scene graph fights you, find out who owns the transform. Animations own it during playback; your rotation fix must happen inside the same update loop, after the animation, not once at load time. Also:/animations for idle/breathing sell the "alive" feeling more than lip-sync quality does — get a subtle idle loop in before polishing anything else.

Lesson 5: Protocol Mismatches Fail Weirdly

The LLM gateway accepts OpenAI-compatible upstreams, and the most confusing bug of the whole project looked like this: a perfectly valid request, upstream answers "model not found." The model name was fine. The real problem: the client appended ?beta=true to the URL, the upstream router keyed its model registry on the full path including the query string.

Strip queries before routing. And if your upstream speaks a protocol that returns assistant messages as separate text and tool_use chunks, merge them — some routers choke on the split form. Protocol bugs don't look like protocol bugs; they look like the upstream lying to you.

What I'd Do Differently

1. Ship the browser extension earlier. Voice-dictating into any web page turned out to be the feature I personally use most, and it was added last
2. Budget display from day one. Per-turn cost, per-user quota — retro-fitting usage analytics onto a system that's already live is miserable
3. Record architecture decisions. I have a CLAUDE.md with design notes, and every time I onboard my future self to a subsystem I thank past me for it

Try It

The repo is here: github.com/lujianfu2008/xiaoqingting — Node 20+, npm install, configure .env, and you have a local assistant. Star it if you like the architecture; issues and PRs welcome.

Next post: the skills system — how the assistant discovers skill packs at runtime and injects them into its own system prompt. Follow me so you don't miss it.