How I built an opt-in Modal (DeepSeek) provider for my desktop AI companion — then benchmarked it and kept the old one

Ankita AI is a desktop AI assistant by Krish Verma — a little companion that lives on your desktop and does things while you are away. Its voice replies felt slow, and DeepSeek V4.1 Flash on Modal looked faster on paper: cheaper tokens, a 1M-token window, and a dedicated deployment with no shared queue. So I built the provider, benchmarked it properly, and the benchmark said no. This is the story of a feature that shipped anyway — as an opt-in, not a default.

The plumbing first: one preset, no special cases

Ankita already had a Baseten provider hard-wired into a few places, so the first job was to stop special-casing providers at all. The Modal preset lives in src/core/provider.mjs as data, not a branch: base URL, model, key, and per-provider request defaults in one map. The voice brain and typed chat resolve base URL, model, and key from that metadata, so adding a provider means adding a row, not an if.

The preset itself: model deepseek-ai/DeepSeek-V4.1-Flash-1017, a 1,048,576-token window advertised by the deployment's /v1/models, a 400-token floor on output (reasoning needs room before any answer tokens), a 4,096-token default, and reasoning effort low — the floor, as it turned out. Auth arrives as two proxy token parts, MODAL_PROXY_TOKEN_ID and MODAL_PROXY_TOKEN_SECRET, joined into the one bearer token the endpoint expects; a combined MODAL_PROXY_TOKEN wins, so rotating the token means changing one variable. The desktop Settings dialog exposes the URL and token, and a 401 now names the expected token shape instead of failing silently. Voice keeps Baseten as the default; Modal is opt-in with VOICE_PROVIDER=modal.

The benchmark that flattered it — then the one that didn't

The trap was measuring the wrong thing first. On bare planner prompts — six spoken-style prompts, three rounds each, 18 requests per planner, 36 out of 36 succeeding — Modal looked like a clear win. Median time to first token: 408 ms against Baseten's 657 ms, 38% faster. Median full reply: 725 ms against 1,477 ms, 51% faster. But Modal produced 68 output tokens against Baseten's 236, so part of that win is a shorter answer, not faster generation.

Then I ran the real voice loop: scripts/verify-voice-latency.mjs drives the actual CLI voice loop with the real planner, the real agent prompt, all 30 tool schemas, and real Sarvam TTS. Six turns per planner. The result flipped completely. Transcript-to-audio-player, median: 2,593 ms on Baseten, 3,414 ms on Modal — 32% slower. The planner's share of that time was 67% slower. And it wasn't one bad sample: Modal's fastest real turn was 2,019 ms and its slowest 3,933 ms, while Baseten ranged 2,340–2,724 ms. The instability is the problem, not the median.

The trace explains why. A real voice request carries 11,793 prompt characters and 39,701 characters of tool schemas. Time to response headers — prefill plus queue, before any generated token — was stable around 1.1 seconds on Baseten across three turns (1,136 / 1,126 / 1,118 ms). On Modal it was 2,229 ms, then 669 ms, then 1,953 ms: fast when its prefix happened to be cached, much slower when it wasn't. The bare-planner win does not survive Ankita's tool catalogue.

I tried shrinking the payload. With an 8-tool, 8.1k-character subset instead of 22 tools and 16.3k characters, Modal still needed 1,662–1,717 ms to headers while Baseten needed 890–902 ms. Generation speed was never the bottleneck; prefill and queue capacity on the deployment is. One more deployment fact: reasoning_effort: "minimal" is rejected with HTTP 400, and passing reasoning_effort: 1 produced more reasoning than low — 78 tokens against 39 — so low stays the floor.

The decision: ship it, don't switch to it

So Modal shipped in Ankita v2.7.5 as an opt-in provider, and Baseten stayed the voice default. The acceptance bar is written down in the benchmark doc: enable prefix caching or add enough prefill capacity that the 30-tool payload reaches headers in a stable sub-1.1 seconds, re-run both benchmark scripts, and accept only if transcript-to-player improves — not just the bare planner.

The same pull request also gave the Plugins page an MCP tab: it searches the official MCP registry, installs a server from its pinned command, and lists, enables, starts, stops, and removes servers. The design choice that matters is what it does not do: DesktopMcp wraps the existing McpStore and McpManager rather than introducing a second execution path, so the approval gate is unchanged. Installing only records the command — starting it asks you to confirm that exact command. The app's own browser and computer-use servers are filtered out of the user-managed list, and remote-only registry entries surface their reason instead of failing.

What I'd argue about

Tell me I'm wrong: six turns per planner is a thin sample, and the Modal numbers swing by 2 seconds depending on whether a prefix was cached. Is the prefix-cache instability a real verdict on this deployment, or did I measure a cold deployment and call it a slow one?

Ankita is open source (MIT) — the provider preset, both benchmark scripts, and the full benchmark doc are in the repo at github.com/akyourowngames/A.N.K.I.T.A, and the project site is ankita-sable.vercel.app.

— Krish Verma, creator of Ankita AI