How I gave my desktop AI companion computer use: a Cua driver, fresh element tokens, and a Rust patch for owned menus

Ankita is a personal AI companion that lives on your desktop — "a little companion, a lot more possible" is the whole pitch. But there was a hole in that pitch you could drive a window manager through: she could chat, she could drive a browser, and she could run commands. She could not click things. Every desktop app on your machine was a place she could see on a screenshot but never touch. This week I closed that hole, and the integration taught me more about trust boundaries than about computer vision.

Keep the agent engine; outsource the hands

The decision that shaped everything: the model and the task loop stay in Ankita. Input, accessibility trees, and screen capture live in an external service — Cua Driver — connected over MCP as a server called ankita-computer. The driver does not call a model; it is a hand, not a brain. This matters because it keeps the number of agent engines at exactly one. There is no second decision-maker to align, audit, or gate.

Discovery is deliberately boring: probe the executable on PATH, then the official installer location relative to LOCALAPPDATA, read its machine-readable manifest (command, args, version) with a 5-second timeout and a 1 MB cap, and only expose the tools that version actually advertises and that Ankita's capability ceiling allows. Browser, maintenance, clipboard, and replay tools are explicitly excluded — the computer-use connection is native desktop capabilities only.

The targeting model: fresh tokens or nothing

The part of the design I'm proudest of is how the agent addresses UI elements. Every observation returns a snapshot with an id and element tokens; a bare control index is invalid. To type into a text box, the agent must present the fresh element_token and the current snapshot_id. After layout changes, old indices are dead. The prompt guidance is blunt about it: never invent coordinates, never reuse stale indices, never claim success from input delivery alone.

Target metadata gets its own protected budget — 24,000 UTF-8 bytes and at most 160 controls per observation — held outside the normal model-context trimming. If trimming would cut the tokens mid-JSON, the reply degrades to guidance text ("request a smaller fresh observation before acting") rather than shipping a partial token the model might hallucinate around. Screen content is untrusted data and cannot authorize actions; a successful set_value only means bytes arrived somewhere, so the agent must read fresh state or use the driver's verify_state before declaring victory. When background typing doesn't land, the fallback is explicit: bring the observed window forward, click the fresh text-box target, type in the foreground, read back.

The approval line

Connecting computer use is an explicit user act — "Connect Computer Use" — not something that happens on startup. Once connected, native desktop actions run without per-action approval prompts. But this is a narrow exemption, not a global one: it's an internal desktop-agent flag (desktopComputerUseDirect) scoped to this connection and its capability ceiling. Explicitly blocked tools stay blocked. The existing approval tiers still apply. And the guide draws one hard line I keep re-reading: login, password, and permission dialogs require the user's direct input — no automation, no workarounds, no exceptions.

The bug that required a fork

Then came the screenshot. A user (well, me, testing) drove Notepad: the animated arrow that follows Ankita's native actions was being painted behind Notepad's File menu. Native investigation showed why: Windows menus are separately-owned, topmost windows (#32768). The driver pinned its overlay above the driven window in the ordinary window band, so any owned topmost surface — menus, dropdowns — covered the arrow. This is a driver ownership/band bug, not renderer CSS, and saving could still work while the arrow was hidden.

So I did the thing you're never supposed to do and patched upstream. vendor/cua-driver/windows-owned-popups.patch (334 lines of Rust) teaches the overlay to follow the driven window's ultimate native ownership family, match the highest owned surface's current band, and demote back to the normal band after dismissal or loss of the driven family. The build script (scripts/build-computer-driver.mjs) clones the pinned upstream revision, applies the diff, runs the native overlay tests, and writes hash receipts: source diff, dependency lockfile, and executable — 3bb823627fcccc071918cdfd870897b715c9673704db2368b1ba62bd1bdf5ffb. The unchanged lockfile preserves the installed 0.20.0 MCP contract, the MIT license is retained, and a receipt file notes that a future official-driver update must retain the ownership correction before replacing this build — "a version number alone is insufficient evidence."

The verification was the fun part: a disposable Notepad document, real agent calls, independent Win32 layer inspection of the exact driver process. Before: menu z=13 topmost=true, overlay z=30 topmost=false — assertion failed as expected. After: overlay z=13 above menu z=14, foreground unchanged, the screenshot shows the complete arrow above the File dropdown, the poem got saved, and Stop left Notepad alive with its text.

What I won't claim

It's Windows-only. Install-from-scratch with a clean user profile, elevated apps, the secure desktop, mixed-DPI and multi-monitor setups are all unverified — the guide says so plainly. The hosted-model checks cover a requested Notepad task, not arbitrary desktop work. And this landed on main this week; it isn't in a release yet, so there is no version number on it.

The open question

The decision I most want argued with: once a user explicitly connects computer use, native actions run with zero per-action approvals — except login, password, and permission dialogs, which demand direct human input. No middle tier, no per-app policy, no "ask once per session" for destructive actions like deleting files or emptying the recycle bin. The existing approval tiers still gate other tools, and the capability ceiling is fixed, but inside this connection the line is binary.

Is that line right? My instinct says the explicit Connect is the meaningful consent moment, and per-action prompts would strangle the feature. But "empty the recycle bin with zero prompts" is a sentence I flinch at. Should the always-require-direct-input list grow — payment dialogs, file deletion, bulk operations — or is a longer list just paternalism that the user already answered by clicking Connect?

Repo: https://github.com/akyourowngames/A.N.K.I.T.A (MIT). Happy to be told my approval line is wrong — bring receipts.