Vercel agent-desktop: computer use for coding agents through the accessibility tree

Vercel Labs released agent-desktop this week, a free, open source tool that gives AI coding agents computer use on macOS, Windows and Linux. Chris Tate, who built Vercel's agent-browser, announced it on October 31 and described it as "agent-browser, for your whole computer." The interesting part is not that an agent can click buttons. It is how it sees the screen: through the operating system's accessibility tree, with no screenshots and no pixel matching.

What actually shipped

agent-desktop is a Rust CLI plus a C library, on npm as version 1.0.2 under Apache-2.0, with native binaries for x64 and ARM64. There is no language model inside it. Any agent that can run shell commands can drive it, so Claude Code, Codex, Cursor's CLI or a plain bash script all qualify. The command set covers observation, clicks and typing, window management, notifications, the clipboard and sessions, about 56 commands in total. Development started in February, led by Lahfir. Version 1.0 with Linux support shipped on October 28, and 1.0.2 followed on October 30.

Refs, not pixels

The core loop is observe, act, observe again:

agent-desktop snapshot --app Finder -i   # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8         # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i   # observe again to verify

A snapshot returns JSON with a ref for every element the agent can act on. The agent acts on a ref, then takes a new snapshot to check the result. Nothing is inferred from pixels. The tool reads the AX API on macOS, UI Automation on Windows and AT-SPI on Linux, so it sees the same UI structure a screen reader sees, with roles and states the OS already maintains.

Errors an agent can actually handle

Refs belong to the snapshot that produced them. When the UI shifts before an action lands, you do not get a wrong click, you get a machine-readable error: STALE_REF or AMBIGUOUS_TARGET. If several windows match, the error lists the candidate window ids in error.details, and you pass one back with --window-id. A bare @e8 without its snapshot prefix is rejected with INVALID_ARGS.

This is the part I would underline for anyone wiring agents to GUIs. Vision-based computer use fails silently: the model clicks the wrong coordinate and you find out downstream, if you find out at all. Here every failure mode has a code and a defined recovery: take a new snapshot, use its refs. Actions are safe to retry because refs stay stable, which is exactly the property you want in an unattended loop.

Dense apps: skeleton first, then drill down

A full accessibility tree of an Electron app is big. The skeleton traversal work merged into the project measured it: a Slack skeleton snapshot is about 3.6KB versus roughly 17.3KB for the full tree, a 78-96% token saving on dense Electron apps. The pattern is to map first, then drill in:

agent-desktop snapshot --skeleton --app Slack -i --compact
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact

The skeleton gives a shallow overview with child counts on truncated containers. --root starts a traversal from a previously discovered ref, so you only pay tokens for the region you are working in. For an agent on a token budget, this is the difference between a tool you can leave running and one you babysit.

Wiring it to an existing agent

Install is npm install -g agent-desktop. The binary carries its own docs, matched to its version:

agent-desktop skills get desktop    # the loop, refs, verification and retry rules
agent-desktop skills get platform   # what differs on the OS you run on

You can install the bundled skill into a harness with npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop. Two details worth knowing before you script it. Actions run headless by default: they go through the accessibility API without moving the pointer or stealing focus, and --headed opts into real input. And on PowerShell, quote every ref, because a bare @token is the splatting operator.

Platform reality check

The platform table in the repo readme is honest about gaps. macOS needs an Accessibility grant, and agent-desktop permissions --guide walks you through it. Windows needs no grant for apps at your terminal's integrity level. Linux needs at-spi2-core in your graphical session; mouse input is X11 only or --headed, the Wayland clipboard needs wl-clipboard, and notifications go through GNOME Shell. Apps without an accessibility tree are invisible to it, and it does not automate web pages, that is what agent-browser is for.

Take the data exposure seriously

Snapshots, screenshots and clipboard reads can contain private data, and the project's own docs warn against routing them into public logs. An optional helper called JEV sends screen descriptions to a configured model provider. Sit with that for a second: this tool reads your screen the way a screen reader does, and it hands what it reads to whatever agent invoked it. An agent with agent-desktop plus shell access effectively holds your user account.

I wrote about where that boundary sits when Nvidia moved agent execution into the kernel, and about the guardrail details to check before letting Copilot click your desktop. The same logic applies here: run it in a dedicated environment where the blast radius is a test account, not your daily driver.

When I would reach for it

Not for anything with an API. If the thing you are driving exposes an MCP server, a CLI or an HTTP endpoint, use that instead, it will be cheaper and more reliable. agent-desktop earns its keep on apps with no automation surface at all: installers, vendor tools, that one dialog box your CI cannot get past. The observe-act-observe loop with structured errors is the same verification discipline I want in every agent workflow, and now it extends to GUIs. That is the real news here, not the clicking.