I let coding agents use my Android phone, then made one run on the phone's NPU
For the last six weeks I've been building Buddy, an open-source Android app where you tell an agent what you want ("reply to Sam that I'm late", "add a Switch 2 to my Amazon cart") and it reads the screen, taps and types
For the last six weeks I've been building Buddy, an open-source Android app where you tell an agent what you want ("reply to Sam that I'm late", "add a Switch 2 to my Amazon cart") and it reads the screen, taps and types until it's done. Here's how it works, and what I learned getting a model to run on the phone itself.
The setup
-
The phone tools. An accessibility service reads the screen as a tree of elements and does the taps, swipes and typing. Buddy serves that as about 50 tools over MCP on localhost:
read_screen,tap,type_text, notifications, calendar, alarms, a headless browser, and so on. - The agents. Claude Code, Codex, OpenCode, Google's Antigravity CLI and Grok Build run inside the app, in an Alpine Linux userland (proot, no root). You sign in with your own subscription, or use an API key. Buddy has no server of its own.
- Safety. Before anything is sent, posted, bought or deleted, the agent has to ask on an approval card, and Buddy also stops on buttons like that by itself. Every tool has an on/off switch, globally or per chat.
- Background mode. With Wireless debugging, a small helper runs as Android's shell user and gives each chat a hidden virtual display. The agent works there while you keep using your phone.
Running the model on the phone
Cloud models do the real work, but I wanted a fully offline option. Buddy downloads llama.cpp's official Android build and runs it as a local OpenAI-style server. Some numbers from a OnePlus 12 (Snapdragon 8 Gen 3) with Gemma 4 E2B:
| reading (tokens/s) | writing (tokens/s) | |
|---|---|---|
| CPU | 128 | 12 |
| GPU (OpenCL) | 691 | 18 |
| NPU (Hexagon) | ~2,400 | ~23 |
The NPU is the difference between "wait a minute" and "answers right away", and it works in airplane mode.
Things that bit me on the way:
-
You can't just exec a binary. With targetSdk 29+, Android won't run executables from the app's data folder. Starting the server through
/system/bin/linker64works. -
The working directory matters. The plain (non-Snapdragon) build loads its CPU backends from the current directory. On a Pixel, that meant "no backends are loaded" until the server ran from its
libfolder. -
The prompt was the bottleneck. An agent's first request was ~24k tokens: the system prompt plus 50 tool descriptions. A small model rereads that every step. A "lite mode" with a short prompt and a dozen core tools, plus a
more_toolscall to load the rest, brought it to ~4k. -
Thinking loops. llama.cpp turned on Qwen3.5's thinking by default, and a simple "write a title" request thought for 3,000 tokens. With
--reasoning off, a step takes a second or two. - Context comes from places you don't expect. One user's chats hit 254k tokens on a 49k model. The cause was an MCP server in their own OpenCode config, whose tool descriptions came along with every request.
Honest limits: small on-phone models handle one-step requests (the battery, a timer, opening an app) and make mistakes on longer tasks, like tapping the wrong calculator key.
Distribution
Google Play rejected the app: "Accessibility API cannot be requested for performing autonomous tasks". So it's on GitHub and F-Droid. Then, when I added a read-only SMS and call log tool, Play Protect started blocking the APK as harmful: an accessibility app with READ_SMS looks exactly like banking malware. Those tools now read through the background-mode helper (content query as the shell user), and Buddy itself holds no SMS permission.
Try it
- Code (Apache 2.0): https://github.com/ghorbelhamdi/buddy-android
- F-Droid: https://f-droid.org/packages/app.buddy.assistant/
Android 11+, 64-bit. If you build something with it, I'd love to see it: https://github.com/ghorbelhamdi/buddy-android/discussions/77
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.