Dev.to AI 🤖 Ai 👁 0 📖 3 min read

I let coding agents use my Android phone, then made one run on the phone's NPU

For the last six weeks I've been building Buddy, an open-source Android app where you tell an agent what you want ("reply to Sam that I'm late", "add a Switch 2 to my Amazon cart") and it reads the screen, taps and types

I let coding agents use my Android phone, then made one run on the phone's NPU

Buddy: Claude Code, Codex, OpenCode, Antigravity and Grok in a bubble on your Android phone

For the last six weeks I've been building Buddy, an open-source Android app where you tell an agent what you want ("reply to Sam that I'm late", "add a Switch 2 to my Amazon cart") and it reads the screen, taps and types until it's done. Here's how it works, and what I learned getting a model to run on the phone itself.

The setup

  • The phone tools. An accessibility service reads the screen as a tree of elements and does the taps, swipes and typing. Buddy serves that as about 50 tools over MCP on localhost: read_screen, tap, type_text, notifications, calendar, alarms, a headless browser, and so on.
  • The agents. Claude Code, Codex, OpenCode, Google's Antigravity CLI and Grok Build run inside the app, in an Alpine Linux userland (proot, no root). You sign in with your own subscription, or use an API key. Buddy has no server of its own.
  • Safety. Before anything is sent, posted, bought or deleted, the agent has to ask on an approval card, and Buddy also stops on buttons like that by itself. Every tool has an on/off switch, globally or per chat.
  • Background mode. With Wireless debugging, a small helper runs as Android's shell user and gives each chat a hidden virtual display. The agent works there while you keep using your phone.

Running the model on the phone

Cloud models do the real work, but I wanted a fully offline option. Buddy downloads llama.cpp's official Android build and runs it as a local OpenAI-style server. Some numbers from a OnePlus 12 (Snapdragon 8 Gen 3) with Gemma 4 E2B:

reading (tokens/s) writing (tokens/s)
CPU 128 12
GPU (OpenCL) 691 18
NPU (Hexagon) ~2,400 ~23

The NPU is the difference between "wait a minute" and "answers right away", and it works in airplane mode.

Things that bit me on the way:

  • You can't just exec a binary. With targetSdk 29+, Android won't run executables from the app's data folder. Starting the server through /system/bin/linker64 works.
  • The working directory matters. The plain (non-Snapdragon) build loads its CPU backends from the current directory. On a Pixel, that meant "no backends are loaded" until the server ran from its lib folder.
  • The prompt was the bottleneck. An agent's first request was ~24k tokens: the system prompt plus 50 tool descriptions. A small model rereads that every step. A "lite mode" with a short prompt and a dozen core tools, plus a more_tools call to load the rest, brought it to ~4k.
  • Thinking loops. llama.cpp turned on Qwen3.5's thinking by default, and a simple "write a title" request thought for 3,000 tokens. With --reasoning off, a step takes a second or two.
  • Context comes from places you don't expect. One user's chats hit 254k tokens on a 49k model. The cause was an MCP server in their own OpenCode config, whose tool descriptions came along with every request.

Honest limits: small on-phone models handle one-step requests (the battery, a timer, opening an app) and make mistakes on longer tasks, like tapping the wrong calculator key.

Distribution

Google Play rejected the app: "Accessibility API cannot be requested for performing autonomous tasks". So it's on GitHub and F-Droid. Then, when I added a read-only SMS and call log tool, Play Protect started blocking the APK as harmful: an accessibility app with READ_SMS looks exactly like banking malware. Those tools now read through the background-mode helper (content query as the shell user), and Buddy itself holds no SMS permission.

Try it

Android 11+, 64-bit. If you build something with it, I'd love to see it: https://github.com/ghorbelhamdi/buddy-android/discussions/77

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.