Dev.to AI 🤖 Ai 👁 0 📖 7 min read

I planned to copy Composio's "reactions" into my agent dispatcher. My logs said it would have caught 1 redo out of 19.

I run a homemade dispatcher called agent-dispatch. Claude plans the work and writes a work order. A cheaper model does the coding in a macOS sandbox, on its own git worktree and branch. Claude Sonnet reviews the diff rea

I run a homemade dispatcher called agent-dispatch. Claude plans the work and writes a work order. A cheaper model does the coding in a macOS sandbox, on its own git worktree and branch. Claude Sonnet reviews the diff read-only. Then I merge locally. It's about 14,000 lines of stdlib Python in one file, 124 commits since October 11, and since November 1 it has handled 64 orders.

This week I compared it with the mature tools: Composio's Agent Orchestrator, OpenRouter's new subagent and advisor server tools, Claude Code subagents, and the hosted "assign an issue, get a PR" agents (Codex cloud, Copilot coding agent, Jules). I made a list of four things to borrow. Then I checked the list against my own order logs, and half of it fell over.

What the mature tools do better

Composio's orchestrator is the closest match to mine: one git worktree per agent, a dashboard, and a YAML block called reactions. CI fails, the logs go back to the agent, it retries N times, and only then does a human get pinged. Review comments get the same treatment. Their pitch is "push, not pull": you hear about an order only when a decision is yours to make.

OpenRouter shipped two server tools that point in opposite directions. openrouter:subagent lets a strong model hand routine subtasks to a cheap worker. openrouter:advisor lets a cheap model ask a strong one for a single-turn second opinion mid-run. Both are beta and both are plain API calls. Neither touches a repo, a test suite or a branch.

Claude Code lets each subagent run a different model in its own worktree. OpenCode does the same with an agent-to-model map in its config. The hosted agents are the most polished, but each one locks you to its vendor's models.

What I have that they mostly don't: any vendor's CLI as the worker (Xiaomi MiMo, GPT-6.1 Sol through the Codex CLI, DeepSeek), per-worker price cards with a monthly USD cap, automatic fallback when a lane is down, and a replay script that re-runs old hard orders on a new model.

The plan

  1. Reactions: when review blocks, send the blocking items back to the worker automatically.
  2. Advisor: when a cheap worker gets stuck, let it consult a strong model once instead of escalating the whole order.
  3. Self-contained orders: lint work orders for references to a conversation the worker can't see.
  4. Per-order cost ledger.

What the logs said

Item 4 already existed. I had built it on October 31 and forgotten (stats already reports cost per worker and per project). Since November 1: $35.74 total, $32.24 of it on GPT-6.1 Sol across 49 orders.

Item 1 was the one I was most excited about. The review log since November 1: 50 pass, 1 block. Sol, my default worker since November 3, passed first time on 36 of its 43 accepted orders (84%). Review almost never blocks, so a review-block reaction would almost never fire.

But second rounds still cost me. I pulled every attempt since November 1:

Attempt kind Count Tokens
first 63 145.7M
redo (written by me) 19 46.1M
escalate 4 14.5M
auto retry (check failed) 7 8.8M
stall retry 14 3.2M

(All 14 stall retries were MiMo streams hanging mid-run. That's one reason I stopped paying for MiMo.)

The 19 manual redos are where a reaction would have to earn its keep. I read every feedback file:

Why I sent it back Redos
My diff review found a bug the reviewer passed 7
A run on real data or the real network found a problem the sandbox couldn't 5
The worker stopped and asked permission (usually to change an old test) 4
The base branch moved and the rebase broke 2
My work order was wrong 1

One of the 19 was a genuine review block that an automatic reaction could have handled. One.

The five real-environment failures are the interesting row. My sandbox blocks the network and my real databases on purpose, so a worker literally can't see that Wikidata returns Traditional Chinese labels for 開封市, or that one feature in Natural Earth's river file has an empty geometry. No reaction fixes that. Someone has to run the thing for real.

The four "may I?" stops look like an advisor problem at first. They aren't. Every one asked for an authorization ("can I change this assertion?"), and an advisor model has no business granting that.

What I'm building anyway, and why

I still dispatched item 1, scoped down: one automatic redo per order, then it stops and waits for me. Sol built it in one round. (The worker is told to argue back in its report if the review is wrong, instead of forcing a change.) The reason is item 2. If a much cheaper model plus an occasional advisor call can match Sol, review blocks will come back, and the reaction should be ready when they do.

I dispatched item 3 as well. On October 29 I scored 61 "must fix" items from first-round reviews of ten orders that needed rework. 55 of them (90%) were gaps in my work order, not the worker ignoring it. The new lint only warns. It flags a missing background section and phrases like "as discussed" that point at a conversation the worker never saw. Run over the 67 implementation orders since November 1, it flagged 6. Four were real (a batch of orders I wrote one evening with no background section at all). Two were false positives: one used "mentioned above" for an earlier line of the same order, and the other was the lint's own work order, which quoted the phrase as an example.

Before submitting, I ran both orders through my "falsifier", a read-only Sonnet pass that attacks the order against the actual repo (35 seconds, about $0.07). It found 7 defects in the first draft of each order, including one CRITICAL: my reaction hook sat inside a try/except that would have turned any crash in the redo into "review unavailable". After I fixed those, a second pass found 7 more in each. These were orders I wrote on the same day I was writing about how 90% of rework comes from gaps in orders (so yes, the lint is partly for me).

The replay set is the part worth copying

On October 31 I replayed five orders that had taken MiMo 3 to 7 rounds each:

  • GPT-6.1 Sol (medium effort): passed review first time on 3 of 4 doable orders, about 17 minutes and $0.8 each.
  • GPT-6.1 Terra (xhigh): 2 of 4, about 26 minutes and $1.0 each, using 3 to 4 times Sol's tokens.
  • Both stopped within minutes on the fifth order and pointed out that my order contradicted itself.

That replay is why Sol is my default today. Leaderboards told me Terra was the better deal. My own five orders said otherwise.

This week I replayed the same five orders on three cheaper models: GPT-6 Luna (direct from OpenAI), and GLM-5.3 and Tencent Hy4 through OpenRouter, each with its own API key and a hard credit limit so the bill per lane is exact.

Lane Finished Passed first round Passed with no human input Avg minutes Cost
GPT-6.1 Sol (Oct 31) 5/5 3 of 4 doable 3 of 4 17 $0.66 per order
GPT-6 Luna 5/5 2/5 5/5 52 $0.24 per order
GLM-5.3 1/5 1/1 1/1 68 ~$6.30 spent, then the $7 key ran dry
Tencent Hy4 2/10 (alone and with advisor) 1/2 2/2 70 ~$3.60 spent, then the $4 key ran dry

(The new runs went four at a time with reviews queued behind each other, so their minutes are inflated a bit. Sol's ran two at a time.)

Three things I didn't expect.

GLM and Hy4 look cheap on the price page and weren't cheap here. They chewed through tokens and got almost no cache hits, so a finished order cost as much as Sol or more. Both keys hit their limits before the round was done. If I hadn't set per-key limits, I'd have found out on the invoice.

Luna finished all five at about a third of Sol's cost, and nobody touched it. Two orders failed their checks and retried automatically. Two got blocked by review, and the new review-block reaction sent them back. Both came back clean. So the reaction I almost didn't build fired twice on its first day. But Luna took three to four times as long, and on the order that contradicts itself, Sol and Terra both stopped and asked. Luna didn't notice, built something, and the reviewer passed it. That's the one result I'd want a human to see before trusting the cheap lane.

The advisor barely got used. I told the cheap models in plain words that a stronger advisor was available and when to ask. Across the four advisor-lane runs that got going before the keys ran out, one Hy4 run asked once. It was a good question (whether its reading of three ambiguous spec clauses held up), and the advisor's bill for the whole day was $0.095. One call isn't a result. What it does suggest is that "ask when you're stuck" doesn't happen on its own. If I keep the advisor, the dispatcher will have to make the call at fixed points, before delivery or after a second failure, rather than leave it to the worker.

If you run something similar

Count your redo reasons before you build automation for them. Mine were mostly "the reviewer missed a bug" and "the sandbox can't see reality", and neither one is a reaction. And keep a handful of your own past hard tasks as a replay set, with one API key per model and a hard limit on each. This round cost me about $13 across four models. That's cheap for finding out that two "cheap" models weren't, and that the cheap model that worked needs a human to look at its output.

Happy to compare notes if you're running a dispatcher of your own.

Zhuojun Li

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.