Dev.to AI 🤖 Ai 👁 0 📖 5 min read

27,280 Agent Runs Prove Your Harness Matters More Than Your Model

27,280 agent runs. 83 billion tokens. 22 harness configurations, 5 models, 62 terminal tasks, 4 attempts each. The headline is not which model won. It is that the same harness moved eleven of twenty-two places when the

27,280 agent runs. 83 billion tokens. 22 harness configurations, 5 models, 62 terminal tasks, 4 attempts each.

The headline is not which model won. It is that the same harness moved eleven of twenty-two places when the model underneath it changed.

Codex CLI ranked first on cost per task with GLM-5.3. With Kimi-K3, it ranked sixteenth. If your harness choice came from a launch post, you are running someone else's pair.

THE EXPERIMENT, IN PLAIN ENGLISH

FrontierHarness Eval v1.1 ran 22 harness configurations across 5 models on 62 terminal tasks, 4 attempts each: 27,280 runs and 83 billion tokens, 76 times the size of v1.0. The earlier version ranked nine harnesses on one model, where pass rates sat within 17 points and median cost per pass differed by 17 times. The spread needed a second variable.

A harness is everything around the model that decides what it sees, what it can call, and when it stops: prompt, tool schema, loop, retries, and stop condition. The model is a parameter inside it.

THE 11-PLACE SWING

The median harness moved 11 of 22 places by cost per task. Best value means lowest cost per pass, unless another harness costs at most 30 percent more and passes 5 points more often. That picked Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse, and Alma with Kimi-K3.

The extremes are worse. DeepSeek Harness: Creator went from first with Muse to twenty-first with GLM Flash. OpenCode v1 went from first with DeepSeek to thirteenth with Muse. Codex CLI, cheapest per pass overall at $0.937, was twelfth with Muse.

Reruns do not explain it. Split the attempts in half, and the mean rank correlation is 0.71 within one model, 0.26 across two.

THE MEASUREMENT THAT LIES

The model still sets the level. A two-way analysis of variance puts 92.8 percent of the cost spread on the model, 2.3 percent on the harness, 4.9 percent on the pair.

Hold the model fixed, and the spread is still brutal. The most expensive harness cost 1.8 to 5.0 times as much per task as the cheapest, 1.9 to 4.9 times per pass. One decision, worth several times your inference bill.

Paying more buys nothing. Cost per task and pass rate correlated from minus 0.39 to plus 0.22. With four of five models, the most expensive harness per pass passed less often than the cheapest.

Then follow the money. 64.5 percent of total cost went to trials that did not pass, 52.7 percent to failures, and 11.8 percent to timeouts. Your agent is not expensive because it works. It is expensive because it flails.

WHY THE HARNESS DECIDES

Four factors were tested against cost per pass. Prompt size did not predict it. Tool count did not. Cache hit rate did not. Steps did, at plus 0.51.

Every step resends the context, so a trial burned 2.0 to 4.8 million input tokens. mini-SWE-agent had the smallest first request at 882 tokens and still ranked seventeenth of twenty-two at 38 steps per pass. You are not paying for your prompt. You are paying for your loop.

The five best-value harnesses shared three choices. One shell tool handled 61 to 93 percent of tool calls. Each request only appended to the last, so 91.1 to 94.0 percent of input in passed trials came from cache. They shared a weakness too: about three in four failed runs ended with a confident final answer rather than a timeout, 68 percent with Pi and 82 percent with Codex CLI.

That is why review became the bottleneck. Writing code stopped being scarce. Trusting what came back is not, and a confident wrong answer on turn forty hands the cost to a human.

THE SCORECARD TO RUN THIS WEEK

You cannot copy the leaderboard. Their tasks are not yours, the prices come from one provider, and the ranking moves on the next release.

  1. Pull 10 backlog tasks a senior engineer would take half a day on.
  2. Run 2 or 3 harness and model pairs on all 10, 3 attempts each, defaults.
  3. Record pass rate per pair.
  4. Record cost per solved task, not cost per task.
  5. Record review minutes per pull request and rework rate at seven days.

An illustrative record shape:

# illustrative scorecard, ten tasks, three attempts each
runs = load("eval_runs.json")

def solved(r):
    return r["pass"] and r["rework_hours"] < 0.5

for pair, rows in group_by(runs, "harness+model").items():
    ok = sum(solved(r) for r in rows)
    print(pair,
          "$%.2f per solved task" % (sum(r["cost_usd"] for r in rows) / max(ok, 1)),
          "%.0f review minutes" % (sum(r["review_min"] for r in rows) / max(ok, 1)))

The line that matters is r["rework_hours"] < 0.5. That clause separates a pass your dashboard counts from a pass your team pays for. Set it to zero, and the cheap harness that generates work loses to the expensive one that finishes it.

Decision rule: keep the pair, not the model. Re-run the eval on every release, because an eleven-place median swing makes last quarter's ranking stale.

WHAT TO DO TODAY

Stop benchmarking models in isolation and benchmark pairs on your own tasks. If you cannot name the pair you run in production, you have a habit, not a configuration.

Add review minutes to the scorecard. A harness that passes 5 points more often and adds 20 minutes per pull request is not better; it is more expensive wearing a better pass rate.

Re-test the harness before you switch models. A model swap moves the harness ranking more than a harness swap moves the model ranking.

Instrument rework rate on agent-written pull requests for two weeks before you believe any productivity claim, including your own. Lines generated is activity. Rework is the cost of it.

THE UNCOMFORTABLE QUESTION

You can name your model version from memory. Can you name your harness's cost rank, or are you flying on vibes and a launch post?

If the argument that the pair matters more than the model stuck with you, take a deeper look at PhantomByte's Guide to Agent Harness Architecture.

It breaks down what a harness is, the four dimensions of its design (composition, context sharing, approval gating, and tool integration depth), and why changing the harness while keeping the model constant can have a greater impact on cost per task than switching between models.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.