GPT-6 Astra Passed 69.9% of Hidden Checks. It Fully Finished 26.4% of Real Edit Tasks

• Book: Ship It — the pragmatic startup tech stack
• Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

Hand GPT-6 Astra a working web app and a real user's change request, and it clears an average of 69.9% of the hidden acceptance checks per task. It clears every check on 26.4% of tasks.

Those numbers come from Co-Design Bench, a paper submitted to arXiv on October 26 by researchers from Alibaba and several Asian universities. The Decoder wrote it up on November 8 and drew the practical conclusion: "so anyone using these tools almost always has to clean up by hand."

GPT-6 Astra was the top scorer of the ten model endpoints tested. The runner-up, opus-5, fully finished 18.3%.

What Co-Design Bench measured

The benchmark starts from real usage. The authors took 39,325 production requests across 26,314 coding sessions, deduplicated them to 37,463 candidates, and built 273 tasks. Each task gives the agent a running web app and asks for a change.

100 tasks are Multimodal Editing, localized edits that can attach images, audio, video or documents. 173 are Repository Editing, coordinated changes across multi-file frontend repos.

The agent never sees the 2,331 acceptance criteria. They stay hidden until grading, so it can't work to a checklist.

All ten models ran under one fixed harness, Claude Code, with runs capped at three hours and 120 turns. That keeps the model as the variable.

Grading wants evidence. One AI judge reads source diffs, renders and browser traces recorded with Playwright, and a second challenges Pass and Partial verdicts, looking for stubs and hardcoded results. The strict score counts a criterion only when it earns a Pass, so a Partial adds nothing.

Most of the work is a long way from all of it

Here is the leaderboard from the project README, with the repository column from the paper's main table:

Repository tasks carry a mean of 10.3 criteria each. Miss one and the task drops out of the full-pass column. That is how the top model gets 66.7% of the checks on repository edits and fully passes 17.9% of those tasks.

Most checks pass. One miss keeps the task out of the full-pass column.

A 70% score doesn't mean 70% of your tickets are done. Track both numbers on your own agent runs. Each inner list below is one task's checks:

runs = [
    [True, True, True, False],
    [True, True, True, True],
    [True, False, True, True],
]
per_run = [sum(r) / len(r) for r in runs]
average = sum(per_run) / len(runs)
done = sum(all(r) for r in runs) / len(runs)
print(f"checks: {average:.1%}  fully done: {done:.1%}")

The average tells you how close the agent gets. The second number tells you how often you can merge without touching the result.

Swap the harness and the scores move

The main table holds the harness still on purpose. Appendix H runs the opposite experiment: two models, kimi-k3 and Qwen3.8-27B, on all 273 tasks under three different agents, Codex, opencode and qwen-code.

On the repository track, Table 12 reports these average completion rates:

Qwen3.8-27B's lowest score here came from Qwen's own CLI. Its highest came from opencode, and the 18.9-point gap between them is the biggest swing in the experiment. On the Multimodal Editing track, the harness order reverses between the two models, though kimi-k3's three scores there sit within 2.1 points.

Compare these numbers only inside this table. The paper doesn't label the metric as strict or graded, and kimi-k3's Multimodal scores here land around 83% to 85% against 66.2% strict in the main table. The experiment also swaps whole harnesses, so it can't pin a difference on any single tool feature.

The authors say the results "motivate reporting the model and harness together." That covers the leaderboard above too: every number in it is a Claude Code number. Two models on three harnesses is a small test, and the spread ran from 2.1 points to 18.9 with no change of model. Don't assume a score from one harness carries over to the CLI you use.

Read the fine print before you quote it

A leaderboard row can't show how the tasks were picked or how they were graded, and both change what 26.4% means.

• The set skews hard. Candidates were ranked partly on structural difficulty and "historical failure signals" such as rollbacks and rephrased requests. The selection favors hard tasks, so don't read 26.4% as the rate for your ticket queue.
• Alibaba is on the author list. Alibaba Group is among the affiliations, Qwen comes from Alibaba Cloud's Qwen team, and Qwen3.8-Max placed third on both main columns. Read it the way you'd read any benchmark from a participant.
• You can't rerun all of it. The public release has 96 Multimodal tasks and no Repository Editing tasks, and its requests are mostly in Chinese. I haven't run it myself.

The setup also differs from how you work. Agents get one shot: each task is a fixed request with no follow-up feedback, and a session with you in it doesn't go that way. The judges are models too. The evidence trail helps, but the paper says it "cannot eliminate judgment error." Two frontend experts checked rubric scores on 200 outputs, and the paper gives no agreement figure.

What to change in your own setup

Write the acceptance checks before you hand over the task. Keep them where the agent can't edit them, which is the idea behind the benchmark's hidden criteria. Browser checks in Playwright work well for web apps because they test the running app.

Keep tasks narrow. Repository edits spanning three or four kinds of change scored 6.3 points lower on strict completion than edits spanning one or two. The paper calls task-group comparisons descriptive, not causal, but splitting a ticket is cheap to try.

Compare model and harness as a pair, on your repo. git worktree gives each attempt its own branch and folder:

git worktree add ../app-codex -b try/codex
git worktree add ../app-opencode -b try/opencode

Run one CLI in each folder with the same prompt, then score both with the same checks.

I build Hermes IDE, so weigh this paragraph with that in mind. Hermes runs each agent in its own terminal, each task on its own branch and worktree, so the same ticket can run in Codex and OpenCode side by side. ⌘N (Ctrl+Shift+N on Windows and Linux) starts a task, ⌘G opens the Review Desk by turn or by file, and your own checks can decide when a task is done. Any command-line agent you add gets a terminal too. Hermes isn't affiliated with Nous Research, OpenAI, Anthropic or any agent vendor.

Hermes IDE 2.3.4: Codex and Claude Code side by side, each in its own hermes/ worktree.

What's your fully-passed rate? Count last week's agent tickets you merged without editing a line. Tell me the number in the comments.

If this was useful

The fix in this post is unglamorous: acceptance checks that run before a change reaches production. Ship It is the book I wrote for everything between localhost and production on a small team, including hosting, auth, payments, CI/CD and monitoring, and that pipeline is where unmet requirements get caught. If you want to try model and harness pairs side by side, Hermes IDE is free and source-available on macOS, Windows and Linux, with the code on GitHub.