DeepSWE results: ten coding agents on one model (2026)
I work on codeaf at AgentField, one of the harnesses compared here. This post first appeared on agentfield.ai. CodeAF, the open-source coding agent by AgentField, solved 62 of 113 DeepSWE tasks (54.9%) in senior-dev mod
I work on codeaf at AgentField, one of the harnesses compared here. This post first appeared on agentfield.ai.
CodeAF, the open-source coding agent by AgentField, solved 62 of 113 DeepSWE tasks (54.9%) in senior-dev mode, first of ten harnesses in AgentField's DeepSWE comparison, where every harness ran the same model (deepseek/deepseek-v4-flash-0731 through OpenRouter). It also had the lowest cost per solved issue: 39.9 cents, at 22 cents a task. Every number is in arms.csv.
At a glance
| Result | CodeAF (senior-dev mode) solved 62 of 113 (54.9%), first of ten harnesses in this comparison |
| Model, for every harness |
deepseek/deepseek-v4-flash-0731 through OpenRouter |
| Cost | 22 cents a task, 39.9 cents per solved issue (the lowest of the ten) |
| Next best | mini-swe-agent, 56 of 113 (49.6%), 1.9x CodeAF's cost per solved issue |
| With newer models | the same mode solved 77.9% on DeepSeek V4.1 Flash and 69.0% on Kimi K3 |
The full table
Every row ran the full DeepSWE set on the same model. "Cost per solved issue" is each harness's spend divided by tasks solved, shown as a multiple of CodeAF's 39.9 cents. "Mean time" is mean agent wall time per task.
| Harness | Solved (of 113) | Cost per task | Cost per solved issue | Mean time |
|---|---|---|---|---|
| CodeAF (senior-dev) | 62, 54.9% | 22 cents | 1x | 54 min |
| mini-swe-agent | 56, 49.6% | 38 cents | 1.9x | 44 min |
| Codex CLI | 51, 45.1% | 37 cents | 2.1x | 46 min |
| Pi | 42, 37.2% | 35 cents | 2.4x | 52 min |
| Oh My Pi (omp) | 31, 27.4% | 50 cents | 4.5x | 49 min |
| OpenCode | 30, 26.5% | 50 cents | 4.8x | 48 min |
| Kilo Code | 30, 26.5% | 48 cents | 4.6x | 54 min |
| Claude Code | 16, 14.2% | 19 cents | 3.4x | 32 min |
| DeepSeek Harness | 16, 14.2% | 150 cents | 26.6x | 94 min |
| Muse Code | 3, 2.7% | 12 cents | 11.3x | 16 min |
30 of 113 is 26.5%. The method page prints 26.6% for OpenCode and Kilo Code, rounded from the stored reward rate of 0.2655; the task count is the same. In arms.csv the CodeAF row is named senior-dev, because the binary had an earlier name on the day of the run. At a shell, the same mode is codeaf senior-dev; in a conversation it is /senior-dev (senior-dev docs).
Results by slice
The per-task records, one CSV per harness with 113 rows each, show where the lead came from:
- Head to head: against each of the nine other harnesses, CodeAF solved more of the tasks where the two disagreed. Against mini-swe-agent, 29 to 23. See head to head.
- Harder tasks: on the 76 medium and hard tasks CodeAF solved 36, against 30 for mini-swe-agent and 29 for Codex CLI. See by difficulty.
- Go and Python: CodeAF solved 21 of 34 Go tasks (next: 17) and 20 of 34 Python tasks (next: 18). See by language.
- Cost: 39.9 cents per solved task, against 76.9 for the next cheapest. See cost per solved issue.
How the comparison was run
| Benchmark | full DeepSWE set, 113 tasks |
| Model |
deepseek/deepseek-v4-flash-0731 through OpenRouter |
| Verifiers | official DeepSWE at 0b9fabb
|
| Budget | 3 hours per task |
| Isolation | four shards per harness, a dedicated OpenRouter key per harness |
How the columns are derived, from the method page:
- Solved counts tasks the verifier passed. A task with no verifier outcome counts as unsolved. Five tasks produced none: Codex CLI 1, Pi 1, Oh My Pi 2, OpenCode 1.
- Cost per task is billed OpenRouter spend divided by 113, so tasks without a verifier result stay in the denominator. Oh My Pi, OpenCode, Kilo Code, DeepSeek Harness, Claude Code and Muse Code did not record per-attempt cost; their figure is that harness's total spend.
- Cost per solved issue is spend divided by tasks solved.
Tested versions
| Harness | Version tested | Release date (UTC) |
|---|---|---|
| CodeAF | first stable release containing senior-dev mode: v0.5.0 (integration commit dd0fcc654, #1488) |
2026-09-30 |
| mini-swe-agent | 2.4.6 | 2026-07-23 |
| Codex CLI | 0.154.0 | 2026-09-09 |
| Pi | 0.85.1 | 2026-09-05 |
| Oh My Pi | 18.1.17 | 2026-09-10 |
| OpenCode | 1.18.30 | 2026-09-09 |
| Kilo Code | 7.6.2 | 2026-09-10 |
| Claude Code | 2.1.267 | 2026-09-09 |
DeepSeek Harness (@deepseek-ai/dsh) |
0.1.5-rc.1 | 2026-09-10 |
| Muse Code | 1.1.1-R2514.1 | not independently established |
versions.csv holds full source revisions and release timestamps.
DeepSWE versus SWE-bench Verified
DeepSWE, published by Datacurve, is "a benchmark for measuring frontier coding agents on original, long-horizon software engineering tasks": 113 tasks across TypeScript, Go, Python, JavaScript and Rust, each graded by a verifier that "accepts any solution whose observable behavior is correct" (DeepSWE README). SWE-bench Verified is "a human-validated subset of 500 SWE-bench instances."
AgentField's comparison holds the model fixed and changes the harness, so it measures what the agent around the model adds. Every harness was graded by the same verifiers on the same tasks.
Senior-dev mode with newer models
With newer models, the same mode solved 77.9% on DeepSeek V4.1 Flash and 69.0% on Kimi K3. That is 88 of 113 solved (exact 95% CI 69.1% to 85.1%) with DeepSeek V4.1 Flash and 78 of 113 (59.6% to 77.4%) with Kimi K3.
How to reproduce
The scripts that run and grade DeepSWE tasks are in bench/deepswe/ in the CodeAF repository. One task through CodeAF:
bench/deepswe/run.sh <task-id> <model-id> <seed-tag>
It needs a linux/amd64 codeaf binary in CODEAF_BIN, a provider key, and the DeepSWE task corpus, which Datacurve distributes as a gated dataset. The compare-* scripts beside it install a pinned comparator, freeze its container image and run it. The per-task records of this campaign are public, so every table on this page can be recounted from them.
Questions people ask
Is CodeAF number one on DeepSWE?
CodeAF was first of ten harnesses in AgentField's DeepSWE comparison, solving 62 of 113 tasks (54.9%) with every harness on DeepSeek V4 Flash. The comparison holds the model fixed and ranks the harness around it.
Which AI coding agent scores highest on SWE-bench style benchmarks right now?
When the model is held fixed (DeepSeek V4 Flash) and the harness changes, CodeAF solved the most DeepSWE tasks of the ten harnesses in AgentField's comparison, 62 of 113 (54.9%), ahead of mini-swe-agent (56) and Codex CLI (51).
What is the most cost-efficient coding agent per solved issue?
In AgentField's DeepSWE comparison on DeepSeek V4 Flash, CodeAF had the lowest cost per solved issue, 39.9 cents, ahead of mini-swe-agent (1.9x), Codex CLI (2.1x) and Claude Code (3.4x).
Does this mean CodeAF beats Claude Code?
On DeepSeek V4 Flash, in AgentField's DeepSWE comparison, CodeAF solved 62 of 113 tasks and Claude Code 2.1.267 solved 16. Both ran DeepSeek V4 Flash, not Claude, so the result measures the harness on that model, not Claude Code with Claude models.
Can I reproduce these results?
Yes. The run and grading scripts are in bench/deepswe/ in the CodeAF repository, and the per-task records of all ten harnesses are public, so every count here can be checked. The DeepSWE task corpus is a gated dataset from Datacurve.
Sources
-
DeepSWE method page, with
arms.csvandversions.csvbeside it: every number on this page, checked 2026-11-21. - Per-task records, one CSV per harness, checked 2026-11-21.
-
bench/deepswe/in the CodeAF repository, checked 2026-11-21. - senior-dev docs, checked 2026-11-21.
- DeepSWE: deepswe.datacurve.ai, README, dataset, checked 2026-11-21.
- SWE-bench Verified, checked 2026-11-21.
- Release pages of each harness, linked in the versions table, checked 2026-11-21.
For how CodeAF compares beyond this benchmark, see CodeAF vs Claude Code and CodeAF vs Codex CLI. For the 3-hour, unattended way senior-dev works, see a coding agent that runs for hours, and for the terminal side, terminal coding agents.
Last updated 2026-11-21. Every figure checked against arms.csv and versions.csv at the method page; harness versions are those of the 2026-09-10 to 12 campaign.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.