Dev.to AI 🤖 Ai 👁 0 📖 6 min read

AI Coding Scores Hit 80%. Then the Tests Failed Their Own Audit. Twice.

The benchmark behind most "best coding model" claims was retired in February. Its replacement lost its recommendation in July. Here is what went wrong, and what to trust instead. Every model launch comes with a chart

The benchmark behind most "best coding model" claims was retired in February. Its replacement lost its recommendation in July. Here is what went wrong, and what to trust instead.

Every model launch comes with a chart. One bar is a little taller than the others, and under it is a number like 80.9%. That number has decided which coding assistant a lot of teams pay for.

This story was written with the assistance of an AI writing program. The figures come from the published audits and papers named in the text.

This year, the two most quoted tests behind those charts were audited by the people who rely on them. Neither came out well. The story is worth knowing before you read another leaderboard.

The exam everyone studied for

SWE-bench arrived in 2023 from researchers at Princeton. The idea was good: take real bugs from real open-source Python projects, hand a model the issue, and check whether its fix passes the project's own tests.

In 2024 OpenAI cleaned it up. Expert engineers reviewed 1,699 problems, three reviewers each, and kept 500. That set was called SWE-bench Verified, and it became the standard. For the next year and a half, every major lab reported it.

Then the scores stopped moving. Over six months the best result crept from 74.9% to 80.9%. OpenAI asked a fair question: are the remaining failures the model's fault, or the test's?

The first audit

They took 138 problems that their o3 model could not solve reliably across 64 runs. At least six experienced engineers reviewed each one.

59.4% of those problems had something materially wrong with them.

The largest group, 35.5%, had tests that demanded one specific implementation. A fix could be correct and still fail, because it did not match the way the original developer happened to write it. Another 18.8% had tests that checked for behaviour the problem description never asked for.

So some of the "failures" were correct answers marked wrong.

The second finding was worse. The problems come from public repositories, and public repositories are training data. OpenAI set one model to interrogate three others: GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash. All three could reproduce details of the original human fix, or the exact wording of the problem, for some tasks.

In one case Claude Opus 4.5 recalled the four-line change, the file, the method, and quoted the inline comment word for word. In another, Gemini 3 Flash was given nothing except the task's ID and produced the task description and the fix, down to the new regular expression and the line numbers.

OpenAI's own comparison is hard to improve on. It is like handing students the questions and answers before the exam. They may not memorise everything, but they will beat the students who never saw them.

On 23 February 2026, OpenAI said it would stop reporting SWE-bench Verified and asked other labs to stop too.

The warning signs were there

None of this came from nowhere. A 2024 study of the original SWE-bench found that in 32.67% of the successful fixes, the solution was sitting in the issue text or its comments. Another 31.08% passed only because the tests were weak. When those were removed, one agent's score fell from 12.47% to 3.97%.

In 2025, researchers at Microsoft and Purdue tried a neat trick. They gave models only the issue description, with no access to the code at all, and asked which file contained the bug. On SWE-bench Verified, models named the right file up to 76% of the time. On repositories outside the benchmark, the best they managed was 53%.

You cannot reason your way to a file path in a project you are not allowed to see. You can only remember it.

The replacement

The fix was supposed to be SWE-Bench Pro, released by Scale AI in September 2025. It has 1,865 tasks from 41 repositories. The public part uses code under strict copyleft licences, which labs tend to keep out of training data. Another part comes from private startup codebases that no model could have seen. The tasks are bigger too, averaging 107 lines changed across four files.

The first results were sobering. Models that scored above 70% on Verified managed about 23% on Pro. On the private codebases, the best two scored 17.8% and 14.9%.

That gap, 70 against 23, was the most honest number the field had produced in a while. OpenAI recommended that everyone switch.

The second audit

Eight months later, the top score on Pro's 731 public tasks had climbed from 23.3% to 80.3%. That is a very fast climb, so OpenAI ran the same kind of audit again.

An automated filter flagged 286 tasks. A review by AI agents under human supervision judged 200 of them broken, which is 27.4% of the public set. A separate review by engineers, five per task, put it at 249, or 34.1%. OpenAI settled on roughly 30%.

One example shows how small the problems can be. A task about formatting a table of contents told the model, to the character, how the output should look: one leading space. The hidden tests expected two. A model that followed its instructions exactly would fail.

On 8 July 2026, OpenAI withdrew its recommendation. The replacement had held it for less than five months.

What the number actually measures

It is tempting to read this as "the scores are inflated". That is half of it.

Contamination and weak tests push scores up. Over-strict tests and misleading prompts push them down. Both happen in the same benchmark at the same time, so the result is not simply too high. It is noisy, in a direction you cannot work out from the outside.

And when the top models sit two or three points apart, the noise is larger than the gap you are being asked to care about.

What to do instead

Read a leaderboard as a rough filter. A model at 30% and a model at 75% are different. A model at 78% and one at 80% may not be.

Check what was run and who ran it. Scores from a lab's own harness and scores from a neutral one are not comparable.

Build a small test from your own code. Take ten or twenty tasks your team already finished: real bugs, real features, from a private repository. Reset to the commit before each one and let the candidate models try. No model has seen your code, so nothing is memorised.

Read the diffs. Passing tests told these benchmarks the wrong thing in both directions. For twenty tasks, a person can read every result. That is the grading the big benchmarks cannot afford.

Run it again when you switch models. Twenty tasks take an afternoon. A wrong choice of tool costs a team months.

Caveats

OpenAI is not a neutral referee. It competes on these benchmarks, and it audited one built by another company. But it also retired a benchmark it created itself, and it published its method and examples both times.

The 59.4% figure describes problems a strong model kept failing, not the whole of Verified. The share across all 500 problems is lower.

And the models really have improved. Nobody who used a coding agent in 2024 and uses one now doubts that. The issue is narrower: the public yardstick can no longer tell you by how much, or which one is ahead.

The takeaway

For years the industry pointed at one number. It turned out that a good share of the hard questions were unfair, and a good share of the answers had been seen in advance. The replacement repeated the first problem within months.

A benchmark is a test that someone else wrote, on someone else's code, graded by a script. The only version that reflects your work is the one you build from it. It is twenty tasks and an afternoon, and it is the one score nobody can study for.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.