GPT-6.2 Astra vs Sol vs Claude Opus 5.5: Sol 6.2 Scores Below 6.1 on SimpleBench
GPT-6.2 Sol scores 69.9% on SimpleBench. GPT-6.1 Sol, which costs exactly the same per token, scores 82.9%. That's a 13-point drop between point releases. OpenAI shipped both 6.2 models on November 17 with a wall of benc
GPT-6.2 Sol scores 69.9% on SimpleBench. GPT-6.1 Sol, which costs exactly the same per token, scores 82.9%. That's a 13-point drop between point releases. OpenAI shipped both 6.2 models on November 17 with a wall of benchmark charts, and four days later a Hacker News thread titled "Astra 6.2 and Sol 6.2 flop at Simple Bench" had 175 points and 162 comments.
I wanted the GPT-6.2 Astra vs GPT-6.2 Sol vs Claude Opus 5.5 picture in one place: the SimpleBench score, what each vendor claims, what you pay, and which one I'd actually put in production. I didn't run any of these tests myself. Every number below comes from the SimpleBench leaderboard (via the sites that mirror it), from OpenAI's and Anthropic's own docs, or from the launch coverage, and each one is linked.
The SimpleBench 2026 numbers
SimpleBench is a multiple-choice test of "over 200 questions" covering spatio-temporal reasoning, social intelligence and trick questions. Each score is the average of 5 runs (AVG@5) at temperature 0.7. Only 10 sample questions are public on GitHub; the rest aren't released. The site's human baseline is 83.7%, taken from "our small sample of nine participants".
The official table is drawn by JavaScript, so I took the rows from LLMLearner (updated 2026-11-20) and checked them against Respan (dated Nov 19). The two sites agree on the scores below except for the GPT-6 Astra row, which I come back to under the table. The ranks are LLMLearner's; Respan puts GPT-6.1 Sol and GPT-6.2 Astra one place higher, and its visible top 25 stops before GPT-6.2 Sol.
| Model | SimpleBench | Rank | API $/1M in / out | Vendor headline claim |
|---|---|---|---|---|
| Claude Fable 5.2 | 94.9% | 1 | $8 / $40 | FrontierCode 2.0 49.1% |
| Gemini 4 Neon | 94.4% | 2 | — | — |
| Claude Opus 5.5 | 88.4% | 6 | $4 / $20 | Terminal-Bench 4.0 66.4%, FrontierCode 2.0 45.5% |
| GPT-6 Astra | 83.6% (86.5% Pro) | 9 | $10 / $50 | — |
| GPT-6.1 Sol | 82.9% | 10 | $2 / $10 | — |
| Human baseline | 83.7% | |||
| GPT-6.2 Astra | 77.3% | 16 | $8 / $40 | GPQA Diamond 97.0%, Agents' Last Exam +5.2 pts vs GPT-6 Astra |
| GPT-6.2 Sol | 69.9% | 30 | $2 / $10 | GPQA Diamond 96.0%, FrontierCode 2.0 within 0.5 pts of Astra |
Sources for the prices: OpenAI pricing and Anthropic pricing. The GPQA figures come from the GPT-6.2 system card. The Opus 5.5 figures come from Anthropic's Opus 5.5 page and the Fable 5.2 announcement, which re-reports Opus 5.5 on FrontierCode 2.0.
About the GPT-6 Astra row: LLMLearner lists Astra at 83.6% and Astra Pro at 86.5%, while Respan gives 86.5% to plain Astra. Whichever is right, 6.2 Astra scores below its predecessor.
A few things in that table bother me more than the headline:
- Both 6.2 models score well below the 83.7% human baseline. Their predecessors sat within a point of it (83.6% and 82.9%).
- Opus 5.5 costs half of what Astra 6.2 costs per token and beats it by 11 points.
-
gpt-6.1-solandgpt-6.2-solcost the same: $2 in, $10 out, $0.10 cached, with the same long-context tier above 272K tokens. If you upgraded by changing the model string, you got a model that SimpleBench rates 13 points worse at common sense, for the same money.
"Flop" is a strong word, and HN argued about it
The top pushback in the thread came from ilaksh, who argued the gap to the leaders is smaller than the title makes it sound: "we are talking about 83% (very near human average) versus 94%. So 'flop' overstates it."
But 83% is where the previous generation sat. Astra 6.2 is at 77.3% and Sol 6.2 at 69.9%, so the comparison that matters is with their own predecessors. jeremyjjj made the point I agree with: "If 6.2 is worse than 6.1, something weird is going on." Losing to Gemini is a competitive problem; losing to your own last release is a regression.
The other recurring complaint is that SimpleBench is "mostly trick questions" and says little about coding. Partly true. But the questions are built so the obvious reading is wrong and one buried detail overrides it, and plenty of real tickets work the same way.
Why vendor benchmarks and SimpleBench disagree
OpenAI's launch page reads like a different model. Astra 6.2 is "5.2 points higher than GPT‑6 Astra" on Agents' Last Exam, and up 6.2 points on GDPval v2 with 22% fewer output tokens. The announcement also says Sol comes within 0.5 points of Astra on FrontierCode 2.0 at about a fifth of the cost. I don't think those numbers are fake, and I can see three ways both sets hold up at once.
First, benchmarks that labs train toward stop telling you much. GPQA Diamond at 97.0% (Astra) and 96.0% (Sol) is close to saturated. GDPval v2 is OpenAI's own eval. In the thread, gessha described labs employing professionals to annotate their job workflows "over and over until they have a vast dataset of different tasks", and throwup238 named the companies selling that data: Mercor, Surge AI, Scale AI and others. If your training mix looks like agent tasks, your agent scores go up. Nobody outside OpenAI can prove that's contamination, but the incentive is plain.
Second, a private, adversarial test can't be studied for. SimpleBench keeps most of its questions private, which GeorgeTirebiter summed up as both its strength and its flaw: they "can't contaminate models", and "you cannot know what it is asking." The questions are also meant to punish pattern-matching. A model tuned hard toward "recognize the task type, run the playbook" will do worse on a question whose whole point is that the playbook is wrong.
Third, token budget might be part of it. LLMLearner lists 69 output tokens per task for Astra 6.2 and 54 for Sol 6.2. I don't know what reasoning effort the SimpleBench harness used or whether hidden reasoning tokens are counted, so treat this as an open question. If 6.2 defaults to thinking less, it would explain some of the drop.
The anecdotes in the thread line up with the third explanation better than I expected. nsingh2 found 6.2 good at code review and math but "relatively worse at deriving intention". logicchains said Astra 6.2 picked a bad architecture where "Claude 5.5 would have asked me for clarification". On the other side, adam_arthur called Astra 6.2 "amazing to work with" but said 6.2 Sol "has been significantly worse", and ChadMoran called Sol 6.2 Ultrafast "the star of this release". The split follows the line SimpleBench draws: well-specified tasks fine, ambiguous ones worse.
Which to pick: GPT-6.2 Astra, GPT-6.2 Sol or Opus 5.5
For coding with clear specs and tests, I'd go with Sol 6.2 or stay on Sol 6.1. If your tasks come with a failing test and a file path, Sol's $2/$10 price and the FrontierCode result make it hard to beat. If your prompts are vague, pin gpt-6.1-sol until you've compared the two on your own tickets. It costs the same.
For agents running unattended, Opus 5.5. Astra 6.2 has the better vendor agent numbers, and explainx reports it at 1,291 on arena.ai's Agent leaderboard versus 1,241 for Sol. But an agent with nobody watching is exactly where "deriving intention" matters, and Topfi in the thread said the 6.2 models "outright claim to have completed work they didn't". At $4/$20, Opus 5.5 is also half the per-token price of Astra 6.2.
For plain reasoning, support bots and anything user-facing, Opus 5.5 is the pick of these three. It scores 88.4% to Astra's 77.3%. If you're not limited to these three, Fable 5.2 (94.9%, $8/$40) and Gemini 4 Neon (94.4%) are the top of the board.
One caveat on price. Opus 5.5 is cheaper per token, not per task. LLMLearner puts the total cost of its SimpleBench run at $2.74 with 575 output tokens per task, against $0.96 and 69 tokens for Astra 6.2. If your workload is lots of short answers, Astra's lower token count can win the bill even at twice the rate.
Before you change the model string
- Pin dated model IDs in config, not "latest". A point release can go backwards, and this one did on at least one axis.
- Write 20 trick prompts from your own domain: tickets where the obvious fix is wrong, specs with one buried constraint. Run old and new models on them side by side before switching.
- When you compare costs, compare cost per completed task, not per token. As aavaa said in the speed sub-thread: "Tok/s is irrelevant if you don't include tokens/task."
- Read the small print under vendor charts. A caption under one of OpenAI's launch charts says its Fable 5.2 results exclude nine tasks that used a fallback model. Anthropic says its Opus 5.5 scores ran with safeguards on. Both labs pick their harnesses.
Your turn
Do you trust vendor benchmarks, or independent ones like SimpleBench, when you decide what to ship? And if you've moved real work to Astra 6.2 or Sol 6.2 this week, has it felt better or worse than 6.1 or Opus 5.5? I especially want to hear from anyone who thinks SimpleBench got this one wrong.
Sources
- SimpleBench: https://simple-bench.com/
- LLMLearner SimpleBench rankings: https://llmlearner.com/rankings/simple-bench
- Respan SimpleBench leaderboard: https://www.respan.ai/models/benchmarks/simplebench-simplebench
- HN discussion: https://news.ycombinator.com/item?id=50588916
- OpenAI, Introducing GPT-6.2 Astra and Sol: https://openai.com/index/introducing-gpt-6-2-astra-sol/
- GPT-6.2 system card: https://deploymentsafety.openai.com/gpt-6-2-astra-sol
- OpenAI API pricing: https://developers.openai.com/api/docs/pricing
- Anthropic, Claude Opus 5.5: https://www.anthropic.com/claude-opus-5-5
- Anthropic, Claude Fable 5.2 and Mythos 5.2: https://www.anthropic.com/claude-fable-and-mythos-5-2
- Anthropic pricing: https://platform.claude.com/docs/en/about-claude/pricing
- explainx launch write-up (arena.ai Agent numbers): https://explainx.ai/blog/openai-gpt-6-2-astra-sol-launch-pricing-benchmarks-november-2026
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.