Don't Let Claude Review Claude's Code. Models That Think Alike Make the Same Mistakes

• Book: Observability for LLM Applications
• Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

Qwen2.5-72B, grading free-form answers from two smaller models, marked its own family's 7B model 71% correct and Llama-3.1-70B 67% correct. On the multiple-choice version of the same questions, the Qwen 7B got 41% right and Llama got 51% (Goel et al.).

That pair comes from Great Models Think Alike and this Undermines AI Oversight, by Shashwat Goel and eight co-authors, a spotlight poster at ICML 2025. The judge preferred the model that got things wrong the way it does over the model that got more things right.

Swap the judge for your review agent and the answers for a diff. If the agent that wrote the code and the agent reviewing it share the same blind spots, the reviewer reads the bug and sees correct code. So don't let a model review its own family's work. That goes for Claude reviewing Claude and for Codex reviewing Codex. Put a different vendor's agent in the reviewer's seat, and read the diff yourself.

CAPA: counting shared mistakes

The authors built a similarity metric called CAPA, short for Chance Adjusted Probabilistic Agreement. It counts how often two models make the same mistakes, beyond what you'd expect from their accuracy alone. Two models that each get 80% right will agree a lot by luck. CAPA subtracts that luck, and it treats two different wrong answers as disagreement.

Two of the findings matter for code review.

Judges favour models like themselves. Nine judge models graded free-form answers to 8,707 MMLU-Pro questions, with no access to the answer key. Every judge's scores went up with the graded model's similarity to it. Holding the graded model's accuracy constant, the partial correlation between similarity and score still ran from 0.35 to 0.65 across judges, and judges under 32B parameters showed the effect more (paper).

Stronger models make more similar mistakes. Across 130 models from the Open LLM Leaderboard, on MMLU-Pro and BIG-Bench Hard, the authors grouped models by capability and compared each one only against models from other developers. The more capable the group, the more alike its mistakes. In the abstract's words, "model mistakes are becoming more similar with increasing capabilities, pointing to risks from correlated failures."

Read the limits before you quote this at your team. The tasks are question answering, mostly multiple choice, and none of them is code. The results are correlational, and the authors say so. The models are open-weight ones from a public leaderboard, not the models behind Claude Code or Codex. Nobody in this paper reviewed a pull request. The step from "judges favour similar models" to "a same-family reviewer agent misses the writer's bugs" is my reading of the evidence. Nobody has measured it on code review yet.

Two reviewers with the same blind spot see the same clean page.

Why a same-family reviewer is the weak spot

A review agent is an LLM judge with a diff as its input. That's the setup the paper tested, with code in place of quiz answers.

Two other papers point the same way. Panickssery, Bowman and Feng (2024) found that GPT-4 and Llama 2 can tell their own outputs apart from other text with non-trivial accuracy, and in their fine-tuning experiments, the better a model recognized its own writing, the more it preferred that writing as an evaluator.

On code, Olausson et al. (ICLR 2024) studied self-repair, where a model reads its own failing program and fixes it. The gains were often modest, and the authors traced the bottleneck to the model's feedback on its own code. When people wrote that feedback instead of GPT-4, the share of repair attempts that passed all the unit tests went from 33.3% to 52.6%, a factor of 1.58.

Put those together. A model grading its own family's work is where you'd expect the most agreement for the wrong reasons: the same training habits, the same gaps, maybe a taste for its own style. When the writer assumes a list is never empty, a reviewer that would make the same assumption has no reason to question it.

The rule: a different vendor reviews, and you read the diff

Here is the setup I'd use:

1. One agent writes. Say Claude Code, on its own branch.
2. An agent from a different vendor reviews. Say Codex, told not to edit anything.
3. You read the diff and the review, and decide which findings go back to the writer.

The second step takes one command. Codex has a /review command that runs "against uncommitted changes, a commit, or a base branch" and "reports prioritized findings without modifying your working tree," with a preset for custom review instructions (Codex CLI docs). The rule runs both ways: when Codex writes, Claude Code reviews.

If the reviewer works in its own git worktree, it can still see the writer's branch, because worktrees share one repository. It can't see uncommitted edits in another worktree, so commit the writer's work first. Then give the reviewer something like this:

Review the changes from:
  git diff main...hermes/add-refund-endpoint
Do not edit any files. List bugs by file and line,
most serious first. For each one, give an input
that breaks it. Skip style comments.

The three dots compare the branch with the point where it split from main, so the reviewer sees only the writer's changes.

The third step is the one people skip, and the paper is why it matters most. Its capability trend was measured across developers: the stronger the open models, the more alike their mistakes, even between companies. If that holds for the models behind coding agents, a second vendor buys you less diversity as both agents get stronger. Treat the reviewer's findings as a list of places to look. You still read the code.

Doing this in Hermes IDE

I build Hermes IDE, so weigh this section with that in mind. You set up the cross-vendor review yourself. What Hermes gives you is the plumbing that makes it less tedious.

Press ⌘N (Ctrl+Shift+N on Windows and Linux) and each task starts on its own hermes/<name> branch, in its own worktree, with the task as the agent's first prompt. You pick the agent, model, effort and account per task. Start the writer in Claude Code. Start a second task in Codex with the review prompt above, pointed at the writer's branch.

Or run the same task twice. Give the same ticket to Claude Code and to Codex, each on its own branch, and read both diffs. Where the two agents wrote different behaviour for the same case, one of them is probably wrong, and that's where you start reading. Where they agree, remember that agreement between strong models is less independent than it looks.

Codex and Claude Code side by side in Hermes 2.3.4, each in its own hermes/ worktree, with status, context use, cost and branch in the sidebar.

Then the human step. ⌘G (Ctrl+Shift+G) opens the Review Desk, where you read changes by turn or by file, with risk flags. Every agent turn is saved, so you can diff or restore any single turn. A line comment you leave there goes back to the agent that wrote that line. So you read Codex's findings, check each one against the code yourself, and send the ones that hold up to Claude Code as line comments. The ones that don't hold up stop with you.

Hermes never answers a permission prompt for you and never types into a terminal on its own, so nothing moves between the two agents unless you move it. In this setup, that's the point.

The Review Desk (⌘G) by turn for a Claude Code task: two turns, turn 1's diff of src/db.ts, the changed files and a commit message drafted from the turns.

Two agents also mean two plans and two usage pools, so cross-vendor review costs real money. Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. The current release is 2.3.4, from October 15, 2026. It isn't affiliated with Nous Research, Anthropic or OpenAI. Download it or read the code on GitHub.

When same-model review is fine

Not every review needs a second vendor. A same-model pass still catches things that don't depend on the writer's assumptions: a file left half-edited, a test that never ran, a TODO that shipped. Linters and type checkers catch a lot of that for free.

The second vendor earns its cost on logic: boundary cases, error paths, concurrency, anything where the bug is a wrong belief about how the world behaves. That's where shared mistakes hide, and where a reviewer that thinks like the writer helps least.

Does your team let the same model write and review? If you've tried cross-vendor review, did the second agent catch what the first one kept missing? Tell me in the comments.

If this was useful

The same bias shows up anywhere one model grades another, including the eval suite behind your own LLM feature. Observability for LLM Applications has a chapter on LLM-as-judge, code-based checks and human-in-the-loop review, which is the part to read if a judge model gates anything you ship.