Five of My Eight Agents Left Out the Same Caveat. A Vote Would Have Made It Official.

I had eight AI agents review the research on multi-agent LLM systems, one subtopic each. None of them messaged another. Six came back with the same figure from the same paper: 17.2x. In five of the six reports it became a plain error multiplier: agents without cross-checks "amplify errors 17.2x". Four reports quoted it inside a design rule.

The paper does not define it that way. In the published version the figure measures extra work inside a run. By the paper's own success rates, those agents fail about 1.2 times as often as one agent.

I never ran a vote. Had I merged the eight reports by counting heads, that reading would have gone into my review with five reports behind it. What caught it was duller than a debate: a second agent per report that reopened the cited sources and compared the main claims with the text.

I ran no benchmark. Every outside number below names its source. The first-hand numbers are about how the review was made, and they do not flatter me.

TL;DR: In the largest matched-budget comparison I found, 260 configurations, six benchmarks, a team of agents changed the score by 0.0% on average, with extremes of +80.8% and -70.0%. That paper's 17.2x figure measures extra work, not failed tasks. Five of my eight agents passed it on as an error multiplier. A check against the paper caught it. A head count would have kept it. The swarms with the biggest reported wins have work that splits into independent parts and, in the coding ones, a referee that is not another agent's opinion. Start with one agent. Without the split and the referee, do not go past a handful.

17.2x measures extra work, not failed tasks

The figure comes from Kim et al., a study that compares one agent with four kinds of team on six agentic benchmarks: 260 configurations, reasoning-token budgets matched. The preprint abstract from December 2025 said independent agents "amplify errors 17.2x", and the text defined the figure as a "relative failure probability". The April revision kept the number and changed what it is. The version published in Nature Machine Intelligence in July 2026 keeps the April definition.

Read as task failures, the figure never matched the paper's own numbers. Mean success is 0.466 for one agent and 0.370 for independent agents, in the preprints and in the journal. That is about 1.2 times the failures (computed from those two rates), close to the paper's 1.1 and nowhere near 17.

The direction survives. Independent agents with no cross-checking were the weakest team design on five of the six benchmarks, and posted a small gain on the sixth. The paper also reports that the trace-level factor predicts performance once it is scaled by the single-agent baseline (robust p = 0.030), though not on its own (p = 0.658).

The paper makes the misreading easy. Its April introduction says independent systems "amplify trace-level errors 17.2x", and the published text still has them "propagate errors to 17.2x baseline".

My eight agents produced 270 claims. A second agent per report reopened the sources for the claims that carried the most weight, 118 of the 270. 46 of the 118 came back changed. I count checker changes, not proven errors. By the checkers' own notes, most were about what a number means, not the number itself.

Six reports cited 17.2x. The run log shows that all six agents had the April preprint open and printed the same sentence from its introduction, with the label "trace-level" in it. Three wrote the label into the claim and left it out of their own summary and design rule. One dropped it at once. One quoted the December abstract and said so. A checker with the paper open changed all five, and four of the five changes said the figure is not an error rate. The sixth report kept the label, and its checker marked it confirmed.

This is one run of a literature review, with no control arm and no repeat. Five of eight is not a statistic. It is an anecdote with an audit trail. The agents shared a model, a prompt template and a source sentence, and I cannot separate the three. Whichever it was, the count was worth less than it looked: five reports that lean on one sentence are one report.

Human group research is most useful as a list of test designs

I started this review with the theory of human cooperation, as background.

These are pairs, not replications. Only the hidden-profile row, where each member holds part of the facts, tests the same kind of task as its human study.

That row carries the fix I would try first. Neither model scale nor individual accuracy reliably predicted group accuracy. A protocol raised it: each agent shared one or two facts and one reason the leading answer might be wrong, for two rounds, and only then voted. On an 18-task subset with four agents, GPT-4.1 went from that subset's baseline of 0.04 to 0.80.

The first row is where the analogy breaks. Ingham et al. (1974) had men pull a rope alone while believing others pulled with them, and they still pulled about 15% less. Agents do not ease off because they think someone else is pulling. Their loss is redundancy: correlated outputs that add cost and little independent evidence.

Thirty debating copies of a 7B model count as one or two

Bertalanič and Fortuna put Ringelmann's name on a preprint that measures effective team size: how many independent answers a team of N really contains. Thirty copies of Qwen2.5-7B debating math word problems contained about 1.8. On two multiple-choice sets the figure was 1.12 and 1.15. The fitted ceilings for that model, a separate estimate, are 1.2 to 1.8 effective agents. They tested up to 30.

Thirty copies that only rechecked their own work, with no peer messages, reached about 6 on the math set by the same measure. On that set the paper treats the measure as an upper bound, for the 1.8 as well as the 6, because scattered wrong answers also count as diversity. The two setups stayed within five points of each other on accuracy, and debate spent 64K prompt tokens per effective agent against 1.4K. Mixing three model families raised the debate ceiling, modestly, to between 1.8 and 2.9. The HiddenBench protocol above works on a different problem: facts that are split, not answers that are copies.

Nobody ships 30 copies of a 7B model on quiz questions, and the objection is fair: this is one preprint, question answering plus a small code pilot, with plurality voting. The only closed model run in a team was Gemini 3.1 Flash-Lite, and the paper lists frontier-class debate, tool-using agents and multi-step planning as untested.

I have to be fair to debate as well. In Choi et al., 2025, averaged over seven benchmarks with five Qwen2.5-7B agents, one agent scored 0.72, the best standard debate 0.74 and voting with no debate 0.77, and the voting edge comes from two arithmetic tasks. In Wynn et al., 2025, all 10 three-agent groups scored lower after debate than their own initial vote on CommonSenseQA. On MMLU 7 of 10 fell. On GSM8K 6 of 10 rose. So it would overstate the case to say debate destroys accuracy. In these studies debate does not reliably beat a plain vote, and you pay for every round.

In the largest matched-budget study, the average team effect is 0.0%

The result in Kim et al. that deserves the fame is the average. Across six benchmarks and four team designs, the mean change against one agent is 0.0%. The extremes are +80.8% for a centralized team on a financial analysis task that splits into parts and -70.0% for independent agents on sequential planning, where every team design lost 39 to 70%. The person selling you a swarm and the person telling you swarms are a scam can both cite it.

The match is on reasoning-token budgets, and the paper's own token-efficiency measure still separates the designs. One agent gets 67.7 successes per 1,000 tokens, and the three designs that communicate get 13.6 to 23.9.

Other head-to-head results lean against the team. They are not on one scale: only the first matches budgets, and the third compares cost.

At 10,000 reasoning tokens, three of Tran and Kiela's five team designs draw level with the single agent. Its lead is widest at 1,000.

Wall-clock speed is a separate question, and it matters when tokens are cheap. Mieczkowski et al., 2026 tested Amdahl's law on teams of one to five agents. Pre-assigned tasks gave a median speedup of 1.36x. Self-organization gave 0.88x, slower than one agent. On sequential tasks, pre-assigned teams spent 5.83x the tokens for a 1.13x speedup.

When a multi-agent result arrives without the single-agent score at the same tokens, ask the question from my harness post. Which one did you benchmark: the structure, or the budget?

The coding swarms with the big numbers have a referee

None of that makes swarms hype. Some of the largest numbers in the review come from many attempts or many agents. No two rows below use the same measure.

Read the last column first: two preprints, two vendor reports, and no row that compares at equal cost. The last source names the confound itself. Multi-agent systems "work mainly because they help spend enough tokens to solve the problem", and token usage alone explained 80% of the performance variance on BrowseComp, a browsing benchmark.

Even so, two things recur. The work splits into parts that do not depend on each other: attempts at one issue, failing tests of a compiler, branches of a research question. And in the first two rows, a test suite that is not an agent has the last word. In both rows that suite is the benchmark's own grader, and in the first it also works as an answer key. Your tests will be a weaker referee than that.

The compiler project shows both from the inside. Parallelism stalled when all 16 agents hit the same bug while compiling the Linux kernel, which is one giant task. It came back when a known-good compiler served as an oracle, a trusted reference, and split the work by file. A person built that referee. Cursor reports a similar stall: 20 equal agents coordinating through a shared file with locks had the throughput of two or three (Cursor, 2026, a vendor report).

So the coding swarms with the big numbers are not teams that talk their way to an answer. They are closer to a search with a referee.

Isolation is not independence. Five agents in my run, none of which messaged another, repeated the same claim, and only the check changed it.

What I would build

Two questions first. Is there an automatic check on the result: tests, a verifier, an oracle? Does the work split into parts that do not depend on each other? Two yeses, and a large swarm is worth a pilot. Without two yeses I would not go past a handful, because most of the head-to-head evidence stops there: teams of three or four in Kim et al.'s main grid, one to five agents in the speedup tests. If the first answer is no, build the check before you build the swarm.

1. One agent first, at the same budget. Match the thing you are short of, tokens, money or time, and report the other two. If the team got more budget, you benchmarked the budget.
2. A check against something outside the agents wherever their outputs are combined. What counts: tests, an oracle, or the current version of the source. Another agent's agreement does not. An agent that reopens the source counts for the passage it finds, and its own opinion still does not. In my run that check changed 46 of 118 claims.
3. Isolation first, then a structured exchange when the agents hold different facts. Agents work alone, then each posts one or two facts and one reason the leading answer may be wrong, and only then do they vote. The vote pools what the agents know and does not replace the check in rule 2.
4. For many agents, assign the work instead of letting them negotiate it. In Cursor's reports planners do not write code and workers do not talk to each other, and in the speedup tests pre-assigned tasks beat self-organization. The evidence is thin and not one-sided: two of the three agent teams in the swarm table ran with no orchestrator. With planners above workers, the cost lever is the model mix. On one SQLite build Cursor reports similar quality for $1,339 with a strong planner and cheap workers, against $10,565 with one frontier model in both roles (Cursor, 2026, a vendor report).
5. Measure the noise before the gain. In Kaliyev and Maryanskyy, 2026, a single preprint, a coordination protocol and its no-coordination baseline had equivalent setups on the first trial, where coordination was not yet active. Their success rates on that trial still differed by 10 points in one run of 100 tasks and by 0 in a rerun of the same tasks. Run the same setup twice before you compare two.

What this post might get wrong

1. I reproduced nothing, and I skipped my own first rule. My numbers are counts from one review run. I never ran the review with one agent at the same budget, so I cannot say the eight earned theirs.
2. The evidence is uneven. The big-swarm results are vendor reports and single preprints. The debate studies ran mostly small models on question answering, and the wins are on coding and research, so I cannot separate the structure from the model or the task. My review turned up no equal-cost comparison for swarms above ten agents, and the matched-budget study uses teams of three or four, with a side experiment up to nine. One report I cite cuts against the referee claim: Cursor says its SQLite swarm went on to pass 100% of a test suite it was never told about, and suspects that stacked agent review was a major contributor to quality.
3. My referee was another agent, and it missed things. The checkers were the same model as the researchers, and both had the same paper open. What differed for the checkers was the brief: take one claim, find the passage, compare. Two of them still left "trace-level" unexplained, and 152 claims got no verdict. One of my own briefs described the paper by its December abstract. Two rounds of checks on this post confirmed a mean of -0.3% against the April preprint and took it for the latest version. The published version says 0.0%. It surfaced only because one checker found the journal named in one of the eight original reports.
4. Matched budgets answer one question. They say whether a structure beats one agent at the same spend. They do not cover a job that will not fit in one context, or a team split to keep tools and permissions apart. Nothing here measures either.

The swarm results I trust most are the ones where a test can say no. Where nothing can, the studies here mostly found one model agreeing with itself, at several times the price.

What sits where your agents' outputs are combined: a test, a source, or another agent's opinion? And if you have an equal-cost number for a swarm above ten agents, post it. I could not find one.

Sources

• Kim et al., 2026, "Capable language models can outgrow the benefits of collaboration", Nature Machine Intelligence - 260 configurations, six benchmarks, matched reasoning-token budgets: mean team effect 0.0%, from +80.8% (centralized team, Finance Agent) to -70.0% (independent agents, PlanCraft). The 17.2x is a trace-level factor. The paper puts the task-level error ratio at about 1.1 and names no team design.
• The same study as a preprint, "Towards a Science of Scaling Agent Systems" (v3, April 2026) - the version my agents read. Its mean team effect is -0.3%. Versions 1 and 2, from December 2025, carry the abstract with "amplify errors 17.2x".
• Bertalanič and Fortuna, 2026, "The Ringelmann Effect in Multi-Agent LLM Systems: A Scaling Law for Effective Team Size" - 30 debating copies of Qwen2.5-7B carry 1.1 to 1.8 effective agents; 30 self-correcting copies reach about 6 on math word problems, an upper bound by the paper's own account. Preprint, question answering plus a small code pilot.
• Choi et al., 2025, "Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?" - one agent 0.72, best standard debate 0.74, plain vote 0.77 with five Qwen2.5-7B agents, averaged over seven benchmarks.
• Wynn et al., 2025, "Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate" - on CommonSenseQA all 10 group compositions scored lower after debate than their own first vote; on MMLU 7 of 10 did, and on GSM8K 6 of 10 rose.
• Stasser and Titus, 1985, "Pooling of Unshared Information in Group Decision Making: Biased Information Sampling During Discussion" - the original hidden profile: 83% of groups chose the best candidate with full information, 18% when each member's facts favored a rival.
• Li et al., HiddenBench (v4, 2026), "Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs" - 80.7% for one agent with all facts, 30.1% for a group with split facts; a share-then-decide protocol lifts GPT-4.1 from 0.04 to 0.80 on an 18-task subset.
• Tran and Kiela, 2026, "Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets" - one agent 0.418 against 0.333 to 0.388 for five team designs at 1,000 reasoning tokens.
• Khatua et al., 2026, "CooperBench: Why Coding Agents Cannot be Your Teammates Yet" - two agents in one repository had on average 30% lower success rates than one agent doing both features, per the abstract; the paper's Table 5 puts the pooled loss at 41%.
• Vals AI, 2026, "Do Agent Teams Pay Off? A Case Study on Vibe Code Bench" - teams cost 1.8 to 5.1x; one of four score differences is significant, a 7.3-point gain for the team.
• Mieczkowski et al., 2026, "Language Model Teams as Distributed Systems" - 1.36x median speedup with pre-assigned tasks, 0.88x with self-organization.
• Brown et al., 2024, "Large Language Monkeys: Scaling Inference Compute with Repeated Sampling" - 15.9% to 56% on SWE-bench Lite with 250 attempts, counting an issue as solved when any attempt passes the benchmark's unit tests. Preprint.
• Zhan et al., 2026, "Agensh: Scaling Organizational Intelligence to 1,024 Agents" - mean test-pass rate 19.3% with one agent and 28.8% with 128 on the five hardest ProgramBench tasks, six hours each. Single preprint, no cost accounting.
• Anthropic, 2026, "Building a C compiler with parallel Claudes" - 16 agents, 100,000 lines, about $20,000. Vendor report.
• Anthropic, 2025, "How we built our multi-agent research system" - +90.2% over one agent on an internal evaluation; token usage alone explained 80% of the variance on BrowseComp. Vendor report.
• Cursor, 2026, "Scaling long-running autonomous coding" - 20 equal agents coordinating through a shared file with locks had the throughput of 2 or 3. Vendor report.
• Cursor, 2026, "Agent swarms and the new model economics" - $1,339 against $10,565 for similar quality on one SQLite build. The same post says the swarm passed 100% of a test suite it was never told about. Vendor report.
• Kaliyev and Maryanskyy, 2026, "How Much Coordination Gain Is Real? A Paired Noise-Floor Protocol for Multi-Agent LLM Benchmarks" - a 10-point gap between equivalent configurations on one run of 100 tasks, 0 on a rerun. Single preprint.
• Human classics cited by name and year only: Smith 1776, Ringelmann 1913, Ingham et al. 1974, Brooks 1975.

AI disclosure: the literature review and this post were written with AI assistance (a multi-agent research run in Claude Code). Separate checker agents reopened the sources for the main claims, and again for the numbers in this post. I chose the framing and I stand behind the conclusions. Any errors that remain are mine.