Only one of our 24 questions split with nobody alone
Most of the interesting results in our divergence study involved a lone model — a claim exactly one model made, or a stance exactly one model took against the rest. But the study's buckets leave one question in a categor
Most of the interesting results in our divergence study involved a lone model — a claim exactly one model made, or a stance exactly one model took against the rest. But the study's buckets leave one question in a category of its own, and I have come to think it may be the most honest result in the set.
The setup: we sent the same 24 questions, unchanged, to five mid-tier models — GPT, Claude, Gemini, Grok and Llama. Six pricing/strategy, six factual with a trap, six coding approach, six recommendations with tradeoffs. A Council read flagged material divergences, and we then hand-classified every question into one of three mutually exclusive buckets:
- Broad agreement: 8 of 24 (33%) — no flagged divergence among the judged answers.
- Lone-model outlier present: 15 of 24 (63%) — at least one sole-source claim or lone dissent.
- Material split with no lone model: exactly 1 of 24 (4%).
That one question was Q22, a recommendation: should a novelist who writes long documents daily use a mechanical keyboard or a laptop's built-in one? The Council flagged 3 divergences on it, and on the core point the judged answers divided 2–2. Nobody was alone. There was no outlier claim to check first, and no majority to defer to. (One method note, as in the study itself: the Council reads a rotating four of the five answers per question — Claude was the omitted family for Q22 — so "2–2" is the split among the judged four.)
Why this matters is what a single-model workflow, and even a synthesis workflow, would do with it. Ask one model and you get one side of a 2–2 split presented as a recommendation, with no sign the other side had equal support. Ask for a synthesis and you get a smoothed answer that has to either invent a winner or flatten the tradeoff into a both-sides paragraph — and in flattening it, deletes the actual finding, which is that competent models, given the same question, landed evenly on opposite sides of a preference-dependent choice.
That is a different signal from the other two buckets, and it calls for a different response. When the panel agrees, the useful reading is narrow: no model saw a reason to dissent — a consensus signal, not proof of correctness. When one model stands alone, the useful move is to check that lone claim or read that lone dissent first, because across this study, of 14 sole-source claims hand-checked, 8 were the highest-value line in their set and 3 were wrong or invented. And when the panel splits evenly with nobody alone, there is nothing to check first and no winner to crown. The split itself is the result: it tells you the question turns on a tradeoff or a preference that the models cannot settle for you, and the decision has to be made on your own criteria.
Across all 24 questions there were 52 material divergences, and exactly half of them (26) involved a lone model — which means the other half were multi-way splits like this one. We have spent most of this series on the lone half, for good reason: it is where both the best lines and the invented ones live. Q22 is a reminder that the other half carries information too, of a quieter kind. An even split is not a failed answer. It is a map of where the genuine disagreement sits.
My practical rule now has three branches: if the models agree, treat it as consensus, not truth; if one model stands alone, check it first; and if the panel splits evenly, stop looking for the winning answer and decide which side of the tradeoff you actually live on.
Full study, method and caveats (including the rotating judging sets, that one judge was also a panel member, and a memory-bleed incident we found and disclosed — which, fittingly, started with this question's hypothetical novelist): https://multio.ai/research/model-divergence
Disclosure: I build MultiO (multio.ai) — One subscription. Every top AI model… Auto sends each message to the most cost-efficient model that can handle it. Compare asks up to three models side by side, and every message shows its exact token charge on a receipt. Free trial: 3,000,000 tokens, no card required.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.