LLM-as-a-Judge Position Bias: Swap A and B and the Winner Flips

My new prompt beat the old one 63 to 37. I had the chart ready for the team channel.

Then, mostly out of paranoia, I reran the exact same eval with one change: the new prompt's answer went in slot B instead of slot A. Same 200 questions, same judge model, same rubric. The new prompt now lost.

Nothing about the answers had changed. Only their order. That is LLM-as-a-judge position bias, and if you run pairwise evals without controlling for it, some of your "wins" are just a record of which answer you happened to paste first.

TL;DR

• LLM-as-a-judge position bias means the judge model prefers an answer because of where it sits in the prompt (first or second), not because of what it says.
• Measure it by running every pair twice with the order swapped. The share of pairs where the verdict follows the slot instead of the answer is your flip rate.
• Only count a win when the judge picks the same answer in both orders. Everything else is a tie.
• Pointwise scoring against a rubric removes the ordering problem but adds its own calibration drift, so use it alongside swapped pairwise, not instead of it.
• Position bias stacks with length bias and self-preference bias. Check all three before trusting a leaderboard you built yourself.

What is LLM-as-a-judge position bias?

LLM-as-a-judge position bias is a systematic preference for the answer in a particular position of a pairwise comparison prompt, independent of answer quality. Depending on the model and the prompt, the judge may favor the first answer or the second. Either way, the verdict carries information about your prompt layout that you did not intend to measure.

A typical pairwise judge prompt looks like this:

Question: {question}

[Answer A]
{answer_a}

[Answer B]
{answer_b}

Which answer is more helpful and correct? Reply with "A" or "B".

When A is clearly better, the judge usually gets it right. The trouble lives in the middle: two decent answers, different in style, roughly equal in substance. That is exactly the zone where you are testing prompt tweaks, and exactly the zone where the judge falls back on a default.

Why does an LLM judge prefer one position?

The judge prefers one position because a pairwise comparison is not symmetric for a decoder-only model. It reads A first with no context, then reads B in the light of A, then produces a token. Three things push on that token:

1. Causal attention is one-directional. The representation of Answer A is built before the model has seen B. B's tokens can attend to A, but A's tokens never attend to B. The two answers are literally processed differently.
2. Training data has order habits. Human preference data, forum threads, and graded examples all have their own quirks about where the "good" answer tends to appear. The model soaks those up.
3. Low confidence collapses to a prior. When the logits for "A" and "B" are close, a tiny structural nudge decides the outcome. On near-ties, the nudge is the whole verdict.

That third point matters most. Position bias is not the judge being dumb on easy cases. It is the judge being uncertain on hard cases and resolving the uncertainty with the slot.

How much can position bias distort an eval?

It can manufacture a winner out of a tie. Here is the arithmetic that convinced me.

Say your two prompts are genuinely equal. Out of 200 questions, the judge is confident on 120 and splits them fairly, 60 each. On the other 80 it is unsure and picks slot A about 75% of the time.

If the new prompt always sits in slot A:

• Confident cases: 60 wins
• Uncertain cases: 60 wins (75% of 80)
• Total: 120 of 200, a 60% win rate for a prompt that is no better

Swap the order and the same math hands the old prompt the 60%. A one-directional eval would have shipped a change that does nothing. If you randomize order per question, the bias averages out in the headline number, but you still can't tell a real 55% from noise, because each individual verdict is still contaminated.

How do I measure position bias in my own judge?

Run every pair in both orders and count how often the verdict tracks the slot instead of the answer. This is cheap to add to any existing pairwise harness:

def judge(question, first, second) -> str:
    """Returns 'first' or 'second'. Wrap your judge call here."""
    ...

def swap_eval(items):
    consistent_new, consistent_old, flipped = 0, 0, 0
    for q, new, old in items:
        v1 = judge(q, new, old)   # new in slot A
        v2 = judge(q, old, new)   # new in slot B

        new_wins_1 = v1 == "first"
        new_wins_2 = v2 == "second"

        if new_wins_1 and new_wins_2:
            consistent_new += 1
        elif not new_wins_1 and not new_wins_2:
            consistent_old += 1
        else:
            flipped += 1  # verdict followed the slot

    n = len(items)
    return {
        "new_wins": consistent_new / n,
        "old_wins": consistent_old / n,
        "flip_rate": flipped / n,
    }

Two numbers come out of this that you never had before:

• Flip rate: the share of pairs where the judge contradicted itself. This is your judge's noise floor for this task.
• Consistent win rate: wins that survived the swap. This is the number you can actually defend.

In my eval, the flip rate was large enough that the "63% win" turned into a lopsided pile of ties with a small, honest edge for the new prompt. Less exciting chart. Much more accurate chart.

A useful rule of thumb from running this a few times: if the gap between consistent wins is smaller than the flip rate, you have not shown a difference. Get more samples or a sharper rubric.

How do I fix LLM-as-a-judge position bias?

You fix it by making order irrelevant to the final score, then shrinking the uncertain zone where bias operates. In rough order of payoff:

1. Swap and agree. Score a pair as a win only when both orders agree, as in the code above. This doubles judge calls. It is also the single change that turns a pairwise eval from a vibe into a measurement.

2. Allow an explicit tie. Forcing "A" or "B" on a near-tie guarantees the slot decides. Give the judge "A", "B", or "tie", and tell it when a tie is correct. Some verdicts that used to be coin flips become honest ties.

3. Put reasoning before the verdict. Ask the judge to list the specific differences first, then decide. The output format should end with the choice, not start with it:

First list concrete factual or completeness differences between the answers.
Then output exactly one line: VERDICT: A | B | TIE

This does not remove the bias, but it gives the model something besides position to anchor on.

4. Average probabilities, not labels. If your API exposes logprobs, read P("A") in both orders and average P(new wins) across them. You get a graded score instead of two brittle labels, and the position effect cancels in expectation.

5. Add a pointwise pass. Score each answer alone against a rubric (say, 1 to 5 on correctness, completeness, format). No second answer means no order. The catch: pointwise scores drift and cluster, so a judge might give everything a 4. Use it as a cross-check on the pairwise result, not a replacement.

What other judge biases stack with position bias?

Position bias rarely travels alone. Two others show up in the same evals and compound it:

• Length bias. Judges tend to favor longer, more thorough-looking answers even when the extra text adds nothing. If your new prompt makes outputs longer, part of its "win" may be word count. Check win rate bucketed by length difference.
• Self-preference bias. A judge can favor outputs that sound like its own. If the judge and one candidate come from the same model family, use a judge from a different family for at least a spot check.

Stacked together, these can make a flat change look like a clear improvement. The swap test catches the first one. The other two need their own slice of the data.

A checklist before you trust a pairwise eval

• [ ] Every pair was judged in both orders
• [ ] Reported win rate counts only order-consistent verdicts
• [ ] Flip rate is reported next to the win rate
• [ ] The judge can answer "tie"
• [ ] Verdict comes after reasoning in the output
• [ ] Win rate checked across length buckets
• [ ] Judge model family differs from at least one candidate, or was spot-checked

So does swapping A and B really flip the winner?

Yes, often enough to matter. LLM-as-a-judge position bias makes a pairwise judge favor whichever answer sits in a particular slot, and it hits hardest on near-ties, which is exactly where prompt and model comparisons live. The fix is to judge every pair in both orders, count a win only when both orders agree, report the flip rate beside the win rate, and let the judge say "tie." If your eval harness runs each comparison once, the winner you see may just be the answer you pasted first.

Written by the developer behind Preterview, an interview prep platform.