How I calibrated an LLM judge to grade like me, 25 cheaper

Businesses that sell technical products answer the same kind of question every
day. "What's the accuracy on this range?" "Can it measure through a coating, and
how thick?" "Does it come with a calibration certificate?" The answers sit in
product datasheets and manuals.

I took 27 of those PDFs, three of them scans with no text layer, and wrote 47
questions the way customers actually ask them. Then I had four AI systems answer
every question and graded all 188 answers.

Running the systems took an afternoon. Building a judge I could trust took the
rest of the work, and that is what this post is about.

The result

My pipeline is Claude Sonnet over the API with page citations. The Claude app ran
Opus with the documents in a project, ChatGPT ran with thinking off, and Chatbase
was on the free plan with its default model.

Every system got the same PDFs and the same instructions. One run each, so I read
a one-question gap as a tie.

No system invented a spec value. I checked every claim twice: a claim checker
read each one against the page it came from, and I went through the ones it was
unsure about by hand. The weaker systems failed more quietly. They said an answer
wasn't in the documents when it was.

Where the misses came from

Scanned pages. Systems that only read the text layer saw blank pages. Every
question answered only by a scan came back "not specified".

Answers that need a second look. A product advertises one range on the front
page. A different mode of the same product stops at a tenth of it, and the
customer's question is about that mode. The same pattern showed up in accuracy
tables split by range and in specs that change with the material.

Unit conversions. The customer asks in one unit. The datasheet lists the
other unit in one row and a separate, lower limit in the customer's unit in
another. Converting the first number gives a confident wrong answer.

The website and the datasheet disagreeing. The product page said one value
and the datasheet said half of it. My rule: the datasheet wins, and the reply says
the page may be wrong. Two systems missed it.

Rules that live in no PDF

Before running anything I reviewed every question. I dropped three that no
customer would ask, corrected two gold answers, and turned the "not in the
documents" cases into rules:

• Datasheet beats product page. Give the datasheet value and say the page may have an error.
• Never a bare "not in our documents". Say it isn't in the published documents, and give an email to ask.
• Certifications only if the document says so. Otherwise, email for confirmation.
• Canned answers for the two most common questions.

After grading I added one more: answer what was asked. Extra conditions
confuse buyers and invite more questions.

That short list did more for answer quality than any prompt trick.

Why the judge needs a judge

Grading 188 answers by hand takes hours, so the usual move is an LLM judge: give
a model the question, the gold answer and the answer, and ask for correct, partial
or wrong. You only know the judge is any good if it agrees with the person whose
standard matters. Here, that's me.

So I graded 20 answers blind: no system names, five per system, with some likely
mistakes mixed in so the judge would be tested on errors too. Then I compared.

First try: 13 of 20. Most of the gap was in my own answer key. Three gold
answers were wrong. Grading real answers showed me that I answer from the
datasheet as printed, even when a conversion on it looks off, and flag the
document separately. My key hadn't been written that way. Once I fixed it,
agreement jumped.

Plain agreement flatters a judge when most answers are correct. A judge that
marks everything "correct" would agree 80% of the time here and understand
nothing. Cohen's kappa corrects for that.

Kappa asks how far above luck the judge got, as a share of how far above luck it
could have got. 0 means no better than random; 1 means it matched me every time.

Two judges

The hybrid gives each part the job it does best.

Code pulls the numbers out of the gold answer and the answer being graded,
and records which match. It never confuses ±(1.2% + 5) with ±(0.8% + 5).

Jev handles meaning. It's a System One model from
TypeSafe: it reads natural language and returns typed answers with
probabilities, with no prose to parse. One request per answer asks four
questions at once:

{
  "verdict": {"type": "choice",
    "instructions": "Grade `answer` to `customer_question` against `gold` and `notes`. Use `code_checks` for which gold numbers the answer contains.",
    "criteria": {"correct": "...", "partial": "...", "wrong": "..."}},
  "brush_off": {"type": "noul",
    "instructions": "Does `answer` brush the customer off, such as a bare 'not in our documents' with no next step?"},
  "unasked_extra": {"type": "noul",
    "instructions": "Does `answer` add specs, conditions or other models that `customer_question` didn't ask about?"},
  "rule_followed": {"type": "noul",
    "instructions": "Does `answer` do what `rule` says?"}
}

A Choice picks one of the grades and returns a probability for each. A Noul is
the probability that a statement holds. The judgment comes back as numbers, so
the policy stays in code where I can see it and change it. One real answer from
the run:

verdict        correct 0.98 · partial 0.02 · wrong 0.00
unasked_extra  0.91  → at or above 0.9: "too much information" flag

About 2,000 tokens per answer, under a second, and all 188 answers for about three
cents.

Sonnet with the original PDF only sees the few claims Jev isn't sure about.

Keeping myself honest

I wrote the pass line down before I graded a second set of 20: at least 16 of
20, kappa at least 0.6. That second set got used once, as a test, and never for
tuning.

It caught a mistake. On the first 20, I had tuned a penalty: if Jev was 90% sure
an answer added unasked details, the grade dropped to partial. It looked perfect
there. On the fresh set it caused two of the three disagreements. The judge
passed the line (17 of 20, kappa 0.63), but barely.

Reading those two answers showed why. I grade substance and length separately.
An answer can be right and too long. The detector itself was right: it flagged
exactly the four answers across both sets that I had found too long, and no
others. The penalty was the wrong policy. So it became a flag beside the grade.
Without it the judge agrees with me 19 of 20, kappa 0.85. That change came after
I'd seen the second set, so the next fresh set is where it has to hold.

The loop I'd run every time

I did it in a worse order: ran the systems, graded everything with an untested
judge, reported numbers, then calibrated. Next time:

1. Build the golden set: questions as asked, reviewed gold answers.
2. All systems answer, same documents, same instructions.
3. I grade 20 answers blind.
4. The judge grades the same 20. I read every disagreement and fix the cause, which is usually the answer key.
5. Write the pass line down, then test on a fresh 20. Use it once.
6. Only then grade everything and report.

A golden set isn't finished until a person has checked real answers against it.

What it means

On product datasheets, a frontier model with the PDFs attached is already very
good: my pipeline and the Claude app tied. Accuracy is the starting point. The
hard part is getting that accuracy into the place where buyers ask, across a
catalogue too big for one prompt, while datasheets change and the website drifts
away from them. Then proving it on the business's own questions, with a judge
calibrated to the person who answers them. That is what I'm building now.

The whole run cost about $8 in API calls. Most of that went on mistakes I won't
repeat. Now I estimate every paid run from a one-question trial, counting every
path that can call a paid model.