You Cannot Unit Test a Summary. You Can Still Test It.

Every developer I know has the same reflex when a feature is done: write a test that feeds it an input and asserts the output. That reflex is correct for almost everything we build. Click save, the record is saved. Parse the date, get the date.

Then you ship your first AI feature and the reflex quietly stops working. Ask a model to summarise the same lecture twice and you get two different summaries, both fine. Ask it to classify a support message and it is right ninety-nine times and confidently wrong on the hundredth. expect(output).toBe(expected) has nothing to hold on to.

What I see teams do next is the dangerous part. They stop testing. The feature gets "tested" by someone on the team trying a handful of prompts they picked themselves, reading the answers while hoping it works, and calling it good. That is a demo, not a test. It passes because the person running it chose the inputs.

The thing that replaces the unit test is an evaluation set, usually just called evals. I build these for client AI features at my studio, and this is the version of it I wish someone had handed me the first time.

What an eval set actually is

Three things, nothing more exotic:

1. A collection of real inputs.
2. A written definition of a good answer for each one.
3. A score you record every time something changes.

The score matters less as an absolute number than as a trend. You are not trying to prove the feature is perfect. You are trying to know, when you change a prompt on Thursday, whether Thursday made things better or worse.

Real inputs, not the ones you would have picked

The examples have to come from the actual workload: real customer messages, real documents, real call transcripts, real lecture recordings. Invented examples are always cleaner than reality, and a feature tuned on clean examples disappoints on messy ones.

If there is no production data yet, take the closest thing you have. Old support tickets. Sample documents from an early customer. Recordings from a pilot. Anything a human produced without knowing your model would read it.

Then go looking for the uncomfortable cases on purpose:

• Ambiguous requests, where the right behaviour is a clarifying question, not a guess.
• Out-of-scope requests, like a caller asking a booking assistant for legal advice. The right answer is a polite refusal or a handoff.
• Bad input: blurry scans, half-finished forms, background noise, mixed languages.
• Adversarial input: text that tries to make the system ignore its instructions. This matters most once the system can take actions, which I wrote about in giving an AI agent write access one verb at a time.
• High-stakes cases, where a wrong answer costs money, trust or safety.

You need fewer of these than you think. A few dozen well-chosen examples that someone actually reads will surface more real problems than thousands of synthetic ones that nobody looks at. The set grows from production later.

Writing down "good" is the real work

For each example, someone who understands the business writes down what a good response looks like. Sometimes it is exact: the invoice total is this number, the caller wants Tuesday at 3pm. More often it is a short checklist: mentions the deadline, does not promise a refund, stays under a paragraph, hands off if the customer is upset.

This is where most of the value lives, and it has nothing to do with code. Writing the definition forces the team to agree on what the feature is for. If two people want opposite behaviour from the same assistant, you want to find out while filling in row twelve of a spreadsheet. That argument is free in week two and expensive after launch.

Three ways to score, and you need all of them

Code checks. Anything a function can verify, a function should verify. Did extraction return the right fields? Is the date valid? Did the voice agent call the booking tool with the correct slot? Is the reply under the length limit? Did it include a link it was told never to share? These are fast and cheap and run on every change. They are also the closest thing to your old unit tests, so they are the easiest place to start.

Human review against a rubric. Some qualities need a person: is this summary accurate, is the tone right. The trick is a rubric specific enough that two reviewers grading the same output agree. "Rate quality from one to five" produces noise. "Does it state any fact not found in the source?" produces signal.

A model grading a model. Using a second model to grade outputs against the rubric scales where humans do not, and it is genuinely useful. It is also wrong in its own consistent ways. I treat it as an assistant to human review: check its grades against a human sample regularly, and never let it be the only judge of anything high-stakes.

The harness can be embarrassingly small

People hear "evals" and picture a platform. The first version I recommend is a spreadsheet of inputs and expected checks, plus a script that runs them and prints a score next to the last score. Something shaped like this (illustrative, not lifted from a client repo):

type EvalCase = {
  id: string;
  input: string;
  checks: Array<(output: string) => boolean>;
  mustNotFail?: boolean; // e.g. never quote a price
};

async function runEvals(cases: EvalCase[], run: (input: string) => Promise<string>) {
  let passed = 0;
  const hardFailures: string[] = [];

  for (const c of cases) {
    const output = await run(c.input);
    const ok = c.checks.every((check) => check(output));
    if (ok) passed++;
    else if (c.mustNotFail) hardFailures.push(c.id);
  }

  const score = passed / cases.length;
  console.log(`score ${(score * 100).toFixed(1)}%  hard failures: ${hardFailures.length}`);
  return { score, hardFailures };
}

That is it. The interesting part is not the loop, it is the mustNotFail flag, which brings me to the mistake I see most.

Decide the launch bar before you see the number

Once a team has seen a score, it is very easy to argue the score is acceptable. 87% sounds fine when you are the one who has been working on it for three weeks.

So agree the bar in advance with whoever owns the outcome:

• What score on typical cases is required to launch.
• Which failures are unacceptable at any rate: medical or legal advice, a wrong price, another customer's data. These are the mustNotFail cases, and one of them blocks the release no matter what the average says.
• What happens when the system is unsure. Often the right design is a handoff to a person, which turns a potential failure into a safe outcome. Confidence-based routing is its own topic, and I went deep on it for document extraction in your extraction is 98% accurate, one document in three still needs a human.

It earns its keep after launch

An AI feature changes in ways ordinary code does not, and most of those changes do not show up in a diff review:

• Prompt edits. A small wording fix for one problem can quietly break three others.
• Model updates. Providers update and retire models. The same model name can behave differently month to month, and eventually you may be forced to switch.
• Data changes. If the feature answers from your documents, reorganising the documents changes the answers.
• Cost changes. Dropping to a cheaper model to control spend is sensible (I covered the economics in model calls are metered), but only if you can show quality held.

Run the set on every one of these and a nervous judgement call becomes a comparison. Put the old score next to the new one where the team sees it, and let a regression block the change the way a failing test would.

Production is the eval set's real source

Pre-launch evals are an educated guess about traffic. Production is the real thing, so the set should keep learning from it:

• Log enough to investigate, within your privacy rules: input, output, what the system decided.
• Watch for failure signals: a user rephrasing the same question, asking for a person, a staff member correcting a result, a thumbs down.
• Turn every real failure into a new case with the correct answer written down, so the same mistake gets caught automatically next time.

Voice is where I feel this most. With the voice agents we built for CallGuard AI and CallSetter AI, and the multilingual voice and SMS intake behind Fortell AI, replaying realistic calls (interruptions, changed minds, requests the business does not handle) is part of how changes get checked. Calls are fast and there is no undo, which is why I called a related post there is no repro for a phone call.

It applies outside voice too. For LectureNotes AI, my own app that turns lecture recordings into summaries and outlines, the useful eval questions are concrete: did the summary drop a key takeaway, or add something the lecturer never said? Those two checks catch more than any general "quality" score.

The honest cost

A first eval set is not a big project. It is a few days inside the build, and most of that effort comes from someone who knows the business writing down what good looks like. The part that actually costs something is attention. Somebody has to own the definition of good and review failures regularly. If that somebody is nobody, the set goes stale, and you are back to shipping on impressions.

If you are wiring AI into a product that already exists, the wider picture of where it fits is in nobody opened your product to chat with it. This post started life as a client guide on the Null Studio blog, reworked here for people who write the code.

What does your eval setup look like right now? I am especially curious whether anyone is trusting a model-as-judge for anything high-stakes, and how you check it.