The Null Run Is the Cheapest Control in Your Benchmark

A benchmark number is a claim, and a claim is only as strong as the control that could have falsified it. Before an agent score reaches a README, the harness should be able to return a zero for a system that deserves a zero, and a full score for a system that is not allowed to fail. Those two runs cost less than a coffee break, and they are the difference between a measurement and a slide deck.

Public agent comparisons skip this step with remarkable consistency. A team wires a model into a task set, runs it once, and reports the fraction of green cells without ever asking whether the grader would have accepted an empty string. When that same harness later grades a competitor or a regression, the missing control turns a debugging problem into a marketing problem, and nobody can tell which one they are looking at.

The timing is not accidental. Benchmarking challenges and leaderboard recaps keep landing on DEV's front page, and every one of them makes the same implicit promise: that the scoring pipeline was validated somewhere upstream. The method below is the smallest version of that validation that still produces evidence you can hand to a skeptical reviewer.

The harness is the thing under test first

The first control exists because graders fail silently. A negative control, often called a null run, swaps the model for a function that returns an empty string and asserts that the harness scores it at the floor. If an empty answer earns partial credit, the task set is measuring formatting compliance rather than task completion, and every later number inherits that leniency.

# harness.py — no vendor SDK, no network, runs anywhere
from dataclasses import dataclass
from typing import Callable, Iterable


@dataclass(frozen=True)
class Task:
    task_id: str
    prompt: str
    check: Callable[[str], bool]


@dataclass
class Result:
    task_id: str
    output: str
    passed: bool


def run(model: Callable[[str], str], tasks: Iterable[Task]) -> list[Result]:
    results: list[Result] = []
    for task in tasks:
        try:
            output = model(task.prompt)
        except Exception as exc:  # a crash is a failure, never a skipped row
            output = f"<error: {type(exc).__name__}>"
        results.append(Result(task.task_id, output, task.check(output)))
    return results


def score(results: list[Result]) -> float:
    return sum(r.passed for r in results) / len(results)

Two details inside that snippet carry more weight than they appear to. Exceptions become failed results rather than dropped rows, because a harness that silently loses crashing tasks will report a higher score for a flakier model. The scorer also divides by the full task count instead of the completed subset, which keeps a partially finished run honest.

Three controls, written as tests

The positive control is the mirror image, and it is the one teams forget because it feels redundant. An oracle function returns the known-good answer for every task, and the harness must score it at the ceiling; any task the oracle fails is a broken check, not a hard question. A third, optional control returns fluent prose that answers nothing at all, which is the cheapest way to catch a grader that rewards tone.

# controls.py
from harness import Task, run, score


def null_model(prompt: str) -> str:
    return ""


def verbose_null_model(prompt: str) -> str:
    # plausible shape, zero content — catches graders that grade vibes
    return "Here is a careful analysis of the problem and its trade-offs."


def oracle_model(expected: dict[str, str]):
    def call(prompt: str) -> str:
        return expected[prompt]
    return call

Wiring those into the test suite means the controls run on every commit, not once during a launch week. The assertion messages matter, because the next engineer will read them at 2 a.m. and needs to know which assumption broke.

# test_controls.py
from harness import Task, run, score
from controls import null_model, verbose_null_model, oracle_model

TASKS: list[Task] = load_tasks("tasks.jsonl")  # task_id, prompt, check


def test_null_run_hits_the_floor():
    assert score(run(null_model, TASKS)) == 0.0, \
        "harness awarded credit for an empty answer"


def test_verbose_null_stays_near_the_floor():
    s = score(run(verbose_null_model, TASKS))
    assert s <= 0.05, f"fluent non-answers scored {s:.3f}; checks are too loose"


def test_oracle_run_hits_the_ceiling():
    expected = {t.prompt: t.expected for t in TASKS}
    assert score(run(oracle_model(expected), TASKS)) == 1.0, \
        "a known-good answer was graded wrong; the check is broken"

Run it with python -m pytest test_controls.py -q and treat a red control as a stop-the-line event. A benchmark published while the oracle test fails is a benchmark with unknown grading semantics, and no amount of sample size repairs that.

Ablation turns one score into an attribution

Controls prove the harness works; they do not explain why a model scored what it scored. That requires changing exactly one variable per run and reporting the delta, which is slower and far more informative than sweeping six settings at once. The table below is the shape to publish, with the null and oracle rows pinned at the top so the reader can see the floor and ceiling before any model appears.

Two rules keep that table from turning into decoration. Each delta needs a repeat count and an interval, because a one-run delta of two points is indistinguishable from noise on a fifty-task set. And the baseline row must name the endpoint, date, and prompt hash, otherwise the ablations describe a system nobody can rebuild.

Where a free tier actually fits

Controls and ablations multiply runs, and multiplication is where starter budgets usually die. The null run, the verbose null, the oracle, and three ablations are six passes over the same task set before a single headline number exists, and that count grows quickly when a task set has two hundred items.

This is the narrow place where free model access and a free server option earn their keep, because none of these control runs need the strongest model on the market. A null model needs nothing, an oracle needs a lookup table, and initial ablations only need something cheap enough to run six times without an approval thread. MonkeyCode's free model access and free server option (the announced free allowance is on the order of ten million tokens; confirm the current terms yourself before you plan around it) fit that calibration stage rather than the final reporting stage. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The distinction matters more than the allowance. A free tier is a place to prove the harness before you pay for representativeness, and a control grid produced on a free endpoint should be re-run on the endpoint you intend to publish. Treat any free-tier number as a calibration artifact, never as a leaderboard entry, because the serving stack, rate limits, and queueing behavior are part of what the score is measuring.

Limitations and who should skip this

If the deliverable is a production SLA rather than a research comparison, the control grid is necessary but not sufficient, since free capacity says little about p95 latency under load. If the task set holds fewer than roughly thirty items, a single flaky check moves the score by several points and the controls will dominate the variance you are trying to study. And if nobody on the team can construct an oracle — because the correct answers are genuinely unknown — then the honest move is to publish the task set and the checks without a score, rather than a number whose ceiling was never demonstrated.

Teams shipping a single internal regression suite should also be careful about cost. Six ablation passes per commit is affordable on a free server, but it is wasteful when one fixed prompt and a null run would catch the regression you actually care about; reserve the full grid for release candidates and model swaps.

The practical order is unglamorous: write the null run, watch it fail, fix the checks, add the oracle, then add ablations one variable at a time. If the control grid is the step that keeps getting postponed, the free models and free server are the cheapest way to run it before anyone commits a budget line to the benchmark.