Run It Ten Times Before You Write One Characterization Test

Conclusion first: run your entrypoint ten times and count distinct outputs. If that count is not one, you have no baseline. You have a noise source, and any test you write on top of it will flake.

Most messy-repo refactors stall right here. People write characterization tests against an unstable output, the tests fail randomly, and the tests get deleted. The fix is not better assertions. The fix is measuring nondeterminism before you assert anything.

This post is a workflow for that measurement, plus the smallest safe change that follows it.

Step 0: Pick one entrypoint and one working directory

Pick the smallest command that produces the artifact you care about. Do not start with the whole pipeline.

cd sandbox && ./run_report.sh > /tmp/out1.txt
cd sandbox && ./run_report.sh > /tmp/out2.txt
diff /tmp/out1.txt /tmp/out2.txt | head -40

If the diff is empty, stop reading and go write the test. If it is not empty, you need a census, not a diff.

Step 1: Census the output across N runs

The script below copies the work directory fresh for every run. That kills state leaking between runs, which is the most common false source of drift.

#!/usr/bin/env python3
"""nondeterminism_census.py - count distinct outputs of one command across N runs.

Usage:
  python nondeterminism_census.py --runs 10 --workdir ./sandbox \
      --report census.json -- ./run_report.sh
"""
import argparse
import json
import re
import shutil
import subprocess
import sys
import tempfile
from collections import Counter
from pathlib import Path

CANDIDATE_MASKS = [
    ("iso8601", re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?")),
    ("uuid", re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}")),
    ("hex", re.compile(r"\b[0-9a-f]{16,}\b")),
    ("int", re.compile(r"\b\d+\b")),
]


def run_once(cmd, workdir, timeout):
    with tempfile.TemporaryDirectory() as tmp:
        sandbox = Path(tmp) / "repo"
        shutil.copytree(workdir, sandbox)
        proc = subprocess.run(cmd, cwd=sandbox, capture_output=True,
                              text=True, timeout=timeout)
        return {"code": proc.returncode, "out": proc.stdout, "err": proc.stderr}


def distinct(texts):
    return len(set(texts))


def fit_masks(outputs):
    chosen, masked, best = [], list(outputs), distinct(outputs)
    improved = True
    while improved:
        improved = False
        for name, pattern in CANDIDATE_MASKS:
            if name in chosen:
                continue
            trial = [pattern.sub("<%s>" % name, t) for t in masked]
            if distinct(trial) < best:
                chosen.append(name)
                masked, best, improved = trial, distinct(trial), True
                break
    return chosen, masked, best


def main(argv=None):
    parser = argparse.ArgumentParser()
    parser.add_argument("--runs", type=int, default=10)
    parser.add_argument("--workdir", default=".")
    parser.add_argument("--timeout", type=int, default=120)
    parser.add_argument("--report", default="census.json")
    parser.add_argument("cmd", nargs=argparse.REMAINDER)
    args = parser.parse_args(argv)

    cmd = args.cmd[1:] if args.cmd and args.cmd[0] == "--" else args.cmd
    if not cmd:
        parser.error("need a command after --")

    workdir = Path(args.workdir).resolve()
    runs = [run_once(cmd, workdir, args.timeout) for _ in range(args.runs)]
    outs = [r["out"] for r in runs]
    codes = Counter(r["code"] for r in runs)

    before = distinct(outs)
    masks, masked, after = fit_masks(outs)

    report = {
        "command": cmd,
        "runs": args.runs,
        "exit_codes": dict(codes),
        "distinct_stdout_before_masks": before,
        "distinct_stdout_after_masks": after,
        "masks_fitted": masks,
        "masked_sample": masked[0][:2000] if masked else "",
    }
    Path(args.report).write_text(json.dumps(report, indent=2))
    print(json.dumps({k: v for k, v in report.items() if k != "masked_sample"}, indent=2))
    if after != 1:
        print("NOT STABLE: %d distinct masked outputs. Find the hidden state." % after,
              file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

To see the shape of the report, use a deliberately noisy demo entrypoint. This is a demo script, not a benchmark of anything.

#!/usr/bin/env bash
set -euo pipefail
printf 'generated=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
printf 'run_id=%s\n' "$(cat /proc/sys/kernel/random/uuid)"
printf 'rows=3\n'
printf 'total=41.50\n'

The report shape from that demo looks like this. Run it yourself before trusting the numbers.

{
  "command": ["./run_report.sh"],
  "runs": 10,
  "exit_codes": {"0": 10},
  "distinct_stdout_before_masks": 10,
  "distinct_stdout_after_masks": 1,
  "masks_fitted": ["iso8601", "uuid"]
}

Ten runs, ten outputs, two masks, one stable baseline. That is the state you want before writing a single assertion.

Step 2: Read the result with a decision table

Do not interpret "stable" loosely. Use three outcomes.

A non-unique exit code set is the one people skip. A command that returns 0 eight times and 1 twice is not a stable baseline either.

Step 3: Treat every mask as a written claim

The mask list is code. It asserts that a field carries no contract worth testing.

That claim needs an owner and a reason. Keep the masks in a reviewed file, not inline in a test.

# masks.py - every entry drops a field from the characterization baseline.
MASKS = [
    ("iso8601", r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?"),
    # reason: wall-clock generation time, no consumer reads it
    ("uuid", r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"),
    # reason: per-run id, asserted separately for format only
]

A mask that swallows a business field is a bug painted over. Review int masks hardest, because they hide totals, counts, and money.

Step 4: Freeze one golden file, then write the test

With a stable masked output, the test is boring and that is the point.

def test_report_matches_frozen_baseline(tmp_path, run_entrypoint):
    output = run_entrypoint(tmp_path)
    assert mask_text(output) == FROZEN_REPORT

One test per artifact. If the golden file is huge, split it by section, not by regex.

Step 5: Make the smallest safe change, then re-run the census

This is where the census pays off twice. After the extraction or rename, census again and compare the fitted mask set.

python nondeterminism_census.py --runs 10 --workdir ./sandbox \
    --report after.json -- ./run_report.sh

If masks_fitted grew from ["iso8601", "uuid"] to include int, your change introduced numeric drift. That is a regression signal the tests cannot give you, because the tests are already masked.

A change is small and safe when three things hold: the mask set did not grow, the golden diff is explainable line by line, and the exit code set stayed identical.

Step 6: Where free model access and a free server fit

Two parts of this loop are tedious rather than hard: drafting mask candidates, and running enough repetitions to trust the count.

MonkeyCode, an open-source coding project, exposes free model access and a free server option. Its operator also describes a free token allowance (reported to me as 10M); confirm current figures on their site before planning around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Here is the part I would actually use a model for, and the part I would not.

1. Drafting normalizers from diff pairs. Feed the census the ten distinct outputs and ask for a mask list with one justification per entry. Then delete half of it. Models over-mask.
2. Running the repetition loop off your laptop. A ten-run census on a slow entrypoint blocks your machine. A free server is a reasonable place to run the loop and publish census.json so a teammate can reproduce it.
3. Not this: letting a model decide that a drifting field is "noise." If the field is under investigation, masking it destroys the evidence.

Guardrails, in order of importance. Never send proprietary output or secrets to a hosted model. Keep the mask file in version control so every dropped field is reviewable. Re-run the census after applying any model-drafted mask; acceptance is a measured distinct_after_masks == 1, not a convincing explanation.

Limitations, stated plainly

• Mask fitting here is greedy and order-dependent. It can mask more than necessary, and int is dangerous.
• Ten runs is weak evidence. A field that drifts once in fifty runs will pass this census.
• copytree isolates files, not environment. Env vars, locale, network, and clock still leak.
• Masked golden files cannot catch regressions in masked fields. Accept that trade or quarantine the artifact instead.
• A shared free tier may queue, rate-limit, or change terms. Do not put it on a release-critical path.

Who should not use this

If the nondeterminism is the bug itself, such as corruption under concurrency, masking hides your only evidence. Stop and debug instead. If the output is contractual bytes, like signed invoices, do not mask anything. And if one run of the entrypoint takes an hour, a ten-run census is not affordable; sample fewer runs and lean on logs.

For everything else, the order is fixed. Ten runs, one stable masked baseline, one test, then the smallest change you can review. If you want to try the mask-drafting step without standing up a local model, MonkeyCode's free model access and free server option are one way in.