Pin the Envelope, Not the Model Name

A regression that arrives because someone else rerouted traffic is still a regression you shipped, and the only durable defense is a manifest that records the exact request you sent and the exact envelope you graded. Free and trial endpoints are the most volatile place to run evaluations, because the routing label in the model field usually survives while the weights behind it quietly change. What follows is the tripwire I use to keep that volatility from silently rewriting my scores: three digests per call, graders that replay offline, and a paired sign test that refuses to call a small flip a trend. The listings below are stdlib-only sketches rather than a released package, so treat the plumbing as replaceable and the decision rule as the part worth copying.

The model field is a routing label, not an identity

Every call should produce three independent digests, because each one answers a different question about who changed what. A request fingerprint proves the prompt you sent is byte-identical to the baseline, which rules out your own edit as the cause of a flip. An envelope digest covers the graded text plus the structural fields that determine whether the text is even comparable, such as the finish reason and which usage keys the server returned. A declared stamp records whatever identity the endpoint claims for itself, and its main value is showing how weak that claim is.

# tripwire.py - stdlib only, Python 3.10+
import hashlib, json, re, time, urllib.request

def canonical(obj):
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)

def fingerprint(obj, n=16):
    return hashlib.sha256(canonical(obj).encode()).hexdigest()[:n]

def envelope_digest(resp):
    choice = resp["choices"][0]
    usage = resp.get("usage") or {}
    shape = {"finish": choice.get("finish_reason"),
             "usage_keys": sorted(usage.keys())}
    return fingerprint({"shape": shape, "text": choice["message"]["content"]})

def call(base_url, key, model, messages, temperature=0, timeout=45):
    body = {"model": model, "temperature": temperature, "messages": messages}
    req = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions",
        data=canonical(body).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {key}"})
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            raw = json.loads(r.read())
    except Exception as exc:
        # a transport failure is data, not an exception to swallow
        return {"req": fingerprint(body), "transport": type(exc).__name__,
                "ms": int((time.perf_counter() - t0) * 1000)}
    return {"req": fingerprint(body), "env": envelope_digest(raw),
            "finish": raw["choices"][0]["finish_reason"],
            "stamp": raw.get("model"),
            "ms": int((time.perf_counter() - t0) * 1000)}

Write one JSON line per case per run, and never overwrite the file. When a case fails, the first comparison is req against the baseline: if the request digest moved, the change is yours and no statistical machinery is needed. Only when the request digest matches, the envelope digest differs, and the stamp stayed the same do you have the interesting case, which is a swap that did not announce itself.

Graders must be replayable without the network

A grader that needs another model call to produce a verdict adds a second volatile endpoint to every measurement, so keep the canary set graded by deterministic checkers only. Exact match, anchored regex, and JSON-subset checks cover far more of a prompt eval than people assume, especially when you write the expected output yourself instead of harvesting it from a previous run. Harvesting expectations from the last run is how a silent regression becomes the new gold standard.

def grade(case, text):
    kind, value = case["check"]["kind"], case["check"]["value"]
    if kind == "exact":
        return text.strip() == value
    if kind == "regex":
        return re.search(value, text) is not None
    if kind == "json_subset":
        try:
            got = json.loads(text)
        except json.JSONDecodeError:
            return False
        return all(got.get(k) == v for k, v in value.items())
    raise ValueError(kind)

Classification happens before grading, and it is the step most harnesses skip. A closed socket, a rate-limit body, or a finish_reason of length is a transport or budget outcome, not a content miss; I argued that point in a separate post and the rule has not changed. Recording those as INVALID instead of FAIL keeps them out of the sign test, where they would otherwise masquerade as a quality drop.

Decide with a paired sign test, not a vibe

With a dozen canary cases, the difference between two flips and five flips is the difference between flake and drift, and eyeballing it will mislead you. Compare the same cases across two windows, count only the discordant pairs, and test them against a fair coin. The rule below flags a regression only when at least three cases moved from pass to fail and the one-sided binomial probability is under five percent.

from math import comb

def one_sided_p(b, c):
    n = b + c
    if n == 0:
        return 1.0
    return sum(comb(n, k) for k in range(b, n + 1)) / 2 ** n

def verdict(pairs, min_flips=3, alpha=0.05):
    regressed = sum(1 for prev, now in pairs if prev and not now)
    recovered = sum(1 for prev, now in pairs if not prev and now)
    p = one_sided_p(regressed, recovered)
    flagged = regressed >= min_flips and p < alpha
    return {"regressed": regressed, "recovered": recovered,
            "p": round(p, 4), "flag": "REGRESSION" if flagged else "inconclusive"}

A single flip on a deterministic case is a signal, but it is not a verdict, which is why the minimum flip count exists. Run the canary twice before escalating, and keep both windows in the ledger so a later reader can reconstruct what you saw. If the same case flips on two consecutive runs with an unchanged request digest, escalate immediately regardless of the aggregate p-value.

Where the free tier actually helps

A canary is only useful if it runs on a schedule, and the cheapest way to keep a small runner alive is a hosted box that is not your laptop. One option worth evaluating here is MonkeyCode, an open source project that offers free model access and a free server option for small workloads, which makes it a reasonable host for a scheduled tripwire and a useful second endpoint to add to the manifest. Disclosure: This article was prepared as part of MonkeyCode's product outreach. At the time of writing the project lists an operator-stated token allowance on the order of ten million tokens, and quotas and terms are exactly the kind of detail that changes, so read the current documentation before you plan a budget around it.

The honest reason to use free model access in this workflow is not to save money on production traffic, but to widen the manifest. Running the same golden cases against two independently operated endpoints tells you whether a flip is your prompt or the platform, because two providers rarely drift in the same direction on the same day. If you want a starting point, the project's repository and its free tier are the places to look, and the tripwire is roughly eighty lines of standard library code that you can port anywhere.

Limits and who should not use this

A sign test over twelve cases has very little power, so it will miss slow drifts that move one case per week in the same direction; pair it with a weekly trend chart if that matters to you. Cases with a nonzero temperature are not deterministic, so either fix temperature at zero for the canary set or run enough repetitions to estimate per-case variance first. Free tiers can throttle you mid-window, which corrupts paired comparisons, so cap concurrency and discard windows with a high INVALID rate rather than averaging through them.

This approach is a poor fit if you need contractual guarantees, per-token cost accounting, or certification evidence for a compliance review, because a tripwire narrows the search space and does not replace a maintained evaluation suite. A saturated canary set is the other failure mode: when every case passes for weeks, the alarm has become a smoke detector with a dead battery. Rotate in new cases from real bug reports, and remove cases that no plausible change could break.