Your Agent Patch's Green Didn't Travel: A Portability Ledger for Two-Machine Test Evidence

A test result is a claim about a machine, not a claim about a patch.

When an agent hands you a diff plus a green suite, the suite ran somewhere. That somewhere had a Python version, a locale, a timezone, a case-sensitive filesystem, and a set of installed package versions. Change any of those and a subset of your assertions changes meaning. The patch didn't get better or worse — the evidence moved.

This article is about fixing that with a small, reproducible artifact: a portability ledger. It answers one question per test — does this green survive a second machine? — and refuses to let environment drift hide behind a passing CI badge.

It is deliberately not about flaky-test freezing policy, diff coverage, or mutation strength. Those are separate ledgers. This one only measures whether your evidence transfers.

Why a second run on the same box is not a control

Running the suite twice locally is still useful, but it measures a different thing. Two local runs separate nondeterminism from environment difference. If run 1 and run 2 disagree, you have a flake and no evidence either way. If they agree and the second machine disagrees, you have a portability gap — a much more interesting failure, because it is usually systematic and fixable.

So the ledger needs three columns, not two.

The bottom row is the one people skip. A test that only passes on the borrowed machine is not a weaker signal, it is an inverted one.

Step 1 — Capture an environment manifest

A manifest is the smallest set of facts that explain most portability gaps. Keep it boring and stdlib-only so it runs anywhere.

#!/usr/bin/env python3
# env_manifest.py — stdlib only; writes JSON to stdout
import json, os, platform, subprocess, sys
from importlib.metadata import version, PackageNotFoundError

WATCH = ("pytest", "hypothesis", "pydantic", "sqlalchemy")

def pkg(name):
    try:
        return version(name)
    except PackageNotFoundError:
        return None

def case_probe(tmp="."):
    p = os.path.join(tmp, ".probe_abc")
    open(p, "w").close()
    try:
        return os.path.exists(os.path.join(tmp, ".PROBE_ABC"))
    finally:
        os.remove(p)

manifest = {
    "python": sys.version.split()[0],
    "impl": platform.python_implementation(),
    "platform": platform.platform(),
    "machine": platform.machine(),
    "cpus": os.cpu_count(),
    "locale": os.environ.get("LC_ALL") or os.environ.get("LANG"),
    "tz": os.environ.get("TZ") or subprocess.run(
        ["date", "+%Z"], capture_output=True, text=True
    ).stdout.strip(),
    "fs_case_insensitive": case_probe(),
    "packages": {n: pkg(n) for n in WATCH},
    "env_keys": sorted(os.environ),
}
print(json.dumps(manifest, indent=2, sort_keys=True))

Run it on both machines and keep the files next to the patch:

python env_manifest.py > ledger/local.env.json
# on the second machine
python env_manifest.py > ledger/runner.env.json

The env_keys field is intentional. It records which variables exist without recording secrets. A missing TZ on one side and a set TZ on the other explains more timestamp failures than any amount of test rewriting.

Step 2 — Diff the manifests and rank the drift

Flatten nested keys, then diff. Thirty seconds of output beats a day of guessing.

# parity_diff.py
import json, sys

def flat(d, prefix=""):
    out = {}
    for k, v in d.items():
        key = f"{prefix}{k}"
        if isinstance(v, dict):
            out.update(flat(v, key + "."))
        else:
            out[key] = v
    return out

a, b = (flat(json.load(open(p))) for p in sys.argv[1:3])
for key in sorted(set(a) | set(b)):
    if a.get(key) != b.get(key):
        print(f"{key}\n  local : {a.get(key)!r}\n  runner: {b.get(key)!r}")

Rank what comes out, highest suspicion first:

1. Package versions — a minor bump in a serializer or ORM changes error messages and default ordering.
2. Locale and timezone — the classic source of date, float-formatting, and sort-order failures.
3. Filesystem case sensitivity — invisible on Linux, fatal for import or fixture-name collisions on macOS.
4. CPU count — anything with a worker pool or a concurrency assertion.
5. Python minor version — dict ordering is stable, but re behaviour and error text are not.

If a test fails on the second machine and none of these five differ, you probably do not have a portability gap. You have a genuine bug that your local machine was hiding.

Step 3 — Emit the ledger as JSON Lines

One row per test, one file per patch. JSON Lines keeps it appendable and diffable.

{"test": "tests/test_orders.py::test_roundtrip", "local_r1": "pass", "local_r2": "pass", "runner": "fail", "env_sensitive": false}
{"test": "tests/test_clock.py::test_expiry", "local_r1": "pass", "local_r2": "pass", "runner": "fail", "env_sensitive": true, "quarantine_until": "2026-11-01"}

The env_sensitive flag is a declaration by the author, not an observation. It says: this test's meaning legitimately depends on time, locale, or hardware. Declaring it does not make the failure acceptable — it moves the test from "blocking" to "must be fixed or removed by a date".

Step 4 — Gate on the ledger, not on the badge

# gate.py — non-zero exit when evidence does not travel
import json, sys

rows = [json.loads(l) for l in open(sys.argv[1]) if l.strip()]
blocking, deferred = [], []

for r in rows:
    a, b, c = r["local_r1"], r["local_r2"], r["runner"]
    if a != b:
        blocking.append((r["test"], "flaky: local runs disagree"))
    elif a == "pass" and c != "pass":
        bucket = deferred if r.get("env_sensitive") else blocking
        bucket.append((r["test"], f"portability gap: local={a} runner={c}"))
    elif a != "pass" and c == "pass":
        blocking.append((r["test"], "runner-only pass: unexplained"))

for name, why in blocking:
    print(f"BLOCK  {name}: {why}")
for name, why in deferred:
    print(f"DEFER  {name}: {why}")

sys.exit(1 if blocking else 0)

Wire it into CI as a required check and store ledger/*.jsonl as a build artifact. The value compounds: after a few weeks you can grep which tests flip most often and delete or rewrite them with data in hand instead of vibes.

Step 5 — Split property checks by portability

Property-based checks are the strongest evidence an agent patch can carry, but only if the property itself is portability-neutral. Divide them explicitly.

Portable properties hold for any machine: round-trip encode/decode, idempotency of a pure function, invariants over a parser, ordering stability of a sorted output.

Environment-bound properties do not: anything asserting on wall-clock time, on str(float) formatting, on locale collation, on filesystem ordering, or on a thread count. Those belong behind a freeze fixture with a fixed input, not behind a random generator.

An illustrative split (not executed here, presented as a pattern):

# portable: no clock, no locale, no ordering assumptions
@given(payloads())
def test_invoice_roundtrip(inv):
    assert decode(encode(inv)) == inv

# environment-bound: pin the clock and the timezone explicitly
def test_invoice_expiry(monkeypatch):
    monkeypatch.setenv("TZ", "UTC")
    with freeze_time("2026-01-01T00:00:00Z"):
        assert is_expired(make_invoice(days=30)) is False

If a property needs a pinned clock or locale to hold, say so in the test name. Future readers will otherwise re-learn it the hard way.

Where a second machine and free model access fit

The ledger above needs two environments. Your laptop is one; something else is the other. For the second one, this account's tooling is MonkeyCode, which the operator describes as offering free model access (a token allowance — 10M tokens at the time of writing) plus a free server option. Both are vendor claims with a shelf life, so verify the current terms before you build a habit on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In the workflow above, MonkeyCode is not the subject — it fills two slots. The free model access is where I ask an agent to generate candidate property checks from a diff, which I then review and keep only if they are portability-neutral. The free server option is where the second manifest comes from: a different machine, a different locale, a different package set, for zero marginal cost.

That framing keeps the method honest. Any second machine works. A container works. A colleague's laptop works. The point is that the second environment must be different and recorded, not that it must be any particular product.

Limitations and who should skip this

The two-environment ledger is a lower bound, not a proof. Passing on both machines tells you a test is not obviously environment-bound; it does not tell you it is correct, and it says nothing about production traffic, data volume, or race conditions under load.

Specific cases where this is the wrong tool:

• Your second environment must mirror production hardware. A general free server is a data point, not a load-test rig. Do not infer capacity from it.
• Strict data-residency or reproducibility rules. If you cannot send code or fixtures off your infrastructure, run the second manifest on a VM you control.
• Suites that require real services. No manifest diff will explain a failure that comes from a missing database or a blocked network call. Fix the test double first.
• Teams that will treat env_sensitive as a permanent skip. Quarantine entries need a date and a name, or the ledger becomes a graveyard.

Start with the manifest and the three-column table. That alone will tell you whether the last agent patch you merged was validated by evidence, or merely by a machine that happened to agree with it. If you want to try the second-runner slot on a free server, the operator's current terms are the place to check — and record the manifest before you celebrate the green.