Green in One Order, Broken in Another: A Shuffle-and-Bisect Harness for Agent Patches

A suite that passes in your runner's default order is evidence about exactly one schedule. If an agent patch adds a module-level cache, a lazy singleton, or mutates a default argument, the defect surfaces only when some other test runs first. Coverage, mutation score, and fixture fingerprints will all stay green.

That is why order sensitivity deserves its own gate. It catches a class of regression that isolated reruns cannot reproduce and that pytest -k debugging actively hides: you re-run the victim alone, it passes, and the failure looks like noise.

The harness below is plain pytest plus one plugin file with no third-party dependencies. It runs the suite in seeded orders, builds a per-test pass matrix, then shrinks each failure to the smallest prefix that reproduces it. No invented statistics: whatever numbers you get depend on your suite and your patches, and that is the point.

One distinction to hold onto. A flake fails in any order. An order-dependent test fails in a specific order. Merging the two categories into one @flaky bucket is how shared state survives review for months.

Why the default order is the worst order for review

Pytest collects alphabetically by module and directory. Agent patches usually add a new module near the top of the tree or a helper imported by many others, which means the patch itself changes collection order. The suite then passes because the new import lands early and primes whatever global it touches, or fails because it lands late and the cache is already warm.

Neither result tells you whether the production change is correct. Both tell you something about initialization order that no test asserts.

The gate needs three properties: a recorded order, a recorded outcome per test, and a comparison against the pre-patch tree. Steps 1 to 3 build the first two.

Step 1: record outcomes without a report plugin

Drop this file at the repo root. Running python -m pytest from there puts the root on sys.path, so -p order_log resolves without installation.

# order_log.py — pytest plugin, no third-party dependencies.
import json
import os

_LOG = os.environ.get('ORDER_LOG')


def pytest_runtest_logreport(report):
    if not _LOG:
        return
    if report.when == 'call':
        outcome = report.outcome          # passed / failed / skipped
    elif report.failed:
        outcome = 'error'                 # setup or teardown blew up
    else:
        return
    with open(_LOG, 'a') as fh:
        fh.write(json.dumps({'id': report.nodeid, 'outcome': outcome}) + '\n')

Setup errors matter here more than in normal CI. A fixture that leaks state often fails in setup of a later test, and a call-only hook would record that test as never having run.

Step 2: sweep seeded orders and score sensitivity

The runner passes explicit node ids so the argument order is the execution order. That is portable and does not require a randomizing plugin.

# shuffle_sweep.py
import json, os, random, subprocess, sys, tempfile

PYTEST = [sys.executable, '-m', 'pytest', '-p', 'order_log', '-q', '--no-header']


def collect_tests():
    out = subprocess.run([sys.executable, '-m', 'pytest', '--collect-only', '-q'],
                         capture_output=True, text=True, check=True)
    return [line.strip() for line in out.stdout.splitlines() if '::' in line]


def run_order(order, seed, stop_on_fail=False):
    fd, log = tempfile.mkstemp(suffix='.jsonl')
    os.close(fd)
    env = dict(os.environ, ORDER_LOG=log, PYTHONHASHSEED=str(seed))
    cmd = PYTEST + (['-x'] if stop_on_fail else []) + order
    subprocess.run(cmd, env=env, capture_output=True, text=True)
    result = {}
    if os.path.exists(log):
        with open(log) as fh:
            for line in fh:
                row = json.loads(line)
                result[row['id']] = row['outcome']
    os.unlink(log)
    return result


def sweep(tests, runs):
    matrix = {t: [] for t in tests}
    for i in range(runs):
        order = tests[:]
        random.Random(i).shuffle(order)
        res = run_order(order, seed=i)
        for t in tests:
            matrix[t].append(res.get(t, 'not-run'))
    return matrix


def sensitivity(matrix, min_runs=3):
    scores = {}
    for test, row in matrix.items():
        seen = [r for r in row if r != 'not-run']
        if len(seen) < min_runs:
            continue
        bad = sum(1 for r in seen if r in ('failed', 'error'))
        if bad:
            scores[test] = round(bad / len(seen), 2)
    return scores

Keep -x off during the sweep. Early exit truncates the matrix and every later test becomes not-run, which is exactly the data you need to attribute the failure to a predecessor.

Five runs is a reasonable starting budget. The cost is linear in suite time, so a two-minute suite costs ten minutes per revision, and a forty-minute suite is out of reach without sharding.

Step 3: shrink the failure to a polluting prefix

A failing order of 900 tests is not a bug report. Shrink it. Binary search on the prefix, then verify, then delete one test at a time while the reproduction still holds.

def reproduces(prefix, victim, seed, repeats=3):
    hits = 0
    for k in range(repeats):
        res = run_order(prefix + [victim], seed=seed * 100 + k)
        hits += res.get(victim) in ('failed', 'error')
    return hits, repeats


def bisect_prefix(order, victim, seed):
    lo, hi = 0, len(order)      # invariant: order[:lo] does not reproduce
    while hi - lo > 1:
        mid = (lo + hi) // 2
        hits, n = reproduces(order[:mid], victim, seed)
        if hits == n:
            hi = mid
        else:
            lo = mid
    return order[:hi]


def shrink(prefix, victim, seed):
    cur = prefix[:]
    changed = True
    while changed:
        changed = False
        for i in range(len(cur)):
            cand = cur[:i] + cur[i + 1:]
            hits, n = reproduces(cand, victim, seed)
            if hits == n:
                cur, changed = cand, True
                break
    return cur

Be honest about the assumption: binary search assumes monotonicity, meaning that if a prefix of length k reproduces, every longer prefix does too. Shared state breaks that assumption. A test can prime a cache that a third test then invalidates. reproduces() therefore demands three of three hits, and the shrink pass after the bisect exists to repair a bad split. If the bisect returns a prefix that does not reproduce, fall back to a linear scan from the shortest failing length.

Step 4: assign ownership from the shrunk prefix

The prefix is the finding. Decide who fixes it before touching the test.

The third row is the one people miss. If the victim fails with an empty prefix, order shuffling did not find the bug; it found that the test was never hermetic. That belongs in a quarantine ledger with a named owner and an expiry date, not in a shuffle sweep.

Step 5: turn the polluter into a property check

Once the shrunk prefix names the culprit, write an in-process invariant that fails in milliseconds instead of after a full sweep. The prefix gives you the material for the assertion.

For a polluted-input bug, the invariant is non-mutation:

# Illustrative — adapt the call to the actual polluter.
import copy
from app.handlers import handle


def test_handle_does_not_mutate_inputs():
    payload = {'items': [1, 2, 3]}
    before = copy.deepcopy(payload)
    handle(payload)
    assert payload == before

For a module-level cache, the invariant is referential stability across repeated calls in one process:

from app import config


def test_config_reload_is_not_cached():
    first = config.load('a')
    second = config.load('a')
    assert first == second
    assert config.load('b') != first

Property checks like these run in the default order against every future patch. The sweep is a one-time diagnostic; the property check is the permanent regression net.

Step 6: gate on the delta, not the absolute score

Every real suite has some order sensitivity. Blocking on a nonzero score blocks all merges. Compare the patched tree against its parent and gate only on new offenders.

git worktree add ../parent HEAD~1
(cd ../parent && python shuffle_sweep.py --runs 5 --out ../parent.json)
python shuffle_sweep.py --runs 5 --out patched.json

import json

parent = json.load(open('parent.json'))
patched = json.load(open('patched.json'))
new = {t: s for t, s in patched.items() if t not in parent}
assert not new, f'new order sensitivity introduced: {new}'

Any entry in new is a merge blocker unless the same patch adds the offender to the quarantine ledger with an owner and an expiry. A freeze that never expires is a rename, not a fix.

Where a hosted run actually helps

Local sweeps are noisy. Environment variables from your shell, a warm ~/.cache, a globally installed pytest plugin, and leftover databases all become candidate polluters, and the shrink pass will happily report them as the patch's fault. Running the sweep in a clean container on MonkeyCode's free server option, whose availability is operator-supplied rather than verified here, removes that confounder: the same five seeds against a pristine image, with the parent worktree built the same way. The free model access is useful at one specific point too. Hand it the shrunk prefix and the victim's assertion, and ask for an invariant in the style of Step 5. The prompt is short, the output is checkable, and a wrong answer costs one failed test rather than a bad merge.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat both as conveniences. The harness is dependency-free and runs anywhere pytest runs, so nothing here stops working if a free tier changes shape. Verify current availability and limits yourself before you build a review process on top of it.

Limitations and who should skip this

• Suite time. The sweep multiplies wall-clock by the run count. If the suite takes longer than a few minutes, shard it first or restrict the sweep to the modules the patch touches.
• Intentional order. Integration suites that execute a workflow in sequence will light up as offenders. Allowlist them explicitly with a comment, not silently.
• Parallel runners. xdist distributes tests across workers and destroys the ordering the matrix depends on. Run the sweep serially.
• Unpinned dependencies. If shuffle results correlate with network access, the sweep is measuring your internet connection, not your patch.
• Skip it entirely if the suite runs in two seconds in the default order and has no module-level state. The gate pays for itself on patches that touch shared initialization, not on leaf functions.

What to do with the finding

Paste the shrunk prefix into the pull request. A three-line reproduction — two polluters plus the victim — explains more to a reviewer than any pass rate, and it converts an intermittent complaint into a fixable claim about initialization order.

Run the sweep once against an agent patch you are currently reviewing, and if it produces a new offender, the value is in the prefix, not in the score.