Agent Patches That Pass Alone and Fail in the Suite: Shuffle Before You Quarantine

State the rule first: when an agent patch fails only in the full suite, it earns triage, not a quarantine. A quarantine is an expiring ledger entry that hides a known-unknown. An order-dependent test is not a known-unknown. It is a defect with a seed number attached, and a seed number is a reproduction recipe.

Shuffle first. Freeze last. That ordering is the whole article; everything below is the mechanics.

Two failure signatures that look identical in CI

A red suite tells you almost nothing on its own. You need the solo/suite cross. Two shapes matter most:

1. Solo-green, suite-red. The test passes in isolation and fails in a full run. Shared state is the usual cause: a module-level cache, a database row, a fixed port, a temp directory keyed by a constant name.
2. Solo-red, suite-green. The test only passes because a sibling test ran first and left state behind. This is worse, because your CI is green for a reason nobody wrote down.

A third shape sits between them: the test is intermittent in both contexts, with no correlation to run order. That one may be genuinely nondeterministic (clock, network, scheduler), and it is the only shape that is ever eligible for a freeze.

Agent patches surface all three because they change the shape of the run. Add eleven tests in front of an existing one and a latent dependency that has been invisible for a year becomes a weekly red. The patch did not create the flake. It changed the arithmetic that was suppressing it.

Step 1: Fingerprint the churn between two runs

Before touching fixtures, get a machine-readable diff of what changed status. Most runners emit JUnit XML; both pytest and most CI collectors can produce it. This script is a proposed reference skeleton, not a validated tool for your reporter — check namespace handling against your own XML.

#!/usr/bin/env python3
# fingerprint.py <baseline.xml> <patched.xml>
# Reports status churn between two runs of the same suite.
import sys
import xml.etree.ElementTree as ET


def collect(path):
    root = ET.parse(path).getroot()
    status = {}
    for case in root.iter("testcase"):
        key = f"{case.get('classname', '')}::{case.get('name', '')}"
        state = "passed"
        for child in case:
            if child.tag in ("failure", "error"):
                state = child.tag
            elif child.tag == "skipped":
                state = "skipped"
        status[key] = state
    return status


def main(baseline_path, patched_path):
    baseline, patched = collect(baseline_path), collect(patched_path)
    keys = sorted(set(baseline) | set(patched))
    regressed = [k for k in keys
                 if baseline.get(k) == "passed" and patched.get(k) in ("failure", "error")]
    fixed = [k for k in keys
             if baseline.get(k) in ("failure", "error") and patched.get(k) == "passed"]
    added = [k for k in keys if k not in baseline]
    removed = [k for k in keys if k not in patched]
    print(f"regressed={len(regressed)} fixed={len(fixed)} added={len(added)} removed={len(removed)}")
    for label, group in (("REGRESSED", regressed), ("FIXED", fixed),
                         ("ADDED", added), ("REMOVED", removed)):
        for key in group:
            print(f"{label}\t{key}")


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

The REMOVED and ADDED lines are not decoration. An agent that deletes a failing test and adds a passing one produces the same fixed count as a real fix. You want that visible in one line of output before anyone reads a diff.

Step 2: Replay under shuffled orders

Now replay the suite under many orders. Thirty runs of a two-minute suite is an hour of machine time, which is the part people refuse to pay — so run it once, on a machine you are not sitting in front of.

# pytest, with a per-run seed artifact you can replay later
mkdir -p .shuffle
for seed in $(seq 1 30); do
  pytest -q -p randomly --randomly-seed="$seed" \
    --junitxml=".shuffle/seed-$seed.xml" -p no:cacheprovider || true
done

# Go: -shuffle prints the seed it chose, so failures are replayable
for i in $(seq 1 10); do
  go test ./... -shuffle=on -count=1 -json > ".shuffle/go-$i.json" || true
done

If you do not use pytest-randomly, most modern runners expose some shuffle-or-seed flag; recent Jest majors ship --randomize and --seed, for example. Verify the flag exists in the version you actually run rather than trusting a blog post, including this one.

Then tally. A test that fails in two or more distinct seeds but passes in the fixed-order run is order-dependent, full stop. Reproduce one failing seed directly to confirm:

pytest -q -p randomly --randomly-seed=17 tests/test_ingest.py -x
go test ./internal/ingest -shuffle=1098234519 -run TestRetryBackoff -v

The counts in these examples are illustrative shapes, not measurements from a specific suite. Substitute your own artifacts.

Step 3: Classify, then decide

Classification is where the freeze decision gets made, and it should be mechanical enough to put in a table.

Two of those rows are the common outs. Teams reach for a freeze when the answer is "make the fixture function-scoped," and they reach for a retry when the answer is "the test asserts on wall-clock ordering."

Step 4: Repair the fixture, not the assertion

Once a test is classified as order-dependent, the fix is almost always in setup, not in the assertion. Five patterns cover most of it:

1. Narrow the fixture scope. A module- or session-scoped fixture that mutates anything is a shared mutable global with extra steps. Move it to function scope and measure the runtime cost.
2. Give every test its own state directory. Derive the path from the test ID or a per-test temp directory, never from a constant like /tmp/app-test.
3. Inject the clock. If the test asserts that event A precedes event B, inject a monotonic fake clock and advance it explicitly instead of sleeping.
4. Allocate resources dynamically. Bind port 0 and read the assigned port. Lock files and fixed ports are the classic source of solo-green/suite-red.
5. Wrap mutating tests in a rollback. A transaction per test is cheaper than a database per suite, and it makes leftover rows impossible by construction.

Step 5: Quarantine only what survives the shuffle

If a test is genuinely nondeterministic after fixture repair, quarantine it in a file, not in a decorator. A manifest forces an owner, a class, and an expiry date into the diff.

# test-quarantine.yaml
- test: "tests/test_ingest.py::test_retry_backoff"
  class: clock          # clock | network | race | unknown | order
  owner: "finley"
  quarantined_on: 2026-09-22
  expires: 2026-10-22
  evidence:
    shuffle_runs: 30
    failing_seeds: 0    # no seed correlation -> not order-dependent
    solo_replay: passes
    issue: "https://tracker.example/ISSUE-481"

Then gate the manifest in CI. Two rules do most of the work: expired entries fail the build, and class: order is rejected outright because an order-dependent test has a reproduction recipe and therefore is not eligible.

#!/usr/bin/env python3
# check_quarantine.py
import datetime
import sys

import yaml

TODAY = datetime.date.today()
ALLOWED = {"clock", "network", "race", "unknown"}
problems = []

for row in yaml.safe_load(open("test-quarantine.yaml", encoding="utf-8")):
    name = row["test"]
    if datetime.date.fromisoformat(row["expires"]) < TODAY:
        problems.append(f"expired: {name}")
    if row["class"] not in ALLOWED:
        problems.append(f"class '{row['class']}' must be repaired, not quarantined: {name}")
    if not row.get("evidence", {}).get("issue"):
        problems.append(f"no tracked issue for quarantine: {name}")

print("\n".join(problems) if problems else "quarantine ledger OK")
sys.exit(1 if problems else 0)

Step 6: Add properties that make order dependence structural

Shuffling finds order dependence. Properties make it harder to introduce. Three are worth writing for any stateful subsystem an agent is allowed to touch:

1. Permutation invariance. Applying the same logical work in two orders produces equivalent observable state.
2. Idempotence under repetition. Running the operation twice with the same key yields the same result as running it once.
3. Write containment. No operation writes outside its assigned state directory — a filesystem-level property you can assert with a temp root.

# Illustrative only; adapt names and normalization to your codebase.
def test_ingest_is_order_invariant(tmp_path):
    forward = ingest_all(["a", "b", "c"], state_dir=tmp_path / "forward")
    reverse = ingest_all(["c", "b", "a"], state_dir=tmp_path / "reverse")
    assert normalize(forward) == normalize(reverse)

A property like this fails loudly at the call site. A quarantine fails silently for a month.

Where free compute fits in this workflow

The shuffle matrix is embarrassingly parallel, embarrassingly boring, and CPU-bound. It is exactly the workload that does not deserve a laptop fan or a metered runner. MonkeyCode offers free model access and a free server option, and this workflow is one place both can be used honestly: run the multi-seed replay on the free server instead of your own machine, and use the free model access for the low-stakes text work around it — clustering thirty failure logs into a handful of distinct stack traces, or drafting test-quarantine.yaml rows from the XML your script already parsed.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the boundary strict. The model can group logs and draft a ledger row. It cannot decide order dependence, because that verdict comes from a seed replay that either reproduces or does not. Treat any model output as a proposal a human checks against the artifact. Free tiers and their limits also change, so verify the current terms yourself before wiring them into CI, and redact logs before sending them anywhere.

If you want to try the parallel shuffle matrix without provisioning a runner, the free server option is the natural starting point.

Limitations and who should not use this

• Suites that cannot be shuffled. If global state is created at import time and cannot be isolated, shuffling will produce a flood on the first run. Budget a cleanup sprint before adopting the gate.
• Suites under a few seconds. Thirty replays of a five-second suite is not worth automation. Run the shuffle once by hand when a flake appears.
• No machine-readable run artifacts. Without JUnit XML or JSON output, step 1 degrades to reading CI logs, which is where triage goes to die.
• Hard real-time or hardware-in-the-loop tests. Order and timing may be genuinely inseparable from the system under test; a freeze with a short expiry is the honest answer there.
• Anyone treating the quarantine manifest as a backlog. An expiring ledger only works if expiry actually fails the build.

Pre-freeze checklist

1. Solo and suite runs both captured, with a fingerprint.py churn diff attached to the PR.
2. At least twenty shuffled orders executed, with seeds stored as artifacts.
3. Two or more failing seeds replayed to confirm the class.
4. Fixture scope, state directories, clock, and resource allocation reviewed before any quarantine.
5. Ledger entry — with owner, class, expiry, and a tracked issue — only if the shuffle shows no seed correlation.

An agent patch is allowed to expose an old ordering dependency. It is not allowed to convert that dependency into a permanent skip. Shuffle, classify, repair, and freeze only the residue.