Replay One Span, Not the Whole Run

A failed agent run is usually cheaper to diagnose as a single span than as a whole re-run. Pull the span out of the trace, replay it twice, compare fingerprints. That one habit collapses 'it broke again' into a small, decidable answer: the world drifted, the tool is nondeterministic, or the model chose badly while the tool behaved perfectly.

Most trace writing stops at reading spans. Reading is the easy half. Replay is the half that produces evidence you can paste into a bug report and re-run in CI a week later.

Whole-run replay lies to you

A full re-run changes too many variables at once. The working tree moved, the cache warmed, the feature flag flipped, the upstream API returned a slightly different shape. You get a new trace with a new failure and no way to attribute it to any one of those changes.

So stop re-running the run. Re-run the span, where the state is small enough to enumerate.

One span, four fields

The harness below assumes a JSONL trace where each line is a span carrying an id, the tool invocation, the tool result, and the model decision that requested it. If your schema differs, change the accessors, not the idea.

{'span_id': '7f3c', 'tool_call': {'argv': ['git', 'diff', '--stat'], 'env_digest': 'sha256:9a1e', 'timeout_s': 30}, 'tool_result': {'exit_code': 0, 'stdout': ' 3 files changed, 41 insertions(+)', 'stderr': ''}, 'model_decision': {'tool_name': 'git_diff_stat', 'args': {}, 'model_id': '<redacted>'}}

Keep env_digest in the record. Without it, drift analysis is guesswork, because a different PATH or a different tool version reproduces the same argv with different output.

Normalize, then fingerprint

The useful comparison is never raw stdout. Raw stdout carries UUIDs, temp paths, durations, and timestamps that change every run for no diagnostic reason. Normalize first, hash second, and keep the denylist short enough that a reviewer can read it in one screen.

# normalize.py
import hashlib, json, re

VOLATILE = ('ts', 'timestamp', 'started_at', 'ended_at', 'duration_ms', 'trace_id', 'span_id', 'request_id')
UUID = re.compile(r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', re.I)
TMP = re.compile(r'/tmp/[A-Za-z0-9_.-]+')

def normalize(obj, drop=VOLATILE):
    if isinstance(obj, dict):
        return {k: normalize(v, drop) for k, v in sorted(obj.items()) if k not in drop}
    if isinstance(obj, list):
        return [normalize(v, drop) for v in obj]
    if isinstance(obj, str):
        return TMP.sub('<tmp>', UUID.sub('<uuid>', obj))
    return obj

def fingerprint(obj):
    blob = json.dumps(normalize(obj), sort_keys=True, separators=(',', ':'))
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

Sixteen hex characters is enough to spot a mismatch and short enough to paste into a commit message. Widen it if you need collision safety across a large corpus; nobody needs that in a debug line.

Two questions, two invocations

Replay has two modes because there are two distinct questions, and mixing them is how people end up blaming the model for a flaky filesystem. Determinism asks whether the tool behaves the same twice. Drift asks whether the recorded result still matches today's world.

# replay.py
import json, subprocess, sys
from normalize import fingerprint

def load(path):
    with open(path) as f:
        return json.load(f)

def run_live(span):
    call = span['tool_call']
    p = subprocess.run(call['argv'], capture_output=True, text=True, timeout=call.get('timeout_s', 30))
    return {'exit_code': p.returncode, 'stdout': p.stdout, 'stderr': p.stderr}

def main(path, mode):
    span = load(path)
    recorded = fingerprint(span['tool_result'])
    if mode == 'determinism':
        first = fingerprint(run_live(span))
        second = fingerprint(run_live(span))
        print(f'recorded={recorded} run1={first} run2={second} stable={first == second}')
    elif mode == 'drift':
        fresh = fingerprint(run_live(span))
        print(f'recorded={recorded} live={fresh} drift={recorded != fresh}')
    else:
        sys.exit(f'unknown mode {mode}')

if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2])

Usage is a short extraction followed by two commands against the extracted file.

python - <<'PY'
import json
spans = [json.loads(l) for l in open('trace.jsonl') if l.strip()]
json.dump(next(s for s in spans if s['span_id'] == '7f3c'), open('span.json', 'w'), indent=2)
PY
python replay.py span.json determinism
python replay.py span.json drift

Run determinism first. Replaying a nondeterministic tool against a recording tells you nothing except that clocks exist.

Reading the two booleans

Each mode answers yes or no, and the pair is more informative than either alone. The top-left cell is the one people skip, and it is the most common real answer in my experience of these loops.

When the tool is stable and the world has not moved, the failure lives in the decision layer, and no amount of tool debugging will find it. That is where a second model earns its keep.

Replay the decision with a second model

Take the recorded prompt and tool schema for that span, ask a different model for the tool call, and compare the emitted call's fingerprint to the recorded one. A different call means the decision is model-dependent. An identical call means the decision is robust and the bug sits downstream. Keep the schema byte-identical, or you are testing the schema.

This is a one-call, cheap check, which makes it a reasonable fit for a free model tier rather than a paid production lane. MonkeyCode is where I run that second-opinion step: its free model access covers the occasional decision replay, and its free server option is enough to host the replay worker itself. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The worker shape matters more than the vendor. Replay is bursty, I/O-bound, and needs no SLA: it runs when a trace fails and idles the rest of the week. Putting that worker on the same host as the production runner is how a debugging tool takes down a service. Keep the lanes separate and let the replay lane be disposable. If you point a client at a model endpoint, read the base URL and model ids from current docs rather than from a blog post, and keep credentials in an environment variable.

Limitations, stated plainly

Span replay cannot see side effects that already happened. If the tool wrote to a database or opened a pull request, replaying it live will write again; stub those tools and compare the stub's arguments instead of the result. Redact before you store, because a trace file is a data-retention decision you make once and regret for years.

Free tiers are not capacity plans. A free token allowance and a free server option are fine for replaying tens or hundreds of spans, and they are the wrong foundation for an audit archive or a latency-sensitive loop. Treat quotas as operator-supplied and verify current terms yourself; they change without notice.

Skip this approach if your artifacts cannot leave your environment, if your tools cannot be stubbed, or if your failures are dominated by concurrent state rather than single-span behaviour. Single-span replay is a narrow instrument, and it is sharp only in that narrowness.

For everyone else the loop is two commands wide. A trace fails, you extract the span, you run determinism, you run drift, and you have a defensible answer before you touch a prompt. If you want to try the second-opinion step without provisioning anything, the free token allowance and the free server option are enough to start; confirm the current terms first.