Map Every Working-Tree Delta to a Closed Tool Span

An agent run is finished when the working tree and the trace agree, not when the model emits a final sentence. git status --porcelain is the ledger. Closed tool spans are the journal entries. If a path changed and no span owns it, the run failed. If a span claims a write and the bytes are missing, the run failed. The summary paragraph is not evidence.

This gap is getting easier to miss. Public debate this month keeps returning to whether generated code is already “better than most developers,” and whether a green test suite is enough to call the session engineering. Those arguments skip a more primitive check. Tests observe outputs someone chose to assert. They do not observe every file the process touched. An agent can skip a tool, narrate the skip as success, and still pass a unit file that never opened the mutated path.

Think of a warehouse cycle count. The packing list the intern typed is the assistant message. The scanner at the dock is the trace. The pallets on the floor are git status. Nobody ships because the intern wrote “all packed.” You ship when every pallet id is on a scan record and every scan record points at a pallet that still exists. Agent tooling that treats the chat transcript as the dock scan will leak mutations into the next human commit.

The failure has two directions, and they are not symmetric. A ghost write is a dirty path with no closed span: a formatter, a shell redirect, a compiler, or a helper script that ran outside the traced adapter. A phantom write is a closed span whose after_hash is not on disk: the tool returned ok, the model thanked it, and a later step reverted the file, or the tool wrote a temp path and reported a source path. Either break should fail the run. A third, quieter break is a clean tree after a task that required a mutation. No-op success is still a false success if the prompt demanded an edit.

A reusable control is a bijection between path-level deltas and closed write spans. The harness below is a worked example, not a production trace backend. It records JSONL, hashes file bytes with SHA-256, and compares that set to git status --porcelain. It does not replace OpenTelemetry. It does claim that a run without this map is an anecdote.

Label the schema as an example. Each tool attempt appends one record. status must become closed before reconcile. op is write, delete, or read. Reads do not have to match porcelain. Writes and deletes do. path is repo-relative POSIX. before_hash and after_hash are hex or null.

# span_schema.py — example record shape, not a shipped SDK
import hashlib, json, time, uuid
from pathlib import Path

def sha256_file(path: Path):
    if not path.is_file():
        return None
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def open_span(trace_path, op, relpath, before_hash):
    rec = {
        "span_id": str(uuid.uuid4()),
        "ts_open": time.time_ns(),
        "op": op,
        "path": relpath,
        "before_hash": before_hash,
        "after_hash": None,
        "status": "open",
        "error": None,
    }
    with open(trace_path, "a", encoding="utf-8") as f:
        f.write(json.dumps(rec) + "\n")
    return rec["span_id"]

def close_span(trace_path, span_id, after_hash, error=None):
    rows = []
    with open(trace_path, encoding="utf-8") as f:
        for line in f:
            rec = json.loads(line)
            if rec["span_id"] == span_id:
                rec["ts_close"] = time.time_ns()
                rec["after_hash"] = after_hash
                rec["status"] = "error" if error else "closed"
                rec["error"] = error
            rows.append(rec)
    with open(trace_path, "w", encoding="utf-8") as f:
        for rec in rows:
            f.write(json.dumps(rec) + "\n")

Wrap every mutating tool. Do not wrap the model. The model can say anything. The adapter is the only process allowed to touch the tree. If the agent is given a raw shell, the shell itself must emit spans for every path it intends to change, or the later git diff will fail the run on purpose. That is the point. Untraced sed -i is not a clever escape. It is an open drawer.

# traced_write.py — example adapter
from pathlib import Path
from span_schema import sha256_file, open_span, close_span

def traced_write(repo: Path, trace_path: str, relpath: str, data: bytes):
    dest = (repo / relpath).resolve()
    if not str(dest).startswith(str(repo.resolve())):
        raise ValueError("path escapes repo")
    dest.parent.mkdir(parents=True, exist_ok=True)
    before = sha256_file(dest)
    span_id = open_span(trace_path, "write", relpath, before)
    try:
        dest.write_bytes(data)
        close_span(trace_path, span_id, sha256_file(dest))
    except Exception as exc:
        close_span(trace_path, span_id, sha256_file(dest), error=str(exc))
        raise

Collect the ledger from git, not from in-memory guesses. Porcelain is stable enough for a gate. Untracked files are mutations. Ignored files are a policy choice: if the agent is allowed to write node_modules, the bijection is already blind. Keep generated noise out of the repo, or accept that the map cannot see it.

# git_delta.py — example ledger
import subprocess
from pathlib import Path

def porcelain_paths(repo: Path) -> dict:
    out = subprocess.check_output(
        ["git", "status", "--porcelain", "-uall"],
        cwd=repo, text=True,
    )
    deltas = {}
    for line in out.splitlines():
        if not line:
            continue
        code, rest = line[:2], line[3:]
        if " -> " in rest:
            rest = rest.split(" -> ", 1)[1]
        deltas[rest.strip()] = code.strip()
    return deltas

Reconcile is a set comparison plus a hash check. Open spans fail first. Then every write or delete span must match a porcelain path. Then every porcelain path must match a span. Then after_hash must equal the bytes on disk for writes, and the path must be absent for deletes. Print the leftover sets. Do not summarize them in prose and call the run “mostly fine.”

# reconcile.py — example gate
import json, sys
from pathlib import Path
from span_schema import sha256_file
from git_delta import porcelain_paths

WRITE_OPS = {"write", "delete"}

def load_spans(trace_path):
    with open(trace_path, encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]

def reconcile(repo: Path, trace_path: str, require_mutation: bool) -> int:
    spans = load_spans(trace_path)
    open_ids = [s["span_id"] for s in spans if s.get("status") == "open"]
    if open_ids:
        print("FAIL open spans:", open_ids)
        return 2

    mutating = [
        s for s in spans
        if s.get("op") in WRITE_OPS and s.get("status") == "closed"
    ]
    disk = porcelain_paths(repo)
    span_paths = {s["path"] for s in mutating}
    ghost = sorted(set(disk) - span_paths)
    phantom = sorted(span_paths - set(disk))

    hash_miss = []
    for s in mutating:
        p = repo / s["path"]
        if s["op"] == "write" and sha256_file(p) != s.get("after_hash"):
            hash_miss.append(s["path"])
        elif s["op"] == "delete" and p.exists():
            hash_miss.append(s["path"])

    if require_mutation and not disk and not mutating:
        print("FAIL no-op tree on a mutating task")
        return 3
    if ghost or phantom or hash_miss:
        print("FAIL ghost:", ghost)
        print("FAIL phantom:", phantom)
        print("FAIL hash:", hash_miss)
        return 1
    print("PASS span-tree bijection")
    return 0

if __name__ == "__main__":
    repo = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
    trace = sys.argv[2] if len(sys.argv) > 2 else "run.jsonl"
    require = "--require-mutation" in sys.argv
    sys.exit(reconcile(repo, trace, require))

The CI step should stay boring. That is the desired property. After the agent process exits, the job fails on any leftover path. Do not scan the final assistant message for “done” or “fixed.” Those strings are not a working tree.

python reconcile.py . run.jsonl --require-mutation

The following cases are unexecuted examples, not measured production results. They pin the three failure classes the gate is meant to catch.

# test_reconcile_cases.py — unexecuted examples
def test_ghost_write_fails(tmp_repo):
    # an untraced formatter touches extra.py; porcelain has it, spans do not
    assert reconcile(tmp_repo, "run.jsonl", False) == 1

def test_phantom_write_fails(tmp_repo):
    # a closed span claims src/app.py; git is clean
    assert reconcile(tmp_repo, "run.jsonl", False) == 1

def test_noop_on_required_mutation_fails(tmp_repo):
    assert reconcile(tmp_repo, "run.jsonl", True) == 3

Wire the agent so it can only call traced_write, a traced delete, and a traced shell that predeclares output paths. Isolation matters more than the hash function. A laptop already mid-refactor will fail the gate for human reasons: your own dirty files become ghosts. Run the agent in a clean clone. Keep the developer tree out of the ledger.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is a practical place to run that clone-plus-harness loop when the point is to keep the agent's tree off the developer's working copy. The scripts do not depend on it. They run against any model endpoint and any VM. The free options only remove the excuse that isolation needs a dedicated budget before the gate exists.

Limitations are sharp. The harness does not see mutations that never hit git, including ignored paths, Docker volume writes, and database rows. Line-ending conversion and formatters that rerun after the last span will produce hash misses even when the semantic edit is correct; trace the formatter or disable it during the run. Concurrent agents on one working tree break the bijection because porcelain cannot say which span owned a path. Case-insensitive filesystems alias paths the JSONL treats as distinct. Binary assets hash correctly but make traces noisy; keep them out of agent workspaces or accept large JSONL. This control also does not prove the change was right. It proves the change was accounted for. A well-traced, well-hashed, wrong patch still merges if tests are weak.

Do not use this approach for exploratory sessions where the human is supposed to edit the same tree. The gate will fight the human. Do not use it as a secrecy boundary. Hashes of file bytes are not redaction; secrets written into a traced file are still secrets in whatever store keeps the JSONL. Do not use it to rank models. A cheaper model that emits fewer ghost writes is not “better at coding.” It is better at staying inside the adapter. Those are different claims, and mixing them is how generated output gets called engineering.

The durable rule is small. Close every mutating span. Reconcile against porcelain. Fail on leftovers in either set. Until that map is closed, the agent did not finish, no matter how confident the last sentence reads.