Make Resume Equivalence an Invariant Before a Free Server Restarts Your Agent Step

Here is the event order I use as a default counterexample whenever someone asks me to review an agent that runs on borrowed compute. Step 6 of 8 calls the model, receives a tool intent, applies a patch hunk, and flushes that hunk to the workspace disk. The host restarts before the runner records that step 6 finished, so the runner resumes from its last durable checkpoint at step 3. Steps 4 through 8 replay, step 6 lands twice, and the diff now contains the same hunk in two places. The test suite still passes, because a duplicated hunk is still a valid patch — and that is exactly why this bug survives code review.

The invariant the common implementation fails to preserve is boring to state and hard to hold: a resumed run must produce the same artifact as an uninterrupted run of the same event order. Not a similar artifact, not a passing build — the same fingerprint. If you cannot compute that fingerprint, you cannot tell a correct resume from a silent double-apply, and you will find out from a reviewer instead of a test.

1. Assumptions I declare before the review

I review free-tier agent builds against these assumptions, and I write them down because half of the disagreements I have with teams come from undeclared ones:

1. The compute domain is volatile. Assume the box can vanish at any instruction boundary, not at a convenient step boundary.
2. The model endpoint is shared and non-deterministic. Assume re-calling a step returns different output, so replay must not depend on re-calling it.
3. Artifact writes are the only externally visible effect. Anything else — a pushed commit, a webhook, a ticket comment — is a second failure domain with its own commit boundary.
4. Resume is a normal path, not an exception path. If you have never run it in CI, assume it does not work.

That fourth assumption is the one people resist, and I understand why: it feels like admitting the infrastructure is unreliable. But the admission you are actually making is that your recovery logic is untested, which is a different and much worse confession.

2. The state model and the two durability domains

Almost every broken resume I have reviewed has the same shape: two pieces of state, stored in two places, assumed to move together. The artifact log records what the agent produced; the receipt set records what the runner believes it already applied. If those two live in different durability domains, a crash can advance one without the other, and the gap is either a duplicate or a loss.

        model call (non-deterministic, expensive, not replayable)
                 |
                 v
  [ step intent ] --> apply() --> artifact log  (domain A: durable?)
                 |                      ^
                 +--> receipt set  -----+   (domain B: durable?)
                              ^
                       cursor / checkpoint

The data flow matters more than the diagram, so let me state it as a rule: the artifact write and the receipt write must be one commit on one domain. If they are two commits, the crash window between them is the bug — and it is not a small window, because the receipt write is usually the network call and the artifact write is usually the local one.

3. The invariant: one fingerprint, four modes

I do not want to argue about whether resume is correct, so I compute a fingerprint and compare. Below is a 60-line model of the problem using only the standard library — no network, no framework, nothing to install. It is a model of the commit boundary, not a mock of anyone's server.

# resume_equivalence.py — stdlib only, deterministic, no network
import hashlib

STEPS = [{"id": f"s{i}", "hunk": f"+line-{i}"} for i in range(8)]
CHECKPOINT_EVERY = 3


class Box:
    """One instance of the compute domain, with two durability switches."""

    def __init__(self, log_durable, receipts_durable):
        self.log_durable = log_durable
        self.receipts_durable = receipts_durable
        self.log = []          # artifact content
        self.receipts = set()  # what the runner believes it applied
        self.cursor = 0        # how many steps the runner thinks are done

    def apply(self, step):
        if step["id"] in self.receipts:   # idempotency guard
            return
        self.log.append(step["hunk"])
        self.receipts.add(step["id"])
        self.cursor += 1

    def crash(self):
        b = Box(self.log_durable, self.receipts_durable)
        b.log = list(self.log) if self.log_durable else []
        b.receipts = set(self.receipts) if self.receipts_durable else set()
        b.cursor = (self.cursor if self.receipts_durable
                    else (self.cursor // CHECKPOINT_EVERY) * CHECKPOINT_EVERY)
        return b


def run(box, start=0, crash_after=None):
    for i, step in enumerate(STEPS[start:], start=start):
        box.apply(step)
        if crash_after is not None and i + 1 == crash_after:
            return box
    return box


def fingerprint(box):
    return hashlib.sha256("\n".join(box.log).encode()).hexdigest()[:10]


REF = fingerprint(run(Box(True, True)))
print(f"reference={REF}")

for log_d in (True, False):
    for rec_d in (True, False):
        for k in range(1, len(STEPS)):
            box = run(Box(log_d, rec_d), crash_after=k).crash()
            box = run(box, start=box.cursor)
            verdict = "SAME" if fingerprint(box) == REF else "DIVERGED"
            print(f"log_durable={log_d:<5} receipts_durable={rec_d:<5} "
                  f"crash_after={k} hunks={len(box.log)} {verdict}")

Run it and the output collapses into four behaviours, which is the whole argument in one table:

The fourth row is my favourite, because it inverts the usual instinct. Full volatility can be safer than partial durability — replaying everything from scratch is deterministic, while replaying from a stale checkpoint is not. That result only holds because the model confines side effects to the box; the moment a step pushes a commit or sends a webhook, row four becomes the worst row in the table.

4. Failure injection, beyond the restart

A single crash point is not a test plan. The properties I actually want to assert, in the order I would add them to CI:

1. Crash at every step boundary — for k in range(1, len(STEPS)) — and require fingerprint equality.
2. Crash between the artifact write and the receipt write — the honest version of this test needs the two writes to be separable, which is precisely the defect you are hunting.
3. Duplicate delivery of the same step intent — run apply() twice with the same id and require exactly one hunk.
4. Stale checkpoint — resume from a cursor older than the durable log and require either correct completion or a loud refusal, never a silent double-apply.
5. Model re-call — assert that resume never calls the model for a step whose receipt exists, because the second answer will differ.

Property five is where most agent builds quietly cheat: they re-plan on resume instead of replaying a recorded decision. If your recovery path re-asks the model what to do, you have not resumed — you have started a second, correlated run.

5. Tradeoffs I would accept, and ones I would not

I will happily pay full replay on a cheap box for a short pipeline, because the failure mode is obvious and the fix is local. I will not accept checkpointing as a correctness mechanism, because a checkpoint is a performance decision wearing a durability costume.

6. What I would change next

1. Move the artifact out of the compute domain entirely — object storage or a git remote — so the box holds nothing you would miss.
2. Make the receipt the recorded decision of the step, not a boolean, so resume replays a decision instead of re-deriving one.
3. Write the receipt in the same commit as the hunk, or refuse to write either.
4. Add the fingerprint property test to CI at the same tier as the unit tests, and fail the build on divergence.
5. Give the runner a loud refusal path: if the checkpoint and the log disagree, stop and report, never guess forward.

7. Where the free tier actually fits

I use this pattern when I run agent steps against a free model endpoint and the project's free server option, which is where I want the failure to happen first. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The two availability claims I am relying on are operator-supplied: free model access, and a free server option — the project page also advertises a free token allowance (10M at the time of writing), and I would check that number on the page rather than budget against it here, because allowances move. What makes the environment useful for this exercise is not the price; it is that a volatile box and a shared endpoint are exactly the constraints under which resume equivalence breaks, and free access lets me crash the thing forty times in an afternoon.

Limitations, plainly: a free tier is not a durability guarantee, and nothing here should be read as an SLA. My simulator models the commit boundary and treats apply() as atomic — real writes are not, so the durable/durable cell in the matrix still needs a genuine atomic primitive behind it. And if you have hard latency SLOs, regulated data, or a requirement that state live on the box, this whole approach is the wrong shape for you; run it on infrastructure you own and keep the fingerprint test anyway.

If you want to try the loop end to end, the free model access and free server option are the cheapest place I know to break the invariant on purpose — run the script, then run it against a real pipeline and see which row of the matrix you land in.

Acceptance rule and the question I would leave you with

Accept the design only if, for every crash point in the trace, fingerprint(resume(crash(run))) equals fingerprint(run()) — or the runner refuses loudly. Anything else is a coin flip you have decided not to look at.

So: which event order breaks your invariant — a restart between the hunk write and the receipt write, a duplicate step intent, or a re-plan that quietly starts a second run? And when it happens, should the system reject the attempt, replay the recorded decision, or compensate for the effect it already committed?