The Runbook Replay Gate: A Fail-Closed Handoff Checklist You Can Run in 45 Minutes

A service is not production-ready because its tests pass. It is production-ready when someone who has never opened the repo can restore it on an empty machine using only what is committed.

That is the gate. The rest of this article makes that single sentence executable: a pre-flight script, a 45-minute blind drill, and a decision table with fail-closed criteria you can copy into your own pipeline.

The failure this gate catches

Most readiness reviews check the happy path. They confirm the deploy works, the tests are green, the health endpoint returns 200. None of that survives contact with the one scenario that actually costs you a night:

• The person who built the service is unreachable, and the only description of how to restore it lives in their head.
• The recovery steps live in a chat thread, a DM, or an assistant session that nobody exported.
• The rollback command has never been executed, so its first run happens during an incident.

A replay gate converts all three into a binary result you can argue about in a pull request instead of at 3 a.m.

The seven gates

Every row has evidence attached. No evidence, no pass — that is what "fail-closed" means here.

Gate 2 is the one teams argue about most, so state the rule plainly: if a step requires a human to interpret intent, it is a note, not a step. Notes are allowed. They just do not count toward a pass.

Gate 4 deserves a second rule: the timestamp must come from a command that failed loudly at least once. A rollback nobody has ever broken is indistinguishable from a rollback that does nothing.

Artifact 1: the replay pre-flight script

This is a proposal, not a measured result — run it against your repo and expect the first run to fail. That is the point. Save it as scripts/replay_check.sh and call it from CI on any change under docs/, deploy/, or your config directory.

#!/usr/bin/env bash
# replay_check.sh — fail-closed pre-flight for a handoff replay drill
set -uo pipefail

FAIL=0
pass() { printf 'PASS  %s\n' "$1"; }
fail() { printf 'FAIL  %s\n' "$1"; FAIL=1; }

need_file() {
  if [ -s "$1" ]; then pass "file: $1"; else fail "file: $1 (missing or empty)"; fi
}

need_file .env.example
need_file docs/runbook.md
need_file docs/rollback.md
need_file docs/owners.md

# 1. No hardcoded model or API endpoints in source; they belong in config.
if grep -RIn --exclude-dir=.git --exclude-dir=node_modules \
     -E 'https://[^ "'\'']*(api|model|inference)' \
     --include='*.py' --include='*.ts' --include='*.go' . >/dev/null 2>&1; then
  fail "hardcoded endpoint in source — move it to config"
else
  pass "no hardcoded endpoints in source"
fi

# 2. The runbook must not outsource steps to memory or a chat window.
if grep -qinE 'ask the (author|builder)|ping |see (slack|thread|dm)' docs/runbook.md; then
  fail "runbook contains non-reproducible instructions"
else
  pass "runbook steps are self-contained"
fi

# 3. Rollback must carry a date and a paste-able result.
if grep -qE '^[0-9]{4}-[0-9]{2}-[0-9]{2}' docs/rollback.md; then
  pass "rollback log is dated"
else
  fail "rollback log has no dated execution"
fi

exit "$FAIL"

Two properties matter more than the checks themselves. It exits non-zero, so it can block a merge. And it fails on absence of evidence, not on presence of bugs — which is the difference between a checklist and a test suite.

Artifact 2: the 45-minute blind replay

The script checks that the paperwork exists. Only a human can check that the paperwork works. Run this drill with three roles and a hard clock.

1. Pick a runner who has never deployed this service. Not a rotation swap — someone genuinely unfamiliar.
2. Give them three things: the repo URL, an empty server, and the requirement to reach a working instance.
3. Silence the builder. Every question the runner asks is a defect in the docs, not in the runner. Log the first question verbatim.
4. Record the clock. When did they reach a running instance? When did they first need help?
5. Paste the log into docs/replay/YYYY-MM-DD.md, including the questions and the commands that failed.
6. Fix the repo, not the runner. Each question becomes a line in the runbook before the next drill.

Write the log before you discuss the drill. Discussions converge on goodwill; logs do not.

What counts as a pass

• Pass: the runner reached a working instance without a single blocking question, and every command in the path is committed.
• Conditional pass: they needed help, but the missing step was added to the repo during the drill and re-run successfully.
• Fail: help was needed and the fix lives anywhere other than the repo.
• Not a pass: the builder narrating over a screen share. That is a demo, and demos are how runbooks rot.

AI-assisted runbooks are the new failure mode

Assistants are good at drafting restore steps from repo evidence: read the config, read the migrations, propose an order. They are also good at producing steps that only work inside the session that generated them.

The rule that keeps this honest is boring. Every generated command gets executed once, and its real output is pasted under the command. A step reading "regenerate the config with the assistant" is not a step, because the next person has no assistant state, no context, and no reason to trust it.

If a step depends on a model call at runtime, gate 5 applies: name the fallback and the timeout in the runbook, next to the command.

Where a disposable box and free model access fit

The drill needs a machine you are willing to delete afterwards — a box that is not the staging server, not production, and not something you will be tempted to keep. That is exactly the shape of a throwaway environment, and it is the least glamorous part of the whole process.

MonkeyCode's free model access and free server option are useful here in two narrow ways: the server gives you somewhere to run the blind replay without touching shared infrastructure, and the free model access can draft the first version of docs/runbook.md from your repo, which you then execute line by line before committing. Neither replaces the drill. The drill is a person, a clock, and a log.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat every availability and allowance claim — from any vendor, including this one — as a snapshot with a date on it. Check the current terms yourself before you build a dependency on them, and put that check on gate 7's freshness schedule.

Limitations, and who should skip this

• The script verifies presence, not correctness. A runbook full of wrong commands passes the pre-flight and fails the drill — which is why you need both.
• A blind replay costs one engineer half a day. If you ship several services a week, batch them into a single monthly slot instead of gating every merge.
• This gate is wrong for a service that cannot be rebuilt from the repo at all, such as a stateful box with hand-tuned data. Those need a snapshot and restore test, not a runbook review.
• Skip it if your team is one person and the service has no users yet. Write the runbook anyway; just do not pretend the drill happened.

The takeaway

Write down the gate, run the script, book the 45 minutes, and keep the log. If the free server option saves you from rehearsing on staging, use it for that and nothing more.

Then ask the only question that matters: who is the next person to restore this, and can they do it without you?

If the answer needs a conversation, the gate is closed.