Pin the First Command, Not the Paragraph: A Testable 03:00 Diagnostic Path

Most on-call runbooks fail at exactly one moment: the ninety seconds between the page and the first command you trust. A document that explains the architecture can be excellent and still leave you scrolling, and the person who wrote it is asleep three time zones away. So here is the conclusion up front: pin one command per alert, render it from the alert payload, and let a test suite fail when it stops working. Everything below is the smallest version of that idea I would actually ship.

Runbooks describe; incidents need a command

I have read plenty of runbooks that open with a paragraph about what the service does. That paragraph is valuable during a design review, and it is nearly useless when the pager fires at 03:00 with a payload containing seven labels. What you need instead is a deterministic, copy-pasteable line that scopes itself correctly and prints evidence you can drop into an incident channel.

The industry conversation has been drifting this way for a while, with a lot of recent writing arguing for executable runbook steps over prose. I agree with the direction, but I want something narrower and stricter than put bash in a wiki. A command is only dependable if the alert supplies the variables, and if a test proves the substitution still works after somebody renames a label.

So the artifact I want has two halves. The first is a small renderer that turns alert labels into the exact command: it makes no network calls and never invents a flag. The second half is a suite that fails CI when an alert has no pinned command, when a fingerprint fixture stops rendering, or when a cluster-scoped check silently inherits the caller's kube-context.

Define the payload fields you are willing to depend on

Before writing any code, write down which labels you will render from, because that list doubles as your contract. My rule is: only labels that exist on every alert routed to this runbook, plus annotations that are strictly informational and never parsed into a command.

{
  "alert_name": "CheckoutLatencyHigh",
  "labels": {"service": "checkout", "namespace": "prod-payments", "cluster": "eu-1"},
  "annotations": {"runbook_url": "https://runbooks.internal/checkout"},
  "startsAt": "2026-11-09T03:12:44Z"
}

Then the runbook itself, with a command template per alert. Keep the template free of shell plumbing, so the rendered output is reviewable in a diff and short enough to read on a phone.

# runbook/alerts.yml
alerts:
  - name: CheckoutLatencyHigh
    service: checkout
    owner: payments-oncall
    severity: page
    escalation: payments-secondary
    scope: cluster
    expect_contains: [Ready, restarts]
    command: >-
      kubectl --context {labels[cluster]} -n {labels[namespace]}
      rollout status deploy/checkout --timeout=30s
  - name: CheckoutErrorBudgetBurn
    service: checkout
    owner: payments-oncall
    severity: ticket
    escalation: payments-secondary
    scope: namespace
    expect_contains: [burn_rate]
    command: >-
      kubectl --context {labels[cluster]} -n {labels[namespace]}
      logs deploy/checkout --tail=200 | grep -c 'status=5'

Yes, those folded scalars become multi-line strings, which is exactly why the renderer normalizes whitespace before printing. That is a detail worth fixing once in code instead of three times at 03:00. The expect_contains list is not decoration either: it tells the responder what a healthy output should look like, and it gives the linter something concrete to require.

The renderer: no model in the critical path

Here is the whole tool, roughly sixty lines, and the important part is what it does not do. It never calls a model, never queries the API server, and never falls back to a default namespace when a label is missing.

#!/usr/bin/env python3
'''firstcmd: render the pinned first command for an alert. Offline, deterministic.'''
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

import yaml

ROOT = Path(__file__).resolve().parents[1]
RUNBOOK = ROOT / 'runbook' / 'alerts.yml'


def load_alerts(path: Path = RUNBOOK) -> list[dict]:
    return yaml.safe_load(path.read_text(encoding='utf-8'))['alerts']


def render(alert: dict, payload: dict) -> str:
    scope = {
        'labels': payload.get('labels', {}),
        'annotations': payload.get('annotations', {}),
    }
    try:
        raw = alert['command'].format(**scope)
    except KeyError as exc:
        name = alert.get('name', '?')
        raise SystemExit(f'{name} needs label {exc} that the payload did not carry')
    return re.sub(r'\s+', ' ', raw).strip()


def main(argv: list[str]) -> int:
    if len(argv) != 3:
        print('usage: firstcmd <AlertName> <payload.json>', file=sys.stderr)
        return 2
    alerts = {a['name']: a for a in load_alerts()}
    alert = alerts.get(argv[1])
    if alert is None:
        print(f'firstcmd: no pinned command for {argv[1]}', file=sys.stderr)
        return 3
    payload = json.loads(Path(argv[2]).read_text(encoding='utf-8'))
    print(render(alert, payload))
    return 0


if __name__ == '__main__':
    raise SystemExit(main(sys.argv))

Exit code 3 matters more than it looks. Unknown alerts are the normal failure mode after a routing change, and a loud non-zero exit in the pager script is what stops somebody from improvising a command under pressure. Exit code 2 is a usage error, which tells you the pager wiring broke rather than the runbook.

Usage is deliberately boring:

python -m venv .venv && . .venv/bin/activate
pip install pyyaml==6.0.2 pytest==8.3.3
./bin/firstcmd CheckoutLatencyHigh tests/fixtures/checkout-latency-eu1.json
# kubectl --context eu-1 -n prod-payments rollout status deploy/checkout --timeout=30s

Tests that turn a convention into a rule

Four checks catch almost every way this small system rots, and each one is a handful of lines. Without them you have a convention, which is a polite word for a thing that will be wrong in six weeks.

1. Completeness. Every alert entry has name, owner, escalation, scope, expect_contains, and command.
2. Renderability. Every fixture payload renders without a leftover brace or a KeyError.
3. Scope binding. A cluster-scoped command must reference {labels[cluster]}, so nobody inherits their local kube-context by accident.
4. Orphan fixtures. A fixture whose alert name no longer exists fails loudly, which is how you notice a rename before the page does.

# tests/test_first_command_contract.py
import json
from pathlib import Path

import pytest
import yaml

from firstcmd import render

ROOT = Path(__file__).resolve().parents[1]
ALERTS = yaml.safe_load((ROOT / 'runbook' / 'alerts.yml').read_text())['alerts']
FIXTURES = sorted((ROOT / 'tests' / 'fixtures').glob('*.json'))
REQUIRED = {'name', 'owner', 'escalation', 'scope', 'expect_contains', 'command'}


def by_name(payload_path: Path) -> str:
    return json.loads(payload_path.read_text())['alert_name']


@pytest.mark.parametrize('alert', ALERTS, ids=lambda a: a.get('name', '?'))
def test_entry_is_complete(alert):
    name = alert.get('name', '?')
    missing = sorted(REQUIRED - alert.keys())
    assert not missing, f'{name} missing {missing}'


@pytest.mark.parametrize('alert', ALERTS, ids=lambda a: a.get('name', '?'))
def test_command_renders_from_every_matching_fixture(alert):
    name = alert['name']
    matches = [f for f in FIXTURES if by_name(f) == name]
    assert matches, f'{name} has no fingerprint fixture'
    for fixture in matches:
        out = render(alert, json.loads(fixture.read_text()))
        assert '{' not in out and '}' not in out, out


@pytest.mark.parametrize('alert', ALERTS, ids=lambda a: a.get('name', '?'))
def test_cluster_scoped_commands_bind_the_cluster(alert):
    if alert['scope'] != 'cluster':
        return
    assert '{labels[cluster]}' in alert['command'], 'inherits caller kube-context'


def test_no_orphan_fixtures():
    known = {a['name'] for a in ALERTS}
    orphans = [f.name for f in FIXTURES if by_name(f) not in known]
    assert not orphans, orphans

Freeze status is a separate input, and I keep it out of this suite on purpose, because mixing 'is the command correct' with 'are changes allowed right now' makes both harder to reason about. A one-line CI check that refuses command: diffs during a declared freeze is enough; the rest of the enforcement belongs with your change process, not with your renderer.

Where a free model actually helps, and where it must not

Drafting is the honest use case. When a new alert lands, ask a model for three candidate first commands, then diff them against the alert's labels and reject anything that invents a namespace, a container name, or a flag you have never run. I used MonkeyCode's free model access for exactly this loop: paste a redacted payload, ask for candidates, paste the winner into alerts.yml, add a fixture, run the suite. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I would not do is put a model inside firstcmd. A stochastic step between the page and the command is a new failure mode at the worst possible time, and 'the model said so' is not an incident note. Generate offline, review as a human, pin the result, and let the test suite be the thing you trust at 03:00.

The rejection rules I use are boring and effective. Reject any candidate that omits --context on a cluster-scoped check, and reject any candidate that pipes into grep without a count or a fixed pattern. Those two rules alone removed most of the plausible-looking drafts I have seen, because fluent shell is not the same as correctly scoped shell.

Where the free server helps

The suite is only a rule if it runs on every pull request, and runbook repositories are exactly the kind of low-traffic project that gets quietly dropped from a paid CI plan. The free server option is a reasonable host for a self-hosted runner that does nothing but run pytest -q tests/ and exit. Check the current terms and capacity before you depend on it, and keep a plain fallback command in the README so a future maintainer is never stuck.

# .github/workflows/runbook.yml
name: runbook-contract
on: [pull_request]
jobs:
  contract:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - run: pip install pyyaml==6.0.2 pytest==8.3.3
      - run: pytest -q tests/

The escalation table belongs next to the command

Pinning commands is half the job; the other half is knowing when to stop. Keep this table in the same repository, and make the linter assert that every escalation value names a rotation that actually exists in your paging tool.

The useful property here is that every row references something already in the repo. That means the escalation path is reviewable in a pull request instead of living in somebody's memory of how the last incident went.

Limitations and who should skip this

• If your alerts lack stable labels, fix routing first; a renderer cannot invent scope it never receives.
• If you have three alerts and one responder, a YAML file plus four tests is more machinery than you need.
• If nobody reviews the drafted command, you have automated guessing; pin only what a human has read and run.
• If your commands need secrets, do not template them here; keep credentials in the environment and template only names.

None of this makes an incident easy. It makes the first ninety seconds repeatable, which is the part you can actually control before the caffeine kicks in. So what would your runbook look like if the only thing you were allowed to ship was one tested command per alert?