Docs That Expire With the Code: Binding Every Generated Claim to a Source Digest

A generated documentation line is only as trustworthy as its provenance, and fluent prose carries no provenance at all. The workflow below forces every model-written line to cite a concrete code artifact together with that artifact's digest at draft time. When the code moves, the digest changes, the citation goes stale, and the build fails before any reviewer spends attention on the page. This is a generation-time gate rather than a review-time ceremony, and it needs two standard-library scripts and one decision table.

Why Review Is the Wrong Place to Catch Drift

A signoff gate answers a policy question about who approved a sentence, and it says nothing about whether the sentence is still true. Reviewers rarely reopen the source file a paragraph describes, so stale prose passes approval and then rots quietly for months. Digests invert the incentive: the doc has to prove freshness mechanically, and the human reviews only the judgement calls that code cannot settle. You do not need an LLM provider to adopt this, but if you already draft with one, the citation rules below are cheap to add.

Step 1 — Build a Claim-Addressable Evidence Index

The index converts your repository into a list of addressable facts, each with a stable id and a short content digest. Public function and class signatures, environment variable lookups, and CLI flags give you most of what a reference page actually asserts. Save this as evidence_index.py and note that ast.unparse requires Python 3.9 or newer.

#!/usr/bin/env python3
"""Build a claim-addressable evidence index from a Python repo."""
import ast, hashlib, json, pathlib, re, sys

ENV_RE = re.compile(r"""os\.(?:environ\.get|getenv)\(\s*["']([A-Z][A-Z0-9_]+)["']""")
CLI_RE = re.compile(r"""add_argument\(\s*["'](--[a-z0-9][a-z0-9-]*)["']""")
SKIP = {".venv", "venv", "build", "dist", "node_modules"}


def rec(kind, name, path, line, text):
    digest = hashlib.sha256(f"{path}\n{text}".encode()).hexdigest()[:10]
    return {"id": f"{kind}:{name}", "kind": kind, "name": name,
            "path": path, "line": line, "text": text, "digest": digest}


def scan_file(path):
    src = path.read_text(encoding="utf-8")
    rel, out = str(path), []
    for node in ast.walk(ast.parse(src)):
        if isinstance(node, ast.FunctionDef) and not node.name.startswith("_"):
            out.append(rec("fn", node.name, rel, node.lineno,
                           f"def {node.name}({ast.unparse(node.args)})"))
        elif isinstance(node, ast.ClassDef) and not node.name.startswith("_"):
            bases = ", ".join(ast.unparse(b) for b in node.bases)
            out.append(rec("class", node.name, rel, node.lineno,
                           f"class {node.name}({bases})" if bases else f"class {node.name}"))
    for lineno, line in enumerate(src.splitlines(), 1):
        for key in ENV_RE.findall(line):
            out.append(rec("env", key, rel, lineno, line.strip()))
        for flag in CLI_RE.findall(line):
            out.append(rec("cli", flag, rel, lineno, line.strip()))
    return out


def main(root="src", out="build/evidence.json"):
    records, seen = [], {}
    for path in sorted(pathlib.Path(root).rglob("*.py")):
        if SKIP & set(path.parts):
            continue
        for r in scan_file(path):
            n = seen.get(r["id"], 0) + 1
            seen[r["id"]] = n
            if n > 1:
                r["id"] = f"{r['id']}~{n}"
            records.append(r)
    dest = pathlib.Path(out)
    dest.parent.mkdir(parents=True, exist_ok=True)
    dest.write_text(json.dumps({"count": len(records), "records": records}, indent=2))
    print(f"{len(records)} evidence records -> {dest}")
    return 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:3]))

Run it from the repository root and inspect the index before trusting it, because duplicates and dead code both show up there plainly. A record looks like this, and the digest is what the doc will have to quote back:

{"id": "env:APP_API_KEY", "kind": "env", "path": "src/client.py",
 "line": 42, "text": "key = os.getenv(\"APP_API_KEY\", \"\")", "digest": "8c1d0b7e2a"}

Step 2 — Mark the Drafting Surface So Regeneration Cannot Touch Human Text

Model output must land inside explicit region markers, and everything outside those markers belongs to a named human. This makes the ownership question structural instead of procedural: a regeneration pass literally cannot overwrite text it has no marker for. Every file also carries a file-level owner, and the checker refuses files with no owner at all.

<!-- owner: avery -->
## Configuration

<!-- model:start -->
The client reads its API key from `APP_API_KEY`, and an empty value disables uploads [[ev:env:APP_API_KEY@8c1d0b7e2a]].
Timeouts default to 30 seconds through the `--timeout` flag [[ev:cli:--timeout@41ff9c02d3]].
<!-- model:end -->

## Support window

We answer configuration questions during business hours; escalation steps are in `SUPPORT.md`.

Keep a short allowlist of files where model regions are forbidden outright, such as security, pricing, SLA, and licensing pages. The allowlist is one constant in the checker, and changing it should require a review from the owner of that class of document.

Step 3 — Fail the Build on Uncited Lines and Stale Digests

The checker walks only the model regions, so human paragraphs are never linted for citations. Each model line must carry at least one citation, and each citation's digest must match the freshly rebuilt index. Save this as check_docs.py and run it after every index rebuild.

#!/usr/bin/env python3
"""Fail the build when a generated claim is uncited or no longer true."""
import json, pathlib, re, sys

CITE = re.compile(r"\[\[ev:([a-z]+:[^@\]]+)@([0-9a-f]{10})\]\]")
OWNER = re.compile(r"<!-- owner: ([^>]+?) -->")
HUMAN_ONLY = {"SECURITY.md", "PRICING.md", "SLA.md"}


def load_index(path="build/evidence.json"):
    data = json.loads(pathlib.Path(path).read_text(encoding="utf-8"))
    return {r["id"]: r for r in data["records"]}


def check(path, index):
    text = pathlib.Path(path).read_text(encoding="utf-8")
    problems, in_model = [], False
    if pathlib.Path(path).name in HUMAN_ONLY and "model:start" in text:
        problems.append(f"{path}: model region in a human-only file")
    if not OWNER.search(text):
        problems.append(f"{path}: missing <!-- owner: ... --> marker")
    for i, raw in enumerate(text.splitlines(), 1):
        line = raw.strip()
        if line == "<!-- model:start -->":
            in_model = True
            continue
        if line == "<!-- model:end -->":
            in_model = False
            continue
        if not in_model or not line:
            continue
        cites = CITE.findall(line)
        if not cites:
            problems.append(f"{path}:{i}: model line has no evidence citation")
        for base, drafted in cites:
            r = index.get(base)
            if r is None:
                problems.append(f"{path}:{i}: unknown evidence id {base}")
            elif r["digest"] != drafted:
                problems.append(f"{path}:{i}: stale {base} -> code changed at {r['path']}:{r['line']}")
    return problems


if __name__ == "__main__":
    idx = load_index(*sys.argv[1:2] or [])
    targets = sys.argv[2:] or [str(p) for p in pathlib.Path("docs").rglob("*.md")]
    total = sum((print(p) or len(v)) for t in targets for p, v in [(t, check(t, idx))])
    print(f"{total} problem(s) across {len(targets)} file(s)")
    sys.exit(1 if total else 0)

A failing run reads like a repair queue rather than a style opinion, which is the point. Each line names the file, the stale claim, and the exact source location that moved:

$ python evidence_index.py src build/evidence.json
214 evidence records -> build/evidence.json
$ python check_docs.py build/evidence.json docs/config.md
docs/config.md:5: stale env:APP_API_KEY -> code changed at src/client.py:42
docs/config.md:6: model line has no evidence citation
2 problem(s) across 1 file(s)

Step 4 — Draft With a Free Model and Rebuild on a Free Server

The drafting call is the only place a model participates, and it receives the fact sheet rather than the repository. A prompt contract that works with small models is deliberately dull, and it forbids the two failure modes that matter: unsupported sentences and invented specifics.

Draft text for the regions between <!-- model:start --> and <!-- model:end --> only.
State a fact only if it appears in the fact sheet below.
End every factual sentence with exactly one citation of the form [[ev:<id>@<digest>]].
If the fact sheet does not support a sentence, omit the sentence instead of guessing.
Never write about support, security, pricing, or dates; those sections are human-owned.

MonkeyCode's free model access covers this drafting step, and its free server option can run the nightly evidence_index.py plus check_docs.py pair so drift surfaces without a CI budget. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Both availability claims are the operator's, so confirm the current terms yourself before depending on them for a scheduled job; the workflow is provider-neutral and the scripts run anywhere Python 3.9 or newer exists.

The scheduled ordering matters more than the tooling, so keep it fixed and boring. Rebuild the index first, then draft new model regions, then run the checker, and never let the checker run against a stale index. If the drafting step fails, publish nothing and leave the previous regions untouched, since a partially regenerated page is worse than an old one.

What the Model May Draft, and What a Human Must Own

The table below is a starting policy, not a universal rule; adjust it to your risk surface and record the adjustment in the repository. Anything with a date, a number about a third party, or a legal commitment stays with a human who rechecks the primary source.

Limitations and Who Should Skip This

The signature-level digest catches renames, parameter changes, and flag removals, but it does not catch a changed default buried in a function body. Regex scanning for os.getenv and add_argument misses dynamic access such as os.environ[key] or flags built from a config file, so treat index coverage as partial and stated rather than complete. Duplicate symbol names get suffixed ids, which keeps the build green but can still make a page cite the wrong twin. Prose-heavy documents such as architecture narratives gain little here, because most of their value is judgement that no artifact can anchor.

Skip this approach if a machine-readable schema already generates your reference docs, since the digests would duplicate a stronger guarantee. Skip it if your team cannot sustain a nightly rebuild, because a stale index produces confusing failures and erodes trust in the gate. Skip it as well if reviewers treat a red build as noise; a gate that everyone mutes is worse than no gate. If you want to trial the drafting half without hosting inference yourself, MonkeyCode's free model access is one place to start, and the checker will tell you honestly whether the output was worth keeping.