Docs Rot When Facts Change, Not When Drafts Are Wrong: A Fact-Fingerprint Drift Gate

The most expensive documentation bug is not a badly written paragraph. It is a correct paragraph whose facts quietly changed three releases ago, still sitting on page one of your docs with a green build badge next to it. Review gates catch bad writing at the moment of drafting; almost nothing catches a page that was accurate when merged and is wrong today.

This article describes a small gate that targets that second failure mode. The idea is to stop treating every documentation block as prose that a reviewer approves once, and start treating the machine-checkable subset as output that carries a fingerprint of its inputs, so staleness becomes a build failure instead of a support ticket.

Measure your own drift before building anything

Do not take my word for the size of the problem. You can measure it locally in about two minutes, using only commands that already exist in your repository.

# 1. How often do the docs change at all?
git log --oneline --since="12 months ago" -- docs | wc -l

# 2. How often does the thing the docs describe change, without docs changing?
git log --oneline --since="12 months ago" -- ':!docs' -- src/ schemas/ cli/ | wc -l

# 3. Where do the two histories separate?
git log --format='%h %ad %s' --date=short -- docs | head -20

If the second number dwarfs the first, your documentation is not being reviewed on the wrong schedule; it is not being reviewed on any schedule at all. That asymmetry is the whole argument for a gate that runs on every change to the source of truth, not on every change to the Markdown.

Step 1: Split each page into four block kinds

Before any tooling, decide what each block on the page actually is. The distinction that matters is not "AI-generated versus human-written", it is "recomputable from a source of truth versus asserted by a person".

Write that table down in the repository, because it is the only place where the model's drafting rights are defined. Anything not in the first two rows is out of scope for generation, and a reviewer can reject a generated block on classification grounds alone rather than arguing about tone.

Step 2: Mark each derived block with the facts it depends on

A derived block declares its inputs in an HTML comment, so the marker is invisible on the rendered page and parseable in the source:

<!-- facts: cli.flags, schema.user.email | sha256=REPLACE_ME -->
| Flag | Type | Description |
|---|---|---|
| `--dry-run` | bool | Print the plan and exit without applying it. |
<!-- /facts -->

The sha256 value is not a checksum of the Markdown; it is a hash of the extracted fact values. That distinction is what makes the gate cheap and stable: cosmetic edits to the table never invalidate it, but a new flag or a changed field type always does.

Step 3: A reference implementation of the gate

The script below is a reference implementation, not a packaged tool. The extractor functions at the top are deliberately stubs that call your project's real commands; I have not run it against your repository, and the only part you should trust without testing is the control flow.

# docfacts.py — reference implementation, Python 3.11+
from __future__ import annotations

import hashlib
import json
import re
import subprocess
import sys
from pathlib import Path


def run(cmd: list[str]) -> str:
    return subprocess.run(cmd, capture_output=True, text=True, check=True).stdout


def cli_flags() -> list[str]:
    out = run(["python", "-m", "myapp", "--help"])
    return sorted(re.findall(r"^\s{2}(--[a-z0-9-]+)", out, re.M))


def schema_field(dotted: str) -> dict:
    node = json.loads(Path("schemas/user.json").read_text())
    for part in dotted.split(".")[1:]:
        node = node["properties"][part]
    return {"type": node["type"], "required": node.get("required", False)}


EXTRACTORS = {
    "cli.flags": cli_flags,
    "schema.user.email": lambda: schema_field("user.email"),
    "schema.user.age": lambda: schema_field("user.age"),
}

BLOCK = re.compile(
    r"<!-- facts: (?P<ids>[^|>]+?) \| sha256=(?P<fp>[0-9a-f]{16}) -->\n"
    r"(?P<body>.*?)\n<!-- /facts -->",
    re.S,
)


def fingerprint(ids: list[str]) -> str:
    payload = {i: EXTRACTORS[i]() for i in ids}
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()[:16]


def scan(path: Path, write: bool = False) -> list[str]:
    text = path.read_text()
    stale: list[str] = []
    for m in BLOCK.finditer(text):
        ids = [i.strip() for i in m["ids"].split(",")]
        want = fingerprint(ids)
        if m["fp"] == want:
            continue
        stale.append(f"{path}: sha256={m['fp']} -> {want} ({', '.join(ids)})")
        if write:
            text = text.replace(m[0], m[0].replace(m["fp"], want, 1))
    if write and stale:
        path.write_text(text)
    return stale


if __name__ == "__main__":
    write = "--write" in sys.argv
    drift = [row for p in sorted(Path("docs").rglob("*.md")) for row in scan(p, write)]
    for row in drift:
        print(row)
    raise SystemExit(1 if drift and not write else 0)

Two design choices are worth defending. First, the fingerprint is truncated to sixteen hex characters, which is far beyond collision needs for a docs gate and keeps the marker readable. Second, an unknown fact id raises KeyError instead of being skipped, so deleting an extractor is a loud failure rather than a silent blind spot.

Step 4: Make the human own the bump, not the paragraph

This is the part that decides whether the gate helps or just annoys people. When docfacts.py reports drift, the workflow is:

1. Run python docfacts.py --write to refresh fingerprints locally.
2. Open the resulting diff. The changed lines are, by construction, only the derived blocks.
3. Rerun the extractors against the new source and compare values, so the diff shows which fact moved, not merely that something moved.
4. Edit the block body so the prose matches the new values.
5. A named human commits the fingerprint bump with a message that says what changed, for example docs: user.email is now required.
6. CI runs python docfacts.py with no --write and fails on any remaining mismatch.

The model is welcome at steps 3 and 4, because rewriting a table row to match a type change is mechanical work with the answer already in the diff. The model is not welcome at step 5, because accepting a fingerprint bump is a claim that the new facts are correct, and that claim belongs to someone whose name is in the log.

Step 5: Give the model the derived blocks only

A generation pass that respects the split is small and boring, which is the point. Feed the model the extracted fact JSON and the current block body, and constrain it to emit a replacement body with the same fact ids and an untouched <!-- /facts --> boundary. Reject any output that introduces a sentence not grounded in the fact JSON; that sentence belongs in a narrative block, and narrative blocks are not generated.

Where I find MonkeyCode useful in this loop is exactly at that boundary: the free model access covers the mechanical rewrite of derived blocks, and the free server option is a natural host for docfacts.py as a pre-commit hook, since the check is a short CPU-bound script that needs no accelerator and no persistent storage. Both of those are availability claims from the operator rather than benchmarks, so confirm the current limits in the product before you make the gate depend on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations, and who should not adopt this

Fingerprints verify consistency, not truth. If your extractor reads the wrong file, every block agrees with a wrong world, and the gate stays green. Narrative claims about performance, compatibility, and security guarantees are outside the gate entirely, and treating a green run as approval for them would be worse than having no gate.

Fact ids also coarsen over time. A block that depends on forty facts will bump constantly and train reviewers to ignore it; keep blocks to a handful of facts and split pages instead. Finally, if your prose is the product, as in design documents, RFCs, or legal text, this approach is a poor fit, because there is no source of truth to hash against.

Do not adopt it if you have fewer than roughly twenty pages, no machine-readable schema or CLI help output, or no CI that runs on pull requests. In those cases a quarterly read-through is cheaper than the tooling, and honesty about that is part of shipping documentation infrastructure people keep.

Start with a single page, the one that has been wrong most recently, and mark only its flag table. If the gate catches a real drift within a month, extend it to the next page; if it never fires, you have learned that your docs problem lives somewhere else.