Three Runs, One Verdict: Use Regeneration Drift to Decide Which Doc Sections a Model May Own

Delegating documentation to a model is usually argued as a policy question. It is easier to settle as a measurement question. Run the same generator three times against pinned inputs, hash every section, and the ownership boundary mostly draws itself. Sections that reproduce identically are plausible candidates for model ownership, while sections that change between runs belong to a named human. Below is a harness that makes that measurement in roughly sixty lines of standard-library Python.

One caveat shapes everything that follows. Reproducibility answers "can this text be re-derived from pinned sources", not "is this text true". A deterministic generator will happily reproduce the same wrong timeout value three times. The harness therefore decides maintainability, while humans still decide accuracy, and those two decisions deserve separate owners.

Rule one: delegability is measured, not negotiated

Most doc-ownership debates stall because nobody can point at evidence. A section assigned to a model by decree gets re-reviewed forever, and a section assigned to a human by habit rots anyway. Drift measurement replaces that argument with a three-run experiment whose result you can attach to the pull request. If a section cannot survive three runs unchanged, no reviewer will keep it current at scale.

The measurement also fails in a useful direction. When a generator drifts on a section you believed was mechanical, you learn that the section depends on something you never pinned: unsorted keys, timestamps, locale formatting, or a model call with no determinism control. Each of those is a fixable input problem rather than a reason to abandon generation.

Step 1 — Declare every section, owner, and source in one manifest

Put the ownership decision in a file that CI can read, and keep it small enough to review in a minute. Each entry names a section marker, an intended owner, and the sources that section must be derivable from. Those same sources are what the harness fingerprints to prove the inputs did not move mid-run.

{
  "generator": ["python", "gen_docs.py"],
  "inputs": ["spec/openapi.json", "docs/_install.md", "src/app/errors.py"],
  "runs": 3,
  "noise_profile": "profiles/default.txt",
  "sections": {
    "install":        { "owner": "auto",  "sources": ["docs/_install.md"] },
    "api-errors":     { "owner": "auto",  "sources": ["spec/openapi.json"] },
    "data-retention": { "owner": "human", "sources": [] },
    "sla":            { "owner": "human", "sources": [] }
  }
}

Note the deliberate asymmetry: auto-owned sections must list sources, and human-owned sections must not. A human-owned section with sources tempts someone to automate it later without repeating the measurement. Keeping that field empty makes the temptation visible during review instead of invisible after a rewrite.

Step 2 — Make the generator emit markers and write to stdout

The harness needs stable boundaries, so require one HTML comment per section and one output stream. Writing the document to stdout keeps the harness out of your file layout and makes the generator trivial to run by hand. Human-owned sections must be absent from that stream entirely.

# gen_docs.py — emits the generated portion of the docs to stdout
import json
from pathlib import Path

spec = json.loads(Path("spec/openapi.json").read_text())

def section(name: str) -> None:
    print(f"<!-- section: {name} -->")

section("install")
print(Path("docs/_install.md").read_text().strip())
print()

section("api-errors")
for name, body in sorted(spec["components"]["responses"].items()):
    print(f"- `{name}` — {body['description']}")

Hand-written text such as the retention policy stays in its own file and is assembled at build time. If somebody later pastes that paragraph into gen_docs.py, the harness reports the section as shadowed rather than stable. A machine-owned copy of a human promise is exactly the failure worth catching early.

Step 3 — Normalize conservatively and count what you hid

Normalization is where a drift harness becomes dishonest, because an aggressive profile can hide real changes. Keep the pattern list short, review it like code, and print both raw and normalized variant counts so a reviewer can see the gap. Sorting belongs in the generator, not in the harness, since sorted output is a real fix rather than a mask.

[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:.]+Z?
^generated at .*$

When a section is stable only after normalization, treat it as unverified instead of safe. The harness labels that state noise-only and refuses to promote it automatically, because deciding which characters carry meaning is a human judgment about your documentation. Any team that skips that review ends up with a green build and silently rewritten numbers.

Step 4 — Run three times, hash, and classify

The harness below splits the stream on markers, fingerprints inputs before and after the runs, and assigns each declared section one of four measured states. Exit codes rank by severity rather than by discovery order, so a single number tells the author which problem to fix first: mutated inputs, then shadowing, then missing sections, then drift.

#!/usr/bin/env python3
"""docdrift.py — classify generated doc sections by regeneration drift.

  0  every auto-owned section reproduced identically, nothing shadowed
  2  an auto-owned section produced more than one normalized form
  3  a declared section never appeared in the generator output
  4  an input file changed during the run
  5  a human-owned section was emitted by the generator
"""
from __future__ import annotations

import hashlib
import json
import re
import subprocess
import sys
from pathlib import Path

MARKER = re.compile(r"^<!-- *section: *([A-Za-z0-9_.:-]+) *-->$", re.M)

def sha(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]

def normalize(text: str, patterns: list[re.Pattern[str]]) -> str:
    for pattern in patterns:
        text = pattern.sub("<N>", text)
    lines = (line.rstrip() for line in text.splitlines())
    return "\n".join(line for line in lines if line).strip()

def split_sections(doc: str) -> dict[str, str]:
    parts = MARKER.split(doc)          # [preamble, id1, body1, id2, body2, ...]
    return {parts[i]: parts[i + 1] for i in range(1, len(parts) - 1, 2)}

def fingerprint(root: Path, paths: list[str]) -> dict[str, str]:
    return {p: sha((root / p).read_bytes().decode("utf-8", "replace")) for p in paths}

def main(manifest_path: Path) -> int:
    manifest = json.loads(manifest_path.read_text())
    root = manifest_path.parent
    profile = (root / manifest["noise_profile"]).read_text().splitlines()
    patterns = [re.compile(p) for p in profile if p.strip()]
    runs = int(manifest.get("runs", 3))

    before = fingerprint(root, manifest["inputs"])
    samples = []
    for _ in range(runs):
        proc = subprocess.run(manifest["generator"], cwd=root,
                              capture_output=True, text=True)
        if proc.returncode != 0:
            print(proc.stderr[-2000:], file=sys.stderr)
            return 1
        samples.append((proc.stdout, split_sections(proc.stdout)))

    if before != fingerprint(root, manifest["inputs"]):
        return 4

    report, status = {}, 0
    for sid, meta in manifest["sections"].items():
        bodies = [s[sid] for _, s in samples if sid in s]
        if len(bodies) != runs:
            report[sid] = {"state": "missing", "owner": meta["owner"]}
            status = max(status, 3)
            continue

        raw = {sha(body) for body in bodies}
        norm = {sha(normalize(body, patterns)) for body in bodies}
        state = "stable" if len(raw) == 1 else ("noise-only" if len(norm) == 1 else "drift")
        report[sid] = {"state": state, "owner": meta["owner"],
                       "raw_variants": len(raw), "norm_variants": len(norm)}

        if meta["owner"] == "human":
            report[sid]["state"] = "shadowed"
            status = max(status, 5)
        elif state != "stable":
            status = max(status, 2)

    print(json.dumps({"runs": runs, "sections": report}, indent=2, sort_keys=True))
    return status

if __name__ == "__main__":
    sys.exit(main(Path(sys.argv[1] if len(sys.argv) > 1 else "docdrift.json")))

Two details in that listing are intentional. The human-owned branch overwrites the measured state with shadowed, because presence in the stream is the problem and variance is not. The max(status, ...) calls keep the exit code deterministic when several sections fail at once, which makes CI logs comparable across runs.

Step 5 — Gate CI on the verdict, not on the prose

Run the harness on every pull request that touches a declared source, and let the exit code block the merge. A drift failure tells the author to fix ordering or pin a value, while a shadowing failure tells them a human paragraph has been absorbed into the generator. Neither message asks a reviewer to read prose, which is the entire point of the exercise.

Sample report from a deliberately unstable run:

{
  "runs": 3,
  "sections": {
    "api-errors":     { "state": "stable",   "owner": "auto",  "raw_variants": 1, "norm_variants": 1 },
    "data-retention": { "state": "shadowed", "owner": "human", "raw_variants": 1, "norm_variants": 1 },
    "install":        { "state": "noise-only", "owner": "auto", "raw_variants": 3, "norm_variants": 1 },
    "sla":            { "state": "missing",  "owner": "human" }
  }
}

Read that report in three passes. shadowed means ownership has leaked, missing means the build is incomplete, and noise-only means somebody must approve the normalization profile. Only stable needs no human decision, and even that verdict covers form rather than accuracy.

The ownership matrix

The bottom row is the one teams miss. A human-owned section can be perfectly stable and still wrong to generate, because the generator will silently overwrite an edited promise on the next build. Measured stability is evidence about mechanism, never a license to own a commitment.

Limitations and who should not use this

• Narrative docs, tutorials, and changelogs have sources too soft to pin, so drift stays constant and the harness becomes noise.
• Any workflow that cannot fix the model id, prompt file, and source snapshot will measure provider nondeterminism instead of section quality.
• Teams without a machine-readable source of truth should build one first, because the harness measures re-derivability and there is nothing to re-derive yet.
• Three runs catch frequent drift, not rare drift, so raise the count for sections that change quarterly.

Skip this approach if your documentation is mostly persuasion or design rationale. Those sections genuinely belong to humans, and the fastest correct decision is to mark them human once and stop measuring them.

Where free model access changes the cost

If cost is the reason you have not measured drift yet, the experiment is cheap to start. MonkeyCode offers free model access and a free server option, according to the operator's description of current availability, so you can run the three-run harness against a hosted model and a scheduled job before spending anything. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Check quotas, limits, and terms on the product's own pages rather than trusting this article for them, since availability claims age quickly.

The workflow itself survives any provider, and that portability is the strongest argument for building it. If you extend the report with the model id, prompt hash, and source fingerprints, then swapping providers becomes a diff you can read instead of a rewrite you must trust.