Draft Only What a Probe Can Fail: A Section Ownership Ledger for Model-Written Docs

Split documentation at the point where a claim can be falsified by a command. Everything on the machine-checkable side of that line may be drafted by a model; everything on the other side needs a named human author, not a reviewer. This post turns that rule into two small scripts and a tab-separated ledger you can run in CI.

Most doc-generation failures I have seen are not hallucinated sentences. They are correctly written sentences about a surface that changed, sitting in a section nobody was assigned to own. A reviewer checkbox does not fix that, because reviewing prose is cheap and owning a fact is expensive. The ledger makes the ownership explicit and makes the split mechanical.

The split rule, stated precisely

A section is model-draftable only when it satisfies three conditions at once. Its factual content must be extractable from a program (CLI help text, config schema, HTTP headers, exit codes). Every number in the drafted prose must appear in that extracted bundle. And it must carry zero unresolved judgment claims, meaning no sentence of the form "you should prefer X because Y".

Everything else is human-owned. That includes migration advice, security guarantees, deprecation rationale, performance expectations, and any statement about intent. Those sections are not reviewed by a human after drafting; they are written by a human and merely proofread by whoever is available.

The split is worth doing because the two failure modes have different costs. A stale flag table wastes a reader's afternoon. A wrong migration recommendation wastes a reader's week.

Step 1: Write the section ledger before any prompt

Create docplan.tsv next to your docs directory. Five columns, tab-separated, one row per section. The probe column points at a test file relative to the plan; handwritten_claims counts the judgment statements you already know the section needs.

section facts_class probe   owner   draft   handwritten_claims
cli-flags   machine tests/probe_cli.py::test_help_text  @docs-cli   model   0
config-keys machine tests/probe_config.py::test_schema_dump @docs-cli   model   0
exit-codes  machine tests/probe_exit.py::test_exit_matrix   @docs-cli   model   0
migration-guide judgment    -   @avery-lin  human   4
security-model  judgment    -   @avery-lin  human   6

The important column is handwritten_claims. It is a declaration that this section contains sentences no probe can produce, and it forces the row into the human bucket. If a section starts at 0 and later grows judgment content, the count changes and the gate re-files it.

Step 2: Gate the plan in CI with one dependency-free script

The gate does not know whether the docs are true. It only enforces that the plan is coherent and that every model-drafted section has a probe file that actually exists. That is enough to catch the common failure where a section is marked model-draftable and the test was deleted six weeks ago.

#!/usr/bin/env python3
"""ownership_gate.py - fail the build when a section is model-drafted without a probe,
   or human-drafted without declaring its judgment claims."""
from __future__ import annotations
import csv, sys
from pathlib import Path

COLUMNS = ["section", "facts_class", "probe", "owner", "draft", "handwritten_claims"]
VALID_CLASS = {"machine", "judgment"}
VALID_DRAFT = {"model", "human"}


def load(plan_path: Path):
    with plan_path.open(newline="", encoding="utf-8") as fh:
        reader = csv.DictReader(fh, delimiter="\t")
        missing = [c for c in COLUMNS if c not in (reader.fieldnames or [])]
        if missing:
            raise SystemExit(f"plan header missing columns: {missing}")
        return list(reader)


def check(rows, root: Path):
    errors, seen = [], set()
    for i, row in enumerate(rows, start=2):
        name = row["section"].strip()
        where = f"line {i} ({name or 'unnamed'})"
        if not name:
            errors.append(f"{where}: empty section name")
            continue
        if name in seen:
            errors.append(f"{where}: duplicate section")
        seen.add(name)
        if row["facts_class"] not in VALID_CLASS:
            errors.append(f"{where}: facts_class must be one of {sorted(VALID_CLASS)}")
        if row["draft"] not in VALID_DRAFT:
            errors.append(f"{where}: draft must be one of {sorted(VALID_DRAFT)}")
        probe, owner = row["probe"].strip(), row["owner"].strip()
        try:
            claims = int(row["handwritten_claims"])
        except ValueError:
            errors.append(f"{where}: handwritten_claims must be an integer")
            claims = -1
        if not owner.startswith("@"):
            errors.append(f"{where}: owner must be an @handle, got {owner!r}")
        if row["draft"] == "model":
            if row["facts_class"] != "machine":
                errors.append(f"{where}: model draft requires facts_class=machine")
            if probe in ("", "-"):
                errors.append(f"{where}: model draft requires a probe")
            elif not (root / probe.split("::")[0]).exists():
                errors.append(f"{where}: probe file not found: {probe}")
            if claims != 0:
                errors.append(f"{where}: model draft requires handwritten_claims=0")
        if row["draft"] == "human" and claims < 1:
            errors.append(f"{where}: human draft must declare at least one judgment claim")
    return errors


def main(argv):
    plan = Path(argv[1])
    rows = load(plan)
    errors = check(rows, plan.parent)
    for err in errors:
        print(f"FAIL {err}", file=sys.stderr)
    model = sum(1 for r in rows if r["draft"] == "model")
    print(f"sections={len(rows)} model={model} human={len(rows) - model}")
    return 1 if errors else 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Run it as python3 ownership_gate.py docplan.tsv. On the table above it prints sections=5 model=3 human=2 and exits zero. Rename a probe file and it exits one with a line-pointer and the missing path.

Step 3: Generate the fact bundle before the prompt

Each probe writes a JSON bundle of the facts it can observe. The model receives the bundle and nothing else as a source of numbers. Keeping the bundle plain text matters: it makes the input diffable, so a reviewer can see that the flag list shrank by two entries this week.

mkdir -p .docfacts
for probe in tests/probe_cli.py tests/probe_config.py tests/probe_exit.py; do
  name=$(basename "$probe" .py)
  python3 "$probe" > ".docfacts/${name}.json"
done
ls -1 .docfacts

This is also where a hosted environment saves time. MonkeyCode's free model access and free server option are operator-supplied availability claims, and they are relevant here in a narrow way: the drafting loop and the probe pipeline can both run on a throwaway machine instead of on the laptop that holds your credentials.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 4: Audit numbers after drafting, not before

A model can produce a fluent section that cites a real fact bundle and still misattribute a number, for example by pairing a timeout value with the wrong endpoint. The audit is deliberately narrow: it checks only that every numeral in the drafted section appears somewhere in that section's bundle. It cannot catch misattribution, which is why step 5 exists.

#!/usr/bin/env python3
"""fact_audit.py - every numeral in a drafted section must exist in its fact bundle."""
import re, sys
from pathlib import Path

NUM = re.compile(r"(?<![\w.])-?\d+(?:\.\d+)?(?![\w.])")


def numbers(text):
    return {m.group(0) for m in NUM.finditer(text)}


def main(facts_path, draft_path):
    facts = numbers(Path(facts_path).read_text(encoding="utf-8"))
    draft = numbers(Path(draft_path).read_text(encoding="utf-8"))
    unbacked = sorted(draft - facts)
    for n in unbacked:
        print(f"UNBACKED {n} in {draft_path}", file=sys.stderr)
    print(f"draft_numbers={len(draft)} backed={len(draft & facts)} unbacked={len(unbacked)}")
    return 1 if unbacked else 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1], sys.argv[2]))

Call it per section: python3 fact_audit.py .docfacts/probe_cli.json docs/cli-flags.md. The exit code is the gate. A section that reports unbacked=0 is not correct; it is merely anchored.

What the model may draft, and what a human must own

Use this table when you are unsure which side of the line a section belongs on. The right column is not a review duty; it is authorship.

Step 5: Let the human-owned sections stay stubs

The most useful behaviour of this pipeline is that it does not fill the judgment sections. The generator writes stubs/migration-guide.md containing the four declared claims as open questions and the owning handle, and CI fails while those files are missing. The model never touches them, which removes the temptation to accept a plausible paragraph because it arrived on time.

That is the whole discipline: the model is fast where a test can disagree with it, and silent where only a person can.

Limitations you should plan for

Probe existence is not probe quality, so a probe that always emits an empty bundle will pass the gate and produce hollow docs. The numeric audit catches invented figures and nothing else, meaning wrong pairings of real facts survive it. The ledger also cannot verify that the named owner read anything, so treat the handle as an accountability statement rather than proof of work.

Finally, do not wire a free tier into a release-blocking path as a permanent assumption. Availability, model behaviour, and limits can change without notice; keep a fallback where the three machine sections are generated from the fact bundle by a local template when the drafting step is unavailable.

Who should not use this

Skip the ledger if your documentation is primarily positioning, tutorials, or narrative explanation, because almost every section lands in the human bucket and the plan becomes paperwork. Skip it if your repository has no executable surface to probe, or if the docs change less often than the ledger itself would. Teams of one or two may also find the tab-separated file heavier than a single OWNERS note in the README.

If you want to try it, the two scripts above are the entire dependency list; start by classifying five existing sections and see how many survive as model-draftable.