Pin the Evidence, Then Let the Model Draft: A Resolvable-Reference Gate for Generated Docs

Documentation drafted by a model fails in a predictable pattern: each sentence was accurate when generated, and nothing rechecks it later. The remedy is not a stronger prompt but a contract that binds every doc section to evidence a script can resolve on each pull request. The workflow below splits drafting rights per section, pins model-written claims to concrete files, symbols, and tests, and blocks publication when any pin stops resolving. It is a reference design you run and adapt, not a benchmarked product claim.

The failure mode is drifting evidence, not sloppy prose

A model writes confidently from whatever context you supply, and that context decays the moment the code moves. Six weeks after generation, the retry default in your docs may contradict RETRY_LIMIT, and no reviewer will notice because the paragraph reads well. Reviewers catch grammar; they rarely re-derive configuration defaults from source. The gate has to be mechanical, cheap, and run on every change.

Step 1 — Declare a section contract

Store ownership and evidence in one YAML file committed next to the docs. Each entry names a section, the file path plus anchor, an owner, and the evidence that justifies the text. Human-owned sections carry no model instructions and require an attestation later.

# docs.contract.yaml
version: 1
sections:
  - id: config-retries
    path: docs/config.md#retries
    owner: model
    max_age_days: 30
    evidence:
      - kind: symbol
        file: src/config/schema.py
        pattern: '^RETRY_LIMIT\s*=\s*\d+'
      - kind: test
        id: tests/test_config.py::test_retry_limit_default

  - id: threat-model
    path: docs/security.md#threat-model
    owner: human
    requires_attestation: true
    evidence:
      - kind: file
        path: SECURITY.md

Step 2 — Resolve the pins with a script

The resolver is deliberately small: file existence, a regex anchor, and a pytest collection check. Collection is a liveness proxy, so it proves the referenced test still exists without paying for a full run.

# tools/docpin.py  — reference implementation, Python 3.9+
import pathlib, re, subprocess, sys
import yaml

ROOT = pathlib.Path('.').resolve()

def pin_file(ref):
    ok = (ROOT / ref['path']).is_file()
    return ok, f"file:{ref['path']}"

def pin_symbol(ref):
    p = ROOT / ref['file']
    if not p.is_file():
        return False, f"symbol:{ref['file']} (missing file)"
    hit = re.search(ref['pattern'], p.read_text(encoding='utf-8'), re.M)
    return bool(hit), f"symbol:{ref['file']}::{ref['pattern']}"

def pin_test(ref):
    out = subprocess.run(
        [sys.executable, '-m', 'pytest', '--collect-only', '-q', ref['id']],
        capture_output=True, text=True,
    )
    return ref['id'] in out.stdout, f"test:{ref['id']}"

PINNERS = {'file': pin_file, 'symbol': pin_symbol, 'test': pin_test}

def main(contract_path='docs.contract.yaml'):
    contract = yaml.safe_load(pathlib.Path(contract_path).read_text())
    failures = []
    for section in contract['sections']:
        for ref in section.get('evidence', []):
            ok, label = PINNERS[ref['kind']](ref)
            if not ok:
                failures.append(f"{section['id']}: unresolved {label}")
    for f in failures:
        print('FAIL', f)
    print(f"{len(failures)} unresolved pin(s) across "
          f"{len(contract['sections'])} section(s)")
    return 1 if failures else 0

if __name__ == '__main__':
    sys.exit(main(*sys.argv[1:]))

Run it locally before you wire CI, and fix every unresolved pin by hand first:

python -m pip install pyyaml
python tools/docpin.py docs.contract.yaml

Step 3 — Draft only the model-owned sections

Feed the model the section id and its evidence refs, and nothing else. MonkeyCode's free model access is enough for this step, because the drafting prompt is short and the output is a single section.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Draft section "config-retries" for docs/config.md in 120 words or fewer.
You may assert only what these evidence entries support:
  [ev:1] src/config/schema.py :: ^RETRY_LIMIT\s*=\s*\d+
  [ev:2] tests/test_config.py::test_retry_limit_default
End every sentence with the pin it relies on, e.g. "Retries default to 3 [ev:1]."
Omit any behavior no listed evidence covers. Do not invent env vars or limits.

Then reject output that carries an unpinned sentence, so padding cannot slip through unnoticed:

import re

def unpinned(text):
    sents = [s for s in re.split(r'(?<=[.!?])\s+', text.strip()) if s]
    return [s for s in sents if not re.search(r'\[ev:\d+\]', s)]

Strip the [ev:n] markers before publishing, but keep them in the pull request description so a reviewer can trace each claim back to code.

Step 4 — Attest human sections against a content hash

An attestation without a hash survives edits it never approved. Hash the section body and store the digest, signer, and date so that any silent rewrite invalidates the approval automatically.

# hash the section body extracted by your anchor tool
python - <<'PY'
import hashlib, pathlib
body = pathlib.Path('docs/security.md').read_text().split('## Threat model')[1]
print(hashlib.sha256(body.strip().encode()).hexdigest()[:16])
PY

# docs/attestations.yaml
threat-model:
  sha256_16: "REPLACE_WITH_COMPUTED_DIGEST"
  signer: "@your-handle"
  date: "2026-11-08"
  scope: "threat model section body only"

Extend the resolver with a check that recomputes each digest and fails on mismatch. That single rule converts "someone approved this once" into "someone approved exactly this text."

Step 5 — Gate the pull request

Run the resolver and the hash check as one job, and keep the job under a minute so nobody routes around it. The free server option is adequate here because the job is I/O-bound and short; treat that availability as operator-supplied rather than guaranteed.

# .github/workflows/docpin.yml
name: docpin
on: [pull_request]
jobs:
  pins:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install pyyaml pytest
      - run: python tools/docpin.py docs.contract.yaml
      - run: python tools/docpin.py --attestations docs/attestations.yaml

What the gate cannot tell you

• A resolving regex proves a symbol exists, not that the surrounding prose describes it correctly.
• Test collection proves the test is present, not that it passes, so pair this with your normal suite.
• Symbol patterns break on refactors, so prefer AST lookups once a file passes a few hundred lines.
• Hash checks fire on formatting-only edits, which is noisy on sections that change often.
• Nothing here verifies tone, ordering, or whether the section belongs in the documentation at all.

Who should skip this

If your repository holds fewer than five doc pages, a link checker plus a review checklist costs less than this pipeline. Teams whose docs require legal or compliance signoff need a policy engine with per-clause approval, not a hash of a section body. Projects without CI should fix that first, because a gate with no enforcement point is just documentation about documentation.

A realistic adoption order

Start with symbol and file pins on the two sections that have actually burned you, and add test pins only where a test names the behavior. Attestations come second, because they add friction that only pays off on security, pricing, or compatibility statements. Run both in report-only mode for two weeks and count unresolved pins before you make the job blocking, since that count is the honest measure of how far your docs had already drifted.

If you adopt some version of this contract, the most useful signal is which pin kind fails most often in your repo — that tells you where generation should stop and a human should start.