Make the Draft Refuse: An Abstention Ledger for Model-Written Docs

Most teams measure a model-written doc page by what it contains, which is the wrong signal to optimize first. The cheaper, earlier signal is where the draft refused to answer, because a refusal is a named human decision waiting to happen. This article describes a workflow I use for documentation generation with a strict split: the model may draft prose, structure, and cross-links, while a human must own every command, version number, limit, and interface guarantee. The artifact is an abstention ledger — a marked-up draft plus a parser that turns refusals into an assignment table your review can act on.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The drafting pass described below is one place where MonkeyCode's free model access fits (availability as described by the operator; verify current terms before depending on it), and the free server option is only relevant if you want the check reachable from CI without a self-hosted runner.

The failure mode is silent confidence, not bad prose

A generated page usually fails in one of two directions, and only one of them is visible in review. The visible failure is a placeholder like TODO or a dangling link, which a reviewer catches in seconds. The expensive failure is a sentence that reads perfectly, asserts a version pin or a rate limit, and is simply not true. Reviewers skim prose for tone and miss those assertions, because nothing in the sentence signals that it needs verification.

Forcing abstention inverts that default. Instead of asking the model to be cautious in prose, you give it a syntactic obligation: when a claim depends on a fact it cannot cite from your repository, it must emit a marker and stop. The marker converts an invisible risk into a countable object with a line number and an owner.

Step 1 — Contract the draft with a marker the parser can read

Write the marker once in the system prompt and keep it stable across drafting runs. HTML comments survive Markdown rendering, so they disappear from the published page while remaining greppable in the source. The three required fields are deliberately minimal, because every extra field is another thing the model can get wrong.

When a sentence would need a fact you cannot cite from the provided files,
replace the claim with exactly this comment and continue:

<!-- NEEDS id="<short-slug>" owner="<team-or-person>" question="<one question>" -->

Do not guess the fact. Do not soften it with "typically" or "usually".
Emit the marker and move on.

A drafting run that produces zero markers across a long page is a warning, not a success. It usually means the model absorbed unstated assumptions from its training data instead of from your repository, and those assumptions are exactly the ones a reviewer cannot check quickly.

Step 2 — Parse the draft into a routing table

The parser does two jobs: it extracts every marker, and it finds silent assertions — version numbers, shell commands, URLs, and unit-bearing figures that appear without a marker. The ratio between those two counts is the metric used in Step 4.

#!/usr/bin/env python3
"""abstention.py: turn a model-drafted page into a review routing table."""
from __future__ import annotations

import json
import re
import sys
from dataclasses import asdict, dataclass
from pathlib import Path

MARKER = re.compile(r"<!--\s*NEEDS\s+(?P<body>[^>]*?)-->", re.S)
FIELD = re.compile(r'(?P<key>[a-z_]+)\s*=\s*"(?P<val>[^"]*)"')

RISK = {
    "version": re.compile(r"\bv?\d+\.\d+(?:\.\d+)?\b"),
    "command": re.compile(r"^\s*(?:\$ |npm |pip |docker |curl |git )", re.M),
    "url": re.compile(r"https?://[^\s)]+"),
    "quantity": re.compile(r"\b\d+(?:\s?%|\s?ms|\s?MB|\s?GB|\s?rps)\b"),
}


@dataclass(frozen=True)
class Need:
    line: int
    id: str
    owner: str
    question: str


def parse_needs(text: str) -> list[Need]:
    needs: list[Need] = []
    for match in MARKER.finditer(text):
        fields = {f["key"]: f["val"] for f in FIELD.finditer(match["body"])}
        missing = {"id", "owner", "question"} - fields.keys()
        if missing:
            raise ValueError(f"marker at line offset {match.start()} missing {sorted(missing)}")
        line = text.count("\n", 0, match.start()) + 1
        needs.append(Need(line, fields["id"], fields["owner"], fields["question"]))
    return needs


def silent_assertions(text: str) -> list[dict]:
    hits: list[dict] = []
    for kind, pattern in RISK.items():
        for match in pattern.finditer(text):
            hits.append({
                "kind": kind,
                "line": text.count("\n", 0, match.start()) + 1,
                "text": match.group(0).strip()[:80],
            })
    return hits


def build_report(text: str) -> dict:
    needs = parse_needs(text)
    bare = silent_assertions(text)
    by_owner: dict[str, list[str]] = {}
    for need in needs:
        by_owner.setdefault(need.owner, []).append(need.id)
    return {
        "lines": text.count("\n") + 1,
        "needs": [asdict(n) for n in needs],
        "by_owner": by_owner,
        "silent_assertions": bare,
        "abstention_density": round(len(needs) / max(len(bare), 1), 3),
        "unowned": [n.id for n in needs if n.owner.lower() in {"", "tbd", "docs"}],
    }


def main(argv: list[str]) -> int:
    report = build_report(Path(argv[1]).read_text(encoding="utf-8"))
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 2 if report["unowned"] else 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Run it as a normal check and keep the exit code meaningful, so CI fails on unowned markers rather than on aesthetics:

python abstention.py docs/queue.md > abstention-report.json
echo "exit=$?"   # 2 means at least one marker has owner="", "tbd", or "docs"

Pin the behaviour with a small fixture instead of trusting the regex by inspection. The test below is the one I keep next to the script; it is deliberately boring, because a parser that surprises you is worse than no parser.

# test_abstention.py
from abstention import build_report

DRAFT = '''
Set the retry window to 30s.
<!-- NEEDS id="retry-window" owner="platform" question="What is the default retry window?" -->
Run `npm i [email protected]` before starting.
'''


def test_counts_marker_and_silent_assertions():
    report = build_report(DRAFT)
    assert [n["id"] for n in report["needs"]] == ["retry-window"]
    kinds = {hit["kind"] for hit in report["silent_assertions"]}
    assert "version" in kinds and "command" in kinds
    assert report["unowned"] == []


def test_rejects_malformed_marker():
    import pytest
    from abstention import parse_needs
    with pytest.raises(ValueError):
        parse_needs('<!-- NEEDS id="x" -->')

Step 3 — Route each need to an owner, not to "the docs team"

The parser is only useful if owner maps to someone who can answer in one message. Reject generated owners like docs, team, or tbd in the contract itself, and let the exit code enforce it. The table below is a template; replace the middle column with the roles that actually exist in your repository's ownership file.

Two rules keep this from decaying into a second backlog. First, every marker must resolve into either a sentence with a citation in the page or a linked registry entry. Second, resolved markers are deleted, never left in place as commentary, so the ledger always reflects open work only.

Step 4 — Gate the re-draft on abstention density

Abstention density is markers divided by silent assertions, and it is a prompt-quality signal rather than a quality score for the page. A low density means the model asserted facts it could not source, which usually points to a prompt that omitted the repository context it needed. A very high density with few assertions means the draft is thin and probably dropped sections you expected, which a heading-coverage diff will confirm.

Use it as a routing decision, not a number to chase:

• density < 0.3 and assertions present: re-draft with narrower scope and the specific files attached, rather than sending it to review.
• density >= 0.3 with assertions clustered in one section: send that section to its owner first and hold the rest.
• Zero markers on a page longer than roughly 150 lines: treat as suspicious and diff the headings against the previous version before reading prose.
• Markers whose questions repeat across two drafts: the fact belongs in the registry, not in the prompt.

Step 5 — Feed resolved needs back into the registry

The last step closes the loop, and it is the one teams skip. When an owner answers a marker question, write the answer as a dated entry in the facts file the drafting prompt already receives, then delete the marker. The next draft starts with one fewer unknown, so the ledger shrinks instead of resetting every release.

This is also where the free-server option is worth considering: exposing abstention.py as a small endpoint lets a webhook run the check on a pull request without provisioning a machine for it. If that is more moving parts than your team wants, a pre-commit hook calling the same script is equivalent, and the ledger does not care which one you pick.

What this does not fix

Abstention markers catch ungrounded claims, not wrong ones: a model can cite a stale file and produce a confidently sourced error. The approach also says nothing about structure, tone, or whether the page answers the reader's actual question. And it adds real friction — every marker is a human round-trip, so a page with twenty markers can take longer than writing it by hand.

Skip this workflow when the model already has a narrow, tested context: a generated reference page rebuilt from a schema, a changelog assembled from merged pull requests, or any page where examples are extracted from executed test files. In those cases the facts are already constrained, and a marker layer mostly adds ceremony. Skip it too if your repository has no ownership file, because routing needs an owner that exists.

If you try one thing from this article, make it the contract in Step 1 and the exit code in Step 2 — a draft that names its own gaps is easier to review than a fluent one that hides them.