Lint your technical drafts for AI writing tells before you publish

A draft about your retrieval layer can read like a launch announcement and still say nothing you could test. Checking the prose is cheap. Checking the engineering claim is a different job. I treat AI-writing lint the way I treat a style checker on code: fast feedback on patterns, and zero proof that the logic is right.

Why does fluent technical writing fool you first?

A large language model is trained to produce likely text, not verified text, so a polished explanation can be wrong and still read like documentation. Dan Klein, co-founder and CTO of Scaled Cognition, described the problem in his episode: older language technology often showed its mistakes, because a translation that read badly also tended to be unfaithful. A technology that always sounds right but often is not makes it hard to know what to trust.

Like I said, one of my biggest fears is that, as a field, we are building systems which produce outputs indistinguishable from the truth.

Dan Klein, on Chain of Thought ep 54

The Chain of Thought writing page makes the same point with a made-up paragraph about a retrieval layer. It marks four patterns from the open-source avoid-ai-writing catalog, and then notes that fixing them makes the paragraph read better while telling you nothing about whether the retrieval layer reduces hallucinations.

Which four patterns should you mark?

The page labels these:

1. Significance inflation: the sentence claims historic weight without saying what changed. Ask what the before and after are, in a number or a behavior.
2. Vague attribution: a crowd of unnamed experts agrees. Ask which experts. With no source, nobody can check it.
3. A contrast frame that rebuts a claim nobody made, the "it is not merely X, it is Y" shape. Ask who said it was merely X. Usually nobody did, and the sentence is adding drama instead of a mechanism.
4. A generic conclusion that could end any paragraph about anything, usually an upbeat line about the future. Ask what forecast it commits to.

When you edit, highlight every instance. If you cannot attach something verifiable on the next pass, cut the sentence or rewrite it into a claim a reviewer could prove wrong.

How do you run a quick pass yourself?

Two passes on a technical post, in this order:

• Pattern pass: search for inflation openers, unnamed authorities, contrast frames, and closing lines that praise the future without a number or a boundary condition.
• Mechanism pass: for every claim left, write down what you would measure and what result would prove you wrong. If the answer is nothing, the sentence is marketing.

On a long post I would rather let a script find the pattern families first. This sketch is mine, and deliberately small; a maintained detector is the better tool for real use.

# Sketch: flag a few AI-writing tells in a markdown draft
import re
from pathlib import Path

text = Path("draft.md").read_text(encoding="utf-8")
rules = [
    (r"\b(experts agree|studies show|researchers say)\b", "vague attribution"),
    (r"\bnot (just|only|merely)\b[^.]*\bit'?s\b", "contrast frame"),
    (r"\bthe future (looks|is)\b", "generic conclusion"),
    (r"\b(watershed|sea change|in the evolution of)\b", "significance inflation"),
]
for pattern, label in rules:
    for m in re.finditer(pattern, text, re.I):
        line = text.count("\n", 0, m.start()) + 1
        print(f"{label} ~ line {line}: {m.group()[:60]}")

Treat the hits like linter warnings: review each one, because some will be false positives, and fix the real ones.

Why can't a clean result mean the post is true?

A detector only sees shape. Removing an unnamed expert does not add evidence. Replacing a claim of historic weight with a concrete before and after does, but only if the before and after are real.

That is why the writing page sits next to three episodes on the gap between plausible and true: Dan Klein on plausibility engines (episode 54), Sudhir Hasbe on hallucinations as a data architecture problem (episode 55), and Alex Ratner on the evaluation gap, where everyone says their system works and almost nobody can say what "works" means in a number (episode 57). Lint gets you to honest prose. Those conversations are about what honest prose is still allowed to claim.

If you want to see the patterns in a real draft, the essays include a piece where an AI agent's first draft runs with formatting fixes only, so you can read it for tells yourself. More about the show and how it is made is on the about page.

Checklist

• Highlight all four pattern families before you fact-check anything.
• Replace every unnamed authority with a named source or a metric, or cut the line.
• Run a quick regex or detector pass, then review every hit by hand.
• Keep "reads well" and "is true" as separate checks, and schedule the second one for every claim that matters.
• If the post describes AI product behavior, define what "works" means in a number before you publish.

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.