I planted typos to catch AI models editing quotes. 0 of 962 got "fixed" the real failure is sentence boundaries

`
This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

There's a piece of folk wisdom about language models that I believed until yesterday: if you ask one to quote text verbatim, it will quietly clean the text up. Fix your typos. Americanize your spellings. Repair your grammar. The RLHF polish instinct, overriding the instruction to copy.

If that's true, it matters. A journalist who gets back a "quote" with a silently corrected word has a correction to print, maybe a retraction. A lawyer misquoting a contract clause is in sanctions territory. A developer piping "cite the exact line from the log" into an incident report is lying about their own system without knowing it.

There's a famous number floating around this topic: the Tow Center at Columbia found 153 of 200 publisher quotes misrepresented by ChatGPT. But that study was about retrieval: the model had to find the text first. I wanted the harder, cleaner question: what happens when the text is right there in the context window and the only job is to copy one sentence?

So I set a trap. And I'm writing this up honestly: the trap caught nobody. What I found instead is, I think, more interesting.

The benchmark

I wrote 10 short original documents (200 to 250 words each): an internal memo, release notes, an incident postmortem, hosting terms, a travel letter, meeting minutes, a hardware spec, an email thread, a README, and a press release. Ordinary workplace prose, written fresh for this benchmark, so there's no copyright issue and no training-data contamination.

Into them I planted 18 irregularities, on a temptation gradient:

Each dirty sentence is paired with clean control sentences from the same documents: 37 items, 18 dirty, 19 clean. Every item asks one thing:

Quote verbatim the sentence that [uniquely identifies one sentence]. Output only that sentence, exactly as it appears in the document, and nothing else.

Each item runs twice: once with the plain instruction above, and once with an explicit warning bolted on: "Reproduce it character-for-character, preserving any typos, spelling choices, grammatical errors, or unusual wording exactly as written - do not correct, complete, or clean up anything." If models fix text under the plain instruction but stop when warned, that's an instruction-following gap with a one-line mitigation. If they fix text even when warned, the cleanup happens somewhere prompts don't reach.

Scoring is deterministic. No LLM judge anywhere. A quote is faithful if it matches the planted sentence exactly (I track raw byte-equality and a conservative normalized match that only forgives curly quotes, unicode dashes, and whitespace). Everything else is classified: corrected (the planted error got fixed), truncated (a partial quote), wrong_span (real text, wrong boundaries), or rewritten (text that exists nowhere in the document). The scorer is about 60 lines of Python with unit tests, and you can audit any single verdict with Ctrl+F.

My Benchmark

https://www.kaggle.com/benchmarks/saharshtibrewala/exact-words

Everything is public in the task notebook: all 10 documents, all 37 items, the scorer, and every one of the 962 model responses.

The Models I Tested and Why

13 models, 74 prompts each, all run on Kaggle Benchmarks: open-weight models because it's Hacktoberfest and self-hosters deserve to know (Gemma 4 31B and 26B-A4B, GLM-5, Qwen3-Next-80B, DeepSeek-R1, GPT-OSS-20B and 120B), and closed models as reference points (Claude Haiku 5.5 and Sonnet 5.5, GPT-5.4-mini and GPT-6-sol, Gemini 3.5 Flash-Lite and 3.7 Flash). I deliberately included small/large pairs within three families, because my prior was that more polish means more silent correction, and family pairs isolate that.

What I Found (And What Surprised Me)

Finding 1: the trap caught nobody. 0 silent corrections in 962 quotes.

My headline hypothesis is dead, and I have receipts. Across 13 models, two instruction arms, and 962 scored responses, not one model fixed a planted error inside a quote. "Recieve" survived 26 encounters. So did "more simpler", "This library have", the reversed DNS definition, and "the Pacific, the world's second-largest ocean", which I was certain someone would quietly promote back to first place.

It's stronger than that. Of 911 faithful quotes, every single one was byte-identical to the source. Not one needed even my gentle normalization. When a 2026 model copies, it copies exactly.

And the twelve responses that produced text existing nowhere in the document? Five of them contained a planted error, and all five preserved it. Even while mangling a quote's boundaries, no model touched the errors. Whatever typo-fixing machinery lives inside these models (there's published mechanistic work on dedicated typo-correction structure in LLMs), the quoting context switches it off. Folk wisdom: falsified, at least for short documents in 2026.

Finding 2: what actually fails is the sentence boundary.

51 of 962 quotes failed, and the taxonomy is lopsided: 26 truncated, 13 wrong-span, 12 rewritten, 0 corrected. Nearly every failure is a disagreement about where a sentence starts and ends:

• The colon problem. The hardest item in the whole benchmark (13 failures out of 26 attempts) was: "But a part of it is our own doing: customers who contact us twice about the same order currently recieve two separate case numbers, and about a tenth of those end up double-refunded." Models split on the colon. Gemini 3.5 Flash-Lite started the "sentence" at "customers" and capitalized it to "Customers", producing a sentence that exists nowhere in the memo (typo intact!). Claude Sonnet 5.5 went the other way and glued on the previous sentence. GPT-OSS-120B stopped mid-clause. One sentence, three different theories of its edges.
• The neighbor-sentence problem. Asked for "the final sentence of the letter", models failed 9 times out of 26: Sonnet 5.5 and Qwen3 returned the final two sentences ("Some trips end with a landmark. This one ended with..."), and Flash-Lite once returned only "Some trips end with a landmark.", which is the second-to-last sentence, alone. The instruction said one sentence; they knew where the text was and disagreed about which one it was.
• Section labels. GPT-OSS-20B prepended "4.3" (and elsewhere "4.2") to terms-of-service quotes, faithfully copying labels that aren't part of the sentence.
• Fabricated punctuation. DeepSeek-R1 cut a long sentence after its first clause and appended a period that isn't in the source, with the reasoning trace visibly talking itself into it: "The period after 'water' is part of the original sentence structure." It isn't; the source continues with a comma. The one true paraphrase of the whole run also lives here: Flash-Lite, in the strict arm, restructured "The extended battery delivers a full two-shift day, defined as..." into "A full two-shift day is defined as..." — the only failure in 962 that reads like the folk-wisdom rewrite, and it's 1 case, not a pattern.

Finding 3: telling it not to correct does nothing (because there's nothing to prevent).

Plain arm: 95.0% faithful. Strict "do not correct anything" arm: 94.4%. A null result, but a clean one: the explicit warning neither rescued the boundary failures nor prevented corrections that were never going to happen. If you've been adding "preserve all typos exactly" to your prompts, on this evidence it's a lucky charm.

Finding 4: bigger is worse, twice out of three.

• GPT-OSS-120B: 82.4%. GPT-OSS-20B: 89.2%. The 6x larger sibling is 7 points worse at copying, mostly through truncation. Both siblings share a family tic: swapping plain hyphens for U+2011 non-breaking hyphens inside otherwise perfect quotes ("day‑old", "hot‑swappable"), a single invisible character that makes the quote no longer exist in the source.
• Claude Sonnet 5.5: 93.2%. Claude Haiku 5.5: 94.6%. The smaller model wins again.
• Gemma 4 31B and 26B-A4B: both a perfect 100%, every byte, both arms, alongside Gemini 3.7 Flash, GPT-6-sol, and GLM-5.

Five models walked through my trap untouched, and two of them are open-weight models you can run yourself. The scaling intuition ("the big model is better, use it for precision work") points exactly the wrong way on this task.

The grader is always the first suspect

Two confessions, because this genre earns its keep through self-audit.

First: my initial leaderboard showed DeepSeek-R1 at 0.0%, with 100% of its quotes "not in the source". Alarming headline, completely false. R1 wraps its answers in <think>...</think> reasoning, my scorer was grading the thinking instead of the answer, and the real answer was sitting right after the closing tag. One line of stripping later, R1 scores 91.9%. If I had shipped the first number, the first commenter who knows R1 would have ended this post. (Its remaining failures are real: truncating clauses and inventing terminal periods, reasoning aloud about punctuation the whole time.)

Second: this isn't even the benchmark I started building yesterday. The first one asked models to compute cron firing times across daylight-saving transitions. I killed it after discovering that cron implementations themselves disagree about DST semantics (Vixie cron skips the nonexistent hour, Quartz handles it differently), which means my "ground truth" would have been one dialect's opinion. And while building that grader I inverted PEP 495 fold semantics and spent twenty minutes confidently marking correct answers wrong. Both experiences set this benchmark's design rule: the correct answer must be a string I planted myself, findable with Ctrl+F, with no dialects and no judge model.

Limitations (read before quoting my numbers)

• 37 items, 13 models. Big enough that 0/962 corrections is a real signal; small enough that I hand-verified every item. Per-class splits (4 typos, 3 wrong facts) are directional.
• Short documents (~250 words). Copy fidelity at 200 words deep tells you nothing about 50,000 tokens deep. I built a combined-document variant (same questions, all ten documents fused into one 2,225-word distractor field) that has not run yet; it is my first follow-up.
• Run-to-run variance is real. I ran the full suite more than once at provider-default sampling; per-model scores wobble by 1–4 points between runs, while the headline findings held every time: zero corrections, the same five models at 100%, byte-identical faithful quotes. Numbers in this post are from one complete 962-call run; per-item CSVs are public in the notebook output.
• English workplace prose only. Code, poetry, non-Latin scripts: open questions.
• Exact-match is strict on purpose. Several failures differ from the source by a single character, like a non-breaking hyphen or an invented terminal period. I publish the full taxonomy so you can draw the line elsewhere; the CSVs have every response. A handful of "rewritten" verdicts are models adding a lead-in like "The sentence is:" despite "output only that sentence", which I count as failures of the task as instructed.
• My sentence boundaries follow period-to-period convention; the colon item shows models hold other theories. I'd call that a finding, but you're free to call it item ambiguity, and the item-level data lets you recompute without it.

What I'd Measure Next

• Depth stress: the combined-document variant above, then real long-context distances.
• Code: quote the exact line from a stack trace or diff, where an invisible "fix" is maximally dangerous.
• The colon census: a benchmark made entirely of sentences with mid-sentence colons, semicolons, and conjunctions, since that's where models' sentence-boundary theories diverge.
• Trailing-punctuation fidelity as its own tracked class; it may explain a chunk of "near-miss" exact-match failures in other benchmarks.
• Reasoning-model output hygiene: R1's failures all happened inside otherwise-correct answers; how much benchmark noise across the industry is really answer-extraction noise?

Honesty section

The documents, planted errors, benchmark design, and analysis are mine; I used AI coding tools (Claude Code) to accelerate the harness, scorer tests, and charts, as the challenge rules permit. Every item was hand-verified against its document, and every number in this post comes from the public run on Kaggle. The project was built during the contest window.

Benchmark: https://www.kaggle.com/benchmarks/saharshtibrewala/exact-words. All code, documents, and per-item results are public. If you find a scoring bug, comment; as established above, the grader is always the first suspect.
`