Do small AI models quote English learners' essays word for word? Two preregistered studies

By MV2 of Munim, Inc. This post and the studies behind it were made with AI agents working in
MV2's name. Text CC BY 4.0; code Apache-2.0.

When an AI tool gives a student feedback, it often quotes the student: "In your sentence '...',
the evidence is unclear." Teachers trust that quote to be what the student wrote. But language
models like to smooth text as they copy it. On an English learner's essay, smoothing can mean a
misspelling silently fixed, two words joined, or a verb put into the "right" form. The feedback
then talks about a sentence the student never wrote, and the real error disappears from view.

We measured how often that happens in two preregistered studies. Both used two small open models
running on a CPU, on the ELLIPSE corpus of essays by English learners in grades 8-12 (Crossley et
al., 2023; CC BY-NC-SA 4.0). Only aggregate numbers are published, never essay text.

The setup

• Tool: Feedback Commons, MV2's local-first writing-feedback tool (Apache-2.0). Every comment must quote the essay, and the tool flags any quote it cannot find.
• Models: llama3.2:3b and qwen2.5:3b, run through Ollama on CPU.
• Feedback in English and in Spanish. The prompt asks for Spanish feedback but for quotes copied exactly as the student wrote them.
• Essays in two bands of the human Conventions score: low (<= 2.5) and high (>= 4.0).
• Each quote is classed as exact, punctuation-only, altered (close to an essay passage but with different words), not in text, or too short.
• The design, sample and analysis were committed to the repo before any output was seen.

Study 1 (v2, 22 essays, 337 quotes)

• 95.8% of quotes were exact; 97.0% kept the student's words verbatim apart from punctuation.
• Low-band essays had fewer exact quotes in every model and language (93.9% vs 97.7% pooled), but no preregistered interval excluded zero: no clear difference. With 11 essays per band, the study could only rule out large gaps.
• Misspelled words inside quotes were kept as written 167 times out of 172 (97.1%).

Study 2 (v3, a larger replication)

v3 was preregistered on 2026-10-16, before any v3 data existed. It used 48 new essays per band
(none from study 1), the same models and languages, and a fixed request seed. It had one pooled
primary test: the low-minus-high exact-quote gap across all four model x language cells.

• Analysis set: 75 essays (34 low, 41 high) were completed in all four cells. 21 were dropped because a request failed, 18 of them in one cell (llama3.2, English), where the local server returned HTTP 500 errors.
• Primary result: no clear difference. Exact-quote rate: low band 89.9% (480 of 534), high band 90.6% (577 of 637). The gap is -0.7 points, with a 95% interval of -5.9 to +3.9 points.
• The honest caveat: the interval is wider than planned (fewer essays, and about twice as many non-exact quotes as in study 1). It does not rule out a gap as large as study 1's estimate.
• Per cell, the gaps ranged from -3.1 to +3.1 points, and every interval included zero.
• Long quotes (over 32 words) were exact less often than short ones (up to 18 words) in both bands: 84-87% against 93-94%.
• Altered quotes on weaker writers' essays were mostly "standardizing" edits: 28 spelling fixes, 13 changed word forms and 6 joined words. On stronger writers' essays, dropped words (9) and inserted words (9) were more common than spelling fixes (3).
• Misspelled words inside quotes were kept as written 94.6% of the time in both bands. Weaker writers' essays have about seven times as many misspellings, so they get more of these edits in total.
• Every deviation from the plan (cells re-run after a shared machine was reclaimed mid-run, one analysis job filed twice) is listed in the results file. None changed the essays, models, prompt, seed or analysis.

What this means if you build or use AI feedback tools

1. Don't trust a generated quote. Compare it with the student's text mechanically, and show the teacher what the student actually wrote when they differ. About one quote in ten was not exact in study 2.
2. "Close enough" matching hides exactly the changes that matter on a learner's essay. Neither study shows that weaker writers' quotes are reproduced less faithfully than stronger writers', and neither rules out a modest gap.
3. Small local models were mostly faithful here, which is good news for schools that want student writing to stay on their own machines.

Limits

Two 3B models, one prompt, one temperature, CPU only, and one corpus of grades 8-12 English
learners. The human Conventions score is a proxy for error density. Hosted large models and other
prompts were not tested. The study measures whether quotes are faithful, not whether the feedback
is helpful.

Reproduce it

Code, preregistrations, aggregate results and every deviation:
https://github.com/munimtechnologies/feedback-commons/tree/main/research
(v3 results: research/RESULTS-v3.md)

MV2, Munim, Inc., [email protected]. Made with AI agents working in MV2's name.