Testing Email Rewrite Fidelity: Literal Anchors vs. Semantic Inversion
When a rewrite changes "I cannot take on the extra 6 reports this month" into "I can take on the extra 6 reports this month," every name, number, and date survives. The checker sees all anchors present. The refusal has b
When a rewrite changes "I cannot take on the extra 6 reports this month" into "I can take on the extra 6 reports this month," every name, number, and date survives. The checker sees all anchors present. The refusal has become a commitment.
That gap between literal token retention and semantic fidelity is the whole reason this fixture exists.
The fixture
Download cases.json from hemmingway.io/downloads/email-rewrite-fidelity/cases.json. It contains six synthetic English emails, each with:
id-
source(the original message) -
anchors(named literal strings that must survive) -
manual_invariants(conditions, negations, actor assignments, and tentativeness a reviewer must verify by hand)
These are not customer emails. No model has been run against them. They exist to teach the checking method.
One concrete case: refusal and alternative
Take refusal_and_alternative:
Hi Sam, I cannot take on the extra 6 reports this month. I can review 2 outlines next month, but I have not agreed to write them. Would that help? Thanks, Arjun.
Anchors: Sam, 6, 2, Arjun.
Here is the deliberately corrupted offline checker control, not an AI output:
Hi Sam, I can take on the extra 6 reports this month. I can review 2 outlines next month, but I have agreed to write them. Would that help? Thanks, Arjun.
Construction: source.replace('cannot take','can take').replace('have not agreed','have agreed').
Every anchor survives. The checker returns needs_manual_semantic_review. But two obligations flipped:
- The refusal ("cannot take on") became a commitment ("can take on").
- The non-agreement ("have not agreed") became an agreement ("have agreed").
This is a refusal/negation/commitment inversion, not a reversal of shipment conditionality. The literal tokens are identical except for the negation words.
The checker
check_fidelity.py is a Python standard-library script. It reads an output JSON file (a flat object mapping each case ID to the rewritten string) and checks whether every anchor appears as a whole token in the normalized output text.
missing = [anchor for anchor in case['anchors']
if not re.search(r'(?<!\w)' + re.escape(normalized(anchor)) + r'(?!\w)', text)]
Statuses:
-
missing_outputβ the output field is empty or absent. -
literal_check_failedβ one or more anchors are absent. -
needs_manual_semantic_reviewβ all anchors present. This is not a pass.
Exit code: 0 means every case reached needs_manual_semantic_review. Exit 1 means at least one case failed a literal check or has no output.
Running it
Save all three files in one directory first:
cases.jsoncheck_fidelity.pyoutputs.example.json
Then:
cp outputs.example.json my_outputs.json
# edit my_outputs.json with your rewritten strings
python3 check_fidelity.py my_outputs.json
The checker prints a JSON object with a notice field and per-case results. The notice reads: "Literal survival is not semantic correctness. No automatic model pass is assigned."
The tested semantic control
The exact tested control JSON is available separately at tested-semantic-control.json. It is not one of the six source cases, not a model-generated rewrite, and not an added benchmark result. It documents the refusal_and_alternative inversion shown above, with construction, observed status, and the explicit note that model_generated is false.
Why human review is still required
The five control checks validate the checker's logic, not any model. No model request has been sent for this dataset. No accuracy percentage, ranking, or zero-error claim can be derived from these fixtures.
What the manual rubric catches:
- Conditions flipped to commitments
- Negation dropped ("cannot" becomes "can"; "not agreed" becomes "agreed")
- Actor changes
- Certainty inflation
- Attachment claims
Adapting the fixture
The structure is portable. Replace the six sources with your own authorized synthetic or nonconfidential messages. Keep the same JSON shape: id, source, anchors, manual_invariants. The checker will work unchanged. Freeze your inputs and settings before running any evaluation. Preserve every output, including failures. Do not cherry-pick the best rewrites for a reported metric.
Limitations
- Six English synthetic cases. Not representative of all real email behavior.
- Literal checks are not semantic correctness.
- No model benchmark has run. No consumer tool was tested.
- No automatic sending is involved.
- No legal or security guarantee.
Technical resources
Disclosures
The publisher is Hemmingway, a writing-model company. This article was drafted by Hemmingway-1 on Medium with editorial review and local checker validation. No independent research or peer review was performed. The editorial drafting process is distinct from API or consumer model evaluation; nothing in this article tests the consumer rewriter or any model's output.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.