The Prompt Contract Test: Failing the Build When the Prompt and Its Fixtures Disagree

Last post: three weeks of decision-log tallies showed 55% of my rejected LLM outputs were my own prompt, not the model — and the pattern was drift, inputs moved while the prompt stood still. I promised the fix: a prompt contract test that fails the build when the prompt and its fixtures disagree. This is that test.

The idea in one sentence

A prompt contract test checks one thing: does every instruction in the prompt still have a fixture that exercises it, and does every fixture still match what the prompt expects? If not, the version bump fails before it reaches production. Drift can't accumulate silently because the gate refuses to ship it.

This is the mechanical version of the prompt-diff ritual from the tally post. The ritual asked me to read my own diff like a code review. The test asks the machine to do it, because my reading discipline has a three-week half-life and the test doesn't.

What counts as a disagreement

The contract lives in a small YAML file next to the prompt. Three checks, nothing more:

1. Orphaned instructions. Every clause in the prompt must map to at least one fixture. I once added "never repeat the customer's name back to them" to the prompt after a bad output, pushed the version, and the regression suite stayed green for a week. Of course it did — no fixture contained a customer name. The instruction was untested theater. Now an instruction with zero covering fixtures fails the build.

2. Orphaned fixtures. Every fixture must exercise at least one instruction clause. This catches the poisoned-fixture case from the tally post in the other direction: a fixture from an older ticket format still in the suite, matching nothing the current prompt expects. The gate was comparing real outputs against a ticket that no longer exists in production. Stale fixtures don't fail loudly — they pass everything and test nothing.

3. Output shape agreement. The prompt contains a worked example of the expected output (it has to — that's how the model knows the shape). The contract test parses that example and validates it against the schema the downstream n8n node enforces. Four fields in the example, five required in the schema: build fails. This one bit me exactly once, when I tightened the schema and updated the docs but not the example the model actually reads.

How it fails (the output matters)

A failing build doesn't just say "contract violation." It names the disagreement class, because each class has a different fix:

• Orphaned instruction: "never repeat the customer's name" → write the fixture, don't relax the prompt.
• Orphaned fixture: "ticket-0042 angry customer" → replace it with a real recent ticket or delete it.
• Shape mismatch: example has 4 keys, schema requires 5 → fix whichever one is wrong, which the test can't tell you and you have to decide.

The classification matters because the failure is a teaching moment for the workflow, not just a red X. Over a few weeks, the failure log reads like a diary of your drift patterns — which I now tally monthly next to the review-queue tags. The orphans cluster exactly where the prompt tags cluster. Same bug, two instruments.

Where it runs

It runs at one point and one point only: the version-bump gate, right next to the review-queue sweep from three posts back. Same rule as the tally: one gate, one job. The pre-publish script owns the mechanical contract, the review queue owns prose quality, the tally owns the direction of blame, and the contract test owns prompt-fixture agreement. Four layers now, each still with exactly one job.

It does not run on every edit, because prompts get edited constantly while you're thinking. It runs when you ask for a version number. That's the moment you declared "this is stable," and the test takes you at your word.

What the tally said after two weeks of the contract test

Small, honest data: two weeks, four version bumps, two contract failures.

• Failure one was an orphaned instruction — the "don't summarize, extract" clause I added after a bad rejection. Zero fixtures. The fix took eleven minutes: write two fixtures, one clean extraction, one deliberately messy source. The suite caught a real edge case the next day.
• Failure two was an orphaned fixture — a ticket sample in the old format that had been green for a month while testing nothing. Deleted in three minutes.

Two failures, fourteen minutes, one real edge case found. The test pays for itself by finding the fixtures that were never testing anything — the quiet rot that no dashboard shows.

Next in this series: the contract test keeps the prompt honest, but the model still gets a vote — what I do when the tally actually shows two consecutive weeks of model-tag growth. Spoiler: it's not what I used to do.

I'm building AI Automation Lab — n8n workflow templates for AI automation — and keeping a free lab on GitHub. I also send occasional notes on n8n reliability patterns: get them here.