We broke production with a one-word prompt edit and git could not tell us

The change was a single word. We softened "must" to "should" in a system prompt to make the tone friendlier. The agent stopped emitting one JSON field consistently, a downstream parser threw, and the incident review had no commit to point at - because the prompt did not live in the repo. It lived in a config row.

That is the uncomfortable truth about a lot of agent code: the behaviour is defined by a prompt, and the prompt is treated like copy, not like code. It has no version history, no diff, no test, and no rollback. You can refactor a service all afternoon with guards around it, then ship a one-line prompt edit with nothing watching. The blast radius is the same.

Why git-by-hand was not enough

My first fix was to put the prompt in a .txt file and commit it. That bought us history and a rollback path, and it was worth doing. But a line diff on a long prompt is hard to read - a reworded paragraph shows up as a wall of changed lines - and, more importantly, a text diff says nothing about behaviour. Two prompts can differ in one word and behave completely differently on the same input. The thing I needed to test was not the text; it was what came out. The dangerous edit is the one that shifts a format or a refusal behaviour just enough that nothing throws and no one looks.

So I kept a small fixed set of inputs that mattered to us, the ones whose output shape the rest of the system depends on. The check I wanted was simple: given the same inputs, does the edited prompt still produce the shape and constraints the old one did? That is a regression check, not a quality score. It does not say the new prompt is better, only that it did not silently break the cases we already rely on.

What I built

I built AgentPrompt to keep prompts under the same discipline as code. The declared features are version history, a diff view between any two versions, a regression check, and rollback. That is the whole set. It does not grade prompt quality, does not predict what a model vendor will change, and does not run your agent for you.

The diff view is built for reading a prompt, not a source file: it shows you what changed between two saved versions so a one-word edit is visible as a one-word edit. Version history and rollback are the pair that make an experiment reversible. If a prompt change passes the checks and still feels wrong in practice, rolling back should be a click, not a reconstruction from a chat log.

The regression check is the honest part. It covers the cases you give it. If you have not saved an input for a behaviour, the check has no opinion about that behaviour, and it should not pretend to. I would rather it report "no cases for this" than imply the prompt is safe.

The limits, stated plainly

A regression check over a fixed set of inputs cannot prove a prompt is correct, only that it has not broken the cases you recorded. Model behaviour can shift for reasons outside the prompt - a vendor update, a different temperature, a changed tool response - and this tool does not observe those. The diff is a text diff; it does not tell you two phrasings are semantically equivalent. None of this certifies a prompt or guarantees output.

I also want to be clear about scope. AgentPrompt manages the prompt artifact itself. It does not replace integration tests for the rest of your system, and it is not a monitoring layer for live traffic. It is the version-and-check discipline that a config-row prompt was missing.

If your agent's behaviour currently lives somewhere git cannot see, AgentPrompt gives you history, a readable diff, a regression check, and a way back. If a case you care about is not covered by your saved inputs, that is a gap in your set, not a false result.

FAQ

What does AgentPrompt do?

It keeps version history, shows a diff between any two versions, runs a regression check, and lets you roll back. You manage the prompt in AgentPrompt instead of a config row.

Does the regression check prove my prompt is correct?

No. It only covers the input cases you saved. It cannot detect a behaviour you have no case for, and it does not grade quality.

Will it catch a model vendor update breaking things?

Not by itself. It checks the prompt against your recorded cases; it does not observe vendor changes, temperature, or tool responses.

Is this a monitoring tool for live traffic?

No. It manages the prompt artifact - history, diff, checks, rollback. It does not watch production requests.