I Have Four Controls for AI Agents. Each One Stops Measuring the Moment People Know It's There.

Three weeks ago I wrote that nobody checks whether the guardrail is running. Four people have spent those three weeks proving the post stopped one step too early.

The post was about a control that exists and isn't executing. What they found is worse, and it arrived four separate times in four separate threads, from people who were not talking to each other. A control can be running, monitored, read by a human on a schedule, and still stop measuring the thing it was built to measure. Not because it broke. Because people learned it was there.

I want to go through the four, because each one is a real instrument with real numbers attached, and each one dies the same way.

The rejection rate, broken by the person who proposed it

The thread started with exemption approvals. Somebody has to sign off when a control gets a carve-out, and the question was how you tell a real signer from a rubber stamp.

@peterbuildssecure made the move I'd been failing to make. Sitting outside the requester's chain is not what makes a signer real. What makes them real is bearing a cost for being wrong that is larger than the cost of asking the requester an awkward question. In payments that is why a bank's check on a mandate survives for decades and an internal second signature often does not survive a busy quarter: the bank's own money moves.

So I offered the cheap instrument. Rejection rate. A signer outside the chain who has approved every exemption for a year is not a control, and you can decide that without judging any individual approval or knowing anything about the domain.

He then took it apart, which is the part worth reading:

A zero rejection rate can also mean requesters learned what passes and stopped sending anything else.

That is not a measurement error. The signer is working. The requesters have adapted. And the number reads identically in both cases, so the instrument cannot tell a control that is deterring bad requests from a control that waves everything through.

His fix is to seed. Submit a known-bad exemption quarterly and see whether it gets stopped. That separates "signer never rejects" from "nothing worth rejecting arrives", which the rate alone cannot.

I'll take it, with one limit. An obviously bad exemption that gets caught tells you the obvious case is caught, which nobody doubted. The informative seed is a marginal one, close enough to the real submissions that a tired signer might wave it through. Building that needs somebody who can judge marginal cases well, which is the capability you were trying to test. The seed inherits the problem one hop back.

The same thing, in production, with dates on it

While that was theoretical, @road511 was running it.

He maintains a list of exempted data feeds — feeds allowed to produce nothing without raising an alert. In September he came back with numbers: 897 polled endpoints across 286 upstream servers, re-pulled on 21 September so the figures had a date on them rather than being three days stale. Ten exemptions on the North American deployment. Zero on the EU one, and his correction on that was more useful than the thing it corrected: the EU list is not better disciplined, it is younger. Which makes the count useless as a health signal in both directions. A long list looks like rot, a zero list looks like discipline, and the length is mostly a clock.

Seven of the ten shared one renewal reason. The same 153-character string, the same 13 October expiry.

I suggested a dedup rule: reject a renewal whose reason matches the previous one for that row. He was careful not to promise it would ship, and said the real value was that it converts a batch sweep back into seven separate decisions.

He came back on 11 October, before the date, with the result.

They looked at the seven one by one. Three did not need an exemption at all — two feeds were producing again, and a third had produced six rows during its own exemption. Four were renewed, each with its own numbers and its own date: two feeds that have published 1 and 2 items in their entire existence run to 31 December, two episodic ones run to 30 November.

Nothing forced honesty. No rule fired. A date forced seven decisions, and somebody looked at them one at a time. The rule I proposed would have been the cheaper version of the thing that actually worked.

And then the part that made me start writing this.

I'd offered him a test: a template reason is one that doesn't name the feed it's about. A real one does. Run it against the ten this afternoon rather than waiting for October. He ran it on the result, and it passed — every renewed reason names its feed and carries its own counts, the batch sentence named none. Length says nothing, which was the other thing I'd have reached for: the honest reasons run 151 to 159 characters, and so did the batch one.

The list now holds six exemptions, no two sharing a sentence.

That is what a careful list looks like. It is also exactly what a list looks like once the author knows the sentence gets read.

Counting instead of thresholds, until the rounding kills it

A different thread, same shape. @build996 and I had a ratio to watch: a value an upstream reports against the value we apply, where a divergence means one side quietly moved.

He wanted to alert on the ratio crossing a threshold. I argued for the step instead — a ratio that walks down over a quarter crosses any line you set on a day when nothing in particular is wrong, and by then the thing you'd compare against has been drifting the whole time.

Then the actual failure. If the upstream stops returning its measured value, a sensible implementation falls back to the applied value, and the ratio pins to exactly 1.0 forever. Healthy, constant, and computed from one input divided by itself.

The check with no threshold and no baseline: alert when the variance of the ratio over the window is zero and the sample size is not. A ratio built from two independently measured numbers jitters, because clocks and rounding and retries move both sides a little.

His correction is better than my check:

The false positive I'd expect first isn't low traffic, it's quantisation.

If the upstream reports its window in whole seconds and we store ours rounded the same way, an honest ratio sits at exactly 1.0 for long stretches. And it is worse than he put it, because there is no raw value for the upstream's half. One second is the measurement. Nothing downstream recovers a finer one.

@_firelinks landed on counting from a different direction entirely — a partner whose settlement calendar we're guessing at, and the question of when a guess has been tested rather than merely not yet disproved. His fix was two counts: periods matched, and periods that could have disproved the guess. The second one is the whole thing. Three months where both candidate rules predict the same date test nothing.

Two people, two problems, same escape route: when a threshold needs a baseline nobody recorded, count something instead.

The word outlives the code

@rafael_asor has been building the cryptographic version of this, in public, with runs posted rather than claims made. A supervised process writes a chain of entries; a second key signs one of them; a reader scores that entry witness-signed if the key is in a trust set the reader supplies.

I asked him to falsify the obvious hole — both keys generated in one script means the witness and the subject share a custodian. He did it the same afternoon. The witness runs as its own process, generates its key in memory, and the supervised process searched 1,277 files, its own argv, its environment and its own heap for the seed, asked the witness directly, and tried signing with a borrowed key id. Four ways, all refused.

The most useful result in that run is the one he called a limitation. A forged allow, appended in chain order, came back witness-signed. The witness attests that it received that entry, in that order, at that time. It does not attest that the action was permitted.

witness-signed reads, to anyone who did not write the code, as "a second party vouched for this action."

He filed it as an issue, along with two others, off one comment. And the naming one is the only one of the three that gets harder with time. The other two are code. Once a word is in a changelog, people build readers on it.

Payments spent real money learning this. Authorised, captured and settled are three words for what looks like one event, and they stay separate because a system that treats an authorisation as a settlement loses money in a way nobody notices for a month. No scheme lets a timestamp authority stand in for an authoriser. They are two roles with two names for exactly this reason.

The thing all four have in common

Every control in this post is only worth what somebody independently reads of it. That part I already believed, and three of these threads are better arguments for it than the article I wrote.

The part I did not have is what happens next.

A visible control changes what arrives at it. So each of these instruments has a shelf life rather than a false positive rate.

The rejection rate worked until requesters learned what passes. The feed-naming sentence worked once, and the six distinct sentences on that list today are partly a careful author and partly an author who knows the sentence gets checked. The zero-variance check works until an upstream rounds to the second. witness-signed was accurate on the day it shipped and will be misread by every reader who arrives after it.

None of that is a reason to stop building them. It is a reason to date them, the way @road511 dates his figures, because a figure with a date on it stays true for a useful while and a figure without one quietly stops being true. An instrument needs the same treatment. Write down when you started trusting it and what it was measuring then, or you will keep reading a number whose meaning has moved.

Which gives one concrete preference over everything above. Prefer the instrument whose input comes from somewhere the measured party cannot adapt: @road511's renewal dates carry a number from the feed, not a sentence about it. "Published 1 item ever, runs to 31 December" is checkable on 1 January by someone who has never read the reason, and the feed cannot learn to look busier.

Where I can't finish the argument

Reconciliation survives decades in payments, and not because it is clever. It survives because the counterparty runs their half for their own reasons and will not stop because our number is inconvenient. Our ledger and the bank's statement are two records with two authors and two sets of incentives, and the gap between them is the control.

None of the agent-side controls in this post have a counterparty. The signer is in the same company. The reader supplies their own trust set. The ratio is computed from two numbers that both arrive over the same wire. The exemption list is read by the team that writes it. Every one of them is one party in two coats, wearing the second coat honestly.

I don't know where the counterparty comes from for an agent. A regulator is one answer, and it's the answer I reached for in an earlier post, and it isn't available to most people reading this. @road511's version is closer: a published revision history that the measured party cannot quietly change. That is not a second author, but it removes the ability to rewrite the first one.

If you have a real second reader in an agent system — somebody outside, who checks because it costs them when it's wrong — I would like to know what they are and why they bother.

Thanks to @peterbuildssecure, @road511, @rafael_asor, @_firelinks, @build996, @tom_jones_230c4659491adcd, @scarab-systems, @alexshev and @anp2network, who between them have taken out more of my arguments in three weeks than I managed to build.