Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 8 min read

Every test passed and the agent was still wrong

The example is a fictional health plan's prior-authorization intake. A request comes in for an MRI, a CT scan or a spinal injection. The suite reads the request and the attached notes, finds the plan's policy for that se

The example is a fictional health plan's prior-authorization intake. A request comes in for an MRI, a CT scan or a spinal injection. The suite reads the request and the attached notes, finds the plan's policy for that service, checks each criterion against the notes and writes a packet for a nurse reviewer. It decides nothing. A person makes every decision, and the packet only has to be right about what the notes say.

I built it as the fifth worked example in a skill pack for coding assistants, and I tested it the way I'd been testing the others. Then I handed it to a reviewer that had written none of it, and the reviewer found three serious defects in a build where every test was green.

This post is about why that happened, what caught the defects in the end, and what I now think each kind of test can and can't show.

The test that said everything was fine

The pack has a test I call the removal run. Every safety rule the plan marks as met is written as a clause in the code. One example: "a rule set can't run until a named person approves it". Another: "provider text never reaches the packet". The removal run deletes each clause in turn and runs the test suite. If no test fails, the clause isn't really protected: you could delete it and nobody would notice.

On the prior-authorization build, the removal run deleted 36 clauses, one at a time. A test failed 36 times out of 36. Perfect score. By that measure, every control in the plan was held in place by a test.

What the reviewer found

The reviewer reported twenty findings. Three were high. Here they are.

First, the suite chose the service by word overlap. A request for "CT lumbar spine" shared enough words with the MRI policy that the suite applied the MRI policy to it, and a request for a lumbar spine injection got the same policy. In both cases the packet said the criteria were met. Wrong policy, confident packet.

Second, the static check that runs before a rule set is released skipped any market or month that had no rule at all. That is exactly the gap the check was supposed to catch: a market or a month where a service has no rule.

Third, the provider's free-text description of the service, and the sender's labels on attached documents, reached the packet, the intake record and the log. Anything typed into those fields, a patient's name included, went wherever the packet went.

All three were in the build. The removal run still passed 36 of 36. The run passed before the fixes and after them.

Why the green run meant less than it looked

The removal run answers one question: if this clause disappears, does some test notice? It doesn't ask whether the tests send the inputs that would break the clause in the first place.

The word-overlap defect is the clearest case. There was a clause that matched the service to a policy, and there were tests for it. Delete the clause and the tests failed. Nothing matched any more. But every test used a request whose wording matched its own policy. No test sent a CT request worded close to the MRI policy. The clause was protected against removal and wrong for a whole class of inputs at the same time.

This is the known limit of mutation testing, the older idea the removal run borrows from. A mutation score tells you how well the tests detect the changes you chose to make. It says nothing about inputs nobody wrote.

The tester that only changes one thing

What I added after that review is a tester whose whole job is near misses. I call it tester h in the pack. It reads the plan, the rule files and the function signatures. It never reads the existing tests. Then it writes twins of passing cases that differ in one field: a service code that is nearly right, a negation inside a clause, a name where a code belongs, a date one day past the last rule period.

Each twin asks one question. If a field changes in a way that should change the outcome, the outcome has to change, and if it changes in a way that shouldn't matter, the outcome has to stay where it was.

On the prior-authorization build it found three more defects, two of them high. A diagnosis code field printed whatever was typed into it, so a name typed there reached the packet. An unrecognized service code was printed the same way. The third was a negated claim: a note saying a treatment "is not supported after six weeks" was reported as confirming the six weeks.

The blind run

The obvious objection is that I wrote the near-miss tester's brief after seeing the review, so of course it found that kind of defect. To check that, I ran it blind: a fresh copy of the tester, with no access to the review, against a copy of the build with ten known defects put back in.

It caught seven of the ten, including all six high ones, and then it found eight more problems that nobody, me included, had listed anywhere. Four were high. A case ID was used as a folder name without checking, so an ID like "../../ESCAPE" wrote outside the cases folder. Placeholder names such as "TBD", "n/a" and "Unknown", and a role written out as "Medical Policy Owner", passed as the named approver on an override and on a rule-set release. The run-time gate checked only the release record's hash, so a record with no approver let cases run. And a negation inside a clause wasn't bound to what followed it: "and no foot drop" read as a red flag, and "did not complete 8 weeks" read as 8 weeks.

Every one of those now has a test that failed before the fix and passes after it. I keep the defects themselves as patches that put each one back, so I can check that a later version of the tester still catches them.

Across the other examples

I then ran the near-miss tester on the pack's other four worked examples, which had been through the same removal runs and review rounds. The task digest had 6 defects. The customs filing had 7. The insurance claims suite had 9. With the 3 and 8 in prior authorization, that's 33 defects across five example systems, each fixed with a test.

Some were small. Others matter a lot in a regulated workflow. In the customs filing, a job title was accepted as the approver's name. In the insurance claims suite, reworded prompt-injection lines got past the filter, and local phone numbers were stored.

None of these showed up as a failing test before the tester wrote its twins. Not one.

What each kind of test can and can't show

Here is how I read the three checks now, after five examples, two review rounds and one blind run that turned up eight problems nobody had written down.

A unit test shows the code does what its author expected on the inputs its author thought of. It can't show anything about inputs the author didn't think of, and on a rules system that is most of the risk.

The removal run shows each safety clause is held in place by at least one test. It can't show those tests reach the inputs that break the clause. A 100 percent removal score with three high defects standing is not a contradiction. It's what that score measures.

The near-miss tester shows how the decision behaves next to the cases you already have. It can't show anything far from them, and it is only as good as the fields it chooses to vary. It also isn't independent in any strong sense: the same model family wrote the build, the reviewer and the tester.

What changed in practice is the order. I now run the near-miss tester before I trust a green removal run. And a release report built on labels the builder wrote, or on a scorer that shares code with the step it measures, says self-consistent. It no longer says pass.

Where the approval check has to live

One more lesson. It's about the human in the loop. In every one of these examples a person approves before anything is paid, sent or released. Several of the defects above were not in the model at all. They were in the plumbing around the approval: a placeholder accepted as a name, a release record with no approver, a job title in the approver field.

A prompt can tell a model to wait for approval. It can't stop code from accepting "TBD" as an approver. Only code can. So the check belongs in code: an approval is a record with a real name and a date that isn't in the future, the gate refuses anything else, and each approval works once. In the insurance claims example a payment with no valid approval never reaches the payment system, and tests check that. Those tests passed long before the near-miss tester found the job-title gap in the customs example, which is the whole point of this post.

What this doesn't show

Every example here runs on fictional data. No run against a real health plan, insurer or customs broker is on record. One person built them, with one model family doing the building, reviewing and testing, so a defect that model family tends not to see may still be in there. The counts are small. And I wrote the near-miss tester's brief after the review that inspired it. So the blind run is the result to weigh.

The prior-authorization README says it directly: those defects are fixed and tested, and others like them may remain.

If you build agents that act on rules

Four things I'd do differently from the start:

  • Write near-miss cases for every key, threshold, scope field and selector the decision reads, one field at a time.
  • Have someone who didn't write the tests write them, from the plan and the rules rather than from the existing tests.
  • Treat a passing removal or mutation run as evidence that the controls exist. Whether they work is a separate question.
  • Put the approval check in code. Test it with bad approvers as well as missing ones: a placeholder, a job title, a future date.

I build and sell Agent Suite Builder, a skill pack for coding assistants that includes these five worked examples and the removal run and near-miss tester described here. The docs are free to read: design-upstrategy-az-llc.github.io/agent-suite-builder

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.