Your AI agent wrote the tests too

An agent builds a feature, writes the tests, and tells you they all passed. The thing I want to know is whether those tests checked what we actually asked it to build.

If the agent misunderstood a requirement, that misunderstanding can end up in both the code and the tests. They agree with each other, but we still have a problem.

I think this needs more attention as we hand more work to coding agents. I want to be able to look at a requirement and see what was checked, what the result was, and what we still don't know.

Take a CSV export

Let's say you're adding a CSV export to a product with multiple workspaces. Members should be able to export reports from their own workspace.

The agent builds the endpoint and adds a test that creates a member, creates a report, and checks the CSV response. That test passes, but we haven't checked what happens if the member requests a report from a different workspace.

Now make the export large enough that it needs a background job. You've got an endpoint to request it, another to check its status, and a way to download the file.

I'd want the agent to follow that whole flow. Checking permissions when someone requests an export doesn't tell us whether another user can download the result.

OWASP's authorization guidance calls for checking permissions on every request. In this example, that means looking at all the ways someone can get to the exported data.

The CSV can contain exactly the right columns and still be available to the wrong person.

What I'd want to see before calling it done

For this example, a review could look like this:

That last one is something I'd want worked out before the agent picks an implementation. If I remove someone from a workspace, should a download link they already have keep working? For how long?

Those answers affect what we build and how we test it. A passing test can't settle a product decision we haven't made.

This gives me enough information to decide what still needs work. A single “verified” label would hide most of it.

What about having another agent review it?

I'd want the reviewing agent to read the requirements and check the work against them. If we only give it the first agent's summary, we're asking it to find problems using an explanation that may have already left those problems out.

Running the tests in CI is useful too. We get a recorded run in a particular environment, but an authorization case we never tested is still missing.

I'd give the reviewer a prompt like this:

Go through the requirements and show me what was checked for each one.

Read the actual tests and results. Explain what they cover
and which important cases are still missing.

If a requirement is unclear, explain the situation we need
to decide on. Don't choose an interpretation and then
report that the requirement passed.

Include the commit and environment each result applies to.

I wouldn't do the same amount of review for a wording change as I would for a feature that exports customer data. But for the latter, I want to know that someone checked the access rules across the whole flow.

How are you doing this with your agents? Can you get from a requirement to the result of a check, or are you reconstructing that while reviewing the PR?