Write the spec, then keep an independent human review of the change

When an agent writes most of the code, your job is to turn a messy problem into a precise, repeatable solution in language, then keep an independent human review of the change. Laurie Voss said the best AI code review tools he has seen have like an 80% hit rate, and that this is not enough to put software into production. Write the spec, derive acceptance cases from it, let the agent implement, and have a person review the change plus anything that was never written down.

Laurie Voss, head of developer relations at Arize and a co-founder of npm, made that case on Chain of Thought in September 2026.

How do you write a spec an agent can execute?

Arrive at the precise version in language before any code is generated. Agents write code faster than we could ever have written code, so somebody else will type it. Get to the precise, repeatable solution in your thoughts. Pseudocode is fine if it helps.

Interview the stakeholder one question at a time. Domain experts usually lack the skill of figuring out what a stakeholder needs. They can say what they want one question at a time, but not how to make it work and repeatable. In his bakery example, you have no idea how to make the perfect croissant, and the person who runs the bakery has no idea how to write bakery software and is not interested in finding out. Making that precise and repeatable was a systems analyst's job in the 60s: translating business requirements into software requirements without writing any software. He expects that job back as a product engineer, who figures out the real requirements and works with agents to implement them, even if that person writes no code.

For each behavior, write down:

• the user and the job they are trying to finish
• the inputs they will have
• the outcome that counts as done
• the cases that must fail
• what this change will not build

If a sentence still depends on an unstated habit, split it and ask one more question.

They could tell you probably, if you interview them one question at a time, what it is that they want to be able to do and what they want the software to be able to do, but they're not going to have any sense of how to put that together and how to make it work and how to be repeatable.

Laurie Voss, Head of Developer Relations at Arize, on Chain of Thought ep 74

AI Codes: Product Engineers De… - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts

Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · September 22 · 56m

podcasts.apple.com

Where should a human still review agent-written code?

Put a review tool on the first pass and keep an independent human review of the change. The best code review AI tools he has seen have like an 80% hit rate. He said that is really good, and that it is not enough to put stuff into production.

He does not see anyone asking an agent to deploy a finished change, figure out what hardware it should run on, or connect those pieces. He expects agents to be good at that about 10 years from now. Deploying, refactoring, architecting, and scaling are tricky, subtle areas that need a lot of knowledge nobody wrote down. He called that tacit information and said agents will continue to be bad at it for the next five years. In the two to five year frame, senior code review and that operational work still sit with people.

I would say the best code review AI tools that I've seen have like an 80% hit rate. And an 80% hit rate is really good, but it's not enough to put stuff out into production, right? You can't mess up 20% of the time and ship that's not going to work out. So there's still scope for senior level code reviews, obviously AI assisted, you can get the 80% done and focus on the last 20%, which is very important given how much more code is being produced.

Laurie Voss, Head of Developer Relations at Arize, on Chain of Thought ep 74

How do you turn that spec into tests the code must pass?

Turn each behavior in the spec into a case the generated code must pass, and block the change on a failure. Do not loosen a case to match a surprise result unless the stakeholder's answers support that result. If the case was wrong, fix the spec, then run again.

A minimal sketch to adapt, not a drop-in library.

def cases_from_spec(spec):
    cases = []
    for behavior in spec["behaviors"]:
        cases.append({
            "name": behavior["name"],
            "given": behavior["given"],
            "action": behavior["action"],
            "expect": behavior["expect"],
        })
    return cases

def change_is_acceptable(spec, run_case):
    for case in cases_from_spec(spec):
        if not run_case(case):
            return False
    return True

run_case stands in for whatever runner you already trust. It does not review the deploy, the hardware plan, or the tacit rules.

FAQ

Is an 80% hit rate from an AI code review tool enough to ship?

No. He said that is not enough to put stuff into production. Have a person independently review the change.

Is a code-focused bootcamp still a good bet?

He is not sure he would advise spending 15 grand on a bootcamp anymore, because those programs center on a particular framework. He would pick a more theoretical course, including complexity and the business of software. Architecture, operations, scaling, complexity, and design can all be taught, he said, and people are already teaching them if you look carefully.

Does the retooling all have to happen this year?

He said you do not have to completely reskill overnight, and programming jobs are not all going to vanish overnight. Over the next five years, a lot of people do need to retool.

Takeaway

• Interview the stakeholder one question at a time until each need is one testable behavior.
• Write the precise, repeatable solution in language before the agent writes code.
• Derive acceptance cases from that spec and block the change when a case fails.
• Let review tools take the first pass, and keep an independent human review of the change.
• Keep deployment, hardware, architecture, scaling, and unwritten operational knowledge with a person while agents are still weak there.

The full conversation, with the transcript, is on Chain of Thought.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.