AI-Generated Code Passed Its Tests. Did the Tests Check the Right Thing?

A coding agent can produce a function, add a few examples, and report that the tests pass. Passing tests is useful evidence, but one question remains: did the tests check the behavior you actually need, or only the examples the agent expected?

A research release published today, November 10, makes that gap easier to see. Google DeepMind says its AlphaProof Code framework helped prove selected security properties of a compiled AArch64 build of Signal's libsignal reference implementation. The project used human-reviewed specifications and machine-checked Lean proofs. The paper describes about 6.5 million lines of generated proof code built on roughly 26,000 lines of trusted specifications and AArch64 semantics. This is a large, specialized verification effort, not a feature you can switch on in an ordinary coding-agent session. But it points to a practical habit any team can adopt: make the required behavior explicit, then test it across more inputs than a few hand-picked examples. (Google DeepMind research paper, announcement)

This tutorial uses Python and Hypothesis to turn a small requirement into executable properties. Property-based tests are not mathematical proofs. They are a way to check many generated inputs against rules you choose, and to discover examples your hand-written tests missed.

Start with a rule, not a prompt

Suppose an agent is asked to write a function that splits an amount of money evenly among a number of people. "Split it evenly" is too vague to review. A useful specification should answer questions such as:

• Is the amount stored as whole cents, rather than floating-point dollars?
• Must every share be non-negative?
• Must the shares add up to the original amount?
• When cents do not divide evenly, how large may the difference between shares be?
• What should happen if the number of people is zero or negative?

These rules are the contract. The implementation can change, but the contract gives you a stable way to judge it.

Use integer cents. Floating-point values can introduce rounding surprises in money calculations, while integer cents make the required total exact for this example. The function below gives any leftover cents to the first shares:

def split_cents(total_cents: int, people: int) -> list[int]:
    if total_cents < 0:
        raise ValueError("total_cents must be non-negative")
    if people <= 0:
        raise ValueError("people must be positive")

    base, remainder = divmod(total_cents, people)
    return [base + (i < remainder) for i in range(people)]

In Python, bool is a subtype of int, so base + (i < remainder) produces integers. If that expression is unclear to your team, use base + int(i < remainder) for readability.

A few examples can check the obvious cases:

assert split_cents(10, 3) == [4, 3, 3]
assert split_cents(5, 2) == [3, 2]
assert split_cents(8, 4) == [2, 2, 2, 2]

But those examples do not cover every amount and recipient count. A boundary bug might only appear when the amount is smaller than the group, or when the remainder is one less than the group size.

Express the contract as properties

A property describes something that must hold across a range of inputs. For the split function, the core properties are:

• The result contains exactly one share per person.
• Every share is non-negative.
• All shares sum to the original amount.
• The largest and smallest shares differ by at most one cent.

Hypothesis generates inputs for a regular Python test function. Its quickstart documents @given and strategies such as integers(). By default, the library tries 100 generated examples for a test, and it can shrink a failing case to a smaller example. (Hypothesis quickstart, Hypothesis introduction)

Install it as a development dependency, following the current Hypothesis instructions for your environment. Then write a test like this:

from hypothesis import given, strategies as st

@given(
    total=st.integers(min_value=0, max_value=10**9),
    people=st.integers(min_value=1, max_value=10_000),
)
def test_split_contract(total, people):
    shares = split_cents(total, people)

    assert len(shares) == people
    assert all(share >= 0 for share in shares)
    assert sum(shares) == total
    assert max(shares) - min(shares) <= 1

The bounds are deliberate. A test suite should cover the operational range your application supports, not pretend to test every integer Python can represent. The example uses large limits to demonstrate the strategy. A real system should select limits from its domain rules and resource budget.

Give the agent the contract first

A useful coding-agent request can include both the rules and the test command:

Implement split_cents(total_cents, people). Reject negative totals and non-positive people counts with ValueError. Return one integer share per person. Shares must be non-negative, sum to the input total, and differ by no more than one cent. Add Hypothesis tests for these properties and run the test suite. Do not weaken a property to make the implementation pass.

That last instruction is not a technical control. An agent can still misunderstand or ignore it. The reviewer must inspect the test assertions and the implementation. Check that the tests independently express the requirements instead of mirroring the implementation or quietly narrowing the promised behavior.

A property suite is most useful when it comes from requirements independently of the code. If the agent writes both the implementation and the properties, ask a reviewer to compare the rules with the product requirement. Otherwise, one wrong assumption can appear consistent in both.

Add explicit edge cases too

Generated tests complement ordinary examples; they do not replace them. Keep named tests for important boundaries and errors:

import pytest

@pytest.mark.parametrize(
    "total, people, expected",
    [(0, 1, [0]), (1, 3, [1, 0, 0]), (10, 3, [4, 3, 3])],
)
def test_named_splits(total, people, expected):
    assert split_cents(total, people) == expected

@pytest.mark.parametrize("total, people", [(-1, 1), (1, 0), (1, -2)])
def test_invalid_inputs(total, people):
    with pytest.raises(ValueError):
        split_cents(total, people)

These examples make expected behavior easy for a teammate to read. The generated properties explore combinations that are tedious to enumerate. Together they provide stronger evidence than either style alone.

Check that the properties can catch a defect

A test suite that passes is not automatically a useful test suite. Try a mutation: change the return expression to base + int(i <= remainder). The extra equals sign gives one too many shares when the remainder is positive. The sum property should fail.

This is a small manual mutation check. In a larger codebase, mutation-testing tools can make similar changes automatically. The key idea is simple: if a meaningful defect does not make a test fail, the test suite may not be checking that behavior.

Also test the test itself with known invalid inputs. The property test above excludes people <= 0, so it cannot establish the error behavior. The named rejection test covers that separate requirement. Inputs excluded by a strategy are not covered by that property.

What these tests can and cannot tell you

Hypothesis can explore many inputs from the strategy you describe, remember failing examples, and reduce some failures to simpler cases. That can make a bug easier to understand. It does not establish that every possible input was tested, or that your properties capture the correct business or security requirement.

Formal verification makes a stronger kind of claim, but only relative to a formal specification and its trusted assumptions. The AlphaProof Code paper is careful about scope: its Signal results cover specified properties, an executable setup, models, and trusted components. It does not claim that every part of Signal's applications or deployment was proved free of defects. The paper also describes assumptions and areas outside its machine model. (Technical paper)

That distinction matters. A property test is not a formal proof, and a proof of the wrong specification would not give the guarantee you wanted. In both cases, deciding what must be true remains a human responsibility.

For routine application code, property-based tests are a practical middle step between a handful of examples and a full formal-methods project. They work especially well for functions with clear invariants: parsers, serializers, encoders, format conversions, pagination, allocation, and state transitions. They are less effective when expected behavior is subjective or depends on an external system you cannot model.

A review checklist for agent-written tests

Before accepting an agent's test suite, check these points:

• Requirements: Does each important requirement map to at least one assertion?
• Range: Do the strategies cover the real input domain and meaningful boundaries?
• Invalid input: Are rejection rules tested separately from valid-input properties?
• Independence: Were the expected properties reviewed independently of the implementation?
• Failure signal: Would a plausible defect make at least one test fail?
• Limits: Does the article or code describe testing honestly, without calling sampled tests a proof?

Treat the agent's "all tests pass" message as a report, not a conclusion. Read the properties, run the suite, and ask what important behavior remains outside them.

The AlphaProof Code result is notable because it links reviewed specifications to proofs checked by a small trusted kernel. Most teams will not recreate that system for a routine feature. They can still borrow its most useful discipline: agree on what correct means before asking an agent to implement it.

I write about software engineering and AI. Subscribe for more practical guides. Have you added property-based tests to an AI-generated function? What did they catch? Save the checklist above for your next code review.

Sources: Google DeepMind: AlphaProof Code paper, Google DeepMind announcement, Hypothesis quickstart, Hypothesis tutorial.