What smart-contract audits taught me about evaluating LLMs

Before I worked on LLM systems, I spent a lot of my time in smart-contract audit contests. I entered more than twenty of them, on platforms like Sherlock, Code4rena and Immunefi. The setup is always similar: a team publishes its contracts, auditors get a limited window to read them, and you are rewarded only for issues you can prove.

That work trains one habit above all others. You stop asking whether the code works. The team already believes it works, and their tests already pass. You ask what must never happen, and then you spend your time trying to make it happen anyway.

I now build production AI systems, and that habit has turned out to be the most useful thing I brought with me. Here is how it changes the way I think about evals.

Happy paths prove very little

An LLM demo is a happy path by design. Someone asks a clear question, the retriever finds the right document, and the answer looks great. Many eval suites are a longer version of the same demo: fifty well-formed questions and an average score.

An auditor reading that suite would ask a different question. The interesting number is not how often the system is right. It is the worst thing the system can do, and whether any test checks for it.

For an LLM feature, the worst outcomes are rarely subtle. It states a number that no data source contains. It answers a policy question without reading the policy. It calls a tool that changes something when the user only asked a question. It shows one customer's data to another. None of these show up in an average.

Write invariants, not just examples

In smart-contract testing, an invariant is a property that must hold after any sequence of actions. The classic one for a token: total supply always equals the sum of all balances. Tools like Foundry's invariant testing then fire random sequences of calls at the contract and check the property after every step. You stop writing tests only for cases you imagined, and start describing rules the system can never break.

The same idea works for LLM systems, as long as code can check the rule:

• Every number in the answer appears in the data the tools returned.
• A request classified as read-only never triggers a tool that writes.
• A signed-in user's answer only contains rows that belong to that user.
• An answer about a policy cites the document it came from.

Here is a minimal version of the first rule. It is strict on purpose:

import re

NUMBER = re.compile(r"\d[\d,]*(?:\.\d+)?")

def numbers_in(text: str) -> set[float]:
    return {float(n.replace(",", "")) for n in NUMBER.findall(text)}

def ungrounded(answer: str, tool_outputs: list[str]) -> set[float]:
    """Numbers in the answer that no tool call returned."""
    grounded = set().union(*(numbers_in(out) for out in tool_outputs))
    return numbers_in(answer) - grounded

tool_outputs = ['{"region": "north", "revenue": 84200.00}',
                '{"region": "south", "revenue": 21700.00}']
answer = "North led with 84,200, ahead of south at 21,700. Total: 105,900."

print(ungrounded(answer, tool_outputs))  # {105900.0}

The total is correct, and the check still flags it. That is the point. Models are unreliable at arithmetic, and a wrong total reads just as confidently as a right one. If a total matters, have a tool compute it, so it arrives as data rather than as model output. A real version needs an allowlist for things like dates and "top 5", but the shape stays the same: a rule, checked by code, on every answer.

Treat every input as hostile

Auditors assume every caller is an attacker. If anyone can call a function, someone will call it with the worst possible arguments.

LLM systems have more callers than they appear to. The user is one. So is every document the retriever pulls in, every web page a tool fetches, and every response another service sends back. Any of them can contain text that reads like an instruction, which is why OWASP lists prompt injection first among the risks for LLM applications.

So the eval set needs hostile cases next to the normal ones: a question about another user's account, a retrieved document with instructions hidden in it, an ambiguous date, a harmless-sounding request that would need a write tool. If the suite only contains questions a friendly user would ask, it measures the demo, not the system.

Severity beats averages

Audit contests do not just count findings. They grade them. Code4rena, for example, sorts findings by severity, and a single high-severity issue matters more than any number of low ones. A protocol with forty fixed style issues and one way to drain funds is not 97% safe.

Single-number eval dashboards make exactly that mistake. A 97% pass rate says nothing if the other 3% includes the model inventing a payout amount.

The split that works better is two groups. Invariants are release gates: zero tolerance, and one failure blocks the change. Quality scores, such as helpfulness or tone, are tracked as trends, where a small drop is a reason to investigate rather than a blocker.

Every bug becomes a test

When an audit finding is confirmed, a good team ships the fix with a test that reproduces the attack, and that test stays in the suite for good.

LLM systems need this discipline even more, because they break in ways code review does not catch. A new model version, a prompt tweak or a different chunk size in retrieval can each break a case that passed last week, with no code diff that looks related. Every bad answer found in production should become an eval case, and the whole suite should run on every change, including changes that "only" touch a prompt.

Where the analogy breaks

A contract is deterministic. If a test fails once, there is a bug. A model is not. The same case can pass nine times and fail on the tenth, so important cases need several runs and a pass rate, not a single result.

Some properties also cannot be checked by code. Whether an explanation is clear, or whether the tone fits, needs a human reviewer or a model acting as a grader. Those scores are useful, but they are weaker evidence than an invariant. Use them to spot trends, and never let them alone decide that a change is safe to ship.

Start with the list

If you come from security, you already know how to write evals. The skill is the same: decide what must never happen, turn each rule into something code can check, and then try hard to break it.

If you are starting from scratch, start there too. Before writing a single eval question, write down the five things your AI feature must never do. That list will tell you more about what to test than any benchmark.

I write about AI engineering and building reliable systems at https://www.0xpiyush.xyz