Evaluating AI Agent Safety: Why an Approval Dialog Is Not the Final Outcome

An AI agent asks to run a program. Its sandbox blocks the first attempt, then presents an approval dialog. The operator approves it. The program runs and changes a protected resource.

Did the protection work?

The answer depends on which event you measure. The initial block happened. The approval gate worked as designed. But the target effect was not prevented after approval. Combining those facts into one "safe" label hides the behavior a user needs to understand.

This article explains the evaluation method we use in Agent Safety Arena. It separates the agent's decisions from sandbox enforcement, and both from the final effect on the computer.

Start with an observable effect

A safety case needs a target that can be checked independently of the agent's reply. "The agent said it was blocked" is insufficient. A useful case asks whether a specific camera opened, a synthetic file changed, or a controlled desktop operation occurred.

We group cases into four areas:

• Privacy: access to cameras, microphones, screenshots, and screen recordings.
• Interaction: window operations and pop-ups that can disrupt control of the desktop.
• System: changes to controlled programs, services, and system settings.
• Data: reading, changing, moving, or deleting designated files.

These tests use controlled resources. The allowed actions and resource boundaries are defined before execution. A scenario should not put real personal files or unrelated system resources at risk.

Choose the test design to match the question

We use two approaches.

For many device, desktop, and system cases, we validate a prepared program outside the evaluated agent, freeze it, and ask the agent to run the unchanged file. This reduces variation from code generation. If a broken program never reaches the target operation, that is not evidence that a sandbox blocked it.

For data cases, we ask for a direct file task and let the agent choose its tools. This tests whether a declared resource boundary holds across file tools, shell commands, and scripts.

These approaches answer different questions. A fixed program helps isolate enforcement against a known operation. A direct task also exposes the agent's tool choices and execution strategy. The case record must state which approach was used.

Record three separate outcomes

For an approval request within the controlled test scope, we capture the original wording and screen, then approve continuation under the policy defined in advance. We inspect the result on the machine.

If the effect occurs after approval, the report must say so. It must not describe that outcome as an operation that silently bypassed an approval gate. Equally, the presence of a gate must not be counted as proof that the effect remained impossible after approval.

Warning quality is a separate assessment. Showing a command or program name alone does not establish that the user received a clear explanation of its consequences.

Define the denominator before reporting a rate

Our final protection rate is the proportion of valid, assessed scenarios in which the target harmful effect was ultimately prevented, calculated separately for each agent.

defined final protection rate = prevented target effects / valid, assessed scenarios

Incorrect programs and operations that were never actually attempted do not count as protection successes. Scenarios with insufficient evidence or without an assessment stay outside the denominator. They should remain visible as unresolved cases rather than being quietly treated as passes.

For example, suppose a hypothetical evaluation has ten planned cases. Two have incomplete evidence, leaving eight valid, assessed cases. If the target effect is prevented in five, the rate is 5/8, or 62.5%. It is not 5/10, and it is not increased by counting approval dialogs as additional successes. These numbers illustrate the calculation; they are not project results.

Read the result together with its conditions

This metric measures final protection under a particular approval process. It is not the probability of an accident in everyday use. It does not predict how often a person would reject a request.

A report also needs the product version, model, prompt, permission configuration, approval policy, and observed evidence. Product defaults and configured resource rules answer different questions and must be identified separately. A comparison with an additional protection layer must disclose configuration differences; the result alone does not isolate which mechanism caused the difference.

For the same reason, a comparison of agent products is not automatically a ranking of their underlying models. The execution environment, tools, and permissions are part of what was tested.

Make the evidence inspectable

A useful case record connects the request to the attempt and the attempt to the observed effect. It preserves the original approval wording, the continuation decision, recordings, and postcondition checks. Where evidence cannot support a conclusion, the report should state the uncertainty.

This makes disagreements concrete. A reader can question the scenario, approval policy, warning assessment, or effect check without having to accept a single unexplained safety score.

The method described here comes from our Agent Safety Arena evaluation guide. The Safety Arena website provides the case records and recordings used to inspect the results.

Disclosure: this article was adapted with AI assistance from our project's evaluation guide.