Your AI agent eval needs a third answer

Most agent evaluations are built around two outcomes: pass or fail.

That sounds reasonable until the evaluator cannot establish what happened. A missing tool result, the wrong time window, stale permissions, or an incomplete trace is not proof that the agent passed. It is not always proof that the agent failed either.

It means the evidence is incomplete.

That needs its own result: INDETERMINATE.

A small example

Suppose a financial-data tool is expected to return five trading sessions. It returns five rows. A count-only check says the result matches.

But the five rows can cover entirely different dates. The shape is right while the evidence is wrong.

Nobulex has a small offline example built around exactly this case. It uses fictional inputs and checks three outcomes:

• matching sessions return PASS
• silently missing sessions return FAIL_UNSAFE
• the same row count covering different dates returns INDETERMINATE

The third result matters. The evaluator can show that the windows disagree, but that alone does not prove which system is wrong. Calling it a failure would overclaim. Calling it a pass would be worse.

Why binary scoring hides problems

A green result can mean two very different things:

1. The check ran against the intended evidence and found no problem.
2. The check never received the evidence it needed.

If both become PASS, a dashboard can look healthy because the safety check was blind.

A failed check and a check that never ran should also stay separate. The first says something about the system under test. The second says something about the evaluation itself.

That distinction is useful during incident review because it tells you whether to investigate the agent, the evidence pipeline, or both.

Try the smallest version

The example is here:

https://github.com/arian-gogani/nobulex-registry/blob/main/examples/wrong_window.py

After cloning the repository:

python3 examples/wrong_window.py

It runs locally with fictional fixtures. It does not contact a live tool and it is not a registry attestation. PASS covers one probe only. It does not establish that prices are correct.

That limit is part of the result, not fine print.

The repository is MIT licensed:

https://github.com/arian-gogani/nobulex-registry

If you work on agent evaluation, try to break the example with a small input. If you want to follow the work, star the repository.