Who Judges the AI Judge? The Problem With Trusting LLM Evaluations

I've been building an open-source AI verification project called Orinyx, and there's one question I keep coming back to:

If we're using an LLM to evaluate another LLM, who verifies that the judge is actually correct?

It's a question that sounds simple until you start designing an evaluation system around it.

We spend a lot of time discussing AI accuracy, hallucinations, benchmarks, and agent reliability. But what happens when the system responsible for identifying those failures has its own limitations?

An LLM judge can be confidently wrong, too.

And if that judge is responsible for deciding whether an AI system passed or failed, we have a bigger problem than inaccurate outputs.

We have an evaluation system that might be producing false confidence.

The problem with AI judging AI

Imagine two agents completing the same task.

Agent A produces a convincing answer that's technically incorrect.

Agent B recognizes that critical information is missing and refuses to provide a definitive answer.

If an LLM judge primarily rewards completeness, confidence, or apparent correctness, Agent A might receive the higher score.

But is Agent A actually more reliable?

Not necessarily.

I recently raised a question about our upcoming Orinyx Arena in a Reddit discussion, and several developers brought up points worth exploring.

One suggested that evaluations should explicitly measure appropriate refusals and abstentions, not just correct answers.

Another argued that verification only matters if the checks are capable of detecting actual failures.

And a third challenged the idea of shared evaluation scenarios altogether.

Their argument was that developers should define what success means for their own agents, rather than relying exclusively on generic benchmarks.

That last point particularly interested me.

A benchmark might tell us which agent performs better on a particular set of scenarios. But does that mean either agent meets the requirements of the application it was built for?

Those are two different questions.

So how do we keep the judge accountable?

This is something we're exploring as we design Orinyx's broader evaluation architecture.

One concept we've discussed is a gold-standard evaluation gate, where authoritative expected outcomes and evidence requirements are established independently of the LLM being evaluated.

The idea is that an LLM judge shouldn't get to invent the criteria for success and then declare that those criteria were satisfied.

Where possible, deterministic checks should verify observable facts and requirements. Independently reviewed reference cases should help establish expected outcomes. LLM-based evaluation can contribute to interpreting ambiguity, but its conclusions should remain challengeable.

We're also considering how regression testing, held-out scenarios, judge disagreement, and human adjudication should fit into that process.

But there's another important distinction.

A regression test can tell you whether your system's behavior changed. It doesn't necessarily tell you whether the original behavior was correct.

And even a carefully constructed reference dataset can become less useful if developers repeatedly tune their systems against the same examples.

So maintaining evaluation integrity is not just about selecting a better judge model.

It's about the surrounding methodology.

Who defines the expected outcomes?

Who reviews disagreements?

What evidence is required before a system can receive a passing result?

How do we prevent our own verification engine from becoming overconfident?

And when the evidence is insufficient, should the most trustworthy answer simply be inconclusive?

Where Orinyx fits into this

Orinyx currently focuses on open-source verification tooling for healthcare-related AI workflows.

We're working toward an API for programmatic verification and an Arena where AI systems can be evaluated against structured scenarios.

But I don't want the Arena to become another leaderboard that produces impressive scores without making the underlying evidence understandable.

I'd rather build something where developers can examine why a system passed, why it failed, where the evaluator was uncertain, and whether the results can be reproduced.

That also means considering developer-defined requirements alongside standardized evaluation scenarios.

We're still working through these design decisions, and I don't think there's a single model, benchmark, or evaluation framework that automatically solves the problem.

Here's what I'd love to hear from other developers

If you're building AI agents, evaluation pipelines, or LLM-as-a-judge systems:

How are you verifying that your evaluator is actually trustworthy?

Are you relying on human-reviewed golden datasets? Deterministic assertions? Independent judge models? Adversarial test cases? Some combination of these?

And perhaps the more difficult question:

What evidence would you need before trusting an AI verification platform enough to accept its verdicts?

I'm particularly interested in approaches that work beyond simple pass/fail scoring.

Because if we're going to build systems that tell developers whether their AI is reliable, we need to be able to demonstrate why those systems themselves deserve to be trusted.

For anyone interested in the project, here's the open-source Orinyx MCP:

https://github.com/lexthirteen-ai/orinyx-mcp

I'd genuinely like to hear how other teams are approaching this problem.