Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 8 min read

Beyond the Final Answer: Evaluating the Reliability of Tool-Using AI Agents

An AI agent can give you the correct answer and still have done the wrong thing to get there. It might skip a required tool, pass incorrect arguments, encounter an execution error, or reach the right answer by coinciden

Beyond the Final Answer: Evaluating the Reliability of Tool-Using AI Agents

An AI agent can give you the correct answer and still have done the wrong thing to get there.

It might skip a required tool, pass incorrect arguments, encounter an execution error, or reach the right answer by coincidence.

In a simple chatbot, evaluating the final response may be enough for a basic quality check. But once an agent starts interacting with tools, the final response tells only part of the story.

How do we know whether an agent actually completed a task correctly, followed the expected execution path, and handled tool failures appropriately?

I wanted to explore this problem by building a small evaluation framework for tool-using AI agents.

That project became Agent Reliability Lab.

The problem: A correct answer doesn't always mean a successful execution

Consider an agent asked to calculate:

Use the calculate tool to compute 125 * 8 + 37.

The expected answer is 1037.

There are several ways an agent might respond:

  1. It invokes the calculator with the correct expression and returns the correct result.
  2. It skips the calculator and produces the answer directly.
  3. It invokes the calculator with invalid arguments.
  4. The tool executes successfully, but the agent returns an incorrect answer.
  5. The tool encounters an error, but the agent successfully recovers.

If we evaluate only the final response, some of these scenarios look identical.

But from an engineering perspective, they represent very different outcomes.

For example, if a task explicitly requires a tool invocation, producing the correct answer without using that tool should not count as a successful execution.

This led me to a different approach:

Evaluate the outcome and the execution evidence independently.

The approach: Make agent execution observable and measurable

The Agent Reliability Lab captures the full execution lifecycleβ€”from task input and tool interactions to independent evaluation and reliability reporting.

Agent Reliability Lab architecture diagram

Agent Reliability Lab is a local-first evaluation and observability framework for tool-using AI agents.

The initial implementation uses:

  • Python for the evaluation harness.
  • Ollama to run a locally hosted Qwen3 8B model.
  • A deterministic calculator tool.
  • Structured JSON execution traces.
  • An independent evaluator.
  • Repeated experiments and aggregated metrics.

The idea is to capture what happened during execution and evaluate it against explicit expectations.

1. Building a bounded agent execution loop

The first component is the agent runner.

It loads a task, sends the prompt to the local model, exposes the calculator tool, and processes any tool calls returned by the model.

The execution loop is deliberately bounded:

MODEL = "qwen3:8b"
MAX_STEPS = 5

for step in range(MAX_STEPS):
    response = chat(
        model=model,
        messages=messages,
        tools=[calculate],
    )

    messages.append(response.message)
    tool_calls = response.message.tool_calls or []

    if not tool_calls:
        trace["final_response"] = response.message.content or ""
        break

Why impose a step limit?

Tool-using agents can sometimes continue calling tools without making meaningful progress. A bounded loop prevents unbounded execution and gives us a way to classify tasks that exceed the execution budget.

The runner also records tool arguments, results, errors, execution duration, and the final response.

This makes the execution path inspectable instead of treating the agent as a black box.

2. Capturing execution traces

A final answer alone doesn't provide enough information to understand an agent's behavior.

For every task, Agent Reliability Lab captures a structured execution trace.

The trace includes:

  • Task identifier.
  • Model used.
  • Execution start time.
  • Tool calls and arguments.
  • Tool results and errors.
  • Individual tool execution durations.
  • Final response.
  • Total execution duration.
  • Execution errors.

A simplified trace structure looks like this:

{
  "task_id": "math-001",
  "model": "qwen3:8b",
  "tool_calls": [
    {
      "step": 1,
      "tool": "calculate",
      "arguments": {
        "expression": "125 * 8 + 37"
      },
      "result": "1037",
      "error": null,
      "duration_ms": 0.5
    }
  ],
  "final_response": "The answer is 1037.",
  "error": null
}

This is an illustrative example of the trace structure.

The implementation persists traces as JSON files, making them available for later inspection and failure analysis.

I wanted to preserve execution evidence separately from evaluation results so that the evaluator could inspect what actually happened.

3. Evaluating the agent independently

This is one of the most important design decisions in the project.

The agent should not decide whether it successfully completed its own task.

Instead, a separate deterministic evaluator examines the execution trace.

It checks three important things:

Answer correctness

Does the final response contain the expected answer?

The evaluator normalizes formatted numbers and recognizes explicit answer patterns, rather than simply searching for an arbitrary matching number anywhere in the response.

It also rejects contradictory answer candidates.

Tool compliance

Was the required calculator tool successfully executed?

A task that requires a calculator call should not pass just because the model returned the expected number without invoking the tool.

Execution status

Did the agent encounter an execution error or exceed the maximum number of steps?

The task passes only when the answer is correct, the required tool execution succeeded, and the agent did not encounter an execution error.

The core evaluation logic is intentionally simple:

passed = (
    answer_correct
    and tool_requirement_met
    and not error
)

Keeping evaluation independent makes the results more trustworthy than relying on the agent's own statement that it completed the task.

4. Classifying failures instead of reporting only pass or fail

A binary pass/fail result is useful, but it doesn't tell us what to fix.

For example, these failures need different investigation paths:

  • The agent never called a required tool.
  • The tool was called but returned an error.
  • The agent returned an incorrect answer.
  • The execution loop reached its step limit.
  • The agent encountered an execution-level exception.

Agent Reliability Lab classifies unsuccessful tasks into categories such as:

Failure category Meaning
tool_use_failure Required tool was not used
tool_execution_failure Required tool did not execute successfully
incorrect_answer Final answer did not match the expected result
execution_error Agent execution encountered an error
step_limit_exceeded Agent reached the maximum execution steps

This distinction matters because the solution to an incorrect answer may be different from the solution to a tool execution failure.

It also gives us a foundation for analyzing failure patterns across repeated runs.

5. Measuring more than answer accuracy

Once execution traces and independent evaluation are available, we can start measuring agent behavior.

The framework generates several metrics:

Metric What it tells us
Pass rate Percentage of benchmark tasks completed successfully
Answer correctness Whether the final answer matches the expected result
Tool success rate Successful tool calls divided by total tool calls
Tool failure count Number of tool calls that returned errors
Recovery count Tasks completed successfully after a tool failure
Failure categories Why unsuccessful tasks failed
Mean latency Average task execution time

I particularly wanted to capture tool reliability and recovery separately from answer correctness.

For example, an agent that successfully recovers after a tool failure behaves differently from one that stops immediately.

The evaluator can identify successful recovery when a task passes despite an earlier tool failure.

These metrics help make those differences visible.

6. Why repeated evaluations matter

One successful run doesn't necessarily mean an agent is reliable.

LLM behavior can vary across executions, even when the task and configuration remain unchanged.

To explore this, I added a repeated evaluation runner.

The current implementation executes the benchmark three times:

RUNS = 3

for run_number in range(1, RUNS + 1):
    result = subprocess.run(
        [sys.executable, "scripts/run_eval.py"],
        cwd=ROOT,
        text=True,
        capture_output=True,
    )

Each run gets its own report and archived execution traces.

The framework then aggregates:

  • Pass rate for each run.
  • Mean pass rate.
  • Mean execution latency.
  • Tool success rate.
  • Total tool calls and failures.
  • Total recovered tasks.

This is useful because a single pass rate can hide variability.

For example, if an agent passes a task in one run but fails it in another, that inconsistency is worth investigating.

Repeated evaluations provide a starting point for understanding that behavior.

One important limitation: three runs over a small dataset are useful for experimentation, but they are not enough to establish statistically strong reliability claims.

7. Keeping the evaluation environment practical

I chose a local-first setup using Ollama and Qwen3 8B.

This makes it possible to experiment without depending on a hosted model API for every run.

The calculator is also deterministic. It uses Python's AST parser and an explicit allowlist of supported arithmetic operations rather than evaluating arbitrary Python expressions.

The current benchmark contains six arithmetic tasks, each requiring the calculator tool.

This is intentionally a small starting point.

The project also separates automated tests from model inference. The deterministic test suite validates the calculator, evaluator, failure classification, execution behavior, and recovery logic without requiring an actual model run.

GitHub Actions runs these tests on pushes and pull requests.

That separation makes it easier to validate the evaluation infrastructure without repeatedly paying the cost of model execution or waiting for inference.

Current limitations

Agent Reliability Lab is an early-stage prototype, not a production-grade agent evaluation platform.

There are several limitations worth calling out:

Small benchmark

The current benchmark contains six arithmetic tasks. It is useful for validating the evaluation pipeline but does not represent complex, multi-step agent workflows.

Limited tool coverage

The initial implementation exposes one calculator tool. Real-world agents interact with multiple tools, APIs, databases, and external systems.

Limited failure scenarios

The framework records and classifies tool and execution failures, but the current benchmark is not a comprehensive fault-injection suite.

Model variability

Local model behavior can vary between executions. Repeated runs help observe this, but a small number of runs cannot establish general reliability.

Limited semantic evaluation

Numeric answer matching works for the current arithmetic benchmark. It is not sufficient for evaluating arbitrary natural-language responses.

These limitations define the next areas of exploration rather than something the current implementation claims to solve.

What I'd explore next

There are several directions I'd like to investigate:

  1. Expand the benchmark to multi-step tasks involving multiple tools.
  2. Introduce controlled tool failures and evaluate recovery behavior.
  3. Compare different models and prompts against the same benchmark.
  4. Improve execution traces with more detailed step-level observability.
  5. Explore OpenTelemetry and agent observability platforms.
  6. Add regression detection across model and prompt changes.

I'm particularly interested in making agent evaluation part of the engineering development lifecycle.

When a prompt, model, or tool implementation changes, we should be able to run the same evaluation suite and understand whether agent behavior improved, regressed, or simply became less predictable.

Final thoughts

Building Agent Reliability Lab helped me think about agent engineering from a testing and observability perspective.

The important shift is moving away from evaluating only what an agent says and toward evaluating what it actually does.

A reliable agent should produce the expected result, use its tools correctly, handle execution failures, and behave consistently enough that engineers can understand and improve it.

The principle I keep coming back to is:

Don't just evaluate the answer. Evaluate the execution that produced it.

I'm continuing to explore practical approaches to AI agent reliability, developer productivity, and engineering workflows.

If you're building tool-using agents, I'd be interested in hearing how you're evaluating tool behavior, execution failures, and consistency across repeated runs.

Explore the project:

Agent Reliability Lab on GitHub

Feedback and suggestions are welcome!

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.