I Built AgentLab: An Open-Source Tool to Test and Understand AI Agents

AI agents can call tools, execute workflows, and generate complex outputs—but how do we know whether they are actually reliable?

I built AgentLab, an open-source platform for testing, evaluating, and understanding AI agents locally.

What AgentLab does

AgentLab lets developers define evaluation cases using YAML, execute a local agent or application, and inspect clear PASS, FAIL, and ERROR results.

Current capabilities include:

• YAML-based test suites
• Secure local command execution
• Explicit permission before executing programs
• Exact output evaluation
• PASS / FAIL / ERROR analysis
• JSON reports
• Standalone HTML reports
• Local web GUI
• No database, cloud upload, or telemetry

Example test suite

yaml
suite: Greeting Tests

tests:
  - id: greeting_001
    input: "Hello"
    expected: "Hello"

  - id: greeting_002
    input: "How are you?"
    expected: "I'm fine."