Observing the Observer: Building a Self-Tracing SRE Agent and Surviving ClickHouse & OpenAI Bottlenecks

AI Agents are Black Boxes:
Last week, I started building an AI-powered SRE Copilot for a hackathon. The goal: an autonomous agent that intercepts system alerts, queries recent traces, and runs remediation playbooks.

But during my first test runs, I ran into a major roadblock. The agent would output: Agent run complete. Status: SUCCESS

Agent run complete. Status: SUCCESS

Problems:
Yet my services remained unhealthy. Because the agent dynamically chains tool calls fetching data, querying API logs, calculating refunds I had zero visibility into where the chain of reasoning broke. Traditional console.log statements are useless when the execution path is generated dynamically at runtime.

I realized I needed to instrument my agent so it could observe itself.

I’m currently 60% through building this system. Here is a transparent look at the engineering hurdles I’ve solved so far using SigNoz and OpenTelemetry, and the challenging roadmap I’m tackling next.

📸 **The Screenshots: What I Built & How SigNoz Monitors It:**
To make the agent’s operations visible, I built a custom Glassmorphic Dashboard and hooked the backend directly into SigNoz. Here are the three key visuals that explain how this system works:

1) The Interactive WebGL Loading Eye:
Press enter or click to view image in full size

Evil Eye Observing Everything!
The Interactive WebGL Loading Eye:

(

) Before the dashboard loads, this interactive WebGL “Evil Eye” loader tracks the cursor position.

The Tech: Built using native browser ES Modules and esm.sh/[email protected] (bypassing legacy UMD CDN 404s).
How it works: The shader coordinates are scaled to 0.55 inside a 500px canvas, allowing the outer flame particles to expand and fade naturally without clipping.

2)The Glassmorphic Observability Console

The Glassmorphic Observability Console:
The Tech: Includes a rolling memory logger buffer of the last 50 entries exposed at /api/dashboard/logs.
Trace-to-Log Link: Every error log links to its active OpenTelemetry traceId using a custom Winston formatter. Clicking a log line transports you directly to its telemetry trace inside SigNoz.

3) The SigNoz Telemetry Traces UI

The SigNoz Telemetry Traces UI:
I get deep, nested trace spans for every LLM reasoning step.

Rate & Latency Tracking: I ran a traffic generator (constant-load.js at 2 req/s) to populate real-time latency and operation rates.
The 429 Fail-Safe: When the OpenAI API key hit quota limits (429), I wrote an offline agent simulator that executes the actual telemetry tools locally. This kept our success rates at 100% and generated full-trace timelines in ClickHouse.

4) The Technical Hurdles Solved So Far:
ClickHouse XML Overrides: Solved a schema migration crash (DB::Exception: Cannot parse JSON) by upgrading ClickHouse to 25.12.5-alpine and enabling the experimental JSON flag in users.xml.
Graceful LLM Degradation: Intercepted rate limits to simulate agent reasoning while running actual instrumented backend systems, keeping the Otel collector populated.

5) The 40% Roadmap: What’s Next?
GET /api/agent/history (Medium): Persist agent runs in a DB so engineers can audit past AI remediation decisions.
Interactive Chat Console (Hard): Build a live terminal to chat directly with the Copilot during active incident triage.
Alert Webhook Hookup (Hard): Automatically trigger the agent whenever a SigNoz alert manager rule fires, letting the AI debug the alert before a human is paged.
Built as part of the Agents of SigNoz Hackathon by WeMakeDevs. Follow the repository on GitHub (https://github.com/praveshhh/signoz)