Tracing for coding agents that redacts before it writes — and never reports an unknown as a zero

When I looked at MLflow, Langfuse and LangSmith for my coding agents, the gap wasn't tracing. All three will happily record spans; two of them will even evaluate outputs with an LLM judge. The gap was the workload. A coding session is not an LLM call. It is a tree of tool calls, file edits, retries and nested agents that runs for hours, and the moment you decide to store transcripts, redaction stops being a footnote and becomes the storage design. Your telemetry now contains source code, .env files and the occasional production token pasted into a terminal.

I ended up building the analytics layer into my own harness, code-copilot-team, instead of adopting a stack. Not because those products are bad, - because each rule I needed broke one of their defaults. This post is about the three rules, the bugs they caught, and what I'd tell anyone building observability for coding agents.

The workload is not an LLM call

My store currently holds 65 real sessions, 72,223 turns and 22,401 tool calls from months of daily work, plus 32 sessions excluded as noise — probe runs in temp directories and two-turn smoke tests that outnumber real work on a developer machine. That exclusion count is printed on every page, because a dashboard that silently drops half the corpus is lying about its denominator.

A flat list of prompts and responses answers "what did the model say". It cannot answer "why did this session take six days", "which tool burns the most turns", or "did my last change to the harness make anything better". Those questions need structure: session → turn → tool call, with latency and cost attached to each level, and lineage across resumed sessions — which session was resumed from which, proven by turn identifiers the transcript preserves, never inferred from timing.

That is the shape MLflow's gap list pointed at, and the reason I extended my harness instead of bridging to theirs: A1 trace tree, A2 search, A3 feedback, A4 harness-version comparison, A5 expectations, A6 threads. Six features, one data model, and every one of them depends on the redaction rule below.

Redact before you write

Transcripts are the most sensitive telemetry a developer produces. So the store has three redaction modes — none, code, metadata-only — applied at ingest, not at read. code strips fenced blocks, which is where keys and curl commands live. metadata-only keeps counts and drops text. The mode is recorded per session, so you can tell later what a number was computed from.

The judge creates a second path out: excerpts of your steering files go into a prompt. That path takes the stricter of two modes — the one recorded when the session was ingested, and the project's current effective setting — because a project tightened after ingest must not leak under the stale mode.

And then there is the rule that cost me a review finding: redact the whole input, then truncate. My excerpt builder read the first 1,200 characters of AGENTS.md, then redacted that slice. A code fence that opened at character 1,100 and closed at 1,400 never matched the fence pattern, so its contents — a secret, in the worst case — went to the model verbatim under a mode that promised fences were stripped. The fix is one reordered statement. The lesson is not. Truncation looks like formatting; redaction looks like security; when they meet, the security step must consume the entire input, or "mostly redacted" becomes a euphemism. The test now writes exactly that case: a fence straddling the boundary, and an assertion that the secret is gone.

Null is not zero

The most dangerous defect in an observability tool is a confident zero. A rate with nothing evaluated is not 0%; it is unknown. An unpriced cost is not $0.00; it is unpriced. These distinctions sound pedantic until you make a budget decision on one.

So the rule in this harness: every rate carries its coverage. A cost tile says "priced turns only — unpriced turns are excluded, not zeroed". A response-time card says "4 of 236 measured". An expectation is met, not_met or unknown, and an unevaluated requirement is never counted as a failure — the rate is null when nothing was evaluated. The lineage view opens with "No ancestor found. All 1 known sources were read and assessed", so an empty answer distinguishes "none exists" from "I could not check".

The judge is a secondary signal

Deterministic scoring tells you whether the code built and the tests passed. It cannot tell you whether the error handling was thoughtful. v2.0.0 adds an LLM judge for that second question, under three constraints.

It is validated before it counts: judge labels are compared against human labels with Cohen's kappa, and the deterministic verdict stays authoritative. It is grounded: the tuning analysis receives the project's actual steering files — AGENTS.md when that is what the project uses, not a guess that it must be CLAUDE.md — plus a harness stamp (version, commit, digests of instructions and provider config), with excerpts redacted as above. And it validates what it proposes: a suggested file path is checked against the inventory of files that actually exist, as membership in that list, never as a filesystem probe — a real file the inventory leaves out must not read as "found", and the check must not become an oracle for what else is on disk.

Failure semantics matter as much as success semantics. A judge run that labels nothing out of a non-zero number of attempts fails the step, with the last error; a mostly failed run leads its message with the failure count instead of burying it after the successes. And a failed attempt is not a label: the next run retries those turns, so one backend outage doesn't leave a permanent red mark in the funnel.

What the trace shows

Under each turn, a trace tree: the tool calls with their results, the time until each result, and subagent runs nested under the turn that started them. Per session, a harness stamp, so "did the new skill change anything" is a comparison rather than an opinion. And in the analysis pipeline, a rule worth stealing: the user picks sessions before anything is read, and a selection that resolves to nothing is refused, never widened to the whole store. Intent is a constraint, not a hint.

The numbers that survived audit

Two, with their qualifiers. The graph rebuild: 67 minutes and 9.2 GB peak memory on a 90k-turn store before the rewrite, 2.7 seconds and 568 MB after, on a 96k-turn store — recorded local runs, not a benchmark. The local-vs-hosted comparison: a 27B on a DGX Spark against a hosted frontier model on one coding task; both passed, the local one ~3.3x slower. One task. The point is not the ranking; it is that the harness made the comparison reproducible and kept the sample size attached.

Try it

The whole thing is open source and local-first: github.com/gosha70/code-copilot-team, released as v2.0.0. Studio and auto-build are beta, routing is experimental, and upgrading from 1.1.0 recreates the analytics store — the remedy is printed at startup.

If you build observability for agents yourself, take the three rules, not the code: redact the whole input before you truncate; a rate without coverage is null, never zero; and give the model a seat, not the gavel.