Dev.to AI 🤖 Ai 👁 0 📖 6 min read

CrossTrace – crash-test your AI agent's model and MCP calls in CI

TL;DR. After my earlier post I ran CrossTrace, the open-source tool that breaks a model call in your agent and watches what happens, on the OpenAI Agents SDK, pydantic-ai and CrewAI. Two of them (pydantic-ai and CrewAI)

TL;DR. After my earlier post I ran CrossTrace, the open-source tool that breaks a model call in your agent and
watches what happens, on the OpenAI Agents SDK, pydantic-ai and CrewAI. Two of them (pydantic-ai and CrewAI) handled a failure in a way my tool had
labelled as a bug: they re-prompted the model, refused a broken tool call, or summarised and carried on. Those aren't bugs, so I fixed the
tool, not the frameworks. (The Agents SDK simply raised an error, which the tool had judged right.) The fix changed how it decides pass or fail, and the changes are below. There are also real findings (cut answers
printed as complete, hangs, a tool call that ran twice), and it can now fault MCP tool calls, where the damage is a customer refunded
twice. https://github.com/mac146/crosstrace

What the tool does, in one paragraph

A local proxy sits between your agent and the model API (the agent only reads OPENAI_BASE_URL). CrossTrace records one healthy run,
replays it offline, and breaks one model call per trial at the byte level: a stream cut mid-tool-call, a stall, a 500, an empty reply.
Then it watches the agent's next move. No LLM judge.

Where it went wrong

My first version decided what the agent meant by looking at the first request it sent after the fault. If it didn't retry the same
call, and it carried on with something else, I called it SILENT: "the agent hid the failure". That fails your build.

Then I ran real frameworks:

  • pydantic-ai got a tool call cut in half. It refused to run it and told the model its arguments were invalid. That's the right behaviour. I labelled it SILENT.
  • CrewAI got a context-length error. It summarised the conversation and continued. I labelled it SILENT.
  • A re-prompt ("your last reply was empty, try again") looks exactly like an agent carrying on as if nothing happened, because both are "the failed reply plus one more user message".

I patched each one as it showed up, and each new agent needed another patch. That's when I stopped and asked what was common to all of them.

The root problem

The replay is a closed world: it can only answer the healthy conversation. Any request that is legitimately different gets an error,
the run dies there, and the verdict is a guess from the shape of one request.

Looking back at what I'd patched, the facts I measured on the wire never needed a patch: a hang, a retry storm, a retry that ignored
Retry-After, a surfaced error. Everything that inferred intent from a request's shape needed one, every time.

So two changes:

  1. A verdict fails the build only if it rests on something measured. Where all I can see is a different-looking request, the result is DIVERGED or REPROMPTED, shown for review with the facts: what the framework sent, what the error was, what the model was told.
  2. The run continues past the fault. After the fault, requests that match nothing recorded get the next healthy reply, so the agent reaches its real end. Then the verdict comes from how it ends: it finishes, it stops with an error, or it never stops (capped at 8 different requests, then RETRY_STORM).

The limit I put in the README: continuation shows that an agent stops, and how. It doesn't show that its re-prompt would work with a real
model. So I added --live: those requests go to the real provider with your own key (the agent never sees it, capped at 6 calls per
trial). On two frameworks, the recovery works:

  • pydantic-ai refused the cut tool call, the real model answered the re-prompt, and the run finished after 3 more requests.
  • CrewAI summarised after the context-length error, and finished after 4 more requests answered by the real model.

I ran only one or two trials per case, and a real model isn't deterministic, so treat those as demonstrations rather than a guarantee.

What the frameworks do (after the labels were fixed)

Framework Cut stream or cut tool call Stalled response
LangGraph ran the tool on the cut arguments hangs
OpenAI Agents SDK raised an error hangs
pydantic-ai refused the cut call and told the model; printed a cut answer as complete hangs
CrewAI printed a cut answer as complete hangs unless you pass a timeout

Each stall fix was run, not assumed (timeout= on the client, 10 s in my test). Three more things worth knowing: CrewAI retried a
persistent outage 7+ times in about 3 seconds, an empty reply made pydantic-ai re-prompt, and on the Agents SDK, --live showed the same
thing as the unlive run: it stops with an error and doesn't retry.

Tool calls: the failure that costs money

A model call that fails is annoying. A tool call that succeeded but whose reply was lost can be expensive: the email went out, the
card was charged, the response never reached the agent, and a retry does it again.

CrossTrace can now do this to MCP servers. The server is launched through crosstrace mcp-wrap, or for a remote server over HTTP you
list its URL in crosstrace.toml. In a test run it never starts or calls the real server. One tool call is made and counted, its reply is
replaced by a transport error, and every write the agent makes afterwards is counted against the healthy run.

On the example agent in the repo, a refund tool and a naive retry loop:

✗ lost_response   DUPLICATE 1/1
                  refund({"amount":129.99,"customer_id":"42",...}) ran 2 time(s), 1 in the healthy run;
                  the agent recovered and finished, but the write was repeated

The agent printed the right answer and exited 0. It also refunded the customer twice. I also ran it against the official Playwright MCP
server with a real Chrome and a local order form: recording placed one real order, replays placed none, and losing the response to the
click made the retrying agent a DUPLICATE. (That agent was scripted, so it tests the tool-call handling, not how well a model drives a
browser.)

For the claim "it really happens twice" there's --real-tools: the lost-reply trial runs the real server, and the report quotes what the
server answered to the repeat. I tested that against a remote server over HTTP, and the server refunded twice. It's opt-in because it does
the work for real, so point it at a sandbox.

What it still can't do

  • It tests model and MCP calls, not an agent's own HTTP calls to other APIs.
  • An agent that checks state before retrying ("did the refund go through?") gets the recorded answer in replay, so it shows as stopping with an error rather than as recovering safely.
  • MCP over stdio and streamable HTTP, but not servers that push messages on their own stream.
  • A real model's answer to a re-prompt varies, so --live is a demonstration, not CI.
  • I have no outside users yet. Everything above was run on agents I wrote or recorded myself.

Try it

pip install git+https://github.com/mac146/crosstrace
crosstrace init --command "python agent.py {input}"
crosstrace record      # runs your agent for real, once per input
crosstrace test        # replays with faults, offline; exit code 1 if it hides a failure

A demo agent with a recording is in the repo (examples/broken_agents), so you can see the whole thing without a key. There's also a
GitHub Action for the report in a PR summary.

If it can't start your agent, or the replay isn't stable, it says why and, for a value that changes every run, prints the mask to paste.
If you try it on your agent, I'd really like to know what it found, or where it's wrong. Given this post, wrong verdicts are the most
useful thing you could send me.

https://github.com/mac146/crosstrace

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.