Dev.to AI 🤖 Ai Fri, 13 Nov 2026 19:30:38 +0000 👁 0 📖 9 min read

Fixing X-Ray Service Maps for Multi-Agent Systems on AWS

I built a multi-agent system on Amazon Bedrock AgentCore. An orchestrator agent takes each request and hands the work to specialist worker agents: one checks orders, one handles refunds, one writes the reply to the custo

Every agent worked. But the first time I opened the X-Ray service map, it showed the orchestrator, a box for Bedrock, and nothing else. No workers. No knowledge bases. Just a map that made my system look far simpler than it was.

Getting a map that matched the real call chain took me much longer than building the agents. This post is everything I wish I'd known before I started.

In this post:

What a service map is, and how X-Ray decides what to draw
The basic setup
Why worker agents and knowledge bases go missing, and how to fix it
Why parallel calls break the trace
How to check it worked

First, some quick definitions

A trace is the story of one request. When a customer asks for a refund, the trace records everything that happens because of it: the model calls, the tool calls and the work each agent does.

A trace is made of spans. A span is one step in the story, such as "asked the refund agent" or "searched the policy knowledge base". Each span records how long it took and whether it failed.

A service map turns traces into a picture. Each service is a box (a node), and each call between them is a line. For a multi-agent system, it's the quickest way to see which agent is slow, which one failed, and whether a request took the route you expected.

Here's the part that caught me out: the map only draws what your traces describe as separate services. If your code doesn't say "this is a call to a different thing", X-Ray has no reason to draw a separate box for it.

The basic setup

1. Turn on CloudWatch Transaction Search (once per account)

AgentCore sends traces through CloudWatch. Nothing appears until Transaction Search is on. You only do this once per AWS account and region.

In the CloudWatch console:

Go to Application Signals (APM) and choose Transaction search.
Choose Enable Transaction Search.
Tick the box to ingest spans as structured logs.
Save.

Then give it about ten minutes before you expect to see anything.

2. Add OpenTelemetry to your agents

AgentCore uses OpenTelemetry, an open standard for recording traces, through the AWS Distro for OpenTelemetry (ADOT).

The setup depends on where your agent runs:

On AgentCore Runtime: the runtime has built-in observability and records basic runtime data for you. To also see the detailed spans from inside your agent (model calls, tool calls, AWS SDK calls), add ADOT to your requirements.txt, and the runtime instruments your code when it starts. You don't need to run anything extra.
Anywhere else (your laptop, a container, a VM): add ADOT too, and start the agent through the OpenTelemetry wrapper yourself.

aws-opentelemetry-distro
strands-agents[otel]

If you run the system on your own machine while testing, start it like this:

export AGENT_OBSERVABILITY_ENABLED=true
export OTEL_PYTHON_DISTRO=aws_distro
export OTEL_PYTHON_CONFIGURATOR=aws_configurator

opentelemetry-instrument python agent.py

3. Trace every request while you're developing

Two separate settings decide whether you see a test request, and it's easy to mix them up:

Sampling decides which requests are traced at all.
Indexing decides which traces become searchable trace summaries. By default, Transaction Search indexes 1% of spans as trace summaries for free.

With a low sampling rate and only a handful of test requests, yours may never be traced, and it looks as if nothing happened.

While you're developing, set your sampling rate to 100% so every request is captured. You can keep a lower indexing percentage if you like. Before production, review both settings, because tracing and indexing more data costs more.

With all of this in place, you'll see traces. But you may still get the "orchestrator plus Bedrock" map I got. Here's why.

Problem 1: Worker agents don't show up on the map

A common multi-agent pattern is agents as tools: each worker agent is wrapped in a tool, and the orchestrator calls the tool when it wants that worker's help.

@tool
def ask_refund_agent(request: str) -> str:
    """Send refund requests to the refund agent."""
    return str(refund_agent(request))

It's a clean design. The catch is that the worker runs inside the orchestrator's process. To the tracing library, it's just more code running in the same service. The worker's model calls show up, but they're filed under the orchestrator. There's no separate refund agent box.

The fix: wrap each worker call in its own named span, and mark it as a call to something downstream.

Two details matter:

The span's kind is CLIENT, which means "I'm calling something else".
The span has a peer.service attribute naming the thing being called. X-Ray uses this name for downstream calls.

Be clear about what this does. Your workers are still running inside the orchestrator's process, so they're logical parts of one program rather than separate services. These spans make each worker call visible and named in the trace. In X-Ray they're recorded as downstream calls, which is usually what puts a separate, named box on the map. How exactly the map draws them depends on how X-Ray interprets the exported spans, so check the result against your own map.

Here's a small helper:

from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode

tracer = trace.get_tracer("orchestrator")


def traced_call(node_name, fn):
    """Run fn as a call to a separate node on the service map."""
    with tracer.start_as_current_span(
        node_name,
        kind=SpanKind.CLIENT,
        attributes={"peer.service": node_name},
    ) as span:
        try:
            return fn()
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR))
            raise

And here's the tool using it:

@tool
def ask_refund_agent(request: str) -> str:
    """Send refund requests to the refund agent."""
    return traced_call("refund-agent", lambda: str(refund_agent(request)))

Do the same for every worker, with a clear and different name for each one: orders-agent, refund-agent, reply-agent. Each worker call now has its own name in the trace, and a failure is recorded against the right worker.

Problem 2: All the knowledge bases merge into one box

Knowledge base searches use the AWS SDK. When AWS SDK instrumentation is enabled (ADOT does this for boto3), these calls can appear in your traces. But every search goes to the same AWS service, so they can be hard to tell apart. If you search three knowledge bases, you can't easily see which one is slow or failing.

The fix is the same idea: put a named span around each search, so you can see exactly which knowledge base was queried.

bedrock_agent_runtime = boto3.client("bedrock-agent-runtime")


def search_kb(kb_id, kb_name, query):
    return traced_call(
        f"KnowledgeBase:{kb_name}",
        lambda: bedrock_agent_runtime.retrieve(
            knowledgeBaseId=kb_id,
            retrievalQuery={"text": query},
        ),
    )

Now each knowledge base gets its own node, such as KnowledgeBase:returns-policy and KnowledgeBase:shipping-policy.

Problem 3: Parallel calls split into separate traces

To save time, I searched several knowledge bases at once using a thread pool. Afterwards, the parallel searches didn't show up under the request at all. Some appeared as small traces of their own.

Here's why. OpenTelemetry keeps track of "which trace am I in?" in something called the context. A new thread starts with an empty context by default. So each search started a brand-new trace instead of joining the request's trace.

The fix: copy the context into each thread.

import contextvars
from concurrent.futures import ThreadPoolExecutor

KNOWLEDGE_BASES = [
    ("KB_ID_1", "returns-policy"),
    ("KB_ID_2", "shipping-policy"),
    ("KB_ID_3", "premium-benefits"),
]

with ThreadPoolExecutor() as pool:
    futures = [
        pool.submit(contextvars.copy_context().run, search_kb, kb_id, kb_name, query)
        for kb_id, kb_name in KNOWLEDGE_BASES
    ]
    results = [future.result() for future in futures]

contextvars.copy_context() takes a copy of the current context, including the trace, and .run starts the search inside that copy. Notice that the list makes a new copy for each task. Keep it that way: don't make one copy and share it between threads.

Each parallel search now joins the same trace, and the map shows the knowledge bases branching off together.

Problem 4: Agents in separate runtimes don't connect

If some of your agents run in separate AgentCore runtimes rather than as tools, a different problem appears. Each agent shows up, but as an island with no line between them.

The caller has to pass the trace along in the request headers (traceparent or X-Amzn-Trace-Id). If it doesn't, the receiving agent starts a new trace.

If you call the other runtime with boto3 (invoke_agent_runtime), the instrumentation adds the header for you.
If you call it over plain HTTP, add the header yourself:

import requests
from opentelemetry.propagate import inject

headers = {"Content-Type": "application/json"}
inject(headers)  # adds the current trace to the headers

requests.post(WORKER_URL, json=payload, headers=headers)

inject() writes the header formats your global propagator is set up for. ADOT configures this for you. If you set up OpenTelemetry yourself, make sure the propagators match what the receiving side expects, for example OTEL_PROPAGATORS=tracecontext,xray for both traceparent and X-Amzn-Trace-Id.

Two small things that cost me time

Traces take a moment to appear. Give it 30 to 60 seconds after a request before you decide it didn't work.
Check the time range and region. Open CloudWatch → X-Ray traces → Service map, pick the region your agents run in, and choose Last 5 minutes. An old time range shows an old map.

Check that it worked

Send a few requests that should take different routes. For example:
- a refund request, which should go orchestrator → orders agent → refund agent → reply agent
- a policy question, which should go orchestrator → several knowledge bases in parallel → reply agent
- a simple question, which should go straight from the orchestrator to the reply agent
Wait about a minute.
Open CloudWatch → X-Ray traces → Service map and choose Last 5 minutes.

You should see the orchestrator in the middle, a named connection to each worker agent, and each knowledge base search called out separately. Open a single trace too. One request should be one trace, with every worker and knowledge base search inside it.

[Screenshot: service map before the fixes]

[Screenshot: service map after the fixes]

The map also works as a test. If a simple question shows a line to the refund agent, your routing is wrong, not your tracing.

Quick checklist

Transaction Search is on, and you waited about ten minutes.
OpenTelemetry (ADOT) is in your requirements, and agents outside AgentCore Runtime are started with opentelemetry-instrument.
Sampling is at 100% while you're developing (indexing is a separate setting).
Each worker-agent call is wrapped in a CLIENT span with its own peer.service name.
Each knowledge base search has its own named span.
Parallel work copies the context into each thread.
Calls between separate runtimes pass the trace in their headers.
You're looking at the right region, with Last 5 minutes selected.

The takeaway

A service map can only draw what your traces describe. Auto-instrumentation records a lot, but it can't know that a function call is really "the refund agent", or that three searches belong together.

So if your map looks simpler than your system, it doesn't mean your system is broken. It means you need to tell the tracer where the boundaries are: name each agent, name each knowledge base, and carry the trace along wherever your code runs.

Have you fought with traces in your own multi-agent system? Tell me what tripped you up in the comments.