An AI agent believes whatever its tools send back

It asks a tool for data, gets JSON, and acts on it. If that JSON says a price is "twelve" where a number was promised, the agent doesn't stop to argue. It carries on with whatever it got.

For the last few weeks I've been working on one question: how do you check structured output, with plain code, before anything acts on it? My first two open-source fixes came out of that question, a few days apart, in two different projects. On their own they look unrelated. Together they taught me one thing I now apply everywhere:

Check what comes back. Then check the thing doing the checking.

Part 1: nobody was checking what the tool returned

MCP (Model Context Protocol) is the standard way for an AI agent to talk to tools. A tool can publish an output schema, a description of what its result will look like: these fields, these types, these are required. The spec is clear that a server must return results that match it.

I was reading the Quarkus MCP Server code to see how it handled this. It built the schema from the Java return type and sent it to the agent. Then, when the tool ran, it turned the result into JSON and sent that back too. Nothing compared the two.

That gap is easy to fall into. The schema is generated by one library and the JSON is written by another. A naming setting, an ignored field, a custom serializer, and the two quietly disagree. The agent was told one shape and handed another.

Here is what that looks like from the agent's side. The tool promised this:

{
  "type": "object",
  "properties": { "amount": { "type": "number" }, "currency": { "type": "string" } },
  "required": ["amount", "currency"]
}

and sent this:

{ "amount": "12.50", "currency_code": "EUR" }

A string where a number was promised, and a field under a different name. Nothing failed. The agent got it as if it were fine.

Fail by default, not log by default

I opened an issue first. The maintainer asked a fair question: what do the other MCP SDKs do? So I read their code. The TypeScript, Python and Java SDKs all check the result against the schema, and all return an error when it doesn't match. None of them has a switch to turn it off.

That settled the most important design question for me. A check that only writes a warning to a log is a check nobody reads. So the default became fail: if the result doesn't match, the agent gets a tool error that names what went wrong, something like

Output validation error: Invalid structured content for tool price: [/amount: ...]

instead of the bad data. An agent can handle a clear error. It can't handle data that looks right and isn't.

There is still a setting for teams who need time to adjust:

# fail (default) | log | off
quarkus.mcp.server.schema-validator.tool-output=fail

Because this changes behaviour for existing tools, it is listed as a breaking change in the release notes, not hidden among new features. That was the maintainer's call in review, and he was right: if you change what happens on a mismatch, people deserve to be told plainly.

The review also caught something I hadn't thought about: two MCP servers in one app can each have a tool with the same name, so the check has to know which server's schema it is looking at. Small detail, real consequence. The PR was merged on November 6, about a day after I opened it.

Part 2: the checker itself was wrong

Every check like this rests on a JSON Schema validator. If the validator says yes, we treat the data as good.

I build an open-source Java library for checking structured LLM output, Outputgate, and its schema check uses networknt's json-schema-validator, a widely used Java validator. So I wanted to know how much I could trust it.

I ran it against the newest version of the official JSON Schema test suite, the shared set of cases every validator is meant to pass. Some of the hostname tests failed.

The hostname format was checked by one regular expression. It looked at which characters were allowed and stopped there. So it said yes to:

• names longer than 253 characters, which the rules for DNS names don't allow;
• international names in their encoded form (labels that start with xn--) that don't decode to anything valid, like xn--X.

Hostnames sound like a niche corner. But think about where they show up in structured output: URLs, email domains, the target of a webhook, the server an agent is told to call. If a field is marked as a hostname and the validator waves through anything that starts with xn--, then "it passed the schema" means less than you think.

A million names against a reference

The fix itself is short to describe: enforce the length limits, and for encoded labels, decode them and apply the rules for international names (which characters are allowed, which can appear only next to certain others, how right-to-left scripts must be arranged).

Getting it right was not short. It took eight rounds of review with the maintainer, and every round found something real. One example: Java's built-in decoder follows the 2003 version of these rules, so it rejects names the newer rules allow, such as names with the German ß. I had to write a small decoder that follows the current standard instead.

What changed the outcome was testing differently. At first I checked a sample of about 2,600 names, and it passed. Then I generated every assigned Unicode character in every context the rules care about, about 460,000 labels, and compared my answers with Python's reference library for international domain names. That run found seven differences the sample had missed completely. By the end, the comparison covered about a million labels and names, and the maintainer ran another 1.2 million random names on his side.

I also overcorrected once. In one round I applied a rule for international names to plain hostnames too, and it started rejecting names that are perfectly valid. The reference comparison didn't catch it, because I had copied the same wrong expectation into it. The maintainer did. Lesson: when you make a check stricter, check which standard that specific field follows, and make sure your test isn't sharing the check's assumptions.

The fix was merged on November 7. Outputgate still bundles an older version of the validator, so upgrading it is on my list.

What the two fixes have in common

When an agent works, data passes through layers:

1. The model produces output.
2. A tool sends a result back.
3. A validator decides whether either of those is acceptable.

Most guardrail advice is about layer 1. My two fixes were in layers 2 and 3, and both were the same kind of problem: something that looks like a check, isn't fully checking, and fails quietly. No exception, no log line, just data that's treated as good.

Here's what I do differently now:

• Fail closed. If a check can't confirm the data, the answer is no. A warning in a log is not a check.
• Check at the boundary where data enters. Tool results are input to the agent, the same as user input. Treat them that way.
• Test the checker with a reference, at scale. A sample of a few thousand cases told me everything was fine. A million cases told me otherwise.
• Don't let the test share the check's assumptions. My one regression got through because the expected answers were wrong in the same way the code was.
• Make the change visible. If a stricter check will start rejecting data that used to pass, say so as a breaking change.

A JSON Schema tells you the output has the right shape. It doesn't tell you the output is right, and it only tells you the shape is right if the validator is right. I now ask that question about every check in the path, including the ones I didn't write.

When did you last check the checker in your own pipeline?

Links

• Quarkus MCP Server, tool results checked against their output schema: https://github.com/quarkiverse/quarkus-mcp-server/pull/1127
• networknt json-schema-validator, stricter hostname format: https://github.com/networknt/json-schema-validator/pull/1377
• Outputgate, deterministic checks for structured LLM output in Java: https://github.com/ravismash/outputgate