The AI agent wasn't confused. It was drowning in 200 tools.

How I cut wrong answers and token usage by giving the model 5 tools instead of 200.

The ticket, if you can call it that, was one sentence:

Make the AI agent give fewer wrong answers.

No spec. No acceptance criteria. Just an agent that kept picking the wrong action.

I assumed the obvious first: it's a prompt problem. It wasn't.

The overlooked number

Before touching the prompt, I logged what the agent was actually sending to the model on each request.

Every single call carried around 200 tool definitions.

Think about what that means for the model. A user asks something simple, and before it can answer, it has to read the name, description and parameters of 200 tools and pick the right one. Most of them have nothing to do with the question.

That explained two things:

• The wrong answers: The model wasn't bad at reasoning. It was choosing from a menu that was far too long, full of noises.
• The token bill: Most of every request was spent re-sending the same tool list, whether the question needed it or not.

So the real question became: why does the model see tools it will never use?

Attempt one: smarter, but not smart enough

My first idea was to stop sending full tool metadata up front. Tool details moved into a database, and the agent only fetched metadata for the tools the LLM loop decided to call.

First attempt. Metadata is fetched only for the tools the model decides to use.

It helped with token usage. But when I measured, three problems were still there:

1. The count was the real cost, not the size. The model still had to choose among all 200 tools, so most of the overhead remained.
2. Adding a tool still meant a deployment. The tool functions lived inside the agent's runtime, so every new tool was a code change.
3. The wrong answers didn't go away. Of course they didn't: the menu was just as long.

I had optimized the wrong thing. The goal wasn't cheaper tool descriptions. It was fewer tools per decision.

The catch: a database row can't hold a function

The obvious next step was to store the entire tool in the database, not just its description. Then the agent could look up only the tools that matched a request.

The problem: a tool's function was a reference inside the running agent. You can put a name and a description in a row. You can't put a function pointer in one.

The only way around it was to move every tool's implementation out of the agent and behind an endpoint. That's not a refactor you quietly slip into a pull request; it's an infrastructure decision.

Let's say, we can move the tools to external APIs and AWS Lambda. That was the missing piece: once every tool was just "metadata plus an endpoint," a tool could be a database row.

The final design: retrieval for tools

The core idea is simple. Treat tools the way RAG treats documents: retrieve the relevant ones, ignore the rest.

Final design. Tools live in a vector database; each request gets only the few that match.

Here's how a tool gets into the system, and how a request finds it.

Adding a tool (no deployment). Admins add tools through a Control Plane API behind company SSO. When a tool is added, its name, description, example queries and tags are embedded in document mode and stored in a vector database along with its endpoint.

Answering a request.

1. The user's query is embedded in query mode.
2. A vector search finds the tools whose embeddings are closest.
3. Only the top k, usually 4 or 5, are handed to the LLM loop.
4. When the model calls a tool, a tools executor sends the request to that tool's external API through an egress proxy.

In simplified form, the per-request flow looks like this:

// Simplified sketch of the idea, not the production code
async function toolsFor(query: string) {
  const vector = await embed(query, { mode: "query" });
  return vectorDb.search(vector, { topK: 5 }); // 4–5 tools instead of ~200
}

const tools = await toolsFor(userQuery);
const answer = await runLlmLoop(userQuery, [...tools, searchTool]);

The safety net for vague questions. Retrieval only works if the query says enough to match on. "Can you fix it?" doesn't. So the model always has one extra tool available: a search tool. If the matched tools don't fit, the model can call it with a more detailed query of its own, and whatever it finds gets added to the tool set for that request. The agent is never stuck with only the first guess.

What changed

Things I learned

Check what you're actually sending before you tune the prompt. The fix was hiding in the request payload, not in the wording. Five minutes of logging beat days of prompt tweaking.

Fewer choices beat better descriptions. Shrinking each tool's metadata barely moved the needle. Shrinking the number of tools changed everything.

Tools can be retrieved like documents. If your agent has more than a few dozen tools, the problem looks a lot like search, and the same techniques apply.

Always leave an escape hatch. Retrieval will sometimes miss. A fallback search tool lets the model recover instead of guessing.

Building something similar, or fighting an agent with too many tools? I'd like to hear about it: [email protected].