MCP Tool Search: Does the Model Pick the Right Tool Out of 738?
Readers asked whether 738 connected tools make the model pick the wrong one. I ran 71 tasks three times, twice with every server connected and once with 12 servers blocked. In 189 scored picks it never took a tool from t
Readers asked whether 738 connected tools make the model pick the wrong one. I ran 71 tasks three times, twice with every server connected and once with 12 servers blocked. In 189 scored picks it never took a tool from the wrong server. It took the shell 19 times.
With 738 MCP tools connected, Claude Code picked a tool from the wrong server 0 times in 189 scored tries. It went to the shell instead of a connected server 19 times. MCP tool search is what makes that many tools usable: the client keeps almost all full definitions out of the context window, shows the model the tool names, and loads a definition only when the model asks for it. The tools come from 19 servers. Readers of my last two articles asked whether that many make the model pick the wrong one. I measured it.
The question
Ten of my servers offer tools whose names start with "search", 25 tools in all. Three offer an image generator. A reader suggested that past some count the model starts choosing a plausible neighbour in place of the right tool. I had planned to disconnect the 12 servers I did not call in 30 days, which removes 284 tools and 10 of the 25 search tools. Another reader asked me to take the "before" numbers first. A third told me how to read them: run the unchanged setup twice, because the tasks whose pick differs between two identical runs are the noise floor, and then look at the direction of the tasks that flip after the change. In a two-sided sign test at 5%, the smallest result that passes is six flips, all one way.
The test
The set has 71 tasks in plain words: "Find the emails from last month that mention an invoice", "Which notes in my vault are tagged #project?", "Find open issues about hydration errors in vercel/next.js". For 63 of them the right tool was written down before the first run: one tool, a short list of acceptable ones, or for seven tasks any tool of the right server. All 63 point at the seven servers I keep, and 24 of them at a search tool. The other eight are open tasks with no single right answer, such as "Find the meeting where we discussed hiring".
The rule is strict. Only the first call counts, and only what was written down is right. A shell command that would have answered the task is scored as a miss.
Each task ran in its own fresh session of Claude Code 2.1.325 with the default model on my account, Claude Opus 5.5, on November 22, 2026. The session started in an empty subfolder of one of my projects, so my usual instruction files were loaded. A hook let tool search through and denied every other call, and the session stopped there. The first such call in the session journal is the pick. No tool was executed: no mail was read and nothing was created.
There were three passes: two with all 19 servers, and one with the tools of the 12 idle servers denied with --disallowedTools. Those servers stayed connected and their instructions stayed in the window, so this is blocking, not a real disconnect. That is 213 sessions. In every one of them all the expected servers were connected.
The result
| Pass | MCP tools | Right tool, of 63 |
|---|---|---|
| All servers, first run | 738 | 53 |
| All servers, second run | 738 | 55 |
| 12 servers blocked | 454 | 53 |
Between the two identical runs, two tasks changed from wrong to right. That is my noise floor, from a single pair of runs. Between the second run and the blocked run, four tasks changed: one from wrong to right and three from right to wrong. Against the first run it is also four, two each way. Four mixed flips next to a floor of two is not a result.
So on this set I could not detect a change from removing 284 tools. That is not proof of no change. The test can only see a shift of about six tasks out of 63, close to ten points. Anything smaller is invisible in it.
What the misses were
Across the three passes there were 189 scored picks and 28 misses. I sorted them by what the model called instead.
| What the model called | Misses |
|---|---|
| A tool from another MCP server | 0 |
| Another tool on the right server | 4 |
| A built-in tool of the client | 24 |
Not once in 189 picks did the model take a tool from the wrong server. On the 24 search tasks it was right 20 times in every pass, and none of those misses was a search tool from another server. The four same-server misses are a first step that was not on the list, such as asking for the list of teams before asking for DNS records.
The 24 built-in misses are the finding. In 12 of them the task was about GitHub, the GitHub server was connected, and the model ran the gh command-line tool in the shell. Asked who I am logged in as on GitHub, it ran gh auth status. In seven more it listed or searched my local files with the shell when the task meant a connected service, three of them on a task that did not name the service. In the other five it opened a built-in feature of the client first: its own list of saved pages when the task meant the design tool, three times, and an installed skill before it touched the presentation server, twice.
By the rule these are misses. Several of them look like they would have produced the right answer. I cannot say more, because nothing was executed. The instruction files loaded in these sessions do not mention gh. The client's own description of its shell tool may: I have not checked it.
The competition these misses show is between an MCP server and the shell that can do the same job. The 25 search tools on ten servers did not compete. In these 189 picks the model did not mix up two servers with similar tools. Given the GitHub server and the gh command-line tool, it took the command line in 12 of 48 GitHub picks, in every pass, with 738 tools and with 454. That is a preference, not an error. It matters if you want the work to go through the server, for example because of its permissions.
One caveat. Most of the tasks name the service: mail, vault, a repository. A vaguer task gives the model more room to pick a neighbour, and the open tasks below show it.
How the model finds a tool
The journals suggest why the count mattered so little in this test. In 213 sessions the model made 197 tool-search requests. 196 of them asked for tools by exact name. One was a keyword search. In these sessions the model almost never searched the definitions. It read a list of names, and a name carries the server in it: a mail search and a calendar search do not look alike in that list.
A few tools load in full up front. The meeting-notes search was called with no tool search at all.
This holds for a client that defers definitions and shows names. I did not test a client that loads every definition into the window.
What blocking the servers did change
Two things moved.
The first request of a session carried about 82,000 input tokens, cache included, with every server connected and about 67,100 with the 12 blocked, by the usage the API reported. That is about 14,900 tokens. In my last article the client's own estimate for the same change was 9,300, on a base of 43,300. One number is what the API counted and the other is what the client estimated. I have not reconciled the two.
The open tasks moved as well. The table shows the first call.
| Task | All servers, both runs | 12 servers blocked |
|---|---|---|
| Find the meeting where we discussed hiring | Meeting-notes search | Notes vault search |
| Find my notes about the planning meeting | Meeting-notes search | Shell search in local files |
| What's on my schedule this week? | Calendar | Mail search |
All three went to servers I had not called once in 30 days. With those servers blocked, the model's first move was not to report a missing calendar. It went to search my mail for a schedule. The test stops at the first call, so I did not see the final answer. A server with no calls in a month can still be the right place for a question I have not asked yet, and without it the model reaches for a different tool.
There was only one blocked pass, so the noise in these three is not measured.
The other five open tasks did not move. "Generate an image of a lighthouse at dusk" went to the same image generator all three times: one of three in the full runs, one of two with the servers blocked.
What I take from it
On my setup, tool count has now failed to explain two things. In the last article it did not predict the token cost. Here, on 63 tasks, one model and one day, I could not detect any effect on wrong picks.
On my setup disconnecting idle servers saved tokens, and this test gives no reason against it. It gives one reason for care: a server with no calls can be the only good place for a kind of question. The calendar is one.
The overlap that showed up is the server against the shell. If I want GitHub work to go through the server, the next thing to try is a line in my instructions, or removing one of the two. I have not tested either.
Do this on your setup
The method rests on one hook. It lets tool search through, and stops the session at the first real call. This is a shortened version of the one I ran. The code is mine, use it freely. Keep it in a separate settings file: in your normal settings it blocks every tool.
import json, sys
call = json.load(sys.stdin)
if call.get("tool_name") == "ToolSearch":
sys.exit(0)
print(json.dumps({
"continue": False,
"stopReason": "pick recorded",
"hookSpecificOutput": {
"hookEventName": "PreToolUse",
"permissionDecision": "deny",
"permissionDecisionReason": "measurement run",
},
}))
- Register it as a
PreToolUsehook for every tool in a settings file, and pass that file to a headless session with--settings. - Write the tasks and the right tool for each before the first run.
- Run each task in its own session with
-p --input-format stream-json --output-format stream-json --verbose. Give the servers time to connect, then write the task to the session's standard input. I waited 25 seconds. The first tool call in the output that is not tool search is the pick. - Run the whole set twice before you change anything. Then change one thing and run it again. I blocked the servers with
--disallowedTools.
One pass of 71 tasks took about 10 minutes with four sessions in parallel. Each session is a full model request with every tool list loaded, so a pass uses real quota or API spend. Check your limits first.
Sources
- MCP Tools: I Have 738 Connected and Called 39 in a Month, BESK TECH
- Reader comments with the proposed method, DEV Community
- Reader comments under the earlier article, DEV Community
- Connect Claude Code to tools via MCP, Claude Code documentation
- Hooks reference, Claude Code documentation
Vladimir Beskorovainyi, Enterprise AI Architect & CTO, author of the Amazon bestseller "Enterprise AI: The KATA Way", founder of Besk Tech (besk.tech)
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.