I cut my AI agent's reading by 78%. It lost 2 answers to a missing folder name.

TL;DR

• A context-shrinking MCP tool cut what my agent sent to the model by a median 78%, on every question.
• It answered 7 of 10 correctly, against 9 of 10 with full files.
• Every miss traced back to one small detail: a file shown without its folder.
• Along the way, I found a bug the founder fixed within two days, and a flaw in my own test.

Coding agents read a lot of code. Every file you paste into the prompt costs tokens, time and money, so a new kind of tool has appeared: send it your files, and it hands the model a trimmed view instead.

Showing that the token count goes down is easy. The harder question is the one that matters:

Does the agent still get the right answer?

I picked one of these tools, Hexum, a hosted MCP server, and set up a fair fight to find out.

The fight

I wrote 10 questions about real open-source Python and TypeScript libraries: things like "which function calls this helper?" and "what does this function return for an empty input?"

Then I asked Gemini each question three ways:

1. Full files. Everything goes into the prompt.
2. Hexum. The model gets Hexum's trimmed view, plus a tool to expand any function it wants to read in full.
3. No files at all. Just the question, to see how much the model already knows from memory.

To keep myself honest, I froze the grading rule, the prompts and all 10 questions in a git tag before the first real call, and checked every answer key against the source code by hand. That way, I couldn't move the goalposts after seeing results.

The rule was simple: Hexum counts as "not worse" if it gets at most one question fewer right than the full files.

The results

On tokens, it's a clear win. Hexum sent less to the model on every single question, with a median cut of 77.7%, and up to 97% on some.

On correctness, it lost two answers. Under my own frozen rule, "not worse" does not hold.

So the interesting part isn't the score. It's why it lost.

The misses had one thing in common

I went through every failed answer, step by step. All three Hexum misses looked the same.

Hexum's view shows each file with its path. Except, in every miss, the file holding the answer was shown by its bare name:

The answer lived in source/utils/body.ts. The view just said body.ts.

So the agent guessed. It looked for the function in source/core/Ky.ts, got back NOT_FOUND with no hint, tried again, and ran out of steps. The same thing happened with recipes.py and core.py in the other two misses.

One missing folder name, and the agent was lost. A full path in the label, or a hint in the error message, would likely have fixed at least some of these.

Twist 1: the tool forgot my code

Before any of this, my first pilot run showed something stranger.

When I asked Hexum to shrink the same files a second time, even in a brand new session, it returned an empty view. It remembered what it had already sent and skipped it, so the agent saw no code at all.

I wrote up an exact reproduction (the arguments, the verbatim responses, the timestamps) and sent it to the founder, Vijay Harre. He shipped a fix within two days. I confirmed it with a probe, restarted the whole run on the fixed server, and kept the earlier run in the repo, labelled "before the fix".

Twist 2: my own test was part of the problem

It would be easy to stop here and blame the tool. But my harness had flaws too:

• The agent never saw the file list. In the Hexum arm, it only saw the view. An agent working inside a real editor would know every path in the project.
• My harness blocked one of Hexum's chances. In one miss, the agent tried to expand recipes.py, the exact name Hexum had shown it. My harness only accepted full paths, so it refused the call before it ever reached Hexum. Whether Hexum would have found the file, I don't know.

I also caught a bug in how my harness handled Hexum's NOT_FOUND errors partway through the run. It was fixed before the affected question was rerun, and it's documented in the repo.

5 things I'd check before trusting any context-shrinking tool

1. Measure correctness next to tokens, on the same questions. A token chart alone tells you nothing about whether the answers survived.
2. Run a no-files baseline. The model got 3 of 10 right with no code at all. On famous libraries, part of any "success" may just be memory.
3. Look at what the agent can navigate. Here, one missing folder name decided the outcome. Check paths, labels and error messages.
4. Count tokens on both sides. Hexum itself received the full files on every call. Fewer tokens to the model is not automatically a lower bill, and I didn't measure cost.
5. Freeze your rules before you look. Otherwise it's too easy to "fix" the test until it says what you hoped.

What this doesn't show

It's 10 questions, one model and one run: a signal, not a verdict. I didn't test edits, Hexum's other tools, or an agent that greps and reads a few files instead.

Everything is public

The harness, the frozen rules, every raw run record and the full results are on GitHub: https://github.com/sravya520/hexum-eval

If you've measured accuracy with a context tool yourself, I'd like to hear what you saw. Did your agent lose answers too?