Reducing AI Agent Token Usage with Deterministic Retrieval: A Real-World Case Study
While working with an AI coding agent, I encountered a simple but interesting optimization problem. Our .NET Core test scenarios depend on predefined database data stored in a large SQL seed file. The file had grown to
While working with an AI coding agent, I encountered a simple but interesting optimization problem.
Our .NET Core test scenarios depend on predefined database data stored in a large SQL seed file.
The file had grown to approximately 40 MB, containing tens of thousands of lines.
The AI agent might need information related to only one particular test scenario, yet searching a very large file can introduce a significant amount of unnecessary context.
That raised a simple question:
Β«Does the AI really need to do the searching?Β»
Moving Deterministic Work Outside the LLM
Instead of relying on the AI agent to search through the large SQL file, I created a lightweight PowerShell retrieval utility.
The workflow became:
40 MB SQL File β PowerShell Retrieval β Relevant Context β AI Agent
The PowerShell utility performs the deterministic work:
- Searches for the required value
- Finds the relevant location
- Extracts only the required section
The AI agent then receives a much smaller amount of information and focuses on what LLMs are actually useful for β understanding the context and reasoning about the problem.
The Result
In my specific scenarios, this approach reduced AI token usage by approximately 80β95%.
This isn't intended as a universal benchmark. The reduction depends on the file, task and agent behavior.
But the experiment highlighted a broader engineering principle for me:
Β«Let deterministic tools do the searching. Let AI do the reasoning.Β»
Why This Matters
As AI coding agents become more integrated into development workflows, optimization shouldn't only be about better prompts or better models.
Sometimes the simplest optimization is reducing what we ask the model to process in the first place.
And PowerShell isn't important to the pattern itself.
The retrieval layer could be implemented using PowerShell, Python, ripgrep, SQL or another deterministic tool.
The same idea can potentially be applied to:
- Large SQL scripts
- Application logs
- JSON/configuration files
- Generated artifacts
- Large repositories
The architecture remains the same:
Large Dataset β Deterministic Retrieval β Minimal Relevant Context β AI Reasoning
For me, that's the key takeaway:
I'm continuing to explore this approach with AI-assisted development workflows and would be interested to hear how others are approaching token and context optimization.
ai #githubcopilot #powershell #dotnet
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.