How I Cut My LLM API Costs by 70% Without Touching My Code
I was staring at my Stripe dashboard in disbelief. $208 for API calls last month. Not my infrastructure. Not my database. Just the LLM APIs powering my side project's features — the ones I'd integrated before I even knew
I was staring at my Stripe dashboard in disbelief. $208 for API calls last month. Not my infrastructure. Not my database. Just the LLM APIs powering my side project's features — the ones I'd integrated before I even knew what I was doing.
The "before" version of my code was simple: every time a user interacted with my app, I fired off a prompt to OpenAI with the full conversation history, all my system context, and a bunch of instructions I'd copy-pasted from random blog posts. It worked. It was also the equivalent of paying premium prices for a buffet and only eating the breadsticks.
I've spent the last three months optimizing that pipeline. The result: I'm now spending around $60/month on the same features, same user base, same output quality. Here's exactly how I did it, with numbers, code snippets, and no marketing fluff.
My cost breakdown (the embarrassing part)
Before we get into fixes, let's look at what was bleeding money:
- ~120,000 API calls per month
- Average 1,800 tokens per call (mostly repeated system prompts and conversation history)
- Mixed usage across GPT-4, GPT-3.5, and Claude models
- No caching. At all.
- No model tiering. I was using the most expensive model for everything, including extracting a zip code from a string.
The math was brutal: my effective cost per 1K tokens was way above what it should have been because I was sending massive payloads to a flagship model for trivial tasks.
Step 1: I stopped sending the whole kitchen sink in every request
Here's the thing I finally understood: the system prompt is your biggest silent cost driver. Every token you send gets billed. Every token you send repeatedly gets billed repeatedly.
My original system prompt was 400 tokens of context: "You are a helpful assistant for a project management tool. You have access to the following features..." followed by 800 tokens of instructions, examples, and edge-case handling.
I was sending that 120 times per day on average. That's 1,200 tokens of static overhead per request just for instructions.
The fix: I moved to a "system prompt once, user context forever" pattern. I now inject the system prompt only once per conversation thread (not per message) and keep the instruction overhead minimal in every subsequent call.
# Before: sending everything every time
response = client.chat.completions.create(
model="gpt-4-turbo",
messages=[
{"role": "system", "content": LONG_SYSTEM_PROMPT},
{"role": "user", "content": user_message}
],
temperature=0.3
)
# After: concise system prompt + context switch
SHORT_SYSTEM_PROMPT = "You are a project management assistant. Be concise."
response = client.chat.completions.create(
model="gpt-4-turbo-mini",
messages=[
{"role": "system", "content": SHORT_SYSTEM_PROMPT},
{"role": "user", "content": user_message}
],
temperature=0.3
)
I also started truncating conversation history intelligently. Instead of sending the last 30 messages, I now send a rolling summary (generated once per new session) plus the last 5 messages. That alone dropped my average tokens per call from ~1,800 to ~1,300. A 28% reduction in input size across the board equals roughly a 25% cost cut without changing a single prompt's quality.
Step 2: I embraced prompt caching (the easy win I ignored)
This one is almost embarrassing. Both OpenAI and Anthropic have had prompt caching for a while. I just never turned it on because I assumed it was an enterprise feature.
Turns out, caching your static prefix — the system prompt and any repeated context — cuts costs dramatically. OpenAI's caching gives you a 50% discount on cached input tokens. Anthropic's does 90% off on cached prompts.
For my use case, the system prompt was same across all users of a feature. I just had to structure my message array so the prefix stays identical between calls. Once I did that and enabled caching, my input token cost dropped by roughly 37% overnight.
Here's the pattern I use now (simplified):
messages = [
{"role": "system", "content": SHORT_SYSTEM_PROMPT},
{"role": "user", "content": user_query}, # this varies per user
]
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
temperature=0.3
)
That's it. Same prefix, so the provider caches it. It took me fifteen minutes to restructure a few message arrays.
Step 3: I started treating model selection like a shopping list, not a default
I had this mental model: "GPT-4 Turbo for everything, because it's safe." That's the most expensive way to think.
I analyzed my actual tasks:
- ~60% are simple extraction, classification, or formatting (a zip code, a date, a sentiment label)
- ~30% are moderate reasoning (summarize, rewrite, short Q&A)
- ~10% are complex reasoning (multi-step planning, long-form generation)
So now I route by task difficulty. Simple stuff goes to GPT-4o-mini or Claude Haiku. Moderate uses GPT-4o-mini with a slightly better prompt. Complex stuff gets Claude Sonnet or GPT-4o. I set up a tiny router function (nothing fancy — a few if conditions) and instantly cut my spend by 20% just by not paying flagship prices for trivial work.
def pick_model(task_type):
if task_type == "simple":
return "gpt-4o-mini"
elif task_type == "moderate":
return "gpt-4o-mini"
else:
return "claude-sonnet-4-20250514"
No fancy ML routing. Just a dictionary and some logic. The quality hit was non-existent because simple tasks were never hard.
Step 4: The aggregation layer that saved my sanity (and wallet)
This is where things got interesting. I found myself juggling three providers: OpenAI, Anthropic, and occasionally Google. Each had different pricing, different rate limits, different quirks. I wrote my own thin wrapper to handle fallbacks and routing, but maintaining it was a pain.
Then I discovered that some folks use a unified API gateway that aggregates multiple providers behind one endpoint. The idea is you keep your code exactly as-is — same OpenAI-style interface — but the router decides which provider actually executes the request. You get automatic failover, model fallbacks, and price-based routing without rewriting your integration.
I tried a few. The one I currently use is a pay-as-you-go gateway (which I found when comparing prices). It's essentially a single API key that rotates between underlying providers based on cost and availability. And because it bills per-token with no subscriptions or fees, it aligned with my usage pattern perfectly.
It wasn't just about saving money — it also eliminated the risk of hitting rate limits on one provider during a spike. My uptime improved because if OpenAI was throttling me, the gateway would silently route to Claude or Gemini for that request.
What the actual numbers look like now
After all four changes, here's my current breakdown:
- March (before): $208/month, 1,800 avg tokens per call, 120K calls
- June (after): $61/month, 1,300 avg tokens per call, 115K calls (user base grew 15%)
That's a 70% reduction in spend with zero changes to my actual product logic — just message structure, caching, model selection, and routing.
The most satisfying part: my end-user-facing latency actually went down because I'm no longer sending giant payloads to a high-latency flagship model for trivial requests.
What I'd tell myself six months ago
If I could go back, I'd say: stop assuming your prompts are immutable. Treat the API call as a cost center, not a fixed expense. Every token you send is a line item on your bill.
And if you're juggling multiple providers and hate maintaining custom fallback logic, consider a pay-as-you-go gateway like the one I use — it does the provider switching for you and only charges what you actually use. It saved me hours of DevOps work and keeps my costs predictable.
The lesson is simple: you don't need to cut features or downgrade quality to slash your API bill. You just need to stop paying for tokens you're not actually using.
Now my code does the same things, my users get the same responses, and my Stripe bill looks like a bar tab from a modest dinner instead of a tech conference afterparty. That's a win I'll take every month.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.