I Kept Paying for the Same AI Answers — So I Built a Gateway That Caches Them

Last month I looked at our OpenAI bill and felt sick. Not because the number was huge — because I recognized the questions. We were paying full price, every single day, for answers the model had already given us.

That was the moment RRC Gateway started.

The three leaks in every LLM bill

After digging through our usage, nearly all the waste came from three places:

1. Duplicate prompts. Support bot, internal tools, eval loops — the same questions phrased slightly differently, answered from scratch every time. "What's your refund policy?" and "Tell me about your refund policy" are two full-price API calls.

2. Bloated context. Our system prompt plus retrieved docs meant every request carried ~100K tokens of context. The model needed maybe 2,000 of them. We paid for all 100K, every call.

3. Premium models doing trivial work. GPT-4o was classifying sentiment, formatting JSON, answering FAQs. Work a model 10x cheaper could do identically.

Individually these look small. Multiplied across thousands of requests a day, they are the bill.

Why I didn't just use the existing fixes

Prompt compression (LLMLingua-style) shrinks tokens but you still pay per call. Manual model routing works until someone forgets and hardcodes the expensive model again. Exact-match caching misses every rephrased duplicate. Each fix solves one leak. I wanted one layer that solved all three, without touching application code.

What I built: a drop-in proxy with three layers

RRC Gateway sits between your app and the providers (OpenAI, Claude, Gemini, Mistral, Llama). You change three lines:

from openai import OpenAI

# Before
client = OpenAI(api_key="sk-...")

# After — everything else stays identical
client = OpenAI(
    api_key="rrc_your_key",
    base_url="https://gateway.rrcsolutions.me/v1",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What's your refund policy?"}],
)

No SDK changes, no rewrites. Behind that URL, three things happen per request:

Semantic caching. The gateway embeds each incoming prompt and compares it against cached ones. "What's your refund policy?" and "Tell me about your refund policy" match at high similarity — the second one never reaches the provider. In live testing, a repeat prompt came back as a cache hit: 116 tokens served, $0.00 charged. A cache hit is the cheapest API call there is — the one you never make.

Virtual memory. Instead of resending your whole knowledge base every call, your docs are compressed into vector embeddings once. Each request retrieves only the ~2,000 tokens the model actually needs. Same answers, roughly 95% fewer context tokens billed.

Smart routing. Every prompt is scored for complexity in real time. Genuinely hard reasoning goes to the frontier model; the FAQ, the classifier, the formatter go to a cheap one automatically. Nobody has to remember to pick — the default stops being the most expensive option.

The math

Take a workload doing 50,000 requests/day against GPT-4o with heavy context. Back-of-envelope, illustrative — your numbers will differ:

Stack them and "up to 90% off the total bill" stops sounding like marketing. It sounds like arithmetic.

Try it

RRC Gateway is in free public beta — no credit card, OpenAI-compatible, three lines to wire up: gateway.rrcsolutions.me

I'm the founder and I read every piece of feedback. If you point it at a workload, I'd genuinely love to hear what your cache hit rate looks like — that's the number I obsess over.