Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 3 min read

3 Production Strategies to Dramatically Reduce Your LLM Costs

When deploying AI agents and LLM-powered features to production, API bills can balloon surprisingly fast. Most teams overpay for LLM calls not because the models are inefficient, but because they treat every incoming req

When deploying AI agents and LLM-powered features to production, API bills can balloon surprisingly fast. Most teams overpay for LLM calls not because the models are inefficient, but because they treat every incoming request the same.

Whether a user is asking a simple intent classification question or requesting complex multi-step reasoning, every request often gets sent to the exact same expensive flagship model.

If you want to run LLMs sustainably in production without sacrificing output quality, you need a structured approach to optimization. Here is how to systematically reduce your LLM infrastructure costs step by step.

Step 1: Instrument Cost Tracking Per Feature

You cannot optimize what you do not track. Before changing your prompts or swapping models, instrument loggers to track input/output token counts and associated costs per endpoint or feature.

Here is a lightweight Python wrapper using standard OpenAI SDK semantics to track usage per feature tag:

import logging
from openai import OpenAI

client = OpenAI()
logger = logging.getLogger("llm_cost_tracker")

# Example pricing mapping per 1M tokens
MODEL_PRICING = {
    "gpt-4o": {"input": 2.50 / 1e6, "output": 10.00 / 1e6},
    "gpt-4o-mini": {"input": 0.15 / 1e6, "output": 0.60 / 1e6},
}

def call_llm_with_tracking(model: str, messages: list, feature_id: str):
    response = client.chat.completions.create(model=model, messages=messages)
    usage = response.usage

    input_cost = usage.prompt_tokens * MODEL_PRICING[model]["input"]
    output_cost = usage.completion_tokens * MODEL_PRICING[model]["output"]
    total_cost = input_cost + output_cost

    logger.info(
        f"Feature: {feature_id} | Model: {model} | "
        f"Tokens: {usage.total_tokens} | Cost: ${total_cost:.6f}"
    )
    return response.choices[0].message.content

Logging costs at the feature level quickly identifies which specific parts of your application burn through your API budget.

Step 2: Implement Prompt Caching

In agentic workflows and Retrieval-Augmented Generation (RAG) systems, system prompts and retrieved context documents are often large and static. Recomputing these identical tokens on every user query is wasteful.

Modern LLM providers support prompt caching. Cached reads cost a fraction of fresh input tokens, often saving 40-80% on long system prompts or static document contexts.

To leverage prompt caching effectively:

  • Structure your prompts so static content (system instructions, background context, reference documentation) comes first.
  • Keep dynamic content (user query, conversation history) at the very end of the prompt array.

Step 3: Implement Dynamic Model Routing

Not every prompt needs a flagship model. Simple tasks like intent classification, basic JSON extraction, or entity detection can easily be handled by smaller, cheaper models.

Model routing is usually the single biggest lever, often cutting costs by 40-70%. You route routine queries to a lightweight model and reserve the expensive model for hard reasoning tasks.

Here is a production routing pattern using a lightweight classifier before invoking the primary task:

def route_and_execute(user_query: str, context: str):
    # Step 1: Determine query complexity using a cheap model
    complexity_prompt = [
        {"role": "system", "content": "Classify query complexity as 'SIMPLE' or 'COMPLEX'."},
        {"role": "user", "content": f"Query: {user_query}"}
    ]

    complexity = call_llm_with_tracking(
        model="gpt-4o-mini", 
        messages=complexity_prompt, 
        feature_id="routing_classifier"
    ).strip().upper()

    # Step 2: Route to the appropriate model based on classification
    selected_model = "gpt-4o" if "COMPLEX" in complexity else "gpt-4o-mini"

    task_prompt = [
        {"role": "system", "content": "You are a helpful technical assistant."},
        {"role": "user", "content": f"Context: {context}\n\nQuestion: {user_query}"}
    ]

    return call_llm_with_tracking(
        model=selected_model, 
        messages=task_prompt, 
        feature_id="core_task"
    )

By adding a fast, low-cost classifier step, a large portion of routine traffic can bypass the flagship model entirely.

Step 4: Prompt Compression and Batching

Once tracking, caching, and routing are live, refine payload sizes and execution timing:

  • Prompt Compression: Trim verbose system instructions, strip unnecessary historical messages from conversation memory, and remove redundant formatting tokens.
  • Batch Processing: For non-realtime background workloads (such as nightly document indexing or batch evaluation), use provider batch APIs, which typically offer a 50% discount on token rates.

By applying this structured sequenceβ€”instrumenting cost tracking first, followed by caching, routing, and compressionβ€”you can optimize production LLM workloads effectively without dropping output quality.

About me

Tanmay builds AI agents and automation for businesses and is open to freelance projects. Feel free to connect with me on LinkedIn.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.