Deploying LLM Models on a Kubernetes Cluster: Best Practices and Examples

Introduction

We are building a support ticket triage agent that runs as a containerized service on Kubernetes. It accepts a customer message, classifies urgency and category, drafts a reply, and returns structured JSON. Because Oxlo.ai uses flat per-request pricing instead of token-based billing, feeding the model long logs or full conversation histories does not inflate costs, which makes it a natural fit for a stateless microservice that scales with demand.

What you'll need

Before starting, make sure you have the following ready.

• Python 3.10 or newer
• Docker and a local or cloud Kubernetes cluster
• kubectl configured to talk to your cluster
• The OpenAI SDK installed: pip install openai fastapi uvicorn pydantic
• An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Define the agent behavior

First, we will write the system prompt that tells the model how to classify tickets and draft replies. Keeping this in a separate constant makes it easy to iterate without touching the application code.

SYSTEM_PROMPT = """You are a support ticket triage agent. Your job is to analyze incoming customer messages and produce a JSON object with exactly three keys:
- "urgency": one of "low", "medium", "high", or "critical"
- "category": one of "billing", "technical", "account", or "general"
- "draft_reply": a polite, concise response that acknowledges the issue and sets expectations

Rules:
- Respond ONLY with the JSON object. No markdown, no explanation.
- If the user mentions downtime, data loss, or security, set urgency to "critical".
- Keep draft_reply under 150 words."""

Step 2: Build the FastAPI service

Next, we will write a small FastAPI application that receives a ticket, calls Oxlo.ai, and returns structured JSON. I am using the OpenAI SDK because Oxlo.ai is fully compatible, so the only change is the base URL. I picked Llama 3.3 70B because it handles tool-like instructions reliably, but you can swap in Qwen 3 32B or Kimi K2.6 without changing any other code.

import os
import json
from fastapi import FastAPI
from pydantic import BaseModel
from openai import OpenAI

app = FastAPI()

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY"),
)

SYSTEM_PROMPT = """You are a support ticket triage agent. Your job is to analyze incoming customer messages and produce a JSON object with exactly three keys:
- "urgency": one of "low", "medium", "high", or "critical"
- "category": one of "billing", "technical", "account", or "general"
- "draft_reply": a polite, concise response that acknowledges the issue and sets expectations

Rules:
- Respond ONLY with the JSON object. No markdown, no explanation.
- If the user mentions downtime, data loss, or security, set urgency to "critical".
- Keep draft_reply under 150 words."""

class TicketRequest(BaseModel):
    message: str

class TriageResponse(BaseModel):
    urgency: str
    category: str
    draft_reply: str

@app.post("/triage", response_model=TriageResponse)
async def triage_ticket(req: TicketRequest):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": req.message},
        ],
        temperature=0.2,
    )
    content = response.choices[0].message.content
    data = json.loads(content)
    return TriageResponse(**data)

Step 3: Containerize the application

Create a requirements file and a Dockerfile so Kubernetes can run the service uniformly anywhere.

Save this as requirements.txt:

openai
fastapi
uvicorn
pydantic

Save this as Dockerfile:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Step 4: Write the Kubernetes manifests

We need a Deployment to run the pods, a Service to route traffic, and a Secret to hold the Oxlo.ai API key. Keeping the key in a native Kubernetes Secret keeps it out of the image and out of source control.

apiVersion: v1
kind: Secret
metadata:
  name: oxlo.ai-secret
type: Opaque
stringData:
  api-key: YOUR_OXLO_API_KEY
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: triage-agent
spec:
  replicas: 2
  selector:
    matchLabels:
      app: triage-agent
  template:
    metadata:
      labels:
        app: triage-agent
    spec:
      containers:
        - name: agent
          image: triage-agent:latest
          ports:
            - containerPort: 8000
          env:
            - name: OXLO_API_KEY
              valueFrom:
                secretKeyRef:
                  name: oxlo.ai-secret
                  key: api-key
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: triage-agent-service
spec:
  selector:
    app: triage-agent
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000
  type: ClusterIP

Step 5: Deploy to the cluster

Build the image, load it into your cluster if you are running locally, then apply the manifests. I am using minikube image load here, but if you are pushing to a registry, replace that step with docker push.

# Build the image
docker build -t triage-agent:latest .

# Load into minikube (skip if you use a remote registry)
minikube image load triage-agent:latest

# Apply manifests
kubectl apply -f deployment.yaml

# Wait for pods
kubectl rollout status deployment/triage-agent

Run it

Forward the service port locally and send a test ticket. The agent should return structured JSON with a drafted reply.

# Port forward in one terminal
kubectl port-forward service/triage-agent-service 8080:80

# Send a test request in another terminal
curl -X POST http://localhost:8080/triage \
  -H "Content-Type: application/json" \
  -d '{"message": "Our production database is down and we cannot process orders. This is urgent."}'

Example output:

{
  "urgency": "critical",
  "category": "technical",
  "draft_reply": "We have escalated this to our infrastructure team immediately. A senior engineer is investigating the database outage now. We will update you within 15 minutes with an ETA for restoration."
}

Wrap-up

From here, you can add a HorizontalPodAutoscaler to scale the deployment based on CPU or custom metrics as ticket volume grows. You can also experiment with other models on Oxlo.ai, such as DeepSeek V3.2 for coding-heavy tickets or Kimi K2.6 for advanced reasoning, without touching your cluster infrastructure.