Deploying LLM Models

We are going to build a deployable support ticket triage API that classifies incoming messages, assigns them to the right team, and drafts a first reply. This is a practical service you can place behind your helpdesk webhook to cut response time without adding headcount. The stack is minimal: Python, FastAPI, and Oxlo.ai via the OpenAI SDK.

What you'll need

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK and a few dependencies: pip install openai fastapi uvicorn pydantic python-dotenv
• Docker, if you want to run the final container locally

1. Project scaffolding

Create a project directory and a virtual environment. I named mine ticket-triage. Inside it, create requirements.txt with the packages above and a .env file that holds your Oxlo.ai API key.

# requirements.txt
fastapi
uvicorn
pydantic
openai
python-dotenv

# .env
OXLO_API_KEY=oxlo_xxxxxxxxxxxxxxxx

2. The Oxlo.ai client and system prompt

The entire service depends on a single external call. Point the OpenAI client at Oxlo.ai and define the system prompt that forces structured JSON output.

SYSTEM_PROMPT = '''You are a support ticket triage agent. Analyze the ticket below and output strictly valid JSON with these keys:
- urgency: one of "low", "medium", "high", "critical"
- team: one of "billing", "technical", "sales", "general"
- sentiment: "frustrated", "neutral", or "satisfied"
- summary: max 20 words
- reply: a concise, professional first response

Rules:
- Output only JSON. No markdown fences, no explanation.
- If the ticket mentions payment failure, refund, or chargeback, set urgency to "high" and team to "billing".
- If the ticket mentions outage, crash, or security breach, set urgency to "critical" and team to "technical".'''

from openai import OpenAI
import os
import json
from dotenv import load_dotenv

load_dotenv()

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY")
)

def triage_ticket(ticket_body: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket_body},
        ],
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

I am using llama-3.3-70b because it is reliable for structured JSON and inexpensive per request on Oxlo.ai. If you expect very long ticket threads, swap in kimi-k2.6 to take advantage of Oxlo.ai's flat request pricing on long contexts.

3. Pydantic models and validation

I validate the LLM output with Pydantic before it touches the rest of the app. This catches malformed JSON or missing fields early.

from pydantic import BaseModel, Field
from typing import Literal

class TriageResult(BaseModel):
    urgency: Literal["low", "medium", "high", "critical"]
    team: Literal["billing", "technical", "sales", "general"]
    sentiment: Literal["frustrated", "neutral", "satisfied"]
    summary: str = Field(..., max_length=200)
    reply: str

def triage_ticket(ticket_body: str) -> TriageResult:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket_body},
        ],
    )
    raw = response.choices[0].message.content.strip()
    data = json.loads(raw)
    return TriageResult(**data)

4. FastAPI server

Wrap the triage function in a minimal FastAPI app with a single POST endpoint. Then start it locally with Uvicorn.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI(title="Ticket Triage API")

class TicketIn(BaseModel):
    subject: str
    body: str

@app.post("/triage", response_model=TriageResult)
async def triage(ticket: TicketIn):
    full_text = f"Subject: {ticket.subject}\n\nBody: {ticket.body}"
    return triage_ticket(full_text)

# Start locally
uvicorn main:app --reload

5. Dockerize for deployment

Add a Dockerfile so you can deploy this anywhere that runs containers. Build the image and run it on port 8000.

FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

# Build and run
docker build -t ticket-triage .
docker run -p 8000:8000 --env-file .env ticket-triage

Run it

Send a test ticket with curl and inspect the structured response. The API returns urgency, team assignment, sentiment, summary, and a draft reply in one shot.

curl -X POST http://localhost:8000/triage \
  -H "Content-Type: application/json" \
  -d '{
    "subject": "Refund not processed",
    "body": "I was charged twice on March 10 and the refund still has not appeared. This is urgent."
  }'

Example output:

{
  "urgency": "high",
  "team": "billing",
  "sentiment": "frustrated",
  "summary": "Customer charged twice, refund delayed since March 10.",
  "reply": "I am sorry for the delay. I have escalated this to our billing team and you will see the refund within 24 hours."
}

Wrap-up

You now have a containerized triage API backed by Oxlo.ai. Wire it into your helpdesk webhook so tickets are routed automatically, and add a Redis cache layer to avoid re-triaging duplicate subjects. If your volume grows, Oxlo.ai's request-based pricing keeps costs predictable even when customers paste hundred-line log files into the ticket body. See https://oxlo.ai/pricing for plan details.