A 200 OK Is Not a Ready Model: Four Health-Check Myths and a One-Token Probe

My last deploy looked perfect.

Health check green. Container healthy. Dashboard quiet. Then the first real request timed out.

Nothing was actually broken. My check was answering a different question than the one I cared about.

So here is the FAQ I wish I had read first. Four claims developers repeat about readiness. The evidence behind each. The corrected mental model. And a small probe you can copy today.

What was my health check really testing?

It tested the HTTP listener. That's it.

It never touched auth, routing, queue admission, or upstream quota. Those live on the model path, not the health path.

A route that returns a constant cannot fail when the upstream fails. That is not a bug in the route. That is me asking the wrong route.

Myth 1: "If /health returns 200, the model is up."

The claim. Wire /health into the readiness probe. Ship it. Sleep.

The evidence. Ask one question: does this route make a model call? If no, its 200 describes a process, not a capability.

The corrected model. You have two different signals, and they need two different checks:

1. Liveness — is the process serving HTTP?
2. Readiness — can a real request complete right now?

Use the first for restarts. Use the second for traffic.

Myth 2: "A failed request means the endpoint is down."

Not the same thing. At all.

A 429 means the request was rejected. The service is talking to you. A 401 means your config drifted. A timeout or 5xx means something is genuinely broken.

So collapse nothing. Keep three states: ready, degraded, broken.

• 429 with a Retry-After header → degraded. Don't page. Don't hammer it either.
• 401/403 → broken config, not broken service. Fix the secret.
• 5xx, connection error, or timeout → broken. This one wakes someone up.

Is a retry storm going to make a 429 better? No. It makes you the problem.

Myth 3: "Streaming makes responses faster."

It doesn't. It makes the wait observable sooner.

Total wall time barely moves. Sometimes it goes up a little. What streaming gives you is two distinct numbers:

• Time to first token (TTFT) — did the model accept the work?
• Total time — how long until the answer finished?

That split is why your probe should stream. TTFT alone catches a stuck queue. Total time alone hides it.

Myth 4: "Warm-up requests remove cold starts."

They move the cold start. They don't delete it.

Every warm-up pushes the cold path to a different request. Yours, or your user's. Pick deliberately.

Also, a warm-up that sends a long prompt costs more than the check is worth. A readiness probe should be cheap enough to run every minute.

The corrected model. Measure both calls, and record which one you're reporting. First-call and steady-state numbers are different data.

The probe: one short prompt, three exit codes

This is a template. I have not run it against a live endpoint, so treat the paths and headers as placeholders to verify. Fill in PROBE_BASE_URL and PROBE_MODEL from your own account. Never hardcode a model identifier from a blog post, including this one.

# probe_ready.py - synthetic readiness probe (template; verify paths before trusting it)
import asyncio, json, os, time
import httpx

BASE   = os.environ["PROBE_BASE_URL"]
MODEL  = os.environ["PROBE_MODEL"]
KEY    = os.environ.get("PROBE_API_KEY", "")
HEALTH = os.environ.get("PROBE_HEALTH_PATH", "/health")
LEDGER = os.environ.get("PROBE_LEDGER", "ready.jsonl")
TIMEOUT = float(os.environ.get("PROBE_TIMEOUT", "20"))

# Keep the prompt tiny. A probe should cost almost nothing.
PROMPT = "Reply with the single word: ok"


def append(rec):
    with open(LEDGER, "a") as f:
        f.write(json.dumps(rec) + "\n")


async def health(client):
    t0 = time.perf_counter()
    try:
        r = await client.get(BASE + HEALTH, timeout=TIMEOUT)
        return {"ok": r.status_code < 400, "status": r.status_code,
                "ms": round((time.perf_counter() - t0) * 1000, 1)}
    except Exception as e:
        return {"ok": False, "status": None, "err": type(e).__name__,
                "ms": round((time.perf_counter() - t0) * 1000, 1)}


async def completion(client):
    body = {"model": MODEL,
            "messages": [{"role": "user", "content": PROMPT}],
            "max_tokens": 8, "stream": True}
    t0, first, chunks = time.perf_counter(), None, 0
    try:
        async with client.stream("POST", BASE + "/v1/chat/completions",
                                 json=body, timeout=TIMEOUT) as r:
            if r.status_code >= 400:
                await r.aread()
                return {"ok": False, "status": r.status_code,
                        "retry_after": r.headers.get("retry-after"),
                        "first_ms": None, "total_ms": None}
            async for raw in r.aiter_lines():
                if not raw.startswith("data:"):
                    continue
                data = raw[5:].strip()
                if data == "[DONE]":
                    break
                if first is None:
                    first = round((time.perf_counter() - t0) * 1000, 1)
                chunks += 1
        return {"ok": True, "status": 200, "first_ms": first, "chunks": chunks,
                "total_ms": round((time.perf_counter() - t0) * 1000, 1)}
    except Exception as e:
        return {"ok": False, "status": None, "err": type(e).__name__,
                "first_ms": first, "total_ms": None}


async def main():
    headers = {"Authorization": f"Bearer {KEY}"} if KEY else {}
    async with httpx.AsyncClient(headers=headers) as client:
        h = await health(client)
        # Skip the model call if the listener is dead. Save the quota.
        c = await completion(client) if h["ok"] else {"ok": False, "skipped": True}
        rec = {"ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
               "model": MODEL, "health": h, "completion": c}
        append(rec)
        print(json.dumps(rec))
        if c.get("status") == 429:
            raise SystemExit(2)          # degraded
        raise SystemExit(0 if c.get("ok") else 1)   # ready / broken


asyncio.run(main())

Run it once by hand. Then run it on a timer. A single invocation tells you nothing interesting. A week of lines tells you a story.

export PROBE_BASE_URL="https://your-endpoint.example"
export PROBE_MODEL="the-id-shown-in-your-account"
export PROBE_API_KEY="..."
python probe_ready.py; echo "exit=$?"

And read the ledger instead of the dashboard:

jq -r '[.ts, .health.status, .completion.status, .completion.first_ms] | @tsv' ready.jsonl | tail -20

How do I read the output?

Small decision table. Paste it into your runbook.

Two rules I would add. Never page on a single degraded sample. Never treat a probe skip as ready.

Where I run the probe

A probe is useless if it only runs when I remember to run it. So it needs a host, and it needs to be cheap.

MonkeyCode provides free model access and a free server option (operator-supplied). I use both here: the probe runs on the free server on a fixed schedule, and the one-token model calls come from the free model access. That keeps a per-minute cadence off my laptop and off a metered card.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The honest constraints. Free tiers are shared, thresholds and limits can change, and I would not read a latency number from this setup as a capacity claim about anything. The probe measures your path, at your cadence. That is all it measures.

Who should not use this

• You need real latency numbers. A one-token probe cannot tell you p99 at production concurrency. Load test instead.
• Your deploy gate blocks on it. Free-tier variance can flap. Gate on a two-sample window, or don't gate at all.
• Your question is "is the container up?" Kubernetes probes already answer that. Keep them.
• You'd log prompts or keys. Then don't log at all. Log status, timings, and nothing else.

The one thing I changed

I stopped asking "is it up?" and started asking "can it answer right now?"

Those are different questions. They need different checks, different exit codes, and different on-call responses.

What does your readiness check actually send? If it never reaches a model, you already know the answer.