Dev.to AI 🤖 Ai 👁 0 📖 3 min read

The stream died halfway through. Your proxy treated it as a clean stop.

When you stream from an LLM, the HTTP 200 header belongs to the connection, not to the answer. The stream died halfway through. Your proxy treated it as a clean stop. What changed LiteLLM v1.109.0 patched th

When you stream from an LLM, the HTTP 200 header belongs to the connection, not to the answer.

The stream died halfway through. Your proxy treated it as a clean stop.

What changed

LiteLLM v1.109.0 patched three failure modes that sit inside almost every streaming LLM stack.

From the project's tracking, verbatim:

  1. Stalled streams: "stalled streams now explicitly answer with 408 litellm.Timeout." Previously failed as a generic connection error.
  2. Aborted streams: "an upstream stream ending with an abort finish reason could be treated as a clean stop." Now: returns HTTP 500 instead of cleanly terminating.
  3. Unfinished streams: "a stream that outputs token content but never sends a terminating finish_reason (or OpenAI-style [DONE] marker) is treated as an incomplete stream."

Why did those three lines need a patch across three pull requests (PR #47198, #48328, #47665)? Because in a streaming architecture, the most dangerous failure is the one that looks like success.

The mechanics of the silent mid-stream drop

When you call an LLM with stream=True, the transport is Server-Sent Events (SSE) or HTTP chunked transfer.

# Illustrative streaming client call
stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    stream=True,
)
for chunk in stream:
    # Process delta tokens...

Here is what happens on the wire:

  1. The HTTP request is dispatched to your API gateway / reverse proxy.
  2. Upstream provider acknowledges immediately: HTTP/1.1 200 OK, Content-Type: text/event-stream.
  3. The gateway sees HTTP 200. As far as standard HTTP proxies, load balancers, and middleware are concerned, the request succeeded.
  4. The model emits twenty tokens.
  5. Midway through token twenty-one, a GPU worker times out, a provider resets the socket, or a network partition aborts the stream.
  6. The connection closes.

Where the bug lives

Because chunk one already sent 200 OK, no HTTP 502, 503, or 504 was ever emitted.

If the abort or unfinished stream is treated as a clean termination:

  • Retries never fire: Gateway retry policies look for 5xx or 429 status codes. A connection that returned 200 OK and closed is not in the retry list.
  • Fallbacks are bypassed: The proxy will not failover to an alternative model or secondary provider because it thinks the primary succeeded.
  • Downstream consumers break: If the model was streaming a structured tool call, the client receives half a JSON object (illustrative payload):
  {"action": "transfer_funds", "recipient": "vendor_98", "amount": 450

The client's JSON parser crashes (illustrative error):

  json.decoder.JSONDecodeError: Unterminated string starting at line 1 column 67
  • The wrong team gets blamed: An engineer opens an issue: "The model hallucinated invalid JSON." The model did nothing wrong. The model was cut in half by infrastructure, and the transport layer labeled it a clean delivery.

Making mid-stream failure loud

The architectural fix in LiteLLM v1.109.0 is straightforward:

  • An unfinished stream missing its terminal sentinel (finish_reason or [DONE]) is a failure, not a short generation.
  • An upstream abort becomes an explicit HTTP 500.
  • A stalled stream becomes an explicit 408 litellm.Timeout.

The moment those exceptions are raised visibly, the proxy's resiliency machinery wakes up: retries kick in, provider cooldowns activate, and traffic shifts to healthy fallbacks.

The discipline for streaming LLM pipelines

  1. HTTP 200 proves the handshake, not the generation. Never flag a streaming invocation as successful in your telemetry or audit log until the final finish_reason or [DONE] event has been parsed.
  2. Enforce gateway-level stream retries. If your proxy or API gateway only inspects response headers before streaming starts, streaming effectively disables your retry architecture. Your proxy must support retrying on truncated chunked streams.
  3. Never parse streaming tool calls until generation completes. Wait for finish_reason == "tool_calls" or finish_reason == "stop". If generation ended with length or aborted unexpectedly, route it as an infrastructure failure, not a prompt issue.
  4. Distinguish client cancellations from upstream aborts. When a user closes their tab, that is client disconnect (499). When an upstream provider aborts midway, that is provider failure (500/502). Mixing them hides provider degradation.

Two hard rules

A response without a terminal marker is an incident, not an answer.

If your retry middleware only reads HTTP headers, streaming turns off your retries.

Honest limitation: I am reading LiteLLM's public changelog and PR tracking (#47198, #48328, #47665). The truncated JSON snippet and parser error string above are illustrative. I have not benchmarked live failover latency under packet drops.

Checkable references: LiteLLM v1.109.0 release notes ("Stream Failure Handling") and PRs #47198, #48328, #47665.

What this series is: one primary source a day — a spec, a changelog, a status page — and what actually changed, with the quote. Follow if you prefer the change over someone's summary.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.