Streaming chat completions from an OpenAI-compatible API: the SSE lines, the usage chunk, tool calls and the usual bugs
Disclosure: I run UberLLM; the examples use its API, but the setup works the same with any compatible provider where noted. Streaming is the right default for anything with a long prompt, long output or a high token cap
Disclosure: I run UberLLM; the examples use its API, but the setup works the same with any compatible provider where noted.
Streaming is the right default for anything with a long prompt, long output or a high token cap: the first words show up quickly, and long answers do not hit request timeouts. It is also where most client bugs live. This post walks through what actually comes over the wire from an OpenAI-compatible /v1/chat/completions endpoint, and the mistakes that show up again and again.
Turn it on
Add "stream": true:
curl -N https://api.uberllm.dev/v1/chat/completions \
-H "Authorization: Bearer $UBERLLM_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek/deepseek-v4-flash",
"stream": true,
"messages": [{"role": "user", "content": "Count to five, one per line."}]
}'
-N turns off curl's buffering so you see events as they arrive. Any OpenAI-compatible provider takes the same body; change the URL, key and model id.
What arrives
The reply is Server-Sent Events: lines starting with data:, each a chat completion chunk, separated by blank lines, and a last line data: [DONE]. Schematically:
: ping
data: {"choices":[{"delta":{"content":"1"}}], β¦}
data: {"choices":[{"delta":{"content":"\n2"}}], β¦}
β¦ more chunks β¦
data: {β¦ "usage": {"prompt_tokens": β¦, "completion_tokens": β¦, "cost_usd": β¦}}
data: [DONE]
Three kinds of line, and a client has to handle all of them:
-
Comment lines starting with
:. UberLLM sends a: pingcomment the moment you connect and keep-alives after that, so a long stream survives proxy idle timeouts. They are not data: skip them. -
data:lines with JSON. The text is inchoices[0].delta.content. -
data: [DONE]. The end. It is not JSON, so do not parse it.
Parsing it by hand
If you are not using an SDK, the loop is short. With Python's requests:
import json, os, requests
resp = requests.post(
"https://api.uberllm.dev/v1/chat/completions",
headers={"Authorization": f"Bearer {os.environ['UBERLLM_API_KEY']}"},
json={
"model": "deepseek/deepseek-v4-flash",
"stream": True,
"messages": [{"role": "user", "content": "Write a short poem."}],
},
stream=True,
)
resp.raise_for_status() # refusals come back as a normal HTTP error before the stream
done, usage = False, None
for raw in resp.iter_lines(): # bytes; decode as UTF-8 yourself
line = raw.decode("utf-8")
if not line or line.startswith(":"): # blank separator or keep-alive comment
continue
if not line.startswith("data:"):
continue
payload = line[len("data:"):].strip()
if payload == "[DONE]":
done = True
break
chunk = json.loads(payload)
if chunk.get("usage"):
usage = chunk["usage"]
for choice in chunk.get("choices") or []: # a chunk may carry no text
text = (choice.get("delta") or {}).get("content")
if text:
print(text, end="", flush=True)
if not done:
raise RuntimeError("stream ended without [DONE]: treat the answer as incomplete")
print("\n", usage)
Things this gets right that hand-rolled parsers often miss: comments are skipped, a chunk with no text (or no choices) does not crash the loop, [DONE] is never passed to json.loads, and a stream that closes without [DONE] is treated as cut off, not finished.
With the OpenAI SDK you get the parsing for free:
from openai import OpenAI
client = OpenAI(base_url="https://api.uberllm.dev/v1", api_key="ull_...")
with client.chat.completions.create(
model="deepseek/deepseek-v4-flash",
stream=True,
messages=[{"role": "user", "content": "Write a short poem."}],
) as stream:
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
Note the trailing /v1 in base_url; without it every call is a 404.
The usage chunk
Token counts in a stream come at the end, in a chunk with a usage object. In the OpenAI format you normally have to ask for it with stream_options: {"include_usage": true}. On other compatible providers, send that yourself where it is supported; in the Vercel AI SDK's OpenAI-compatible provider it is includeUsage: true.
UberLLM adds stream_options: {"include_usage": true} for you when the upstream supports it, and when an upstream sends no usage chunk it adds one before data: [DONE]. The usage carries the prompt and completion token counts and cost_usd, what that request is billed in US dollars.
The usual bug: a loop that only looks at delta.content never sees the usage chunk, so token logging silently records nothing. Read usage from every chunk, as above.
On UberLLM a non-streamed reply also has an X-UberLLM-Cost-USD header. A stream does not, because its headers go out before the cost is known: read usage.cost_usd from the last chunk.
Reasoning models
Reasoning text comes in the delta too. UberLLM sends it in both reasoning and reasoning_content, with the same text, so clients that read either one get it. The bug to avoid: printing both doubles every word. Pick one. A reply with no reasoning has neither field.
Tool calls
Pass tools as usual; streaming does not change the request. Things to get right:
-
Use a model that takes tools. The model list (
GET https://api.uberllm.dev/v1/models) markssupports_toolson each chat model. On UberLLM, tools sent to a model that cannot call them are refused with400 tools_not_supportedbefore anything is charged. -
Wait for the end before running the tool. Tool calls arrive in the
deltain pieces. Do not parse the arguments as JSON until the stream says the call is complete. SDK helpers and frameworks (LangChain, the Vercel AI SDK) assemble the pieces for you. -
Check
finish_reason. A reply that calls a tool ends withfinish_reason: "tool_calls". Some providers have ended tool calls with"stop"instead, and clients that only act on"tool_calls"never ran the tool. UberLLM normalises this so it is always"tool_calls", streamed or not; on other providers, it is worth checking for tool calls whatever the finish reason says. -
Send tool calls back exactly as you got them. Some carry an
extra_contentfield (Gemini's thought signature) that the model needs on the next turn. Rebuilding the message from only the fields you know drops it.
Errors, aborts and retries
-
Before the stream starts, a refusal is a normal HTTP status with an OpenAI-style error body (
{"error": {"code": β¦, "message": β¦}}):402 insufficient_credits,404 model_not_found,429 rate_limited,503 no_available_provider. Check the status before you start reading lines. -
Once streaming has started, nothing is retried. UberLLM retries a call that fails before the first content byte (up to 4 attempts), but a failure mid-stream ends the stream, and you are billed only for what was produced. Your client should treat a missing
[DONE]as a failure and decide whether to retry. - Disconnecting is safe. If you close the connection mid-stream, the gateway cancels the upstream request and settles for the tokens actually streamed.
-
Time to first byte. A stream gets 10 seconds to start (more for a long prompt or a usually slow model); a reply without
"stream": truegets 85 seconds, because nothing arrives until it is complete. Another reason to stream long answers.
The other two surfaces
-
Anthropic format (
/v1/messageswith"stream": true): the same idea in Anthropic's event names.max_tokensis required, and on UberLLM the cost is in theusageof themessage_deltaevent. -
Responses API (
/v1/responses): events have atypesuch asresponse.output_text.delta, and the stream ends withresponse.completed(orresponse.failed, nevercompleted, if it stopped early). The same: pingcomment lines appear while a slow model starts.
Checklist
- Skip lines starting with
:. - Never
json.loadsthe[DONE]line; treat a stream without it as incomplete. - Guard for chunks with no choices or no text.
- Read
usagefrom the last chunk, and ask for it where your provider needsinclude_usage. - Print
reasoningorreasoning_content, not both. - Assemble tool calls before parsing them, and send them back unchanged.
Sources
- UberLLM docs, Streaming: https://uberllm.dev/docs/streaming
- UberLLM docs, OpenAI SDK (tool calls, reply shape, streaming usage): https://uberllm.dev/docs/openai-sdk
- UberLLM docs, Models and routing (retries, time to first byte): https://uberllm.dev/docs/models-and-routing
- UberLLM docs, Errors: https://uberllm.dev/docs/errors
- UberLLM docs, Responses API: https://uberllm.dev/docs/responses
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.