Streaming Responses

Scenario

Your LLM pipeline takes 8 seconds to generate a 400-token answer. Users see a blank screen for 8 seconds, then the full text appears at once. Before you read on, predict: does streaming make the model generate faster, or does something else change?

Think about the difference between generation time and perceived wait — then read on.

Mental model

Streaming is the API sending you tokens as they are generated, instead of waiting to wrap everything up first.

A language model generates text token by token. Without streaming, the API buffers all tokens and sends one big response when generation completes — you wait the full 8 seconds. With `stream=True`, the API opens a long-lived HTTP connection and sends each token the moment it is ready. Your app can render it immediately. Total generation time is identical, but users start reading in about 200 milliseconds.

Deep dive

Without streaming: the model generates 400 tokens over 8 seconds; the API buffers everything; you wait 8 seconds; the full text arrives at once. With streaming: the API sends a Server-Sent Events (SSE) stream; you receive tokens as they arrive — the first token might reach you in 200ms. Total generation time is the same, but users start reading immediately.

Streaming = same compute time, tokens arrive progressively. First token in ~200ms instead of 8s.

Each SSE chunk is a delta — not the full text so far, just the new piece. You accumulate them into a growing string. The stream ends when a chunk's `finish_reason` is `'stop'` (or `'length'` or `'tool_calls'`). Until then `finish_reason` is `None`. The token usage count (tokens billed) is only available on the LAST chunk — you cannot count tokens mid-stream without using the `tiktoken` library.

Chunks are deltas. Accumulate them. finish_reason and usage are on the last chunk only.

**When NOT to stream**: if your pipeline needs the complete answer to decide what to do next — parse JSON, route based on classification, extract citations — streaming adds complexity with no benefit. You would still need to wait for the full text before acting. Stream only for user-facing text output where the typing effect improves perceived responsiveness.

Stream for user-facing output. Skip streaming for pipeline steps that parse the full response.

Code examples

Basic streaming — print tokens as they arrive

from openai import OpenAI
client = OpenAI()

stream = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Explain gradient descent in simple terms.'}],
    max_tokens=300,
    stream=True,    # <-- the only change from a normal call
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:                       # last chunk has delta=None
        print(delta, end='', flush=True)

print()  # newline after stream ends

`stream=True` turns the return value from a response object into an iterator. `delta.content` is the new text piece — it is `None` on the last chunk, which carries `finish_reason`. The `flush=True` makes tokens appear immediately in the terminal instead of buffering.

Accumulate the full text and get token usage

from openai import OpenAI
client = OpenAI()

stream = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'List 5 Python best practices.'}],
    max_tokens=400,
    stream=True,
    stream_options={'include_usage': True},  # request usage in final chunk
)

full_text = []
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        full_text.append(delta)
        print(delta, end='', flush=True)  # stream to user
    if chunk.usage:                        # last chunk only
        print(f'\nTotal tokens: {chunk.usage.total_tokens}')

complete = ''.join(full_text)  # parse or store the complete answer here

`stream_options={'include_usage': True}` appends a final chunk with token counts — otherwise `usage` is `None` for all chunks. This is the production pattern: stream to the user AND log costs by collecting the final `usage` object.

Common mistakes

Trying to parse JSON from a streaming response mid-stream: A streaming chunk is a fragment — a JSON object like `{'role': 'assistant'}` may arrive across three separate chunks. JSON is only valid once the full string is assembled. If you need JSON output, disable streaming or accumulate the full text first, then parse.
Not handling delta.content = None on the final chunk: The last SSE chunk carries `finish_reason` but has `delta.content = None`. Concatenating `None` into your string will raise a `TypeError` or produce the literal string 'None' in the output. Always guard with `if delta:` before appending.

Glossary

streaming: A mode where the API sends the AI's answer to you piece by piece as it is generated, instead of waiting for the full answer to be ready first.
Server-Sent Events: The underlying web technology that keeps a connection open so the server can push new text fragments to your application in real time.
delta: A small fragment of new text that arrives in one stream chunk. You accumulate all deltas together to get the full response.

Recall questions

Explain what streaming does, as if to a classmate.
Does streaming reduce the total time the model takes to generate a response?
When is it better NOT to use streaming?
You enable streaming and your citation-extraction code starts failing. Predict the root cause.

Questions & answers

You enable streaming for your RAG pipeline's answer-generation step. Afterwards, your citation-extraction code fails intermittently. Why?

Citation extraction reads the full text to find patterns like [1] and [2]. With streaming, the code tries to parse the response before the full text has accumulated — the text is incomplete and the parser fails or finds partial citations. Fix: accumulate the full streamed text first, then pass it to the citation extractor. Or disable streaming for this pipeline step entirely.

Approach: Identify the streaming-vs-parsing mismatch. Streaming is for user-facing output; programmatic parsing needs the complete string.

A user sees the streaming output stop mid-sentence with no error shown. What are two likely causes?

First: `max_tokens` was hit — `finish_reason='length'` — the model was truncated before finishing. Fix: increase `max_tokens` or shorten the prompt. Second: a network timeout dropped the SSE connection mid-stream. Fix: implement reconnect logic or catch the stream interruption and retry from scratch. Check `finish_reason` on the last chunk to distinguish between the two.

Approach: Two root causes: model-side truncation (finish_reason='length') vs network-side connection drop. They look the same to the user but need different fixes.

Continue learning

Previous: Chat Completions API

Return to AI Engineering Roadmap