Chat Completions API

Scenario

You see three different code shapes for calling GPT in three different blog posts. Before reading further, predict: which field in the response object contains the text you actually want?

Hold your guess, then read on to learn the one canonical shape and the two response fields that matter most.

Mental model

One function call with a `messages[]` list in, one message object out. Every other feature — streaming, tools, structured output — is just one more parameter on top of this single shape.

The chat completions API is the single interface for almost all LLM work: text generation, extraction, classification, agents, and RAG answer generation. It takes a list of role-tagged messages, sends them to a model, and returns the model's reply as another message object. Once you know this shape, every other feature is just one extra parameter.

Deep dive

The request has three required things: a model name, a `messages[]` list, and your API key (passed when you create the client). The messages list is ordered: system first (optional but standard), then alternating user and assistant turns. Every call is stateless — you resend the full history each time. The model only sees what is in this list.

Request = model + `messages[]`. Stateless — send the full history on every call.

The response object has a predictable shape. `choices[0].message.content` is the text you want. `choices[0].finish_reason` tells you why the model stopped: `'stop'` means it finished naturally; `'length'` means it hit `max_tokens`; `'tool_calls'` means it wants to call a tool (and `message.content` will be `None`). `usage.total_tokens` is what you were billed. Ignore everything else for 90% of use cases.

Response: `.message.content` = text. `finish_reason` and `usage.total_tokens` are the two other useful fields.

Key optional parameters: `max_tokens` caps output length — always set this, because uncapped replies can be very long and expensive. `temperature` controls randomness (0 for factual, 0.7 for creative). `n=2` gives you two independent completions for the same prompt, but doubles the cost. `response_format={type: 'json_object'}` asks for JSON output — but does not validate the schema (more on that in the structured-output lesson).

Always set `max_tokens`. Temperature and `response_format` are the two other everyday levers.

Error handling is not optional in production. The API returns HTTP error codes: 400 (your messages list is wrong or over the context limit), 401 (bad API key), 429 (rate limit — wait and retry), 500/503 (server error — retry with backoff). The `openai` SDK raises typed exceptions. Catch them specifically rather than catching all errors silently.

400 = your bug (do not retry). 429 = rate limit (retry with backoff). 500 = server (retry with backoff).

Code examples

The minimal canonical call — learn this shape by heart

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from environment

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'What is idempotency?'}],
    max_tokens=200,
)
text = response.choices[0].message.content
print(text)

This is the shape every other feature builds on. `OPENAI_API_KEY` is read from the environment — never hardcode keys. `max_tokens` prevents expensive runaway replies. The response `.content` is a plain string.

Inspect what you were billed and why the model stopped

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'List 50 countries in Europe.'}],
    max_tokens=100,   # intentionally short to show truncation
)

print(response.choices[0].finish_reason)   # 'length' — hit max_tokens, not done
print(response.usage.prompt_tokens)        # tokens in the input
print(response.usage.completion_tokens)    # tokens in the output
print(response.usage.total_tokens)         # what you are billed

`finish_reason='length'` means the model was cut off mid-answer — raise `max_tokens` or redesign the prompt. `finish_reason='stop'` means it finished naturally. Log `usage.total_tokens` in production to catch runaway costs early.

Production wrapper: error handling and retry logic

import time
from openai import OpenAI, RateLimitError, APIStatusError

client = OpenAI()

def call_with_retry(messages: list[dict], retries=3) -> str:
    for attempt in range(retries):
        try:
            resp = client.chat.completions.create(
                model='gpt-4o',
                messages=messages,
                max_tokens=500,
            )
            return resp.choices[0].message.content
        except RateLimitError:
            wait = 2 ** attempt          # 1s, 2s, 4s
            print(f'Rate limited — waiting {wait}s')
            time.sleep(wait)
        except APIStatusError as e:
            if e.status_code >= 500:     # server error — retry
                time.sleep(2 ** attempt)
            else:
                raise                    # 400/401 — your bug, do not retry
    raise RuntimeError('Max retries exceeded')

Use exponential backoff on 429 and 5xx errors — wait a little longer after each failed attempt. Never retry 4xx errors (except 429) — those are problems with your request that will not fix themselves on their own.

Common mistakes

Not setting max_tokens: Without `max_tokens` the model can generate a very long response. This is slow and expensive. For tasks where you know the output size (classification = 1 word, summary = 3 sentences), cap it tightly.
Reading response.choices[0].text instead of .message.content: The `.text` field is from the old `/v1/completions` endpoint. The chat completions response uses `.message.content`. Reading the wrong field returns `None` or raises an `AttributeError`.

Glossary

stateless: The API does not remember previous calls. You must send the full conversation history every time you make a new request.
max_tokens: A limit you set on how many tokens the model can generate in its reply. Use it to control cost and prevent very long responses.
finish_reason: A field in the response that tells you why the model stopped generating — for example, it finished naturally, hit the token limit, or wants to call a tool.

Recall questions

Explain the chat completions API to a classmate who has never used it before.
What does finish_reason='length' mean and how do you fix it?
Which HTTP status codes should trigger a retry, and which should not?
What happens if you remove max_tokens from your production call?

Questions & answers

A production LLM pipeline occasionally returns None for the response text. Logs show no exceptions were raised. What are two likely causes?

The response text is `None` in two known cases. First: the code is reading `response.choices[0].text` (the old completions field) instead of `response.choices[0].message.content`. Second: the model returned a tool call response (`finish_reason='tool_calls'`), and `.message.content` is `None` when the model wants to call a tool. Always check `finish_reason` before reading content.

Approach: Two classic `None` causes: wrong field name (`text` vs `message.content`) and `tool_calls` finish reason. Check `finish_reason` defensively in production code.

Your pipeline's average response time doubles overnight. Token usage also spiked. What API parameter might explain this?

`max_tokens` was likely removed or raised to a very high value. LLM response time grows with output token count, not just input. Without a cap, some inputs can trigger very long replies, inflating average time and cost. Check that `max_tokens` is set to a reasonable value for your task.

Approach: LLM latency scales with output tokens. Missing `max_tokens` is the most common cause of sudden latency and cost spikes.

Continue learning

Previous: Completion vs Chat Models

Next: Streaming Responses

Next: Structured Output (JSON Mode)

Next: Rate Limits & Retries

Next: Zero-shot vs Few-shot

Return to AI Engineering Roadmap