Structured Output (JSON Mode)

Roadmaps › AI Engineering

Scenario

Your pipeline asks GPT-4o to extract name, date, and amount from invoices. It works in testing. In production, 3% of responses have prose sentences mixed with JSON, or the key is 'Date' instead of 'date', and your code crashes. Before you read on, predict: is JSON mode enough to prevent this?

Think about what JSON mode actually guarantees — then read on to learn the difference.

Mental model

JSON mode guarantees parseable JSON. Structured outputs (with a schema) guarantee your exact schema. They are two different features.

LLMs are text generators — without constraints, they produce whatever text is most probable. Sometimes this includes a preamble, a trailing comment, or a misspelled key. JSON mode constrains the output to valid JSON syntax but does not enforce your field names or types. Structured outputs use constrained decoding to guarantee the response matches a JSON Schema exactly — right fields, right types, every time.

Deep dive

Basic JSON mode: pass `response_format={'type': 'json_object'}` and instruct the model in the system prompt to return JSON. The API guarantees the output will be parseable with `json.loads()` — no trailing text, no markdown code fences. What it does NOT guarantee: your specific keys exist, values have the right types, or required fields are present. An empty object `{}` satisfies JSON mode.

JSON mode = parseable JSON guaranteed. Schema correctness is NOT guaranteed.

Structured outputs (GPT-4o and later): pass your JSON Schema in `response_format={'type': 'json_schema', ...}`. The model uses constrained decoding — at each token step, it can only generate tokens that keep the output valid against your schema. Schema violations are structurally impossible, not just unlikely. Required fields will always appear. Types will always match.

Structured outputs = schema compliance guaranteed via constrained decoding.

In Python, the cleanest approach is to define a Pydantic model and let the SDK generate and validate the schema for you. `client.beta.chat.completions.parse()` accepts `response_format=YourPydanticModel` and returns `response.choices[0].message.parsed` — already a validated typed Python object, not a string. If the model cannot satisfy the schema, `parsed` is `None` and the refusal reason is in `message.refusal`.

Use Pydantic + `.parse()` for the full pipeline: prompt → guaranteed schema → typed Python object.

When to use what: **JSON mode** for simple cases where any valid JSON is fine and you validate downstream. **Structured outputs** for production pipelines where schema correctness is required. **Plain-text prompting for JSON** ('return JSON with keys x, y, z') for quick prototyping only — it fails in production on edge cases at the worst time. Pydantic + structured outputs is the production standard.

Prototype: prompt for JSON. Production: Pydantic + structured outputs. JSON mode is the middle ground.

Code examples

JSON mode — guarantees valid JSON, not your schema

import json
from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[
        {'role': 'system', 'content': 'Extract invoice data. Return JSON with keys: vendor, date, total_usd.'},
        {'role': 'user',   'content': 'Invoice from Acme Corp dated 2024-03-15, total $1,240.00'},
    ],
    response_format={'type': 'json_object'},
    max_tokens=200,
)
data = json.loads(response.choices[0].message.content)
print(data)  # WARNING: keys and types are not guaranteed — validate before using

JSON mode guarantees `json.loads()` will not raise. It does NOT guarantee 'vendor' exists or that 'total_usd' is a float. You still need downstream validation. Always describe the schema in the system prompt as guidance.

Structured outputs with Pydantic — the production pattern

from pydantic import BaseModel
from openai import OpenAI
client = OpenAI()

class Invoice(BaseModel):
    vendor: str
    date: str           # ISO 8601 format
    total_usd: float
    line_items: list[str]

response = client.beta.chat.completions.parse(
    model='gpt-4o-2024-08-06',    # structured outputs requires this version or later
    messages=[
        {'role': 'system', 'content': 'Extract invoice data.'},
        {'role': 'user',   'content': 'Invoice from Acme Corp, 2024-03-15, $1,240. Items: consulting (800), hosting (440).'},
    ],
    response_format=Invoice,       # SDK derives the JSON Schema from the Pydantic model
)

invoice: Invoice = response.choices[0].message.parsed
print(invoice.vendor)      # 'Acme Corp' — typed Python object, not a dict
print(invoice.total_usd)   # 1240.0

The SDK converts the Pydantic class to a JSON Schema automatically. Constrained decoding ensures every field exists with the right type. `.parsed` returns a typed Python object — no `json.loads()`, no `KeyError`, no type coercion bugs. This is the pattern to use in production.

Handle the refusal case

response = client.beta.chat.completions.parse(
    model='gpt-4o-2024-08-06',
    messages=[
        {'role': 'system', 'content': 'Extract invoice data.'},
        {'role': 'user',   'content': 'No invoice here, just a thank you note.'},
    ],
    response_format=Invoice,
)

msg = response.choices[0].message
if msg.refusal:      # model refused — input does not match the schema's intent
    print('Refusal:', msg.refusal)
elif msg.parsed:
    print('Invoice:', msg.parsed)

When the input does not contain data matching the schema (e.g., no invoice to extract), the model may refuse rather than hallucinate values. `message.refusal` explains why; `message.parsed` is `None`. Always handle both branches in production.

Common mistakes

Glossary

JSON mode
A setting that guarantees the model's response will be parseable as JSON — but does not guarantee that the specific fields and types you need will be there.
structured outputs
An advanced feature that uses constrained decoding to guarantee the model's response matches your exact schema — every required field present, every type correct.
constrained decoding
A mechanism that prevents the model from generating any token that would make the output invalid against your schema — schema violations become structurally impossible.

Recall questions

Questions & answers

Your extraction pipeline uses JSON mode and fails in production on 2% of requests with a KeyError. In testing with 50 examples it never failed. What is the root cause and fix?

JSON mode guarantees parseable JSON syntax but not schema compliance. For rare inputs the model may omit a key or use a different name. Testing with 50 hand-picked examples never hit the edge cases. Fix: switch to structured outputs using Pydantic + `.parse()`, which uses constrained decoding to structurally prevent missing keys.

Approach: Name the root cause precisely: JSON mode does not equal schema-valid. The low 2% failure rate is the red flag for 'works in testing, fails in production on edge cases'. Constrained decoding is the correct fix.

A colleague proposes: 'Instead of structured outputs, we'll just ask GPT to return JSON in the system prompt and wrap json.loads() in a try/except.' Why is this fragile in production?

Prompting for JSON fails in several ways at scale: the model adds a preamble or trailing explanation making json.loads() fail; it uses slightly wrong key names; it omits required fields on ambiguous inputs; it returns a string instead of a number. A try/except only catches parse errors — schema errors silently pass and corrupt downstream data. Structured outputs prevent all of these at the generation level.

Approach: List the concrete failure modes: preamble, wrong key names, missing fields, type mismatch. A try/except catches parse errors but not schema errors. Constrained decoding catches all of them.

Continue learning

Previous: Chat Completions API

Next: Function / Tool Calling

Return to AI Engineering Roadmap