Temperature & Top-p

Scenario

You set `temperature=0` for a customer support bot to get consistent answers. Before you test it, predict: will every identical question always get the exact same answer?

Make your prediction, then read on to find out what temperature 0 actually guarantees.

Mental model

Temperature is the dial between 'always pick the most likely word' and 'pick almost any word'. Top-p is the filter that limits which words are even in the lottery.

At every step, the model assigns a probability to every word in its vocabulary — 'what comes next?' Temperature reshapes those probabilities before the model picks. Low temperature makes the highest-probability word much more likely to be chosen. High temperature flattens the probabilities so less-likely words get a real chance. Top-p (called nucleus sampling) cuts the vocabulary before sampling — keeping only enough top words to cover a chosen fraction of the total probability.

Deep dive

Picture the model choosing the next token from a bar chart — probability on the y-axis, candidate tokens on the x-axis. Without temperature, it would always pick the tallest bar. That would be repetitive and robotic. Temperature changes the bar heights before the pick.

Temperature reshapes the probability bars — low = peaked at one word, high = spread across many.

Mathematically, each raw score (called a logit) is divided by the temperature before the final probability step. `temperature=1` leaves the probabilities unchanged. `temperature=0.2` sharpens them: the top token gains probability, others lose it. `temperature=1.5` flattens them: even low-probability words become real options. You do not need the math — just remember: colder = more predictable, hotter = more varied.

Lower temperature → model sticks to the most likely word. Higher temperature → more variety.

`top_p` (nucleus sampling) is a companion filter. Before picking, sort all tokens by probability from highest to lowest. Keep adding tokens until the total reaches `p`. Discard everything else. `top_p=0.9` means 'sample only from the smallest group of tokens that together account for 90% of the probability.' Very unlikely tokens are never picked, even if temperature is high.

`top_p` cuts the vocabulary to the 'nucleus' — the smallest set summing to probability p.

In practice: **tune one dial, leave the other at its neutral value.** Use temperature for creativity vs precision, and leave `top_p=1` (off). Or use `top_p` as the main control and leave `temperature=1`. OpenAI says not to adjust both at once — they interact in confusing ways. For factual tasks (extraction, classification): `temperature=0–0.2`. For creative writing: `temperature=0.7–1.0`.

Tune one: temperature for precision or creativity; top_p for vocabulary ceiling. Not both at once.

The trap: `temperature=0` is NOT fully deterministic on the OpenAI API. At this setting the model uses greedy decoding (picks the most likely token) but tiny rounding differences on distributed computers can occasionally flip two equally-likely tokens. To get the most consistent results, also set the `seed` parameter. Even then, OpenAI does not guarantee identical outputs across model updates.

`temperature=0` ≈ very consistent, but not perfectly deterministic. Add `seed` for best repeatability.

Code examples

Set temperature for a factual extraction task

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[
        {'role': 'system', 'content': 'Extract the total. Reply with just the number.'},
        {'role': 'user',   'content': 'Invoice #4821: subtotal 120, tax 10.80, total 130.80.'},
    ],
    temperature=0.0,
    seed=42,
)
print(response.choices[0].message.content)  # '130.80'

For extraction and classification tasks, use `temperature=0` to maximize consistency. Add `seed` to improve repeatability — but note that a model update may still change outputs.

Higher temperature for creative brainstorming

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[
        {'role': 'system', 'content': 'You are a creative product-naming assistant.'},
        {'role': 'user',   'content': 'Suggest 5 names for a meal-planning app for gym-goers.'},
    ],
    temperature=0.9,   # more variety — different names each run
    top_p=1.0,         # top_p off (1 means keep everything)
)
print(response.choices[0].message.content)

For brainstorming, a higher temperature gives more varied options across runs. Setting `top_p=1` keeps it out of the picture so you are changing only one dial.

Common mistakes

Believing temperature=0 is fully deterministic: Tiny rounding differences on distributed computers and model updates mean `temperature=0` only maximizes consistency — it does not guarantee identical outputs every time. If exact repeatability matters, also set `seed` and pin the model version.
Adjusting both temperature and top_p at the same time: They interact: high temperature spreads probabilities, then `top_p` cuts the spread. Low temperature narrows probabilities so `top_p` cuts almost nothing. Change one, leave the other at its neutral value, so you know what you changed.

Glossary

temperature: A number that controls how creative the AI is. Low temperature makes it pick the most likely word. High temperature makes it pick less likely words too.
top-p: A filter that keeps only the most likely words before the AI picks one, removing the very unlikely options.
greedy decoding: What happens at temperature 0: the AI always picks the single most likely next word, with no randomness at all.

Recall questions

Explain temperature to a classmate who has never heard of it.
What does `top_p=0.9` mean in plain terms?
Is `temperature=0` fully deterministic on the OpenAI API?
What happens if you set `temperature=1.5` for a medical information bot?

Questions & answers

You're deploying a medical info bot. A colleague sets temperature=1.5 to 'make it friendlier'. What problems does this introduce?

High temperature makes unlikely — and possibly incorrect — words more competitive. For a medical context, this raises the risk of the model producing wrong dosages or conditions. For factual, high-stakes tasks, use `temperature=0–0.2`. A friendlier tone is a system-prompt problem, not a temperature problem.

Approach: Explain the probability-flattening effect and link it directly to hallucination risk. 'Friendlier tone' is controlled by the system prompt, not temperature.

A developer runs the same prompt 100 times at temperature=0 and gets the same output 98 times, but 2 different outputs on 2 runs. They conclude the API is broken. What is the real explanation?

`temperature=0` uses greedy decoding — pick the most likely token — but tiny rounding differences on distributed computers can occasionally flip two nearly-equal probabilities. This is expected behavior, not a bug. To improve consistency, also set the `seed` parameter — though OpenAI does not guarantee identical outputs in all conditions.

Approach: Name floating-point rounding on distributed hardware as the cause. Distinguish greedy decoding (what temp=0 does) from true determinism (not what the API guarantees).

Continue learning

Previous: Tokens & tokenization

Return to AI Engineering Roadmap