Chain-of-Thought

Scenario

You ask GPT-4o-mini a multi-step maths problem and it answers instantly — but incorrectly. You add 'Let's think step by step.' to the prompt. Before you read on, predict: does adding this phrase actually help, and why?

Think about what changes when the model writes out steps — then read on to understand the mechanism.

Mental model

An LLM predicts the next token. 'Jump to the answer' means predicting the answer token directly from the question. 'Think step by step' makes the model write intermediate steps that become its working memory.

Transformers have no internal scratchpad — they cannot hold a number in mind. Their computation happens in the forward pass for each token. When you ask the model to write out reasoning steps, those steps become context tokens that the model attends to when generating subsequent steps. More reasoning tokens = more forward passes = more opportunity to get multi-step logic right before committing to an answer.

Deep dive

Zero-shot CoT is the simplest form: append 'Let's think step by step.' to the user message. This phrase reliably triggers more deliberate, structured reasoning. The model generates its reasoning as text, which becomes part of the context for the final answer. On multi-step maths and logic problems, this trigger alone can lift accuracy by 20–40%.

Append 'Let's think step by step.' — the simplest chain-of-thought trigger with surprising accuracy gains.

Few-shot CoT provides explicit reasoning examples. You show the model an (input, reasoning steps, answer) triple rather than just (input, answer). The model learns both that it should reason and how to structure that reasoning for your domain. This is more powerful than zero-shot CoT for domain-specific problems — financial analysis, legal reasoning, or debugging code — where the style of reasoning matters.

Few-shot CoT teaches the reasoning style, not just 'think more'. Show the full reasoning path.

The cost is real. CoT generates many more output tokens. A direct answer might be 5 tokens; a CoT answer is 80–200 tokens. At scale this adds up. The smart approach: classify query complexity first, then apply CoT only to the hard queries. A cheap model can decide if a query is 'simple' or 'hard'. Simple queries get a direct answer. Hard queries get CoT on a stronger model.

CoT trades output tokens for accuracy. Route only hard queries through CoT; simple ones do not need it.

**CoT only helps when reasoning steps are actually needed.** 'What is the capital of France?' gets no benefit from CoT — the model knows 'Paris' in one token and writing steps only adds noise. Use CoT for: multi-step maths, logical puzzles, code debugging, tasks that require aggregating several facts. Skip it for direct recall, classification, and short extraction.

CoT = useful for multi-step problems. Unnecessary noise for direct recall or simple classification.

Code examples

Zero-shot CoT — the 'think step by step' trigger

from openai import OpenAI
client = OpenAI()

# Without CoT — may error on multi-step maths
direct = client.chat.completions.create(
    model='gpt-4o-mini',
    messages=[{'role': 'user', 'content':
        'A train does 120 km at 60 km/h, then 80 km at 40 km/h. '
        'What is the average speed for the whole journey?'}],
    max_tokens=20, temperature=0,
)
print('Direct:', direct.choices[0].message.content)  # often wrong

Average speed = total distance / total time = 200 / (2 + 2) = 50 km/h. Models often shortcut to the arithmetic mean of 50 km/h by coincidence — but on asymmetric distances they fail. Adding CoT forces them to calculate each leg's time before dividing.

Zero-shot CoT trigger added

cot = client.chat.completions.create(
    model='gpt-4o-mini',
    messages=[{'role': 'user', 'content':
        'A train does 120 km at 60 km/h, then 80 km at 40 km/h. '
        'What is the average speed for the whole journey? '
        "Let's think step by step."}],
    max_tokens=300, temperature=0,
)
print('CoT:', cot.choices[0].message.content)  # 50 km/h with correct steps

Adding 'Let's think step by step.' makes the model output the time for each leg explicitly: 120/60 = 2 h, 80/40 = 2 h, total = 200 km / 4 h = 50 km/h. The written steps prevent the arithmetic shortcut error.

Few-shot CoT — show reasoning as examples

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[
        {'role': 'system', 'content': 'Solve word problems. Show steps, then state the answer.'},
        {'role': 'user',      'content': 'A shop has 50 items. 20% sold Monday. 25% of the rest on Tuesday. How many remain?'},
        {'role': 'assistant', 'content': 'Step 1: Monday sold = 20% of 50 = 10. Remaining = 40.\n'
                                         'Step 2: Tuesday sold = 25% of 40 = 10. Remaining = 30.\n'
                                         'Answer: 30 items.'},
        {'role': 'user',      'content': 'A tank holds 200 L. 30% drains in the morning. 40% of the rest in the afternoon. How much is left?'},
    ],
    max_tokens=200, temperature=0,
)
print(response.choices[0].message.content)
# Step 1: Morning = 30% of 200 = 60. Remaining = 140.
# Step 2: Afternoon = 40% of 140 = 56. Remaining = 84.
# Answer: 84 litres.

The example teaches both the reasoning structure (Step 1, Step 2, Answer) and the approach (compute each stage sequentially). This prevents the common mistake of adding percentages directly: 30 + 40 ≠ 70% of 200.

Extract the final answer from CoT output with a delimiter

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[
        {'role': 'system', 'content': 'Think step by step. Then write the final answer '
                                      'on a new line after ANSWER:'},
        {'role': 'user', 'content': 'What is 15% of 240?'},
    ],
    max_tokens=200, temperature=0,
)
full = response.choices[0].message.content
# '15% of 240: 0.15 × 240 = 36.\nANSWER: 36'
final = full.split('ANSWER:')[-1].strip()
print(final)  # '36'

In pipelines, you need a clean final answer, not the full reasoning. The ANSWER: delimiter splits scratchpad from result with no regex needed. Combine with structured outputs for even more reliable extraction.

Common mistakes

Using CoT for simple recall or classification: 'What is the capital of France? Let's think step by step.' generates 50 tokens of reasoning with no accuracy benefit. The model knew 'Paris' in one token. Reserve CoT for tasks that genuinely need multi-step reasoning.
Trusting the reasoning as ground truth: CoT reasoning can sound confident and be completely wrong. The model may generate a fluent but incorrect chain of steps. Always validate the final answer for high-stakes tasks — do not just check that the reasoning 'sounds right'.

Glossary

zero-shot CoT: Adding a phrase like 'Let us think step by step' to make the model write out its reasoning before answering, without giving it examples.
few-shot CoT: Showing the model worked examples that include the reasoning steps, so it learns both that reasoning is needed and how to structure it.
forward pass: The computation an AI does to generate one token. More written reasoning steps = more forward passes = more computation time to work through the problem.

Recall questions

Explain why writing out reasoning steps helps an LLM solve multi-step problems.
What is the difference between zero-shot CoT and few-shot CoT?
What is the main cost trade-off of chain-of-thought prompting?
You apply 'Let's think step by step.' to all 500,000 daily queries to improve accuracy. Predict the cost impact and propose a smarter approach.

Questions & answers

Your financial analysis bot makes arithmetic errors on multi-step compound-interest calculations. You are already using GPT-4o. What prompt change would you try first before switching models?

Add chain-of-thought prompting: instruct the model to show its calculation step by step, and provide a few-shot example of a similar compound-interest problem with explicit reasoning steps. Written intermediate calculations become context that prevents the model from shortcutting to a wrong answer.

Approach: Explain why CoT helps: intermediate tokens act as working memory. This is a prompt engineering fix that costs nothing compared to a model upgrade.

In a production pipeline processing 500k queries per day, your team wants to add CoT to improve accuracy. A colleague says 'just prepend the trigger phrase to everything'. Why is this a bad idea at scale?

Blanket CoT adds 100–200 output tokens per query across all 500k queries. At typical output token prices, this can add hundreds of dollars per day — for reasoning tokens on simple queries that get no accuracy benefit. The correct approach: classify queries by complexity, apply CoT only to the hard fraction, and route simple queries through direct answering.

Approach: Do the math: tokens × calls × price per million tokens. Then propose complexity routing as the solution. This shows you can apply the technique economically at scale, not just in isolation.

Continue learning

Previous: Zero-shot vs Few-shot

Return to AI Engineering Roadmap