Context Window

Scenario

You build a chatbot. After 20 turns the conversation is going well. Predict: what happens at turn 25 if you do not manage the history carefully?

Think about why the model might start forgetting or contradicting itself — then read on to understand the exact mechanism.

Mental model

The context window is the model's working memory. It can only 'see' what fits inside it. Anything outside that window does not exist for the model.

Every time you call an LLM API, it processes all the tokens you send from scratch. There is no memory between calls. The context window is the total number of tokens — input plus output — the model can handle in one call. Text outside the window is invisible to the model, as if it never existed.

Deep dive

Modern context windows range from 8k tokens (older models) to 128k (`gpt-4o`) and even 1M (Gemini 1.5). But a larger window is not free. Cost grows with every extra input token. Speed also slows down. Sending a 128k prompt costs about 16× more than an 8k prompt at the same price per token. 'Just send everything' is not a real strategy.

Larger window = higher cost and slower speed. There is no free 'send everything' option.

The context window holds everything in one call: your system prompt, the conversation history, the retrieved text chunks, the user's question, and space for the model's reply. All of these compete for the same fixed budget. If your system prompt uses 2k tokens and the reply needs 1k, your usable space for history and retrieved text is `window_size − 3k`.

Budget = system + history + retrieved context + user message + output space.

When you go over the limit, the API returns a 400 error. To prevent this, your app must use a sliding window — remove the oldest messages before calling the API. This removal is why chatbots forget old details. The safe approach: count tokens before sending with `tiktoken`, then remove the oldest messages when you get close to the limit.

Overflow causes a 400 error. Always count tokens before sending and remove old messages early.

The 'lost in the middle' problem is more subtle than overflow. Research shows models pay best attention to content at the START and END of the context. Content in the middle is answered less reliably. Sending 50 retrieved chunks into the middle of a 128k context is worse than sending 5 well-chosen chunks. Position inside the window matters, not just token count.

Model attention is strongest at the start and end. The middle gets less attention.

**Context window is not memory.** Even with a 1M-token window, the model starts fresh on every API call. 'Memory' in chatbots is an app-level pattern: you keep a list of past messages and include as many as fit each turn. Cross-session memory — remembering facts from yesterday — needs a separate store (a database or vector store) that you inject into each call.

Context window = one call's working memory. Cross-call memory is your app's job, not the model's.

Code examples

Guard against context overflow before sending

import tiktoken

MODEL = 'gpt-4o'
MAX_TOKENS = 128_000
OUTPUT_HEADROOM = 2_000
enc = tiktoken.encoding_for_model(MODEL)

def count_tokens(messages):
    total = sum(4 + len(enc.encode(m.get('content', ''))) for m in messages)
    return total + 2

Count tokens before every API call. The `4 + 2` overhead counts the role delimiters `tiktoken` adds to each message. Catching overflow here is free; catching it from a 400 API error wastes a round-trip.

Sliding window: evict oldest messages when near the limit

def trim_history(messages: list[dict]) -> list[dict]:
    'Evict oldest non-system messages until we fit within budget.'
    while True:
        total = count_tokens(messages)
        if total <= MAX_TOKENS - OUTPUT_HEADROOM:
            return messages
        # drop the first non-system message
        for i, m in enumerate(messages):
            if m['role'] != 'system':
                messages.pop(i)
                break
        else:
            return messages   # only system message left

This is the standard sliding-window pattern for long conversations. The system message stays pinned; the oldest user and assistant turns are removed when you approach the limit. This is the mechanism behind every chatbot that 'forgets' old turns.

Common mistakes

Treating the context window as cross-call memory: The model starts from zero on every API call. If you want it to remember the previous turn, you must resend that text. Forgetting this is why chatbots built without history management seem to 'forget' the user's name two messages later.
Assuming more context always helps: The 'lost in the middle' effect means filling a large context with barely relevant text can hurt answer quality. Send fewer, highly relevant chunks rather than everything that fits.

Glossary

context window: The maximum number of tokens an AI can process in a single request. It acts like the model's temporary working memory for that one call.
tiktoken: A free tool that counts exactly how many tokens your text uses, so you can check before sending it to the API.
lost in the middle: A known problem where AI pays close attention to the beginning and end of a long text, but misses details buried in the middle.

Recall questions

Explain what a context window is, as if to a classmate who has never heard the term.
What is the 'lost in the middle' effect?
How do you prevent a context-overflow error without calling the API?
What happens if you add a 50-chunk retrieval result to the middle of a large context window instead of the top?

Questions & answers

A chatbot works well for the first 10 turns but gives inconsistent answers by turn 20. You have not changed the system prompt or the model. What is the most likely cause and how do you fix it?

The chatbot is running out of context space. The app is removing old messages to avoid a 400 error, so the model loses important early context. The immediate fix is to count tokens before each call and trim old turns carefully. A better fix is to compress old turns into a running summary before removing them.

Approach: Name context overflow as the root cause. Distinguish silent truncation (model seems confused) from a 400 error (call fails). Propose token counting plus a sliding window or summarization step.

You have 50 relevant chunks from a vector store that all fit in a 128k context window. A colleague says 'just send all 50'. Why might that produce worse answers than sending only the top 5?

Sending all 50 chunks places the truly relevant ones in the middle of the context, where model attention is weakest. Five high-quality chunks placed near the edges get better attention. Sending 50 also increases cost and response time by roughly 10× with little benefit.

Approach: Lead with the attention distribution point ('lost in the middle'), then add cost and latency as secondary reasons.

Continue learning

Previous: Tokens & tokenization

Return to AI Engineering Roadmap