Tokens & tokenization

Roadmaps › AI Engineering

Scenario

You ask an AI to write a 500-word essay. It stops in the middle of a sentence. You check the billing page and the charge is not counted in words at all.

Predict before reading: is a 500-word English essay more than 500 of whatever the AI counts in, or fewer? And when the AI hits its limit, why does it stop mid-word instead of finishing the sentence?

Mental model

Tokens are the syllables that AI models use to read and write.

An AI does not read text letter by letter or word by word. It chops text into small pieces called tokens. A short, common word is one token. A long or rare word might be two or three tokens. In English, one token is about 0.75 words, so 1,000 tokens is roughly 750 words. Every price, context limit and rate limit you will ever see is measured in tokens, not words.

Deep dive

Picture chopping a long word into syllables to pronounce it. The AI does exactly this to read text. It chops sentences into small pieces called tokens. **Models process tokens, not characters or words.** This is the single most important rule of AI text. A short common word is one token. A long or rare word becomes two or three tokens.

Tokens are the small text pieces that AI models read and write.

Let us trace a real WhatsApp message: 'Hi Rahul! See you at 5pm'. Before reading on, guess how many tokens it makes. The real answer is 9: ['Hi', ' Rahul', '!', ' See', ' you', ' at', ' ', '5', 'pm']. Two surprises hide here. 'Rahul' stays as ONE token — it is common enough to have its own entry in the tokenizer's vocabulary. But '5pm' breaks into THREE tokens: a space, the digit '5', and 'pm'. Digits usually tokenize separately from words.

Splits are decided by the tokenizer's vocabulary, and they will surprise you — measure, don't guess.

Why chop into sub-words at all? A dictionary of every full word would need millions of entries and still miss names, typos and new slang. Instead the tokenizer keeps a fixed vocabulary of roughly 200,000 pieces, learned from real text (the method is called BPE — byte pair encoding). Any word it has never seen still splits into pieces it knows. 'programming' becomes ['program', 'ming']. But 'running' is ONE token — it was common enough to earn its own entry. You cannot predict splits from grammar or spelling; only the learned vocabulary decides.

A fixed sub-word vocabulary handles every possible text — but splits follow the vocabulary, not grammar rules.

This matters because you pay per token — for the tokens you send in AND the tokens the model writes back. The model also has a hard cap on tokens per call, and the cap counts input plus output together. If your prompt fills most of the window, the reply gets cut off: the model stops at the token limit, and a token boundary can fall in the middle of a word. That is exactly why the essay in the hook stopped mid-sentence.

Tokens are the unit of billing AND the hard limit — and input plus output share one cap.

**Non-English text and code use more tokens per word than English.** The tokenizer's vocabulary was learned mostly from English text, so a Hindi or Thai word may cost 2-4 tokens where an English word costs one. Code and rare technical terms split heavily too. The same content can cost two or three times more in another language — teams that budget by word count get surprised when they go multilingual.

Token cost varies a lot by language and domain — always measure on your real content.

You never have to guess: the `tiktoken` library runs OpenAI's exact tokenizer on your own computer, free, with no API call. Count before you send, and you will never hit the context limit by surprise or misjudge a bill. This lesson is the ground floor: the next steps in the roadmap — context window and cost budgeting — are both built on counting tokens.

Count with tiktoken before sending. Context window and cost budgeting build directly on this.

Code examples

Count tokens and see the actual pieces

import tiktoken

enc = tiktoken.encoding_for_model('gpt-4o')

tokens = enc.encode('Hi Rahul! See you at 5pm')
print(len(tokens))  # 9

# See the exact pieces the model reads:
for t in tokens:
    print(repr(enc.decode([t])))
# 'Hi', ' Rahul', '!', ' See', ' you', ' at', ' ', '5', 'pm'

This runs OpenAI's real tokenizer locally — no network call, no cost. Notice 'Rahul' stays whole while '5pm' splits into three pieces: the vocabulary decides, not grammar. Every count in this lesson was generated by running this code.

Count a messages[] list — the real billing unit

import tiktoken

def count_messages_tokens(messages, model='gpt-4o'):
    enc = tiktoken.encoding_for_model(model)
    total = 0
    for msg in messages:
        # ~4 tokens overhead per message (role + delimiters)
        total += 4 + len(enc.encode(msg.get('content', '')))
    total += 2  # reply priming
    return total

msgs = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user',   'content': 'Explain RAG in one sentence.'},
]
print(count_messages_tokens(msgs))  # 23

The API bills the full messages[] array, not just your latest question — every message carries a few overhead tokens for its role tags. In a long chat this adds up faster than you expect.

Key points

Common mistakes

Glossary

token
The basic building block of text for an AI. It can be a whole word or just a part of a word.
tokenization
The process of chopping normal text into tokens so the AI can read it.
tiktoken
A free tool from OpenAI that counts exactly how many tokens your text uses, on your own computer, without any API call.

Recall questions

Questions & answers

You are building a chat app that summarizes long meeting transcripts. Your API costs are much higher than expected, even though the word count is low. The transcripts are in Spanish. What is the root cause?

Token-per-word ratios are language dependent. The tokenizer's vocabulary is mostly English, so Spanish text splits into more tokens per word — you are billed for a much higher token count than the word count suggests. Measure tokens directly with tiktoken on real transcripts, never estimate from words.

Approach: Identify that tokenization efficiency varies by language. The trap is assuming English token math applies globally.

A team estimates their RAG prompt at '3,000 words' and is surprised when the API throws a context-length error. What are two likely causes?

First, words are not tokens: 3,000 English words is roughly 4,000 tokens. Second, the context limit counts input plus output together — if the reply needs headroom, the usable input budget is smaller than the advertised window. Fix: count with tiktoken and reserve output budget explicitly.

Approach: Convert words to tokens (about 1.3x) and flag that the limit is shared by input and output. These two points explain almost every 'why did I hit the limit?' surprise.

Continue learning

Next: Context Window

Next: Temperature & Top-p

Next: Embeddings

Next: Cost & Latency Budgeting

Return to AI Engineering Roadmap