Chunking Strategies

Roadmaps › AI Engineering

Scenario

You have a 50-page PDF. You slice it into fixed 100-word blocks. A user asks about 'pricing'. The retriever returns a block that says 'It costs $50 per month' — but misses the previous sentence: 'For the Enterprise tier:'. The price has no context.

Before you read on, predict: what simple change to the splitting strategy would prevent this problem?

Mental model

Chunking is like framing photographs of a landscape. A frame too small shows nothing. A frame too large loses all detail.

Before you embed documents, you must cut them into smaller pieces called chunks. A chunk must be large enough to contain full context — so the LLM understands it — but small enough that retrieving it is precise and affordable. To prevent losing context at boundaries, chunks intentionally overlap at their edges.

Deep dive

You cannot embed an entire 50-page manual as a single vector. A single vector for 50 pages is too diluted — it averages out all the specific details. If you search for 'reset password', that entire document will not rank highly because 'password' is only 0.1% of its content. You must split it into chunks.

Documents must be split into chunks so their embeddings stay specific and retrievable.

The simplest method is fixed-size chunking: count a set number of characters or tokens and slice the text at that count. It is fast and easy, but it cuts blindly through sentences and paragraphs, destroying context at the boundary.

Fixed-size chunking is fast but breaks sentences in half without warning.

To fix the boundary problem, use overlap. Instead of starting the next chunk exactly where the last one ended, go back a little and repeat some text. If chunks are 500 characters, overlap them by 50 characters. A sentence that falls on a boundary will be fully included in either the chunk before or the chunk after.

Overlap ensures sentences at the boundary appear whole in at least one chunk.

A smarter approach is recursive character chunking. Instead of counting tokens blindly, it tries to split on natural boundaries first: double newlines (paragraphs), then single newlines, then spaces. It only cuts inside a word if a single paragraph is longer than your maximum chunk size. This is the standard starting point for most RAG systems.

Recursive chunking respects paragraph and sentence boundaries where possible.

**Chunk size is a trade-off.** Small chunks (100 tokens) are precise but lack surrounding context. Large chunks (1,000 tokens) have great context but add noise, use more prompt tokens, and cost more. Most systems start at about 500 tokens with 10% overlap and tune from there. The right size depends on the data — a dense legal contract may need sentence-level chunks, while a blog post may work fine with paragraph-level chunks.

Pipeline: Docs → Chunk → Embed → Store. Chunking determines exactly what text the LLM will eventually read.

Code examples

Naive fixed-size chunking — shown so you know what to avoid

def fixed_chunk(text: str, size=500) -> list[str]:
    return [text[i:i + size] for i in range(0, len(text), size)]

chunks = fixed_chunk('This is a very long document...')

This method has no awareness of sentences or paragraphs. It will often cut an important word right in the middle, producing a chunk with broken context at both ends.

Recursive chunking with overlap — standard practice

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Split by paragraphs, then sentences, keeping chunks under 500 chars.
# Overlap by 50 chars so context at the boundary is not lost.
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", " ", ""]
)

chunks = splitter.split_text(long_document_string)

This is the default starting point for most RAG systems. It respects natural language boundaries (paragraphs, then sentences) before it resorts to splitting words.

Key points

Common mistakes

Glossary

chunking
Splitting a large document into smaller pieces called chunks so that each piece can be embedded and searched individually.
overlap
A technique where the end of one chunk is repeated at the start of the next chunk, so important context at the boundary is not lost.
recursive character chunking
A splitting method that tries to keep natural boundaries — paragraphs and sentences — intact, only cutting inside a sentence as a last resort.

Recall questions

Questions & answers

A developer reports that their RAG system often retrieves the right paragraph, but the LLM still gives poor answers because the paragraph relies on a pronoun defined in the previous unretrieved paragraph. How do you fix this?

The problem is a loss of context across chunk boundaries. Increase the chunk overlap so the preceding sentence — which defines the pronoun — is included in the retrieved chunk. Alternatively, increase the chunk size to naturally include the surrounding context.

Approach: Identify the root cause as missing adjacent context. Overlap is the immediate fix. Larger chunk size is the alternative.

You are building a RAG pipeline for a large codebase to help developers search for functions. Standard recursive character chunking is returning chunks that do not compile. What is wrong?

Text splitters split on newlines and spaces, which destroys code structure — a function body can be separated from its definition. Code requires a specialized chunker that splits on code boundaries, such as an AST-aware splitter that keeps each function or class intact.

Approach: Recognize that code has structural boundaries that text splitters ignore. Propose AST-based splitting that respects function and class definitions.

Continue learning

Previous: Grounding with Retrieval

Next: Embed & Retrieve Top-k

Return to AI Engineering Roadmap