Context Bloat: Pruning and Compacting Agent State
Overview
Techniques for managing an agent's growing state by summarizing old context (compaction) and dropping irrelevant tool outputs (pruning) before calling the LLM.
Agents accumulate state at every step. Without pruning, this unbounded state leads to context collapse, massive latency, API limits, and ultimately agent failure.
Where used: LangGraph state machines, Multi-agent systems, Persistent AI assistants
Why learn this
- Unbounded state tracking causes context collapse and fatal API errors in production.
- Most agent state is meant for programmatic routing, not the LLM's context window.
- Anthropic treats context management as a correctness requirement, not just a speed optimization.
Code walkthrough
def pruned_node(state):
recent_msgs = state['messages'][-3:]
summary = state.get('summary', '')
return llm.invoke(f"{summary}\n{recent_msgs}")Focus: Notice how the node extracts only a slice of `messages` and a pre-computed `summary`, purposefully ignoring the rest of the massive `state` object.
Common mistakes
- Assuming all state must be in context: Developers often believe every variable in the agent's state machine must be passed to the LLM on every turn. In reality, most state is for programmatic routing. Only a pruned version should go to the LLM.
- Treating pruning as only a speed optimization: Many view pruning as a 'nice-to-have' for latency. But because agents run out of usable context window, it is a correctness requirement. An unpruned agent won't just get slower; it will hit a hard limit and fail.
Recall questions
- What is the difference between agent state pruning and agent state compaction?
- Why is it dangerous to pass the entire agent state object to the LLM prompt on every turn?
- An agent retrieves 50 documents, storing them in its state. The next node simply calls `llm.invoke(str(state))`. What happens?
Understanding checks
If an agent retrieves 50 documents and runs this code, what is the most likely outcome?
The API request fails with a `context_length_exceeded` error, or suffers massive latency.
Passing the entire unbounded state dictionary directly to the LLM causes context bloat. The state must be pruned before invocation.
Why would an architecture keep raw documents in the `state` dictionary if they aren't being passed to the LLM?
Because the state machine uses them for programmatic routing, metric logging, or passing to specific downstream tools.
State is the system's database, while the LLM prompt is just its working memory. They do not need to be perfectly synchronized.
Practice tasks
Implement a Compaction Reducer
Write a function that takes a list of messages. If the list has more than 5 messages, replace the oldest messages with a 'summary' string, keeping the 3 most recent messages intact.
Continue learning
Previous: State machines for agents
Previous: Agent Memory: Short vs. Long Term
Previous: Context Window