Agent Memory: Short vs. Long Term

Scenario

You build a personal assistant chatbot. On Monday, the user says 'I prefer Python over JavaScript.' On Friday, the user asks 'What language should I use for this script?' The bot says 'It depends on your preferences.' Before you read on, predict: why does the bot not remember Monday's conversation?

Think about how the API works per request — then read on to understand what 'memory' really means for an LLM.

Mental model

Short-term memory is RAM — fast and temporary, lost when you restart. Long-term memory is a hard drive — persistent, but you have to explicitly write and read from it.

An LLM is completely stateless. It has no memory between API calls. 'Memory' is just data you choose to inject into its prompt. Short-term memory is the messages array you append to and resend on every call — it holds the current conversation. Long-term memory is saving important facts to a database and actively retrieving them in future sessions to inject into the system prompt.

Deep dive

Every LLM API call is independent — the model knows nothing from previous calls unless you include that history in the current request. To have a 'conversation', you must send the entire array of past messages back on every call. This running list of messages is the agent's short-term memory. If you do not include a message, the model literally cannot know it happened.

Short-term memory is the messages array you resend on every call. The model has no other memory.

Short-term memory has two limits: the model's token limit and cost. A conversation that includes 100 past messages sends hundreds of tokens of history on every single call. Eventually you hit a `context_length_exceeded` error. The fix is a **sliding window** — keep only the last N messages plus the system prompt. Older messages are dropped. This gives the agent amnesia about early conversation details.

The context window is finite. Use a sliding window to keep the messages array from growing forever — but always preserve the system prompt.

To remember facts across sessions, weeks, or after the sliding window drops them, you need **long-term memory**. The pattern: equip the agent with a `save_fact(fact)` tool. When the user reveals important information ('I prefer Python'), the agent calls the tool. Your app writes the fact to a database. In future sessions, you query the database with the user's new message as a search key, and inject relevant retrieved facts into the system prompt.

Long-term memory is RAG applied to user facts — save to a database, retrieve on future sessions.

Code examples

Short-term memory with a sliding window

from openai import OpenAI
client = OpenAI()

MAX_MESSAGES = 10

def chat_with_memory(new_user_text: str, message_history: list[dict]) -> str:
    message_history.append({'role': 'user', 'content': new_user_text})

    # Sliding window: always keep system prompt + last MAX_MESSAGES messages
    system_prompt = message_history[0]  # index 0 is always the system prompt
    recent = message_history[-MAX_MESSAGES:]
    truncated = [system_prompt] + recent

    response = client.chat.completions.create(
        model='gpt-4o',
        messages=truncated
    )
    reply = response.choices[0].message
    message_history.append({'role': 'assistant', 'content': reply.content})
    return reply.content

The sliding window keeps token count stable. The system prompt at index 0 is always preserved — without it, the agent forgets its core instructions. Messages older than MAX_MESSAGES are dropped silently.

Long-term memory — saving and retrieving user facts

# Equip the agent with a tool to save facts
tools = [{
    'type': 'function',
    'function': {
        'name': 'save_user_fact',
        'description': 'Save a fact about the user for long-term retention across sessions.',
        'parameters': {
            'type': 'object',
            'properties': {
                'fact': {'type': 'string'},
                'category': {'type': 'string', 'enum': ['preference', 'background', 'project']}
            },
            'required': ['fact', 'category']
        }
    }
}]

# On a new session, retrieve and inject relevant facts:
# facts = db.query('SELECT fact FROM long_term_memory WHERE user_id = ?', user_id)
# system_prompt += f'\nRemember about this user: {facts}'

Long-term memory is not magic — it is a tool call combined with a database read. The agent calls `save_user_fact` when it learns something important. On the next session, your app reads relevant facts from the database and injects them into the system prompt before the first message.

Common mistakes

Thinking the model remembers previous API calls: Models are perfectly stateless. If you do not explicitly include a fact in the `messages` array of the current API call, the model cannot know it. There is no hidden memory on the server between requests.
Letting the message history array grow without limit: Appending messages forever causes a `context_length_exceeded` error once the total token count exceeds the model's limit. More subtly, it also makes every call increasingly expensive because more history is billed as input tokens. Always implement a sliding window or summarization strategy.

Glossary

short-term memory: The recent messages you send back to the model on every API call so it remembers the current conversation. It lives only in the messages array.
long-term memory: A system where you save important facts about a user to an external database and retrieve them in future sessions to inject into the model's context.
sliding window: A simple memory management strategy where you keep only the last N messages and discard older ones to prevent the context window from overflowing.

Recall questions

Explain why an LLM chatbot cannot remember Monday's conversation on Friday, as if to a classmate.
What is the primary limitation of short-term memory in LLM applications?
When implementing a sliding window, which message must you always keep and never discard?
A user tells the bot 'I prefer Python' on Monday. On Friday, a new session starts with an empty messages array. Predict how you would ensure the bot still knows this preference.

Questions & answers

Your chatbot works great for 10 minutes, but then requests start failing with a 400 error about 'context length'. How do you fix this?

The messages array is growing infinitely and hitting the token limit. Two fixes: implement a sliding window — keep only the last N messages, always preserving the system prompt. Or implement summarization — use the LLM to compress older messages into a short summary paragraph and prepend it to the recent messages. The sliding window is simpler; summarization preserves more context.

Approach: Identify the stateless nature of LLMs and the hard context window limit. Propose truncation or summarization as the standard strategies.

Design a long-term memory system for a personal assistant agent that needs to remember user preferences across sessions.

Three parts: (1) Equip the agent with a `save_fact(fact, category)` tool. When the user states a preference, the agent calls the tool and your app writes the fact to a database keyed by user ID. (2) On future sessions, query the database using the new user message as a search key — embed the message and retrieve semantically similar stored facts. (3) Inject the retrieved facts into the system prompt before the first call. This is RAG applied to user facts.

Approach: Combine tool calling (writing memory) with RAG (reading memory). Long-term memory is a specialized database integration — not a built-in model feature.

Continue learning

Previous: Tool Use & The Call Loop

Return to AI Engineering Roadmap