Caching & Cost Control

Roadmaps › AI Engineering

Scenario

Your AI feature goes viral. You wake up to a $4,000 OpenAI bill because 10,000 users asked the exact same question: 'How do I get started?' Before you read on, predict: how many times did the API actually need to compute a new answer?

Think about what 'stateless API' means — then read on to understand how caching eliminates redundant computation.

Mental model

Caching is memorization — don't pay to compute what you already know. Model routing is hiring the right employee for each task — don't pay a senior engineer to sort mail.

Every LLM API call is stateless. If you send the same prompt 100 times, the provider computes it 100 times and bills you 100 times. In production, a large portion of traffic is repetitive (FAQs, onboarding questions). Caching returns a stored answer instead of re-computing. For non-identical but simple queries, model routing sends them to a cheaper model so you spend frontier-model money only on hard problems.

Deep dive

LLM API calls are stateless. The provider does not save results between calls. If you send the same prompt 100 times, you are billed 100 times and wait 100 times for generation. In production, onboarding FAQs, help docs questions, and template-based queries follow predictable patterns — recomputing them is wasted money and wasted latency.

Providers do not cache your repetitive queries. You must build caching yourself.

The simplest defense is **exact match caching**. Hash the user's prompt (or the full messages array) and use it as a key in Redis. If another user sends the exact same string, return the cached response instantly — zero API cost, millisecond latency. It is 100% safe because it only hits on mathematically identical strings.

Exact caching (Redis) is safe, instant, and free — but only hits on identical phrasing.

When User A asks 'How to start?' and User B asks 'How do I start?', exact match fails. **Semantic caching** fixes this: embed the user's question and search a vector store for previously answered questions with cosine similarity above 0.98. If a match is found, return the cached answer. The risk: if your threshold is too low, a semantically similar but meaningfully different question gets the wrong cached answer.

Semantic caching catches phrasing variations but risks false positives if the similarity threshold is too low.

When caching misses and you must generate, **model routing** controls costs. Do not use GPT-4o for everything. Use cheap, fast models (GPT-4o-mini, Claude Haiku) for summarization, classification, and simple RAG. Reserve expensive models for complex reasoning and long-context tasks. A two-tier setup — mini for 80% of traffic, a frontier model for the hard 20% — typically cuts total API cost by 60–80%.

Route simple tasks to cheap models. Save frontier models for hard reasoning only.

Always set `max_tokens`. This is a hard circuit breaker — if the model gets stuck in a repetitive loop or the task produces a longer-than-expected output, `max_tokens` cuts off generation before it burns your budget. For a 2-sentence summary, `max_tokens=100` is a reasonable cap.

max_tokens is a cost and latency circuit breaker. Always set it.

Code examples

Exact match caching with Redis

import hashlib

def get_cached_response(redis_client, prompt: str) -> str:
    prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()

    cached = redis_client.get(prompt_hash)
    if cached:
        return cached.decode('utf-8')  # cache hit — no API call

    response = call_llm(prompt)  # cache miss — call the API
    redis_client.setex(prompt_hash, 86400, response)  # cache for 24 hours
    return response

If 10,000 users ask the exact same question, you only pay the LLM provider once. The SHA-256 hash converts any-length prompt into a fixed-length key. Safe because it only hits on mathematically identical strings — no risk of serving the wrong answer.

Model routing based on task complexity

def route_query(user_input: str) -> str:
    # Use cheap model to classify intent first
    intent = call_llm(user_input, model='gpt-4o-mini',
                      system='Classify as SIMPLE or COMPLEX. Reply with one word.')

    if intent.strip() == 'SIMPLE':
        return call_llm(user_input, model='gpt-4o-mini')   # ~15x cheaper
    else:
        return call_llm(user_input, model='gpt-4o')        # frontier model

You pay a tiny fraction of a cent for the routing classification. If 80% of traffic is SIMPLE, you slash your bill by roughly 80% while preserving quality on hard queries.

max_tokens as a budget circuit breaker

from openai import OpenAI
client = OpenAI()

resp = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Summarize this article in 2 sentences.'}],
    max_tokens=150,  # <-- the circuit breaker
)

Even when you ask for 2 sentences, models can get stuck in repetitive loops. `max_tokens` ensures the API physically stops generation before it burns your budget. Always set it — never omit it in production.

Key points

Common mistakes

Glossary

semantic caching
A smart cache that recognizes when two questions mean the same thing and reuses the stored answer — even if the wording is slightly different.
max_tokens
A hard limit you set on how long the model's reply can be — preventing runaway generation that burns your token budget.
model routing
Sending simple queries to cheap, fast models (like gpt-4o-mini) and saving expensive frontier models only for the queries that genuinely need them.

Recall questions

Questions & answers

You implement semantic caching. Users report getting answers that don't quite match their specific question constraints. How do you fix this?

The similarity threshold is too low — questions that are different in a meaningful way are being treated as cache hits. Fix: raise the cosine similarity threshold (e.g., from 0.90 to 0.97 or 0.98) so only nearly identical questions return cached answers. For precision tasks like code generation or medical queries, fall back to exact match caching entirely.

Approach: Identify semantic caching as a source of false positives. The primary lever is the similarity threshold. For precision-critical tasks, exact match is safer.

Your app answers general knowledge questions but appends 'Have a great day, [Username]!' at the end. Can you cache the full response?

No — the username makes every response unique per user. Cache only the generic LLM answer (the knowledge part), and use standard application code (string interpolation) to append the personalized greeting separately. LLM caches must not contain user-specific state or PII.

Approach: Show understanding of data separation. Cacheable content = generic knowledge. Non-cacheable = any user-specific state, PII, or dynamic values.

Continue learning

Previous: Cost & Latency Budgeting

Return to AI Engineering Roadmap