Cost & Latency Budgeting

Scenario

Your RAG chatbot prototype works great. You show it to the team. A product manager asks: 'How much will this cost at 10,000 requests per day?' Before you read on, predict: what information do you need to calculate this?

Think about what drives cost in an LLM API call — then read on to learn the formula.

Mental model

Every LLM call is a taxi ride: you pay per token (the meter), and the bigger the car (model tier), the higher the rate.

Cost = (prompt tokens × input price + completion tokens × output price) per million tokens. The two levers are model choice (gpt-4o-mini is about 15x cheaper than gpt-4o per token) and token count (smaller prompt + smaller output = linear cost reduction). Latency for LLMs is dominated by output tokens — more tokens take more time to generate. Time-to-first-token (TTFT) is a separate metric driven by server load and prompt prefill.

Deep dive

The pricing formula: `cost = input_tokens × input_$/MTok + output_tokens × output_$/MTok`. For gpt-4o (mid-2025): $2.50 per million input tokens, $10.00 per million output tokens. For gpt-4o-mini: $0.15 input / $0.60 output. A 2k-token prompt + 500-token reply on gpt-4o costs about $0.010. The same call on gpt-4o-mini costs about $0.0006. At 10k requests/day that is $100/day vs $6/day.

Cost = input tokens × rate + output tokens × rate. Output tokens are priced 4× higher than input on most models.

The single most impactful cost decision is model choice. Use the smallest model that meets quality requirements for your task. Use gpt-4o-mini for classification, routing, extraction, and short summarization. Reserve gpt-4o for complex reasoning, long-context tasks, and multimodal. A two-tier setup — mini for 80% of traffic, 4o for the hard 20% — typically cuts total cost by 60–80%.

Model tier is the biggest lever. Never default to the frontier model for every task.

Latency has two components. Time-to-first-token (TTFT): the wait before any output arrives — driven by server load and prompt length (longer prompts take longer to prefill). Tokens-per-second (TPS): the generation speed after the first token — roughly 40–80 TPS for gpt-4o at normal load. Total latency ≈ TTFT + output_tokens / TPS. Cutting `max_tokens` is the most direct latency lever you control.

Total latency = TTFT + output_tokens / speed. Limit max_tokens to cut tail latency.

Prompt compression is the overlooked cost lever. Your system prompt repeats on every API call. If your system prompt is 1,000 tokens and you make 10k calls/day, that is 10 million input tokens of system-prompt cost alone per day. Techniques: compress the system prompt (fewer words, same intent), use prompt caching (OpenAI's cached input pricing is 50% off for repeated prefixes), and trim retrieved RAG context aggressively.

Repeated system prompts are a hidden cost multiplier. Cache or compress them.

Code examples

Estimate cost before sending a request

import tiktoken

PRICING = {
    'gpt-4o':       {'input': 2.50,  'output': 10.00},
    'gpt-4o-mini':  {'input': 0.15,  'output': 0.60},
}

def estimate_cost(messages: list[dict], expected_output_tokens: int, model='gpt-4o') -> float:
    enc = tiktoken.encoding_for_model(model)
    input_tokens = sum(4 + len(enc.encode(m.get('content', ''))) for m in messages) + 2
    p = PRICING[model]
    return (input_tokens * p['input'] + expected_output_tokens * p['output']) / 1_000_000

messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user',   'content': 'Summarise the following in 3 bullet points: ' + 'x ' * 500},
]
print(f'gpt-4o:       ${estimate_cost(messages, 150, "gpt-4o"):.5f}')      # ~$0.0040
print(f'gpt-4o-mini:  ${estimate_cost(messages, 150, "gpt-4o-mini"):.5f}') # ~$0.0002

Run this before choosing a model. For a 1k-token prompt + 150-token reply, gpt-4o costs about 20x more than gpt-4o-mini. If mini's quality is acceptable, use it. Build this into your observability pipeline to track actual vs estimated costs per request type.

Log actual cost per call from the usage object

from openai import OpenAI
client = OpenAI()

PRICING = {
    'gpt-4o':       {'input': 2.50, 'output': 10.00},
    'gpt-4o-mini':  {'input': 0.15, 'output': 0.60},
}

def call_and_log(messages: list[dict], model='gpt-4o-mini') -> str:
    resp = client.chat.completions.create(
        model=model, messages=messages, max_tokens=300,
    )
    u = resp.usage
    p = PRICING[model]
    cost = (u.prompt_tokens * p['input'] + u.completion_tokens * p['output']) / 1_000_000
    print(f'[cost] ${cost:.6f} | in={u.prompt_tokens} out={u.completion_tokens} | model={model}')
    return resp.choices[0].message.content

Log token counts and cost on every production call. A spike in `prompt_tokens` means your RAG context is growing too large. A spike in `completion_tokens` means `max_tokens` is not set. The `usage` object is the ground truth for cost debugging.

Common mistakes

Using gpt-4o for every task by default: gpt-4o is 15–20x more expensive per token than gpt-4o-mini. For classification, extraction, and short summarization, gpt-4o-mini achieves equivalent quality at a fraction of the cost. Benchmark both before defaulting to the expensive model.
Not accounting for output token cost: Output tokens are typically priced 4x higher than input tokens per million. A prompt that asks for a 2,000-word essay pays far more in output cost than in input cost. Setting `max_tokens` is both a latency and a cost control measure.

Glossary

Time-to-first-token: The delay before the AI outputs its first word — driven by prompt length and server load. Streaming makes this the user-visible wait.
Tokens-per-second: The speed at which the model generates text after the first token arrives — determines how fast the full response appears.
prompt caching: A provider feature that discounts the price of repeated prompt prefixes — like your system prompt — that appear at the start of every call.

Recall questions

Explain the LLM API cost formula to a classmate.
What are the two components of LLM API cost, and which is typically priced higher?
What is the primary latency lever you control as an application developer?
Your team's LLM API bill doubles in a week with no change in user traffic. Predict three things to check first.

Questions & answers

Your team's LLM API bill doubles in a week with no change in user traffic. What are three things you check first?

Three things: prompt_tokens per call — did retrieved context size or system prompt size grow? Completion_tokens per call — was max_tokens removed or increased, causing longer outputs? Model tier — did someone switch from gpt-4o-mini to gpt-4o for a high-volume endpoint? Log usage on every call to answer these questions instantly.

Approach: Three levers: prompt size, output size, model tier. Frame it as a logging gap — you cannot debug cost spikes without per-call usage logging.

Design a cost-optimized pipeline for a support chatbot handling 100k requests per day, where 75% are simple FAQs and 25% are complex multi-step questions.

Two-tier routing: a cheap classifier (gpt-4o-mini, under 50 tokens I/O) labels each request as 'simple' or 'complex'. Simple requests go to gpt-4o-mini with a short answer prompt (max_tokens=200). Complex requests go to gpt-4o with full RAG context. This routes 75k per day to mini (~$0.15/MTok) and 25k per day to gpt-4o (~$2.50/MTok), reducing overall cost by roughly 70% compared to using gpt-4o for everything.

Approach: Key insight: routing classification — use the cheapest model to triage which calls need the expensive model. The cost savings scale linearly with the fraction of traffic routed to mini.

Continue learning

Previous: Tokens & tokenization

Next: Caching & Cost Control

Next: Observability & Tracing

Return to AI Engineering Roadmap