Rate Limits & Retries

Roadmaps › AI Engineering

Scenario

You launch a RAG pipeline to process 5,000 documents. Within 30 seconds you start getting 429 errors. Your retry loop immediately retries on each 429. Before you read on, predict: does retrying immediately make the situation better or worse?

Think about what a 429 means and what happens when you send more requests while rate-limited — then read on.

Mental model

A rate limit is a traffic light. Hammering the accelerator while the light is red does not make it green faster — you waste fuel and block the intersection.

LLM APIs enforce two simultaneous limits: Requests Per Minute (RPM) and Tokens Per Minute (TPM). Hit either and you get a 429 response. The correct response is exponential backoff — wait longer after each failure, with added jitter so multiple parallel workers do not all retry at the same moment. The OpenAI SDK has built-in retry logic; configure it rather than write it from scratch.

Deep dive

Two limits apply simultaneously: RPM (max requests per minute) and TPM (max tokens per minute, counting prompt + completion tokens). Both are tracked server-side per API key. A 429 error tells you which limit you hit in the error body. In RAG systems with large prompt contexts, TPM is usually hit first — not the raw request count.

Two limits: RPM and TPM. RAG systems usually hit TPM first due to large prompt contexts.

Exponential backoff with jitter is the correct retry strategy. After a 429: wait 1 second, retry. If it fails again: wait 2 seconds, retry. Then 4 seconds, 8 seconds — doubling each time, up to a cap of about 60 seconds. The jitter (random ±20% variation on the wait time) staggers parallel workers so they do not all retry at the same millisecond — which would just recreate the traffic spike.

Double the wait after each retry, cap at ~60 seconds. Add jitter to prevent a thundering herd of simultaneous retries.

The OpenAI Python SDK includes built-in retry with exponential backoff via the `max_retries` parameter. Setting `max_retries=3` at client initialization means the SDK automatically retries on 429 and 5xx errors with its own backoff logic — no manual code needed for simple cases. For fine-grained control (custom backoff, circuit breakers, per-request overrides), use the `tenacity` library.

Use SDK max_retries for simple cases. Use tenacity for production-grade retry control.

For bulk jobs (processing thousands of documents), the better solution is the **Batch API**. It accepts up to 50,000 requests in a JSONL file, processes them asynchronously over 24 hours, and costs 50% less per token. No rate limit errors — it queues internally. Use the synchronous API only for real-time, user-facing requests where latency matters.

Batch API for offline bulk work: 50% cheaper, no rate limits, async. Sync API only for real-time use.

Code examples

Built-in SDK retry — the simplest fix

from openai import OpenAI

# SDK automatically retries 429 and 5xx with exponential backoff
client = OpenAI(max_retries=3)

response = client.chat.completions.create(
    model='gpt-4o-mini',
    messages=[{'role': 'user', 'content': 'Summarise this doc.'}],
    max_tokens=200,
)
print(response.choices[0].message.content)

Setting `max_retries=3` on the client handles transient 429s and server errors with zero boilerplate. The SDK uses exponential backoff internally. This is enough for 90% of real-time pipeline situations.

Manual exponential backoff with jitter

import time
import random
from openai import OpenAI, RateLimitError

client = OpenAI(max_retries=0)  # we handle retries ourselves

def call_with_backoff(messages: list[dict], max_retries=5) -> str:
    wait = 1.0
    for attempt in range(max_retries):
        try:
            resp = client.chat.completions.create(
                model='gpt-4o-mini',
                messages=messages,
                max_tokens=300,
            )
            return resp.choices[0].message.content
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            jitter = random.uniform(0.8, 1.2)        # ±20% jitter
            sleep_time = min(wait * jitter, 60.0)    # cap at 60 s
            print(f'Rate limited. Waiting {sleep_time:.1f}s (attempt {attempt + 1})')
            time.sleep(sleep_time)
            wait *= 2                                 # exponential backoff
    raise RuntimeError('Max retries exceeded')

Jitter prevents a thundering herd — if 50 workers all hit a rate limit and retry at the same moment, they trigger another wave of 429s. The random multiplier staggers their retries. The 60-second cap prevents infinitely long waits.

Batch API for bulk offline processing

import json
from openai import OpenAI
client = OpenAI()

# Build a JSONL file of requests
requests = [
    {'custom_id': f'doc-{i}', 'method': 'POST',
     'url': '/v1/chat/completions',
     'body': {'model': 'gpt-4o-mini',
              'messages': [{'role': 'user', 'content': f'Summarise document {i}'}],
              'max_tokens': 200}}
    for i in range(1000)
]

with open('batch_input.jsonl', 'w') as f:
    for req in requests:
        f.write(json.dumps(req) + '\n')

# Upload and submit
batch_file = client.files.create(file=open('batch_input.jsonl', 'rb'), purpose='batch')
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint='/v1/chat/completions',
    completion_window='24h',
)
print(f'Batch submitted: {batch.id} — poll status, results ready within 24h')

The Batch API processes requests asynchronously at 50% of the synchronous price with no rate limit errors. Ideal for nightly pipelines, bulk embeddings, and offline evaluation runs. Poll `batch.status` until 'completed', then download results by `output_file_id`.

Common mistakes

Glossary

exponential backoff
A retry strategy where you wait progressively longer after each failure — 1 second, then 2, then 4 — to avoid overwhelming the server.
jitter
A small random amount added to your wait time so that multiple workers do not all retry at the exact same millisecond.
Batch API
A cheaper, slower way to send thousands of requests to the API at once for jobs that do not need instant responses.

Recall questions

Questions & answers

Your bulk document-processing job fails after 200 of 5,000 documents with repeated 429 errors. Your retry logic retries immediately on each failure. What is wrong and how do you fix it?

Immediate retries amplify the rate limit problem — each retry is another request against the same exhausted quota. Two fixes: first, implement exponential backoff with jitter so retries are spaced out. Second, and better: switch to the Batch API — submit all 5,000 as a batch, get results asynchronously at 50% cost with no rate limit errors.

Approach: Two fixes at different levels: tactical (backoff for synchronous API) and strategic (Batch API for bulk offline work). The Batch API is the correct tool for this use case.

Two services both use gpt-4o-mini with max_retries=3. Service A uses a shared client singleton. Service B creates a new client per request. Under load, a colleague claims Service B gets fewer 429s because it uses separate client instances. Is this correct?

No. Rate limits are enforced server-side per API key — not per client instance in your code. Creating a new client object per request does not reduce the number of requests hitting the server. Both services share the same API key quota. To reduce 429s you need server-side throttling, request batching, or the Batch API.

Approach: Identify the misconception: rate limits are server-side per API key, not per Python client object. Client isolation has no effect on 429 rates. The fix is request-level throttling or the Batch API.

Continue learning

Previous: Chat Completions API

Return to AI Engineering Roadmap