Observability & Tracing

Scenario

A user complains that your RAG bot told them to 'reboot the router' in response to a software password reset question. You check the logs. You see the user's question and the bot's wrong answer. Before you read on, predict: what information do you need to diagnose why it went wrong?

Think about the gap between 'user asked X, bot said Y' and the full internal decision process — then read on.

Mental model

Observability is the flight data recorder for your AI. It doesn't just record where the plane landed — it records every dial, switch, and decision along the way.

Standard software logging ('INFO: User logged in') is insufficient for AI systems. An agent may do ten things before replying: embed the query, search the database, call a tool, summarize results, and more. If the final answer is wrong, you need a Trace — a structured tree of the entire request — broken into Spans (individual steps), each capturing the exact prompts, retrieved chunks, tool outputs, latency, and token counts at every point.

Deep dive

In a basic LLM script, you just print the response. In production, a single user request triggers a cascade of hidden operations: chunk retrieval, prompt assembly, tool calling, and multiple model calls. If you only log the final output, you cannot tell whether a wrong answer came from bad retrieval, a bad prompt, or bad model reasoning. You are flying blind.

AI pipelines are non-deterministic and multi-step. Final-output logging is not enough.

The solution is tracing. A **Trace** represents the full lifecycle of one user request. A trace is made of **Spans**. Each span represents one unit of work — 'Embed Query', 'Query Vector DB', 'Call LLM' are all separate spans. Each span records its start time, end time, inputs, outputs, and metadata.

Trace = the whole request. Span = one step inside the request.

Inside each LLM span, four things are essential: the **exact fully assembled prompt** (including injected context — not just the user's input), the **exact model output**, the **token usage** (prompt + completion tokens), and the **latency**. Without the exact prompt, you cannot reproduce the bug. Without token counts, you cannot debug cost spikes.

Log the fully assembled prompt, model output, token usage, and latency — for every LLM call.

For RAG, also trace the retrieval span. Log the search query (which an agent may have rewritten), the IDs of chunks returned, and their similarity scores. When a model hallucinates, the first question is always: 'Did the vector database return the right chunks?' The retrieval trace answers this — letting you separate 'retrieval failure' from 'model reasoning failure'.

Trace the retrieved context to separate database failure from model failure.

You do not build this from scratch. Observability platforms (LangSmith, Helicone, Phoenix, Datadog LLM observability) wrap your functions in decorators or SDK hooks and automatically capture inputs, outputs, and timing — building the visual trace tree for you.

Use existing observability platforms — they build the trace tree automatically.

Code examples

The manual way — what you are trying to avoid

def rag_pipeline(user_query):
    logger.info(f'Query: {user_query}')

    docs = retrieve(user_query)
    logger.info(f'Retrieved {len(docs)} docs')  # Which docs? What similarity scores?

    prompt = build_prompt(user_query, docs)
    response = call_llm(prompt)
    logger.info(f'Response: {response}')  # How many tokens? How long did this take?

Flat text logs lose the structure. You cannot reconstruct what the model actually saw, which chunks were injected, what the similarity scores were, or how many tokens the prompt contained. When the answer is wrong, you have no path to the root cause.

Tracing with spans — conceptual pattern

from tracer import trace, span

@trace(name='RAG_Request')
def handle_request(user_query):

    @span(name='Retrieve_Context')
    def get_docs(q):
        docs = db.search(q)
        span.log_metadata({'retrieved_text': [d.text for d in docs],
                           'scores': [d.score for d in docs]})
        return docs

    @span(name='LLM_Generation')
    def generate(q, docs):
        prompt = assemble_prompt(q, docs)  # full prompt logged automatically
        return call_llm(prompt)            # tokens and latency logged automatically

    return generate(user_query, get_docs(user_query))

Wrapping functions in spans builds a tree. When an answer is wrong, you click the 'RAG_Request' trace, open 'Retrieve_Context' to see exactly which chunks were returned, then open 'LLM_Generation' to read the exact prompt the model received.

Key points

Traces vs spans: A trace is the full request. A span is a single step (retrieval, tool call, LLM call) within that trace.
Log the fully assembled prompt: The prompt template is not enough. You need the exact text sent to the API, including the injected RAG context.
Track tokens and latency: Every LLM span must log prompt_tokens and completion_tokens so you can debug cost spikes.
Blame assignment: Tracing separates 'The model hallucinated' from 'The vector DB returned the wrong chunks'.

Common mistakes

Logging only the user input: If a user types 'summarize this' and you only log 'summarize this', you cannot reproduce the bug — you did not log the 5-page document injected into the prompt alongside it. Log the fully assembled prompt.
Ignoring tool inputs and outputs: If your agent uses a calculator tool, log exactly what equation the model sent to the tool and what number the tool returned. Otherwise you cannot tell whether the model did bad reasoning or the tool returned a wrong result.

Glossary

observability: The ability to understand what your AI system did internally during a request — not just what output it produced.
trace: A complete tree-shaped record of everything that happened during one user request, from the first database search to the final model reply.
span: One individual step within a trace — for example, 'query vector DB' or 'call LLM'. Each span records its own inputs, outputs, timing, and token counts.

Recall questions

Explain the difference between a trace and a span, as if to a classmate.
Why must you log the fully assembled prompt rather than just the user's original input?
Besides the prompt and response, what two quantitative metrics must be logged for every LLM span?
Your model gives a wrong answer to a user question. Your tracing shows the retrieval span returned 5 chunks — all about a different product. Predict the root cause and fix.

Questions & answers

A user reports your RAG system gave an entirely wrong answer to a simple question. You have full tracing enabled. Walk through how you debug this trace.

First, open the trace for that specific request. Look at the retrieval span: did the vector DB return the correct context chunks? If no, the bug is in the retrieval layer — fix embedding, chunking, or metadata filters. If yes, look at the LLM generation span: did the fully assembled prompt correctly include those chunks? If yes, the model failed to reason over the correct context — adjust the system prompt or upgrade the model.

Approach: Demonstrate how tracing allows blame assignment. Isolate the retrieval step from the generation step to find the root cause systematically.

You notice a massive spike in your OpenAI bill with no increase in user traffic. How do you use observability tools to find the cause?

Sort traces by token usage descending. Look for spans with unusually high completion_tokens — indicating the model got stuck in a generation loop or max_tokens was removed. Also look for high prompt_tokens — indicating a bug injecting too much context. Filter by date range to identify which deploy introduced the spike.

Approach: Show that observability is the primary tool for cost debugging, not just correctness. Token usage logging on every call makes cost spikes trivially diagnosable.

Continue learning

Previous: Cost & Latency Budgeting

Return to AI Engineering Roadmap