Normalizing Agent Traces Before Scoring
Overview
Converting raw, messy agent execution logs—including retries, tool errors, and framework-specific formatting—into a clean, standardized schema before passing them to a grading system.
Feeding an unedited execution transcript to an LLM judge causes it to grade inconsistently. Normalization isolates the actual decisions and outcomes, preventing the grader from penalizing agents for irrelevant noise.
Where used: Building LLM-as-judge evaluation harnesses, Comparing traces from different agent frameworks (e.g., LangGraph vs. custom ReAct loop), Corpus-level trace diagnostic tooling
Why learn this
- You will understand why feeding an unfiltered execution trace to an LLM judge leads to brittle, unreliable evaluations.
- You will learn to extract signal (system prompt, terminal state) from noise (retries, framework formatting) to ensure fair cross-run grading.
Code walkthrough
def run_eval_pipeline(raw_trace: list):
# Stage 1: Normalize a raw provider-specific trace into a canonical shape
canonical_trace = {
'system_prompt': extract_system(raw_trace),
'tool_schema': extract_tools(raw_trace),
'ordered_message_list': filter_noise_and_retries(raw_trace),
'terminal_state': extract_final_output(raw_trace)
}
# Stage 2: Feed only the canonical dict to the LLM-judge prompt
judge_prompt = f"""
Evaluate the agent based on its final state and key decisions.
System Context: {canonical_trace['system_prompt']}
Clean Execution Path: {canonical_trace['ordered_message_list']}
Final Output: {canonical_trace['terminal_state']}
"""
return llm_judge.score(judge_prompt)Focus: The pipeline is two stages: stage 1 takes a raw provider-specific trace and maps it into a canonical dictionary, filtering out noise. Stage 2 feeds only that canonical shape to the LLM-judge, ensuring the judge evaluates a clean, normalized representation. Helper names are placeholders — the control flow is the point.
Aha moment
# TEMPTING: Feed the raw trace directly to the judge.
def grade_agent_run(raw_trace):
# The judge sees 4 failed API retries and penalizes the agent.
return llm_judge.score(f'Evaluate this trace: {raw_trace}')Prediction: The judge will grade the run inconsistently, penalizing the agent for environmental noise like retries.
Common guess: The LLM judge can easily read the raw transcript and understand what happened.
LLM graders should evaluate 'what the agent produced, not the path it took'. Unfiltered path detail makes graders inconsistent; traces must be reduced to a canonical shape first.
Common mistakes
- Raw transcript is gradeable: You can hand an LLM judge the complete, unedited execution transcript (every retry, every failed tool call, every internal planning token) and trust it to correctly identify what actually happened. In reality, graders should evaluate 'what the agent produced, not the path it took'. Unfiltered path detail makes graders inconsistent and penalizes agents for irrelevant noise.
- Traces are format-uniform: Every agent framework's execution trace already comes in one consistent shape, so a single grading prompt template works across any agent you evaluate. In reality, different frameworks emit different raw event shapes (tool_use blocks vs. function_call objects). Traces must be imported through adapters into one shared schema before any cross-run comparison or scoring is meaningful.
Recall questions
- Why can't an LLM judge simply read the raw execution transcript of an agent run?
- What does 'normalizing' a trace into a canonical schema actually mean?
- What happens if a developer's agent retries a flaky search tool 4 times before succeeding, and the raw trace is given to an LLM judge?
Understanding checks
A developer's agent retries a flaky search tool 4 times before succeeding on the 5th call. What happens when the raw, unfiltered trace is fed directly to an LLM judge?
The LLM judge scores the run inconsistently across repeated grading passes, sometimes penalizing it for the 4 failed attempts and sometimes not.
Nothing in the raw trace tells the judge those attempts are temporary environmental noise rather than a core logic failure. Unfiltered detail confuses the grader, requiring the trace to be reduced to a canonical shape first.
A team reuses the same LLM-judge grading prompt they built for their LangGraph agent on a new agent built directly on the Anthropic Messages API. The judge silently misparses the new trace's tool-call fields and produces scores that look plausible but are grading the wrong spans. Why?
Different frameworks and providers emit different raw event shapes for the same logical step (e.g., `tool_use` blocks vs. `function_call` objects).
Without normalizing the two frameworks' traces into the same shape first, the grading prompt is essentially reading the wrong structure, leading to silent failures and meaningless cross-run comparisons.
Practice tasks
Design a Canonical Trace Schema
You are given a raw trace containing multiple tool-use retries and framework-specific metadata. Write a Python function `normalize_trace(raw_trace: dict) -> dict` that extracts only the `system_prompt`, `tool_schema`, `ordered_message_list` (stripping out failed retries), and `terminal_state` into a canonical dictionary.