Building Eval Harnesses for Multi-Step Tasks
Overview
An agent evaluation harness is an automated testing infrastructure that combines mock environments, execution trace normalization, and code-based graders to measure if an agent successfully achieved a multi-step task.
Because agents take actions that alter state (like deleting rows or writing files), simply asking an LLM judge 'did the agent succeed?' based on a chat transcript is unreliable. You must use deterministic unit tests to verify real environment changes.
Where used: Enterprise agent testing pipelines, Continuous integration for AI systems, Debugging multi-step agent trajectories
Why learn this
- You cannot safely deploy an autonomous agent if you cannot mathematically prove it reliably succeeds.
- Relying purely on LLM-as-judge for agents leads to silent failures where agents hallucinate success.
- Evaluating agents requires structuring raw, messy execution logs into standard schemas so graders can consistently read them.
Code walkthrough
def test_agent_refund_task(mock_db):
# 1. Run the agent in a mocked environment
agent = RefundAgent(db=mock_db)
trace = agent.run("Refund user 123 for their last purchase")
# 2. Extract normalized steps from the trace
steps = normalize_trace(trace)
# 3. Trajectory Eval (LLM Judge)
assert llm_judge(steps) == "SUCCESS", "Agent failed to plan correctly"
# 4. Outcome Eval (Code Grader)
# The actual source of truth: did the DB change?
user_record = mock_db.execute("SELECT status FROM users WHERE id = 123")
assert user_record['status'] == 'refunded', "Agent claimed success but DB didn't update"Focus: The separation between trajectory eval (llm_judge) and the outcome eval (code grader asserting the database state). Helper names are placeholders — the control flow is the point.
Common mistakes
- Assuming LLM-as-judge is sufficient for evaluation: Many developers pass the final conversation transcript to an LLM-as-judge and ask if the agent succeeded. But agents alter state. An LLM judge cannot verify if a file was actually written or a database row deleted. Harnesses use code-based graders (unit tests) to assert environment state, combined with LLM judges for qualitative trajectory analysis. Helper names are placeholders — the control flow is the point.
- Scoring raw logs directly without normalization: It's a mistake to build graders that read directly off whatever raw event log a framework emits. Different frameworks emit structurally different traces. Harness tooling explicitly builds a modular importer to bring heterogeneous raw traces into one shared schema before any grader runs against them.
Recall questions
- What is the primary difference between a trajectory evaluation and an outcome evaluation?
- Why are code-based graders essential for evaluating agent outcomes?
- A developer reuses a code-based grader built for Agent A on Agent B's raw tool-use API, but the grader crashes. What architectural component is missing?
Understanding checks
If you only use an LLM-as-judge on an agent's raw chat transcript, what happens when the agent hallucinates calling a refund tool?
The judge will likely score the agent as successful based on the hallucinated text, even though no refund occurred.
An LLM judge reading a transcript cannot verify external state changes. It only evaluates what the agent claims happened.
You wrote a code-based grader for a LangGraph agent, but it silently fails to read trace data when evaluating a custom REST agent. What step was skipped?
The raw execution traces were not normalized into a shared schema before grading.
Different frameworks emit structurally different trace logs. A modular importer must normalize these logs into a consistent format so the grader knows where to look for data.
Practice tasks
Build a Code-Based Grader
Write a Python function `grade_refund_task(db_connection)` that queries the mock database for user ID 123 and asserts their `status` is 'refunded'.