A Taxonomy of Production Agent Failures
Overview
A systematic classification of how and why multi-agent systems fail, moving beyond generic `hallucination`.
To correctly diagnose an agent failure so you can apply the right fix (e.g., fixing a specification issue rather than arbitrarily tuning the prompt).
Where used: Debugging failing agents, Evaluating agent traces, System design interviews
Why learn this
- Debugging a failing agent without a shared vocabulary leads teams to blindly tune prompts regardless of the root cause.
- Anthropic and the research community have identified failures like `endless search` and `redundant coordination` that are invisible if you only check the final answer.
Code walkthrough
def evaluate_agent_trace(trace_data):
failures = {'specification': 0, 'misalignment': 0, 'verification': 0}
for step in trace_data:
if 'missing_context' in step.get('error_tags', []):
failures['misalignment'] += 1
elif 'schema_validation_failed' in step.get('error_tags', []):
failures['verification'] += 1
elif 'unclear_task_boundary' in step.get('error_tags', []):
failures['specification'] += 1
return failures
# Example usage:
# trace = [{'error_tags': ['schema_validation_failed']}]
# print(evaluate_agent_trace(trace))
# Output: {'specification': 0, 'misalignment': 0, 'verification': 1}Focus: This function walks an agent's execution log and labels each step against MAST's three top-level categories instead of just returning a single pass/fail bit.
Common mistakes
- Failures are all hallucination: The belief that every agent failure traces back to the LLM `hallucinating` a wrong fact or tool call. In reality, the MAST taxonomy shows failures fall largely into `specification issues`, `inter-agent misalignment`, and `task verification`, which are mostly not `hallucination`.
- Final answer correctness is the whole story: If an agent run's final output was acceptable, developers assume nothing meaningfully went wrong during execution. In reality, early agents made errors like spawning 50 `subagents` for simple queries or scouring the web endlessly, driving up costs significantly despite a correct final answer.
Recall questions
- According to the MAST taxonomy, what are the three main categories of multi-agent failure?
- A team's multi-agent pipeline keeps producing wrong final answers. They spend a sprint tightening prompts against hallucination, but the worker agent's output schema had silently changed and the supervisor never re-validated it. What kind of failure is this?
- A developer ships a research agent that gives a correct final answer in testing, unaware it spawned 40 redundant subagents to get there. Why is this problematic?
Understanding checks
If an agent correctly identifies a task but the supervisor fails to check whether the claimed outcome actually happened, which `MAST` category does this fall under?
`Task verification failure`.
Task verification failures occur when there is a lack of proper validation of the agent's work, rather than an issue with the task definition (specification) or communication between agents (misalignment).
What happens if a developer only monitors the final output of an agent pipeline without tracing the intermediate steps?
They will miss hidden failures like `endless search loops` or redundant `subagent` spawning, leading to unexpectedly high API costs and latency.
Outcome-only evaluation cannot detect trajectory-level inefficiencies and coordination failures that resolve before the final output is produced.
Practice tasks
Implement a Trace Labeler
Write a Python function that iterates through an agent's execution log and labels each step with one of the three MAST failure categories, producing a histogram of failures.
Continue learning
Previous: Multi-agent orchestration patterns