Source-linked AI summary
Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems
Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, Qingyun Wu
TL;DR
Automated failure attribution in LLM multi-agent systems is proposed to address the labor-intensive gap between benchmark evaluation and identifying responsible agents and steps. Using the Who&When dataset and three evaluated methods, the paper finds that attribution remains difficult, especially for decisive error steps, motivating further research.
Problem
Failure attribution is an overlooked, labor-intensive process for identifying system components that directly lead to task failures and guiding improvements.
Method
The paper introduces Who&When, with annotated failure logs from 127 LLM-powered multi-agent systems, and evaluates three automated failure attribution methods.
Results
8.77% accuracy was achieved by the best-performing method when identifying decisive error steps in the hand-crafted agentic system.
Takeaways & Limitations
The findings underscore the significant difficulty and complexity of automated failure attribution and the need for further research.
Takeaways & Limitations
Annotating decisive error agents and specific error steps requires substantial human effort, including parsing complex logs and assessing agent actions.
Abstract
from arXiv · showhide
Failure attribution in LLM multi-agent systems-identifying the agent and step responsible for task failures-provides crucial clues for systems debugging but remains underexplored and labor-intensive. In this paper, we propose and formulate a new research area: automated failure attribution for LLM multi-agent systems. To support this initiative, we introduce the Who&When dataset, comprising extensive failure logs from 127 LLM multi-agent systems with fine-grained annotations linking failures to specific agents and decisive error steps. Using the Who&When, we develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons. The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps, with some methods performing below random. Even SOTA reasoning models, such as OpenAI o1 and DeepSeek R1, fail to achieve practical usability. These results highlight the task's complexity and the need for further research in this area. Code and dataset are available at https://github.com/mingyin1/Agents_Failure_Attribution
1. Introduction
Failure attribution is a labor-intensive, expertise-dependent step in refining increasingly complex LLM multi-agent systems, yet it remains underexplored. The paper proposes automating this step and introduces a benchmark with fine-grained failure annotations to evaluate that goal.
- Motivation: Failure attribution links benchmark failures to the system components that directly cause them, guiding subsequent system refinement.It is part of an iterative cycle of evaluation, attribution, and refinement.
- Motivation: Manual attribution requires analyzing complex historical logs and navigating technical system details, while mapping results to components depends heavily on domain expertise.The growing number of system components makes this challenge increasingly difficult.
- Motivation: The paper formulates automated failure attribution as a research problem, drawing on LLM judgment to bridge evaluation results and failure analysis.The proposed automation aims to reduce time-intensive diagnostics so human effort can focus on improving system functionality.
- Contributions: The Who&When benchmark contains failure logs from 127 LLM-powered multi-agent systems with annotations for the responsible agent, decisive step, and failure reasons.It includes algorithmically generated and hand-crafted systems spanning realistic scenarios.
- Findings: The best-performing method achieved only 8.77% accuracy in identifying decisive error steps within hand-crafted agentic systems.The paper evaluates methods across model variations, historical context lengths, and query-label conditions.
2. Problem Formulation: Automated Failure Attribution in Multi-Agent Cooperation
The paper defines failure attribution as identifying the agent and time step whose corrected action changes a failed trajectory into a successful one. With turn-based agents and potentially multiple decisive errors, it selects the earliest decisive error as the principal cause.
- System Model: The formal setting models N agents acting at discrete time steps under a turn-based protocol, with exactly one agent acting at each step.The active agent is determined by the turn function ϕ(t).
- System Model: A trajectory records alternating states and actions until a terminal time step or terminating state, and its outcome indicates whether the system ultimately fails.The trajectory is represented as s0, a0, s1, a1, …, sT.
- Decisive Error: A decisive error is an agent-time pair whose corrected action changes the trajectory outcome from failure to success.The correction replaces the action at that step while later actions are adjusted to preserve correctness.
- Decisive Error: The failure-responsible agent is the agent in a decisive error pair, while the decisive error step is the pair’s exact time step.These labels define the two prediction targets for automated attribution.
- Objective: When multiple decisive errors exist, the objective selects the decisive pair with the earliest time step as the principal cause of failure.The objective identifies the pair (i*, t*) with the earliest t among decisive errors.
3. The Who&When Dataset
Who&When is a dataset of annotated failure logs from 127 LLM multi-agent systems, designed to identify the responsible agent and decisive error step. It combines diverse system types and evaluates the labor and difficulty involved in producing these annotations.
- Dataset composition: Who&When contains failure logs from 127 LLM multi-agent systems, including algorithm-generated and human-crafted systems.Annotations identify responsible agents, decisive error steps, and natural-language reasons, yielding 184 failure annotation tasks.
- Dataset composition: Each dataset instance pairs a real-world query with a system failure log, agentic-system information, and failure annotations.System information can include prompts, tools, and agent names tailored to the query.
- Evaluation: Who&When evaluates failure attribution using Agent-Level Accuracy and Step-Level Accuracy, with step identification imposing higher requirements.The metrics measure correct identification of responsible agents and decisive error steps.
- Agentic systems: The dataset includes automatically generated systems built with CaptainAgent and a mature hand-crafted Magnetic-One system.Magnetic-One contributes failures from AssistantBench and selected GAIA instances, while the remaining GAIA data is excluded because its logs are too long to annotate.
- Annotation process: Manual annotation requires experts to inspect complex logs, trace agent reasoning, and judge which action directly caused failure.Three annotators spent 30.9, 30.2, and 23.2 human hours, respectively.
- Annotation process: Uncertainty ranged from 15% to 30% across annotators, with disagreement remaining before consensus discussions.Multiple rounds used standardized guidelines, uncertainty categorization, discussion, and voting to reach agreement.
4. Can LLMs help identify When and Which agent causes task failures?
The study evaluates whether LLMs can automate identification of the agent and step responsible for multi-agent task failures. Results show a trade-off: broader context improves agent-level attribution, while incremental processing improves step-level attribution, but performance deteriorates with longer logs.
- Overall Performance: All-at-once achieves higher agent-level accuracy than step-by-step, while binary search falls between them.All-at-once exceeds step-by-step by 19.13 and 20.69 percentage points with ground truth, and by 25.1 and 20.69 points without ground truth.
- Overall Performance: Step-by-step performs best for step-level accuracy, whereas all-at-once performs worst and can fall below random.Step-by-step wins in 3 of 4 cases, consistent with its more focused, incremental processing.
- Overall Performance: Ground-truth labels improve attribution accuracy for all three methods across all reported metrics.The final query answer provides a useful reference signal that can alert judge LLMs when an agent leads the system in a wrong direction.
- Performance Across Varying Context Lengths: Attribution performance declines as failure-log context length increases, with step-level accuracy more sensitive than agent-level accuracy.At maximum context length, all three methods converge to near 0%; the step-by-step decline is particularly pronounced.
- Step-Level Accuracy Under Different Tolerances: Tolerance changes the preferred method: step-by-step is strongest at tolerance 0 or 1, while all-at-once becomes more competitive as tolerance increases.Step-by-step remains better aligned with accurate predictions when high precision is required.
- Combined Methods and Statistical Attribution: The hybrid method outperforms the individual methods on step-level accuracy by narrowing candidate steps, but requires two algorithms sequentially.The three baseline methods also provide more actionable information statistically than at the individual-instance level.
5. Related Works
Prior work uses LLMs as evaluators and reward models to reduce human effort and assess outputs or intermediate reasoning. However, automated failure attribution for multi-agent systems remains insufficiently addressed.
- LLMs-as-judges evaluate tasks against predefined standards, including chat conversations and text summarization.
- Agentic-system research has used LLMs to analyze feedback signals, guide corrective actions, and evaluate agentic systems.
- Most reward models predict preferences or evaluate reasoning processes step by step rather than assessing only final answers.
6. Conclusion
The paper formulates automated failure attribution in LLM multi-agent systems as an overlooked research area and introduces resources and methods to study it. Its findings emphasize the task’s difficulty and the need for further research.
- The paper proposes and formulates automated failure attribution in LLM multi-agent systems as a new research area.
- The Who&When dataset contains 127 multi-agent systems with extensive failure logs annotated with failure details.
- Three automated failure attribution methods are developed and evaluated to highlight the task’s challenges and complexities.
Impact Statement
The paper describes potential benefits and risks of improving LLM multi-agent systems through failure attribution. Its methods analyze failure logs to identify responsible agents and decisive error steps.
- Failure attribution mechanisms could support efficient development and improvement of LLM-powered multi-agent systems across domains.
- Granting multi-agent systems the ability to modify external environments, such as executing computer code, could produce unintended consequences.
- The Step-by-Step method supplies the query and progressively longer failure-log prefixes to an LLM until it indicates an error.
- Binary Search provides log segments to an LLM, narrows the search interval based on its error judgment, and identifies the responsible agent at the resulting step.
B.1. Ablation of Reasoning Prompts
The ablation examines explicit reasoning prompts in all-at-once and step-by-step attribution methods. Removing these prompts causes a significant performance drop, while the benchmark includes varied task and log characteristics.
- B.1. Ablation of Reasoning Prompts: The all-at-once and step-by-step approaches explicitly ask LLMs to provide reasons alongside failure-attribution judgments.
- B.1. Ablation of Reasoning Prompts: Binary Search is excluded from the reasoning-prompt ablation because its prompt uses a simple classification task without reasoning mechanisms.
- B.1. Ablation of Reasoning Prompts: Figure 7 compares ablations of explicit reasoning prompts for the all-at-once and step-by-step methods.
- B.1. Ablation of Reasoning Prompts: Removing explicit reasoning prompts produces a significant drop in failure-attribution performance.
- C.1. Overview: Table 5 reports task totals by category and the maximum and minimum numbers of agents and log lengths.
- C.1. Overview: The Who&When dataset contains 184 failure-annotation tasks from hand-crafted and algorithm-generated agentic systems.
- C.1. Overview: Its failure logs cover diverse scenarios with varying numbers of agents and interaction lengths.
C.2. Data Distribution
Figure 8 shows the number of agents and total failure-log length for each Who&When instance; overlapping tasks can make some plotted data points appear sparse.
- The visualization jointly represents agents involved and total failure-log length for Who&When instances.
C.3. Data Example
A Who&When example annotates the agent responsible for failure, the corresponding error step, and a natural-language explanation of the failure reason.
- Each example links a failure-responsible agent to its corresponding error step in the failure log.
- Annotations also include a plain-language explanation of why the failure occurred.
D. Brief Cost Analysis
The cost analysis compares input-token requirements across all-at-once, step-by-step, and binary-search attribution methods. All-at-once is cheapest, while the other methods depend on log structure and decisive-error location.
- All-at-Once: All-at-once sends the full context in one input and requires only a single inference step, giving it the lowest cost.
- Step-by-Step: Step-by-step processes log segments incrementally until the decisive error step i* is found, so cost depends on that step’s location.
- Binary Search: Binary search repeatedly halves the failure log and requires ⌈log2(n)⌉ iterations to narrow the search to one step.
- Cost Comparison: The three methods’ costs depend on log size n, average tokens per entry T_l, and decisive-error position i*, so method choice should match budget and use case.