Source-linked AI summary
Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation
Yeonjun In, Mehrab Tanjim, Jayakumar Subramanian, Sungchul Kim, Uttaran Bhattacharya, Wonjoong Kim, Sangwu Park, Somdeb Sarkhel, Chanyoung Park
TL;DR
MAS failure attribution is commonly evaluated as if each failure had one deterministic root cause, despite plausible alternatives arising from inter-agent dependencies and execution strategies. The paper introduces multi-perspective failure attribution, MP-Bench, and an evaluation protocol that assesses diverse attributions with rationales. Experiments show that modern LLMs can produce diverse, human-aligned attributions when evaluated under this formulation, while the benchmark remains limited in domain and framework coverage.
Problem
Existing MAS failure-attribution benchmarks assume a single deterministic failure-inducing step, although failures can admit multiple plausible attributions from different analytical perspectives.
Method
The paper proposes multi-perspective failure attribution and introduces MP-Bench with expert annotations, rationales, and an evaluation protocol for attribution quality and reasoning.
Results
Modern LLMs produce diverse, human-aligned failure attributions under multi-perspective evaluation, challenging prior conclusions based on restrictive deterministic benchmarks.
Takeaways & Limitations
Reliable MAS debugging requires benchmarks and evaluation protocols that represent attribution ambiguity rather than treating one oracle label as uniquely correct.
Takeaways & Limitations
MP-Bench currently focuses on general-purpose assistant tasks and has limited scale and multi-agent framework diversity.
Abstract
from arXiv · showhide
Failure attribution is essential for diagnosing and improving multi-agent systems (MAS), yet existing benchmarks and methods largely assume a single deterministic root cause for each failure. In practice, MAS failures often admit multiple plausible attributions due to complex inter-agent dependencies and ambiguous execution trajectories. We revisit MAS failure attribution from a multi-perspective standpoint and propose multi-perspective failure attribution, a practical paradigm that explicitly accounts for attribution ambiguity. To support this setting, we introduce MP-Bench, the first benchmark designed for multi-perspective failure attribution in MAS, along with a new evaluation protocol tailored to this paradigm. Through extensive experiments, we find that prior conclusions suggesting LLMs struggle with failure attribution are largely driven by limitations in existing benchmark designs. Our results highlight the necessity of multi-perspective benchmarks and evaluation protocols for realistic and reliable MAS debugging.
1 Introduction
Existing MAS failure-attribution work assumes one deterministic failure-inducing step, although failures often support multiple plausible perspectives. The paper proposes a multi-perspective paradigm, MP-Bench, and a tailored evaluation protocol, finding that LLMs can produce diverse, human-aligned attributions under this setting.
- Existing approaches automate failure attribution but typically label each failed execution with one deterministic failure-inducing step.This inherits the single-oracle assumption of benchmarks such as Who&When.
- MAS failures often admit multiple plausible attributions because interdependent agents can follow diverse valid trajectories toward the same goal.Different analysts may accept different strategies, such as web-first or code-first execution.
- Deterministic benchmarks can misjudge reasonable predictions and fail to capture diverse causal patterns, reducing evaluation reliability and hindering debugging.
- The paper proposes multi-perspective failure attribution, pairing multiple plausible failure-inducing steps with explicit reasoning and actionable diagnostic signals.
- MP-Bench uses expert annotations and evaluates both multi-perspective step identification and the reliability of supporting reasoning.Its annotations include failure-inducing steps and explicit rationales from multiple expert perspectives.
- Extensive experiments find that repeated sampling lets state-of-the-art LLMs identify diverse attributions whose reasoning aligns closely with human experts.The findings challenge conclusions from deterministic benchmarks that LLM attribution performance is near random.
2 Related Works
Related work frames MAS failure attribution as a practical debugging problem while exposing limitations in existing benchmarks. This paper extends perspectivism to technically grounded MAS diagnosis through expert multi-perspective annotations and evaluation.
- MAS failure attribution identifies failure-inducing steps from execution traces to support system debugging and refinement.
- Benchmarks for Failure Attribution of MAS: Existing benchmarks differ in scope, with MAST restricted to execution-level failure modes rather than step-level attribution.
- Benchmarks for Failure Attribution of MAS: MP-Bench is introduced as the first multi-perspective MAS failure-attribution benchmark, built with expert recruitment and rigorous labeling for high-quality annotations.
- Benchmarks for Failure Attribution of MAS: The benchmark includes annotator rationales and an evaluation framework assessing both attribution quality and the reasoning underlying each attribution.
- Perspectivism and Annotator Disagreement: Unlike perspectivist work focused mainly on socially subjective judgments, this study examines disagreement arising from technical reasoning and MAS diagnostic complexity.
3 Task Formulation
The task formulation contrasts deterministic attribution, which selects a fixed oracle set of failure steps, with a multi-perspective formulation that represents several plausible steps and rationales. Success therefore means recovering multiple expert-aligned causal explanations rather than matching one label.
- 3.1 Preliminaries: An MAS execution is modeled as a finite trajectory produced by turn-based interactions among LLM-powered agents.The system is defined over agents, states, actions, transition dynamics, and an action-selection function.
- 3.1 Preliminaries: Deterministic attribution maps each failed trajectory to a fixed set of responsible time steps, usually simplified to one step.This treats attribution as an objective labeling problem with a single correct attribution set.
- 3.2 Multi-Perspective Failure Attribution: The paper defines MAS failure attribution as inherently non-deterministic because valid execution trajectories and expectations of correct strategy can differ.
- 3.2 Multi-Perspective Failure Attribution: Multi-perspective attribution outputs plausible failure-inducing steps together with rationales explaining each judgment.
- 3.2 Multi-Perspective Failure Attribution: Each attribution pairs a time step with an explicit perspective-specific rationale, and the number of attributions can vary across executions.
- 3.2 Multi-Perspective Failure Attribution: Different valid attributions may identify different responsible steps and agents, so evaluation targets recovery of multiple reasonable explanations aligned with expert judgments.
- 3.2 Multi-Perspective Failure Attribution: The formulation reframes attribution from selecting one failure-inducing step to characterizing a space of plausible causal explanations.
4 MP-Bench: Multi-Perspective Failure Attribution Benchmark
MP-Bench is an expert-annotated benchmark designed to capture multiple perspectives on MAS failure attribution. It converts independent step-level judgments and rationales into consensus-based debugging priorities.
- Dataset construction: MP-Bench collects execution logs from diverse MAS configurations and tasks to represent varied failure patterns.The benchmark includes hand-crafted and fully automated systems evaluated on GAIA and AssistantBench tasks.
- Annotation process: Each execution step receives independent failure-inducing labels, failure reasons, and ideal-action descriptions from expert annotators.Annotators review complete execution traces and task descriptions without being given guiding perspectives.
- Annotation process: MP-Bench preserves attribution ambiguity by allowing annotators to identify different failure-inducing steps and explain their judgments independently.This design captures perspective-dependent interpretations of correct execution trajectories.
- Annotation consolidation: Steps are ranked by annotator consensus rate, turning multi-perspective labels into an actionable prioritization for debugging.Steps identified by every annotator receive the highest priority, while lower-consensus steps represent perspective-dependent failures.
- Annotation quality and effort: 289 execution logs receive triple expert annotation, prioritizing annotation quality and reasoning coverage over dataset scale.The annotation effort totals approximately 346 expert-hours, while the authors acknowledge that this limits immediate scalability.
- Annotation analysis: 56.1% of steps are labeled failure-inducing by only one annotator, compared with 16.2% labeled by all three.The distribution indicates substantial perspective dependence in MAS failure attribution.
5 Evaluation Protocol
The evaluation protocol treats failure attribution as ranking under ambiguity rather than single-label classification. It separately evaluates whether systems recover diverse failure steps and whether their rationales align with expert judgments.
- Evaluation goals: MP-Bench evaluates whether LLM systems capture failures from multiple perspectives and provide reliable attribution reasoning.These are the protocol’s two explicit evaluation questions.
- Ranking-based attribution: The protocol replaces deterministic single-step classification with ranking-based evaluation because one execution can admit multiple plausible attributions.This formulation aligns the task with attribution ambiguity.
- Ranking-based attribution: LLMs are sampled N times at temperature τ, producing independent failure-step rankings with associated reasons and ideal actions.The resulting outputs are compared with rankings derived from human annotations.
- Ranking-based attribution: nDCG@K measures agreement with human rankings while emphasizing highly salient failure steps relevant to debugging priorities.The protocol reports ranking quality rather than only exact single-step matches.
- Reasoning evaluation: An LLM-as-a-Judge framework scores consolidated attribution reasoning for consistency with human judgments, execution grounding, and explanatory adequacy.GPT-5.1 serves as the primary judge, with GPT-4.1 and Claude-Sonnet-4.5 used to test robustness.
6 Experiment
Experiments show that stochastic and cross-model diversity helps LLMs recover multi-perspective failure attributions with human-aligned reasoning. Increasing temperature, sample count, and model-family diversity improves the evaluation profile.
- Main results: Multiple LLM samples produce disagreement patterns that closely mirror human annotator disagreement, supporting non-deterministic failure attribution.The result indicates that sampled outputs can represent distinct plausible perspectives rather than one fixed root cause.
- Main results: With N=3 and τ=1, LLMs identify multiple reasonable failure-inducing steps, and all models score substantially above a random baseline.Claude-Sonnet-4.5 performs strongest among evaluated models, while Qwen3-8B shows limited attribution and reasoning capability.
- Main results: LLM-generated attributions receive high LLM-as-a-Judge scores for reasoning that aligns with human judgments and supports MAS debugging.The reasoning is presented as a meaningful diagnostic signal alongside the predicted failure steps.
- Sampling temperature: Performance at τ=0 is consistently worse than at higher temperatures, while introducing stochasticity at τ≥0.3 substantially improves nDCG@5 with exponential gain.The analysis attributes the difference to deterministic decoding constraining the output space and limiting attribution diversity.
- Number of LLM runs: Increasing the number of LLM runs from N=3 to N=10 consistently improves nDCG@5 with exponential gain across models.More samples produce more stable and reliable aggregated failure-attribution rankings.
- Multi-LLM collaboration: Multi-LLM systems consistently outperform single-LLM counterparts, with the largest gains from combining models across families.Cross-family combinations introduce qualitatively different attribution perspectives and complementary reasoning patterns.
- Practical guidance: The experiments motivate aggregating multiple perspectives, using nonzero temperature, and combining different model families for more informative failure diagnosis.These recommendations follow the observed relationships between diversity and attribution performance.
7 Conclusion
The paper argues that MAS failure attribution should be evaluated as an inherently multi-perspective task rather than a deterministic labeling problem. MP-Bench and its evaluation framework support assessment of both attribution quality and reasoning for reliable MAS debugging.
- Conclusion: Deterministic formulations fail to capture the inherently multi-perspective nature of real-world MAS failures.The conclusion frames attribution ambiguity as central to realistic benchmark design.
- Conclusion: MP-Bench combines expert annotations with multi-perspective evaluation to assess failure attribution quality and reasoning.The benchmark is presented as a foundation for reliable MAS diagnosis and debugging.
8 Limitations and Future Work
MP-Bench’s current scope is limited to general-purpose assistant tasks and a relatively small, less diverse set of MAS frameworks. Future work should broaden domains and scale while preserving annotation quality.
- Domain Coverage: MP-Bench currently focuses on general-purpose assistant tasks, limiting coverage of specialized domains such as scientific research, software engineering, and creative tasks.Future expansion to these domains would broaden applicability.
- Scale and Framework Diversity: The quality-first annotation approach limits the benchmark’s scale and the diversity of covered multi-agent system frameworks.The authors suggest hybrid expert–automated annotation pipelines and additional frameworks as possible directions.
- Future Work: MP-Bench provides a gold-standard foundation for multi-perspective failure attribution research despite these limitations.The paper presents it as a resource for advancing this research area.
C.1 Sensitivity Analyses on LLM choice for Consolidation and LLM-as-a-Judge
Sensitivity analyses show that relative model rankings remain robust across consolidation and judge model choices, even though absolute scores vary with judge stringency. Agreement is strong for both hand-crafted and automatic MAS executions.
- Hand-Crafted MAS Executions: Average τ=0.896, with 31 of 36 configuration pairs statistically significant and all 36 satisfying p<0.1 for hand-crafted MAS executions.Observed τ values range from 0.733 to 1.00.
- Automatic MAS Executions: Average τ=0.904, with 30 of 36 configuration pairs satisfying p<0.05 and all 36 satisfying p<0.1 for automatic MAS executions.Observed τ values range from 0.733 to 1.00.
- Conclusion: Relative model rankings remain highly stable across configurations, although absolute scores vary with the selected judge model’s evaluation stringency.This supports interpreting ranking differences as attribution-quality differences rather than artifacts of particular consolidation or judge choices.
C.2 Scaling 𝑁in Multi-LLM Failure Attribution
The multi-LLM scaling analysis varies N from 1 to 10, corresponding to 3 to 30 total LLM runs, and evaluates attribution using average nDCG@5 across hand-crafted and automatically generated MAS datasets. The supplied passages describe the setup and evaluation materials but do not report the resulting scaling outcome.
- Scaling Setup: N varies from 1 to 10, corresponding to 3 to 30 total LLM runs when three LLMs collaboratively perform failure attribution.Performance is evaluated using average nDCG@5 with exponential gain across hand-crafted and automatically generated MAS datasets.
- Scaling Results: Figure 5 reports failure attribution performance across varying numbers of LLM runs N on MP-Bench, including OSS and Sonnet models.OSS denotes GPT-OSS-120B, while Sonnet denotes Claude-Sonnet-4.5.
- Evaluation Procedure: The LLM-as-a-Judge prompt compares model predictions with human expert annotations using the execution log, failure reason, and ideal action.The prompt evaluates whether reasoning is plausible and grounded rather than exactly worded.
- Evaluation Criteria: The evaluation jointly considers failure reason and ideal action through reasoning alignment, execution-context faithfulness, coverage and completeness, and plausibility of ideal action.These criteria compare model reasoning with human annotations and the execution trace.