Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attribution

1 Introduction

• Multi-agent systems can compound early mistakes, so identifying the initial error’s source at both agent and step level is crucial to mitigating failures.The paper describes traces as involving interdependent interactions, large contexts, and subtle, context-dependent errors.Source: §1 · pp. 1–2
• All-at-once, step-by-step, and binary-search approaches analyze interaction logs differently, but complex patterns remain difficult to assess consistently.The paper notes that even leading closed- and open-source LLMs struggle on the Who&When benchmark [9].Source: §1 · pp. 1–2
• ECHO combines hierarchical representations of entire traces, independent objective analyses across contexts, and consensus voting to attribute errors.Source: §1 · p. 2

2.1 Evaluation of LLM Agents

• Agent evaluation benchmarks cover task completion, realistic multi-turn tasks, machine-learning experimentation, coordination, and tool-use capabilities.Source: §2.1 · p. 2

2.2 Error Attribution

• Error-attribution work examines failure types and reasoning steps, while Who&When [9] and ECHO address multi-agent cases by identifying responsible agents and erroneous steps.Source: §2.2 · p. 2

3 ECHO Methodology

• The methodology combines context understanding, error analysis, and decision synthesis through hierarchical representation, decoupled analysis, and confidence-weighted voting.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Hierarchical Context Representation

• Four context layers range from complete immediate interactions (L1), through tactical and strategic compression (L2–L3), to milestone-based global patterns (L4).The hierarchy is intended to capture both local and global interaction patterns.Source: §3.1 · p. 3

3.2 Objective Analysis

• A panel of analysts independently evaluates all steps through the hierarchy, using distinct perspectives and structured evidence, likelihood scores, conclusions, and alternatives.The analyst roles include conservative, liberal, detail-focused, pattern-focused, skeptical, and general perspectives.Source: §3.2 · p. 4

3.3 Consensus Voting

• Consensus voting weights analyst attributions by confidence above a threshold, resolves conclusions hierarchically, and examines disagreement across analysts.High disagreement or conflicting high-confidence attributions can prompt additional review.Source: §3.3 · pp. 4–5

4.1 Experimental Setup

• The Who&When evaluation covers algorithm-generated and hand-crafted failure logs, tested with and without ground truth at agent and step levels.The implementation used Claude 3.5 Sonnet v2 and three randomly sampled analysis agents; it assumes position-based context importance, independent analysts, and primarily additive errors.Source: §4.1 · p. 5

4.2.1 Performance of ECHO

• ECHO achieved 68.4% agent-level accuracy on hand-crafted data and 68.8% on algorithm-generated data, with statistically significant performance across both datasets.The results report minimal degradation when ground truth is withheld.Source: §4.2.1 · p. 6, Table 1 · p. 6
• Exact step matches were 27–28%; accuracy reached 42.1% on hand-crafted data with ground truth at ±3 steps and 61.4% at ±5 steps.ECHO still outperformed prior methods with statistical significance on step-level attribution.Source: §4.2.1 · p. 6

The Question of Unifying or Decoupling Objective Analyses

• For shorter algorithm-generated traces, unified analysis reached 65.1% agent-level and 46.1% step-level accuracy, while decoupling raised agent accuracy to 68.8% but lowered step precision to 28.8%.The unified and decoupled analyses used approximately 13K and 6K+7K tokens, respectively.Source: The Question of Unifying or Decoupling… · p. 7
• For longer hand-crafted traces, decoupling raised agent-level accuracy from 61.0% to 68.4% and step-level accuracy from 23.2% to 28.1%, while reducing tokens from approximately 67K to 13K+19K.The reported results indicate that trace length affects which attribution setup performs better.Source: The Question of Unifying or Decoupling… · p. 7

4.2 Comparative Analysis of Implementations

• The comparison progresses from fixed-window to hierarchical, objective, and decoupled analysis; objective analysis cuts token use by 60–110× on hand-crafted cases.It also reduces token use by 25–30× on algorithm-generated cases while improving accuracy.Source: §4.2 · pp. 5–6, The Computational Overhead Resolution… · p. 7

Switching from Fixed-Window Context to Hierarchical Context

• Hierarchical context improved hand-crafted agent-level accuracy by 16.1% over fixed ±1-step context; algorithm-generated accuracy rose by 6.2% at agent level and 3.8% at step level.The hand-crafted step-level improvement was 1.9% and was not statistically significant; that analysis covered shorter traces due to computational constraints.Source: Switching from Fixed-Window Context to… · p. 8

Additional Ablation Studies

• On hand-crafted data with ground truth, Claude-3.7 Sonnet reached 78.8% agent-level accuracy versus Haiku’s 61.8%; expanding the analyst panel from three to six yielded minimal gains while doubling token use.LLM-based context extraction reached 75.0% agent-level and 43.8% step-level accuracy but used approximately 166K tokens, including approximately 145K for extraction.Source: Additional Ablation Studies · p. 8, Table 3 · p. 8

• ECHO combines hierarchical context representation, decoupled objective analysis, and confidence-weighted consensus voting for error attribution.Source: §5 · p. 8
• The authors report substantial improvements over existing baselines, particularly on longer traces where traditional approaches become prohibitive.Source: §5 · p. 8
• The authors point to reinforcement learning and single-agent prompt refinement as broader applications, and propose future work on context preservation, debate protocols, severity metrics, and benchmark expansion.Source: §5 · p. 8

A.1 ECHO Algorithm

• For each agent, hierarchical extraction creates four context levels, from full local context and key decisions to compressed summaries and remaining-trace milestones.Source: §A.1 · p. 12
• The algorithm separately evaluates agent and step attribution with k analysts, optionally supplying ground truth, and records each analysis with a confidence score.Source: §A.1 · p. 12
• Consensus voting retains analyses meeting the minimum confidence threshold, then computes weighted aggregates and analyzes disagreement before returning results.Source: §A.1 · p. 12

A.2 Fixed-Window Context

• The extractor takes a conversation history and returns tuples containing the previous, current, and next agent turns.Source: §A.2 · p. 13
• It iterates through each position in the conversation history to build one context tuple per turn.Source: §A.2 · p. 13
• At the first and last positions, the missing previous or next turn is represented as None.Source: §A.2 · p. 13

A.3 Hierarchical Context Extraction

• The routines cap extracted key decisions and milestones using a maximum-word limit.Source: §A.3 · pp. 13, 15
• Context extractors use regex patterns for handoff, decision quality, error propagation, or general content, then fall back to sentence extraction or truncation.Source: §A.3 · p. 15
• The hierarchy stores immediate, nearby, distant, and milestone context levels for each current agent.Source: §A.3 · pp. 16–17
• Agents at distance one retain full content, those two to three steps away are represented by key decisions, and more distant agents receive summaries or milestones.Source: §A.3 · p. 17

A.4 Context Step Aware Agent

• The basic context includes the previous, current, and next agents, while the hierarchical version considers the entire conversation.Immediate context provides full adjacent-agent details, nearby context summarizes agents two to three steps away, and distant context briefly summarizes agents four or more steps away.Source: §A.4 · pp. 18, 20
• The agent is instructed to analyze information received, its own output, effects on the next agent, and ground-truth evidence to argue that its step caused the wrong answer.Its output is organized under Purpose, Assumptions and Information, Errors, and Evidence headings.Source: §A.4 · p. 18
• Both prompt variants ask the model to make a strong argument for why the selected agent caused the final wrong answer.The hierarchical prompt also asks it to consider how information and errors propagated across the conversation.Source: §A.4 · pp. 19–20
• The method records the agent’s analysis with its step ID and agent name, and the hierarchical variant additionally labels the context type.Both records include token usage.Source: §A.4 · pp. 20–21

A.5 Objective Analysis Agent

• The agent evaluates all agents objectively, identifies likely contributors and the mistake step, and returns confidence scores and reasoning.Its instructions include considering whether no single agent is clearly at fault.Source: §A.5 · pp. 21–22
• The analysis uses a conversation summary and hierarchical context with full details for immediate agents, key decisions for nearby agents, and brief summaries for distant agents.The summary preserves original step indices and agent names, roles, and content.Source: §A.5 · pp. 22, 24
• The output schema accommodates single-agent or multi-agent conclusions, an attribution, a mistake step, confidence, reasoning, and alternative hypotheses.It also requests individual agent evaluations with error likelihoods and evidence.Source: §A.5 · p. 22
• Analyst focus can be conservative, liberal, detail-focused, pattern-focused, skeptical, or general, with different attribution preferences and evidence priorities.The general focus calls for a balanced perspective and assessment of impactful mistakes.Source: §A.5 · p. 25

A.6 Judge Agent

• The Final Judge weighs competing Context-Aware Step Agent arguments, tracing evidence and causal reasoning backward from the wrong answer to identify the mistake agent, step, and reason.Arguments are structured around each step’s purpose, assumptions and information, errors, and supporting evidence; the judge considers agents actively making decisions or providing information.Source: §A.6 · pp. 25–26

A.7 Consensus Voting

• Consensus voting uses weighted-confidence totals to select among objective analysts’ single-agent or multi-agent conclusions, considering only conclusions meeting the 0.3 confidence threshold.It aggregates agent evaluations and alternative hypotheses, and derives agent and mistake-step attributions from votes supporting the winning conclusion type.Source: §A.7 · pp. 27–29
• With conversation history, the implementation keeps only integer step predictions within its bounds; without history, it selects the highest-scoring available step.If no step votes are available, the selected mistake step is None.Source: §A.7 · p. 31
• Review is required when confidence spread exceeds 0.5 or when more than two conclusion types each have votes.The disagreement analysis records the number of conclusion types, confidence spread, and whether the high-disagreement condition is met.Source: §A.7 · p. 32
• For the selected conclusion, synthesized reasoning reports the analyst count and average confidence, then uses the first reasoning and notes other supporting analyses.The implementation truncates the primary reasoning to 200 characters when additional reasoning entries are present.Source: §A.7 · p. 32