PaperlayerView exact revision on arXiv

Source-linked AI summary

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems

Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu, Xu Zhang, Zheming Zhuang, Dawei Feng, Bo Ding, Huaimin Wang

arXiv:2610.11600v1cs.AI

TL;DR

Failure attribution in multi-agent systems is difficult because observed symptoms do not necessarily reveal the decisive error, whose correction would recover a failed execution. EMFA models propagation across dependent steps and persistent interaction loops, then screens and verifies candidate errors. On Who&When, it achieves state-of-the-art step-level attribution, improving previous best results by 3.45 and 4.40 percentage points on the two subsets.

  • Problem

    Failure attribution must identify the decisive agent–step pair whose correction would recover a failed execution, despite corrected deviations and downstream symptoms that can obscure it.

    Source: Introduction · p. 2, Problem Formulation · p. 3

  • Method

    EMFA structures failed trajectories, models cascading error propagation and persistent interaction loops, and uses candidate screening followed by counterfactual verification to identify decisive errors.

    Source: Introduction · p. 2, Method · p. 3

  • Results

    EMFA achieves state-of-the-art step-level attribution on Who&When, improving over the previous best by 3.45 and 4.40 percentage points on the Hand-Crafted and Algorithm-Generated subsets, respectively.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Modeling propagation and persistent loops helps distinguish decisive errors from downstream failure symptoms in failed multi-agent executions.

    Source: Introduction · p. 2

    EMFA adds inference-time overhead from explicit trajectory structuring and propagation modeling.

    Source: Conclusion and Limitations · p. 7

Abstract

from arXiv · show

LLM-based multi-agent systems (MASs) are increasingly used to solve complex tasks through coordinated reasoning, tool use, and interaction with external resources. However, attributing failures in such systems remains challenging because the observed outcome often does not directly reveal the error responsible for the failed execution. In this work, the attribution target is the decisive error, defined as the agent--step pair whose correction would recover the failed execution. Existing approaches largely identify suspicious steps without explicitly modeling how errors propagate across interactions or persist in unresolved loops, making decisive errors difficult to distinguish from downstream failure symptoms. We propose \textbf{E}rror-Propagation \textbf{M}odeling for \textbf{F}ailure \textbf{A}ttribution (\textbf{EMFA}). EMFA constructs a structured representation of the failed trajectory, models both cascading propagation and persistent interaction loops, and uses propagation-aware candidate screening followed by counterfactual verification to identify the decisive agent--step pair. On the Who\&When benchmark, EMFA achieves state-of-the-art step-level attribution accuracy and remains competitive at the agent level. It improves the previous best step-level results by 3.45 and 4.40 percentage points on the Hand-Crafted and Algorithm-Generated subsets, respectively.

Introduction

Failure attribution must distinguish decisive errors from corrected deviations and downstream symptoms by tracing how errors propagate or persist in interaction loops. EMFA structures failed trajectories, screens candidate errors, and verifies decisive agent–step pairs counterfactually; on Who&When, it improves step-level attribution over prior best results.

Introduction

  • Failed trajectories can contain corrected deviations and downstream symptoms that inherit earlier errors, so decisive-error identification requires tracing error influence across execution.The paper focuses on cascading propagation across dependent steps and persistent interaction loops in which agents repeat actions without task progress.Source: Introduction · p. 2
  • EMFA structures execution logs into dependency representations, models cascading errors and persistent loops, screens candidate errors, and applies counterfactual verification to identify decisive agent–step pairs.Its error-propagation perspective aims to distinguish decisive errors from downstream failure symptoms.Source: Introduction · p. 2
  • EMFA improves the previous best step-level attribution results by 3.45 and 4.40 percentage points on the Hand-Crafted and Algorithm-Generated Who&When subsets, respectively.It achieves state-of-the-art step-level accuracy and remains competitive at the agent level.Source: Introduction · p. 2
  • The paper introduces an error-propagation-oriented perspective for failure attribution in LLM-based multi-agent systems.It explicitly considers errors propagating across dependent steps or persisting in unresolved interaction loops.Source: Introduction · p. 2

Related Work

Prior LLM-based multi-agent systems use manually specified workflows or automatically optimized interaction structures, while failure-attribution methods span four broad paradigms. EMFA is closest to structure-guided work but explicitly models error propagation before candidate selection and uses counterfactual verification.

Related Work

  • LLM-based multi-agent systems use manually specified workflows or automatically optimized interaction structures, but their dependencies can let errors propagate or remain unresolved in repeated interactions.The paper links these dependencies to difficulty identifying the decisive error.Source: LLM-Based Multi-Agent Systems · p. 2
  • Failure-attribution methods include direct inference, replay-based, fine-tuning-based, and structure-guided approaches, with EMFA most closely related to the last category.The paper contrasts EMFA with CHIEF and FALAT, which do not explicitly model complete error propagation before candidate selection.Source: Failure Attribution in LLM-Based… · p. 2
  • Direct LLM inference can struggle with fine-grained dependencies, replay-based methods incur substantial computational costs, and fine-tuning methods require additional supervision and training resources.The cited examples are FAMAS for replay and AgenTracer and GraphTracer for fine-tuning.Source: Failure Attribution in LLM-Based… · p. 2
  • Structure-guided methods include CHIEF, which ranks candidates using hierarchical causal graphs, and FALAT, which analyzes propagation after candidate pruning.The paper says this sequencing may leave propagation paths incomplete or confuse decisive errors with downstream symptoms.Source: Failure Attribution in LLM-Based… · p. 2

Problem Formulation

The formulation defines failure attribution as recovering a decisive agent–step pair from a failed turn-based trajectory. The observable final failure symptom may be downstream from that pair.

Problem Formulation

  • The system is modeled as turn-based, with exactly one agent acting at each execution step.The setting follows a formulation consistent with prior work to remain compatible with the Who&When benchmark.Source: Problem Formulation · p. 2
  • A trajectory is an ordered sequence of steps, each pairing an acting agent with its action, observation, reasoning, or result.The task outcome is represented as success or failure.Source: Problem Formulation · p. 2
  • The final failure symptom is where the unsuccessful outcome becomes observable, but it need not coincide with the decisive step.Examples include an incorrect final answer, premature termination, or an unresolved task requirement.Source: Problem Formulation · p. 3
  • The decisive pair is identified by correcting a step and allowing the subsequent execution to unfold, rather than simply locating the final failure.The attribution function recovers this pair from the task query and observed failed trajectory.Source: Problem Formulation · p. 3
  • The attribution target is the earliest decisive error: an agent–step pair whose correction would recover the failed execution.A failed trajectory may contain multiple pairs meeting the counterfactual criterion.Source: Problem Formulation · p. 3

Method

EMFA traces a failed multi-agent execution from its final symptom through dependencies and persistent loops, then screens and counterfactually verifies candidate errors to identify the decisive agent–step pair.

Method

  • EMFA identifies the decisive error by tracing the final failure symptom through five stages, ending with counterfactual verification of candidate agent–step pairs.The stages are final failure step identification, trajectory structuring, propagation modeling, dual-branch screening, and verification.Source: Method · p. 3

Final Failure Step Identification

  • The final failure step anchors backward dependency tracing but may be only a downstream manifestation, not the decisive error or propagation source.For explicit answers, EMFA scans the last five steps in reverse; if execution ends without an answer, it selects the last step.Source: Final Failure Step Identification · p. 3

Structured Trajectory Construction

  • EMFA represents the trajectory as semantic subtasks linked by step-level producer–consumer dependencies and cross-subtask reuse links.Deterministic post-processing adds an instruction–response dependency for an immediate addressed response, without treating temporal adjacency alone as evidence of dependency.Source: Structured Trajectory Construction · p. 3

Error Propagation Modeling

  • EMFA traces relevant information backward from the final failure step, then annotates changes chronologically and separately extracts unresolved repetitive interaction loops.Propagation annotations distinguish introduced, propagated, transformed, or corrected deviations; a propagation source need not be the decisive error.Source: Error Propagation Modeling · pp. 4–5

Dual-Branch Candidate Error Screening

  • Propagation screening prioritizes early unrecovered deviations, while loop screening selects the earliest step that initiates or sustains unresolved repetition.Each branch contributes at most two candidates, so full EMFA evaluates up to four.Source: Dual-Branch Candidate Error Screening · p. 5

Counterfactual Decisive Error Verification

  • For each candidate, model-based counterfactual assessment tests whether minimal repair prevents downstream failure and satisfies the query-derived success criteria.When multiple repairs could recover execution, responsibility-boundary principles distinguish introducing an error from later executing, propagating, or exposing it.Source: Counterfactual Decisive Error… · p. 5

Experiment Setup

The evaluation uses Who&When’s two subsets, compares ten representative approaches, and reports strict Top-1 agent- and step-level accuracy without task ground truth unless otherwise specified.

Dataset

  • Who&When’s test set includes 58 Hand-Crafted logs and 126 Algorithm-Generated logs, with expert-consensus annotations of failure steps, agents, and reasons.The logs derive from Magentic-One and CaptainAgent architectures, with tasks from GAIA and AssistantBench.Source: Dataset · p. 5

Baselines

  • Ten representative methods span random, LLM prompting, fine-tuning, and structure-guided attribution; FAMAS is excluded because it requires task ground truth.The comparison includes methods from Who&When and other cited work across these four paradigms.Source: Baselines · p. 5
  • The main-results table reports Who&When accuracy percentages and marks best and second-best results in each column.Parenthetical EMFA values indicate its gap to the second-best result when it ranks first, or to the best result otherwise.Source: Table 1 · p. 6

Evaluation Metrics

  • Evaluation uses strict Top-1 agent-level and step-level accuracy, averages EMFA over three runs, and generally excludes task ground truth.Published baseline values follow their original reporting protocols.Source: Evaluation Metrics · p. 6

Implementation Details

  • Experiments use GPT-4o by default at temperature 0 with a 4,096-token generation limit, and some baselines use stronger published results instead of reproductions.The implementation uses Python 3.11 on Ubuntu.Source: Implementation Details · p. 6

Results

EMFA leads step-level attribution on both Who&When subsets; further analyses examine trajectory length, backbone choice, and dual-branch screening.

Main Results

  • 3.45 and 4.40 percentage points: EMFA improves over CHIEF, the previous best, in step-level attribution on the Hand-Crafted and Algorithm-Generated subsets, respectively.Source: Main Results · p. 6
  • EMFA achieves the best agent-level accuracy on the Hand-Crafted subset and remains competitive on the Algorithm-Generated subset.Source: Main Results · p. 6

Further Analysis

  • GPT-4o achieves the best results on both subsets; its Algorithm-Generated step-level accuracy is 50.00%, versus 39.68–40.48% for open-weight models.Source: Table 2 · p. 7, Further Analysis · p. 7
  • Full EMFA outperforms direct LLM screening and either single branch; removing propagation particularly hurts step accuracy, while removing loops especially degrades Algorithm-Generated performance.Algorithm-Generated step-level accuracy falls from 50.00% to 33.33% without the Loop Branch.Source: Table 3 · p. 7, Further Analysis · p. 7

Main Results

Further Analysis

  • The full dual-branch design provides the strongest end-to-end attribution performance, with propagation and loop analysis offering complementary candidate coverage.Source: Further Analysis · p. 7

Conclusion and Limitations

EMFA models cascading error propagation and persistent interaction loops to identify decisive errors in LLM-based multi-agent systems. It achieves state-of-the-art step-level attribution on Who&When, but adds inference-time overhead.

Conclusion and Limitations

  • EMFA traces failure-related dependencies, models cascading propagation and persistent interaction loops, and applies counterfactual verification to identify decisive agent–step pairs.Source: Conclusion and Limitations · p. 7
  • Explicit trajectory structuring and propagation modeling introduce additional inference-time overhead.Source: Conclusion and Limitations · p. 7
Loading 2610.11600v1…