PaperlayerView exact revision on arXiv

Source-linked AI summary

DeFA: Dependency-Guided Failure Attribution for LLM Agents

Bo Deng, Xinlei Zheng, Yi Wei, Kang Zhou, Chongyang Tao, Renzhao Liang, Xuanren Chen, Lifan Guo, Chi Zhang

arXiv:2610.01256v1cs.AI

TL;DR

LLM-agent failures can become visible many steps after the decision that caused them, making decisive-error attribution depend on tracing step content and dependencies. DeFA connects those dependencies and follows suspicious events through their effects to identify the decisive error and responsible agent. It reports strong attribution across text and multimodal evaluations, and its diagnostic feedback improves downstream skill evolution.

  • Problem

    Agent failures can have visible consequences many steps after the responsible decision, so attribution requires tracing the error and its effects through the trajectory.

    Source: §1 · p. 1

  • Method

    DeFA combines protocol and semantic dependencies into an event dependency graph, traces suspicious events’ sources and effects in a failure propagation graph, and uses connected evidence to identify the decisive error and responsible agent.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3

  • Results

    DeFA leads in responsible-agent and exact-step accuracy across three backbones on Who&When and Who&When Pro text, with reported gains on multimodal trajectories and 6–15-point Trace2Skill test-accuracy improvements over the native pipeline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DeFA’s diagnostic feedback can guide skill improvements for subsequent tasks in the evaluated Trace2Skill skill-evolution setting.

    Source: §5 · p. 9

    Relation recovery and propagation analysis depend on the model’s trajectory interpretation, and downstream utility has been evaluated for spreadsheet skill evolution; other domains and forms of agent improvement remain to be assessed.

    Source: §5 · p. 9

Abstract

from arXiv · show

Errors in LLM agent executions and their visible consequences can be separated by many steps, making decisive-error localization a matter of understanding both step content and step dependencies. We introduce DeFA, a dependency-guided framework for agent failure attribution. DeFA first combines protocol relations and semantic dependencies into an event dependency graph spanning the trajectory. It then identifies events that may violate task requirements and traces their sources and subsequent effects to construct a failure propagation graph. Finally, DeFA uses step evidence and the steps' roles in failure propagation to identify the decisive error, responsible agent, and error category. To support long trajectories, DeFA partitions executions into segments and combines the current segment's detailed content with summaries of the other segments, giving local diagnosis access to global execution context. Across Who and When and the Who and When Pro text subset, DeFA achieves the highest responsible-agent and exact step accuracy with all evaluated backbones, and the highest failure-mode accuracy among taxonomy-aligned methods on Pro. Further experiments on image and video trajectories demonstrate its applicability to multimodal failure attribution. Ablations support the contributions of segmentation, the event dependency graph, and the failure propagation graph. Using DeFA's diagnostic feedback for skill evolution in Trace2Skill improves downstream task accuracy by 6-15 percentage points over the native pipeline, showing that the diagnoses can also support agent improvement on subsequent tasks.

1 INTRODUCTION

DeFA attributes agent failures by connecting suspicious steps to their upstream sources and downstream effects, then identifying the decisive error and responsible agent. Across text and multimodal evaluations, it reports more precise attribution and shows that diagnostic feedback can improve subsequent skill evolution.

Motivation

  • The decisive error may occur well before its visible consequence, so attribution must trace how an error propagates through the execution.A misread retrieved value can be used in a later calculation and repeated in the final answer; relevant evidence may also be distributed across a long trace.Source: §1 · p. 1
  • Existing approaches include global judgments, sequential checks, binary search, constraint validation, multi-agent investigation, and hierarchical diagnosis.DeFA focuses on connecting suspicious events with the execution that made them consequential, including upstream sources and later actions that adopted, amplified, or corrected effects.Source: §1 · p. 1

Contribution

  • DeFA combines protocol and semantic dependencies in an event dependency graph, traces suspicious events’ sources and effects in a failure propagation graph, and uses that evidence to identify the decisive error, responsible agent, and error category.For long trajectories, segmented analysis combines detailed content from the current segment with summaries of the others.Source: §1 · p. 2

Results

  • DeFA leads in responsible-agent and exact-step accuracy across three backbones on Who&When and Who&When Pro text, with further gains on multimodal trajectories and downstream skill evolution.The introduction reports a 7.6–9.8 percentage-point exact-step improvement on Who&When and 6–15 points in Trace2Skill test accuracy over the native pipeline.Source: §1 · p. 2

Contribution

  • The introduction presents DeFA’s framework and propagation graph as the basis for tracing how errors are introduced, transmitted, and adopted to support decisive-error attribution.It also reports gains in precise attribution across text and multimodal evaluations and improved downstream skill evolution using diagnostic feedback.Source: §1 · p. 2

2 RELATED WORK

Related work spans trajectory evaluation, failure attribution, and the use of diagnostic feedback for agent improvement. DeFA evaluates its feedback within a skill-evolution process.

Execution trajectory evaluation and analysis

  • Trajectory evaluation covers response judging, intermediate-work and competency assessment, and reasoning correctness and error localization.Other work characterizes coordination failures (Cemri et al., 2025), tool-use risk, and coding-agent failure development and recovery (Zhao et al., 2026).Source: §2 · p. 2

Failure diagnosis and attribution frameworks

Diagnostic feedback for agent improvement

  • Agent-improvement methods carry feedback into later outputs or trials, while Trace2Skill consolidates trajectory lessons into reusable skills; DeFA’s feedback is evaluated within this skill-evolution process.The passage cites Self-Refine, Reflexion, and spreadsheet-task applications of Trace2Skill.Source: §2 · p. 3

3 METHOD

DeFA builds connected evidence across an agent trajectory and uses it to identify the decisive error. Its pipeline combines dependency-guided candidate recall, propagation analysis, and adjudication.

3 METHOD

  • DeFA constructs an event dependency graph to guide candidate recall, traces candidate sources and effects in a failure propagation graph, then adjudicates the decisive error using the resulting steps and evidence.Figure 1 summarizes the pipeline.Source: §3 · p. 3, Figure 1 · p. 3

3.1 PROBLEM FORMULATION

  • For a failed trajectory, DeFA predicts the decisive error step as the earliest erroneous step whose correction would make execution succeed, and assigns its responsible agent and error category.The category follows the Who&When Pro taxonomy, including hallucination and tool parameter or invocation error.Source: §3.1 · p. 3

3.2 EVENT DEPENDENCY GRAPH CONSTRUCTION

  • The event dependency graph combines protocol relations and semantic dependencies to represent how steps pass and use instructions, results, and state.Protocol relations capture framework-defined interactions such as delegation and execution or tool requests and results; semantic dependencies capture later use of earlier content, such as a retrieved value used in a calculation. Segmentation preserves related interactions where possible and pairs current-segment detail with summaries of other segments, retaining broader execution context and original step indices.Source: §3.2 · pp. 3–4

3.3 CANDIDATE EVENT RECALL

  • For each segment, candidate recall examines steps using task requirements, the analysis view, and relevant graph relations, returning candidate records with original step indices and evidence-based reasons for possible failure.Records can identify, for example, information inconsistent with task constraints or an omitted necessary operation.Source: §3.3 · p. 4

3.4 FAILURE PROPAGATION ANALYSIS

  • Failure propagation analysis traces candidate sources and how later execution continued, amplified, or corrected their effects, then adds steps found through propagation to the attribution graph.It selects relevant dependency edges and infers additional propagation edges from trajectory evidence; the merged graph includes candidates, exact edge endpoints, and cross-segment source steps.Source: §3.4 · p. 4

3.5 ROOT CAUSE ATTRIBUTION

  • An adjudicator uses the task, propagation graph, and trajectory evidence to compare plausible errors, assess whether a correction would turn failure into success, and select and classify the decisive step.The prediction includes the step, its error category, and the actor recorded at that step. In an example, connected steps 12, 16, and 19 lead DeFA to attribute the failure to step 16, where the assistant omits a retrieved film from its comparison.Source: §3.5 · p. 5

4 EXPERIMENTS

The experiments evaluate DeFA against seven baselines on text and multimodal failure-attribution benchmarks, then test its diagnostic feedback in skill evolution. DeFA leads on responsible-agent and exact-step accuracy across the text benchmarks and evaluated backbones, and its feedback improves downstream test accuracy over native Trace2Skill.

4.1 EXPERIMENTAL SETUP

  • The text evaluation uses 184 Who&When trajectories and 300 Pro text trajectories, comparing seven baselines across three backbones on Agent, Step, and Mode metrics.Mode is evaluated on Pro text; the other methods are excluded from that comparison because they do not use Pro’s failure-mode taxonomy.Source: §4.1 · p. 5

4.2 MAIN RESULTS ON WHO&WHEN

  • DeFA achieves the highest Agent and exact Step accuracy with all three backbones on Who&When, with exact Step gains of 7.6–9.8 percentage points.It reaches 52.7% exact Step accuracy with GLM-5.2.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 MAIN RESULTS ON WHO&WHEN PRO

  • On Pro text, DeFA leads in Agent and exact Step accuracy across all three backbones, and leads taxonomy-aligned methods in Mode accuracy.Exact Step gains are 4.0–8.3 percentage points; Mode reaches 34.0–41.0%, improving over the strongest comparable baseline by 6.7–13.0 percentage points.Source: Table 2 · p. 6, §4.3 · p. 7

4.4 TRAJECTORY LENGTH AND LLM CALLS

  • Exact Step accuracy remains challenging on the longest Who&When trajectories, where DeFA reaches 17.2% for the 33–130-step interval.DeFA uses 8.73 calls on average for 52.7% exact Step accuracy on Who&When and 80.3% on Pro; its largest length-specific gains occur at intermediate lengths.Source: Figure 2 · p. 7, §4.4 · p. 7

4.5 EXTENSION TO MULTIMODAL TRAJECTORIES

  • On 100 multimodal trajectories, DeFA leads Agent and exact Step with both backbones and Mode among taxonomy-aligned methods.Exact Step reaches 80.0% with Qwen3.8-Max and 78.0% with Kimi-K3, exceeding CockpitHAT by 7.0 and 6.0 percentage points, respectively.Source: Table 3 · p. 8, §4.5 · p. 8

4.6 COMPONENT ABLATIONS

  • Removing segmentation lowers Step from 52.7% to 44.6%, while removing the event dependency graph lowers Coverage to 77.7% and Step to 42.4%.Removing the failure propagation graph lowers Step to 40.8%; removing only propagation edges preserves 83.2% Coverage but lowers Step to 40.2%.Source: Table 4 · p. 8, §4.6 · p. 8

4.7 DOWNSTREAM UTILITY: SKILL IMPROVEMENT

  • DeFA’s diagnostic feedback improves test accuracy over native Trace2Skill by 6–15 percentage points across four settings.The average gain is 9.0 points; with Qwen3.5-Plus and a model-generated initial skill, accuracy rises from 63% to 78%.Source: Figure 3 · p. 9, §4.7 · p. 9

5 CONCLUSION AND LIMITATIONS

DeFA traces agent failures through step dependencies, using connected evidence to identify decisive errors and responsible agents. Experiments support attribution gains and downstream skill-evolution utility, while the authors note limits in relation recovery and the domains evaluated.

5 CONCLUSION AND LIMITATIONS

  • DeFA combines protocol relations and semantic dependencies into an event dependency graph, then traces suspicious events’ sources and effects to identify decisive errors and responsible agents.Final attribution compares how steps introduce, transmit, or adopt errors.Source: §5 · p. 9
  • 9.0 percentage points: DeFA’s diagnostic feedback improved test accuracy after Trace2Skill skill evolution over the native pipeline; attribution gains span text and multimodal inputs, backbones, and trajectory lengths.Ablations support segmentation, the event dependency graph, and the failure propagation graph.Source: §5 · p. 9
  • Relation recovery and propagation analysis depend on the model’s trajectory interpretation, so missing or misidentified relations can affect attribution, especially in long executions with dispersed evidence.The authors identify improving relation identification in long trajectories as future work.Source: §5 · p. 9
  • Downstream utility has been evaluated for spreadsheet skill evolution; effectiveness in other domains and forms of agent improvement remains to be assessed.Source: §5 · p. 9

Related papers

Loading 2610.01256v1…