PaperlayerView exact revision on arXiv

Source-linked AI summary

Detect Before You Attribute: Cascade Failure Attribution for Multi-Agent Systems

Jiayi Zhang, Zexin Wang, Degang Sun, Changhua Pei, Fei Sun, Gaogang Xie, Jingjing Li

arXiv:2608.29646v1cs.AIcs.MA

TL;DR

LLM-based agents need precise failure attribution, but structural methods can miss semantics and LLM-based methods degrade on lengthy trajectories. DUOTRACE detects anomalous executions with a dual-view Tree-LSTM VAE and augmentation, then supplies focused evidence to downstream judges, improving agent- and step-level accuracy by 8.7% and 7.0%.

  • Problem

    Existing failure-attribution methods either overlook fine-grained semantics or suffer context dilution on lengthy trajectories, hindering precise diagnosis.

  • Method

    DUOTRACE is a plug-and-play detect-before-attribute filter using dual-view semantic-structural representations, a Tree-LSTM-based VAE, and prefix-chain- and LLM-based augmentation.

  • Results

    8.7% and 7.0% absolute improvements in agent-level and step-level attribution accuracy, respectively, were achieved over existing baselines.

  • Takeaways & Limitations

    DUOTRACE provides focused diagnostic evidence to downstream LLM judges while reducing context dilution, token consumption, and inference latency.

  • Takeaways & Limitations

    Failure attribution still relies on downstream LLM judges whose intrinsic societal or cognitive biases may skew diagnostic results in specific contexts.

Abstract

from arXiv · show

Large language model (LLM)-based agents have shown strong potential in solving complex tasks through multi-step reasoning, yet they remain vulnerable to execution failures. Accurate failure attribution is therefore critical for improving agent reliability. Existing topology- and spectrum-based methods exploit trajectory structures but often overlook fine-grained semantics, while LLM-based attribution methods capture semantic cues but suffer from long-context degradation over lengthy trajectories. To address these challenges, we propose DUOTRACE, a plug-and-play detection filter for LLM-based failure attribution. DUOTRACE follows a detect-before-attribute paradigm: it first detects anomalous executions and then supplies focused trajectory evidence to downstream LLM-based attribution methods. For effective VAE-based anomaly detection on agent trajectories, DUOTRACE integrates dual-view semantic-structural node representations, a Tree-LSTM-based trajectory encoder, and prefix-chain- and LLM-based data augmentation to handle heterogeneous nodes, hierarchical execution structures, and limited failure data. Experiments with six LLM-based attribution baselines show that DUOTRACE improves agent-level and step-level attribution accuracy by 8.7% and 7.0%, respectively.

1 Introduction

LLM-based agents face cascading execution failures that make precise failure attribution essential, while existing structural and LLM-based approaches each miss important information. DUOTRACE detects anomalous executions first and provides focused evidence to downstream judges, improving attribution accuracy.

  • Early execution errors can cascade through hierarchical branches, corrupt downstream decisions, derail objectives, and waste inference time.
  • Topology- and spectrum-driven methods capture trajectory structure but often overlook fine-grained semantic cues, limiting attribution of hallucination-related failures.
  • DUOTRACE adopts a detect-before-attribute paradigm that filters normal information before downstream LLM judges examine failure-relevant signals.
  • DUOTRACE fuses semantic embeddings with topological adjacency, uses a Tree-LSTM-based VAE, and augments normal executions with prefix chains and LLM simulations.
  • 8.7% and 7.0% absolute improvements in agent-level and step-level attribution accuracy, respectively, were achieved over existing baselines.

2 Related Work

Failure attribution in multi-agent systems has developed from structural statistical methods toward semantic LLM evaluators, but both approaches retain important trade-offs. Structural methods are efficient yet semantic-blind, whereas LLM-centric methods face costs and context dilution on lengthy logs.

  • Failure attribution identifies the exact error step and responsible agent, extending evaluation beyond overall task success or collaboration dynamics.
  • Recent benchmarks annotate failure trajectories, while fault-injection pipelines such as AgenTracer and Aegis synthesize diagnostic data.
  • Topology-driven methods such as FAMAS and CDC-MAS use structural dependencies or causal inference while remaining structurally lightweight and efficient.
  • Topology-driven approaches are semantic-blind because they rely exclusively on trajectory structure, limiting their access to semantic failure cues.
  • LLM-as-a-Judge methods use hierarchical contexts, error caches, or end-to-end fine-tuning to capture semantic nuances in failure attribution.
  • Running LLM-centric evaluators on lengthy multi-agent logs increases token costs and latency while causing context dilution known as Lost in the Middle.

3 Motivation and Problem Formulation

Multi-agent trajectories combine hierarchical dependencies with verbose semantic content, creating a need to select suspicious steps before final LLM-based attribution. DUOTRACE frames anomaly detection as candidate prioritization rather than causal attribution.

  • 3 Motivation and Problem Formulation: Multi-agent trajectories intertwine complex topological dependencies with verbose text, making raw-log evaluation vulnerable to context dilution.
  • 3 Motivation and Problem Formulation: Pure structural analysis may overlook semantic failure signals, motivating an anomaly-based candidate-selection stage before LLM attribution.
  • 3 Motivation and Problem Formulation: DUOTRACE learns regular execution patterns from normal trajectories and uses deviations to prioritize a compact set of suspicious execution steps.
  • 3 Motivation and Problem Formulation: Retained suspicious candidates are examined by downstream LLM methods for the final causal judgment.
  • 3.1 Problem Formulation: An execution trajectory is modeled as a directed hierarchical graph T = (V, E), with V representing its execution steps.
  • 3.1 Problem Formulation: Each node vi represents an agent behavior containing the acting agent identity, invoked operator or tool, and unstructured textual payload.
  • 3.1 Problem Formulation: Edges encode parent-child invocation dependencies when one execution step directly invokes another.
  • 3.1 Problem Formulation: For failed trajectories, DUOTRACE scores each step for anomaly, ranks the scores, and forwards a condensed sub-trajectory to LLM judges.

4 Methodology

DUOTRACE is a progressive, plug-and-play pipeline that detects anomalous execution patterns before downstream failure attribution. It combines augmentation, dual-view representations, a Tree-LSTM VAE, dynamic scoring, and compact sub-trajectory distillation.

  • Data Augmentation: DUOTRACE augments scarce normal-trajectory data through LLM replay generation and variable-length prefix-chain extraction.LLM replay uses stochastic task replays to synthesize fault-free executions, while prefix slicing expands the valid training space.
  • Dual-View Feature Representation: Dual-view node representations fuse role-aware semantic embeddings with adjacency-based topological context for structure-aware anomaly detection.Agent identities and operators are embedded with a Sentence Transformer, while the adjacency matrix records direct invocation relationships.
  • Lightweight Structural VAE: A Tree-LSTM encoder captures hierarchical dependencies while excluding current-node token consumption to prevent information leakage.The decoder models token consumption and operational context from functional role and topological position, then reconstructs dual-view inputs from the latent representation.
  • Lightweight Structural VAE: The VAE jointly checks semantic reconstruction and execution-trace prediction, using reconstruction errors to identify structural anomalies or hallucinations.The objective combines MSE, NLL, and KL divergence; token consumption is modeled with a conditional Gaussian distribution.
  • Dynamic Additive Point Scoring: Dynamic Z-score scoring tracks cumulative anomaly mutations and prioritizes early significant spikes as suspicious execution candidates.Successful traces establish the baseline mean and standard deviation, replacing unreliable static thresholds for long-horizon tasks.
  • Sub-Trajectory Distillation: DUOTRACE distills the execution tree into a top-K sub-trajectory that preserves adjacency dependencies and focuses downstream LLM evaluators on failure origins.Ranking nodes by Z-score constrains the evaluator’s search space while reducing context dilution.

5 Experiments

Across accuracy, efficiency, robustness, ranking, and generalization evaluations, DUOTRACE consistently improves downstream failure attribution while reducing retained-context costs. Ablations show that semantic-structural modeling, augmentation, hierarchical encoding, and failure-relevant filtering are important to these gains.

  • Accuracy and diagnostics: 69.1% overall Agent Accuracy and 49.0% overall Step Accuracy are achieved by CORRECT enhanced with DUOTRACE, outperforming AgenTracer.
  • Generalization: 19.0% and 15.5% absolute Handcraft Agent Accuracy gains are obtained for Qwen3.5-plus and GPT-5, while CORRECT gains 12.1% in Step Accuracy.
  • Data augmentation ablation: 8.62% and 5.56% performance drops follow removal of prefix-chain extraction on Handcraft and synthetic replays on Automated, respectively.The results associate temporal context and valid behavioral variants with stronger anomaly detection.
  • VAE component ablation: 20.69% and 18.97% Handcraft performance drops occur when semantic and structural views are removed, while removing Agent loss causes a 13.80% drop.Removing the KL penalty causes a further 6.90% degradation.
  • Candidate filtering: 72.4% Handcraft Agent Accuracy with DUOTRACE exceeds Random Filter, Rule-Based Matching, and Top-K Filtering at 43.1%, 36.2%, and 10.3%.The comparison indicates that preserving failure-relevant candidates matters beyond merely shortening trajectories.
  • Efficiency: 41% to 76% token reductions and a 2.5× CORRECT speedup demonstrate lower computational overhead and latency.CORRECT inference time falls from 81.4 to 32.3 minutes, while Qwen3.5-plus latency falls from 168.6 to 60.6 minutes.
  • Robustness: Under 20% semantic masking, average degradation falls from 5.9 percentage points without DUOTRACE to 2.6 points with it, assuming observable topology.

6 Conclusion

DUOTRACE is a plug-and-play detect-before-attribute filter that isolates anomalous executions and supplies focused evidence to downstream LLM evaluators. Its Tree-LSTM VAE combines dual-view representations and data augmentation, improving attribution accuracy while reducing inference overhead across six baselines.

  • DUOTRACE enhances LLM-based failure attribution through a detect-before-attribute paradigm.
  • A Tree-LSTM-based VAE integrates dual-view semantic-structural representations and data augmentation to capture hierarchical execution dependencies.
  • Across six baselines, the lightweight front-end consistently improves agent- and step-level attribution accuracy while drastically reducing inference overhead.

Limitations

DUOTRACE is presented as a lightweight, plug-and-play front end that detects anomalies before LLM-based failure attribution. It isolates focused evidence while improving both agent- and step-level attribution accuracy.

  • DUOTRACE uses a “detect-before-attribute” paradigm to isolate anomalous executions before downstream LLM attribution.It supplies downstream LLMs with focused evidence instead of lengthy trajectories.
  • The framework combines dual-view semantic-structural representations, a Tree-LSTM-based VAE, and counterfactual data augmentation.These components support anomaly detection over heterogeneous, hierarchical agent trajectories.
  • Across six baselines, DUOTRACE consistently improves agent- and step-level attribution accuracy.The method functions as a lightweight front-end for downstream attribution systems.

Ethics Statement

The paper frames accurate failure attribution as a tool for improving transparency, reliability, and safety, while acknowledging dual-use and computational concerns. It also notes that downstream LLM judges may introduce bias into diagnostic outcomes.

  • DUOTRACE aims to improve transparency, reliability, and safety in LLM-based multi-agent systems by identifying anomalous-execution root causes.The stated goal is to help developers build more robust AI agents.
  • The authors warn that precise vulnerability-isolation tools could be exploited by malicious actors.They advocate using such debugging tools only in secure, authorized development environments.
  • DUOTRACE reduces downstream inference overhead and token consumption but requires initial computational resources to train its Tree-LSTM-based VAE.The paper identifies this as a computational and environmental trade-off.
  • The failure-attribution process relies on downstream LLM judges whose intrinsic societal or cognitive biases may skew diagnostic results in specific contexts.The experiments use simulated environments and standard open-source logs without PII or offensive content.

A.1 Task Classification Statistics

The task distribution is dominated by Information Retrieval, while the failure taxonomy spans both structural and semantic error domains. These statistics are used to motivate DUOTRACE’s dual-view design.

  • Information Retrieval accounts for 72.3% of the task distribution, whereas complex reasoning tasks account for 27.7%.The complex reasoning examples include Math & Logic tasks.
  • The dataset contains 184 failure instances classified into nine categories.The categories cover both structural and semantic failure domains.
  • Structural errors include Code Error, while semantic errors include Hallucination.The reported distribution empirically supports modeling both types of failure.

B Implementation Details

The implementation uses Tree-LSTM-based VAE modeling, semantic-structural representations, controlled counterfactual augmentation, and top-20% anomaly-based trajectory distillation. Training and evaluation rely on specified architectural, optimization, and reproducibility settings.

  • Model Architecture: DUOTRACE’s semantic view encodes agent identities and operational semantics into 384-dimensional embeddings, concatenates them into 768 dimensions, and projects them to 128.The Child-Sum Tree-LSTM uses a hidden dimension of 128, while the VAE latent space has 64 dimensions.
  • Model Architecture: The decoder reconstructs 768-dimensional semantic embeddings and predicts conditional Gaussian parameters for the latent representation.The decoder uses two ReLU multi-layer perceptrons.
  • Training Hyperparameters: Training uses Adam for 50 epochs with learning rate 2 × 10^-3 and KL-divergence weight β = 0.01.Semantic reconstruction uses MSE, while token consumption is optimized with NLL.
  • Inference and Evaluation Setup: Node anomaly scores sum semantic-vector MSE and token-prediction NLL, and sub-trajectory distillation retains the top 20% of nodes.The candidate set size is fixed this way for both Handcraft and Automated datasets.
  • Data Augmentation: Counterfactual augmentation generates structurally aligned positive samples using constrained agent vocabulary and Human-originated interactions.The prompt template is documented in Table 13.
  • Data Augmentation: Augmentation is restricted to training data, excludes diagnostic ground truth, and produces over 900 verified successful trajectories through five executions per task.Generated final states are programmatically checked against task ground truth, while KL regularization mitigates synthetic artifacts.

C Counterfactual Alignment Analysis

The analysis aligns generated successful trajectories with original failed logs using DTW and evaluates semantic similarity with BERTScore. It assesses augmentation through prefix preservation, failure-node divergence, and post-correction diversity.

  • DTW aligns generated successful trajectories with original failed logs to validate the data augmentation strategy.
  • BERTScore measures the semantic similarity trend between aligned generated and original trajectories.Figure 5 illustrates this trend.
  • Prefix Consistency (X < 0) confirms preservation of operational preconditions before the failure node.
  • Divergence at the Failure Node (X = 0) validates the surgical precision of the correction.
  • Post-Correction Diversity (X > 0) ensures the VAE is exposed to diverse execution variations.
Loading 2608.29646v1…