Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels

1. Introduction

• Failure attribution seeks the decisive step in a failed trajectory, a task that can require tracing dozens or hundreds of steps and costly, inconsistent manual review.The problem matters for debugging, auditing, and improving agents.Source: §1 · p. 1
• Existing approaches rely on costly or potentially unreliable LLM judges, expensive step-level annotations, or verified success-only trajectories that may become stale.OAT [22] learns successful behavior but does not use failure trajectories during training.Source: §1 · pp. 1–2
• The paper studies failed trajectories whose task outcome is known but whose decisive step is unmarked, avoiding the need for decisive-error annotations or verified successful trajectories.The trajectories contain an unlabeled mixture of ordinary steps and decisive errors.Source: §1 · pp. 2–3
• Downstream contamination occurs when later steps inherit an early error and show stronger failure signals, leading independent scorers to select a consequence instead of the root error.A wrong tool argument can trigger visible errors and retries that overshadow the initial action.Source: §1 · p. 2
• SOAP scores steps against a spectral reference space, then uses attention-derived dependencies to propagate downstream error evidence toward likely upstream sources.This targets contamination without step-level training labels or a separate success-only reference set.Source: §1 · pp. 2–3
• SOAP consistently improves decisive-step localization over competitive baselines; on TRACEELEPHANT with DeepSeek-R1-Distill-Llama-8B, it improves step-level Accuracy by 24.66% over the strongest baseline.Source: §1 · p. 3

2. Problem Setup

• The target is the earliest step at which the error responsible for eventual task failure is introduced, with the responsible agent identified by that step.Source: §2 · p. 3
• A frozen language model supplies internal signals, and may be the agent model itself or a separate open-weight model when agent states are inaccessible.The model is not fine-tuned on failure-attribution data.Source: §2 · p. 3
• The task is to estimate a decisive-error index from a task description and failed trajectory; scalar step scores yield a top-ranked step and a ranked list for triage.The paper evaluates top-1 localization and top-k performance.Source: §2 · p. 3
• SOAP assumes a disjoint reference collection of failed trajectories with known task-level outcomes but no step or responsible-agent labels.Gold step labels are reserved for evaluation and, when applicable, hyperparameter selection on a disjoint validation split.Source: §2 · pp. 3–4
• The reference steps are modeled as a mixture of ordinary and failure-related steps, with an unknown contamination proportion; failure-related steps include the decisive error and its downstream effects.Within each trajectory, the earliest failure-related step is the decisive error.Source: §2 · p. 3

3. SOAP: Spectral Scoring with Attention-Guided Propagation

• SOAP combines a spectral base score for each step with attention-derived dependencies that propagate downstream error evidence toward likely upstream sources.The resulting score estimates how likely a step is to have introduced the failure.Source: §3 · p. 4

3.1. Spectral Step Scoring from Unlabeled Failure Trajectories

• Because ordinary steps dominate failure-trajectory representations while failure-related steps are sparse, SOAP scores steps using inverse projection energy in a selected spectral band.Decisive-error steps tend to show lower projection energy, which yields larger base scores.Source: §3.1 · pp. 4–5
• The selected spectral band avoids relying only on leading directions that may encode frequent shared patterns or tail directions that may capture irrelevant variation.Source: §3.1 · p. 5

3.2. Dependency-aware Rescoring

• SOAP builds soft dependencies by aggregating attention from each step to eligible retained predecessors and normalizing the resulting masses into relative dependency weights.These weights come from the frozen LLM, without repeated generative prompting, dependency reconstruction through LLM-based graph reasoning, or task-specific post-training.Source: §3.2 · pp. 5–6
• Dependency-aware rescoring adds dependency-weighted downstream base scores to a step's local error signal, promoting earlier steps linked to anomalous downstream steps.The normalization prevents a step from receiving a large correction merely because it has many downstream dependents.Source: §3.2 · p. 6

4.1. Setup

• The evaluation uses five subsets across three benchmarks, with trajectory-level splits; reference annotations are hidden and used only to construct the spectral matrix.Validation labels select hyperparameters, while test labels are reserved for final evaluation; compared baselines include prompt-based judges and training-based attributors.Source: §4.1 · pp. 6–7

4.2. Main Results

• 47.62% vs. 39.68% for ErrorProbe and 16.72% for OAT [22] on WW-AG with Qwen3.5-9B; SOAP leads every subset with both open backbones.No method receives the gold task answer at inference.Source: Table 1 · p. 7, §4.2 · p. 8
• 44.97 vs. 22.65 for StepFinder [29] on WW-AG with larger backbones; cross-subset transfer and synthetic references also remain effective.Constructing the reference on WW-HC and testing on TE-MAG gives 21.01 versus 23.19 in-domain; GPT-4o-generated references give 46.56 on WW-AG versus 47.62 with original training trajectories.Source: Figure 4 · p. 8, §4.2 · pp. 8–9
• 43.39 vs. 40.74 on WW-AG and 34.48 vs. 19.54 on WW-HC; SOAP remains ahead of ErrorProbe when all methods receive the gold answer.The results use Qwen3.5-9B.Source: Table 2 · p. 8, §4.2 · p. 8

4.3. Analysis

• With twelve WW-AG and six WW-HC reference trajectories, accuracy is within a few points of full-data results: 43.39 vs. 47.62 on WW-AG.The reference is constructed from one third of the 30% training split.Source: Figure 5 · p. 9, §4.3 · p. 10
• 39.15 on WW-AG for the spectral band versus 18.52 for the best alternative; the selected spectral score outperforms seven alternative scoring functions.Alternatives include perplexity, other projection subspaces, and embedding norms.Source: Table 3 · p. 9, §4.3 · p. 9
• Attention-guided rescoring reaches 47.62 on WW-AG versus 40.74 for the strongest alternative, while propagation peaks at γ=0.6 and the earliest attention band performs best.All tested attention bands improve over the base score; later representation layers perform best.Source: Table 4 · p. 9, Figure 5 · p. 9, §4.3 · p. 10
• In the TE-CAP example, rescoring moves the prediction from the final Orchestrator turn to the decisive error four steps earlier.The base score peaks on a turn restating the wrong answer; the earlier error is an extraction expert's misreading of program structure and proposal of the wrong fix.Source: Figure 6 · p. 10, §4.3 · p. 10

• Prompt-based methods use multi-stage judging, retrieval, context restructuring, or dependency reconstruction; examples include RAFFLES [27], ErrorProbe, and CHIEF [20].Other approaches include schema retrieval and reconstructed step dependencies [1] [3] [16] [23].Source: §5 · p. 10
• AgenTracer [24] and StepFinder [29] train on labeled or synthetically corrupted trajectories, while GraphTracer traces root causes over information-flow graphs.These approaches rely on computationally intensive inference or step-level annotations; OAT [22] instead learns from clean successful trajectories.Source: §5 · p. 11
• Unlike OAT [22], SOAP uses failure-conditioned trajectories; unlike Du et al. [8], it adds dependency propagation to projection-based scoring.Liu et al. [13] also use attention signals, but score high-negative-log-likelihood steps for two-stage localization.Source: §5 · p. 11

• SOAP scores each step against a spectral reference constructed on unlabeled failure trajectories.Source: §6 · p. 11
• Attention-guided propagation routes late contamination back toward the decisive error; SOAP requires no step-level annotation and no separate successful-trajectory reference set.The framework can be applied to off-the-shelf LLMs.Source: §6 · p. 11
• Experiments show SOAP consistently outperforms prompt-based and training-based baselines across benchmarks, backbones, and settings.Source: §6 · p. 11

A.1. Datasets and statistics

• Three benchmarks provide five subsets: WHO&WHEN’s WW-AG and WW-HC, CORRECT-ERROR’s seven-pool CE mean, and TRACEELEPHANT’s TE-CAP and TE-MAG.WW-HC averages 52 turns and reaches 130, making exact localization especially difficult there.Source: §A.1 · p. 14

A.2. Input format and context truncation

• Each scored-step prompt contains the task description and steps through the target; at the 8,192-token limit, oldest steps are dropped while the task and target remain.The input uses one user turn, no system prompt, and step text formatted with agent name and step number.Source: §A.2 · pp. 14–15

A.3. Implementation details of SOAP

• SOAP mean-pools each step’s token states, selects a representation layer, and scores steps by projection onto a selected spectral band.Attention mass is averaged across heads and selected layers to estimate dependencies; rescoring retains top-mass predecessors within the context and uses validation accuracy for two-stage hyperparameter selection.Source: §A.3 · pp. 15–16
• The scoring alternatives compare perplexity, spectral and other projection subspaces, and vector norms, while rescoring alternatives include attention-free corrections and position heuristics.Temporal bias penalizes later positions; the earliest-of-top-5 heuristic selects the earliest among the five highest base scores.Source: §A.3 · p. 16

A.4. Baselines

• Prompt-based baselines use released or reproduced implementations, while OAT models successful-trajectory dynamics and StepFinder is supervised on step-labeled failures.OAT uses one-class training on successful MCP-ATLAS trajectories; StepFinder uses a BiLSTM, agent-aware attention, and a scoring head.Source: §A.4 · pp. 16–17

A.5. With-ground-truth setting

• In with-ground-truth evaluation, prompt-based judges receive the answer in their prompt, while SOAP’s performance changes by under five points almost everywhere.SOAP’s only larger reported movement is +6.5 on DeepSeek-8B TE-MAG.Source: §A.5 · p. 17
• SOAP leads every column in both open-backbone blocks with ground-truth answers, tying Step-by-Step only on Qwen3.5-9B TE-MAG.On CE with Qwen3.5-9B, the closest judges reach 59.89 and 59.57 against SOAP’s 60.59.Source: §A.5 · p. 17

A.6. Compute resources and cost

• SOAP’s SVD fit takes 0.08 s on WW-AG and 0.29 s on WW-HC, versus 51–115 s per training seed for OAT and StepFinder.One forward pass takes 0.38 s and 0.76 s, respectively; stored-vector scoring takes 0.7 and 5.5 ms per trajectory.Source: §A.6 · p. 17, Table 7 · p. 18
• SOAP takes 0.38 s on WW-AG and 0.76 s on WW-HC per trajectory, faster than every judge except Binary Search at 0.2 and 0.4 s.CHIEF takes 24 s and ErrorProbe 6 s per trajectory.Source: §A.6 · pp. 17–18, Table 8 · p. 19

A.7. Synthetic reference trajectories

• Synthetic references rerun CaptainAgent and Magentic-One with Qwen3.5-9B and GPT-4o on the benchmark question pools, using unlabeled mixes of successes and failures.The runs share question pools with the benchmarks but not trajectories; the GPT-4o corpus also differs in model version and rebuilt generation pipeline.Source: §A.7 · pp. 18–19

B.1. Context window of the dependency weights

• On Qwen3.5-9B WW-AG, accuracy is 47.62 at w=1–2 versus 40.21 at w=all, while TE-CAP favors wider windows.The preferred window varies by dataset and backbone; a poor choice can fall below the base score, as Qwen3.5-9B TE-MAG at w=1 does (15.94 vs. 21.01).Source: §B.1 · p. 19, Table 9 · p. 20

B.2. Sensitivity to the hyperparameters on all cells

• DeepSeek-8B WW-AG accuracy rises from 38.62 at γ=0 to 45.50 at γ=1.0, while the other cells peak at γ∈{0.1, 0.2}.The selected layer is the per-layer post-rescoring optimum in seven of eight cells, and the configurations lie on broad rather than isolated optima.Source: §B.2 · pp. 19–20

B.3. Pooling of the step representation

• WW-AG base-score accuracy with mean pooling is 39.15 versus 29.10 with last-token pooling on Qwen3.5-9B.Mean pooling also scores 38.62 versus 29.10 on DeepSeek-8B WW-AG and 40.31 versus 24.81 on DeepSeek-8B TE-CAP; rescoring helps last-token representations but starts from a lower base.Source: §B.3 · p. 20, Table 10 · p. 20

B.4. Quantity of unlabeled reference data

• 6–12 unlabeled reference trajectories keep every cell within about four points of its full-data accuracy, with most within one or two.The configuration is re-selected at each data fraction on the validation split.Source: §B.4 · p. 20
• DeepSeek-8B WW-HC and TE-MAG can score above their full-data results at 10% reference data because hyperparameters are re-selected on tiny fits.The authors characterize these cases as selection noise, not a real gain.Source: §B.4 · p. 20
• The reported data-fraction results support the conclusion that a handful of unlabeled trajectories suffices to construct the spectral reference.Source: §B.4 · p. 20

B.5. Length and norm baselines

• SOAP achieves the best accuracy on every cell, including 47.62% versus 35.45% for the best unit-norm score on Qwen3.5-9B WW-AG.The longest-step baseline scores only 8–20%, and the shortest-step baseline falls below random on every cell.Source: §B.5 · p. 21, Table 12 · p. 23

B.6. Effect of the reference corpus

• On WW-AG, the benchmark reference beats every alternative by up to 14.29 points with Qwen3.5-9B and 8.99 points with DeepSeek-8B.MCP-ATLAS failures outperform successes on WW-AG with both backbones; the steps-before-t⋆ reference nearly matches the full reference with DeepSeek-8B.Source: §B.6 · pp. 23–24, Table 13 · p. 24

B.7. Comparison with a supervised probe

• SOAP outperforms the supervised probe on every subset, by 3.70–17.06 points with Qwen3.5-9B and 9.20–21.71 points with DeepSeek-8B.The probe comes closest on Qwen3.5-9B WW-AG, while the largest gap is on TE-CAP with both backbones.Source: §B.7 · p. 24, Table 14 · p. 24

C.1. Agent-level accuracy and variance across seeds

• SOAP has the highest agent-level accuracy on both subsets with both open-weight backbones, while agent-level scoring can credit the right agent despite a wrong step.All-at-Once with GPT-4o reaches 59.77 on WW-HC while locating the step in 6.90% of cases.Source: §C.1 · p. 25, Table 15 · p. 25
• On small subsets, standard deviations of 2–5 points are typical, so differences below that on a single subset should not be read as meaningful.SOAP’s Qwen3.5-9B WW-AG margin over ErrorProbe is about three standard deviations; CORRECT-ERROR is stable to under one point.Source: §C.1 · p. 25, Table 16 · p. 26

C.2. Accuracy at 𝑘

• At k=3, SOAP finds the decisive step in 74.60% of Qwen3.5-9B WW-AG trajectories, versus 56.72% for OAT and 55.24% for StepFinder.At k=5, SOAP reaches 91.01% on WW-AG; its WW-HC gap over OAT is 12 points.Source: §C.2 · p. 26

C.3. Results with GPT-5 as the judge

• SOAP remains ahead of every GPT-5 judge on WW-HC, TE-CAP, and DeepSeek-8B TE-MAG without the answer, while GPT-5 ErrorProbe overtakes it on WW-AG.On WW-AG, ErrorProbe scores 49.21 versus SOAP’s 47.62 without the answer, and 53.44 versus 43.39 with it.Source: §C.3 · pp. 26–27, Table 18 · p. 27

C.4. Larger backbones

• SOAP’s Qwen3.5 accuracy is stable from 9B to 27B: 47.62 versus 44.97 on WW-AG and 34.48 at both sizes on WW-HC.Neither OAT nor StepFinder scales with model size; the Qwen3-14B comparison also crosses model generations.Source: §C.4 · pp. 27–28, Table 19 · p. 28

C.5. Cross-distribution transfer

• The in-distribution reference wins every target column on both backbones, while off-diagonal transfer degrades gradually rather than catastrophically.For DeepSeek-8B, WW-HC→WW-AG retains 39.68 of the in-distribution 45.50, and WW-HC→TE-CAP retains 35.66 of 42.64.Source: §C.5 · p. 28, Figure 9 · p. 28

C.6. Synthetic references, full results

• On WW-AG, the benchmark reference remains strongest at 45.50, while both synthetic corpora are within 6–7 points; neither generator dominates across subsets.On WW-HC, the Qwen3.5-9B-generated corpus slightly exceeds the existing reference, 29.89 versus 28.74, and all synthetic rows exceed training-based baselines.Source: §C.6 · pp. 28–29, Table 20 · p. 29

C.7. Qualitative examples

• Across four trajectories, the base score peaks at a late symptom, while SOAP’s attention-guided rescoring lifts the likely cause.Examples include shifting from the TypeError traceback at S2 to the buggy call at S1, and from a late search dump at S73 to the initiating FedEx mistake at S8.Source: §C.7 · p. 29