PaperlayerView exact revision on arXiv

Source-linked AI summary

ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems

Weilin Jin, Mingyu Wang, Taiyu Zhu, Ziqi Zhou, Wenbo Li, Haoyang Huang, Nan Duan, Yifan Wu, Ying Li, Zhonghai Wu

arXiv:2610.11334v1cs.AI

TL;DR

Failures in LLM-based multi-agent systems can propagate through interactions, while evaluated step representations show limited separation between root-cause and other steps. ReCast learns attribution-oriented representations from frozen-LLM hidden states using layer selection, complementary features, and contextual learning. It achieves the best Hit@1 across four benchmarks, with gains of 5.65 and 9.19 percentage points on Who&When Algorithm and Handcrafted.

  • Problem

    Evaluated step representations show limited separation between root-cause steps and other steps, while multi-agent failures can propagate through interactions.

    Source: §1 · p. 1, §7 · p. 9

  • Method

    ReCast selects attribution-relevant layers, builds complementary pattern and deviation features, then uses contrastive and ranking objectives to contextualize frozen-LLM hidden states.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    ReCast achieves the best Hit@1 across four benchmarks, exceeding the strongest baseline by 5.65 percentage points on Who&When Algorithm and 9.19 percentage points on Handcrafted.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Representation analysis shows improved within-trace and cross-trace separation, highlighting the value of step representations tailored to failure attribution.

    Source: §7 · p. 10

    AgenTracer-8B is not publicly available, so multi-seed results could not be reported.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

In LLM-based agent systems, failures can originate from early steps whose effects propagate through subsequent interactions, making their origins difficult to identify. To trace such failures back to their origin, failure attribution has been formulated as the task of identifying the earliest step responsible for the failure. Recent methods leverage LLM internal signals for failure attribution, typically using hidden states as step representations. We therefore conduct an empirical study to evaluate how effectively these representations distinguish root-cause steps from other steps and find limited separation. Motivated by this observation, we propose ReCast, a step representation learning method that transforms hidden states from a frozen LLM into attribution-oriented step representations. ReCast first selects attribution-relevant layers, then constructs complementary pattern and deviation features, and finally learns contextualized step representations through an encoder trained with contrastive and ranking objectives. We also introduce ReCast-2K, a training dataset for failure attribution. ReCast achieves the best Hit@1 across four benchmarks, surpassing the strongest baseline by 5.65 and 9.19 pp on Who&When Algorithm and Handcrafted, respectively. Code is available at https://anonymous.4open.science/r/ReCast-5FB6 .

1 INTRODUCTION

Multi-agent failures can emerge from early errors whose effects surface only after downstream steps, making root-cause attribution important for debugging and reliability. Because existing step representations show limited separation, the paper proposes ReCast and evaluates it alongside a new training dataset.

1 INTRODUCTION

  • Multi-agent failures can propagate from early errors, so attribution must identify the earliest decisive error rather than only detect failure or locate its final symptom.Accurate step-level attribution is described as essential for debugging agent behaviors, improving orchestration strategies, and constructing more reliable systems.Source: §1 · p. 1
  • An empirical study finds limited separation between root-cause steps and other steps in text embeddings and LLM hidden-state representations.This comparison motivates learning attribution-oriented step representations.Source: §1 · p. 2, Figure 1 · p. 2
  • ReCast selects attribution-relevant internal features, constructs complementary pattern and deviation representations, and learns contextualized step representations with contrastive and ranking objectives.Its three stages address which signals to retain, how to represent them, and how to learn them for attribution.Source: §1 · p. 2
  • ReCast consistently achieves the best Hit@1 across four benchmarks, exceeding the strongest baseline by 5.65 and 9.19 percentage points on Who&When Algorithm and Handcrafted, respectively.The paper also reports ablation studies supporting the contributions of its main components.Source: §1 · p. 2
  • The authors report the first systematic study of step-representation discriminative ability for multi-agent failure attribution, using quantitative separation metrics within and across trajectories.Source: §1 · p. 2
  • The paper introduces ReCast-2K, a training dataset, and reports efficient attribution from precomputed features.The dataset is described as a diverse source training dataset.Source: §1 · p. 2

2 RELATED WORK

Prior failure-attribution work uses observable trajectories or LLM internal signals, with learned approaches drawing on different representations and modeling strategies.

Black-box Failure Attribution Methods

  • Black-box methods use observable trajectories for structured reasoning or learned attribution, but distinguishing root causes from noisy or propagated symptoms remains challenging.CHIEF (Wang et al., 2026) uses hierarchical causal graphs for oracle-guided backtracking and counterfactual attribution; FALAT (Rafi et al., 2026) uses hierarchical dependency-guided search.Source: §2 · pp. 2–3
  • Learning-based methods train on trajectories, using counterfactual replay and fault injection, prefix auditing, semantic dependency modeling, or step-context and agent-interaction modeling.These methods primarily use external trajectory representations, leaving internal diagnostic signals underexplored.Source: §2 · p. 3

White-box Failure Attribution Methods

  • White-box methods use LLM internal signals: MASPrism (Liu et al., 2026b) combines hidden-state-derived negative log-likelihood with attention, while OAT (Yeh et al., 2026) models successful hidden-state dynamics.ReCast instead learns attribution-oriented step representations.Source: §2 · p. 3

3 PROBLEM FORMULATION

The problem formulation treats model actions in an execution trace as attribution candidates and defines the root cause as the earliest decisive error. Its representation study evaluates whether embeddings and hidden states separate root-cause steps from other steps.

3.1 DEFINITION

  • A trace contains a task query, system states, and candidate model actions; only model actions are considered for failure attribution.Task-specific success evaluation divides traces into successful and failed sets.Source: §3.1 · p. 3
  • A step is decisive when replacing its action with a feasible corrective action and re-executing from that point turns the failed trace into a successful one.Source: §3.1 · p. 3
  • The root-cause target is the earliest decisive-error timestamp; all other candidate timestamps are non-root steps, which are not necessarily error-free.Successful traces have no root cause, so all their candidate timestamps are non-root.Source: §3.1 · p. 3

3.2 OBSERVATION

  • The study compares Qwen3 text embeddings with mean-pooled and last-token final-layer hidden states on failed trajectories from the training data and two Who&When subsets.Source: §3.2 · pp. 3–4
  • Within-trace separation asks whether a root-cause step differs from other steps in its trace, while cross-trace separation asks whether root-cause representations differ from non-root representations across traces.ARS and GRS measure local and overall within-trace separation; cosine SI measures cross-trace separation.Source: §3.2 · pp. 3–4
  • Figure 2 presents ReCast as a method for learning attribution-oriented step representations from frozen LLM hidden states.Source: Figure 2 · p. 4
  • The evaluated representations show limited separation: high ARS and GRS indicate similarity to neighboring or other steps, while low SI indicates weak separation across traces.Relative performance varies across the compared representations.Source: §3.2 · p. 4

4 METHODOLOGY

ReCast transforms frozen-LLM hidden states into attribution-oriented step representations. It selects attribution-relevant layers, constructs complementary features, and trains an encoder with contrastive and ranking objectives.

4 METHODOLOGY

  • ReCast jointly trains a contextual encoder and scoring head with root and non-root contrastive losses plus a trajectory-level ranking loss.At inference, the unaugmented trajectory is encoded and the highest-scoring step is predicted as the root cause.Source: §4 · p. 4, §4.2 · p. 6

4.1 FEATURE SELECTION AND CONSTRUCTION

  • A five-fold probe ranks layers by root-cause identification performance, then selects top-ranked layers across contiguous groups spanning model depth.The probe compares root-cause features with non-root features on failed training trajectories.Source: §4.1 · pp. 4–5
  • A fixed sparse Johnson–Lindenstrauss projection forms complementary pattern and deviation branches, with the latter measuring differences from successful trajectories.The pattern branch applies layer-wise normalization to preserve relative activation patterns.Source: §4.1 · p. 5

4.2 ATTRIBUTION-ORIENTED STEP REPRESENTATION LEARNING

  • A bidirectional encoder uses the complete trajectory to produce contextualized step representations for step-level supervision.Training augments projected trajectories with independently perturbed views while protecting root-cause steps from step masking.Source: §4.2 · p. 5
  • The non-root contrastive objective uses steps preceding the root cause as anchors and contrasts them with the root-cause step, averaging eligible trajectories equally.When anchor selections coincide, one is retained; ties in scores are resolved by choosing the earliest step.Source: §4.2 · p. 6

5 EXPERIMENTS

Experiments evaluate ReCast on four failure-attribution benchmarks using benchmark-specific training data and comparisons with heuristic, LLM-based, and feature-based methods. ReCast leads Hit@1 across the reported benchmark settings, while further results examine ranking coverage and representation separation.

5.1 EXPERIMENTAL SETUP

  • Evaluation covers four benchmarks, including 124 algorithm-generated and 58 handcrafted Who&When trajectories, 1,251 Who&When Pro test trajectories, 163 AFTraj-2K test trajectories, and 175 TraceElephant trajectories.The Who&When Pro test set uses an 8:2 task-grouped split; TraceElephant results use the Captain-Agent and Magentic-One subsets.Source: §5.1 · pp. 6–7
  • Training data varies by target benchmark: ReCast-2K is used for Who&When and TraceElephant, while AFTraj-2K and Who&When Pro use their respective training partitions.In in-domain settings, supervised feature-based baselines use the same failed-trajectory training split, except OAT, which follows its positive-only protocol.Source: §5.1 · p. 7
  • Evaluation reports Hit@k for k ∈{1, 3, 5}, MRR, AUROC, and AUPRC, with mean and standard deviation over the same three seeds for all methods.Hit@k measures whether the annotated root-cause step appears among the top-k predictions; MRR reflects its ranking position, while AUROC and AUPRC assess score-level separation.Source: §5.1 · p. 7
  • Step representations are extracted with Qwen3.5-27B, and the bidirectional Transformer encoder uses four layers with dz = 128.Each projection branch has dimension dp = 2,048; training uses AdamW for up to 50 epochs with batch size 32 and learning rate 2 × 10^-4.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • ReCast achieves the best Hit@1 across all four Who&When and TraceElephant subsets.On Who&When Algorithm and Handcrafted, it reaches 45.43% and 33.33%, exceeding CHIEF by 5.65 and 9.19 percentage points; on TraceElephant Captain-Agent and Magentic-One, it reaches 35.69% and 27.04%, exceeding ASCon by 4.71 and 1.11 percentage points, respectively.Source: Table 1 · p. 7, §5.2 · p. 8
  • ReCast achieves 84.05% Hit@1 on AFTraj-2K and 91.42% on Who&When Pro, outperforming the reported GPT-5.6 Luna Step-by-Step and ASCon baselines.The results suggest its gains are not limited to training on ReCast-2K.Source: §5.2 · p. 8, Table 2 · p. 8
  • ReCast has the highest Hit@5 across all four Who&When and TraceElephant settings, with 78.04% Hit@5 and 35.69% Hit@1 on Captain-Agent versus MASPrism’s 75.29% and 18.82%.Figure 3 shows cumulative Hit@k, with stacked ranks 1, 2–3, and 4–5 and error bars for population standard deviation across three seeds.Source: §5.2 · pp. 8–9, Figure 3 · p. 8
  • Learned representations show lower ARS and GRS and higher SI on average, with mean SI rising from 0.17 to 0.59 on Handcrafted and from 0.03 to 0.29 on Algorithm.These results indicate improved within-trace and cross-trace separation.Source: §5.2 · p. 9

6 ANALYSIS

Ablations on Who&When examine ReCast’s components, while backbone and efficiency analyses assess accuracy and attribution time under different configurations.

Ablation Study

  • The full model achieves the highest mean Hit@1 of 41.58% in Who&When ablations; layer selection, joint loss, and augmentation each improve accuracy over specified alternatives.Layer selection exceeds five alternatives by 2.75–4.40 percentage points, joint loss beats ranking-only training by 3.85 points, and removing augmentation lowers Hit@1 by 4.22 points.Source: §6 · p. 9

Backbone and efficiency analyses

  • The 27B backbone has the highest Hit@1 on Who&When, AFTraj-2K, and Who&When Pro, but 0.8B and 9B perform best on Captain-Agent and Magentic-One, respectively.Thus, increasing backbone size does not consistently improve performance.Source: §6 · p. 9
  • With step features precomputed, ReCast enables efficient attribution; its attribution stage is faster than the compared feature-based methods on HC, AG, and Magentic-One.On Captain-Agent, StepFinder and ASCon are faster, while ReCast achieves higher accuracy.Source: §6 · p. 9

7 CONCLUSION

The conclusion reports limited separation in existing representations and presents ReCast as a method for learning attribution-oriented step representations, with ReCast-2K supporting training and results showing stronger within- and cross-trace separation.

7 CONCLUSION

  • ReCast learns contextualized step representations from frozen LLM hidden states using layer selection, pattern and deviation features, and contrastive and ranking objectives.The paper also introduces ReCast-2K as training data.Source: §7 · pp. 9–10

A.1 EVALUATION METRICS

  • ARS and GRS measure within-trace similarity, where lower values indicate stronger separation; SI measures cross-trace separation, where higher values indicate stronger separation.Source: §A.1 · p. 14

A.2 REPRESENTATIONS

  • The evaluated candidate-step representations include text embeddings, mean-pooled last-layer hidden states, and last-token last-layer hidden states.The hidden-state representations use Qwen3.5-27B and the text embeddings use Qwen3-Embedding-0.6B.Source: §A.2 · p. 14

A.3 COMPLETE RESULTS

ReCast separates root-cause representations from other steps more strongly than the compared representations, while basic representations show substantial within-trace similarity and limited cross-trace separation.

A.3 COMPLETE RESULTS

  • Table 4 compares basic representations, projected features with 4,096 dimensions, and learned ReCast representations with 128 dimensions.Source: §A.3 · pp. 14–15, Table 4 · p. 15
  • ReCast’s mean SI is 0.5913 on Handcrafted and 0.2873 on Algorithm, exceeding the best basic-representation values of 0.1735 and 0.0309, respectively.Table 4 reports ARS, GRS, and SI; ReCast values are means ± population standard deviations across seeds 6, 20, and 42, while other representations are fixed.Source: Table 4 · p. 15, §A.3 · p. 15
  • Among basic representations, text embeddings perform best in most comparisons, but all three show high ARS and GRS and low SI.Source: §A.3 · p. 15

B METHOD IMPLEMENTATION DETAILS

The appendix details the layer-selection and projection procedures, benchmark composition, and candidate-step processing used in evaluation.

B.1 ATTRIBUTION-AWARE LAYER SELECTION

  • Five-fold out-of-fold probe results rank layers by Hit@1, with ties broken by MRR and mean root margin, selecting the top layer from each contiguous depth group.Trajectories are assigned to folds deterministically by trace identifier.Source: §B.1 · pp. 15–16

B.2 DUAL-BRANCH PROJECTION

  • The dual-branch projection retains selected-layer columns from a fixed sparse projection and concatenates pattern and deviation features without branch-specific weighting.The projection is shared across branches and training seeds, and the deviation branch uses a successful-reference mean.Source: §B.2 · p. 16

C.1 WHO&WHEN

  • Who&When contains 184 failed trajectories across algorithm-generated and handcrafted settings; two AG trajectories are excluded because their decisive errors are tool outputs, not model actions.Source: §C.1 · p. 17

C.2 WHO&WHEN PRO

  • The experiments use all 6,257 text trajectories from Who&When Pro, whose controlled error interventions retain cases that change successful executions into failures.Task groups are kept intact in an 8:2 training–test split, and successful ReCast-2K trajectories provide reference means and contrastive negatives.Source: §C.2 · p. 17

C.3 AFTRAJ-2K

  • AFTraj-2K contains 2,276 trajectories, with 1,944 for training and 332 for testing; successful trajectories and their injected variants stay in the same partition.Source: §C.3 · p. 18

C.4 TRACEELEPHANT

  • The TraceElephant evaluation uses 175 failed trajectories from Captain-Agent and Magentic-One, with an average of 25.1 and maximum of 59 candidate steps per trajectory.These trajectories do not exactly overlap with the failed trajectories in the training corpus.Source: §C.4 · p. 18

C.5 DATA PROCESSING

  • Candidate processing retains model-generated actions and excludes observations, tool results, and other non-model events; all compared feature-based methods use the same candidate set.For AFTraj-2K test data, processing reduces candidate steps from 1,711 to 1,548 while retaining all 163 trajectories and root-cause steps.Source: §C.5 · p. 18

D.1 TASK SAMPLING AND TRAJECTORY GENERATION

ReCast-2K is generated and filtered before root-cause annotation, and its final dataset contains 1,044 successful and 1,496 failed trajectories with variation in trajectory length and generation model.

D.1 TASK SAMPLING AND TRAJECTORY GENERATION

  • ReCast-2K collection starts from 3,222 trajectories generated with Captain-Agent and Magentic-One on GAIA and AssistantBench, then filters malformed traces and deduplicates answers within task and source configuration.Source: §D.1 · pp. 18–19, Table 6 · p. 19, §D.2 · p. 19

D.2 ROOT-CAUSE ANNOTATION

  • Three independent LLM judges label failed trajectories by step-level voting; 1,112 receive consensus labels and the remaining 384 are manually reviewed.Among consensus cases, 766 have unanimous agreement and 346 have a two-of-three majority.Source: §D.2 · p. 19

D.3 FINAL DATASET STATISTICS

  • Failed trajectories average 25.9 steps and 8,346.6 tokens, versus 17.2 steps and 5,605.5 tokens for successful trajectories.GPT-4o-0806 failed trajectories average the most steps at 32.0, while DeepSeek V4 Flash has the highest average and maximum token counts among failed trajectories.Source: §D.3 · pp. 19–20, Table 7 · p. 20

E.1 EVALUATION METRICS

The section defines ranking and score-level evaluation metrics, reports three-seed variability, and gives ReCast’s hyperparameter settings alongside deterministic and random heuristic baselines.

E.1 EVALUATION METRICS

  • Hit@k and MRR evaluate whether and how highly the root-cause step ranks, while AUROC and AUPRC assess pooled score-level separation.AUPRC is particularly informative under class imbalance.Source: §E.1 · p. 20
  • AUROC and AUPRC pool candidate-level predictions across trajectories, comparing root-cause steps with non-root steps.AUPRC uses a non-interpolated average-precision formulation.Source: §E.1 · pp. 20–21

E.2 HYPERPARAMETER SETTINGS

  • ReCast uses eight selected layers, two 2,048-dimensional feature branches, and a four-layer bidirectional Transformer producing 128-dimensional step representations.The backbone, projection matrix, and successful-reference mean remain fixed while the encoder and scoring head are jointly optimized.Source: §E.2 · p. 21, Table 8 · p. 21

F.1 HEURISTIC BASELINES

  • First deterministically predicts the earliest candidate, while Random uniformly shuffles candidates; both use the original candidate set.First uses no trajectory text, agent identity, or benchmark labels, and has zero seed variance.Source: §F.1 · p. 21

F.2 LLM-BASED METHODS

The section describes LLM and feature-based attribution baselines, then reports ReCast’s ranking, score-discrimination, and backbone-comparison results across benchmarks.

F.2 LLM-BASED METHODS

  • The LLM protocols use either the complete conversation in one request or growing prefixes, with the latter stopping at the first parsed “Yes.”Each run returns at most one prediction and is evaluated only with Hit@1; unparsable or invalid-candidate responses count as incorrect.Source: §F.2 · p. 22

F.3 FEATURE-BASED METHODS

  • OAT ranks steps by reconstruction error against learned successful-trajectory dynamics, while StepFinder refines learned error logits with cross-step differences and an early-position bias.OAT uses squared reconstruction error; StepFinder trains with trajectory-level cross-entropy and an auxiliary next-step representation loss.Source: §F.3 · pp. 22–23
  • MASPrism ranks candidate sources using prefill-derived attention, likelihood contrast, and multi-symptom consensus; ASCon incorporates contextualized step and agent representations.MASPrism uses two prefill passes without decoding or root-cause annotations.Source: §F.3 · pp. 23–24

G.1 MRR ACROSS RANK CUTOFFS

  • ReCast leads all compared methods in MRR@K at every cutoff across four benchmarks; on TraceElephant, MRR@1 is 0.3124 versus ASCon’s 0.2838.At K=5, ReCast reaches 0.4341 versus ASCon’s 0.3860.Source: Figure 4 · p. 24, §G.1 · p. 24

G.2 SCORE-LEVEL DISCRIMINATION

G.3 HIDDEN-STATE BACKBONE COMPARISON

  • Qwen3.5-27B has the highest Hit@1 on both Who&When subsets, AFTraj-2K, and Who&When Pro, while smaller backbones lead on Captain-Agent and Magentic-One.On AFTraj-2K, 0.8B and 9B Hit@1 scores are within 1.23 percentage points of 27B; 9B leads Hit@5 and AUROC.Source: §G.3 · pp. 25–26, Table 11 · p. 26

G.4 TIME EFFICIENCY ANALYSIS

The section compares attribution time and accuracy, illustrates step-level scores on selected trajectories, and describes two LLM prompt protocols.

G.4 TIME EFFICIENCY ANALYSIS

  • ReCast has the highest Hit@1 across four subsets, with mean attribution times of 0.11–8.28 ms per trajectory; measured LLM baselines take 2.88–30.50 seconds.It is the fastest compared feature-based method on HC, AG, and Magentic-One; on Captain-Agent, StepFinder and ASCon are faster but less accurate.Source: Figure 7 · p. 26, §G.4 · pp. 26–27

H CASE STUDY

  • Figure 8’s selected cases show ReCast ranking annotated causes first, including the missing source constraint at HC-37 step 4 and premature completion at HC-42 step 29.The HC-50 case likewise ranks the step-5 location-only search above a later competing score peak.Source: Figure 8 · p. 27, H CASE STUDY · pp. 27–28
  • In the Algorithm cases, ReCast ranks the annotated simulated-data substitution at AG-116 step 7, distance-conversion error at AG-109 step 1, and unverified claims at AG-120 step 8 first.Source: Figure 8 · p. 27, H CASE STUDY · p. 28

I.1 ALL-AT-ONCE PROMPT

I.2 STEP-BY-STEP PROMPT

  • Step-by-Step evaluates growing prefixes and advances unless the model first returns a parsed “Yes.”Source: §I.2 · p. 29

Related papers

Loading 2610.11334v1…