PaperlayerView exact revision on arXiv

Source-linked AI summary

LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures

Yunfei Zhang, Boyu Feng, Changhua Pei, Zexin Wang, Zhihuang Peng, Xinlong Liu, Hengyue Jiang, Difeng Ma, Jiayi Zhang, Yongzhou Yao, Yanan Zhao, Fei Sun, Yintong Huo, Zhaoyang Liu, Jingjing Li, Gaogang Xie, Dan Pei

arXiv:2608.15242v1cs.AIcs.SE

TL;DR

Long-horizon agent failures are difficult to diagnose because outcome evaluation does not identify the responsible role or earliest decisive error. LongRCA Bench and the training-free RCTA method address this gap, with RCTA achieving 51.1% responsible-role accuracy and 24.1% exact root-step accuracy.

  • Problem

    Outcome evaluation rarely identifies where decisive errors entered long agent trajectories or which workflow role was responsible.

  • Method

    RCTA retrieves candidate error steps from segment summaries and traces them backward to earlier handoff instructions for separate role and root-step attribution.

  • Results

    RCTA achieves 51.1% responsible-role accuracy and 24.1% exact root-step accuracy, strongest among evaluated methods under the shared protocol.

  • Takeaways & Limitations

    LongRCA Bench supports evaluating responsible-role attribution and earliest decisive root-step localization as separate targets in long-horizon failure diagnosis.

  • Takeaways & Limitations

    The benchmark does not measure full causal-chain reconstruction because intermediate causal explanations remain unscored.

Abstract

from arXiv · show

When a long-horizon agent execution fails, outcome-level evaluation reveals the unsuccessful result but not where the decisive error entered the trajectory. Developers must then inspect the full execution to identify the responsible role and localize the earliest decisive root-cause step. Existing failure-attribution benchmarks largely focus on shorter traces, leaving diagnosis across hundreds of recorded steps underexplored. We introduce LongRCA Bench, comprising 1,140 failed trajectories across five domains without injected errors. It provides independently scored human labels for the responsible role and earliest decisive root-cause step. The median trajectory contains 145 steps, and the strongest baseline reaches only 13.2% exact root-step accuracy. We further present Root-Cause Trajectory Attribution (RCTA), a training-free method that retrieves candidate error steps from segment summaries and traces them to available earlier handoff instructions. Using the same backbone, benchmark instances, and scoring protocol, RCTA reaches 51.1% responsible-role accuracy and 24.1% exact root-step accuracy. These results highlight the need to evaluate responsible-role attribution and exact root-step localization as separate targets in long-trajectory failure diagnosis.

1 Computer Network Information Center, Chinese Academy of Sciences

The section lists the authors’ institutional affiliations across Chinese, Singaporean, and Alibaba research organizations. It also notes equal contribution and identifies a corresponding author.

  • Institutional affiliations: The affiliations span the Chinese Academy of Sciences, University of Chinese Academy of Sciences, Chongqing University, Singapore Management University, Alibaba Group, and Tsinghua University.The listed units include the Computer Network Information Center, Hangzhou Institute for Advanced Study, Institute of Computing Technology, and Tongyi Lab.
  • Author notes: The paper marks some authors as making equal contributions and identifies a corresponding author.These notes are indicated by an asterisk and dagger, respectively.

1 INTRODUCTION

LongRCA Bench frames long-horizon failure diagnosis as two independent tasks: identifying the earliest decisive root-cause step and the responsible workflow role. It introduces a benchmark of 1,140 failed trajectories and RCTA, a training-free method that improves both attribution targets over baselines.

  • Motivation: Outcome-level evaluation reveals failed results but rarely identifies where the decisive error entered or which workflow role caused it.The diagnostic gap is especially consequential when hundreds of recorded steps separate the initial error from the failed outcome.
  • Problem formulation: Root-cause localization selects the earliest recorded step introducing the evaluator-confirmed decisive error, while responsible-role attribution identifies the accountable trajectory role.Successfully repaired errors are excluded, and valid role labels must match roles recorded in the source trajectory, including possible human roles.
  • Challenge: LongRCA Bench defines a diagnostic regime where excessive trajectory length and large root-to-end distances make decisive roots difficult to locate.The benchmark characterizes this regime through source diversity, trajectory length, and root-to-end distance.
  • Benchmark: LongRCA Bench contains 1,140 observed, non-injected failed trajectories with human labels for responsible roles and earliest decisive root-cause steps.The benchmark spans five agentic domains and 178,137 recorded steps in total.
  • Method: RCTA combines segment-level candidate recall with backward tracing to earlier handoff instructions for focused long-trajectory diagnosis.The method separates broad trajectory search from comparison of the original logged text.
  • Results: 51.1% responsible-role accuracy, 24.1% root-cause exact accuracy, and 37.4% root-cause ±5 accuracy were achieved by RCTA.Using the same inference backbone, trajectories, and scoring protocol, these results exceeded the strongest baseline by 23.6, 10.9, and 12.7 percentage points, respectively.

2 RELATED WORK

Prior trajectory-failure resources vary in supervision targets, annotation protocols, and diagnostic methods. LongRCA Bench evaluates responsible-role attribution and earliest decisive root-cause localization on substantially longer recorded histories.

  • Benchmarks: Existing benchmarks target recurring failure modes, erroneous spans, recovery-impossible steps, initial deviations, responsible agents, decisive steps, or failure-inducing steps.MAST, TRAIL, AgentRx, Failure as a Process, Who&When, Who&When Pro, and MP-Bench exemplify these differing outputs and supervision schemes.
  • Benchmarks: 1,140 observed failures comprise LongRCA Bench, with human annotations for the responsible role and earliest decisive root-cause step.The benchmark places these labels within substantially longer recorded histories than prior resources.
  • Diagnostic methods: Prompting-based approaches search logged text or compact trajectory representations using all-at-once, step-by-step, binary-search, hierarchical-context, consensus, or judge-based strategies.Who&When introduced three prompting modes, while ECHO combines hierarchical context representations with consensus voting and RAFFLES alternates between a central judge and specialized evaluators.
  • Diagnostic methods: Structure-aware methods localize failures through replay and spectrum analysis, causal and Shapley analysis, hierarchical causal graphs, counterfactual screening, or dependency search.FAMAS, CDC-MAS, CHIEF, and FALAT exploit execution dependencies or evidence from previous runs.
  • Diagnostic methods: LongRCA Bench independently evaluates responsible roles and root-cause steps, comparing direct prompting, hierarchical context, and dependency-search strategies.The benchmark separates diagnostic outputs because some methods return only a critical step or failure category, while others return both a role and a step.

3 LONGRCA BENCH

LongRCA Bench evaluates post hoc attribution of failed agent trajectories through two independently scored outputs: the responsible recorded workflow role and the earliest decisive root-cause step. It standardizes heterogeneous executions and finalizes human references across 1,140 trajectories from five task domains.

  • Benchmark interface: Each instance contains a task instruction, a complete failed trajectory, evaluator-supplied outcome, indexed records, and recorded source roles.The reference role must appear in the trajectory, while it need not emit the selected root-cause step.
  • Benchmark interface: Responsible-role and root-cause-step predictions are scored independently, with roles restricted to the trajectory’s recorded role set.LongRCA Bench evaluates both outputs after the failed trajectory concludes rather than testing online failure detection.
  • Root-cause definition: The reference root-cause step is the earliest unrepaired recorded step introducing the decisive error relevant to the evaluator-confirmed failure.Successfully repaired earlier errors and later execution, propagation, or exposure steps are excluded; under handoff, a decisively erroneous instruction step is selected.
  • Data coverage: The benchmark retains evaluator-failed executions from five sources spanning software repair, terminal tasks, travel planning, service-oriented tool use, and web interaction.The sources include SWE-bench Pro, Terminal Bench 2, TravelPlanner, VitaBench, and WebArena Verified; infrastructure, smoke-test, and debug runs are excluded.
  • Human annotation: Twenty-two computer-science graduate students annotated each trajectory’s responsible role, earliest decisive root-cause step, and trajectory-grounded rationale.After checking references and resolving disagreements, the process produced one finalized reference for each of 1,140 trajectories.

4 DATASET ANALYSIS

LongRCA Bench spans 1,140 failed trajectories across five domains, multiple agent organizations, and three generator models, with substantial variation in execution length. Its long traces often place the decisive root far before the end, requiring localization amid hundreds of subsequent steps.

  • Dataset composition: 1,140 failed trajectories span five domains, with TravelPlanner contributing 685 (60.1%) and the other four sources contributing 455 (39.9%).The domains are travel planning, web interaction, software repair, service-oriented tool use, and terminal tasks.
  • Dataset composition: The benchmark includes fixed-role teams, group-chat coordination, sequential agent organizations, and three generator models: MiniMax-M2.5 (681), Kimi-K2.5 (276), and Qwen3.5-Plus (183).Source-level profiles also vary in trajectory length and root-to-end distance.
  • Trajectory structure: 178,137 recorded steps yield an average trajectory length of 156.3 steps, a median of 145, and a maximum of 728; the reference root occurs at median step 55.Median trajectory length ranges from 51 steps in VitaBench to 257.5 in SWE-bench Pro, while median root-to-end distance ranges from 24 to 186 steps across sources.
  • Trajectory structure: 49.0% of trajectories extend more than 50 steps beyond the reference root, 28.4% exceed 100 steps, and 7.7% exceed 200 steps.Root-to-end distance is the number of recorded steps after the annotated root, not a labeled propagation-chain length.

5 ROOT-CAUSE TRAJECTORY ATTRIBUTION

RCTA diagnoses long-trajectory failures by separating broad candidate recall from evidence-based attribution, using segment summaries, original step text, and earlier handoff instructions. It independently predicts the responsible role and earliest decisive root-cause step while validating output provenance.

  • Motivation and overview: RCTA separates broad candidate recall from final evidence-based attribution because relevant fault evidence may span many steps and originate in earlier instructions.The method uses segment summaries for broad search and original logged text for final attribution.
  • Method overview: RCTA applies three stages: trajectory partitioning, candidate error-step recall from segment summaries, and retrieval of relevant earlier handoff instructions.Segments are consecutive blocks of recorded steps, while handoffs are logged instructions from one role to another.
  • Handoff-based attribution: RCTA retrieves the nearest preceding handoff addressed to executor or verifier candidates, or the nearest earlier handoff as plan context for other candidates.The final LLM call compares each retained candidate’s original text with the retrieved handoff context.
  • Candidate recall: One LLM call per segment proposes candidate error steps, a second organizes segment summaries by subgoal, and the method prioritizes cited IDs before retrieving original step text.The outline supplies global progress information, whereas original step text supplies evidence for attribution.
  • Output validation: RCTA predicts the responsible role and root-cause step in separate fields, while validators verify recorded roles, step IDs, and quoted handoff provenance before scoring.Validation establishes output validity and textual provenance rather than semantic diagnosis correctness.

6 EVALUATION

The evaluation independently measures responsible-role attribution and root-cause-step localization across 1,140 failed trajectories. RCTA leads the matched comparison, but exact root-step localization remains substantially harder than role attribution.

  • Evaluation metrics: Responsible-role accuracy and root-cause-step localization are scored independently, with exact localization requiring the predicted 0-based step to match the reference.Missing or invalid role outputs are incorrect for role accuracy; non-numeric or out-of-range step outputs are incorrect for both step metrics.
  • Overall results: 24.1% exact root accuracy is achieved by the strongest method, versus 51.1% responsible-role accuracy.RCTA identifies the responsible workflow role more often than it localizes the exact earliest decisive step.
  • Overall results: RCTA leads all four reported metrics, improving over ECHO by 23.6, 10.9, and 12.7 percentage points on accuracy metrics while reducing MAE by 11.8 steps.ECHO records 27.5% role accuracy, 13.2% exact root accuracy, 24.7% within-five accuracy, and root MAE of 50.4.
  • Baseline comparison: 2.8% exact root accuracy is attained by FALAT, compared with 12.5% within-five accuracy, both below all-at-once prompting.The comparison is limited to the evaluated implementation and does not characterize dependency-based diagnosis methods generally.
  • Stratified analysis: 30.3% exact accuracy for trajectories of at most 100 steps falls to 20.3% for 101–200 steps and 20.2% for 201–400 steps under RCTA.The >400-step bin reaches 31.0% but contains only 42 trajectories and differs in source composition; root-to-end-distance results are non-monotonic.

7 DISCUSSION AND LIMITATIONS

LongRCA Bench separates responsible-role attribution from exact root-cause-step localization, showing that these capabilities can diverge. Its conclusions are bounded by annotation scope, evaluation setting, matched-backbone comparisons, and source-confounded difficulty analyses.

  • Discussion: LongRCA Bench scores responsible-role attribution and root-cause-step localization independently, preventing role-attribution success from obscuring weaknesses in exact event localization.RCTA identifies the responsible workflow role more often than it localizes the exact earliest decisive event.
  • Limitations: Human annotation provides one finalized responsible-role label, one finalized root-cause-step label, and a rationale for every trajectory.The two labels are scored independently; rationales document supporting evidence but are not scored.
  • Limitations: The benchmark scores only the responsible role and earliest decisive root-cause step, so RCTA’s intermediate causal chain is not evaluated as full causal-chain reconstruction.Intermediate causal explanations remain unscored benchmark outputs.
  • Limitations: LongRCA Bench diagnoses failures from completed full logs and therefore does not test early warning or online intervention.Its results should not be compared directly with protocols rewarding detection before failure fully unfolds.
  • Limitations: All methods use the same inference backbone, supporting matched-backbone conclusions rather than model-independent performance claims.Absolute performance may change with a stronger inference backbone.
  • Limitations: Length and root-to-end bins combine heterogeneous source domains, establishing associations within LongRCA Bench rather than isolated causal effects.Controlled matched-source experiments and component ablations are proposed to clarify difficulty trends and RCTA contributions.

8 CONCLUSION

LongRCA Bench evaluates responsible-role attribution and root-cause-step localization in long-horizon agent failures, while RCTA narrows trajectories to candidate error steps and compares them with earlier handoff instructions. The results support treating responsibility and root-cause localization as explicit diagnostic targets beyond final-outcome evaluation.

  • Benchmark: LongRCA Bench contains 1,140 failed trajectories from five heterogeneous source benchmarks with retained human annotations.In many trajectories, the reference root cause is followed by a long sequence of downstream steps before execution ends.
  • Method and results: RCTA compares candidate error steps with relevant earlier handoff instructions and returns separately validated role and root-step predictions.Under the same backbone, benchmark instances, and scoring protocol, it achieves the strongest performance on both targets among evaluated methods.
  • Method and results: 51.1% role accuracy and 24.1% exact root-step accuracy are achieved by RCTA under the shared evaluation setup.These results are reported for responsible-role attribution and exact root-cause-step localization, respectively.
  • Implication: Responsibility and root-cause localization should be treated as explicit targets when diagnosing completed agent failures.Final-outcome evaluation alone cannot provide either form of diagnostic evidence.

A RELEASED DATA SCHEMA · B SOURCE EXECUTION PROVENANCE

The released schema separates model-visible inputs from withheld reference annotations used for scoring, while provenance records document source workflows, generators, and counts. Together, these sections specify both prediction-time inputs and execution-source metadata.

  • A RELEASED DATA SCHEMA: The released records separate model-visible fields from reference annotations withheld during prediction.This separation defines which information evaluated methods can access versus which annotations remain unavailable at prediction time.
  • A RELEASED DATA SCHEMA: Evaluated methods receive the question ID and history, including the task instruction.The table identifies these as the released prediction-time inputs.
  • A RELEASED DATA SCHEMA: Three reference annotation fields are withheld during prediction and support benchmark scoring.The schema distinguishes these reference fields from the inputs provided to evaluated methods.
  • B SOURCE EXECUTION PROVENANCE: Source-execution provenance records implementation-level workflow names and generator counts for each source.These details complement the broader agent-organization descriptions in the main benchmark summary.
  • B SOURCE EXECUTION PROVENANCE: SWE-bench Pro and Terminal Bench 2 archives record custom role workflows and generators without separate third-party scaffold packages.The provenance table explicitly distinguishes these custom workflows from third-party scaffold packages.
  • B SOURCE EXECUTION PROVENANCE: Generator columns correspond to MiniMax-M2.5, Kimi-K2.5, and Qwen3.5-Plus, with dashes marking absent trajectories.The table abbreviates these generators as M2.5, K2.5, and Q3.5+, respectively.

C DIFFICULTY-STRATIFIED NUMERIC RESULTS … F PIPELINE PSEUDOCODE

The paper reports difficulty-stratified root-cause exact accuracy and specifies RCTA’s decision rules, prompting stages, validation behavior, and inference pipeline. These components distinguish root-step selection from responsible-role prediction and govern candidate retrieval, fallback handling, and abstention.

  • C DIFFICULTY-STRATIFIED NUMERIC RESULTS: Both difficulty analyses report root-cause exact accuracy (%) across four trajectory-length bins and four root-to-end-distance bins, with bin sizes reported in each caption.The trajectory-length bins contain 393, 467, 238, and 42 trajectories; the root-to-end-distance bins contain 256, 325, 235, and 324.
  • C DIFFICULTY-STRATIFIED NUMERIC RESULTS: The difficulty-stratified values support descriptive comparisons rather than causal conclusions because source domains and workflows differ across bins.
  • D ROOT-CAUSE DECISION RULES: If a later step implements an erroneous handoff decision, the handoff is selected; if it departs from the handoff or introduces a new decisive error, the later step is selected.
  • D ROOT-CAUSE DECISION RULES: An earlier error is excluded when repaired before evaluator-confirmed failure, while a later verification omission does not replace the step that introduced the error.
  • D ROOT-CAUSE DECISION RULES: RCTA selects the earliest log-supported decisive error that remains unrepaired through evaluator-confirmed failure, while predicting responsible role in a separate output field.Contradictory task and evaluation requirements permit selecting an input step only when both explicitly incompatible texts are cited from the trajectory.
  • E RCTA PROMPTS AND IMPLEMENTATION DETAILS: RCTA uses local summarization, trajectory-outline aggregation, and final attribution; with m segments, this requires m + 2 initial LLM calls before validation-triggered retries.Verifier PASS or FAIL markers provide only optional rule-based segmentation boundaries, not semantic judgments, benchmark labels, or the final evaluator outcome.
  • E RCTA PROMPTS AND IMPLEMENTATION DETAILS: Phase-construction parse failures fall back to one phase per local summary, and 308 of 1,140 outputs contain phase counts outside the requested 3–8 range.The final artifacts do not retain a fallback flag, so 308 should not be interpreted as an exact fallback count.
  • F PIPELINE PSEUDOCODE: The pipeline segments trajectories, summarizes segments, organizes phases, retrieves and prioritizes candidate steps, adds earlier handoff instructions, then validates role and root predictions independently.Invalid fields are removed while valid fields are preserved; the system abstains when essential fields remain missing.
Loading 2608.15242v1…