PaperlayerView exact revision on arXiv

Source-linked AI summary

LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures

Yunfei Zhang, Boyu Feng, Changhua Pei, Zexin Wang, Zhihuang Peng, Xinlong Liu, Hengyue Jiang, Difeng Ma, Jiayi Zhang, Yongzhou Yao, Yanan Zhao, Fei Sun, Yintong Huo, Zhaoyang Liu, Jingjing Li, Gaogang Xie, Dan Pei

arXiv:2608.15242v1cs.AIcs.SE

TL;DR

Outcome-level evaluation reveals when long-horizon agent executions fail, but not where the decisive error entered or which role was responsible. LongRCA Bench evaluates both targets, and RCTA achieves 51.1% responsible-role accuracy and 24.1% exact root-step accuracy, the strongest results among evaluated methods.

  • Problem

    Outcome-level evaluation identifies failed results but rarely reveals the decisive error’s location or responsible workflow role in long-horizon trajectories.

    Source: §1 · p. 1

  • Method

    LongRCA Bench labels responsible roles and earliest decisive root-cause steps, while RCTA retrieves candidate error steps from segment summaries and traces them to earlier handoff instructions.

    Source: §1 · pp. 2–3

  • Results

    51.1% responsible-role accuracy and 24.1% exact root-cause accuracy; RCTA is strongest on both targets among evaluated methods under the shared protocol.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    Long-horizon failure diagnosis evaluates responsible-role attribution and exact root-step localization as separate targets.

    Source: §3.1 · p. 4, §8 · p. 10

    The benchmark does not score intermediate causal-chain reconstruction, so reported results do not measure that capability.

    Source: §7 · p. 10

Abstract

from arXiv · show

When a long-horizon agent execution fails, outcome-level evaluation reveals the unsuccessful result but not where the decisive error entered the trajectory. Developers must then inspect the full execution to identify the responsible role and localize the earliest decisive root-cause step. Existing failure-attribution benchmarks largely focus on shorter traces, leaving diagnosis across hundreds of recorded steps underexplored. We introduce LongRCA Bench, comprising 1,140 failed trajectories across five domains without injected errors. It provides independently scored human labels for the responsible role and earliest decisive root-cause step. The median trajectory contains 145 steps, and the strongest baseline reaches only 13.2% exact root-step accuracy. We further present Root-Cause Trajectory Attribution (RCTA), a training-free method that retrieves candidate error steps from segment summaries and traces them to available earlier handoff instructions. Using the same backbone, benchmark instances, and scoring protocol, RCTA reaches 51.1% responsible-role accuracy and 24.1% exact root-step accuracy. These results highlight the need to evaluate responsible-role attribution and exact root-step localization as separate targets in long-trajectory failure diagnosis.

1 INTRODUCTION

LongRCA Bench evaluates failure diagnosis in long-horizon agent executions, where outcome-level evaluation does not reveal the decisive error step or responsible role. RCTA combines segment-level candidate retrieval with backward tracing to earlier handoff instructions.

Motivation

  • Outcome-level evaluation identifies failure but not the decisive error step or responsible workflow role, a gap compounded by long traces and roots far before execution ends.The diagnostic setting is offline attribution for completed, failed trajectories.Source: §1 · pp. 1–2

Task formulation

  • LongRCA treats responsible-role attribution and earliest decisive root-step localization as independent predictions, rather than deriving responsibility from the selected step’s emitter.Repaired errors are not selected as roots, later steps do not replace the step that introduced the decisive error, and a role label must match a role recorded in the trajectory.Source: §1 · p. 1

Benchmark

  • LongRCA Bench includes 1,140 observed, non-injected failures across five domains, with human role and root-step labels; median trajectory length is 145 steps and median root-to-end distance is 48 steps.The maximum root-to-end distance is 605 steps.Source: §1 · p. 2

Method

  • RCTA recalls candidate error steps through segment-level analysis, then traces each candidate to earlier handoff instructions that may contain the same error.This separates broad trajectory search from focused comparison of the original logged text.Source: §1 · pp. 2–3

Results

  • 51.1% responsible-role accuracy, 24.1% root-cause exact accuracy, and 37.4% root-cause ±5 accuracy put RCTA 23.6, 10.9, and 12.7 percentage points above the strongest baseline, respectively.The comparison uses the same inference backbone, 1,140 trajectories, and scoring protocol.Source: §1 · p. 2

2 RELATED WORK

Prior failure benchmarks and diagnosis methods differ in the outputs they target and how they search histories. LongRCA Bench focuses on responsible-role attribution and earliest decisive root-step localization across long recorded histories.

2.1 BENCHMARKS AND EMPIRICAL RESOURCES FOR AGENT-FAILURE ANALYSIS

2.2 METHODS FOR TRAJECTORY DIAGNOSIS

  • Prompting-based diagnosis ranges from direct step or agent attribution to hierarchical context, evaluator consensus, constraint checks, and selective trace access (Zhang et al., 2025; Banerjee et al., 2025; Zhu et al., 2026a; Barke et al., 2026; Zhu et al., 2026b).These strategies include Who&When prompting, ECHO, RAFFLES, AgentRx, and SAFARI; SAFARI targets traces exceeding effective context budgets.Source: §2.2 · p. 3
  • Structure-aware methods use execution dependencies, causal analysis, or evidence from earlier failures to localize errors and attribute roles or steps (Ge et al., 2025; Ma et al., 2025; Wang et al., 2026c; Rafi et al., 2026; Yu et al., 2025).FAMAS replays failures, CDC-MAS uses causal and Shapley analysis, CHIEF backtracks over causal graphs, FALAT searches execution dependencies, and CORRECT retrieves error schemata.Source: §2.2 · p. 3
  • Diagnosis methods return different outputs, while LongRCA Bench separately evaluates responsible roles and root-cause steps on long trajectories and compares prompting, hierarchical-context, and dependency-search strategies.Some prior methods produce only a critical step or failure category, while others return both a role and a step.Source: §2.2 · p. 3

3 LONGRCA BENCH

LongRCA Bench standardizes failure diagnosis across heterogeneous long-horizon agent workflows by separately labeling the responsible recorded role and earliest decisive root-cause step. It combines evaluator-confirmed failed executions with human-reviewed trajectory references.

3.1 TASK DEFINITION

  • The benchmark independently scores an explicit responsible-role prediction and the earliest decisive root-cause step after a trajectory fails.The root is the earliest unrepaired step introducing the decisive error; repaired errors and later execution or propagation steps are excluded. For handoffs, the instruction is selected if it contains the error, otherwise a recipient’s departure or new error is selected.Source: §3.1 · p. 4

3.2 SOURCE TASKS AND AGENT ORGANIZATIONS

  • LongRCA retains evaluator-confirmed failures from five benchmarks spanning software repair, terminal tasks, travel planning, service-oriented tool use, and web interaction.The sources include fixed-role teams, group-chat coordination, and sequential agent organizations; trajectories were generated by MiniMax-M2.5, Kimi-K2.5, and Qwen3.5-Plus.Source: §3.2 · pp. 4–5

3.3 UNIFIED TRAJECTORY PARSING

  • Different source logs are normalized into complete step-indexed records with role names and content, retaining available tool and verifier metadata.Parsed instances include task instructions and trajectories; reference role and root-step fields are withheld during prediction.Source: §3.3 · p. 5

3.4 HUMAN ANNOTATION AND LABEL RESOLUTION

  • Twenty-two computer-science graduate students annotated trajectories using the task instruction, full trajectory, and evaluator outcome, producing human rationales and labels.Annotation typically took 30–40 minutes per trajectory; role and step references were checked, disagreements reviewed, and one reference finalized for each of 1,140 trajectories.Source: §3.4 · p. 5

4 DATASET ANALYSIS

LongRCA Bench spans five domains and multiple agent organizations and generator models, with trajectories extending hundreds of steps. Its annotated roots can be followed by long stretches of recorded execution.

4.1 SOURCE, AGENT ORGANIZATION, AND GENERATOR MODEL COVERAGE

  • The benchmark covers five domains and 1,140 failed trajectories; TravelPlanner contributes 685 (60.1%), and the other four sources contribute 455 (39.9%).Associated organizations include fixed-role teams, group-chat coordination, and sequential agent organizations.Source: §4.1 · p. 5, Figure 2 · p. 6
  • Trajectories come from MiniMax-M2.5 (681), Kimi-K2.5 (276), and Qwen3.5-Plus (183), with median lengths ranging from 51 to 257.5 steps across sources.Median root-to-end distance ranges from 24 to 186 steps across sources.Source: §4.1 · p. 6

4.2 TRAJECTORY LENGTH AND ROOT-TO-END DISTANCE

  • Across all trajectories, length averages 156.3 steps (median 145; maximum 728), while root-to-end distance has median 48, 90th percentile 183, and maximum 605 steps.The reference root occurs at median step 55.Source: §4.2 · p. 6
  • In 559 trajectories (49.0%), more than 50 steps follow the reference root; 324 (28.4%) have more than 100, and 88 (7.7%) have more than 200.Root-to-end distance counts recorded steps after the annotated root, not labeled propagation-chain length.Source: §4.2 · p. 6
  • The Section 6 stratified analysis is descriptive and does not establish a causal effect of root-to-end distance on model accuracy.Root-to-end distance measures recorded steps after the annotated root.Source: §4.2 · p. 6

5 ROOT-CAUSE TRAJECTORY ATTRIBUTION

RCTA addresses long-horizon attribution by separating broad candidate recall from evidence-based final attribution, since fault evidence may be dispersed and a late error may originate in an earlier instruction. It uses segment summaries to find candidates and logged handoffs to trace earlier context.

5 ROOT-CAUSE TRAJECTORY ATTRIBUTION

  • RCTA separates broad candidate recall from evidence-based attribution, using segment summaries to find candidate errors and earlier handoff instructions to evaluate them.The final attribution uses the original logged text rather than summaries alone.Source: §5 · p. 6

5.1 SEGMENTING THE TRAJECTORY

  • Segments preserve original step IDs, enforce character and step limits, and overlap by up to five preceding steps to retain boundary context.Rule-based boundaries may align with completion or handoff transitions; segmentation uses no language model.Source: §5 · p. 6, §5.1 · p. 7

5.2 RECALLING CANDIDATE ERROR STEPS

  • One LLM call per segment summarizes steps and proposes candidate error IDs; another combines adjacent summaries into a subgoal outline, then RCTA prioritizes candidates and retrieves their original text.The outline summarizes segments rather than repartitioning the trajectory, providing global progress context while original step text supplies attribution evidence.Source: §5.2 · p. 7

5.3 TRACING CANDIDATES TO HANDOFF INSTRUCTIONS

  • RCTA compares each retained candidate with the nearest relevant earlier handoff, selecting the instruction or candidate as root according to where the decisive error entered the trajectory.A repaired error is excluded as the failure root. Role and root-step predictions are separate fields, and programmatic checks validate role, step IDs, and quoted handoff provenance; invalid outputs receive one retry.Source: §5.3 · p. 8

6 EVALUATION

The evaluation scores responsible-role attribution and root-step localization independently; RCTA performs best across the full benchmark, while exact localization varies across trajectory and root-position bins.

6.1 EXPERIMENTAL SETTINGS

  • Role attribution and root-step localization are scored independently across all 1,140 trajectories, with exact matching as the primary localization metric.Role accuracy evaluates the explicit role output, not the role associated with the predicted root step; missing or invalid role and step outputs count as incorrect for their respective accuracy metrics. Root-step scoring also includes ±5 accuracy and source-weighted valid-output MAE; invalid step outputs do not contribute a direct MAE error term. All six methods use DeepSeek-V4-Flash, the same trajectories, and the same scoring rules.Source: §6.1 · p. 8

6.2 FULL-BENCHMARK RESULTS

  • RCTA leads the full benchmark, reaching 51.1% responsible-role accuracy and 24.1% exact root-step accuracy, versus ECHO’s 27.5% and 13.2%, respectively.RCTA performs best on all four reported metrics; versus ECHO it gains 23.6, 10.9, and 12.7 percentage points on the three accuracy metrics and reduces root MAE by 11.8 steps. ECHO reports 24.7% within-five accuracy and root MAE of 50.4.Source: Table 3 · p. 9, §6.2 · p. 9
  • FALAT’s dependency-guided search scores below all-at-once prompting on exact and within-five root accuracy in this evaluated implementation.FALAT attains 2.8% exact root accuracy and 12.5% within-five accuracy; this comparison does not characterize dependency-based diagnosis methods in general.Source: §6.2 · p. 9

6.3 PERFORMANCE BY TRAJECTORY LENGTH AND ROOT-TO-END DISTANCE

  • RCTA’s exact accuracy varies non-monotonically across trajectory-length and root-to-end-distance bins, and the pooled stratification is descriptive rather than causal.By trajectory length, RCTA scores 30.3% for ≤100 steps, 20.3% for 101–200, 20.2% for 201–400, and 31.0% for >400 steps; the last bin contains only 42 trajectories with different source composition. Across the four root-to-end-distance bins, accuracy is 21.5%, 27.1%, 20.9%, and 25.6%.Source: §6.3 · p. 9

7 DISCUSSION AND LIMITATIONS

LongRCA Bench treats responsible-role attribution and earliest root-cause-step localization as separate diagnostic capabilities, revealing that RCTA identifies roles more often than it localizes decisive events. The discussion bounds these findings to human-labeled, full-log diagnosis under a matched-backbone evaluation and identifies targeted follow-up experiments.

7 DISCUSSION AND LIMITATIONS

  • RCTA identifies the responsible workflow role more often than it localizes the exact earliest decisive event, so role attribution and root-step localization must be evaluated separately.This separation prevents success on role attribution from obscuring weaknesses in event localization.Source: §7 · p. 9
  • LongRCA Bench provides one finalized role label, one root-cause-step label, and a human-written rationale per trajectory, requiring careful judgment across long executions.The two labels are scored independently; the rationale documents supporting evidence but is not scored.Source: §7 · pp. 9–10
  • The benchmark targets only the responsible role and earliest decisive root-cause step; RCTA’s intermediate causal chain remains unscored, so full causal-chain reconstruction is not measured.Source: §7 · p. 10
  • Diagnosis uses the full log after a trajectory fails, so the evaluation tests neither early warning nor online intervention and is not directly comparable to pre-failure detection protocols.Source: §7 · p. 10
  • All methods share an inference backbone, supporting matched-backbone comparisons, but absolute performance may change with a stronger backbone and does not establish model-independent performance.Source: §7 · p. 10
  • Length and root-to-end-bin trends are descriptive associations, because bins mix source domains with different workflows, role structures, and failure distributions.The figure notes differing source composition across bins; controlled matched-source experiments are needed before attributing causal effects to either variable.Source: §7 · p. 10, Figure 5 · p. 10
  • Follow-up work could ablate candidate recall, candidate-to-handoff tracing, backward instruction checks, and validation, and compare additional attribution methods under matched backbones.Source: §7 · p. 10

8 CONCLUSION

LongRCA Bench evaluates responsible-role attribution and root-cause-step localization in long-horizon agent failures, while RCTA narrows trajectories to candidate errors and checks them against earlier handoff instructions. The results support treating responsibility and root-cause localization as explicit diagnostic targets beyond final-outcome evaluation.

8 CONCLUSION

  • LongRCA Bench contains 1,140 failed trajectories with retained human annotations from five heterogeneous source benchmarks.In many trajectories, the reference root cause is followed by a long sequence of downstream steps before execution ends.Source: §8 · p. 10
  • 51.1% role accuracy and 24.1% exact root-step accuracy make RCTA the strongest evaluated method on both targets under the same backbone, instances, and scoring protocol.RCTA compares candidate error steps with relevant earlier handoff instructions and returns separately validated predictions for each target.Source: §8 · p. 10
  • Final-outcome evaluation alone cannot provide evidence about responsible roles or root-cause localization in completed agent failures.The benchmark and results support treating both as explicit diagnostic targets.Source: §8 · p. 11

A RELEASED DATA SCHEMA · B SOURCE EXECUTION PROVENANCE

The released schema separates prediction-time inputs from withheld reference annotations, while provenance records identify source-specific workflows and generator coverage. The tables clarify which fields methods receive and how to interpret generator entries.

A RELEASED DATA SCHEMA

  • Released records separate model-visible fields from reference annotations withheld during prediction.The schema defines these fields and clarifies their roles in scoring.Source: A RELEASED DATA SCHEMA · p. 12, Table 4 · p. 13
  • Methods receive the question ID and history, including the task instruction; the three reference annotation fields are withheld during prediction.Source: Table 4 · p. 13

B SOURCE EXECUTION PROVENANCE

  • The provenance section records implementation-level workflow names and generator counts for each source.These names complement the broader agent-organization descriptions in the main benchmark summary.Source: B SOURCE EXECUTION PROVENANCE · p. 13, Table 5 · p. 13
  • SWE-bench Pro and Terminal Bench 2 record custom role workflows and generators but no separate third-party scaffold package.Source: Table 5 · p. 13
  • The provenance table’s generator columns correspond to MiniMax-M2.5 (M2.5), Kimi-K2.5 (K2.5), and Qwen3.5-Plus (Q3.5+).Source: Table 5 · p. 13
  • A dash in the provenance table indicates that the source contains no trajectory from that generator.Source: Table 5 · p. 13

C DIFFICULTY-STRATIFIED NUMERIC RESULTS · D ROOT-CAUSE DECISION RULES

The sections provide bin-level root-cause accuracy reporting and define how RCTA selects a single root-cause step in common boundary cases. The selected step and predicted responsible role are separate outputs.

C DIFFICULTY-STRATIFIED NUMERIC RESULTS

D ROOT-CAUSE DECISION RULES

  • When an erroneous decision is already stated in a handoff, select that handoff; when a later step departs from it or introduces a new decisive error, select the later step.Source: D ROOT-CAUSE DECISION RULES · p. 14
  • Exclude an earlier error if it is repaired before evaluator-confirmed failure; a later verifier’s failure to detect an earlier error does not replace the step that introduced it.The missed verification may be retained as unscored explanatory context.Source: D ROOT-CAUSE DECISION RULES · p. 14
  • Select an input step for contradictory requirements only when the task instruction and official evaluator or test requirement are explicitly incompatible and both are cited from the recorded trajectory.Source: D ROOT-CAUSE DECISION RULES · p. 14
  • Among candidates supported by the original log, RCTA selects the earliest recorded step introducing the decisive error relevant to the evaluator-confirmed failure, provided it remains unrepaired through failure.Source: D ROOT-CAUSE DECISION RULES · p. 14
  • Root-cause-step rules govern only that output: RCTA predicts the responsible role separately, so the selected step’s emitter does not determine the predicted role.Source: D ROOT-CAUSE DECISION RULES · p. 14

E RCTA PROMPTS AND IMPLEMENTATION DETAILS · F PIPELINE PSEUDOCODE

RCTA uses local summaries, a trajectory outline, and final attribution to retrieve candidate errors and trace them to earlier handoff instructions. Its implementation combines rule-based segmentation with validation that treats role and root-step predictions independently, while documenting fallback behavior and limits.

E RCTA PROMPTS AND IMPLEMENTATION DETAILS

  • Segmentation uses rule-based boundaries, with verifier PASS or FAIL markers serving only as optional boundary cues—not semantic judgments, benchmark labels, or final outcomes.Segments contain at most 80 non-overlap steps; subsequent segments can receive up to five preceding steps as overlap context.Source: E RCTA PROMPTS AND IMPLEMENTATION… · pp. 14–15
  • RCTA runs local summarization once per segment, then makes one trajectory-outline call and one final-attribution call, for m + 2 initial LLM calls before retries.The stages are local summarization, trajectory-outline aggregation, and final attribution.Source: E RCTA PROMPTS AND IMPLEMENTATION… · p. 14
  • The frozen prompts provide task rules and output-schema guidance without benchmark instances or demonstrations, and predictions receive no reference role, root step, or rationale.All 20 run snapshots for the final 1,140 predictions contain identical prompt hashes.Source: E RCTA PROMPTS AND IMPLEMENTATION… · p. 15
  • The stages use DeepSeek-V4-Flash with different reasoning settings and default output limits of 4,000, 6,000, and 8,000 tokens, respectively.Truncated final responses are retried with a doubled limit; 11 trajectories used a recorded 16,000-token final limit with a 32,000-token retry.Source: E RCTA PROMPTS AND IMPLEMENTATION… · p. 15
  • If phase-construction output cannot be parsed, the implementation assigns one phase per local summary; 308 of 1,140 outputs fall outside the requested 3–8 phase range.The artifacts do not retain a fallback flag, so 308 is not an exact count of parse failures.Source: E RCTA PROMPTS AND IMPLEMENTATION… · p. 15
  • Role and root fields are validated independently: roles and root steps must occur in the trajectory, and an invalid field does not overwrite a valid one.ENVIRONMENT cannot be a scored role, and system evaluation may appear only as an unscored terminal node in the explanatory failure chain.Source: E RCTA PROMPTS AND IMPLEMENTATION… · p. 15, F PIPELINE PSEUDOCODE · p. 16

F PIPELINE PSEUDOCODE

  • RCTA retrieves source steps from summaries and phases, then adds the nearest earlier handoff instruction for executor or verifier candidates when available.For upstream attribution, a textual provenance check matches the cited text to the logged instruction but does not verify that it contains the same causal error.Source: E RCTA PROMPTS AND IMPLEMENTATION… · p. 15, F PIPELINE PSEUDOCODE · p. 16
  • The inference pipeline retries once with validation feedback when fields are invalid, preserves valid fields, and flags for review or abstains when essential fields remain missing.Role-name-triggered planner–executor recall rules did not affect the reported results because no benchmark trajectory matched their configured signature.Source: E RCTA PROMPTS AND IMPLEMENTATION… · pp. 15–16, F PIPELINE PSEUDOCODE · p. 16

Related papers

Loading 2608.15242v1…