PaperlayerView exact revision on arXiv

Source-linked AI summary

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

Harsh Raj, David Lee, Anas Mahmoud, Renxiong Wang, Razvan-Gabriel Dumitru, Chenguang Wang, Tong Zhao, Yunzhong He, Darvin Yi, Vipul Gupta

arXiv:2609.13463v2cs.AIcs.HCcs.LGcs.SE

TL;DR

Root-cause attribution turns failure outcomes into actionable diagnostic feedback, but long execution records can leave relevant evidence unexplored. Continual Search prompts judges to keep searching across turns; it improves attribution across long-horizon benchmarks, including Opus-4.8 on MegaRCA-Mix.

  • Problem

    Root-cause attribution must locate failure mechanisms in execution records because outcome-level signals alone offer little insight into where to intervene.

    Source: §1 · p. 2

  • Method

    Continual Search iteratively nudges an LLM judge to examine unresolved or previously unexamined diagnostic evidence.

    Source: §1 · p. 3

  • Results

    Across three datasets, Continual Search yields the highest final attribution scores for both judges except GPT-5.5 on MegaRCA-Mix, where it ties Passive Continuation; Opus-4.8 improves from 0.471 F1 to 0.608.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    The authors characterize long-horizon RCA as primarily search-limited; at the same reasoning effort, Sonnet-5 and Fable-5 reach comparable performance using Continual Search.

    Source: §1 · p. 4

    Attribution is limited to the observable record, since a causal failure may occur in unobserved thinking tokens and become visible only later.

    Source: §7 · p. 14

Abstract

from arXiv · show

The increasing deployment of AI agents in long-horizon tasks yields massive execution logs. Diagnosing failures within these records is crucial for reliability, as it transforms outcome-level signals into actionable interventions. The sheer scale of the data renders human review impractical, driving the need for automated root-cause attribution (RCA). However, automated RCA methods using LLMs suffer from low diagnostic accuracy, especially as execution traces grow larger. They struggle because relevant information is often sparse, distributed across distant actions, and disconnected from the visible failure, reducing root-cause attribution to a massive search problem. Existing RCA methods typically rely on one-shot LLM judgments to diagnose failures from execution traces. While effective for shorter trajectories, these judges tend to settle on a plausible diagnosis early, leaving critical evidence in longer traces unexamined. We introduce Continual Search, an iterative framework that nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence. We evaluate Continual Search across four existing RCA benchmarks. Recognizing the lack of massive execution traces in current benchmarks, we introduce MegaRCA-Mix to evaluate RCA at scale. MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks. Across multiple benchmark suites and model families, Continual Search consistently improves attribution performance. On MegaRCA-Mix, for example, it improves Opus-4.8's F1 score by 29\%, from $0.471$ to $0.608$. More interestingly, within the same model family, lower-tier models can even surpass their higher-tier counterparts, demonstrating that effective search supersedes raw model scale.

1 Introduction

Long-horizon agent failures leave sparse, distant evidence that outcome-level signals alone cannot explain, making root-cause attribution a large-scale search problem. Continual Search prompts judges to keep searching across turns, improving attribution on long-log benchmarks and motivating MegaRCA-Mix as a larger testbed.

1 Introduction

  • Execution logs can reach millions of tokens, with sparse evidence distributed far from visible failures, making RCA a search over plausible explanations and contradictions.Misattributions can incur substantial costs or trigger unnecessary interventions.Source: §1 · p. 3
  • On shorter AgentRx and Who&When trajectories, additional turns add little evidence and can flip an initial correct prediction toward an incorrect label.The initial attribution already covers most of the available evidence in these shorter settings.Source: §1 · p. 3
  • On MegaRCA-Mix, Continual Search raises Opus-4.8 F1 from 0.471 to 0.608, compared with 0.479 under Passive Continuation.The benchmark contains 50 human-annotated failure trials with a median execution size of 286K tokens.Source: §1 · p. 3
  • MegaRCA-Mix extends evaluation to 50 human-annotated failures from Harbor Index, with a median execution size of 286K tokens.It includes the full evaluation record beyond the agent trajectory.Source: §1 · pp. 3–4
  • On long-log benchmarks, Continual Search improves attribution by prompting judges to search unresolved or previously unexamined evidence across turns.The framework matches or outperforms evaluated attribution methods, with monotonic gains across turns on benchmarks with large execution logs.Source: §1 · p. 3
  • At the same reasoning-effort setting, Sonnet-5 and Fable-5 reach comparable performance using Continual Search.Experiments across five judge models and multiple reasoning-effort settings indicate that long-horizon RCA is primarily search-limited.Source: §1 · p. 4

2 Related Work

Prior RCA work localizes failures across components, steps, and agents, but many datasets use trajectories that fit within modern context windows. Related research also points to evidence retrieval and the risks of repeated judgment as central concerns.

Root-cause attribution for agent failures

  • RCA benchmarks target component interactions and fault sides (Raj et al., 2026), decisive steps and agents (Zhang et al., 2025), or critical steps and root-cause categories (Barke et al., 2026).Who&When Pro broadens evaluation to more than 12K labeled trajectories across frameworks, domains, and modalities (Liu et al., 2026).Source: §2 · p. 4
  • Many existing datasets remain small enough for modern LLMs to ingest, including Who&When Pro’s 7.5-step average and AgentRx’s 4.9K–16.5K-token averages.Who&When Pro’s longest reported analysis bin starts above 12K tokens.Source: §2 · p. 4

Evidence search in long-horizon trajectories

Agent-as-a-Judge

3 Method

Continual Search treats root-cause attribution as iterative evidence search, contrasting continued exploration with passive reconsideration. The evaluation spans five RCA settings, with benchmark-native tasks and execution records ranging from short trajectories to massive logs.

3.1 Continual Search

  • Both continuation conditions branch from the same turn 1 prediction and judge configuration, while only their subsequent-turn instructions differ.The method extends the benchmark’s native single-turn RCA analysis in an iterative multi-turn setting.Source: §3.1 · p. 5
  • Continual Search prompts the judge to challenge its current diagnosis and examine unresolved evidence, whereas Passive Continuation asks it only to reconsider and reverify.A decision tree assembles turns 2–4 from generic sentences based on evidence storage and answer-scoring properties, without benchmark-specific prompts.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2 Evaluation Setting

  • Evaluation covers five settings—MegaRCA-Mix, TRAIL, TELBench, AgentRx, and Who&When—while retaining each benchmark’s native task formulation and metrics.The settings span taxonomy classification, error-span and step localization, and multi-agent attribution.Source: §3.2 · p. 6
  • MegaRCA-Mix contains 50 Harbor Index failure trials annotated with first consequential errors and provides the complete evaluation record, including logs, verifier outputs, and sandbox artifacts.Its median execution size is 286K tokens, compared with 100K for TRAIL, 39K for TELBench, 7.2K for AgentRx, and 2.4K for Who&When.Source: §3.2 · p. 6
  • Tool-enabled judges selectively inspect evaluation artifacts on demand rather than receiving massive traces directly in the context window.Inlining those traces risks information saturation and context-limit exhaustion.Source: §3.2 · p. 7

4 Results

Continual Search improves attribution across the largest-log benchmarks, while its gains depend on unread evidence. Results also show that lower-tier models can approach or exceed higher-tier models, and that short trajectories may regress.

4.1 Continual Search Improves Long-Horizon Attribution

  • Across MegaRCA-Mix, TRAIL, and TELBench, Continual Search yielded the highest final scores for both judges except GPT-5.5 on MegaRCA-Mix, where it tied Passive Continuation.Opus-4.8 rose from 0.471 to 0.608 F1 on MegaRCA-Mix; Table 2 compares initial single-turn and Turn 4 results.Source: Table 2 · p. 7, §4.1 · p. 7
  • Continual Search expanded its evidence base and reasoning across turns as attribution improved, while Passive Continuation saturated after the initial attribution.The authors report this pattern across MegaRCA-Mix, TELBench, and TRAIL.Source: §4.1 · p. 7, Figure 3 · p. 8

4.2 Varying Model Scale and Reasoning Effort

  • Sonnet-5 reached 0.505 Turn 4 Weighted F1, comparable to Fable-5’s 0.498 at about 64% of its cost; Opus-4.8 did slightly better than Fable-5 at somewhat lower cost.Continual Search improved every judge over its turn-1 attribution.Source: §4.2 · p. 9, Figure 4 · p. 9
  • Across all tested reasoning configurations, Turn 4 Weighted F1 exceeded Turn 1, but reasoning-effort effects showed no consistent pattern.For Opus-4.8, max effort reached 0.543 versus about 0.505 for low and high effort; for Sonnet-5, max was slightly below high.Source: §4.2 · p. 9, Figure 5 · p. 10

4.3 Limited Benefit on Short Execution trajectories

  • Continual Search’s benefit depends on unread evidence; when the initial attribution covers most available evidence, later prompting can act as conversational pressure rather than discovery.The authors link this setting to judge verdict instability and drift from ground truth.Source: §4.3 · pp. 10–11
  • On short AgentRx and Who&When trajectories, Continual Search lacked consistent gains; GPT-5.5 fell from 0.310 to 0.241 on AgentRx and from 0.516 to 0.437 on Who&When.Opus-4.8 also regressed on both benchmarks, and the initial turn typically processed almost the entire execution log.Source: §4.3 · p. 10, Table 3 · p. 10

5 Ablations

Across the ablations, Continual Search outperforms independent resampling and performs best when longer execution records leave evidence to explore. Its performance gains coincide with broader acquisition of previously unseen evidence.

5.1 Continual Search Outperforms Independent Resampling

  • Opus-4.8 reaches 0.543 Weighted F1 with Continual Search versus 0.430 with self-consistency, while GPT-5.5 reaches 0.494 versus 0.383.Self-consistency also regresses below one-shot attribution for both judges, despite using a larger inference budget than Continual Search.Source: §5.1 · p. 11, Table 4 · p. 11

5.2 Effect of Execution-Log Size

  • Continual Search stays above single-turn attribution in every execution-log-size tercile except the largest on MegaRCA-Mix and TELBench, where the results coincide.At the highest tercile, median log size is 3.40 MB on MegaRCA-Mix and 2.60 MB on TRAIL; no similar decline appears on shorter-log AgentRx and Who&When.Source: §5.2 · p. 12, Figure 6 · p. 12

5.3 Evidence Acquisition Across Turns

  • Across MegaRCA-Mix, TRAIL, and TELBench, attribution gains coincide with broader searches over the execution record, as Continual Search acquires more unseen evidence than Passive Continuation.On MegaRCA-Mix, Passive Continuation rapidly saturates while Continual Search continues acquiring unseen evidence.Source: §5.3 · pp. 12–13, Figure 7 · p. 13

6 Using Continual Search for Benchmark Diagnosis

On MegaRCA-Mix, Continual Search lets a taxonomy classifier diagnose benchmark-side defects more effectively than the specialized ABA baseline. Its first continuation turn raises F1, driven by increased recall.

6 Using Continual Search for Benchmark Diagnosis

  • The evaluation tests four benchmark-side defect categories from the interaction-centric taxonomy against Automated Benchmark Analysis, a specialized auditing framework.The four categories are Instruction-Grader Mismatch, Service Failure, Stale State Delivery, and Mistranslation.Source: §6 · p. 13
  • The taxonomy classifier’s F1 rises from 0.10 at turn 1 to 0.67 at turn 2, surpassing ABA’s 0.63 and remaining at 0.67 through turn 4.Both methods receive identical access to complete MegaRCA-Mix evaluation records, including trajectories, verifier outputs, and environment artifacts.Source: §6 · p. 13, Figure 8 · p. 14
  • The classifier’s recall rises from 0.05 at turn 1 to 0.50 at turn 2, while precision stays at 1.00.The reported gains come at a higher inference cost than ABA.Source: §6 · p. 13, Figure 8 · p. 14

7 Limitations

The analysis is bounded by incomplete visibility into agent execution and by challenges in evaluating attribution. Proprietary-model thinking tokens may be hidden, and taxonomy overlap and noisy human labels may affect measured performance.

7 Limitations

  • Because proprietary models may hide thinking tokens, the causal failure can occur in unobserved steps and become visible only later; results therefore concern attribution over the observable record.Source: §7 · p. 14
  • The authors suspect overlapping failure-taxonomy categories could artificially lower classification scores.Source: §7 · p. 14
  • Establishing reliable human ground truth over massive execution logs is inherently difficult and may introduce label noise.Source: §7 · p. 14

A.1 Artifact-Level Evidence Coverage

Continual Search expands artifact coverage over successive turns while Passive Continuation quickly plateaus. Previously unread auxiliary files can reveal evidence that corrects trajectory-based misattributions.

A.1 Artifact-Level Evidence Coverage

  • Both strategies start at 70.5% evidence coverage at turn 1; Continual Search reaches 95.2% by turn 3 and 98.5% by turn 4, while Passive Continuation plateaus at 75.6%.Source: §A.1 · p. 19, Figure 9 · p. 19
  • Although the judge inspects the primary trajectory in 38 of 50 trials by turn 2, it typically accesses secondary files such as system logs and environment configurations only later.Source: §A.1 · p. 19
  • In a GAIA2 example, inspecting config.json and the console log reveals that the environment failed to respond during the wait window, correcting an initial diagnosis of model Satisficing to Stale State Delivery.Source: §A.1 · p. 19, Figure 10 · p. 20

A.2 Stability Across Independent Judge Runs

Across four independent Opus-4.8 runs on TRAIL, Continual Search outperforms Passive Continuation under both metrics. Its gains remain positive despite slight variation in single-turn baseline accuracy.

A.2 Stability Across Independent Judge Runs

  • The stability test independently resamples the Opus-4.8 judge from the initial turn onward for three additional TRAIL evaluations.Source: §A.2 · p. 20
  • Across all four runs, Continual Search outperforms Passive Continuation under both metrics, with Joint Accuracy gains ranging from 0.037 to 0.055.The four runs use independently sampled Opus-4.8 judges on TRAIL with n = 148.Source: §A.2 · p. 20, Table 5 · p. 20
  • Single-turn baseline accuracy fluctuates from 0.129 to 0.153, while Continual Search’s relative advantage remains positive across runs.Source: §A.2 · p. 21

B Judge Configuration and Prompts

This appendix describes the judge configurations, initial and continuation prompts, and benchmark-native scoring rules used to evaluate evidence-grounded diagnosis of long traces. It covers procedures for locating, enumerating, and adjudicating failures, with a shared continuation-search approach adapted to each benchmark’s evidence format and answer type.

B.1 Judge Configuration

  • The primary judges are GPT-5.5 at xhigh reasoning effort and Opus-4.8 with adaptive thinking at max effort; the panel also includes Opus-4.7 and Grok-4.6.All are invoked through the Claude Agent SDK and have access to Bash, Grep, Glob, Write, Edit, and Task.Source: §B.1 · p. 21

B.2 Judge Prompts

B.2.1 Turn 1 Prompts

  • MegaRCA-Mix prompts direct judges to locate the first causal divergence, scrutinize plausible conclusions, and retain non-model faults even when the agent also errs.The governing rule says the terminal error is a symptom, and a broken environment, grader, or tool remains a non-model fault.Source: §B.2.1 · p. 21
  • TELBench uses a bare, single-call prompt that requires reading every ordered span and sparsely marking committed harmful mistakes, not harmless exploration or isolated tool failures.When harm appears only in the final report, the prompt says to mark that span alone; it also specifies how to handle an earlier harmful commitment and its later reliance.Source: §B.2.1 · p. 24
  • AgentRx’s first turn enumerates every plausible candidate failure across the trajectory, quoting the action and contradicted evidence while leaving resolution and root-cause selection to later turns.Judges receive the trajectory and ground-truth action sequence, and identify the failure category and step index.Source: §B.2.1 · pp. 24–25
  • AgentRx’s checklist distinguishes planning deviations, invalid tool invocations, tool-output misinterpretation, intent misalignment, and system connectivity failures.Its taxonomy also includes other categories, with separate checks for whether evidence supports each failure type.Source: §B.2.1 · pp. 24–25
  • AgentRx’s root-cause procedure scans from the beginning, continues past resolved failures, and labels the first failure that remains unresolved as the run’s root cause.The prompt asks judges to record each first failure and check later trajectory evidence for resolution before proceeding.Source: §B.2.1 · p. 25
  • The turn-1 JSON output records the step and taxonomy category alongside verbatim commitment and contradiction quotes, resolution status and evidence, terminal connection, and tracking fields.Source: §B.2.1 · pp. 25–26
  • Who&When’s published all-at-once template asks the judge to identify the responsible agent and first mistake after reading the full conversation, including every chunk and explicit step number.The prompt warns that the first mistake may be early and quiet.Source: §B.2.1 · p. 26

B.2.2 Continuation Prompts

  • A fixed decision tree assembles turns 2–4 from generic prompt sentences, requiring only each benchmark’s evidence format and answer type and yielding prompts with 7% average character-level similarity.The tree is shown in Figure 11.Source: §B.2.2 · p. 26, Figure 11 · p. 27
  • For MegaRCA-Mix and TELBench, later turns open unread evidence; TRAIL sweeps less-covered trace regions, while AgentRx and Who&When test candidates and retain the earliest unfixed cause linked to failure.Single-item labels may change only to an earlier cause supported by an exact quote; TRAIL prompts preserve earlier findings unless given a specific reason to drop them.Source: §B.2.2 · p. 27
  • AgentRx and Who&When continuation checks assess whether each candidate was later fixed and whether an unbroken causal path still leads to the final failure, then verify the earliest qualifying candidate.The prompts require quoted evidence for both checks.Source: §B.2.2 · p. 29

B.3 Evaluation Metrics

  • MegaRCA-Mix scores each prediction by its best F1 match among acceptable gold cluster sets, then averages those trial scores.This set-based metric accommodates failures with multiple valid root-cause categories.Source: §B.3 · p. 30
  • TRAIL reports Joint Accuracy as recall over gold location–category pairs and Weighted F1 over error categories; Joint Accuracy does not penalize extra incorrect pairs.Source: §B.3 · pp. 30–31
  • TELBench uses First-Error Accuracy (FEA), while AgentRx and Who&When use Critical Step-index Accuracy and Step-Level Accuracy, respectively; all require an exact span or step-index match.Source: §B.3 · p. 31
Loading 2609.13463v2…