PaperlayerView exact revision on arXiv

Source-linked AI summary

TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems

Minxing Wang, Xiaofei Xie, Yintong Huo

arXiv:2605.26563v3cs.SE

TL;DR

Long, noisy repository-level coding trajectories make automated failure diagnosis difficult. TrajAudit combines trajectory folding and test-failure-based guidance with an investigator that retrieves details on demand, and RootSE evaluates the approach on annotated coding-task failures. On RootSE, TrajAudit reports up to 21.6% improvement in exact step-level accuracy over the strongest baseline.

  • Problem

    Repository-level coding trajectories are long and contain substantial low-signal observations, challenging automated diagnosis of where and why agents fail.

    Source: §I · p. 1

  • Method

    TrajAudit uses test-failure-based prior reasoning and semantic saliency folding to guide an investigator agent that can retrieve trajectory details on demand.

    Source: §I · pp. 1–2

  • Results

    Up to 21.6% improvement in exact step-level accuracy over the strongest baseline is reported on RootSE.

    Source: §I · p. 2

  • Takeaways & Limitations

    RootSE provides a benchmark of 102 failed repository-level coding trajectories with earliest-error annotations, and TrajAudit consistently outperforms the baselines on it.

    Source: §I · p. 2, §X · p. 10

    RootSE excludes cases primarily attributable to ambiguous task descriptions and currently covers bug fixing and feature implementation, limiting generalizability to other task types and external-condition failures.

    Source: §IX · p. 10

Abstract

from arXiv · show

Agentic systems have been widely studied to automate coding tasks such as bug fixing and feature implementation. As these systems increasingly operate on complex codebases, understanding where and why they fail becomes essential for iterative refinement and operational reliability. Existing automated failure diagnosis approaches leverage \textit{task execution trajectories}, yet they struggle with trajectories produced by repository-level coding agents due to two key properties. First, these trajectories are often long, spanning many execution steps, making it difficult for LLMs to track the causal chain of failure over the execution history. Second, these trajectories are laden with noise, containing substantial low-signal observations such as redundant program structures and verbose code context, which can interfere with LLM reasoning. To address these challenges, we propose \textit{TrajAudit}, an automated failure diagnosis framework specifically for trajectories produced by repository-level coding agents. TrajAudit employs an investigator agent supported by two modules: one reduces failure-irrelevant noisy context through semantic saliency folding, and the other derives preliminary diagnostic guidance from test failure reports as prior knowledge to help LLMs focus on likely failure regions. The investigator agent can further invoke tools to inspect folded content on demand, enabling a focused investigation without losing access to the full trajectory context. We also introduce \textit{RootSE}, a benchmark of 102 real-world instances from repository-level coding tasks, each annotated with the earliest decisive error step and a justification. Experiments on RootSE show that TrajAudit outperforms the strongest baselines by 10.8\% and 21.6\% in exact failure localization accuracy in the with- and without-reference settings, respectively, demonstrating its effectiveness.

I. INTRODUCTION

Repository-level coding agents can fail after long, noisy executions, making it important to identify where and why they fail. TrajAudit guides an investigator through those trajectories, and RootSE provides a benchmark for evaluating earliest-error diagnosis.

I. INTRODUCTION

  • Trajectories often span 20 to over 100 steps, average over 5,000 characters per step, and contain tool observations that are mostly irrelevant to diagnosis.Observations can account for over 70% of trajectory content.Source: §I · p. 1
  • Existing diagnosis methods passively consume entire trajectories, treating all steps as equally relevant, while early mistakes can contribute to opaque failures on complex, multi-file tasks.Understanding where and why agents fail is identified as critical for iterative refinement and trustworthy intelligent software engineering.Source: §I · p. 1
  • TrajAudit combines test-error-based prior failure reasoning and semantic saliency folding with an investigator that retrieves folded content on demand.The prior diagnosis focuses attention on suspicious regions; folding retains failure-relevant context, and interactive retrieval supports top-down investigation.Source: §I · pp. 1–2
  • Up to 21.6% improvement in exact step-level accuracy over the strongest baseline is reported on RootSE.RootSE contains 102 real-world instances annotated with earliest decisive error steps and diagnostic justifications; the paper also reports releasing its artifacts.Source: §I · p. 2

II. PRELIMINARIES

The paper models coding-agent execution as a sequence of interactions with a software environment and defines diagnosis as locating and explaining the earliest decisive error. These definitions establish the task and its trajectory representation.

II. PRELIMINARIES

  • The preliminaries formally define execution trajectories and the failure-diagnosis task as foundations for the benchmark and methods.Source: §II · p. 2

A. Execution Trajectory

  • An execution trajectory comprises system states and agent-environment interaction steps, with each step containing Thought, Response, Action, and Observation.Task success is determined by whether all acceptance tests pass.Source: A. Execution Trajectory · p. 2

B. Failure Diagnosis

  • Failure diagnosis takes a task description, failed trajectory, test code, and test failure output, and returns the first decisive-error step with a natural-language explanation.A decisive error is an interaction whose replacement, with preceding steps unchanged, can yield a successful trajectory; the earliest such interaction is the target.Source: B. Failure Diagnosis · p. 3

1) LLM-based:

Prior approaches diagnose agent failures through LLM inspection, repeated-run spectrum analysis, or learned models, while existing benchmarks differ from repository-level coding trajectories. RootSE is introduced to evaluate earliest-error diagnosis in that setting.

1) LLM-based:

  • LLM-based methods inspect full trajectories using single-pass, sequential, or binary-search strategies, but long coding traces can exceed context windows and noisy tool outputs can interfere with diagnosis.TRAIL prompts with an error taxonomy to locate all failed steps; these approaches can be effective on shorter traces such as web browsing.Source: 1) LLM-based · p. 3, B. Failure Diagnosis Benchmarks for… · p. 3

2) Spectrum-based:

  • FAMAS re-executes tasks, clusters normalized behavior triples, and ranks groups by frequency-based suspiciousness, but repeated runs are costly and action triples omit reasoning traces.The cited work reports FAMAS outperforming prior purely LLM-based methods on general-purpose task failure localization.Source: 2) Spectrum-based · p. 3

3) Learning-based:

  • AgenTracer fine-tunes a lightweight model with reinforcement learning on annotated failure trajectories, but requires training a dedicated model for new settings.Source: 3) Learning-based · p. 3

B. Failure Diagnosis Benchmarks for Agentic Systems

  • Who&When and TRAIL include general-purpose tasks or mostly information-retrieval trajectories, whereas RootSE targets earliest crucial errors in repository-level coding tasks.TRAIL labels all plausible erroneous steps; the paper identifies no existing benchmark for earliest-step diagnosis in the coding setting.Source: B. Failure Diagnosis Benchmarks for… · p. 3, §IV · p. 3

IV. THE ROOTSE BENCHMARK

  • RootSE comprises 102 failed instances from four representative agents, annotated with earliest decisive error steps and justifications, spanning over 5,000 steps and approximately 30 million characters.The benchmark is characterized by long-horizon execution and substantial observation content.Source: §IV · p. 3

A. Task Description

  • RootSE takes the task description, failure trajectory, test code, and error messages as input, and evaluates exact step accuracy, tolerated step accuracy, and justification accuracy.Justification accuracy is assessed by an LLM-as-a-judge for semantic equivalence to the ground truth.Source: A. Task Description · pp. 3–4

B. Data Collection

RootSE draws on four single-agent coding systems and repository-level tasks selected for real-world origins, executable tests, and multi-step reasoning. Its 102 trajectories were retained through staged filtering to support a single earliest-error annotation per instance.

1) Agentic System Selection:

  • RootSE uses four single-agent systems; this scope targets failures from an agent’s own reasoning and actions while allowing individual-agent diagnosis in multi-agent settings.The systems are SWE-agent, Live-SWE-agent, OpenHands, and AutoCodeRover.Source: 1) Agentic System Selection · p. 4

2) Task Selection:

  • Tasks come from real repository issues, have executable test suites, and require multi-step code reasoning and execution.The task sources are SWE-bench, SWE-bench Pro, and SWE-rebench.Source: 2) Task Selection · p. 4

3) Trajectory Generation:

  • 500 candidate trajectories were filtered to 102 by removing ambiguous tasks, mismatched test requirements, and cases with multiple failing tests.The resulting benchmark spans 35 repositories, three programming languages, and four agent architectures.Source: 3) Trajectory Generation · p. 4

C. Annotation Procedure

RootSE annotations identify an earliest decisive error step and justify how it leads to task failure, with disagreements resolved through expert review. Benchmark analysis maps these errors to coding phases and finds failures across the main phases and environment interaction.

C. Annotation Procedure

  • Two experts independently annotated each instance using shared criteria, then discussed inconsistencies; a third expert arbitrated unresolved cases.Each annotation contains the earliest decisive error step and a justification linking it to task failure.Source: C. Annotation Procedure · pp. 4–5
  • 30 of 30 counterfactual corrections led to successful task completion, supporting that the annotated step lies on a causal path, though not proving it is the unique cause.Annotators reported Cohen’s Kappa of 0.78 for earliest decisive error step identification.Source: C. Annotation Procedure · p. 5

1) Failure Phase Distribution Analysis:

  • Localization accounts for 41.2% (42 instances), patch generation 42.1% (43), verification 10.8% (11), and environment interaction 5.9% (6).Instances are mapped by the purpose of the annotated error step, with environment interaction treated as an orthogonal category.Source: 1) Failure Phase Distribution Analysis · p. 5, Figure 4 · p. 5

2) Complexity and Diversity Analysis:

RootSE combines complex, diverse coding trajectories with a diagnosis workflow that uses test failures to guide investigation and folds irrelevant observations while keeping them retrievable. The investigator integrates these aids to focus analysis without losing access to the full trajectory.

2) Complexity and Diversity Analysis:

A. Overview

  • The workflow combines the investigator with prior failure reasoning and semantic saliency folding, and allows on-demand access to folded trajectory content.The investigator integrates both module outputs and dynamically retrieves folded content during a focused investigation.Source: A. Overview · p. 6, Figure 5 · p. 6

B. Prior Failure Reasoning

  • Prior failure reasoning uses test code and error descriptions to identify a likely workflow phase and rationale before the investigator processes the full trajectory.The preliminary diagnosis is not constrained to a predefined set of phases.Source: B. Prior Failure Reasoning · p. 6

C. Semantic Saliency Folding

  • Semantic saliency folding preserves code patches and observations with failure-indicative keywords, replacing other observations with markers whose original content can be retrieved.Patch patterns and a failure-indicator dictionary identify observations to preserve; other observations are folded.Source: C. Semantic Saliency Folding · p. 6

D. Investigator Agent

The investigator integrates the processed module outputs and decides whether it has enough context to diagnose the failure. If not, it retrieves folded observations iteratively before submitting a step and justification.

D. Investigator Agent

  • The investigator combines module outputs and dynamically probes folded observations when needed to make the final diagnosis.Source: D. Investigator Agent · p. 6
  • It first assesses whether the current context suffices; if so, it outputs the failure step index and justification directly.Source: D. Investigator Agent · p. 6
  • When context is insufficient, the agent uses tools to inspect folded observations and expands available context until it can diagnose.One tool retrieves a folded observation at a specific step; another submits the final diagnostic result.Source: D. Investigator Agent · p. 7

VI. EXPERIMENTAL SETUP

The experiments evaluate TrajAudit on RootSE against five baselines, under settings with and without a representative reference patch. The benchmark contains 102 failed repository-coding instances annotated with decisive error steps and justifications.

VI. EXPERIMENTAL SETUP

  • The experiments ask about effectiveness, localization effectiveness versus token cost, backbone robustness, and component contributions.These are the four stated research questions.Source: §VI · p. 7

A. Dataset

  • RootSE contributes 102 failed repository-coding instances, each annotated with its earliest decisive error step and a diagnosis justification.The instances come from representative agentic systems that failed real-world coding issues.Source: A. Dataset · p. 7

B. Baselines

  • Five baselines are compared; methods exceeding an LLM context limit are counted as diagnosis failures.The baselines are All-at-Once, Step-by-Step, Binary Search, TRAIL, and FAMAS; AgenTracer is excluded because its trained parameters are unavailable.Source: B. Baselines · p. 7

C. Evaluation Settings

  • The with-reference setting supplies one representative patch because repository tasks allow multiple valid solutions, while the without-reference setting relies only on execution logs.The two settings represent diagnosis with known reference solutions and practical deployment without one.Source: C. Evaluation Settings · p. 7

D. Metrics

The evaluation uses step-level accuracy under tolerance windows, justification accuracy, and a fixed default LLM setup. In exact localization, TrajAudit beats TRAIL in both reference settings, with a significant paired advantage without reference.

D. Metrics

  • Tol@k denotes step-level accuracy under tolerance k; unless specified otherwise, Tol@0 in the without-reference setting is the primary metric, and the LLM-as-a-judge agrees with consensus labels at 93.3%.Two annotators initially agreed on 97.2% of sampled judgments; Claude Sonnet 4.51 is the default backbone with temperature set to 0.Source: D. Metrics · p. 7, E. Implementation Details · p. 7

1) Exact Step-level Accuracy:

  • 10.8 percentage points with reference (49.0% vs. 38.2%) and 21.6 percentage points without reference (52.0% vs. 30.4%): TrajAudit exceeds TRAIL in exact step-level accuracy.The result is reported on RootSE; Table II presents exact step-level accuracy.Source: 1) Exact Step-level Accuracy · p. 8, Table II · p. 8
  • p < 0.001 in each paired comparison: TrajAudit significantly outperforms all five baselines without reference.The counts of instances diagnosed correctly only by TrajAudit exceed those correct only by each baseline.Source: 1) Exact Step-level Accuracy · p. 8, Table III · p. 8
  • Reference patches help full-trajectory methods, whereas TrajAudit and FAMAS perform better without reference; the paper attributes this difference to focused views reducing reliance on external anchoring.The authors note that a reference patch may bias diagnosis toward patch-related changes and away from earlier planning or exploration errors.Source: 1) Exact Step-level Accuracy · p. 8

2) Tolerated Step-level Accuracy:

TrajAudit leads across relaxed step-localization tolerances and trajectory-length buckets, and also achieves the highest justification accuracy in the reported comparison.

2) Tolerated Step-level Accuracy:

  • 65.7% at ±3: TrajAudit exceeds TRAIL by 15.7 percentage points and FAMAS by 13.7 percentage points in without-reference tolerated accuracy.All methods improve as tolerance increases, but TrajAudit has the highest accuracy at every reported tolerance level.Source: 2) Tolerated Step-level Accuracy · p. 8

3) Performance across Varying Trajectory Lengths:

  • 46.7% on 53–70-step trajectories and 55.0% on trajectories with at least 71 steps: TrajAudit leads the best baseline in both buckets.The corresponding best-baseline accuracies are 30.0% and 35.0%; the result is for the without-reference setting.Source: 3) Performance across Varying… · p. 8, Figure 6 · p. 8

4) Justification Accuracy:

  • 52.9% justification accuracy: TrajAudit exceeds the best baseline, All-at-Once, at 42.4%.Binary Search is excluded because it outputs region labels rather than diagnostic justifications; Step-by-Step reaches 14.7%.Source: 4) Justification Accuracy · p. 9, Table V · p. 9

B. RQ2. How does TrajAudit balance localization effective-

TrajAudit sits at the high-accuracy end of the observed cost–effectiveness frontier, remains effective across backbones, and benefits from both supporting modules. Backbone choice and module removal nevertheless affect performance and token use.

ness and token cost compared with baselines?

  • 52.0% accuracy at 138K tokens per case: TrajAudit lies on the observed Pareto frontier with FAMAS, and strictly dominates Binary Search and Step-by-Step.Compared with FAMAS at 47K tokens and 30.4% accuracy, TrajAudit uses more tokens but improves accuracy by 21.6 percentage points; Figure 7 compares exact accuracy with average API token consumption.Source: ness and token cost compared with… · p. 9, Figure 7 · p. 9

C. RQ3. How robust is TrajAudit across different backbones?

  • Across five backbones, TrajAudit reaches 40.2–55.9% Tol@0 and 61.8–70.6% Tol@3, while the strongest-to-weakest Tol@0 gap is 15.7 percentage points.Localization and explanation quality do not fully align: Gemini 3.1 Pro has the best localization, while Claude Sonnet 4.5 has the highest justification accuracy at 52.9%.Source: C. RQ3. How robust is TrajAudit across… · p. 9, Table VI · p. 9

D. RQ4. How does each component contribute to TrajAudit?

  • Removing PFR decreases exact step-level accuracy by 17.7% for only 12.7K fewer tokens per case; removing SSF saves 52.0K tokens but reduces exact accuracy by 13.8% and justification accuracy by 16.9%.SSF folds observations from 94.6% of steps on average; only 20.2% of folded steps are later expanded on demand, leaving over 75% of all steps never fully expanded.Source: D. RQ4. How does each component… · pp. 9–10, Table VII · p. 9

VIII. TRAJAUDIT ERROR ANALYSIS

TrajAudit localizes early errors less reliably, and its mislocalized predictions tend to fall later than the annotated error. The paper also reports validity boundaries for RootSE and concludes that TrajAudit outperforms baselines on that benchmark.

VIII. TRAJAUDIT ERROR ANALYSIS

  • For errors in the first 20% of steps, accuracy is 19%–26% across backbones, versus 54%–57% for errors in the middle 35%–65%.Early errors lack an observable transition from correct to incorrect behavior; middle errors offer a contrast between preceding correct and subsequent erroneous steps.Source: §VIII · p. 10
  • Two recurring causes of late localization are implicit flawed plans before code changes and early code errors whose effects accumulate silently.TrajAudit may select the later step where the plan becomes externally observable or the effects of the code error become most evident.Source: §VIII · p. 10
  • Successful localization typically involves a distinct shift from correct exploration to an identifiable erroneous decision that directly causes failure.The paper identifies early planning errors with much later observable consequences as a target for stronger reasoning-level localization.Source: §VIII · p. 10
  • 81% of mislocalized predictions are later than the annotated step, with an average delay of 14.5 steps.Figure 8 shows the distribution of prediction offsets among mislocalized cases.Source: Figure 8 · p. 10, §VIII · p. 10

IX. THREATS TO VALIDITY

  • To reduce internal-validity threats, the authors used a unified annotation guideline, independent annotation by two annotators, third-party arbitration, and temperature 0 for LLM methods and evaluators.Temperature 0 was used to reduce sampling variance.Source: §IX · p. 10
  • RootSE’s 102 instances exclude failures primarily attributable to external factors and cover bug fixing and feature implementation, not tasks such as test generation.The authors note that filtering may bias the dataset away from ambiguity-driven failures and that the task coverage may limit generalizability.Source: §IX · p. 10

X. CONCLUSION

  • The paper concludes that TrajAudit combines semantic saliency folding with prior failure reasoning and consistently outperforms baselines on RootSE.RootSE contains 102 failed repository-level coding trajectories annotated with earliest-error steps.Source: §X · p. 10
Loading 2605.26563v3…