PaperlayerView exact revision on arXiv

Source-linked AI summary

Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

Xiangxin Zhao, Han Li, Shuaiting Li, Tianyi Zhao, Earl T. Barr, Federica Sarro, He Ye

arXiv:2607.09510v1cs.SEcs.AI

TL;DR

Existing studies largely treat coding-agent failure as a final outcome, offering limited insight into its temporal development in CLI settings. This study annotates failure trajectories and finds that failures are predominantly epistemic, often begin early, and can remain hidden past the point of recovery.

  • Problem

    Prior studies largely treat coding-agent failure as a static outcome and focus on issue-resolution or multi-agent settings rather than terminal agents.

    Source: §I · p. 1

  • Method

    The study analyzes CLI failure trajectories as processes, annotating decisive errors, empirical lock-in, and first observability across execution traces.

    Source: §I · p. 1, E. Methodology for RQ1 (Error… · p. 4

  • Results

    Failures are predominantly epistemic, typically originate early, and often remain hidden until recovery is no longer possible.

    Source: §VI · p. 11

  • Takeaways & Limitations

    Pass rate alone captures only part of coding-agent reliability, motivating trajectory-aware evaluation, monitoring, and intervention.

    Source: §VI · p. 11

    Cross-system comparisons are observational rather than causal, despite a shared benchmark and evaluation protocol.

    Source: §V · p. 10

Abstract

from arXiv · show

Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a growing concern. Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process, providing limited insight into how failures emerge, evolve, and become unrecoverable. We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories. We first collect 3,843 execution trajectories generated by seven frontier models across three coding-agent scaffolds (OpenHands, MiniSWE, and Terminus2) on Terminal-Bench, then carefully filter them to obtain 1,794 complete and valid trajectories for manual annotation (over 63,000 execution steps), from which we derive 14 findings spanning failure occurrence, root causes, recovery, and cross-system consistency. Our findings show that coding-agent failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is no longer possible, suggesting that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.

I. INTRODUCTION

Prior studies largely treat coding-agent failure as a static outcome and rarely examine terminal agents, leaving limited insight into how failures evolve. This study analyzes CLI failure trajectories as processes and reports findings across occurrence, causes, recovery, and system differences.

I. INTRODUCTION

  • Existing studies treat failure as a static outcome and rarely study terminal agents, leaving limited insight into how errors evolve into unrecoverable failures.Source: §I · p. 1
  • The study analyzes 1,794 valid trajectories from three scaffolds and seven models, selected from 3,843 executions across 89 tasks.Source: §I · p. 2
  • The analysis yields 14 findings spanning failure occurrence, root causes, recovery, and cross-system differences.Source: §I · p. 2
  • The paper argues that coding-agent failure should be studied as a process rather than a final success/failure label.Source: §I · pp. 1–2
  • The authors release an annotated dataset of 1,794 runs and more than 63,000 execution steps.Source: §I · p. 2

II. RESEARCH SETTING AND METHODOLOGY

The paper’s methodology collects and filters CLI-agent trajectories, then manually annotates them to support four research questions. It distinguishes error events from final failure and treats unrecoverability as an empirical observation.

II. RESEARCH SETTING AND METHODOLOGY

  • The study collects trajectories, filters them into a valid dataset, and manually annotates them to answer RQ1–RQ4.Source: §II · p. 2
  • An error is a problematic step event, failure is the final outcome, and an unrecoverable error is one from which no recovery was observed.The authors do not claim recovery is impossible.Source: §II · p. 2

A. Research Questions •

  • The four research questions examine when errors become unrecoverable, their root causes, recovery behavior, and variation across models and scaffolds.Source: A. Research Questions • · p. 2

B. Research Setting

The study uses Terminal-Bench and three diverse CLI-agent scaffolds with seven frontier models. Filtering for complete cross-system coverage yields a 1,794-trajectory dataset for analysis.

B. Research Setting

  • Terminal-Bench provides 240 human-curated containerized command-line tasks with instructions, Docker-based environments, and executable tests.Its outcome-only scoring and common execution harness support trajectory analysis and cross-system comparison.Source: B. Research Setting · p. 2
  • The three scaffolds—MiniSWE, OpenHands, and Terminus2—represent lightweight, platform-based, and terminal-native interaction designs.Source: B. Research Setting · p. 2
  • The study evaluates seven frontier models across all 3 × 7 scaffold–model combinations, a controlled snapshot collected from May 2025 onward.Source: B. Research Setting · p. 3

C. Trajectory Collection & Filtering

  • From 3,843 completed trajectories, the authors retain 89 tasks with complete coverage across 21 systems, producing 1,794 valid trajectories.The raw pool lacks 1,197 runs that terminated abnormally or did not complete; 75 trajectories are unavailable in the complete grid.Source: C. Trajectory Collection & Filtering · p. 3
  • The final dataset contains 1,184 failed and 610 successful trajectories across three scaffolds and seven models.The analyzed corpus is the largest trajectory dataset collected for an empirical study of coding agents on terminal tasks, compared with 93 trajectories in [14] and 120 in [10].Source: C. Trajectory Collection & Filtering · p. 3

D. Trajectory Annotation

The authors use LLM-assisted drafts followed by independent human review to annotate trajectories. For failure timing, they track decisive error, empirical lock-in, and first observability, then evaluate a prefix monitor against lock-in.

D. Trajectory Annotation

  • The 1,184 failed trajectories are distributed across scaffolds and foundation models, with every scaffold and model contributing substantially.Source: Figure 2 · p. 4
  • Claude Opus 4.6 drafts structured annotations, then two independent human annotators verify evidence and finalize every label.Every failed trajectory receives two annotations; agreement ranges from Cohen’s κ = 0.78 to 0.94 across key labels, with disagreements resolved through discussion.Source: D. Trajectory Annotation · pp. 3–4

E. Methodology for RQ1 (Error Occurrence)

  • The three timestamps mark the decisive error, empirical loss of recoverability, and first observable sign; their intervals measure the fix window and observability lag.The decisive error is labeled retrospectively, and observability may occur before or after lock-in.Source: E. Methodology for RQ1 (Error… · p. 4, Figure 3 · p. 4
  • A prefix monitor reads trajectory prefixes without outcome knowledge and predicts whether runs have passed lock-in, comparing task-name-only input with task names plus requirements.It is evaluated on 2,659 prefixes from 600 trajectories stratified across 21 systems.Source: E. Methodology for RQ1 (Error… · p. 4

F. Methodology for RQ2 (Root Cause)

The methodology traces decisive errors to their triggering mechanisms, examines recovery in both failed and successful trajectories, and compares these patterns across agent scaffolds and foundation models.

F. Methodology for RQ2 (Root Cause)

  • Root causes are classified by the trigger of the decisive error at terr, distinguishing information misuse, capability limitations, and external environment conditions.Grouping related triggers supports analysis across agents without case-by-case analysis.Source: F. Methodology for RQ2 (Root Cause) · p. 4

G. Methodology for RQ3 (Recovery)

  • Recovery analysis compares failed and successful trajectories because decisive errors may lead either to failure or to correction and completion.Successful recovery is measured by the steps needed to return to a successful execution path.Source: G. Methodology for RQ3 (Recovery) · pp. 4–5
  • For failed trajectories, post-lock-in behavior is classified by dominant activity and failure awareness into five patterns, measured by prevalence, wasted execution, and median remaining steps.Failure awareness records whether the agent explicitly recognizes that the run has gone wrong.Source: G. Methodology for RQ3 (Recovery) · p. 5

H. Methodology for RQ4 (Model and Scaffold Differences)

  • Across three agent scaffolds and seven foundation models, the study compares decisive-error timing, root-cause distributions, and recovery behavior to separate consistent from system-specific characteristics.The comparison covers results from RQ1 to RQ3.Source: H. Methodology for RQ4 (Model and… · p. 5

III. ANATOMY OF CODING AGENT FAILURES

Failure unfolds across distinct stages: decisive errors often occur early, may remain recoverable briefly, and can stay unobservable until much later. The results quantify this timing and show why early detection matters.

A. Results for RQ1 (Error Occurrence)

  • At the 50% mark, failed trajectories reach decisive error by step 7, lock-in around step 12, and observable evidence around step 16.The cumulative curves cover 1,184 failed trajectories.Source: Figure 4 · p. 5, A. Results for RQ1 (Error Occurrence) · p. 5
  • The median decisive error occurs at step 7 in runs with a median of 27 steps, placing onset within the first quarter of failed trajectories.The authors argue reliability efforts should prevent or detect decisive errors early rather than rely on late-stage repair.Source: A. Results for RQ1 (Error Occurrence) · p. 5
  • The median recovery window after a decisive error is one step, although 60.9% of errors leave at least one recovery step and 239 trajectories remain recoverable for over ten.This window provides an opportunity for intervention before failure becomes irrecoverable.Source: A. Results for RQ1 (Error Occurrence) · p. 5
  • Observable failure signals appear around ten steps after decisive errors, so analyses based only on observable outcomes can misidentify when failure begins.The median first observable failure is approximately step 16, compared with step 7 for the decisive error.Source: A. Results for RQ1 (Error Occurrence) · p. 5
  • A prefix monitor flags locked-in runs at 82% precision, but its median lead time is zero and it flags only 3.7–8.7% of failures before lock-in.Supplying task requirements raises recall on specification-relative failures, while overall real-time recall remains 28.8% at best.Source: A. Results for RQ1 (Error Occurrence) · p. 6
  • In a representative silent failure, a failed repository change at step 2 led to lock-in by step 6, with the first signal appearing at step 23.The agent built the site in the wrong directory, and the failure remained invisible for 17 steps after lock-in.Source: Figure 5 · p. 6, A. Results for RQ1 (Error Occurrence) · p. 6

B. Results for RQ2 (Root Cause)

The taxonomy groups decisive errors into epistemic, competence, and environment causes, with epistemic errors most common. False premises are the largest trigger, as agents act on incorrect beliefs despite information being available.

B. Results for RQ2 (Root Cause)

  • False premises account for 30.7% of decisive errors, the single most common trigger in the taxonomy.Once adopted, an incorrect assumption can make subsequent reasoning internally consistent while directing the trajectory away from correct execution.Source: B. Results for RQ2 (Root Cause) · p. 7
  • Epistemic errors account for 57.9% of decisive errors, exceeding competence errors (32.8%) and environment errors (9.4%).The three categories are mutually exclusive; epistemic triggers involve available information being ignored, forgotten, or misinterpreted.Source: B. Results for RQ2 (Root Cause) · p. 7
  • In the representative sudo example, the agent mistakes “sudo: not found” for evidence of insufficient permission, although the needed information is available.The false premise becomes decisive and shapes later decisions.Source: B. Results for RQ2 (Root Cause) · p. 7, Figure 6 · p. 7

C. Results for RQ3 (Recovery)

Recovery separates successful from failed trajectories: successful runs act on error signals and recover quickly, while many failed runs continue without progress. Failed recovery can consume substantial execution, and fabrication often appears after recovery has become impossible.

C. Results for RQ3 (Recovery)

  • 82% of failed trajectories continue executing after recovery is empirically impossible; only 18% terminate shortly afterward.Continued behaviors include repairing the wrong problem, repeating an unsuccessful strategy, and verification that cannot change the outcome.Source: C. Results for RQ3 (Recovery) · p. 8
  • Repairing the wrong problem contributes 39% of all wasted execution, despite appearing in only 24% of failed trajectories.This behavior has a median continuation of 21 steps; Table III reports behavior share, waste share, and median remaining steps.Source: C. Results for RQ3 (Recovery) · p. 8
  • 71% of successful trajectories recover from at least one error, so making an error alone does not distinguish successful from failed runs.Only 29% of successful runs complete without an observable mistake; the distinction is whether the agent recovers.Source: C. Results for RQ3 (Recovery) · p. 8
  • Successful recoveries have a median duration of 5 execution steps, versus 12 for failed recoveries.When recovery continues substantially longer than typical successful recovery, the authors recommend reconsidering the diagnosis and capping repair effort.Source: C. Results for RQ3 (Recovery) · pp. 8–9
  • Before recovery becomes impossible, 92% of successful trajectories respond to an error signal, compared with 37% of failed trajectories.Observable error signals occur at similar rates in successful and failed trajectories (74% vs. 72%).Source: C. Results for RQ3 (Recovery) · p. 9
  • Fabricated success appears in 26% of failed trajectories and is a response to failure, not its cause.84% of fabrication begins at or after lock-in, its median onset is exactly at lock-in, and it co-occurs with other unproductive behaviors in 41% of runs where it appears.Source: C. Results for RQ3 (Recovery) · p. 9

D. Results for RQ4 (Model and Scaffold Differences)

Final success varies with both the foundation model and the coding-agent scaffold. Across systems, epistemic errors remain the dominant cause of failure.

D. Results for RQ4 (Model and Scaffold Differences)

  • Pass rates range from 19% to 45% across the 21 model–scaffold systems, with both model and scaffold influencing success.Table IV compares models within each scaffold and scaffolds for each model; under Terminus2, pass rates range from 29% (Qwen) to 43% (DeepSeek), while GPT-5 ranges from 34% under MiniSWE to 45% under OpenHands.Source: D. Results for RQ4 (Model and Scaffold… · p. 9, Table IV · p. 9
  • Epistemic errors are the largest failure category in every model–scaffold system, accounting for 44% to 80% of failed trajectories.The reported range runs from 44% for Qwen3 under OpenHands to 80% for Claude under Terminus2.Source: D. Results for RQ4 (Model and Scaffold… · p. 9, Table V · p. 9

A. Trajectory and Failure Analysis of LLM Agents

Prior work established trajectory analysis as useful for studying agent behavior and failures, but remained partial in scale, explanatory structure, or setting. Agent benchmarks increasingly target command-line tasks, while outcome-only scoring provides limited visibility into how agents reach their results.

A. Trajectory and Failure Analysis of LLM Agents

  • Prior trajectory and failure analyses vary in scale and explanatory structure, and most target repository issue-resolution settings rather than general terminal interaction.This study complements them with a larger-scale, mechanism-oriented analysis of terminal coding agents organized around a taxonomy linking why and how failures occur.Source: A. Trajectory and Failure Analysis of… · p. 10

B. Benchmarks and Evaluation for Agents

  • Outcome-only benchmark protocols score final environment state and report success rate, providing limited visibility into how agents reach their outcomes.The passage also notes that outcome-centric reward designs can substantially misestimate agent capability.Source: B. Benchmarks and Evaluation for Agents · p. 10

C. LLM Agents for Software Engineering

Prior work develops LLM software-engineering agents for repository tasks, while this paper examines failure trajectories across CLI agents. Failures are predominantly epistemic, often begin early, and may remain hidden until recovery is no longer possible.

C. LLM Agents for Software Engineering

V. THREATS TO VALIDITY

  • Comparisons across systems are observational rather than causal, and timeout or incomplete trajectories fall outside the study scope.The analyzed set covers 89 Terminal-Bench tasks and 21 model–scaffold systems.Source: §V · p. 10

VI. CONCLUSION

  • The study analyzes 1,794 annotated trajectories across 89 tasks, three scaffolds, and seven frontier models.It examines when failures begin, why they occur, and how they evolve toward recovery or unrecoverability.Source: §VI · p. 10
  • Failures are predominantly epistemic, typically originate early, and often remain hidden until recovery is no longer possible.The conclusion frames failure as a temporal process that includes onset, evolution, and recovery.Source: §VI · pp. 10–11
  • Because pass rate captures only part of reliability, the findings motivate trajectory-aware evaluation, monitoring, and intervention.Source: §VI · p. 11
Loading 2607.09510v1…