PaperlayerView exact revision on arXiv

Source-linked AI summary

Continue or Restart? The Limits of Early Failure Prediction in Coding Agents

Srinath Srinivasan, Lunxiao Li, Tim Menzies, Wesley Assunção

arXiv:2610.35971v1cs.SE

TL;DR

Early stopping for coding agents requires distinguishing task difficulty from attempt-level progress. Across four model-framework settings, the study analyzes execution traces and replays restart policies under shared token budgets. Early prefixes show near-chance within-task discrimination, and uninterrupted continuation outperforms both restart policies across most tested configurations.

  • Problem

    Pooled prediction can reflect baseline task solvability rather than distinguish successful and failed attempts on the same issue, complicating evaluation of whether early signals justify stopping.

    Source: §I · p. 1

  • Method

    The study compares issue context and early execution prefixes across 147,002 traces, then replays learned and matched random restarts against continuation under shared token caps.

    Source: §I · pp. 1–2, F. Budget-constrained restart replay… · p. 5

  • Results

    Early prefixes yield near-chance within-task discrimination (AUCwithin ≈0.50); continuation achieves higher solve rates at lower mean token cost in 216 of 224 tested configurations.

    Source: §VII · p. 10

  • Takeaways & Limitations

    Stopping evaluations should establish attempt-level discrimination and net compute savings relative to uninterrupted continuation.

    Source: §VII · p. 10

    The replay measures transcript tokens rather than provider caching discounts or wall-clock tool latency, and gives zero credit to partial edits that exceed residual budgets.

    Source: §VI · p. 10

Abstract

from arXiv · show

Early stopping promises to reduce compute costs for autonomous coding agents by terminating unpromising runs early. Deciding when to abandon a run requires separating baseline issue difficulty from the trajectory of an individual attempt. We study the distinction in 147,002 execution traces from two public corpora spanning four model-framework settings. Comparing pre-execution issue context against early execution prefixes, we evaluate linear count models, initial-turn text embeddings, task-conditional rankers, and a fine-tuned Qwen3-0.6B monitor. Although early prefixes modestly improve cross-task outcome ranking, within-task discrimination remains near chance across early landmarks (5k-20k tokens): initial-turn embeddings and 20k-token task-conditional models yield AUCs around 0.50, and three-seed neural evaluations across 32,406 attempts at 20k tokens achieve a mean within-task AUC of 0.50 [0.48, 0.51] across tasks of varying solve rates. To evaluate how these predictions translate into stopping decisions, we replay score-guided and matched random restarts under shared token budgets alongside uninterrupted continuation. Under a 75k-token cap, learned restarting improves resolution over matched random restarting by up to 1.98 percentage points. An exact algebraic decomposition resolves this paradox: the gain over random restarting stems almost entirely from between-task stopping allocation (avoiding stops on easy issues), whereas attempt-level selection within a task contributes near zero (-0.11 to +0.24 percentage points). Across all four settings at the 75k cap (and 216 of 224 tested budget-threshold configurations overall), uninterrupted continuation achieves higher solve rates at lower mean token cost than both restart policies. Practical evaluations of early stopping must therefore establish attempt-level discrimination and demonstrate net compute savings relative to continuation.

I. INTRODUCTION

Early stopping is useful only if execution signals justify abandoning an attempt rather than merely identifying difficult issues. The paper evaluates early prediction and restart policies while separating task difficulty from attempt-level progress.

I. INTRODUCTION

  • Pooled prediction can rank outcomes by task difficulty even when a monitor gives identical scores to every attempt on the same issue.Within-task AUC compares successful and failed runs on the same task; 0.50 indicates random ranking.Source: §I · p. 1
  • Under a shared token cap, restart policies must be compared with both matched random restarting and uninterrupted continuation.Abort-only termination can forfeit late resolutions, while restarting spends the residual budget on a fresh attempt.Source: §I · p. 1
  • 147,002 traces from two public corpora are used to compare issue context and execution prefixes across four model-framework settings.The study evaluates landmarks from 5k to 60k tokens, focusing on 20k, when nearly all runs remain active.Source: §I · p. 2
  • Across four settings, early prefixes yield near-chance within-task discrimination (AUCwithin ≈0.50), while static context supports cross-task ranking.The evaluation spans 5k embeddings, 20k count models and task-conditional rankers, and a three-seed neural monitor.Source: §I · p. 2
  • At a 75k-token cap, learned stopping beats matched random restarting by up to 1.98 percentage points, but the gain comes from between-task allocation.Across 224 budget-threshold configurations, continuation has higher solve rates at lower mean token cost in 216 configurations.Source: §I · p. 2

II. RELATED WORK

Prior work motivates controlling for task identity when evaluating trajectory signals and distinguishes cold restarts from state-preserving recovery. This study extends that perspective across four coding-agent settings and analyzes released execution traces.

II. RELATED WORK

  • Prior trace-mining work finds that issue-level properties shape trajectory metrics, motivating tests of whether early observations capture attempt-specific progress.Related monitors use trajectory, sequential, or language-model signals, while this study evaluates ranking among attempts on the same issue.Source: §II · p. 2
  • State-preserving recovery can retain edits from aborted runs, whereas cold restarting discards exploration; the study adapts a terminal-outcome monitor to assess cold replacements.Prior work reports 14.6–20.4% execution-token savings at a 5% false-positive target, and higher resolution with transferred edits than cold restarts in one comparison.Source: §II · p. 2
  • Across four coding-agent settings, the study connects within-task evaluation to restart mechanics and attributes learned stopping’s gain over random restarts to task selection.It also reports that uninterrupted continuation dominates both policies across the shared-budget frontier.Source: §II · p. 3

A. Released runs and labels

  • The analysis covers 147,002 traces from two public corpora, with settings defined by combinations of language models and execution frameworks.The four settings include OpenHands and SWE-agent runs across the Nebius and NVIDIA corpora; task counts overlap across settings.Source: A. Released runs and labels · p. 3
  • Each completed run receives a binary label based on whether the benchmark harness’s final patch resolves the issue.Unlabelled records are excluded.Source: A. Released runs and labels · p. 3

B. Trace representation and decision landmarks

The paper measures execution progress in transcript tokens and evaluates predictions at fixed decision landmarks. Its metrics distinguish pooled ranking from within-task discrimination and account for task clustering in uncertainty estimates.

B. Trace representation and decision landmarks

  • Transcript progress is counted across recorded messages using each setting’s native tokenizer, and tool invocations are normalized into behavioral categories.The categories cover file inspection, codebase search, code editing, and test execution.Source: B. Trace representation and decision… · p. 3
  • 20k tokens is the primary decision landmark, where nearly all runs remain active and the primary corpus has detected tests in 87.51% and edits in 32.80% of active runs.The study evaluates seven landmarks from 5k to 60k tokens, with eligible runs defined by total length exceeding the landmark.Source: B. Trace representation and decision… · p. 3

C. Evaluation metrics and task clustering

  • Pooled AUC mixes cross-task difficulty with attempt-level differences, whereas within-task AUC compares runs on the exact same task.Within-task AUC aggregates successful-failed pairs across mixed-outcome tasks, giving each valid pair equal weight.Source: C. Evaluation metrics and task… · p. 4
  • Uncertainty intervals use cluster bootstraps that resample repositories for pooled and replay metrics, and tasks for within-task AUC.Neural-monitor estimates also resample training seeds.Source: C. Evaluation metrics and task… · p. 4

D. Prediction models for RQ1 and RQ2

The prediction models compare pre-execution issue features with prefix-derived signals, while conditional models isolate differences between attempts on the same task. Supplementary designs test metadata, text embeddings, and task solvability as alternative signals.

D. Prediction models for RQ1 and RQ2

  • Unseen-repository cross-validation compares issue-only and issue-plus-prefix models to measure whether execution features improve prediction.The logistic-regression models use eight issue features and add 18 prefix features, or 16 in the Open-SWE settings.Source: D. Prediction models for RQ1 and RQ2 · p. 4
  • Task-conditional logistic regression removes task-level intercepts and estimates prefix-feature weights from successful and failed attempts on the same issue.Cross-setting transfer, within-task label permutations, and equivalence tests assess the conditional pipeline and near-chance estimates.Source: D. Prediction models for RQ1 and RQ2 · p. 4
  • Supplementary models test whether task categories and programming language improve prediction, and whether attempt discrimination varies with empirical task solvability.Tasks are stratified into low, medium, and high solve-rate tiers.Source: D. Prediction models for RQ1 and RQ2 · p. 4
  • Frozen sentence embeddings of the issue description, initial assistant turn, and first tool observation test for lexical signals missed by summary counts.The embeddings are combined with initial prompt-context metadata in a regularized linear classifier.Source: D. Prediction models for RQ1 and RQ2 · p. 4
  • A task-rate oracle uses a held-out estimate of each task’s solve fraction to test task-level predictability and whether prefix features add value when solvability is known.The diagnostic includes tasks with at least four active attempts and evaluates the rate feature alone and with prefix counts.Source: D. Prediction models for RQ1 and RQ2 · p. 4

E. Adapted neural monitor

The neural monitor is trained to rank attempts within the same task, then evaluated alongside learned restarts, matched random restarts, and continuation under shared token caps. An exact decomposition separates task-level stopping allocation from within-task attempt selection.

E. Adapted neural monitor

  • At the 20k-token landmark, the monitor compares issue-only inputs with issue-plus-prefix inputs, training on matched successful and failed attempts from the same mixed-outcome task.Balanced within-task pairing fixes each task’s training prevalence at 0.50 and combines binary cross-entropy with pairwise ranking loss.Source: E. Adapted neural monitor · pp. 4–5
  • The neural evaluation covers 32,406 attempts across 2,946 tasks, using unseen-repository cross-validation and three training seeds per fold.Validation within-task AUC governs checkpoint selection; intervals use joint repository-and-seed bootstrapping.Source: E. Adapted neural monitor · p. 5

F. Budget-constrained restart replay for RQ3

  • Replay compares score-threshold learned restarts with matched random restarts and uninterrupted continuation, under a shared cap on combined token expenditure.A replacement receives only the residual B − b tokens after the initial run is stopped at landmark b.Source: F. Budget-constrained restart replay… · p. 5
  • The exact solve-rate decomposition distinguishes between-task stopping-rate effects from the within-task covariance between stopping decisions and restart value.The within-task term captures whether policies preferentially stop weaker attempts on the same task.Source: F. Budget-constrained restart replay… · pp. 5–6
  • The cost-solve analysis tests Pareto dominance across 224 policy, setting, cap, and threshold configurations.A policy is dominated when another achieves at least as high a solve rate at no greater mean token cost, with a strict improvement on one dimension.Source: F. Budget-constrained restart replay… · p. 6

A. Baseline task clustering across settings

Outcomes cluster strongly by repository issue, and task solvability also tracks across agent frameworks. Early action counts modestly improve pooled ranking, but initial-turn embeddings add no consistent pooled gain and early prefixes do not improve within-task discrimination.

A. Baseline task clustering across settings

  • Across all four settings, task-level outcome intraclass correlation reaches at least 0.71, so repeated attempts on an issue predominantly share an outcome.This clustering makes baseline task solvability important to distinguish from attempt-level progress.Source: A. Baseline task clustering across… · p. 6
  • Across 7,102 shared tasks, Qwen3.5 SWE-agent and OpenHands task solve rates have a Spearman rank correlation of 0.67.SWE-agent’s mean solve rate is 47.30% versus OpenHands’ 33.24%, a difference of 14.06 percentage points (95% repository-bootstrap CI: [12.26, 15.95]).Source: A. Baseline task clustering across… · p. 6

B. RQ1: Value of observing an early prefix

  • At 5k tokens, prefix action counts raise pooled AUC from 0.56 to 0.58, while at 20k tokens they raise it from 0.57 to 0.61.At 5k, initial-turn embeddings leave pooled AUC at 0.62 in both issue-only and issue-plus-prefix arms.Source: B. RQ1: Value of observing an early… · p. 6
  • Early action counts and static metadata yield modest pooled-ranking gains of 0.02–0.04 AUC, while initial-turn embeddings provide no consistent pooled improvement over issue text alone.Adding task categories raises pooled AUC by approximately 0.02 while leaving within-task AUC unchanged (∆AUCwithin < 0.01).Source: B. RQ1: Value of observing an early… · p. 6

C. RQ2: Ranking attempts at the same task

Early execution prefixes do little to distinguish successful from failed attempts on the same issue, even when pooled ranking looks strong. The task-rate oracle illustrates how task difficulty can drive pooled performance without within-task discrimination.

1) Early action counts and frozen embeddings:

  • At 20k tokens, within-task AUC remains near 0.49–0.50 across all four settings, indicating near-chance discrimination between attempts on the same task.At 5k tokens on the primary corpus, the issue-plus-prefix count model has within-task AUC 0.50 [0.49, 0.51].Source: 1) Early action counts and frozen… · p. 6
  • At 30k and 40k tokens, primary-corpus within-task AUC reaches 0.52 [0.51, 0.53] and 0.54 [0.53, 0.55], with 93% and 69% of runs still active.At 60k, AUC reaches 0.56 while only 23% of initial runs remain active.Source: 1) Early action counts and frozen… · p. 6, Figure 1 · p. 7
  • At 5k tokens, the task-rate oracle yields pooled AUC 0.97 but within-task AUC 0.50; adding prefix counts leaves pooled AUC at 0.97 and within-task AUC at 0.50.This task-rate baseline uses empirical solve fractions from reserved attempts.Source: 1) Early action counts and frozen… · p. 6, Figure 2 · p. 7
  • At 5k tokens, frozen text embeddings have within-task AUC 0.49 [0.48, 0.50], while pooled AUC spans 0.58 to 0.97 across predictors.Within-task AUC clusters around 0.50 despite the wide pooled range.Source: 1) Early action counts and frozen… · p. 7

2) Task-conditional and cross-setting training:

Across task-conditional models, cross-setting transfers, neural monitoring, and solvability tiers, early prefixes provide near-chance discrimination between attempts on the same task. Attempt-level separation appears at run completion, but not at the 20k-token landmark.

2) Task-conditional and cross-setting training:

  • The task-conditional model achieves a within-task AUC of 0.50 [0.49, 0.51] across 38,447 successful-failed pairs from 1,737 mixed-outcome tasks.Its 90% task-bootstrap interval falls within both tested equivalence bounds, and the within-task label-permutation control is centered at chance.Source: 2) Task-conditional and cross-setting… · p. 7

3) Adapted neural monitor:

  • At 20k tokens, the three-seed neural monitor achieves a mean within-task AUC of 0.50 [0.48, 0.51] across 32,406 attempts.Performance remains near 0.50 within every repository fold and training seed; the neural monitor’s 90% interval lies within the [0.45, 0.55] equivalence bound.Source: 3) Adapted neural monitor · p. 7, Table II · p. 7

4) Stratification by task solvability and late-trajectory signals:

D. RQ3: Evaluating restart policies under shared token budgets

Under a shared 75k-token cap, learned restarting can outperform matched random restarting, but uninterrupted continuation outperforms learned restart across all four settings. The learned-versus-random advantage is driven by between-task stopping allocation, while within-task selection contributes little.

D. RQ3: Evaluating restart policies under shared token budgets

  • Under a shared 75k-token cap, learned restart beats matched random restart by up to 1.98 percentage points in the Qwen3.5 settings, while uninterrupted continuation outperforms learned restart across all four settings.The learned-minus-random solve-rate differential spans −0.66 to +1.98 percentage points.Source: Figure 3 · p. 8, D. RQ3: Evaluating restart policies… · p. 8, Table IV · p. 8

1) Between-task selection drives the 75k learned-versus-random differential:

  • The between-task selection term matches the sign of the learned-versus-random solve-rate difference in every setting, while the within-task term remains between −0.11 and +0.24 percentage points.The between-task term ranges from −0.70 to +1.75 percentage points and exceeds the magnitude of the within-task term in every setting.Source: 1) Between-task selection drives the… · p. 8, Table V · p. 9
  • Learned stopping assigns lower scores to harder tasks, where both replacement success and original continuation success are below the corpus average.In the two Qwen3.5 settings, fewer forfeited continuation successes outweigh reduced replacement credit, yielding a positive between-task contribution.Source: 1) Between-task selection drives the… · p. 8

2) Uninterrupted continuation dominates the efficiency frontier:

Under shared token budgets, uninterrupted continuation generally achieves higher solve rates at lower mean token costs than restart policies. The comparison reflects both weak early within-task discrimination and the costs of truncating a run and repeating its exploration.

2) Uninterrupted continuation dominates the efficiency frontier:

  • At the 75k cap, continuation has a strictly higher solve rate and lower mean token cost than all 56 restart configurations; across tested caps, it Pareto-dominates 216 of 224 configurations.At 100k, it Pareto-dominates 55 of 56 configurations; the remaining point adds less than 0.01 percentage points in solve rate at an added cost of 376 mean tokens.Source: 2) Uninterrupted continuation… · p. 9
  • A 20k-token restart leaves only B − 20k tokens for replacement, while continuation lets the original run use the full cap.At 30k and 50k caps, 91%–100% of trajectories require more than the replacement’s residual budget to finish.Source: 2) Uninterrupted continuation… · p. 9

V. DISCUSSION

  • Aborting at 20k forfeits prior repository exploration and makes a replacement repeat setup and localization, increasing token expenditure.The discussion attributes cold-restart underperformance to forfeiting initial exploration while compressing the residual budget.Source: 2) Uninterrupted continuation… · p. 9, §V · p. 10
  • At 20k tokens, early prefixes do not separate same-task attempts well; measurable within-task separation rises from 0.52 at 30k to 0.62 at run completion.By those later stages, a substantial fraction of the token budget has been spent or most runs have terminated.Source: §V · p. 9
  • Static issue signals support pre-execution routing and budgeting; on shared Qwen3.5 tasks, scaffold choice shifts solve rates by 14.06 percentage points.The reported shift is SWE-agent 47.30% versus OpenHands 33.24%, compared with at most 1.98 percentage points from mid-run restarts.Source: §V · p. 10
  • Runtime-monitoring evaluations should report within-task AUC alongside pre-execution issue baselines and permutation controls, then compare policies with random stopping and continuation.The discussion also calls for decomposing outcomes into between-task and within-task selection.Source: §V · p. 10

VI. THREATS TO VALIDITY

The conclusion reports near-chance within-task prediction and finds that uninterrupted continuation dominates restart policies across most tested budget-threshold configurations. The validity discussion bounds these findings to the study’s corpora, progress measure, model settings, and empirical analysis choices.

VI. THREATS TO VALIDITY

  • The study’s labels and trajectories inherit the outcome labels and execution environments of the public benchmark harnesses and corpora.The authors audited task-identifier text consistency and used within-task label-permutation controls; controls centered at chance in the primary and OpenHands settings, while calibration deviation in SWE-agent/Qwen3.5 precluded formal equivalence claims.Source: §VI · p. 10
  • Transcript tokens standardize progress comparisons across corpora but omit provider caching discounts and wall-clock tool latency.Replay measures binary task resolution under hard token caps and gives no credit to partial edits that exceed the remaining budget; the editor-tool heuristic can also include file viewing alongside code modification.Source: §VI · p. 10
  • The findings cover four open-weight model-framework settings on Python-centric repository benchmarks, using linear count and embedding classifiers and a 0.6B-parameter monitor.Within-task discrimination requires mixed-outcome tasks, and monitors using internal representations, sequential verifier feedback, or state-preserving recovery may behave differently.Source: §VI · p. 10
  • Confidence intervals resample out-of-fold predictions by cluster without refitting models, and Pareto dominance across 224 budget-threshold configurations uses empirical replay point estimates.Subgroup and multi-landmark sensitivity checks use Holm–Bonferroni adjustment, while primary 20k-landmark intervals are unadjusted.Source: §VI · p. 10

VII. CONCLUSION

  • AUCwithin ≈0.50 across early landmarks: embeddings, action-count models, task-conditional rankers, and a three-seed Qwen3-0.6B monitor show near-chance within-task discrimination.The neural evaluation covers 32,406 attempts, including tasks with high solvability.Source: §VII · p. 10
  • 111 of 112 configurations at the 75k and 100k frontiers, and 216 of 224 across all tested caps, favor uninterrupted continuation over restarts in solve rate and mean token cost.At a 75k-token cap, learned stopping beats matched random restarting by up to 1.98 percentage points in two settings, but the gain comes from between-task allocation, not within-task failure detection.Source: §VII · p. 10
  • The authors recommend that future stopping and recovery methods demonstrate both within-task discrimination across repeated attempts and net compute savings against uninterrupted continuation.Source: §VII · p. 11

Related papers

Loading 2610.35971v1…