PaperlayerView exact revision on arXiv

Source-linked AI summary

StopLoss: Teaching Agents to Predict Failure and Stop Early

Jiarong Liang, Xueguang Ma, Dongfu Jiang, Yuxuan Zhang, Ping Nie, Wenhu Chen

arXiv:2610.25525v1cs.AI

TL;DR

Coding agents may spend heavily on runs that ultimately fail, while budget rules can stop eventual successes and general-purpose judges add monitoring cost. StopLoss learns to recognize failure from outcome-labeled trajectory windows and applies a calibrated stopping threshold without changing the executor. It lowers cost per retained solve than every judge policy across four settings, with reported savings on Verified and Pro.

  • Problem

    Coding agents can continue spending on runs that will fail verification, while budget-based stopping can cut off eventual successes and repeated general-purpose judging can cost more than it saves.

    Source: §1 · p. 2

  • Method

    StopLoss trains a small external monitor on completed trajectories and verifier outcomes, scores local execution windows, and uses a threshold calibrated on successful runs to stop likely failures without changing the executor.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    StopLoss has lower cost per retained solve than every judge policy across four settings; on Verified/Luna it saves 4.5% versus full execution while retaining 99.0% of otherwise successful runs.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Outcome supervision provides a useful failure signal by the first scheduled check at one to two orders of magnitude lower per-check cost than tested judges.

    Source: §1 · p. 2

    Training uses only GPT-generated trajectories and evaluation is limited to Python tasks, including a Pro subset of three repositories.

    Source: §7 · p. 10

Abstract

from arXiv · show

Coding agents keep working on runs that will fail verification, and asking a frontier model to watch for this is expensive in its own right. We introduce StopLoss, a small external monitor that learns failure recognition from completed trajectories and their verifier outcomes. The name is inspired by the stop-loss order in finance: StopLoss ends a run early when the run is likely to fail, which saves resources. Final outcomes label earlier local windows, so a fine-tuned language model can score the most recent tool steps in a single forward pass, and a threshold calibrated on successful runs turns that score into an abort decision without changing the executor. Trained on GPT-family SWE-rebench trajectories and evaluated on SWE-bench Verified and a SWE-bench Pro subset with GPT and Claude executors, StopLoss is compared against six general-purpose judges reading identical windows. Outcome supervision lifts failure prediction from near chance to a useful signal at the first check, at one to two orders of magnitude lower cost per check than any tested judge. This translates into savings: relative to completing every task, StopLoss lowers cost per retained solve by 4.5\% on Verified while preserving 99\% of successful runs, and by up to 31\% on the costlier Pro subset, reaching 48\% with a Qwen3-8B backbone, and in all four settings its cost per retained solve is below that of every judge policy. Outcome-trained small monitors can make failure recognition cheap enough for early stopping to pay for itself.

1 INTRODUCTION

Coding agents can spend resources on runs that ultimately fail, while budget caps may stop eventual successes and repeated general-purpose judging can cost more than it saves. StopLoss learns failure recognition from completed trajectories and uses a calibrated, inexpensive monitor to stop likely failures without changing the executor.

1 INTRODUCTION

  • Budget caps can cut off runs that would eventually pass, while repeated general-purpose judging can cost more than it saves.Early stopping pays only when avoided execution exceeds monitoring overhead plus the successes mistakenly stopped.Source: §1 · p. 2
  • Final verifier outcomes label earlier trajectory windows, providing supervision to predict eventual failure from local records without enforcing a budget.Because labels come from completed executions, a run that eventually passes remains a success even if it was expensive.Source: §1 · p. 2
  • StopLoss adapts 7B–8B models on GPT-family SWE-rebench trajectories and compares them with six judges reading identical windows on Verified and a three-repository Pro subset.The monitor scores recent actions and observations; a threshold calibrated on successful trajectories triggers an abort while the executor remains fixed.Source: §1 · p. 2, Figure 2 · p. 3
  • Across all four executor–benchmark settings, StopLoss has lower estimated cost per retained solve than every judge policy; on calibration-disjoint Verified/Luna runs, it reduces cost 4.5% versus completing every run while retaining 99.0% of otherwise successful runs.Savings reach 31% on the costlier Pro subset.Source: §1 · p. 2
  • On Verified/Luna, outcome supervision raises Qwen2.5-7B-Instruct’s mean single-model tier AUC from 0.491 to 0.606, with trained ensembles separating outcomes at tool step 5.Each check costs an estimated $0.00024, one to two orders of magnitude below every tested judge.Source: §1 · p. 2

2 STOPLOSS

StopLoss trains a small monitor on outcome-labeled execution windows and uses its failure score to decide when to stop a run. It calibrates a threshold on successful trajectories and leaves the executor unchanged.

2.1 TRAJECTORY WINDOWS AND OUTCOME SUPERVISION

  • At each checkpoint, the monitor reads the step count and the three most recent action–observation pairs as its local window.The paper specifies a 1,600-token sequence limit, with encoding and truncation rules in Appendix A.Source: §2.1 · p. 3
  • Final outcomes label failed-run windows ABORT, the last two successful-run steps DELIVER, and earlier successful-run windows CONTINUE.DELIVER is an auxiliary training label; runtime stopping uses only the ABORT score, and runtime inputs contain only text observed by the current checkpoint.Source: §2.1 · p. 3

2.2 LEARNING AND FAILURE SCORING

  • A LoRA-adapted 7B–8B causal language model is trained on outcome-derived labels, and its failure score is read from candidate-answer logits.Each check requires one forward pass per model without generating an answer or reasoning trace; ensemble scores are averaged.Source: §2.2 · pp. 3–4

2.3 CALIBRATION AND EARLY STOPPING

  • Calibration and deployment use checkpoints 5, 8, 11, and so on; the threshold is chosen so at most a target fraction of calibration successes would be stopped.Calibration uses each successful run’s maximum checkpoint score to represent the event that any crossing stops the run; this is an empirical constraint, not a guarantee under distribution shift.Source: §2.3 · p. 4

3 EXPERIMENTAL SETUP

The study trains monitors on GPT-generated SWE-rebench trajectories and evaluates them on SWE-bench Verified and a three-repository Pro subset with GPT and Claude executors. It compares them with same-family judges and measures stopping costs alongside retained successes, using paired repository-cluster bootstrap intervals.

3.1 DATA, EXECUTORS, AND EVALUATION COHORTS

  • Training uses 4,867 SWE-rebench tasks and 61,315 windows, while external evaluation covers 500 Verified tasks and the 266-task Python subset of SWE-bench Pro.The Pro subset spans ansible, openlibrary, and qutebrowser; evaluation uses GPT-5.6 Luna and Claude Sonnet 4.6 executors.Source: §3.1 · p. 4
  • Stopping-policy evaluation uses 425 Verified and 213 Pro tasks per executor after excluding exploratory tasks and requiring valid verdicts from both executors.Ranking cohorts differ from stopping cohorts, and Figure 1 uses the Pro/Luna stopping cohort.Source: §3.1 · p. 4

3.2 MONITORS AND COMPARISON METHODS

  • Three GPT judges assess GPT-5.6 Luna trajectories and three Claude judges assess Sonnet 4.6 trajectories, while all small monitors read both executors.Methods receive identical local window bodies within each cell; the step-count-only baseline ranks windows by step number without reading trajectory content.Source: §3.2 · p. 5

3.3 METRICS AND COST ACCOUNTING

  • Cost per retained solve charges execution through stopping and every visited check, while false-kill rate measures successful runs stopped and success retention is 1 − FK.Negative ΔCPS indicates savings versus full execution; comparisons report retention because equal nominal targets may not produce equal realized risk.Source: §3.3 · p. 5
  • Tier AUC compares failure ranking within quartiles of accumulated execution cost, while reported monitoring costs include visited checks and ensemble members.Reported per-run costs exclude training, infrastructure, one-off calibration, and offline scoring of unvisited windows.Source: §3.3 · p. 5

3.4 STATISTICAL PROTOCOL AND STUDY SCOPE

  • Pro Python uncertainty intervals warrant caution because the subset contains only three repository clusters.Intervals are conditional on trained models and do not adjust for the decision to expand full judge coverage after initial results were inspected.Source: §3.4 · p. 5

4 MAIN RESULTS

StopLoss reduces cost per retained solve while preserving many successful runs, and outcome-trained monitors provide early failure predictions at low cost. Training improves failure ranking over untuned models and step count alone; the monitor’s per-check cost is far below that of tested judges.

4.1 COST-EFFECTIVE EARLY STOPPING

  • 4.5% lower estimated cost per retained solve than full execution on Verified/Luna; on Pro, reductions are 24.2% on Pro/Luna and 30.6% on Pro/Sonnet.Success retention is 99.0% on Verified/Luna, 72.2% on Pro/Luna, and 91.6% on Pro/Sonnet.Source: §4.1 · p. 6
  • 48.1% lower cost per retained solve on Pro/Sonnet with Qwen3-8B, while retaining 83.2% of otherwise successful runs.Both alternative ensembles have lower estimated cost per retained solve than every judge policy in all four settings.Source: §4.1 · p. 6

4.2 LEARNING EARLY FAILURE RECOGNITION

  • Step-count-only tier AUC ranges from 0.471 to 0.528, below every trained ensemble in its corresponding setting.The baseline uses only the current tool-step index, without examining execution content.Source: §4.2 · p. 6
  • At tool step 5, the primary ensemble’s ordinary AUC is 0.642 on Verified/Luna and 0.645 on Pro/Luna, versus 0.494 and 0.450 for its untuned base.Training improves mean single-model tier AUC by +0.115 on Verified/Luna and +0.148 on Pro/Luna; ensembling adds approximately 0.014–0.016.Source: §4.2 · p. 6

4.3 PREDICTION QUALITY AND MONITORING COST

  • The primary ensemble costs $0.000237 per check; GPT judges cost 9–74 times as much per check, while its tier AUC on GPT-5.6 Luna trajectories is within 0.044 of each judge’s point estimate.On Claude Sonnet 4.6 trajectories, Claude Opus 5 achieves higher AUC.Source: §4.3 · p. 6

5 ANALYSIS AND DISCUSSION

StopLoss can identify failures at an early checkpoint, but score transfer and stopping behavior vary across executors and benchmarks. The cost advantage depends on monitoring overhead, avoided execution, and lost successes.

5.1 WHEN ARE FAILURES RECOGNIZED AND STOPPED?

  • At tool step 5, the primary monitor shows predictive signal on Claude Sonnet 4.6, with ordinary AUCs of 0.604 on Verified and 0.645 on Pro.Across eight correctly stopped Verified/Luna failures, the median fraction of execution cost remaining at the trigger was 64.5%.Source: §5.1 · pp. 6–7
  • At tool step 5, the primary ensemble has higher ordinary AUC than its untuned base in all four settings; later checkpoints use different subsets of runs.Figure 3 is a post-hoc evaluation before the stopping policy is applied, with pointwise 95% repository-bootstrap intervals.Source: Figure 3 · p. 8

5.2 TRANSFER OF PREDICTIONS AND STOPPING THRESHOLDS

  • At tool step 8 on Verified, ordinary AUC is 0.627 on GPT-5.6 Luna versus 0.578 on Claude Sonnet 4.6, consistent with only partial transfer of failure cues.Claude calibration changes the stopping threshold, not the learned scores or their ranking.Source: §5.2 · p. 9
  • Tier AUC is 0.622 on Verified and 0.614 on Pro, yet false-kill rates are 1.0% and 27.8%, respectively, under the frozen threshold.Successful GPT-5.6 Luna runs have a median of 16 tool steps on Pro versus 13 on Verified; longer trajectories may create more threshold-crossing opportunities.Source: §5.2 · p. 9

5.3 EXECUTION COST AND MONITORING OVERHEAD

  • Execution savings must cover monitoring overhead and lost successes; on Verified/Luna, StopLoss’s checks cost $0.44 versus $33.20 for GPT-5.6 Sol’s visited checks.Doubling the ensemble’s inference-cost proxy moves the Verified/Luna cost-per-solve estimate from −4.5% to −2.9%; at five times the charge it becomes +1.8%.Source: §5.3 · p. 9

6 RELATED WORK

StopLoss sits at the intersection of trajectory assessment and cost-efficient agent execution. It learns from final verification outcomes and observable windows, then evaluates stopping under a fixed executor using cost per retained solve.

Trajectory evaluation and early stopping

  • StopLoss learns eventual failure from final verification outcomes and short action–observation windows, without executor internals or intermediate patch-test labels.Related approaches include hidden-state probes, inexpensive execution signals, observable-trace monitors, and monitors combining terminal outcomes with patch-test supervision.Source: §6 · p. 9

Cost-efficient agent execution

  • Unlike budget-oriented methods, StopLoss decides whether to continue a run under a fixed executor using predicted failure rather than budget exhaustion.BAGEN studies remaining-resource estimation, while EET uses execution experience for early termination during patch generation and selection.Source: §6 · pp. 9–10
  • StopLoss evaluates stopping through cost per retained solve, accounting for monitoring charges and lost successes, and compares the trade-offs with prompted judges.Source: §6 · p. 10

7 LIMITATIONS AND CONCLUSION

StopLoss uses verifier outcomes from completed trajectories to stop likely failures during execution without changing the executor. The authors report lower cost per retained solve, while noting limits on evaluation scope and possible cross-executor generalization.

7 LIMITATIONS AND CONCLUSION

  • Training exclusively on GPT-generated trajectories may limit generalization to Claude; evaluation also covers Python tasks and only three SWE-bench Pro repositories.The authors leave broader evaluation across languages and benchmarks to future work.Source: §7 · p. 10
  • StopLoss makes early stopping pay for itself in the evaluated settings, with savings of 4.5% on Verified/Luna and up to 31% on Pro, or 48% with Qwen3-8B.The authors also state that saved execution must exceed monitoring cost and the cost of successes stopped.Source: §7 · p. 10
  • The authors used generative AI tools for language polishing and textual revisions, and take responsibility for the final paper content.Source: AI USE STATEMENT · p. 10
  • An early-stop policy can silently reduce an agent’s success rate, so the experiments report false kills and retained solves alongside resource savings.Source: ETHICS STATEMENT · p. 10

A PROMPTS, LABELS, AND TRAINING DETAILS

This appendix details how StopLoss prompts are constructed, labels are trained and scored, and model and judge configurations are recorded. It also specifies training optimization and evaluation constraints.

A PROMPTS, LABELS, AND TRAINING DETAILS

  • Prompt construction truncates actions and observations, removes middle text when needed to meet a 1,600-token limit, and rejects any remaining overlong sequence.Actions are truncated to 700 characters and observations to 900; the step-count header and monitor question are retained.Source: A PROMPTS, LABELS, AND TRAINING DETAILS · p. 13
  • The monitor prompt presents recent action/observation blocks and asks whether the agent should continue, deliver, or abort, with a one-word answer.Each window contains up to three recent tool-step blocks.Source: A PROMPTS, LABELS, AND TRAINING DETAILS · p. 13
  • The monitor reads its ABORT probability from the softmax over first answer tokens; outcome labels do not certify that a run could not recover under another continuation.Source: A PROMPTS, LABELS, AND TRAINING DETAILS · p. 13
  • Judges receive the same window body as monitors but use a probability-estimation question and provider-specific formatting; configurations remain frozen during full-coverage evaluation.The 1,600-token limit does not restrict judge outputs or reasoning.Source: A PROMPTS, LABELS, AND TRAINING DETAILS · p. 13
  • Recorded configurations distinguish trajectory generation from judging, including cases where one model serves both roles, and set agent and verifier timeouts separately.The recorded scaffold is Terminus-2 version 2.0.0 in Harbor 0.17.0.Source: A PROMPTS, LABELS, AND TRAINING DETAILS · p. 13, Table 3 · p. 14
  • Training supervises every target-answer token with causal language-model loss, while masking prompt and padding labels; scoring uses the first-token readout.The main recipe uses zero label smoothing; the shared encoder checks prompt/answer token-boundary alignment.Source: A PROMPTS, LABELS, AND TRAINING DETAILS · p. 14
  • All three model bases use one epoch of rank-16 LoRA, AdamW, cosine decay, and 3% warm-up, with validation selecting the checkpoint by failure AUC.The recipe uses learning rate 10^-4, weight decay 0.05, and effective batch size 16.Source: A PROMPTS, LABELS, AND TRAINING DETAILS · p. 14

B.1 DATA CONSTRUCTION AND SPLITS

The appendix describes the SWE-rebench training split, external stopping cohorts, and calibration procedure. Thresholds are derived from maximum checkpoint scores on successful calibration runs.

B.1 DATA CONSTRUCTION AND SPLITS

  • The SWE-rebench corpus has 5,086 tasks and 63,173 windows; held-out repositories reserve 219 tasks, leaving 4,867 fitting tasks and 61,315 windows.External Verified and Pro tasks are disjoint from both partitions.Source: §B.1 · p. 14
  • The independent stopping cohort contains 425 Verified and 213 Pro tasks shared across executors, excluding 122 early exploratory tasks and Claude calibration tasks.Source: §B.1 · p. 14

B.2 THRESHOLD CALIBRATION

  • Calibration uses successful runs reaching at least one checkpoint, yielding 348 GPT and 378 Claude checkpoint windows.Two of 76 successful Claude runs do not reach a checkpoint and are excluded from threshold computation.Source: §B.2 · p. 15
  • For target rate ρ, each method sets its threshold just above the selected order statistic of per-run maximum checkpoint scores, stopping at most ⌊ρn⌋ calibration successes.Tied maxima are not split and can make the rule more conservative.Source: §B.2 · p. 15

B.3 EVALUATION COVERAGE

The appendix distinguishes ranking coverage from the independent stopping cohort and describes a later full-coverage extension. It also specifies score checks and the frozen cost-accounting basis.

B.3 EVALUATION COVERAGE

  • The main policy comparisons use an independent stopping cohort; runs without checkpoints remain in stopping denominators but contribute no ranking windows or monitor calls.Infrastructure or verifier failures without verdicts are excluded rather than labeled as model failures.Source: §B.3 · p. 15, Table 4 · p. 15
  • The full-coverage extension was decided after initial results were inspected, while the primary monitor, configurations, windows, thresholds, prices, and comparison rules stayed fixed.It reports 24,153 required judge-window scores, including newly collected and reused accepted scores.Source: §B.3 · p. 15
  • Accepted judge scores require successful parsing, normal completion, usage records, and an allowed backend identity; truncated answers are rejected.Verifier labels, score records, prompt and configuration hashes, and call ledgers are checked for consistency.Source: §B.3 · p. 15

C.1 TOKEN PRICING AND EXECUTION ACCOUNTING

  • The token schedule charges full input rates for cache reads and writes and includes reasoning tokens in output; reported comparisons use frozen list prices, not subscription charges.Tool-prefix spend is aligned to ledger calls, surviving runs pay trailing non-tool calls, and aborts avoid future execution and checks.Source: Table 5 · p. 16, §C.1 · p. 16

C.2 MONITOR INFERENCE COST

The monitor’s scoring cost is estimated with an amortized GPU-time proxy, while judge predictions cost substantially more. Collection accounting distinguishes known charges from unavailable failed-call usage, and sensitivity analysis tests inflated monitoring charges against stopping savings.

C.2 MONITOR INFERENCE COST

  • The monitor-cost proxy averages scoring-loop wall time over 27,001 windows; it excludes model loading, tokenization, deployment, serving overhead, and utilization.The five Qwen2.5-7B-Instruct adapters average 0.211–0.215 seconds per window; this proxy is not online request latency or a latency percentile.Source: §C.2 · p. 16
  • Judge costs range from $0.0018 to $0.0772 per prediction, approximately 8–326 times the primary ensemble proxy of $0.000237 per check.Across trained seeds, ensemble costs per check are $0.0002367 for Qwen2.5-7B-Instruct, $0.0002289 for Qwen3-8B, and $0.0001893 for Qwen2.5-Coder-7B-Instruct.Source: §C.2 · pp. 16–17

C.3 COLLECTION ACCOUNTING

  • The full-coverage extension added 16,029 valid window scores with $319.57 in known recorded costs, while failed attempts with unavailable usage remain unpriced, not zero-cost.Full-coverage collection charges differ from policy replay charges, which cover only checks visited before stopping or completion.Source: §C.3 · p. 17, Table 7 · p. 17

C.4 SENSITIVITY TO MONITORING COST

  • Verified/Luna’s cost-per-solve saving disappears at approximately 3.83× baseline monitoring cost; both Pro settings retain savings at 10× with success retention unchanged.Verified/Sonnet stops no runs and therefore only adds monitoring cost.Source: §C.4 · p. 18, Table 8 · p. 18

D PREDICTION AND STOPPING DIAGNOSTICS

The diagnostics illustrate correct stops, missed failures, and false kills at the same threshold, alongside a table reporting single-model tier AUC for every trained seed.

D PREDICTION AND STOPPING DIAGNOSTICS

  • Table 9 reports single-model tier AUC for every trained seed on the full ranking cohort.Source: Table 9 · p. 19
  • At threshold 0.876, the illustrative correct stop at SymPy-12419 avoided $0.225 of a $0.306 execution cost after a step-17 failure score of 0.894.The logged continuation failed verification; the selected cases illustrate behavior rather than error prevalence.Source: D PREDICTION AND STOPPING DIAGNOSTICS · p. 19
  • The illustrative SymPy-20428 failure was missed: its maximum checkpoint score was 0.550 at step 11 despite passing local tests and readiness to finish.The completed run failed verification.Source: D PREDICTION AND STOPPING DIAGNOSTICS · p. 19
  • The illustrative scikit-learn-25931 false kill scored 0.955 at step 20 during terminal-recovery attempts, although the completed run passed verification.All three illustrative cases use threshold 0.876.Source: D PREDICTION AND STOPPING DIAGNOSTICS · p. 19

Related papers

Loading 2610.25525v1…