PaperlayerView exact revision on arXiv

Source-linked AI summary

Executable Prefix Values for Locating Agent Failures

Bingyu Shen, Jiaqi Tang, Lan Wei, Yi Sheng, J. Jenny Li, Boyang Li

arXiv:2610.17633v1cs.AIcs.CL

TL;DR

The paper asks whether first-mistake labels identify causes or visible symptoms, a distinction that matters for debugging and restart decisions. It estimates success probability from reconstructed intermediate states and uses the resulting curves to evaluate judge labels and restart policies. It finds that labels often coincide with no drop in success probability at all.

  • Problem

    First-mistake labels are commonly treated as ground truth, but their position relative to the cause of failure and visible evidence has not been measured.

    Source: §1 · p. 1

  • Method

    The paper estimates prefix success probabilities by repeatedly restarting agents from reconstructed intermediate states of failed SWE-bench Verified trajectories, then compares judge labels and restart policies with those curves.

    Source: §1 · pp. 1–2

  • Results

    For 66% of labels, the agent’s success probability does not drop at the labeled step, and the labels scatter around rather than consistently identify the decisive drop.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that controllers need the point where the agent’s own prospects collapse, which first-mistake labels do not reliably provide.

    Source: §1 · p. 2

    The evaluation covers two policies from one family on one scaffold and benchmark subset, and prefix value is relative to the agent and its budget.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Work on debugging LLM agents treats the "first mistake" marked by a human or LLM judge as ground truth for where a trajectory failed. We ask whether such labels point at causes or symptoms, and make the distinction executable: we restart the agent from reconstructed intermediate states of a failed trajectory and estimate its prefix value V(h_t), the probability of success from each step, with a certified point of no return (PNR) for a failure. On 150 SWE-bench Verified tasks with two frozen open-weight agents, we build such curves for 47 and 60 failed trajectories and probe every step labeled by four frontier judges under the Traverse rubric. We find that the labels do not mark what a controller needs. Judges of different lineages agree on the exact step in only 21/57 cases, and the newest judges flag mistakes in 8/10 and 10/10 successful runs. For 66% of the labels, the agent's success probability does not drop at the labeled step; only 7/188 answers name the decisive step. Restarting at the measured restart point raises expected success from 0.44 to 0.57 offline, and real restarts confirm the advantage, whereas restarting at a judge's label does not (-0.08 to -0.03). We release our code, curves, and labels.

1 Introduction

The introduction argues that first-mistake labels may identify visible symptoms rather than the earlier causes that determine failure. It makes this distinction measurable through an agent’s prefix success probability and uses that signal to assess judge labels and restart decisions.

1 Introduction

  • Current first-mistake judgments mark the earliest erroneous step in hindsight, without measuring whether the task was lost there or whether the label instead marks where evidence appears.The introduction notes that benchmarks, verifiers, and localizers commonly rely on these labels.Source: §1 · p. 1
  • The prefix value V(h_t) is the probability of success if the agent continues from history h_t with its remaining budget, and its drops indicate lost prospects.The paper estimates V by repeatedly restarting the agent from reconstructed intermediate states and evaluating outcomes with task tests.Source: §1 · pp. 1–2
  • The study builds V-curves for failed trajectories from two frozen open-weight agents on SWE-bench Verified and places four frontier judges’ first-mistake labels on them.It asks where labels fall relative to value drops, how failures and decisive steps are distributed, and how restart policies perform.Source: §1 · p. 2
  • For 66% of judge labels, the agent’s success probability does not drop at the labeled step, so the labels do not carry the decisive signal a controller needs.The paper argues that a controller needs the point where the agent’s own prospects collapse.Source: §1 · p. 2

2 Related work

Prior work studies first-mistake labeling and failure attribution across agent settings, while inference-time controllers use predictions to decide whether to continue, restart, or halt. This paper positions executable prefix values as a way to measure what judgment-based labels capture.

First-mistake labels

  • Traverse (Rahman et al., 2026) labels earliest erroneous steps, evaluates frontier judges, and trains Scout to reproduce those labels.Who&When (Zhang et al., 2025) and LongRCA (Zhang et al., 2026) address attribution in multi-agent and long-horizon settings, while AgentDebug (Zhu et al., 2025) builds a taxonomy and labeled benchmark and uses root-cause feedback for recovery.Source: §2 · p. 2
  • Prior attribution work uses judgments as its reference, whereas this paper asks what those judgments measure against an executable signal.The paper also places V-curves alongside targeted interventions and prefix resampling as a non-targeted intervention used for measurement.Source: §2 · p. 2

Inference-time control

  • Meta-reasoning controllers (Dahal et al., 2026) and Fail-Fast/Restart-Smart (Wang et al., 2026) use predictions to decide which runs to continue, restart, or halt.Both approaches need the value of an agent prefix.Source: §2 · p. 2

3 Definitions

The definitions frame failure localization in terms of an agent’s success probability from each history, then define value drops, failure points, and restart value relative to the agent and its remaining budget.

Setting

  • A run is a stochastic policy’s sequence of actions and observations under step budget B, and replayability assumes the environment state after each history can be reconstructed exactly.The run ends with a binary outcome from the task evaluator.Source: §3 · p. 2

Prefix value

  • Prefix value V(h_t) is the probability of success when the agent resumes from history h_t with the remaining budget.At the initial history it equals the task’s pass rate under the policy; at the terminal history it equals the run outcome.Source: §3 · p. 2

Value drop and responsibility share

  • A step’s value drop and responsibility share grade how much of the lost success probability is assigned to that step.On a failed run, the shares sum to one and can be distributed across steps; a negative share marks partial recovery.Source: §3 · pp. 2–3
  • In a deterministic environment, the value drop is the negative advantage of the action actually taken over the agent’s average behavior at that history.The drops telescope across steps.Source: §3 · p. 2

Failure points

  • The decisive step is the largest resolved drop, while onset and point of no return use statistical criteria on estimated prefix values.A task is a cliff when its loss concentration C is at least 0.5 and erosion otherwise; tasks with initial pass rate at most 1/8 are treated as doomed from the start.Source: §3 · p. 3

Restart value and restart point

  • Restart value estimates fresh-run success within the remaining budget, and a restart point compares that value with continuing from the current history.The PNR is a certified, task-independent counterpart because prefix value is below the threshold there regardless of restart value.Source: §3 · p. 3

4 Method

The method reconstructs intermediate trajectory states, resumes the agent from them, and estimates success probabilities with an adaptive sampling schedule. It checks replay and estimation behavior and uses controls to distinguish value loss from budget effects.

Replay and anchors

  • Replay accepts a trajectory only when re-execution reproduces its observations and state fingerprints, dropping and replacing trajectories at the first mismatch.Anchors are saved as container images, while non-anchor states are reached by replaying from the nearest earlier image under the same checks.Source: §4 · p. 3

Continuations

  • Each continuation resumes the agent with its recorded messages, remaining budget, and a fresh sampling seed, then evaluates its submission with the official harness.Continuations are keyed by trajectory, step, and seed and are not run twice.Source: §4 · pp. 3–4

Schedule

  • The adaptive schedule screens anchors, bisects large drops, confirms candidate drops, certifies a PNR, and directly probes labeled steps; a curve costs 331 continuations on average.Confirmation uses up to 32 cumulative samples at both ends of a bracketed drop, while PNR certification uses a one-sided 95% Wilson upper bound below 0.15.Source: §4 · p. 4

Estimation

  • The estimator uses continuation success fractions with Wilson intervals and reads drops from a weighted isotonic non-increasing fit to reduce spurious positive drops from noise.The decisive step is the largest fitted drop between consecutive resolved points; the estimator was validated on 2,000 synthetic curves with known decisive steps.Source: §4 · p. 4

Controls and exclusions

  • Controls test base-run agreement, repeat screening with fresh seeds, and rerun screening anchors with the full budget to separate value loss from budget loss.Containers run without network under a memory watchdog, and killed or unevaluable continuations are rerun with the same key.Source: §4 · p. 4

5 Experimental setup

The study freezes an agent configuration, samples and filters SWE-bench Verified tasks, and selects trajectories before viewing curves. Four judges and separate annotator-protocol sources label selected runs.

5 Experimental setup

  • The frozen setup used mini-swe-agent 2.4.6 with Qwen3.6-27B-FP8, default SWE-bench configuration and prompts, and a step budget of 100.The same frozen parameters applied to base runs and continuations.Source: §5 · p. 4
  • 150 SWE-bench Verified tasks were each run eight times, yielding 1,200 runs evaluated with the official harness; empty patches failed without a run.The team sampled 170 instances and kept the first 150 whose images pulled cleanly.Source: §5 · p. 4
  • Tasks with two to six successes out of eight were selected; failed runs were stratified by length and end type, with ten successful decoys included blind.Selection was fixed before any curve was seen; capped runs all fell in the top length tercile.Source: §5 · p. 4
  • Four frontier judges each labeled selected trajectories and decoys once, with three-pass self-consistency checks on a 20-trajectory subset; malformed answers were unparsed.GPT-6 Luna was a contrast and excluded from consensus.Source: §5 · pp. 4–5
  • Claude Opus 5.5 and a software-engineering-background human independently labeled the same 40 trajectories from blind static pages under the Traverse rubric.Claude's labels were reported separately and entered no consensus or probe.Source: §5 · p. 5

6 Results

Across two agents, judges’ first-mistake labels often miss where success probability falls, while restarting at a measured restart point improves expected success. The results also show that decisive-step estimates and label-based rankings have important limits.

6.1 The policy and the pool

  • The study selects 47 failed trajectories from 84 failed band runs across 21 tasks, with replay reconstructing 70 of 94 selected, decoy, and spare trajectories.The base agent resolves 0.68 of runs, with only 21 tasks in the informative band.Source: §6.1 · p. 5

6.3 Labels against the curves

  • 67% of probed labeled steps show no drop, and only 7/188 judge answers identify the decisive step; judges also disagree across lineages.The annotators’ labels also often miss the drop: V does not drop at 9/12 model labels and 5/10 human labels.Source: §6.3 · p. 5, §6.2 · p. 5

6.4 A close reading of four trajectories

  • Decisive steps often involve adopting a wrong diagnosis during an innocuous read or probe, while judges may mark its later statement or resulting action.In a successful decoy, GPT-6 judges also flagged a step where no value drop existed.Source: §6.4 · p. 6

6.5 What a restart policy gains

  • 0.44 to 0.57 (+0.13): restarting at the measured restart point raises expected success offline, and real restarts beat continuing.The online check succeeded in 0.39 of cases against a predicted 0.45 and a continuing value of 0.26.Source: §6.5 · p. 6

6.6 Cheaper estimates, and whether judges can guide them

  • 28%: even the full schedule identifies the decisive step exactly in only 28% of replications, while judge-guided probing ends near full-schedule cost at the same accuracy.The full schedule costs 276 continuations; probing GPT-6.1 Sol’s label first costs 255 and verifies that label in only 40/940 replications.Source: §6.6 · p. 7

6.7 Failure shapes and the decisive step

  • The decisive step occurs early (median t*/T = 0.28), often before the first edit, and the full-budget check agrees on its location within three steps in 6/10 trajectories.Of decisive steps, 81% are reads and 6% edits; the point of no return follows t* by a median of six steps.Source: §6.7 · p. 7

6.8 Ranking localizers

  • The best value-capture score is 0.16 for GPT-6 judges within one step, versus 0.04 for a random step; agreement-based and value-based localizer rankings diverge.Agreement rewards matching other judges rather than locating where the task was lost.Source: §6.8 · p. 7, §7 · p. 8

6.9 A second policy

  • On Qwen3.5-9B, every judge-label restart policy loses, while the restart point gains +0.05; 58% of continuations exhaust the remaining budget.The authors note that this second policy fails the budget-control check and leave its restart evaluation to the appendix.Source: §6.9 · p. 8, §6.10 · p. 8

Limitations

The study bounds its conclusions to two policies of one family, one scaffold, and one benchmark subset, while documenting protocol choices and an adaptive schedule for measuring trajectories.

Limitations

  • The evidence is limited to two policies of one family, one scaffold, and one benchmark subset, with V relative to each agent and its budget.The offline restart evaluation assumes a value-based policy never restarts a run that would have succeeded; the judges are from one vendor, and only one human annotator was used.Source: Limitations · p. 8

Contents of the appendix

  • The appendix documents protocol decisions, schedule and method checks, judge analyses, trajectory case studies, estimator settings, restart evaluations, and all measured curves.It also includes the second policy’s tables and curves.Source: Contents of the appendix · p. 10

A Protocol decisions

  • Protocol decisions include raising the per-task trajectory cap from two to three, adding GPT-6 Astra after an initial look, and resampling watchdog-killed continuations.The study set for each policy comprises the trajectories resolved when its schedule, fixed at selection time, was stopped.Source: A Protocol decisions · p. 10

B The schedule

  • The schedule allocates continuations adaptively across screening, bisection, confirmation, point-of-no-return certification, and judge-label probes.A curve costs 331 continuations on average, including 55 label probes; continuations are reused across stages by trajectory, step, and seed.Source: B The schedule · p. 10, Table 3 · p. 10

C Checks on the method: criteria

The method checks meet their stated criteria, and the accompanying analyses describe the second policy, judge agreement, trajectory examples, prompt, estimator settings, and task-selection scope.

C Checks on the method: criteria

  • All method checks pass: replay is 73%, harness disagreements are 0/46, empty-history reproduction is 5/5, median screening gap is 0.00, cap hits are 31% against a <35% threshold, and confirmed-drop or PNR coverage is 47/47.The cap-hit rate is discussed with analyses repeated without the L2 stratum.Source: C Checks on the method: criteria · pp. 10–11

D Judges on the second policy

  • On Qwen3.5-9B, GPT-6 judges flagged 10/10 successful decoys each, while GPT-5.5 and OpenAI o3 flagged 3/10 each.The second policy’s 60 failed trajectories and 10 decoys show a repeated lineage pattern; its runs were longer and more often capped.Source: D Judges on the second policy · p. 11, Table 5 · p. 11

E Judge pairs

  • Across 57 judged trajectories, the GPT-6 pair agreed exactly on 52/57, while the best distinct-lineage pair, GPT-5.5 and OpenAI o3, agreed on 21/57.Agreement tracked early-versus-late labeling style rather than model-family name.Source: Table 6 · p. 11, E Judge pairs · p. 11

F Four trajectories, read closely

  • Case studies show value loss can precede a fully stated diagnosis, spread over twenty steps, or coincide with a probe-script traceback.In matplotlib, the agent’s value dropped when it committed to a wrong hypothesis; in sympy, value eroded while it pursued an idea about cancel().Source: F Four trajectories, read closely · pp. 11–12

G Judge prompt

  • The Traverse prompt treats an error as a state-changing wrong Thought or Action that contributes to an incorrect outcome; observations provide context, not errors.Judges return the earliest error’s 0-based index, or −1 for no mistake.Source: G Judge prompt · p. 12

H Frozen estimator parameters

  • Frozen settings trade sampling cost against resolution: Kscreen=8, Kconfirm=32, Kpnr=24 with ϵ=0.15, and Klabel=16.Confirmed drops have an estimated standard error of about 0.08; curves are coarse away from detected drops and precise at points raised to their required sample counts.Source: H Frozen estimator parameters · p. 12, Table 7 · p. 12

I The base-run pool

  • Only 21 of 150 tasks fall in the informative band of 2–6 successes across eight base runs; pass@8 is 0.51 against pass@1 0.68.The study’s curves use failures from this band; tasks outside it still contribute to restart estimates and decoy calibration.Source: I The base-run pool · p. 13, Table 8 · p. 13

J All trajectories

The trajectory tables place judges’ first-mistake labels alongside the decisive step and certified point of no return. Across the trajectories, failures are decided before the agent stops, and the schedule averages 331 continuations per curve.

J All trajectories

  • The decisive step occurs at a median 28% of each trajectory; all 47 failures are decided before the agent stops, and the point of no return follows by a median of 6 steps.Four trajectories lose the task within five steps, none only at the final step, and the upper quartile gap from the decisive step to the point of no return is 30 steps.Source: J All trajectories · p. 13, Table 9 · p. 14
  • The 47 trajectories comprise 17 L0, 12 L1, and 18 L2 runs, with lengths of 40–100 steps and a median length of 75.Their fitted starting values range from 0.05 to 1.00, with a median of 0.50.Source: J All trajectories · p. 13, Table 9 · p. 14
  • 40 of 47 trajectories show an explicit failure signal after the first edit; the others fail silently, usually by submitting a patch that does not resolve the issue.The tables list each source’s first-mistake label next to the decisive step and point of no return, with “none” denoting no-mistake answers and a dot denoting no label.Source: J All trajectories · p. 13, Table 11 · p. 15, Table 12 · p. 15

K Schedule statistics

  • Each curve averages 24.1 points and costs 331 continuations; screening and label probes dominate, while confirmation and point-of-no-return stages add the rest.Most trajectories have three or four distinct labels because the four judges rarely agree; cliffs and erosions cost about the same to measure.Source: K Schedule statistics · p. 16, Table 13 · p. 16

L Replay

Replay reconstructed 70 of 94 trajectories exactly, while non-replayable runs were excluded rather than patched. The reported mismatches were usually in observations, not filesystem fingerprints.

L Replay

  • 70 of 94 trajectories replay exactly, and the L1-submit stratum has the lowest replay rate at 60%.The tables report replay by role and stratum alongside mismatch types and the comparison levels used for observations.Source: L Replay · p. 16, Table 14 · p. 16, Table 15 · p. 16
  • The first mismatch occurs at a median step 17 and almost always lies in an observation rather than the filesystem fingerprint.Non-deterministic printed values, such as random tokens or hash-based ordering, explain many mismatches; the authors drop these trajectories because continuing from a different state would measure a different history.Source: L Replay · p. 16
  • Fixed normalization makes exact replay attainable for about three quarters of real SWE-bench trajectories, while reported replay failures remain excluded rather than repaired.Sorted-line and character-multiset comparisons absorb reorderings when directory listings or test-output content is otherwise identical.Source: L Replay · p. 16, Table 14 · p. 16, Table 15 · p. 16

M Judge agreement in detail

Judge agreement forms two lineages rather than four independent opinions: the GPT-6 judges align closely, while other judges more often answer “no mistake” and label later. Within-judge sampling noise also contributes to cross-lineage disagreement.

M Judge agreement in detail

  • The four main judges are unanimous on the exact step in 3 of 57 trajectories, while at least two fall within one step in 56.The pairwise table reports exact, within-one-step, and within-three-step agreement across the 47 failed trajectories.Source: M Judge agreement in detail · p. 16, Table 16 · p. 17
  • One pass over the study set cost US$53 across the judges, including three self-consistency passes on a 20-trajectory subset.The judge tables also report label counts, format errors, decoy flags, label positions, and self-consistency.Source: M Judge agreement in detail · pp. 16–17, Table 17 · p. 17
  • The two GPT-6 judges agree far more often with each other than with GPT-5.5 or OpenAI o3, forming a distinct lineage from the looser second group.GPT-5.5, OpenAI o3, and the contrast model more often answer “no mistake” and label later in trajectories.Source: M Judge agreement in detail · pp. 16–17
  • GPT-6 judges return the same answer across three passes on most trajectories, whereas GPT-5.5 and OpenAI o3 do so on a minority.The authors conclude that some cross-lineage disagreement reflects within-judge sampling noise rather than stable differences of opinion.Source: M Judge agreement in detail · p. 17

N Controls

The controls test whether continuations reproduce the base pass rate, repeat screening estimates, and respond to remaining-budget constraints. They identify a measured late-decision bias on long runs, with its effect checked without the L2 stratum.

N Controls

  • Continuations from the empty history reproduce the base pass rate, with no row rejected by Fisher’s exact test at α = 0.05.The sanity check compares empty-history continuation successes with the task’s base runs.Source: N Controls · p. 17, Table 18 · p. 18
  • At K = 8, two screenings of the same history agree exactly in 21 of 30 anchors, and only one anchor differs by more than 0.25.The median absolute gap across the three retested anchors is 0.00; the authors note that a 3/8 gap is within binomial noise at K = 8.Source: N Controls · p. 17, Table 19 · p. 18
  • Using the full budget instead of B − t never moves the decisive step earlier and moves it later in 4 of 10 trajectories.In two cases, the step moves to the last step because a run still viable under B runs out of steps under B − t; the authors interpret late decisions as losses under the remaining budget.Source: N Controls · p. 17, Table 20 · p. 18
  • The known late-decision bias on long runs is bounded by repeating the analysis without the L2 stratum.The conclusion states that the bias’s direction and size are measured, with its effect on the main claims bounded by that analysis.Source: N Controls · pp. 17–18

O Labels against the curves, in full

The judges’ first-mistake labels usually do not coincide with a drop in the agent’s success probability, and their misses vary by lineage. Labels also rarely mark the first visible symptom, and the pattern persists without the L2 stratum.

O Labels against the curves, in full

  • H1 shares range from 63% to 91% across sources: GPT-6 judges tend to miss early, while GPT-5.5 and OpenAI o3 tend to miss after the point of no return.The sources differ mainly in the direction of their miss, not whether they miss.Source: O Labels against the curves, in full · p. 18
  • On symptomatic runs, almost no labels fall within one step of the first symptom, and only a minority fall within one step of the decisive step.Judges usually mark where a wrong idea is stated or acted on, which the curve dates earlier.Source: O Labels against the curves, in full · p. 18
  • Without the L2 stratum, pooled H1 is 85% (83/98), compared with 81% on the full set.This repeat analysis covers 29 trajectories whose base run submitted.Source: O Labels against the curves, in full · p. 18, Table 22 · p. 19
  • For about two thirds of every source’s labels, V does not fall at the labeled step; pooled H1 is 91% on cliffs and 77% on erosion.The table reports labels against the curves for every source and quantity.Source: O Labels against the curves, in full · p. 18, Table 21 · p. 19

P Restart policies, in full

The restart evaluation favors policies driven by measured value over judge labels: measured restart points act earlier on failed runs, while judge labels can arrive too late or trigger on successful runs. Online checks, cheaper schedules, and localizer rankings further show that capturing value matters more than exact-step agreement.

P Restart policies, in full

  • Value-based policies restart failing runs early and outperform judge-label policies, which restart late and can forfeit successful decoys; the pattern persists without L2.The restart policies are evaluated on 47 trajectories, and the no-L2 repeat reports a +0.17 gain at the restart point versus +0.13 on the full set.Source: P Restart policies, in full · p. 20, Table 23 · p. 20, Table 24 · p. 20

Q Online restart check

  • Observed success was 0.39 versus 0.45 predicted and 0.26 for continuing; fresh restarts still beat continuing, while offline estimates were optimistic.The online check used 240 fresh runs across 30 trajectories with restart points.Source: Q Online restart check · p. 22, Table 28 · p. 22

R Cheaper schedules, in full

  • Sparse schedules matched the full schedule within noise at lower cost, while value captured was more forgiving than exact decisive-step agreement; cliffs were located more reliably than erosions.Judge-guided schedules usually fell back to local or full searches, and erosions make the decisive step an expensive, ill-defined target.Source: R Cheaper schedules, in full · pp. 22–23, Table 29 · p. 23, Table 30 · p. 23

S Ranking localizers, in full

  • Rankings by VC1 and A1 correlate at Kendall’s τ of 0.56 (permutation p = 0.04) but select different winners; value captured corresponds to what a controller gains.The table also compares localizers by agreement with t∗, reported for reference because that target is unstable on erosions.Source: S Ranking localizers, in full · p. 23, Table 32 · p. 24

T Synthetic validation of the estimator

Synthetic curves with known shapes and decisive steps test whether the estimator detects and classifies drops, certifies points of no return, and avoids false positives on flat curves. It performs well on cliffs, while single-step localization is inherently less reliable for erosion curves.

T Synthetic validation of the estimator

  • Overall shape accuracy was 92% on 2,000 synthetic curves; on cliffs of size ≥0.4, median |t_hat−t∗| was 0 and 94% were within one step.The estimator’s average cost was 321 continuations per curve.Source: T Synthetic validation of the estimator · p. 24, Table 33 · p. 24
  • Cliffs and plateau-cliffs were detected and classified reliably, with a point of no return certified in all; erosions were classified when no single drop was large enough to detect.Erosions still received a certified point of no return because their value reached zero.Source: T Synthetic validation of the estimator · p. 24
  • Flat curves yielded no certified point of no return, and only a few percent had a detected drop, matching the estimator’s false-positive rate under sampling noise.The synthetic schedule’s cost matched the real cost and set the budget before continuations began.Source: T Synthetic validation of the estimator · p. 24

U The annotators: Claude Opus 5.5 and a human

The annotator-protocol model and the human annotator are cautious about flagging successful runs, but their labels still often miss the curve’s decisive interval. The results distinguish avoiding false alarms from locating where the agent’s success probability changes.

U The annotators: Claude Opus 5.5 and a human

  • Claude Opus 5.5 answered “no mistake” on 11 failed trajectories and flagged none of the 10 decoys under the annotator protocol.The protocol used static pages, alternative candidates, confidence ratings, random ordering, and no outcome information.Source: U The annotators: Claude Opus 5.5 and… · p. 24
  • Against the curves, Claude’s labels were after the point of no return on 7 trajectories and inside the decisive interval on 1; most probed labels showed no value drop.The subset included 30 resolved trajectories, and the protocol model was not closer to the drop than the judges.Source: U The annotators: Claude Opus 5.5 and… · p. 24
  • The human flagged 1 of 10 decoys and answered “no mistake” on 7 of 30 failed runs, but only 2 of 23 labels fell inside the decisive interval.The human agreed most with Claude Opus 5.5, at 15/30 exact; V did not drop at 5 of 10 probed labels.Source: U The annotators: Claude Opus 5.5 and… · p. 25, Table 35 · p. 25

W All V-curves

Across frozen trajectories and the second policy, executable V-curves characterize how failures unfold and assess whether first-mistake labels identify decisive loss or the point at which failure becomes irreversible. The section also examines value-based versus judge-label restarts and limits involving replay, localization, judge agreement, and cap hits.

W All V-curves

  • Cliffs show a single decisive step at a median 35% of the run, while erosion point-of-no-return steps lag t* by a median 13 steps, versus 2 on cliffs.Nine of the 13 wide decisive intervals were erosions.Source: W All V-curves · p. 26
  • Of 47 curves, 17 were cliffs and 30 erosions; 26 stayed flat until the decisive interval, 16 had lost at least 30% before the largest drop, and 13 intervals spanned multiple steps.No curve lost the task only at its last step.Source: W All V-curves · p. 26
  • A curve starts at the task’s pass rate under the policy; its shaded interval marks the largest loss, while the dotted point of no return marks when prospects fall below ϵ.A label after the dotted line marks an event that could no longer matter; a drop before the first edit indicates a mistaken diagnosis before code changed.Source: W All V-curves · p. 26
  • Only 3 of 47 curves had a main-judge label inside the decisive interval; 10 had every label after it and 5 had every label before it.The gallery shows that labels rarely sit on the shaded decisive interval.Source: W All V-curves · p. 26, Figure 4 · p. 27, Figure 5 · p. 28
  • On django-11555 s5, raw estimates recovered by 0.75 from step 60 to 70 with nonoverlapping 95% intervals, a genuine recovery flattened by the monotone fit.The decisive step was 86, after the first edit at 52; the point of no return was 93.Source: W All V-curves · p. 27

X.1 The base-run pool

  • Only 44 of 150 tasks fall in the informative 2–6-success band; pass@8 is 0.09 versus pass@1 0.34, and the second policy fails the cap-hit check.Across 60 trajectories, decisive steps occur at a median 27% of trajectory length; curves average 25.6 points and cost 345 continuations, with cost driven by length and detected drops. The curves are a replication of label results, and restart evaluations carry the cap-hit caveat.Source: §X.1 · p. 31, §X.2 · p. 31, X The second policy, Qwen3.5-9B · p. 31

X.4 Replay

  • Replay exactly matches 80 of 111 trajectories; non-replayable cases largely diverge on nondeterministic observations and are dropped rather than patched.After back-fills, 60 of 81 selected trajectories remain for Stage 4, preserving a stratified sample of the band’s failures.Source: §X.4 · p. 34, Table 43 · p. 34, Table 44 · p. 34

X.5 Judge agreement in detail

  • The four main judges are unanimous on the exact step in 8 of 70 trajectories, with two judge lineages differing in label timing, abstention, and self-consistency.At least two judges label within one step in 68 trajectories, but agreement across lineages is less reliable than within-lineage agreement.Source: §X.5 · pp. 34–35, Table 45 · p. 35, Table 46 · p. 35

X.6 Controls

  • Controls reproduce base pass rates and show screening variability within binomial noise, but using full budget B moves the decisive step later in 10 of 10 trajectories.In four cases it moves to the last step, so a late decisive step under B −t means failure under the remaining budget, not necessarily a reasoning failure.Source: §X.6 · p. 35, Table 47 · p. 35, Table 48 · p. 36

X.7 Labels against the curves, in full

  • At labeled steps, success probability does not fall for about two thirds of every source’s labels; pooled H1 shares are 69% on cliffs and 80% on erosions.Without the L2 stratum, the pooled H1 share is 67% (36/54), versus 76% on the full set.Source: §X.7 · p. 36, Table 50 · p. 36, Table 51 · p. 37

X.8 Restart policies, in full

  • On the full set, the restart point gains +0.05, while judge policies without L2 gain -0.16 to -0.04.Without L2, the restart-point gain is +0.02; value-based policies restart early, while judge policies often restart late, after value has decayed.Source: §X.8 · p. 37, Table 56 · p. 38

X.9 Cheaper schedules, in full

  • Even the full schedule identifies the decisive step exactly in only 30% of replications, while sparse schedules match it within noise and capture value more forgivingly than exact localization.Cliffs are located more reliably than erosions, and judge-guided schedules usually fall back to broader searches rather than verifying the label directly.Source: §X.9 · p. 39, Table 57 · p. 39, Table 58 · p. 39

X.10 Ranking localizers, in full

  • Rankings by value captured and judge-label agreement diverge: their VC1 and A1 rankings have Kendall’s τ = 0.44 (permutation p = 0.12).The paper argues value captured better reflects controller benefit; “first edit” captures as much value as a mid-ranked judge, while “step 1” and “before first symptom” capture almost none.Source: §X.10 · pp. 39–40, Table 60 · p. 40

X.11 All V-curves

The 60 reconstructed V-curves locate decisive intervals in failed trajectories and show how success prospects evolve, from abrupt cliffs to gradual erosion and irrecoverability. They also compare these intervals with judges’ labels, which often fall later than decisive steps, while points of no return or explicit failure signals may occur much later or not at all.

Curve patterns

  • Of 60 curves, 20 are cliffs and 40 erosions; cliffs’ decisive steps sit at a median 53% of the run, while erosion point-of-no-return steps follow t∗ by a median 28 steps.The value stays flat until the decisive interval in 28 curves, while in 16 at least 30% has leaked away before the largest drop; 15 of 19 wide intervals are erosions.Source: §X.11 · p. 41

Raw estimates

  • 43 curves show rises of at least 0.25 between consecutive raw points, usually within K ≤ 32 noise; 5 are genuine recoveries, including django-11999 s0’s 0.50 rise.For django-11999 s0, the 95% intervals do not overlap, and the monotone fit flattens the recovery.Source: §X.11 · pp. 41–42

Reading curves

  • The curve height at t = 0 is the policy’s task pass rate; a cliff marks one decisive decision, whereas an erosion spreads losses across a staircase.The shaded interval marks where to inspect the transcript, and the dotted point-of-no-return line marks the earliest step prospects are certifiably below ϵ; labels to its right cannot matter.Source: §X.11 · p. 41

Label alignment

  • Only 6 of 60 curves have a main-judge label inside the decisive interval; on 18 every label is after it, and on 9 every label is before it.The six curves comprise 1 cliff and 5 erosions.Source: §X.11 · p. 41, Figure 8 · p. 42, Figure 9 · p. 43

X.11 All V-curves

  • Among the cited cases, sphinx-8459 s0, sphinx-9320 s7, sphinx-9658 s1, s3, and s6 each have no explicit failure signal.Source: §X.11 · p. 46
  • At sphinx-9658 s6, raw estimates rose by 0.50 from step 37 (2/16) to 38 (10/16), a genuine recovery whose nonoverlapping 95% intervals the monotone fit flattened.Source: §X.11 · p. 46
  • At sympy-13615 s6, decisive step 9 preceded edit 61 with 40% of value already gone; GPT-6.1 Sol, GPT-6 Astra, and GPT-5.5 labeled 9, while o3 labeled 61.The point of no return was step 41, and the first failure signal came at step 65.Source: §X.11 · p. 46
  • At sympy-19783 s0, all four judges labeled step 33, after decisive step 6 and point of no return 30; at sphinx-9658 s3, both GPT-6 judges labeled edit 85, after decisive step 6 and point of no return 50.Source: §X.11 · p. 46
  • On sympy-22714 s5, decisive step 17 followed 33% of value already being gone despite no repository edits; its point of no return was step 70.All four labels fell near the decisive step: GPT-6.1 Sol, GPT-6 Astra, and o3 labeled 16, and GPT-5.5 labeled 20.Source: §X.11 · p. 46

Related papers

Loading 2610.17633v1…