PaperlayerView exact revision on arXiv

Source-linked AI summary

Monitor Accuracy Isn't the Answer: A Decision-Value Framework for Evaluating AI Agent Restart Monitors

Siddharth Jain, Venkat N. Vedam

arXiv:2610.36544v1cs.AIcs.IR

TL;DR

Restarting an AI agent cannot be judged by failure-classification accuracy alone, because abandoning a live attempt removes its remaining success chance and the replacement has only residual budget. The paper develops decision-value criteria and a model-free trace audit; across its selected executions, restart effects vary and none of 103 live comparisons meets the specified certification rule.

  • Problem

    Failure-prediction rates alone do not establish whether restarting improves success, because the decision also depends on the replacement attempt’s residual-budget success probability.

    Source: §1 · pp. 1–2

  • Method

    The paper derives an alarm-to-action identity, defines decision sufficiency for progress representations, and audits existing traces with simultaneous binomial intervals that can certify an action preference or abstain.

    Source: §1 · p. 2

  • Results

    Zero of 103 live task-time comparisons met the simultaneous certificate, and the apparent 16-call Luna gain across the audited traces came from a single eligible failed trajectory.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Agent monitors should be evaluated through intervention-specific recovery opportunity, decision value, and statistical support rather than classification performance alone.

    Source: §8 · p. 23

    The audit uses task identity and elapsed calls rather than rich trace features, so it does not measure the decision value of embeddings, learned monitors, or semantic summaries.

    Source: §7.3 · p. 23

Abstract

from arXiv · show

Classification accuracy and true-positive and false-positive rates alone do not determine whether restarting an AI agent improves success. Using an exact alarm-to-action identity for an independent clean restart with zero monitoring overhead, we show analytically that a monitor with fixed 90% accuracy can swing from a five-percentage-point loss to a 35-point gain in success rate, depending solely on the replacement attempt's residual-budget success probability. We introduce two techniques for evaluating monitors and progress signals in decision terms. First, a decision-sufficiency criterion, a binary-action specialization of Blackwell's comparison of experiments, identifies which progress representations or context compressions preserve the information needed to choose optimally between the specified actions, independent of trace reconstruction fidelity. Second, a finite-sample, model-free decision audit uses simultaneous binomial intervals over existing agent traces, without additional model calls, to certify a preference for "restart" or "continue" under stated sampling assumptions, or return "abstain" when evidence is insufficient. In a study of 194 externally evaluated model executions on 19 selected SymPy issues from SWE-bench Verified, with a retrospective audit of 130 of these traces, we find a 2.29-point apparent restart gain traced to a single surviving failed attempt, a 42.29-point loss on a second model, no positive replication of the initial gain on prospective task cohorts, and zero of 103 live comparisons meeting the specified certification bar. These results motivate evaluating agent monitors through recovery opportunity, decision value, and statistical support, rather than classification performance alone.

1 Introduction

The paper asks how alarm errors, replacement success within the residual budget, and information retained by progress representations affect restart decisions. It presents analytical decision criteria and a trace-based audit, then reports model-dependent restart outcomes and insufficient evidence for certified per-task preferences.

1 Introduction

  • The motivating distinction is that abandoning a live trajectory removes its remaining success chance, while a replacement attempt receives only residual budget.Thus accurate failure classification alone need not make restarting beneficial.Source: §1 · p. 1
  • With failure-classification accuracy fixed at 90%, changing only replacement success probability shifts restart gain from −5 to +35 percentage points.This is a controlled mathematical example, not an implemented-detector result.Source: §1 · p. 2
  • The decision-sufficiency criterion identifies when a progress representation preserves information needed to choose between continuation and restart, regardless of trace reconstruction fidelity.The paper presents it as a binary-action specialization of classical information comparison.Source: §1 · p. 2
  • The study covers 194 model-driven executions of mini-swe-agent on 19 selected SymPy issues, and does not establish performance beyond its selected tasks and runtime configurations.The paper does not train a monitor or directly compare against learned recovery systems.Source: §1 · pp. 2–3
  • Across 130 audited executions, the apparent 16-call Luna gain came from one eligible failed trajectory, while all 103 live comparisons remained unresolved by the simultaneous certificate.On the second model, every task with a live attempt favored continuation empirically.Source: §1 · p. 2
  • A dependency-free audit estimates both action values and returns simultaneous sign certificates or abstention under explicit assumptions.It uses existing traces and requires no additional model calls.Source: §1 · p. 3

2 Related Work

The related work spans restart optimization, inference-time allocation, agent progress signals, and software-agent evaluation. This paper positions its contribution as an evaluation and evidence audit for a specified action pair, not as a new restarting policy or a comparison against richer learned systems.

2.1 Restarts and inference-time allocation

  • Restart theory and inference-time allocation provide foundations, but finite-budget restart success still depends on completion scale and informative observations.The paper specializes Blackwell’s comparison of experiments to two actions and applies exact binomial intervals with simultaneous comparison [6, 10, 11].Source: §2.1 · p. 3

2.2 Agent progress and externally grounded evaluation

  • Prior work spans progress metrics, semantic stopping, and context management; this audit evaluates post-alarm action and remaining-budget success jointly with failure prediction, using externally grounded binary completion rather than transcript improvement.Source: §2.2 · p. 4

2.3 Software engineering agents and reproducible evaluation

  • The study uses selected SymPy software tasks with external evaluation safeguards, but does not compare agent architectures or report a leaderboard estimate.The benchmark evaluates repository-level issue resolution against executable tests [7], and the study runs the evaluator outside the agent workspace after each action.Source: §2.3 · p. 4

2.4 Position of this work

  • This work contributes an evaluation and evidence audit for a specified action pair; its negative screens do not invalidate systems with different observations, training, or recovery semantics.Table 1 contrasts task-grounded progress decisions with trace-only inference.Source: §2.4 · p. 4, Table 1 · p. 5

3 Black-Box Progress as a Decision Problem

The framework treats black-box progress as a decision problem: a useful representation must preserve distinctions that change the preferred action, and restart value depends on completion-time scale. Survival alone can leave continuation and restart indistinguishable even when their optimal choices differ.

3.1 Setup

  • Progress error incurs regret in proportion to the value gap between actions, so label accuracy alone can obscure consequential mistakes.A representation is decision sufficient when the optimal action is measurable from that representation and the remaining budget; exact sufficiency is not required for usefulness.Source: §3.1 · pp. 5–6

3.2 Fixed restarts depend on the completion scale

  • No fixed proper cutoff uniformly dominates uninterrupted execution, and a finite restart schedule can miss all completion mass beyond its longest episode even when an uninterrupted run succeeds.A positive universal-restart claim therefore requires assumptions about completion times, a prior, or observations that predict residual completion time.Source: §3.2 · p. 7

3.3 The continuation decision is not identified by survival

  • Two completion regimes can produce the same no-success history through call c yet require opposite choices, so survival alone cannot identify the optimal action.Without an identifying sensor or calibrated prior, appearance-based stagnation does not establish residual value.Source: §3.3 · pp. 7–8
  • With aliased histories, any randomized policy has worst-case regret of at least p/(1 + p), approaching 1/2 as p approaches one.The result concerns identification from the observation, not an inherent inability to distinguish regimes with a task-specific sensor.Source: §3.3 · p. 8

4 From Failure Prediction to Restart Decisions

The section evaluates restart decisions by their effect on success under a specified intervention, then develops criteria for information sufficiency and finite-sample decision evidence. Its analyses distinguish a monitor’s classification quality from restartability, the decision value preserved by a representation, and the support available for choosing an action.

4.1 Failure prediction must be combined with restartability

  • The alarm-to-action identity applies to a clean restart that restores the initial state, is independent of the alarm and continuation outcome, and consumes no budget.Edit-preserving recovery changes the intervention, while mixing task populations can induce dependence; action values are therefore estimated within task and runtime before aggregation.Source: §4.1 · pp. 8–9
  • At 90% classification accuracy, the analytical example ranges from a five-point loss to a 35-point gain as restartability changes.The identity assumes an independent clean restart with zero monitoring overhead; these are analytical controls, not measured monitor performance.Source: §4.1 · pp. 8–9

4.2 The decision value of a richer progress signal

  • Compressing a progress signal loses decision value when a coarse observation combines positive and negative restart advantages.Preserving the preferred action can preserve decision value without reconstructing the original trace; token retention and reconstruction quality alone do not measure this loss.Source: §4.2 · pp. 9–10

4.3 A finite-sample decision certificate

  • The audit compares live attempts’ continuation through the horizon with fresh attempts restarted at the remaining budget, without extrapolating beyond observed horizons.Continuation outcomes come only from attempts alive at the decision; a later protocol terminal counts as unsuccessful continuation, while earlier terminals are excluded.Source: §4.3 · p. 10
  • Simultaneous binomial intervals certify restart when the restart lower bound exceeds the continuation upper bound, continue when the reverse separation holds, and otherwise abstain.Under the stated IID, complete-observation, and outcome-independent comparison-family assumptions, each nonabstaining decision has the correct sign with probability at least 1 − α.Source: §4.3 · p. 10
  • Zero-success observation counts bound one success probability under the specified interval procedure, but do not by themselves establish an intervention effect.Conditional continuation observations must also reach the decision while alive.Source: §4.3 · p. 11, Figure 2 · p. 11

5 Experimental Method

The experiments use frozen stage-specific protocols on selected SymPy tasks, externally verify completion, and evaluate restart and recovery decisions under defined budgets. Existing traces support a retrospective decision audit with task-level uncertainty and explicit treatment of censored attempts.

5.1 Design principle

  • The staged experiments froze each task rule, intervention, outcome, and stopping rule before collecting its new outcomes, while allowing earlier results to motivate later experiments.The authors distinguish this sequential falsification design from a single preregistered confirmatory study.Source: §5.1 · p. 11

5.2 Benchmark and runtime tuple

  • The 19 selected SymPy issues reduce language and build-system variation but limit external validity; runtime identity also includes task, model, agent release, route, prompt, and sampling policy.The primary budget is model calls, distinct from valid environment actions, verified completion, and protocol-terminal calls.Source: §5.2 · pp. 11–12

5.3 External verification and completion time

  • Completion time is the first externally verified passing state, checked after each environment action with the evaluator and reference patch kept outside the agent sandbox.Successful attempts were audited for verifier or hidden-test changes, and none modified those assets.Source: §5.3 · p. 12

5.4 Restart replay

  • Restart replay resamples fresh attempts within task under a 30-call budget, releases calls after protocol termination, and evaluates fixed cutoffs against the uninterrupted baseline.Cutoff selection is also assessed leave-one-task-out, since attempts within an issue share task difficulty.Source: §5.4 · p. 13

5.5 Prospective task cohorts

  • Prospective cohorts fixed the transferred 16-call cutoff and 30-call baseline before outcomes; the extension tested four tasks that were 0/5 at 30 calls through a 60-call horizon.The first cohort used seven highest-ID eligible unseen tasks; the difficulty-enriched cohort yielded five eligible tasks.Source: §5.5 · p. 13

5.6 Context-rejuvenation intervention

  • Context rejuvenation preserved repository state but reset conversation after three consecutive invalid tool responses, with a shared 60-call cap and no extra calls or feedback.The expansion rule required at least a five-point gain or two recovered successes, with mean-cost increase no greater than 20%.Source: §5.6 · pp. 13–14

5.7 Failed-branch reuse

  • Failed-branch reuse composed pairs of failed production-code patches from a clean benchmark state, then screened one bounded semantic-resolver attempt per task using failed patches as hypotheses.Patch ranking was frozen before verification; the resolver protocol allowed no retries and opened a second stage only after a verified success.Source: §5.7 · p. 14

5.8 Retrospective decision-value audit

  • The audit reanalyzes 130 traces across 156 task-time comparisons using simultaneous binomial intervals, and treats empty live populations as not reached.The evaluated decision is a single clean restart with residual budget; retrospective oracle values are not deployed policies or validated opportunity bounds.Source: §5.8 · p. 14

6 Results

Across the tested models and interventions, restart value depended on completion and recovery opportunity, while the initial apparent gain did not replicate prospectively. The audits and bounded recovery tests also distinguish visible outcome changes from evidence of improved verified task success.

6.1 A restart scale does not transfer across models

  • At cutoff 16, Luna’s estimated gain was +2.29 points, while the same cutoff reduced DeepSeek success by −42.29 points.Leave-one-task-out selection gave Luna 75.54% versus 82.86% for the 30-call baseline; tested finite schedules also varied sharply by model.Source: §6.1 · pp. 15–16, Figure 3 · p. 16

6.2 The in-sample gain does not survive prospective transfer

  • Fixed 16-call cutoffs did not improve prospectively: easy tasks tied at 100%, while the difficulty-enriched cohort scored 0% versus 4% for fixed 30.Across all 19 Luna tasks, fixed 16 scored 68.21% and fixed 30 scored 68.42%.Source: §6.2 · p. 17

6.3 Policy differences do not certify individual restart decisions

  • None of the 103 live comparisons met the simultaneous sign-certificate rule, and the initial 2.29-point gain came from one surviving failed attempt.All six live DeepSeek cells favored continuation in plug-in estimates; no live decision occurred in the prospective high-ID cohort.Source: §6.3 · p. 17, Table 7 · p. 18, Figure 4 · p. 19

6.4 Additional horizon reveals protocol collapse, not late success

  • Among 20 extended attempts, one passed at call 17 and none first passed during calls 31–60; repeated 30-call replay scored 9% versus 5% for one 60-call attempt.The replay benefit came from reusing calls after observable protocol-terminal events, not censoring live trajectories.Source: §6.4 · pp. 18–19, Figure 4 · p. 19

6.5 Fresh context changes the terminal label, not success

  • Context reset eliminated the RepeatedFormatError label but produced no treatment successes; mean cost rose 56.7% and mean wall time rose 65.7%.Baseline solved one of 20 attempts, compared with none under treatment.Source: §6.5 · p. 19, Figure 5 · p. 20

6.6 Failed branches are diverse but not compositional

  • None of 51 clean, novel three-way patch composites passed, and all four bounded semantic-resolver runs failed verification.The historical clean-context cohort also had no completion within 12 calls, so the resolver screen was not a randomized effect estimate.Source: §6.6 · p. 20, Table 10 · p. 21

7 Discussion

The discussion frames the work as an operational account of decision value and evidential support for specified agent-control actions. It emphasizes boundaries on the selected-task evidence and proposes a focused held-out test of an existing monitor.

7.1 Contribution and scope

  • The contribution connects failure discrimination to residual-budget restartability, decision-preserving compression, and the sampling support for per-task action comparisons.The retrospective audit diagnoses collected traces but does not demonstrate a deployed policy gain; intervention screens concern particular controllers on selected tasks.Source: §7.1 · p. 21

7.2 Implications for evaluation and deployment

  • Evaluate failure signals where an alternative action has recovery opportunity; failure prevalence alone does not establish that restarting can recover value.A ceiling cohort may offer no live decisions, while a hard cohort may contain many failures but little measured restartability.Source: §7.2 · p. 21
  • Fix reset semantics, residual budget, verifier, and monitoring cost when comparing signals, because clean restart, context reset, and edit reuse have different outcome laws.A representation may discard trace detail without losing control value if it preserves distinctions that determine the preferred action.Source: §7.2 · pp. 21–22
  • The audit reports an action only when its specified intervals separate; abstention means insufficient evidence, not that continuation is optimal.Five-run cells are too sparse for this conservative rule, while hierarchical pooling would add assumptions requiring held-out validation.Source: §7.2 · p. 22

7.3 Limitations

  • The 19 selected SymPy issues do not estimate performance across SWE-bench Verified or agent workloads generally; later interventions used one model and one compact harness.Two models on seven shared tasks demonstrate heterogeneity, not broad model coverage.Source: §7.3 · p. 22
  • Five attempts per task leave substantial uncertainty after conditioning on live survival, and the audit’s retrospective decision family is not prospective validation.The zero-success intervention screens do not exclude modest benefits in other populations.Source: §7.3 · pp. 22–23
  • The binomial certificate assumes exchangeable independent provider draws within each fixed task and runtime, and does not transfer across arbitrary distribution or harness changes.External evaluation labels are unavailable to typical production controllers, and reported model costs omit end-to-end deployment costs.Source: §7.3 · p. 23

7.4 A focused next test

  • The next test should freeze an existing monitor and recovery semantics, then compare continuation and baseline policies on held-out tasks while charging monitoring and recovery overhead.A second model or repository is needed to test transfer; more draws on the same four hard tasks cannot establish it.Source: §7.4 · p. 23

8 Conclusion

Across 194 model executions, restart and recovery outcomes depended strongly on runtime, with no prospective replication of the initial cutoff gain. The authors recommend evaluating interventions by recovery opportunity, decision value, and statistical support.

8 Conclusion

  • Monitor value depends jointly on alarm errors and success within the residual budget after an independent clean restart.The conclusion distinguishes decision-oriented retention or compression of progress observations from statistical support for a per-task action preference.Source: §8 · p. 23
  • Across 194 model executions and 19 verifier-only composites, restart and recovery showed strong runtime dependence and no prospective replication of the initial cutoff gain.A retrospective audit of 130 traces found 103 live task-time comparisons and no sign certificate under its simultaneous rule.Source: §8 · p. 23
  • The study’s evaluation standard is to declare the intervention, measure recovery opportunity, account for monitoring cost, and report uncertainty where decisions are made.RestartLab provides the accompanying trace analysis and audit implementation; broader prospective validation remains necessary.Source: §8 · p. 23

B Task Cohorts and Frozen Decisions

The study combines several overlapping cohorts and frozen intervention protocols; its fixed-policy estimates treat five empirical attempts per task as IID, with replay used for episode-dependent residual budgets.

B Task Cohorts and Frozen Decisions

  • The study comprises 194 model attempts across overlapping stages, plus 19 external verifier calls on composed candidates that add no model attempts.The 194 attempts consist of 70 initial, 60 prospective, 20 horizon-extension, 40 context A/B, and four semantic-resolver attempts.Source: B Task Cohorts and Frozen Decisions · p. 27

C Policy Definitions

  • The single-attempt policy runs up to budget B and stops at the first external pass or explicit terminal failure.Source: C Policy Definitions · p. 27
  • Fixed, geometric, and Luby policies allocate total budget through repeated attempts, using fixed caps or frozen cap sequences.Fixed and geometric sequences truncate at the remaining total budget; Luby uses a scaled prefix of the universal sequence [9].Source: C Policy Definitions · p. 27
  • Horizon insurance compares two 15-call probes followed by a reserved 30-call episode against two 30-call episodes at budget 60.Source: C Policy Definitions · p. 27
  • Fixed-policy plug-in estimates treat five empirical attempts per task as an IID distribution, while implementation uses Monte Carlo replay when terminal failures make residual budgets episode-dependent.Source: C Policy Definitions · p. 27

D Additional Operational Results

Operational results use provider-metered trace costs, while the accompanying sanitized package supports offline reproduction of analyses and figures but not new agent collection without credentials.

D Additional Operational Results

  • Context rejuvenation’s large cost and latency change arises because treatment consumes all 60 calls after replacing an early terminal format failure with a fresh episode.Recorded provider-metered costs are not estimates of deployment expense.Source: D Additional Operational Results · p. 27
  • Table 13 reports collection cost and accumulated wall time, with wall time defined as the sum of parallelizable attempts rather than elapsed study duration.Source: Table 13 · p. 28

E Artifact Manifest

  • The artifact package excludes prompts, trajectories, local paths, hidden verifier output, reference patches, and credentials.Source: E Artifact Manifest · p. 28
  • Table 14 lists accompanying analysis artifacts and their roles in the study.Source: Table 14 · p. 28
  • Sanitized artifacts reproduce reported trace analyses and figures offline, but new agent collection requires a model credential; derived audit cells reuse earlier attempts.The package includes stage protocols, composition plans, adapters, replay policies, analysis code, command-line tools, and tests.Source: E Artifact Manifest · p. 28

F Threats to Interpretation

The authors qualify interpretation by task selection, verifier scope, adaptive stage design, multiplicity, and incomplete identification of hosted model runtime.

F Threats to Interpretation

  • Pooled rates describe the selected tasks, not prevalence estimates for SWE-bench Verified, because pilot and deliberately difficulty-enriched cohorts were included.The prospective high-ID cohort used a mechanical ordering.Source: F Threats to Interpretation · p. 28
  • Verifier success is the reported outcome, not semantic issue resolution, because the fail-to-pass suite remains a finite specification.Verification after every environment action provides precise first-pass timing.Source: F Threats to Interpretation · p. 28
  • Later stages were selected after earlier results, so cross-stage narratives can invite hindsight despite each stage having a frozen protocol.The ladder table states each stage’s question and comparison, and null follow-ups are retained.Source: F Threats to Interpretation · p. 28
  • Stage-level bootstrap intervals are not adjusted for simultaneous coverage across the several policies and mechanisms evaluated.The decision-audit intervals alone use explicit family allocation; exploratory maxima are not claimed as significant.Source: F Threats to Interpretation · pp. 28–29
  • Requested model ID and collection date define but do not fully identify the runtime, since hosted behavior can change under a stable identifier.Source: F Threats to Interpretation · p. 29

G Failure-Conditioned Branches: Supporting Examples

Branching’s benefit depends on the epistemic relation among branches, not their textual distance. Conditioning an ensemble on terminal failure can reverse component reliability, while correlated branches limit the value of adding views.

G Failure-Conditioned Branches: Supporting Examples

  • Branching is proposed when continuation is uncertain, but its benefit depends on the epistemic relation among branches.Conditioning ensemble inputs on terminal failure can reverse component reliability.Source: G Failure-Conditioned Branches:… · p. 29
  • A component with unconditional reliability q > 1/2 can become minority-reliable among failed full patches when r > (2q −1)/q.This is an existence result, not an estimate of the patch population.Source: G Failure-Conditioned Branches:… · p. 29
  • When branch correctness indicators have common pairwise correlation ρ = 1, adding branches does not reduce uncertainty.The variance corresponds to that of only a limited number of independent views; the cited passage does not give the number.Source: G Failure-Conditioned Branches:… · p. 29
  • Behavioral diversity, such as different files or diffs, can coexist with insufficient error diversity for aggregation across branches from one model and harness.The paper distinguishes this epistemic independence from behavioral diversity and cites the standard ensemble effect.Source: G Failure-Conditioned Branches:… · p. 29

Related papers

Loading 2610.36544v1…