More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight

1 Introduction

• The unit of verification is how many planned actions each verifier call reads, a deployment parameter whose over-blocking can forfeit usefulness and pressure deployers to disable oversight.The paper studies fixed batches with every step reviewed exactly once, rather than when to invoke the monitor.Source: §1 · p. 1
• Natural trajectories cannot isolate window-length effects because review length co-varies with difficulty, error type, and position, while clean prefixes lack matched bad controls.Thus catch alone cannot distinguish better discrimination from rejecting more clean actions.Source: §1 · p. 1
• The twin-prefix framework pairs an environment-accepted injected error with a clean twin differing in one write, tested at five nested lengths with error type and position fixed.It pre-registers informedness J = catch−FR, supports score-mode AUC with per-length recalibration, and maps performance and per-call costs to a deployed unit L∗.Source: §1 · pp. 1–2
• Informedness peaks at L∈{1, 2} for all six judges in both domains; primary J drops 0.602→0.035 as windows lengthen.Replaying withheld observations raises J(8) from 0.035→0.490, tracing the failure largely to observation deprivation.Source: §1 · p. 2
• No tested label-blind policy consistently beats a calibrated fixed short unit; safety cases should state the unit and co-report the clean series.Source: §1 · p. 2

Oversight and AI control

• AI-control work studies fallible supervision and safety–usefulness trade-offs, while this paper varies the reviewed unit; adaptive attacks limit how far its non-adaptive measurements extrapolate (Terekhov et al. 2026b).Related work also prices supervision and end-to-end control tax (Terekhov et al. 2026a), extends control protocols (Bhatt et al. 2025; Korbak et al. 2025; Baker et al. 2025), and treats the unit as given.Source: §2 · p. 2

Granularity and verifier reliability under length

• Existing granularity studies either confound review length with error type and position, lack irreversible actions or a clean false-rejection axis, or fix the unit while varying enforcement (Fan et al. 2026; Sah et al. 2026).Other cited work examines verifier criticality, growing-prefix monitoring, annotated error corpora, and trained process-reward models (Zhou et al. 2026b; Huang et al. 2026; Zhang et al. 2026; Recchia et al. 2026; Chae et al. 2025; Li et al. 2026).Source: §2 · p. 2

Speculation and triggers

• Speculative-execution methods choose verification units for exact verifiers, where width costs compute rather than accuracy; this work instead measures unit cost for fallible monitoring (Ro et al. 2025; Guan et al. 2025; Zhou et al. 2026a).It inherits single-step ask-or-act triggers and verifier routing (Ren et al. 2023; Aggarwal et al. 2023; Kamath, Jia, and Liang 2020; Gupta et al. 2024; Kiyani et al. 2026).Source: §2 · p. 2

3.1 Problem Formulation

• Window cost is C(L)=F+L·i, while informedness J(L)=catch(L)−FR(L) rewards catching bad prefixes without rejecting matched clean ones.The measure makes reject-everything behavior visible: it yields catch=1 but J=0.Source: §3.1 · p. 2

3.2 Controlled Twin-Prefix Construction

• Each gold plan yields matched clean and bad prefixes at lengths 1, 2, 3, 5, and 8, differing only in an environment-accepted anchor write.In-window steps are unexecuted, so longer windows also withhold their observations; injections are balanced across severity strata and near/far argument distances.Source: §3.2 · pp. 2–3

3.3 Paired Co-evaluation and Pre-registration

• Each twin receives a separate stateless call, and the frozen instruction requires a specific fault for rejection while treating missing observations as no evidence of error.The clean series is a mandatory co-report alongside catch; gold-clean and draft-clean false rejections are distinguished.Source: §3.3 · pp. 3–4

3.4 Selecting the Unit

• All six open-weight cells showed a short-minus-long contrast excluding zero; calibration across held-out task clusters never selected L>2.The recipe selects a length on calibration data using deployment-specific token savings, informedness loss, and error weighting, then fixes that length.Source: §3.4 · p. 4

4.1 Experimental Setup

• The experiments compare five review lengths across retail and AppWorld using fixed-length, pooled, routed, self-signal, and label-aware oracle policies, with paired resampling and clean controls.The primary verifier is Llama-3.3-70B-Instruct; the panel also contrasts Qwen verifiers and three closed-weight judges. Retail has 31 clean gold anchors, while AppWorld has 200.Source: §4.1 · pp. 4–5, §4 · p. 4, Table 1 · p. 5

4.5 RQ4: A Short Unit Is Rarely Beaten

• Only the AppWorld/14B router beats its fixed short unit (+0.105); pooling loses, and draft self-signals reach at most 0.410 J.The retail 70B label-aware oracle reaches 0.903 versus fixed-unit 0.602, with 0.302 headroom; this bound uses labels unavailable at deployment. Fixed short units are near-optimal only within the tested label-blind class.Source: Table 1 · p. 5, §4.5 · p. 7, Figure 5 · p. 7

4.2 RQ1: Longer Windows Buy Rejection, Not Discrimination

• Catch rises from 0.765 to 0.970 as review length grows from 1 to 8, but clean false rejection rises from 0.419 to 0.935 and informedness falls to 0.035.Informedness peaks at L=2 with J=0.602; the 14B and 72B reach catch = false rejection = 1.000 at L=8. The short-versus-long J contrast excludes zero in every open-weight cell under both resampling protocols.Source: §4.2 · p. 5, Table 1 · p. 5, Figure 2 · p. 6

4.3 RQ2: Where Long Windows Fail

• For the 14B, catch is 0.237 for a position-4 error in an L=8 window versus 1.000 at position 8; position-marginal informedness never exceeds 0.093 at measured long lengths.The headline design places errors at the final step, so it overstates long-window catch; the end-position versus position-marginal catch gap at L=8 excludes zero for every verifier.Source: §4.3 · p. 6
• Providing gold observations changes retail L=8 false rejection from 0.935 to 0.290 and informedness from 0.035 to 0.490, with the peak still at L=2.AppWorld loses 38% of its false-rejection slope under replay, with residual causes unattributed; predicted observations repair almost none of the oracle recovery, while inert padding leaves false rejection high (0.968 versus 0.290).Source: §4.3 · p. 6, Figure 4 · p. 6

4.4 RQ3: The Shape Is Robust

• All 15 short-window advantage intervals in the wider robustness battery exclude zero, and short units remain ahead under all 12 anchor/task reweighting schemes.Short-only 14B training does not repair long-window performance: at L=8, clean false rejection is 0.935, J is 0.040, and AUC is 0.672. The section scopes informedness claims to the twin-calibrated distribution.Source: §4.4 · p. 7

• The verifier study covers a zero-shot panel, two tool-agent domains, L≤8, and no second-family process-reward model; its argmax is per-deployment and claims concern per-call proxies.These boundaries limit generalization beyond the tested settings.Source: §5 · p. 7
• Injected errors are single, non-adaptive writes bounding catch; unit-aware planning is untested, and natural-error probes agree on catch rise but not the J peak.The passage points to Appendices B, L, Q, and X for these limitations.Source: §5 · p. 7
• Retail gold-clean prefixes cap at n=31, leaving small false-rejection gaps underpowered and the peak protocol-sensitive; reported costs cover verifier tokens alone.The passage points to Appendices AF and Z for these statistical and cost limitations.Source: §5 · p. 7

6 Conclusion

• Across six judges and two domains, longer review increased rejection without improving discrimination; a calibrated fixed short unit was not consistently beaten by tested label-blind policies.Replay traces the failure largely to observation deprivation, and short-only training does not repair it. The conclusion calls faithful observation prediction the open problem and recommends stating the verification unit and co-reporting the clean series.Source: §6 · p. 7

A Notation

• Appendix A defines score-mode metrics from bad and clean prefix scores, uses average ranks for the Mann–Whitney statistic, and lets the best-threshold measure include reject-nothing, so J∗(L) ≥0.Table A3 collects the paper’s symbols with their defining locations in the main paper.Source: A Notation · p. 10

B Injected-Corpus Design

• Injected bad items replaced each anchor’s final gold write with one non-gold write accepted by the environment, using real database identifiers and deterministic generation.The design defines near versus far argument distance by whether consequential injected identifiers fall within the anchor entity scope; the AppWorld generator likewise instantiates three severity strata with a registered 20/50/30 mixture.Source: B Injected-Corpus Design · p. 10
• The short-minus-long informedness contrast stayed positive across the full severity-mixture range for all three retail verifiers, with peak lengths of L=2/2/1.Worst-case simultaneous task-cluster bootstrap 95% lower bounds were +0.26, +0.13, and +0.35 for the 70B, 14B, and 72B verifiers, respectively; P(minw ∆w > 0) ≥0.9998. This post-hoc reweighting bounds sensitivity to the injected mixture, not transfer to natural errors.Source: B Injected-Corpus Design · p. 10

C Per-Step Protocol Control (Full Curves)

• Both per-step verdict readings produced identical clean false-reject curves, verified with 0 mismatches on both verifiers, while longer windows remained unsafe under either tested protocol.The rise in clean false rejections with review length survived the protocol change. Short-length ordering flipped between verifiers: 70B per-step FR was lower at L=1 (exact McNemar p = 0.008), while the 14B’s was higher.Source: C Per-Step Protocol Control (Full… · p. 10
• Per-step control re-judged the frozen retail corpus by labeling each window step independently in one call and aggregating verdicts by any-reject.Complete-matrix analyses dropped incomplete items (n=174 bad / 29 clean); the per-step re-judge itself had full coverage (200 bad / 31 clean per length) for both verifiers. The alternative closure reading accepts only when the longest all-accepted leading run covers all L steps.Source: C Per-Step Protocol Control (Full… · p. 10

D Position Sweep (Full Grid)

• Catch declined from the window’s end toward its interior for every verifier at both long lengths, monotonically in five of six cells.In AppWorld at L=8, the 14B caught a position-1 error at 0.000 versus 1.000 at the last step.Source: D Position Sweep (Full Grid) · p. 10
• The position-marginal catch is an unweighted mean over sampled positions, estimating the uniform-position marginal under batch deployment rather than an exact integral.It is systematically below headline window-end catch, so the headline is position-optimistic about long windows and the correction strengthens the short-unit conclusion.Source: D Position Sweep (Full Grid) · pp. 10–11
• At L=8, position-marginal informedness was −0.001 for the 70B, −0.367 for the 14B, and −0.100 for the 72B.At L=5, it was 0.093 for the 70B, −0.419 for the 14B, and −0.181 for the 72B. The 95% item-bootstrap CIs were, respectively, [−0.081, 0.103], [−0.418, −0.311], and [−0.139, −0.061] at L=8; and [−0.043, 0.243], [−0.497, −0.323], and [−0.289, −0.055] at L=5.Source: D Position Sweep (Full Grid) · p. 11

E Rejection-Rationale Taxonomy (Detail)

• Nine of 31 clean items flipped from an ARG rejection at L=1 or 2 to an EVID/OBS rejection at L=5 or 8.For one 70B item, a gold action was rejected for a hallucinated goal violation at L=2 and unavailable evidence at L=5; the 70B clean false-reject rate was 0.419 at L=1 and 0.194 at L=2.Source: E Rejection-Rationale Taxonomy (Detail) · pp. 11–12

F AppWorld Panel: Full Registered Series

• J(2)−J(8) was 0.151 for 70B, 0.311 for 14B, and 0.075 for 72B in AppWorld; all three task-cluster 95% CIs excluded zero.The respective intervals were {0.070, 0.234}, {0.215, 0.408}, and {0.024, 0.130}.Source: F AppWorld Panel: Full Registered Series · p. 12

G Registered Contrasts and Peak Probabilities (All Six Cells)

• Every verifier–domain cell’s registered J(2)−J(8) contrast excluded zero under both item-level and task-cluster bootstrap intervals.Table A9 consolidates each cell’s informedness peak, bootstrap peak probability, and registered short-versus-long contrast.Source: G Registered Contrasts and Peak… · p. 12

E Rejection-Rationale Taxonomy (Detail)

• Across all six cells, long windows shift clean-rejection reasons from concrete, often hallucinated argument mismatches toward unverifiable-evidence refusals.Reasons came from the verifier’s controlled eight-code vocabulary and were not independently audited against rationales; free-text explanations were spot-read qualitatively.Source: E Rejection-Rationale Taxonomy (Detail) · p. 11, Figure A2 · p. 13

H Interval Registry for Compressed Main-Text Statements

• The largest in-sample-minus-held-out J* gap was 0.035; held-out estimates refit thresholds on training clusters and evaluated them on held-out clusters.The held-out protocol used 500 seeded cluster-level 50/50 splits in AppWorld; retail had too few clean items per fold for per-fold FPR.Source: H Interval Registry for Compressed… · p. 13
• Against the one-action unit selected in AppWorld, four of five strong judges lost held-out discrimination, with both intervals excluding zero.Against the two-action unit, those judges showed no detectable held-out decay, reproducing the threshold-drift reading.Source: H Interval Registry for Compressed… · p. 14

I Closed-Weight Judge Panel: Transports and Completeness

• The closed-weight robustness panel covered 13 judge–domain cells, with dated prequery amendments and transport-specific serving setups.The panel reused frozen corpora, prompts, and analysis; transports were disclosed but not matched, and the nominal CIs reflect added judges and transports rather than more task samples.Source: I Closed-Weight Judge Panel:… · p. 14
• Five of six closed-weight cells had exact totals and complete availability; DeepSeek-V4-Pro retail had 1,205/1,215 usable rows and per-length availability of 0.986–1.000.No closed transport was snapshot-pinned, so future re-queries of the same floating model ID are not guaranteed to reproduce verdicts.Source: I Closed-Weight Judge Panel:… · p. 14

Verdict Pooling

• The any-reject union attained J ≤0.105 in every cell, with J = 0 exactly in four.The union rejects if any review length rejects, including for clean items.Source: J Oracle Bound, Router, and Verdict… · p. 14
• Majority voting across five lengths was worse than fixed L∗ by ΔJ −0.118 to −0.400, with every 95% CI below zero, despite 5× query cost.No score-pooling rule separated from the best test-tuned fixed length in any cell (ΔJ −0.021 to +0.030; all six task-cluster CIs spanned zero); leave-one-task-out score thresholds fell below that comparator in five of six cells.Source: J Oracle Bound, Router, and Verdict… · p. 15

Router

• The per-tool LOTO router was statistically indistinguishable from fixed L∗ in five of six cells, but improved AppWorld 14B by +0.105 [+0.024, +0.188].The exception was a weak judge over a heterogeneous 29-tool surface, where tool identity carried information about the detectable-at-L profile.Source: J Oracle Bound, Router, and Verdict… · p. 15

Oracle Bound

• Oracle headroom was 0.14–0.53, while mean token log-probability (AUC 0.730; informedness 0.387) and verbalized confidence (AUC 0.709; informedness 0.410) fell below retail fixed-L∗ J (0.431–0.666).The oracle requires bad/clean label information unavailable at deployment; the informedness values are in-sample best single-threshold results and upper bounds over deployable thresholds.Source: J Oracle Bound, Router, and Verdict… · p. 15

K Asymmetric Error Weighting

• The reweighted utility Uw(L) = catch(L) − w · FR(L) covers every linear catch/false-rejection utility, with exact argmax breakpoints computed from measured rates.At w=1, its argmax matches the headline L*; as w tends to infinity, it selects the minimum-FR length, which is 1 or 2 in every cell. These are point-estimate sensitivity breakpoints, not resampled quantities.Source: K Asymmetric Error Weighting · p. 15
• At a false-rejection cost of at least 0.41 times the missed-error cost, every cell keeps a 1–2-action optimum; below threshold, long windows beat always-reject by only 0.03–0.08.The long-window advantage is at most 0.03 in five cells and 0.08 in the mildest cell, AppWorld 70B.Source: K Asymmetric Error Weighting · p. 15
• The weight w = (1−π)cfr/(π cmiss) makes low-w deployments miss-dominant, where the measured results support stronger verification rather than longer review windows.This is the region where long-window gains over free always-reject shrink to 0.03–0.08.Source: K Asymmetric Error Weighting · p. 15
• As review length grows, all six open-weight cells move toward blanket rejection, while only the retail 70B and 14B cells first move toward the ideal at L=2.The two retail cells are the interior-peak cases.Source: Figure A3 · p. 16

L Natural-Error Probes

• +0.097 catch@window-3 −catch@window-1 in deterministic harvest and +0.0707 in sampled harvest directionally replicate, but do not establish, the controlled curve.The 95% CIs are [−0.032, +0.258] and [−0.0101, +0.1515], respectively; the legs are reported side by side and never pooled. Paired n=31 for deterministic harvest and n=99 for sampled harvest.Source: L Natural-Error Probes · p. 16
• The reversibility self-tag matches environment write/read truth at 99.68%, with 5 missed writes and 0 false alarms over 1,562 frames.This is the cheap deployment signal for identifying the cost regime.Source: L Natural-Error Probes · p. 16
• At L ≥3, 83–100% of human-benchmark windows trigger the observation-barrier rule, while spliced weaker-executor errors lose anchor grounding from 43/43 at L=1 to 1/18 at L=8.Both routes therefore measure barrier handling or fail grounding rather than provide a decisive full catch(L) curve; a powered natural-error test needs errors that exist before execution.Source: L Natural-Error Probes · p. 16

M Observation Deprivation: Intervention and Structural Audit

• Long-window collapse persisted among fully groundable items and after replacing all 853 audited retail values with symbolic placeholders, so parameter-level dependence did not drive it.The pointer audit found zero pre-window-evidence violations across 4,869 retail and 8,370 AppWorld frames. In the placeholder leg, retail J peaked at 0.579 at L=2 and fell to 0.065 at L=8; the clean-side delta was 0.000 at every length.Source: M Observation Deprivation:… · pp. 18–20
• Retail’s false-rejection slope fell from +0.516 to −0.129 with observations provided, while AppWorld’s fell from +0.495 to +0.305; informedness still peaked at L=2.The removed share was 1.25 in retail, the registered deprivation-dominant result, and 38.4% in AppWorld, the registered mixed result. In retail, catch stayed 0.74–0.80 across lengths; in AppWorld, clean-side McNemar tests at L≥2 consistently favored the observation-provided leg.Source: M Observation Deprivation:… · p. 18, Table A16 · p. 19

N Inert-Padding Token Controls

• Restoring L=8’s token deficit with inert padding did not lower clean false rejection, which instead fell when real observations were provided.In the retail 70B cell, padding changed L=8 clean FR from 0.935 to 0.968 (paired ΔFR +0.032, CI [0.000, 0.107]); the observation-provision control reduced it to 0.290. The L=2 padding change was not detected, while L=1 fell from 0.419 to 0.226 without a registered interpretation.Source: N Inert-Padding Token Controls · p. 20

O A Three-Way ACCEPT/REJECT/DEFER Probe

• On retail, the three-way probe’s selective informedness peaked at L=2 (0.623) and fell to 0.177 at L=8, while clean-side deferrals reached 0.613.At L=8, 93 of 95 deferrals carried the NEED_OBSERVATION code, and neither mapping deferrals to rejection nor acceptance rescued long windows. The recode bound anticipated substantial epistemic rejection mass but was not a prediction for the re-prompted instrument.Source: O A Three-Way ACCEPT/REJECT/DEFER Probe · pp. 20–21, Table A18 · p. 21
• On AppWorld’s three-way probe, selective J was flat in L (0.498/0.419/0.452/0.378/0.487; point argmax L=1); J(8)−J(2)=+0.068, CI [−0.044, 0.180], including zero.The probe showed no significant short-vs-long discrimination gap in either direction.Source: O A Three-Way ACCEPT/REJECT/DEFER Probe · p. 21

P Clean False Rejection: Intervals and the Draft-Clean Bound

• Gold-clean rates are upper bounds and draft-clean rates are lower bounds on deployment false rejection; both bounds rise with length.Retail’s n=31 intervals are wide enough for adjacent lengths to overlap, so the paper relies on paired within-item contrasts rather than marginal-rate comparisons; AppWorld has no draft stream.Source: P Clean False Rejection: Intervals and… · p. 21, Table A19 · p. 22
• Long-window gold-clean false rejection overstated the deployment-facing draft-clean rate by roughly 3× in the deterministic K=8 harvest.The harvest produced 163 eligible five-step and 119 eligible eight-step windows; at both lengths, the legs fell in the registered branch where draft-clean false rejection was below gold-clean false rejection. Because draft and gold populations differ in content, causal isolation rests on the observation intervention, not this comparison.Source: P Clean False Rejection: Intervals and… · pp. 21–22

Layer 1: exhaustive environment-semantics census

• Environment replay accepted and state-diverged 200/200 retail and 188/188 evaluable AppWorld bad writes; clean twins were accepted 31/31 and 200/200, respectively.The divergence persisted to terminal state for all evaluable suffixes (106/106 retail and 188/188 AppWorld). Retail had 24 bad-write no-ops relative to the pre-state; AppWorld had 2. Two AppWorld items were unevaluable because the gold anchor write returned HTTP 422 on replay, and 16 gold suffixes raised in-suffix errors. Repeated anchor states were byte-identical, and benchmark task success is defined on the same database state.Source: Q Label-Validity Audit: Environment… · p. 22

Layer 2: two-model blind panel

• Panel–census agreement was .588/.679 on bad units and .453/.827 on clean units; the audit treats environment semantics, not model consensus, as operative validity evidence.The blinded panel comprised GPT-5.6 and Claude Opus 4.8; bad-side violation agreement was κ=.705 (raw .86), while clean-side validity agreement was κ=.298 (raw .63), with low clean-side κ attributed to abstention rather than contradiction. The authors explicitly state that the model panel is not a human audit or ground truth.Source: Q Label-Validity Audit: Environment… · p. 22

Layer 3: exclusion stability

• The registered dispute set was empty, so the registered exclusion rerun changed no cell (∆=0).The set combined layer-1 state-equivalent cases with cases where both annotators concordantly vetoed; neither yielded any exclusions.Source: Q Label-Validity Audit: Environment… · p. 23
• Stricter exclusion variants preserved every argmax, although double-affirmed filtering moved rates by up to 0.131 and sharpened the retail peak.A conservative single-veto set moved no rate by more than 0.024. Double-affirmed-only filtering changed retail J(2) from 0.601 to 0.722 and J(8) from 0.035 to −0.028. Reason codes remained verifier-emitted, without human or model re-coding.Source: Q Label-Validity Audit: Environment… · p. 23

Anchor- and task-balanced reanalysis

• Equal-anchor and equal-task reweighting preserve arg maxL J in five of six cells, and the short-versus-long J contrast remains positive in all 12 scheme-by-cell resamplings.The sole peak shift is AppWorld/14B under anchor balancing, from 1 to 2 within its disclosed tie band.Source: R Balanced Reweighting, Matched Twins,… · p. 23

Matched-twin readings

• Within-anchor conditional AUC reproduces the global-pool pattern in all six score cells, raising AppWorld AUC by 0.02–0.10 while retail moves by no more than 0.03.Conditional AUC(2) is 0.811 [0.744, 0.874] for primary and 0.738 [0.683, 0.793] for AppWorld-70B.Source: R Balanced Reweighting, Matched Twins,… · p. 23

Per-anchor influence diagnostics

• Deleting any one retail anchor flips no peak-contrast sign and moves no arg maxL J; the largest single-deletion change in any FR(L) cell is 0.033.The jackknife covers 31 anchors and three retail verifiers; the 70B L=8 FR level stays within [0.933, 0.967] around 0.935.Source: R Balanced Reweighting, Matched Twins,… · p. 23

Where the cost constants come from

• The width experiment fits batched per-call verifier-token cost as 655.1 fixed tokens plus 135.3 tokens per reviewed item, while main-panel median prompt totals are 3,264–4,927 tokens.The experiment starts from an empty state; main-panel totals include the executed pre-window history, so refitting on those totals yields a negative i rather than the fitted marginal cost.Source: R Balanced Reweighting, Matched Twins,… · p. 23

Protocol transfer and sensitivity

• Across three retail-70B protocol variants, no extension has ∆J ≤0, leaving the safety-co-primary verdict “never extend” invariant.Since ¯λ scales linearly with F, a ±20% tokenizer band moves the holistic L∗=2→3 chord from 428.3 to [342.6, 514.0]; including history raises chords 4.98–7.52×, while common rescaling preserves L-ordering.Source: R Balanced Reweighting, Matched Twins,… · pp. 23–24

Judge reasoning settings

• The retail ranking decay already includes GPT-5.6 Sol at maximum reasoning effort, but a controlled low-versus-high reasoning-budget comparison on one judge remains untested.Claude Opus 4.8 used vendor-default adaptive thinking, and DeepSeek-V4-Pro used its API default.Source: R Balanced Reweighting, Matched Twins,… · p. 24

S Same-Backbone Domain-Trained Monitor

• Short-only training does not transfer to long windows: at L=8, J=0.040 and AUC=0.672, while all-length training largely flattens the J(2)−J(8) contrast.For short-only training, J(2)−J(8)=0.462 [0.276, 0.622] and AUC(2)−AUC(8)=0.240 [0.160, 0.284]; with all-length training, those contrasts are 0.043 [−0.064, 0.162] and 0.013 [−0.024, 0.043]. All-length is a seen-length ceiling, not evidence of length generalization.Source: S Same-Backbone Domain-Trained Monitor · p. 24, Figure A6 · p. 24, Table A20 · p. 25
• The trained control uses the same bf16 Qwen2.5-14B-Instruct backbone and rendering envelope as the zero-shot panel, with LoRA and matched bad/clean twin training.The causal-LM readout scores semantic-level digit rows without a newly initialized head; the short-only regime trains on L=1,2, while the all-length regime uses L=1,2,3,5,8.Source: S Same-Backbone Domain-Trained Monitor · p. 24
• The cost-aware optimum is L=1 for short-only and all-length training, but the all-length curve can support L=8 when read as flat for amortized cost.At L=8, all-length training repairs draft-clean false rejection and catches 0.750 at position 4; short-only catches 1.000 there but has gold-clean false rejection .935 versus .290 for all-length.Source: S Same-Backbone Domain-Trained Monitor · pp. 24–25

T Held-Out Validation of the Unit-Selection Recipe

• Across all six cells and both protocols, held-out calibration never selected a unit longer than two actions, and the held-out short-versus-long J contrast excluded zero in every cell.Retail 70B/14B selected L=2 in every fold; retail 72B and all three AppWorld cells selected L=1, with AppWorld 14B selecting L=1 in 52% and L=2 in 48% of repeated splits.Source: T Held-Out Validation of the… · p. 25, Table A22 · p. 26
• Held-out regret is upward biased because the test maximum is a winner’s-curse estimate, so leave-one-cluster-out regret exceeds repeated-split regret on the same data.The passage reports regret up to 0.302 for leave-one-cluster-out and up to 0.031 for repeated splits; it identifies selection into L∈{1,2} and a paired contrast against fixed L=2 as bias-free readings.Source: T Held-Out Validation of the… · p. 25

U Position Sweep: Worst-Case and Randomized-Offset Readings

• Position readings use exhaustive coverage only at L=1 and a coarse three-point grid at L=5 and 8; the randomized-offset reading assumes a hidden uniform offset and each step audited once.At L=2 and 3, only the end position was measured, so no worst-case reading is reported; the simultaneous lower bound combines a Bonferroni one-sided Wilson catch limit with an upper limit on false rejection.Source: U Position Sweep: Worst-Case and… · p. 26
• The one-action bound is positive in all six cells (0.043 to 0.445), whereas the eight-action bound is negative in all six (−0.172 to −0.973).The pre-registered retail contrast Jworst(5)−Jworst(8) is +0.112, +0.095, and −0.103, with task-cluster intervals containing zero in all three cells, so no claim is made.Source: U Position Sweep: Worst-Case and… · p. 26, Table A23 · p. 26

V Deployable Observation Substitutes: Predicted and Inert Arms

• At L=8, predicted and inert arms each repaired 0.104 of oracle-replay recovery; at L=5, prediction repaired 0.559 while inert repair was null.The L=8 oracle denominator was 0.455; predicted catch was 0.985 versus baseline 0.970, while oracle catch was 0.780. Predicted and baseline false-rejection slopes were both +0.484, versus −0.129 for oracle. Table A24 reports catch / FR / J by arm and length.Source: V Deployable Observation Substitutes:… · pp. 26–27, Table A24 · p. 27
• At AppWorld L=8, prediction repaired 0.630 of oracle advantage versus 0.989 for placebo, while neither arm repaired at L=5.The L=8 oracle denominator was 0.164. AppWorld returns empty observations for 43.7% of in-window steps, unlike retail’s full removal; on content-returning steps, field-value agreement was 0.117 and memorization recovery 0.049. AppWorld and retail rows were not pooled.Source: V Deployable Observation Substitutes:… · p. 27
• The deployable substitute’s fidelity fell with rollout depth, and its repair estimates are upper bounds because the predictor reproduced unreachable benchmark content.Predicted-oracle field-value agreement was 0.630 at L=2, 0.270 at L=5, and 0.065 at L=8; these exploratory lexical measures bound semantic agreement from below. AppWorld’s longer prompts also initially timed out under a retail context limit, prompting a rerun with pre-flight context and availability gates.Source: V Deployable Observation Substitutes:… · p. 27

W Reasoning-Budget Dose Controls

• The reasoning-budget controls use independent vendors, families, and knobs and are never pooled.The closed-judge comparison re-ran the same judge and frozen retail corpus at its lowest admissible effort, rather than disabling reasoning; minimal effort was rejected by the endpoint.Source: W Reasoning-Budget Dose Controls · p. 27, §W.1 · p. 27

W.1 Closed judge at its lowest admissible effort

• The closed judge’s AUC(2) − AUC(8) decay was +0.310 at highest effort and +0.325 at lowest effort, with no resolved effort-by-length interaction.The registered interaction was +0.015 (task-cluster CI [−0.067, 0.103]); both arms had complete availability and parse rates. Effort was confounded with file family and a 13-day interval against a floating model id.Source: §W.1 · p. 27

W.2 An open-weight reasoning verifier, thinking on/off

• For Qwen3-32B on retail, J(2) − J(8) was +0.691 with thinking ON and +0.255 with thinking OFF, so the length collapse was larger when thinking was enabled.The registered mode-by-length interaction was −0.471. Thinking improved short-window J (at L=2, 0.721 versus 0.289) but not L=8 J (0.029 versus 0.035); FR(8) reached 0.964 / 0.935. Table A26 reports J(L) by mode and domain.Source: Table A26 · p. 28, §W.2 · p. 28
• For Qwen3-32B on AppWorld, the mode-by-length interaction was −0.002, unlike retail’s larger ON-mode decay.J(2) − J(8) was +0.074 and +0.063 across the two modes, with intervals containing zero; J was highest at L=1 in both modes, an exploratory result. Availability was below the frozen panel’s 1.000 standard, and complete-length analyses selected out hard-to-parse items, limiting generalizability.Source: §W.2 · p. 28

X Compositional Two-Step Errors

• For compositional errors, maxL≤3 J − maxL≥5 J = +0.450 overall; from L=1 to 8, false rejection rises 0.490→1.000 while catch stays 1.000.At L=8, both twins are always rejected and J(8)=0. The corpus represents a weaker compositional form: c2 is harmful alone in all 100 items, and no admitted pair has an individually harmless c2.Source: X Compositional Two-Step Errors · pp. 28–29, Table A27 · p. 29

Y One Frozen Four-Exemplar Protocol and Its Decomposition

• The frozen four-exemplar protocol reduced FR(8) by 0.538 versus control, but ACCEPT-only reduced it by 0.615 while catch fell by 0.223, indicating a relaxed rejection threshold.A token-matched placebo moved FR by exactly 0.000, while the reject-only block also lowered rejection by 0.385. These results concern one frozen protocol, whose clean-side estimates rest on 26 items and whose exemplars share the evaluation generator.Source: Table A28 · p. 29, Y One Frozen Four-Exemplar Protocol… · p. 29

Z Episode-Level Utility Projections

• At N=8, L∈{1,2} minimizes tokens per completed episode in five of six gold-clean cells; retail 70B at L=2 uses 5,158 tokens per completed episode.These are halt-on-reject projections, not end-to-end runs. AppWorld 70B is an exception: L=5 costs 8,260 tokens, versus 13,591 at L=1 and 9,378 at L=8; the draft-clean scenario instead favors L=8.Source: Table A29 · p. 30, Figure A7 · p. 30, Z Episode-Level Utility Projections · p. 30

Verdicts

• At L=8, partial commit accepts 0.524 of clean steps versus 0.065 for whole-window binary, cutting tokens per committed clean action from 3,362 to 414 at the same 0.015 terminal bad-write release rate.The result is from the retail per-step protocol, with the bad write in the final slot; the 14B verifier reproduces the direction, while interior-position results are exploratory.Source: Verdicts · pp. 30–31

Score-Cascade Simulation

• The short-first Ls=2 cascade gains over fixed-Ls in two of six cells: +0.040 for retail 70B and +0.048 for AppWorld 70B, at 12–23% extra clean-side cost.Both thresholds and comparisons are test-tuned; the AppWorld 70B cascade remains below fixed L=1 at 0.526, and no long-first design wins.Source: Score-Cascade Simulation · p. 31

Reproducibility

• The reproducibility audit regenerated 44 of 45 analyses byte-identically, and the novel datasets’ public availability is partial because AppWorld-derived corpora ship as rebuild scripts and checksums.Some analyses require recorded invocation details, closed-model judgments are archived rather than guaranteed to be re-queryable, and an order-unstable artifact is disclosed.Source: Reproducibility · pp. 32–33

Forward-Drafted In-Window History

• On drafted-history items, J peaks at L=2 (0.458) and falls to 0.077 at L=8 as catch rises 0.774→1.000 and false rejection rises 0.387→0.923.The pre-registered L=1→2 contrast is inconclusive (+0.042, CI [−0.208, 0.292]); the between-arm double difference is not evaluable because only 13 of 31 anchors survive at L=8.Source: Forward-Drafted In-Window History · p. 33

Resampling Procedures

• The minimum detectable J(2)−J(8) effects at 80% power were 0.242/0.263/0.171 for the three retail cells and 0.082/0.104/0.046 for the three AppWorld cells.These are planning quantities from a normal approximation; the inferential test remains registered exact McNemar with a profile-likelihood interval.Source: Resampling Procedures · p. 33
• At L=8, clean false rejection rises with catch, reaching blanket rejection for the retail 14B and 72B verifiers rather than improved discrimination.The 70B rejects 94% of clean gold prefixes at L=8; the most severe error stratum is caught at 1.00 at every length, while the length signal lies in subtle strata.Source: Figure A8 · p. 34, Figure A9 · p. 34, Figure A12 · p. 35
• The retail clean-side effective sample size is 21–31, not 31, because task clustering yields design effects of 1.00–1.48.For the 72B at L=1, ICC is 0.60 and effective n is 21.0; AppWorld’s 200 clean items yield effective n of 133–172.Source: Resampling Procedures · p. 34
• The bootstrap places the informedness argmax at L=2 for the 70B and 14B, and L=1 for the 72B, in at least 91.6% of paired resamples.The respective frequencies are 98.4%, 91.6%, and 98.0%.Source: Figure A10 · p. 34
• At nominal 5%, item-level bootstrap rejection rates are .066/.054/.068 across the three retail cells; the task-cluster companion is conservative, not anti-conservative at G=17.The corresponding task-cluster rates are .046/.024/.042; AppWorld rates are .038–.062 for item bootstrap and .002–.006 for the cluster companion.Source: Resampling Procedures · p. 34
• The deployment rule prices any extension beyond the informedness peak against its supervision cost; a safety-co-primary deployment does not extend past the peak.At N=8, the reported cost slopes for the 70B’s L*→3/5/8 chords are 255/264/433 tokens per action per unit J, and the 72B’s 1→2 slope is approximately 1,980.Source: Figure A11 · p. 35

Reason-Code Vocabulary

• The verifier uses eight fixed reason codes in its structured JSON reject output, and the fixed taxonomy does not reclassify them.Three codes carry the reported ARG, EVID, and OBS buckets; the other five account for residual mass.Source: Reason-Code Vocabulary · pp. 35–36
• The verifier handles only the residual semantic boundary after deterministic gates and returns the longest leading prefix whose steps are all safe to attempt before execution.The first semantically bad step truncates the prefix, enforcing prefix closure.Source: Reason-Code Vocabulary · p. 37