PaperlayerView exact revision on arXiv

Source-linked AI summary

How Much Can Best-of-N Help a World Action Model? Measuring Selectable Headroom and Realizing It in Closed Loop

Haojia Gao, Yue Cao, Yumin Liu, Haohua Que, Mingkai Liu, Guanqiu Guo, Zhuo Zou, Bokui Chen, Xueqian Wang

arXiv:2610.37196v1cs.ROcs.CVcs.LG

TL;DR

Test-time selection for world action models is bounded by how much better the best candidate truly is, yet one-shot estimates mix candidate quality with continuation randomness. The paper measures realizable headroom with repeated, held-out counterfactual branching and uses the findings to build a selector that improves closed-loop success.

  • Problem

    The standard single-continuation estimate of selectable headroom conflates candidate quality with randomness in what happens afterward, obscuring how much gain a selector could realize.

    Source: §1 · p. 1

  • Method

    The paper uses reproducible counterfactual branching, repeated continuations, held-out oracles, and common random numbers to measure realizable headroom and build a failure-gated selector.

    Source: §1 · p. 2

  • Results

    The selector raises closed-loop task success from 76.9% to 81.6% on reserved training-task episodes and from 62.6% to 69.8% on unseen tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Structured proposals create headroom where the policy fails, and imagined futures already carry candidate-outcome information that can rank those proposals.

    Source: §1 · p. 2

    Measurements cover four WAMs on two MuJoCo-based benchmarks with bitwise-reproducible branching; physical-robot resets approximate the held-out estimate up to reset noise.

    Source: §7 · p. 8

Abstract

from arXiv · show

World action models (WAMs) predict the visual consequences of the actions they generate, which makes test-time selection a natural way to improve a fixed policy: imagine the futures of candidate actions and execute the best one. The benefit of selection is bounded by the selectable headroom, how much better the best candidate truly is, but the standard estimate, which executes each candidate once and keeps the best outcome, conflates candidate quality with the randomness of everything that follows. We introduce a denoised counterfactual protocol, based on bitwise-reproducible branching, repeated continuations, held-out oracles, and common random numbers, that measures the headroom a selector can realize, and use its measurements to build a fail-gated selector. Across four WAMs, two benchmarks, over 100,000 branched rollouts, and pre-registered tests, we find that (i) the standard estimate overstates the selectable headroom of native samples: bounds from 17 continuations per candidate place its selectable share between 24% and 66% for three WAMs; (ii) structured action perturbations create up to 16 points of selectable headroom; and (iii) in both WAMs with action-conditioned imagination, imagined futures carry candidate-outcome information with no detectable loss relative to real futures, information that the value head of Cosmos Policy leaves unused. Deviating from the policy chunk only when a readout of the WAM predicts failure, the selector raises closed-loop task success from 76.9% to 81.6% on held-out episodes of training tasks, with the largest gain where the policy fails most often, and from 62.6% to 69.8% on unseen tasks.

1. Introduction

World action models make test-time selection possible by imagining candidate-action futures, but its benefit is capped by how much better the best candidate truly is. The paper measures this selectable headroom while separating candidate quality from continuation randomness, then uses the findings to build a fail-gated selector that improves closed-loop success.

1. Introduction

  • WAMs jointly produce action chunks, predicted post-action scenes, and value estimates, making it possible to sample candidate chunks, imagine their futures, and execute the best-looking one.Related sampling-and-verification schemes have been used for language-model reasoning and VLA policies.Source: §1 · p. 1
  • A denoised counterfactual protocol uses reproducible branching, repeated continuations, held-out oracles, and common random numbers to measure realizable headroom across four WAMs and two benchmarks.The evaluation uses more than 100,000 branched rollouts and pre-registered tests.Source: §1 · p. 1
  • With 17 continuations per candidate, native-sample selectable-share bounds are 24%–52% for Cosmos Policy, 33%–66% for Flex-π, and 33%–63% for Fast-WAM.The paper reports that the single-continuation oracle overstates native-sample selectable headroom.Source: §1 · p. 2
  • Structured action perturbations create headroom where the policy chunk fails, while imagined futures carry candidate-outcome information that the Cosmos Policy value head leaves unused.For both WAMs with action-conditioned imagination, imagined-future readouts rank candidates with no detectable loss relative to real-future readouts.Source: §1 · p. 2
  • The selector raises closed-loop task success from 76.9% to 81.6% on reserved training-task episodes and from 62.6% to 69.8% on unseen tasks.It deviates from the policy chunk only when a learned readout predicts failure.Source: §1 · p. 2

2. Related work

Prior work proposes diverse policy selectors and WAM-specific verifiers, while a closely related branching protocol measures test-time planning outcomes. This paper extends that measurement design with repeated continuations, held-out oracles, controlled proposals, and representation-level readouts.

2. Related work

  • Existing robot-policy selectors sample or perturb actions and rank them with VLMs, value functions, calibrated abstention, or other verification strategies; WAM approaches also score imagined futures or internal signals.Other approaches use imagination to refine a chunk or decide when to stop executing it.Source: §2 · p. 3
  • Closest to this work, Yuan et al. [53] use same-state counterfactual branching for WAM test-time planning; this paper adds repeated continuations, held-out oracles, controlled proposals, and representation-level readouts.The paper also uses common random numbers and pre-registered tests.Source: §2 · p. 3

3. A denoised counterfactual protocol

The protocol branches candidates from reproducible simulator states, repeats policy continuations, and uses held-out evaluation to estimate how much of the candidate headroom is selectable. It also decomposes outcome variation and tests structured proposals and readouts of imagined futures.

3. A denoised counterfactual protocol

  • For candidate c, p_c is its task-success probability after execution from the decision state followed by policy actions; selectable headroom compares the best candidate with uniform choice.When the policy chunk is in the pool, the protocol also reports gain over that chunk.Source: §3 · p. 3
  • Bitwise-reproducible branching restores the full simulator, random-number, and fixture states, reproducing seeded replay byte for byte.After executing a candidate chunk, the resulting state is snapshotted so its continuations start from the same state.Source: §3 · p. 3
  • The held-out oracle selects using some continuations and scores on an independent one, estimating realizable selection; the in-sample oracle is an upper bound and the held-out estimate a lower bound in expectation.The single-continuation oracle cannot distinguish candidate quality from continuation luck.Source: §3 · p. 3
  • With shared seed schedules, the variance decomposition separates candidate effects from schedule effects and candidate-by-continuation interaction, which no selector can exploit.For structured proposals, candidate variation is further divided into shared direction effects and state-by-direction interaction.Source: §3 · p. 3
  • The overview pairs repeated branching with a failure gate that imagines and ranks twelve same-noise axis offsets only when the gate score exceeds its threshold.The readouts are trained on branch outcomes rather than taken from the value head.Source: Figure 2 · p. 4
  • Efficient sampling gives every candidate one continuation first, then allocates additional continuations to differing-outcome points and a weighted hash-selected quarter of remaining points.Estimates use only second-stage continuations, weighted by inverse sampling probability.Source: §3 · p. 4
  • Native candidates are independent policy samples; structured candidates displace the policy chunk along robot-base axes at specified scales while leaving orientation and gripper unchanged.The described scales move the end effector by about 2, 5, or 10 cm.Source: §3 · p. 4
  • Linear probes predict within-decision outcomes from final-block tokens at imagined future-frame positions, with state-held-out fitting and permutation-based null testing.Evaluator seeds are independent of candidate sampling.Source: §3 · p. 4

4. Experimental setup

The experiments evaluate four WAMs on RoboCasa and LIBERO-Plus, using selected perturbation categories and branching datasets collected under a pre-registered testing plan.

4. Experimental setup

  • Four WAMs are evaluated on LIBERO-Plus Objects Layout and Robot Initial States perturbations; Cosmos Policy is also evaluated on RoboCasa.The LIBERO-Plus categories were chosen because every model succeeds 30–90% of the time on them; the selector is built on Cosmos Policy.Source: §4 · p. 4
  • The imagination probes cover Cosmos Policy and Streaming-WAM, while headroom measurements include all four WAMs.Streaming-WAM imagines consequences from its first eight executed steps.Source: §4 · p. 4
  • Figure 3 compares headroom as candidate count grows under the Yuan et al. [53] protocol on three RoboCasa tasks with 16 native candidates.Continuation seeds are shared across candidates; the single-continuation oracle rises with K while the held-out oracle stays near zero, with 95% confidence intervals.Source: Figure 3 · p. 5
  • Twelve branching datasets were collected, and every confirmatory main-text test was pre-registered before its data were collected.The datasets comprise four RoboCasa and eight LIBERO-Plus sets, with non-overlapping initial-state hashes across datasets and evaluation or training episodes.Source: §4 · p. 5

5. Results

Across the tested WAMs, native-candidate oracle gains largely reflect continuation randomness, whereas structured perturbations create selectable headroom. Imagined futures retain candidate-outcome information, and a fail-gated selector improves closed-loop success.

5.1. The single-continuation oracle is dominated by continuation randomness

  • At K = 16, the held-out oracle is 0.009 [−0.001, 0.023] versus 0.102 [0.036, 0.190] for the single-continuation oracle; 98% of distinguishing outcome variation is re-drawn across continuation seeds.A selector observing three independent outcomes per candidate realizes 0.086 [−0.009, 0.270] of the measured gain on a fresh continuation.Source: §5.1 · p. 5
  • With 17 continuation schedules, selectable shares are bounded at 24–52% for Cosmos Policy, 33–66% for Flex-π, and 33–63% for Fast-WAM.The standard single-continuation oracle overstates the native selectable share by at least 1.5-fold for Cosmos Policy and 1.27-fold for the other two models.Source: §5.1 · p. 5

5.2. Structured proposals create selectable headroom

  • On TurnOnStove, structured proposals yield +0.158 [0.081, 0.247] held-out oracle gain over the policy chunk at κ = 0.25; 67–95% of structured-candidate gain survives held-out evaluation.That retained share holds on both benchmarks and across all four WAMs, compared with at most 22% for native candidates.Source: §5.2 · p. 6
  • Structured perturbations spread outcomes in both directions, with headroom concentrated where the policy chunk fails and the useful direction depending on state.At κ = 0.5, the average perturbed candidate is 7–14 points below the policy chunk; a fixed direction chosen on other states does not improve on it.Source: §5.2 · p. 6

5.3. The imagination carries the signal that the value head misses

  • Fine-tuning raises value-head AUROC for final success from 0.53 to 0.87–0.90, but its native-candidate outcome ranking remains essentially unrelated (|ρ| ≤ 0.05).With common random numbers the ranking is stable across seeds but not informative; with independent seeds, seed-to-seed noise dominates.Source: §5.3 · p. 6
  • Imagined-future linear probes rank perturbed candidates on TurnOnStove (ρ = +0.31, p = 0.001, κ = 0.25) and LIBERO-Plus (ρ = 0.26, p = 0.020, κ = 0.1), passing each reported pre-registered test.On LIBERO-Plus, Cosmos Policy’s imagined-versus-real probe difference was +0.04 [−0.06, 0.13], within the pre-registered ±0.15 equivalence margin.Source: §5.3 · p. 6
  • Future-frame probes retain candidate-outcome information beyond action-frame probes, while the scalar value-head output yields ρ = 0.01.After removing information explained by the action-frame probe, future frames retain 0.11 [0.05, 0.17], whereas the action frame retains 0.02 [−0.04, 0.07] beyond future frames.Source: §5.3 · p. 7

5.4. A selector built on the findings improves on the policy

  • 76.9% to 81.6% closed-loop task success on reserved training-task episodes, and 62.6% to 69.8% on unseen tasks.The frozen selector's gain is largest under robot initial-state perturbations, where success rises from 65.1% to 75.8%.Source: Table 1 · p. 7, §5.4 · pp. 7–8

6. HeadroomBench

HeadroomBench packages branching measurements and selector training and evaluation data, with reference results for several selector types.

6. HeadroomBench

  • HeadroomBench provides a headroom track and a selector track, including training and test splits, closed-loop evaluation lists, and reference results.On its test split, the held-out oracle exceeds the policy chunk by 11 points, while value-head and random selection each lose 8–10 points.Source: §6 · p. 8

7. Discussion and conclusion

The paper concludes that selection is bounded by selectable headroom and reports gains from a fail-gated selector, while noting that its measurements rely on reproducible branching. Its confirmatory tests were registered before data collection.

7. Discussion and conclusion

  • The standard oracle largely reflects continuation randomness; consequential proposals at likely policy-failure states create headroom that imagined futures can help rank.A selector that deviates only when a learned readout predicts failure improves in closed loop and carries about half its gain to tasks absent from training.Source: §7 · p. 8
  • Measurements cover four WAMs on two MuJoCo-based benchmarks where bitwise-reproducible branching is available.On physical robots, repeated resets can approximate the held-out estimate only up to reset noise; deployment of the selector itself does not require state restoration.Source: §7 · p. 8

A. Pre-registration

  • Every confirmatory test was registered before the first rollout of the dataset it introduced.The paper lists 24 tests in registration order and states that pre-registration texts and repository history will be released with HeadroomBench.Source: A. Pre-registration · p. 12

B. Datasets and protocol details

The protocol validates reproducible branching and rendering, then applies shared state and continuation schedules across models and tasks. Dataset splits and proposal settings define the scope of these measurements.

B. Datasets and protocol details

  • Snapshot branching is checked against seeded replay, byte-identical episode reruns, and online and offline rendering checks.Two genuine dark-camera occlusions were retained; removing them changes no reported result by more than 0.003.Source: B. Datasets and protocol details · p. 12
  • LIBERO-Plus evaluation selected Objects Layout and Robot Initial States, where all models had 30–90% success in a preliminary run.E5 used 80 shared states; E6 used 56 different states.Source: B. Datasets and protocol details · p. 14
  • The protocol uses shared continuation schedules, with E5+ averaging 17 schedules and E5 averaging four, so their estimates of ¯O1 differ.For Cosmos Policy, the example estimates are 7.9 and 7.3 points, respectively.Source: B. Datasets and protocol details · p. 14
  • Native candidates use each model’s chunk length, while structured offsets are scaled to match cumulative commanded displacement across settings.At κ = 0.1 and 0.5, offsets move the end effector about 2 cm and 6–10 cm.Source: B. Datasets and protocol details · p. 14

C. Estimators

The estimators separate candidate effects from continuation noise and bound what a selector can realize. The resulting selectable-share estimates are exploratory when based on a Gaussian approximation.

C. Estimators

  • The in-sample oracle upper-bounds selectable headroom, while the held-out oracle lower-bounds it by selecting on continuations independent of scoring.The standard single-continuation oracle is invariant to candidate-label shuffling within a continuation and cannot distinguish candidate quality from continuation luck.Source: C. Estimators · pp. 14–15
  • Two-stage estimates use inverse-probability weights for decision points selected for further continuations, with state-cluster bootstrap intervals.Points with differing first-stage outcomes receive weight 1; a hash-selected quarter of the others receives weight 4.Source: C. Estimators · p. 15
  • The variance decomposition estimates candidate-effect variance after accounting for residual interaction variance, which includes binomial noise.The model-based selectable-share curve assumes candidate means differ with variance τ^2 and ranks them using m shared schedules.Source: C. Estimators · p. 15
  • 31% [14] [49] of ¯O1 is the moment-based selectable-share estimate for Cosmos Policy; Flex-π and Fast-WAM estimates are 46% [26, 71] and 45% [23, 78].The Gaussian approximation ignores heterogeneity across decision points, so these estimates are reported as exploratory.Source: C. Estimators · p. 15
  • Both ends of the selectable-share bracket remain within reported ranges when any single state is left out for Cosmos Policy and Flex-π.The stability analysis reports ranges of [21, 30]% and [50] [55]% for Cosmos Policy, and [28] [37]% and [64, 71]% for Flex-π.Source: C. Estimators · p. 15

D. Additional results on continuation randomness

Across RoboCasa and LIBERO-Plus, repeated continuations show that native-candidate outcome variation is largely interaction with continuation randomness. Judge-free consensus selection yields little evidence of reliable per-decision gains.

D. Additional results on continuation randomness

  • On TurnOffStove, the held-out oracle gain is −0.003, despite 19% of decision points having mixed outcomes under the first schedule.Removing two trunks flagged by rendering checks leaves the ratio and permutation test unchanged.Source: D. Additional results on continuation… · p. 16
  • On LIBERO-Plus native candidates, candidate effects are at most 0.003, versus interaction variance of 0.015–0.026; Streaming-WAM’s effect is indistinguishable from zero.Its held-out oracle lies at the permutation-null center (p = 0.81).Source: D. Additional results on continuation… · p. 16
  • KeyStone’s main native-candidate analysis finds no estimate different from zero, and its per-decision effect is bounded at about 0.4 points per query.Two positive intervals among twelve reference and sensitivity intervals are treated as exploratory.Source: D. Additional results on continuation… · p. 16

E. Additional results on structured proposals

Structured action offsets create task-dependent headroom, and their gains concentrate where the policy chunk fails. The reported comparisons also show that larger sampling variance alone does not create native-candidate headroom.

E. Additional results on structured proposals

  • On TurnOnStove, TurnOnMicrowave, and PnPCounterToSink, structured proposals have +0.112 to +0.158 headroom; on PnPCounterToCab, headroom is +0.027.Increasing Cosmos Policy’s sampling variance does not create native-candidate headroom on TurnOnStove.Source: E. Additional results on structured… · p. 16
  • At κ = 0.5 on LIBERO-Plus, the average perturbed candidate is 7.0–11.5 points below the policy chunk across the four WAMs.The values are 9.5 for Cosmos Policy, 11.5 for Fast-WAM, 7.0 for Flex-π, and 8.7 for Streaming-WAM.Source: E. Additional results on structured… · p. 16
  • In all four WAMs, the held-out gain over the policy chunk comes from cases where that chunk fails in at least one selection continuation; elsewhere it is close to zero.Streaming-WAM has fewer such cases and the smallest average gain, although its gain when the chunk fails is as large as for the other models.Source: E. Additional results on structured… · pp. 16–17

F. Additional results on the imagination

Action-conditioned imagined futures carry information about candidate outcomes, and their predictive fidelity depends on task and perturbation scale. The value head generally does not reliably rank candidates, while structured perturbations create selectable headroom on several RoboCasa tasks.

Value heads

  • Value-head rankings are generally inconclusive: all within-decision correlation and selection-gain CIs include zero except the 10K head on perturbed candidates (ρ = +0.18 [0.004, 0.356]).With independent candidate seeds, value noise exceeds between-candidate differences by an order of magnitude, and independent argmaxes agree only 15–27% of the time (chance: 1/6).Source: F. Additional results on the imagination · pp. 17–18

Structured perturbations

  • Structured perturbations create selectable headroom on TurnOnStove, TurnOnMicrowave, and PnPCounterToSink, although the average perturbed candidate is worse than the policy chunk.The table caption defines a related split by whether the policy chunk fails across continuations, but supplies no cell values.Source: Figure 8 · p. 18

Fidelity of the imagination

  • Imagined futures decode commanded 2 cm displacement direction at R² = 0.43–0.45, at least as well as real futures, and reproduce consequences distinguishing sibling candidates.The figure examples include the kettle entering view under −y and the higher viewpoint under +z on RoboCasa, and a basket edge under the failing +y offset on LIBERO-Plus.Source: F. Additional results on the imagination · p. 18, Figure 9 · p. 19

Scope

  • The future-readout probe applies only to Cosmos Policy and Streaming-WAM because it requires action-conditioned predicted futures at test time.Flex-π generates its future jointly with its own action, while Fast-WAM does not generate a future at test time.Source: F. Additional results on the imagination · p. 18

When the information appears

  • For TurnOnStove, 2 cm perturbations yield reliable outcome differences (split-half Spearman +0.39 [0.16, 0.64]) that neither real nor imagined futures at tb + 16 or tb + 32 reveal.At 5–10 cm the consequence is visible in both futures; on TurnOnMicrowave it is visible at 2 cm (ρ = +0.31), and whenever real near-future probes rank candidates, imagined-future probes do too.Source: F. Additional results on the imagination · pp. 18–19

Fidelity of the imagination

  • For Streaming-WAM, imagined and real future-token probe rankings are equivalent within the pre-registered band in the initial test and on the E11 repetition.On E11, imagined and real future-token probes reach ρ = 0.157 [0.077, 0.232] and 0.190 [0.113, 0.263], respectively; the paired difference is −0.033 [−0.091, 0.024].Source: F. Additional results on the imagination · p. 19

Readouts

  • Wrist-camera tokens rank candidates for imagined and real futures at ρ = 0.207 [0.089, 0.323] and 0.247 [0.091, 0.382], respectively.Third-person camera correlations are 0.138 (p = 0.062) and 0.207 (p = 0.010); supplied action-expert tokens reach 0.125 (p = 0.069).Source: F. Additional results on the imagination · p. 20

Future versus action frames

  • Across twelve conditions, future-frame predictions retain 0.13 [0.08, 0.18] outcome-ranking information beyond a current-state-by-direction baseline.The analysis residualizes predictions on the leave-one-state-out global direction and compares future- and action-frame predictions within decision points.Source: F. Additional results on the imagination · p. 20

G. Selectors

The selector uses a failure gate to decide when to deviate and a ranker to choose an offset, while HeadroomBench provides branching and evaluation tracks for measuring headroom and selector performance. On unseen tasks, gains are positive on most tasks, but benefits and costs vary across settings.

G. Selectors

  • Ranking with abstention alone selects an infinite threshold within the pre-registered grid and never deviates, because the average offset is worse than the policy chunk.An exploratory wider ridge-strength grid reaches up to +0.02 without nested cross-validation or a CI, so the two selectors were not tuned with matched grids.Source: G. Selectors · p. 20, Table 11 · p. 21
  • The fail-gated selector combines a policy-failure logistic gate with a centered dual-ridge ranker; nested cross-validation estimates +0.025 [0.009, 0.041] gain over the policy chunk.The selected configuration uses all nine imagination frames for the gate, twelve offsets, and a 60th-percentile gate threshold; it deviates at 35% of held-out decision points.Source: G. Selectors · p. 20
  • On E8 episodes of training tasks, success rises from 65.1% to 75.8% under robot initial states, while object-layout results are 87.4% versus 88.6% for the policy.E9 shows the trade-off: 42% of deviations occur where the policy always fails, while deviating where it always succeeds costs 7.6 points.Source: G. Selectors · p. 21
  • The gate’s current-scene representation contains failure information discarded by the scalar value head, while state-dependent ranking approaches full-representation gains within tasks.Under task-held-out cross-validation, the full, imagined-future, and state-by-direction rankers cannot be separated; imagined-future rankers were selected in all four unseen-task folds.Source: G. Selectors · pp. 21–22
  • At 18% deviation, or 2.2 imagination passes per decision point, the selector retains +0.016 [0.004, 0.029] gain; selected thresholds yield +0.025 [0.009, 0.041].Imagining all twelve offsets at every decision point gains +0.009 [−0.010, 0.029], while the maximum gain is +0.026 [0.009, 0.044] when the gate fires about half the time.Source: G. Selectors · pp. 21–22, Figure 10 · p. 22
  • +7.6 points median gain across 40 unseen tasks; the selector improves success on 30 tasks, with larger gains where the policy succeeds less often (exploratory r = −0.44).The selector that never saw the task gains +4.9, +12.4, +10.0, and +1.8 points across four folds.Source: G. Selectors · p. 22
  • Offline RoboCasa selection gains appear on TurnOnMicrowave but remain within noise on TurnOnStove and PnPCounterToSink; state-dependent direction learning needs E7-scale data.With only 15–36 training states per task, the imagined-future probe recovers 80% of TurnOnMicrowave headroom at κ = 0.5, but the gain does not persist when entire scenes are held out.Source: G. Selectors · p. 23

H. HeadroomBench

  • HeadroomBench’s headroom track measures single-continuation and held-out oracle gains, headroom over the policy chunk, and variance decomposition; its selector track scores candidate chunks against the policy chunk.On the E6 selector-track test split, the held-out oracle exceeds the policy chunk by 11 points, while value-head and random selection each lose 8–10 points; the fail-gated selector gains +0.020, with E6 informing its design.Source: H. HeadroomBench · p. 24, Table 14 · p. 24

Related papers

Loading 2610.37196v1…