PaperlayerView exact revision on arXiv

Source-linked AI summary

In CEM, a World Model Is Also a Proposal Mechanism

Oliver Obst, Frieder Stolzenburg

arXiv:2610.00921v1cs.LGcs.RO

TL;DR

CEM model scores affect both current candidate selection and the future search pools, so evaluating only rankings on existing pools leaves the proposal-update role untested. The paper combines crossed rescoring with a matched one-update intervention; proposal distance falls across all task-seed units, and the intervention lowers final selected-sequence cost in the selection units.

  • Problem

    CEM elite decisions determine later candidate pools, so fixed-pool ranking evaluation alone cannot show what lower-cost sequences another proposal might have sampled.

    Source: §1 · pp. 1–2

  • Method

    The study crosses four surrogate scorers with saved pools from each surrogate’s CEM trace, then replaces one first elite update with an environment-ranked update under matched continuation.

    Source: §1 · p. 2

  • Results

    Proposal distance fell from iteration zero to three in all 12 units, and the environment-ranked first update lowered final selected-sequence cost in all six selection units.

    Source: §6.1 · p. 8, §6.3 · p. 9

  • Takeaways & Limitations

    CEM model scores shape both the ordering of current candidates and the distribution of later queries, which the crossed audit and splice examine separately.

    Source: §7 · p. 10

    The splice is an oracle diagnostic, and its finite-horizon selected-sequence cost does not establish higher closed-loop return.

    Source: §4 · p. 6, §7 · p. 10

Abstract

from arXiv · show

The cross-entropy method (CEM) uses world-model scores to select action sequences and fit the distribution sampled in its next iteration. A scoring error can therefore change both the present decision and the candidates considered later. We evaluate these two roles separately. Four types of predictive model generate CEM traces, and every model rescores every saved candidate pool. Executing the same candidates in the environment provides a reference elite set and proposal update. Across twelve independently trained task-seed units on Walker and Cheetah, the pre-specified proposal distance falls from the first to the final CEM iteration in every unit. Proposal widths contract and fitted means separate relative to the remaining search width. Pairwise ranking agreement stays near chance on Walker and declines on Cheetah; elite-set agreement does not improve. This comparison shows greater variation between scorers than between pool sources on Cheetah; Walker has variation in both and in their pairings. We use the original six units to select Random nonlinear for a one-update intervention, without inspecting intervention outcomes. Replacing its first model-ranked update with an environment-ranked update lowers final realised selected-sequence cost in those six units and in six further units held out from the selection.

1 Introduction

CEM model scores affect both which candidates are selected now and which candidates appear in later search pools. The paper separates these roles through crossed rescoring and a matched one-update intervention.

1 Introduction

  • An earlier CEM update can leave the planner searching a higher-cost region, even when a later scorer orders that pool correctly.The illustrative pools show that correct within-pool ranking does not guarantee that low-cost candidates are available.Source: §1 · p. 1
  • CEM creates planner-induced query shift because elite decisions change its future candidate distribution, even when the physical system is unchanged.Logged rollouts and actor-centred tests evaluate chosen action distributions, whereas CEM changes its distribution after each elite decision.Source: §1 · p. 2
  • Crossed rescoring compares scorers on identical pools and a fixed scorer across pool sources, while realised costs show which pools contain lower-cost candidates.A fixed-pool comparison cannot show which candidates another proposal would have sampled.Source: §1 · pp. 1–2
  • Four surrogates replace Dreamer dynamics during planning, share data and CEM settings, and differ in their feature maps.A fixed readout maps predicted future contexts to task variables used in planning cost.Source: §1 · p. 2
  • Each surrogate generates pools through its own CEM history, and every scorer evaluates every pool in a 4-by-4 crossed design.The matrix describes disagreement associated with generators, scorers, and their pairings; it does not assign causal effects.Source: §1 · p. 2, Figure 1 · p. 2
  • The one-update splice replaces the first model-ranked elite set with an environment-ranked one, then continues both branches with the original scorer and matched random samples.The endpoint is the realised cost of the selected sequence after the remaining CEM iterations.Source: §1 · p. 2

2 Related work

Prior work evaluates models through their effects on value, policy, or control, and proposes related diagnostics and planning changes. This paper distinguishes its crossed CEM comparison and update intervention from those approaches.

2 Related work

  • Value-aware learning, DeepMDP, value equivalence, and analyses of likelihood-control mismatch motivate evaluating models through their intended use [Farahmand et al., 2017; Gelada et al., 2019; Grimm et al., 2020; Lambert et al., 2020].The paper holds predictors fixed and examines finite candidate decisions made by CEM.Source: §2 · pp. 2–3
  • Yu et al. [2026] propose interventional action-fidelity, policy-ranking, and optimisation-utility evaluation; Bhamidipaty et al. [2026] analyse policy-preference reversals as model exploitation.This paper measures finite candidate disagreements, not worst-case exploitability over a policy class.Source: §2 · p. 3
  • Wang et al. [2026] measure ranking agreement along CEM search, while this paper also tests scorers across pools from other surrogates and compares fitted proposal means and widths.A rank correlation alone does not specify the proposal refit.Source: §2 · p. 3
  • Li et al. [2026] execute candidates to distinguish failed selection from absent feasible candidates; this paper intervenes on an earlier elite refit and follows later pools.The intervention tests how proposal history changes candidates available at final selection.Source: §2 · p. 3
  • Recent methods alter scoring objectives or proposal updates, and the crossed comparison could test which measured disagreements those changes reduce.The cited approaches include perturbation-ranking training, trajectory-level costs, objective changes, and learned plan-improvement operators [Chahe et al., 2026; Huang et al., 2026; Singh, 2026; Sommer and Schilling, 2026].Source: §2 · p. 3

3 Auditing a CEM update

The audit compares model decisions with environment outcomes on candidates sampled from a fixed planning problem. It measures both within-pool selection and disagreement between the next proposals fitted from model and environment elites.

3 Auditing a CEM update

  • The audit saves CEM pools from a fixed simulator state and Dreamer context, executes each candidate from a restored state, and uses realised cost as the within-pool reference ordering.Restoring the state prevents one candidate from changing another candidate’s starting point.Source: §3 · p. 3
  • Each candidate is a flattened 90-dimensional vector of 15 actions with six coordinates each, and the proposal is a diagonal Gaussian without fitted coordinate correlations.Flattening lists all action entries without discarding or averaging actions.Source: §3 · p. 3
  • The scorer’s 32 lowest-cost candidates form its elite set, while the 32 lowest realised-cost candidates form the environment elite set; precision and pairwise agreement assess pool-level selection.Expected elite precision is 0.125 under chance overlap, and selected-candidate regret measures cost relative to the best available candidate.Source: §3 · pp. 3–4

3.1 Comparing the two proposal refits

  • Because elite membership alone does not determine the magnitude of an update error, the audit applies CEM’s refit to both elite sets.Similar actions can yield similar proposals despite disjoint elites, while a small membership change can move the proposal if sequences are far apart.Source: §3.1 · p. 4
  • The pre-specified proposal distance combines mean and log-width disagreement; its mean term measures refit-centre separation relative to current proposal widths.Equal fitted widths contribute zero to the log-width comparison, while D = 0 means the refits produce the same diagonal Gaussian.Source: §3.1 · p. 4
  • The audit separately measures first-action refit disagreement, selected first-action gaps in raw and proposal-width units, and clipping and width-floor effects.MPC executes only the first of the 15 planned actions before replanning, so full-plan averages can conceal disagreement in that action.Source: §3.1 · p. 5

4 Crossed audit and one-update splice

The crossed audit separates scorer and pool-source patterns by applying each scorer to each generator’s saved pools. A matched splice then changes one early elite decision and tracks the final selected sequence’s realised cost.

4 Crossed audit and one-update splice

  • The crossed matrix scores each generator’s fixed pool with every surrogate, with rows holding pools fixed and columns following a scorer across pool sources.The diagonal is the usual model-driven trace, where the generator and scorer match.Source: §4 · p. 5
  • Subtracting each scorer’s common-pool baseline distinguishes its initial disagreement from later change, while row and column patterns associate variation with pool source or scorer.These matrix patterns describe associations, not causal effects.Source: §4 · p. 5
  • The two-way residual identifies cell-specific pairing discrepancies beyond row and column averages, and variance shares are averaged within task-seed matrices.The shares describe the four fixed families, not population variance estimates.Source: §4 · p. 5
  • Every generator begins from the same proposal and random samples, so later pools differ through preceding elite decisions rather than independent sampling variation.The iteration-zero pools are verified to have identical candidate IDs and environment costs.Source: §4 · pp. 5–6
  • The splice forks from a common pool, fits one proposal from model elites and one from environment elites, then uses the original scorer and matched samples in both branches.Only the first elite update differs between the branches.Source: §4 · p. 6
  • The primary endpoint is branch-E minus branch-M realised incumbent cost after iteration three; a negative contrast favours the environment-elite branch.The audit cannot establish that an early update caused harm from rescoring alone, and the splice is an oracle diagnostic using finite-horizon selected-sequence cost rather than closed-loop return.Source: §4 · pp. 5–6

5 Experimental setting

The audit compares four surrogate models across Walker and Cheetah using independently trained task-seed units, fixed CEM settings, and saved candidate pools. It also separates the original cohort used to select an intervention from the extension and held-out cohorts.

5 Experimental setting

  • Each surrogate generates CEM pools that are executed from restored simulator states and rescored by every surrogate, crossing four generators with four scorers.The actor supplies the initial CEM mean, and the frozen Dreamer backbone supplies contexts for the independently trained units.Source: §5 · pp. 6–7
  • Four surrogate families differ in feature map and decoder; the three compressed families use width 256, while Identity retains the 1280-dimensional context.All four share the action-conditioned transition form; the compressed families also share decoder architecture, auxiliary heads, optimiser, data, and training budget.Source: §5 · p. 6
  • Seeds 0–2 form the original six-unit cohort for splice selection, while seeds 3–5 were fixed in advance as an extension and cannot change that selection.The splice and decision rule were also fixed before inspecting the extension units’ splice costs.Source: §5 · pp. 6–7
  • Across 12 task-seed units, diagonal audit measurements track four CEM iterations while candidate count, elite cut, and horizon remain fixed.Thin lines show six units per task after averaging the four surrogates; heavy lines show task means.Source: §5 · p. 6, Figure 2 · p. 7
  • Each audit run uses four iterations, 256 candidates, 32 elites, and a proposal width starting at 0.35 with a 0.05 floor.Candidates are evaluated over 15 predicted steps; four episode bins supply replay-checked states, and repeated executions verify costs.Source: §5 · p. 7

6 Results

The proposal distance fell across all task-seed units, but ranking agreement did not improve; a one-update environment-ranked intervention lowered final selected-sequence cost in both cohorts. The crossed audit found scorer variation dominant on Cheetah, while Walker showed variation across scorers, pool sources, and their pairings.

6.1 The proposal distance falls without better candidate ranking

  • D fell from 0.254 to 0.235 on Walker and from 0.252 to 0.235 on Cheetah between iterations zero and three.Every task-seed contrast was negative; the two components moved oppositely, as the log-width term fell while normalized mean disagreement rose.Source: §6.1 · p. 8
  • The first-action gap shrank in raw units but grew relative to the proposal width, as CEM’s search region contracted faster than the actions’ separation.The same scale dependence was present at the first planned step.Source: §6.1 · p. 8
  • Only 1.2% of Walker coordinates and 1.4% of Cheetah coordinates reached the width floor by iteration three.The mean displacement from clipping also decreased.Source: §6.1 · p. 8
  • Pairwise order agreement stayed at 0.503 on Walker and fell from 0.535 to 0.523 on Cheetah, while chance-adjusted elite precision did not improve.Neighbouring elite cuts on original seeds 0–2 kept task-mean adjusted precision below 0.10.Source: §6.1 · p. 8

6.2 Scorer variation dominates on Cheetah

  • On Cheetah, scorer columns accounted for 66.2% of descriptive sum of squares, versus 10.0% for pool-source rows and 23.8% for pairings; Walker was mixed.Scorer variation exceeded pool-source variation in all six Cheetah seeds, compared with two of six Walker seeds.Source: §6.2 · p. 8, Figure 3 · p. 9
  • Random nonlinear had a positive own-pool residual in all six original units, so the pre-specified decision rule selected it for the splice.No pool-source row met the cross-task criterion.Source: §6.2 · pp. 8–9

6.3 Replacing the first elite set lowers final selected-sequence cost

  • The best realised candidate had lower cost in every task-seed unit, but the selected-cost benefit appeared only at the fixed final endpoint.The best-candidate contrast was smaller than the selected-cost contrast on Walker; several held-out units had a positive contrast after iteration one.Source: §6.3 · p. 9
  • All 12 task-seed contrasts were negative at the final endpoint, including all six held-out units.The original six selection units had task means of -0.405 on Walker and -0.912 on Cheetah; the six held-out units had means of -0.253 and -0.620, respectively.Source: §6.3 · p. 9, Figure 4 · p. 10

7 Discussion and limitations

The discussion separates current candidate ranking from the proposal that generates later queries, and uses crossed rescoring and a matched splice to examine those roles. The findings support controlled follow-up tests, but the audit and intervention have specific scope limits.

Discussion

  • Proposal audits should report mean separation relative to remaining search width and width ratios alongside elite recovery, then use selected-sequence cost to test outcome changes.These measures answer different questions; contraction alone should not be read as improved decision agreement.Source: §7 · p. 10
  • The crossed comparison can guide tests of scoring or proposal constraints, but does not select a remedy or guarantee that one will work.Cheetah differences follow scorer columns, while Walker also shows row and pairing variation.Source: §7 · p. 10

Limitations

  • The Random nonlinear splice’s direction repeats on held-out units, but the experiment does not compare interventions across all four surrogate families.It therefore establishes the consequence of the selected update, not that the residual identified the best target.Source: §7 · p. 10
  • The audit covers two tasks, six backbones per task, four states per backbone, one horizon, and one CEM configuration; other dynamics classes may yield different matrices.Identity also retains a wider context than the three compressed families, so differences cannot be assigned to feature type alone.Source: §7 · p. 10
  • The audit requires a resettable environment and execution of every candidate, while the splice uses information unavailable to an ordinary deployed planner.Lower finite-horizon selected-sequence cost does not establish higher closed-loop return.Source: §7 · p. 10

Discussion

  • A matched one-update splice tests whether an elite decision changes final selected-sequence cost, beyond what crossed rescoring alone establishes.The splice changes the first update and follows its consequence through later CEM iterations.Source: §7 · p. 10

A Follow-up rule and experiment chronology

The follow-up rule selected Random nonlinear for a one-update splice without using splice costs; the extension then tested that fixed intervention on held-out seeds. The section also describes the matrix attribution calculation, crossed patterns, and leave-one-state-out checks.

A Follow-up rule and experiment chronology

  • The original gate required replicated directions across both tasks, at least two of three seeds per task, and stability after each one-state omission; it was not a replication-probability estimate or significance test.The gate tested pre-specified distance and attribution patterns; only the second question could trigger the splice.Source: A Follow-up rule and experiment… · p. 13
  • D decreased rather than increased, no pool-source pattern qualified, and Random nonlinear’s positive own-pool residual selected its splice without using splice costs.Its iteration-three residual was positive in all six gate units and retained its sign under every one-state omission.Source: A Follow-up rule and experiment… · p. 13, Table 4 · p. 16
  • All six held-out full-state final selected-cost contrasts were negative, and the pre-specified rule passed under every omission despite one slightly positive Walker contrast.Seeds 3–5 were fixed as an audit precision extension before their audit outcomes were inspected; the splice decision rule was fixed before inspecting splice costs.Source: A Follow-up rule and experiment… · p. 13
  • The first update was chosen because traces shared the same pool there, and iteration three was the fixed four-iteration search endpoint rather than a stage selected by realised gain.Intermediate contrasts, including positive ones, were also reported.Source: A Follow-up rule and experiment… · p. 13

B Descriptive attribution calculation

  • Within each task-seed matrix, row, column, and residual sums of squares partition total variation; their shares are averaged across six matrices, not interpreted as population or causal effects.The calculation is performed within each task-seed matrix rather than on a six-seed mean matrix.Source: B Descriptive attribution calculation · pp. 13–14

C Seed-level crossed matrices

  • D and raw first-action gap decrease from the common initial pool to the final iteration, while width-normalised distance increases; ranking stays near chance on Walker and declines on Cheetah.Chance-adjusted elite precision does not improve.Source: Table 2 · p. 14

D Leave-one-state-out checks

  • Every leave-one-state-out omission preserved a negative iteration-three-minus-zero distance contrast on both tasks; Random nonlinear passed the interaction trigger while no generator passed the support trigger.The checks recomputed task-seed summaries from the remaining three states.Source: Table 3 · p. 16, Table 4 · p. 16, D Leave-one-state-out checks · p. 16

E Seed-level elite-splice results

The seed-level splice results report lower final selected-sequence costs for the environment-elite branch across the selection and held-out cohorts. The held-out omission check mostly preserves that pattern under the pre-specified rule.

E Seed-level elite-splice results

  • The environment-elite branch differs only in the first update; later iterations use Random nonlinear scores in both branches with the initial pool and base samples matched.The table describes possible follow-up tests from row, column, and diagonal patterns but says those patterns do not establish that a response will work.Source: E Seed-level elite-splice results · p. 16, Table 7 · p. 17
  • All 12 iteration-three selected-cost contrasts are negative for the Random nonlinear splice units.The 12 units comprise seeds 0–2, which selected the pairing without splice costs, and seeds 3–5, which tested the fixed intervention.Source: Table 5 · p. 17, E Seed-level elite-splice results · p. 17
  • The held-out splice rule passes under every one-state omission, although omitting bin 3 makes Walker seed 5 slightly positive.The pre-specified rule still has two negative Walker seeds and three negative Cheetah seeds in that omission.Source: Table 6 · p. 17

F Elite-set size

The elite-size check recomputes final proposal distance and chance-adjusted precision from saved pools at four elite cuts. Distance decreases as more candidates enter the refit, while adjusted precision remains low.

F Elite-set size

  • Task-mean chance-adjusted elite precision stays below 0.10 at every elite cut of 8, 16, 32, and 64, while proposal distance decreases as more candidates enter the refit.These are alternative refits of saved pools, not new CEM traces; 32 elites is the primary update setting.Source: F Elite-set size · p. 17, Figure 7 · p. 18

G Clipping diagnostics

Clipping frequency declines along generator traces, while pool-level clipping displacement and candidate-level clipping rate capture different effects. A finite-pool random-refit reference contextualizes observed proposal distances.

G Clipping diagnostics

  • Candidate-level clipping frequency and clipping-induced pool-mean displacement measure different effects, since opposing shifts can cancel.Source: G Clipping diagnostics · p. 18
  • Mean clipping displacement decreases, and most final proposal coordinates remain above the width floor in the combined six-seed audit.Source: Table 8 · p. 18
  • 0.48–0.50 at iteration zero falls below 0.10 by iteration three for every task–generator mean in the original seeds 0–2.Source: Figure 8 · p. 19

H Random-elite reference

  • The random reference fits 4,096 without-replacement subsets of 32 candidates from each saved pool to form pool-specific proposal-distance distributions.Source: H Random-elite reference · p. 19
  • 28 of 32 task–surrogate–iteration summaries place the observed refit above the random median; Cheetah Identity supplies all four exceptions.The reference scales D within a finite pool and does not measure realised cost.Source: Figure 9 · p. 19

I Reproducibility settings

The reproducibility section specifies the settings and approximate compute required for the reported audit, while noting that code and data are not public with the preprint.

I Reproducibility settings

  • Table 9 lists settings for the frozen models, surrogate fits, readouts, and audit costs; its architecture strings give layer widths from input to output.Source: Table 9 · p. 20
  • Approximately 179 GPU-hours reproduce the reported training, surrogate fitting, crossed audit, and splice work.The estimate comprises 133 hours for twelve 500k-step Dreamer backbones, 4.6 hours for exports and surrogate fits, and 41 hours for the crossed audit and twelve splice units.Source: I Reproducibility settings · p. 20
  • Compute totals sum wall-clock time across five GPU types without speed adjustment and exclude unused tasks, other evaluations, failed jobs, and exploratory runs.Source: I Reproducibility settings · p. 20
  • The code, trained checkpoints, and saved simulator states are not public with the preprint; release is planned with an archival publication.Source: I Reproducibility settings · p. 20
Loading 2610.00921v1…