Second Thoughts: Exploring and Planning through Comparative Imagination in World Models

1 INTRODUCTION

• Planning can exploit errors in an imperfect model, and a high-return proposal may not warrant intervention when the actor would perform similarly.During training, planning also changes the experience from which the actor learns, so policy effects and individual interventions matter separately.Source: §1 · pp. 1–2
• Prediction error, novelty, and model disagreement can encourage unfamiliar experience, but task-directed exploration needs to distinguish action-dependent return spread from stochastic variation under fixed behavior.Source: §1 · p. 2
• SDI measures task-return variation across imagined action sequences relative to repeated rollouts of the actor’s fixed sequence, using that reference for exploration and plan acceptance.This comparison separates action-dependent return variation from stochastic variation under fixed behavior, and requires no additional disagreement ensemble.Source: §1 · p. 2
• SDI achieves the highest observed task-average score among compared methods in each of three benchmark suites and matches or exceeds the strongest baseline mean on 15 of 18 tasks.Source: §1 · p. 3

2 RELATED WORK

• Dreamer executes a learned actor without further search, while planners search imagined futures; SDI adds a post-search comparison with repeated actor rollouts and reuses that reference for exploration.LOOP constrains search relative to an actor, whereas SDI explicitly compares the selected proposal against actor repeats.Source: §2 · p. 3
• Repeated actor-sequence rollouts estimate stochastic return variability, but cannot capture model bias shared across those rollouts.Source: §2 · p. 3
• Unlike Plan2Explore’s next-observation disagreement signal, SDI compares task-return predictions across action sequences relative to actor-repeat variability.This ties its exploration signal to the same reference used for plan acceptance.Source: §2 · p. 3

3 METHOD

• The base agent’s world model, actor, and critic support imagined return estimates, while SDI changes imagined rewards for exploration and environment actions for selective planning.Planning changes training experience; evaluation executes the learned actor’s mode without planning.Source: §3 · pp. 3–4, §3.1 · p. 4

3.1 ACTOR-REFERENCED IMAGINED RETURNS

• SDI compares sibling action-sequence returns with repeated rollouts of a fixed actor sequence, using their relative spread as a task-directed exploration signal.The bonus is normalized by its running mean, clipped to [0, 3], and added to the first imagined reward with weight β = 0.5; it anneals after the first environment success over 20k optimizer updates.Source: §3.1 · p. 4

3.2 SEARCH, FRESH RESCORING, AND PLAN ACCEPTANCE

• After MPPI search, SDI freshly re-scores the selected plan and fixed actor sequence from the same latent state, accepting the plan only when its advantage exceeds κ times actor-return variability.The re-scoring uses Rg = 4 fresh rollouts per sequence; the actor sequence stays fixed, while a rejected plan leads to a fresh actor sample.Source: §3.2 · p. 5

3.3 TRAINING SCHEDULE AND PLANNER CONFIGURATION

• The planner is disabled for the first 20k environment steps, then becomes available when return improvement falls below .05 and remains available thereafter.Planner availability is distinct from planned-action acceptance; the configuration uses N = 256 candidates, H = 8, I = 4 refinement iterations, E = 32 elites, and R = 4 rollouts per candidate.Source: §3.3 · p. 6

Benchmarks and budgets

• Meta-World v3 uses 64 × 64 observations and sparse success rewards, while Adroit tasks use dense training rewards and success-based evaluation.Meta-World runs for 700k steps except pick-out-of-hole at 500k; Adroit uses a 1M-step budget, and DM-Control comparisons use each task’s common observed depth within a nominal 500k budget.Source: §4 · pp. 6–7

Ablation study

• Table 2 reports mean evaluation success at 350k and 700k steps on hand-insert, assembly, and disassemble, with the mean equally weighting the three 700k task means.Occupancy measures planner availability, not planned-action execution; controls are R2-Dreamer and DreamerV3.Source: Table 2 · p. 7, §4 · p. 7

Baselines and evaluation

• Meta-World and Adroit endpoints are mean success over the final 100 evaluation episodes, while DM-Control averages episode return over five evaluation calls at a common task-specific scoring depth.Baseline results are from the authors’ runs; crossing means reaching a last-10 evaluation success rate of at least 0.5 at the task budget.Source: §4 · p. 7

5.1 CROSS-DOMAIN BENCHMARK PERFORMANCE

• SDI has the highest observed suite mean at both budgets in Meta-World (.43 to .84 versus R2-Dreamer’s .42 and .60), Adroit (.60 and .76 versus .33 and .40), and DM-Control (359.3 and 546.2 versus 241.8 and 521.6).Across all three domains, SDI matches or exceeds the strongest reported baseline mean on 15 of 18 tasks.Source: §5.1 · p. 7

5.2 TRAINING DYNAMICS ACROSS DOMAINS

• Training curves show task-dependent routes to endpoint performance, including early saturation, delayed transitions, and differences in plateau stability.SDI’s late gains follow earlier progress by other methods on some tasks, while its advantage appears early and is maintained on reacher-hard.Source: §5.2 · p. 8

5.3 ABLATION STUDY

• At matched .97 planner occupancy, removing the acceptance gate lowers the three-task mean from .53 to .32; reducing occupancy to .47 lowers it to .16.Always-on bonus, reset, and spread-prioritized replay reach means of .06, .08, and .13, respectively; no individual control reproduces the full method’s cross-task result.Source: §5.3 · p. 8

5.4 PAIRED-STATE ANALYSIS

• Across 1,800 restored states, gated planning recovers 4.2% of actor failures versus 2.4% for ungated planning and preserves 98.3% of actor successes versus 95.8%.The pooled net success change is +.0172 for gated planning and −.0033 for ungated planning; Figure 3 summarizes paired-state outcomes and operating points.Source: §5.4 · pp. 8–9, Figure 3 · p. 9

• SDI uses repeated actor rollouts as a shared reference for task-directed exploration and selective planning, without an additional disagreement ensemble.Planning guides training data collection, while evaluation uses the learned actor alone.Source: §6 · p. 9

• Generative AI tools assisted with methodological and experimental feedback, code scaffolding, experiment orchestration, analysis, literature search, figure drafting, and manuscript work.Source: AI USE STATEMENT · p. 10
• The tools did not generate synthetic datasets or autonomously collect experimental data, and the work makes no theoretical-proof claims.Source: AI USE STATEMENT · p. 10
• The authors ran and inspected AI-assisted code, checked analyses and numerical claims against experiment records, verified citations, manually reviewed generated text and figures, and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

• The study concerns simulated control benchmarks and does not involve human participants, animal subjects, personal or sensitive data, or physical-system deployment.Source: ETHICS STATEMENT · p. 10
• The authors warn that improved planning and autonomous decision making could be misapplied in safety-critical or harmful settings.Source: ETHICS STATEMENT · p. 10
• Benchmark results do not establish safety, robustness, or real-world suitability; world-model bias and planning error remain limitations.The authors recommend domain-specific risk assessment, human oversight, fail-safe constraints, and distribution-shift testing before physical or high-stakes transfer.Source: ETHICS STATEMENT · p. 10

• Section 3 specifies the return construction, exploration bonus, acceptance rule, planning schedule, and search settings.Source: REPRODUCIBILITY STATEMENT · p. 10
• Section 4 defines the task budgets, endpoints, and replication used for the main claims; evaluation uses the learned actor without planning.Source: REPRODUCIBILITY STATEMENT · p. 10
• The supplementary material contains only results with complete reported cohorts, and code, configurations, and instructions are provided in the anonymous repository.Source: REPRODUCIBILITY STATEMENT · p. 10

• The actor-repeat reference does not capture model bias shared by all rollouts, and its acceptance threshold κ is an empirical operating point, not a confidence level.Source: LIMITATIONS · p. 14
• Even the normalized margin’s highest decile improves on the actor in fewer than half of paired decisions, so it is not a calibrated probability.Source: LIMITATIONS · p. 14
• Search and fresh re-scoring add acting-time computation even when a proposal is rejected; paired-state analysis requires state restoration and common random numbers.The experiments use finite continuations from frozen checkpoints and do not isolate the long-term learning contribution of each accepted or rejected intervention.Source: LIMITATIONS · p. 14

A.1 SEARCH AND RESCORING

• The planner samples 256 candidate sequences over horizon 8, with 4 refinement iterations, 32 elites, and 4 independent stochastic rollouts per candidate.Source: §A.1 · p. 14
• Candidate actions are sampled around the search mean with Gaussian noise and clipped to environment action bounds.The initial mean follows the actor’s imagined mode sequence; subsequent acting steps shift the prior sequence forward, with episode boundaries resetting initialization. Scores are standardized separately for each environment.Source: §A.1 · p. 14
• After search, the plan and actor mode sequence receive 4 fresh rollouts each; the gate compares mean returns using the actor rollouts’ standard deviation.Only the first planned action is executed when accepted; otherwise, a stochastic actor action is used, while evaluation uses the actor’s mode alone. Candidate rollouts, actor repeats, and fresh re-scoring use independent model noise.Source: §A.1 · p. 14