Source-linked AI summary
Scope-WM: Scoped Computation for Efficient Visual World Models
Chunzheng Li, Zesheng Jia, Hongda Zhang, Jiaying Tang, Yuntian Wang, Siao Liu, Jin Wang
TL;DR
Dense visual world-model planning incurs high latency and peak memory because it propagates many tokens across long horizons and candidate action sequences. Scope-WM allocates computation to relevant tokens, lightweight background updates, and promising action sequences. On Push-T, its aggressive configuration achieves a 6.97× planning speedup while retaining competitive task performance.
Problem
Dense world-model planning repeatedly propagates all visual tokens over long horizons for many candidate sequences, creating high latency and peak memory that hinder real-time deployment.
Source: Introduction · p. 1Method
Scope-WM selects prediction-relevant tokens, updates remaining tokens with pooled foreground-state and change context, and reuses high-quality action sequences during later planning.
Source: Introduction · p. 2Results
On Push-T, Scope-WM reduces planning time to 14.3% and peak GPU memory to 18.1% of dense DINO-WM, achieving a 6.97× planning speedup with only a modest change in task performance.
Source: Conclusion · p. 7Takeaways & Limitations
Scope-WM offers flexible performance–efficiency operating points, with moderate sparsity preserving or improving task performance and aggressive sparsity prioritizing latency and memory reduction while retaining competitive control performance.
Source: Introduction · p. 2
Abstract
from arXiv · showhide
Visual world models enable robotic planning by predicting future observations, but dense latent-state propagation and sample-intensive trajectory optimization incur high inference latency and peak memory usage, limiting real-time deployment on resource-constrained platforms. Existing sparse world-model acceleration methods either rely on unguided token sparsification, which may discard planning-relevant information and restrict achievable sparsity, or introduce heavy auxiliary modules and cumbersome multi-stage training pipelines. In this work, we present Scope-WM, an efficient visual world model that scopes computation to prediction-relevant latent regions and promising action sequences. Scope-WM distills prediction relevance into a lightweight action-conditioned selector and applies full dynamics prediction only to a compact subset of selected tokens. It updates the remaining tokens using a compact summary of foreground states and their changes, allowing the background to perceive foreground dynamics without costly token-to-token interactions. During planning, Scope-WM preserves and reuses high-quality action sequences discovered during the initial MPC search, focusing subsequent search under reduced rollout budgets. The resulting pipeline requires only a one-off selector distillation followed by a single joint training stage for the sparse world model. On the challenging Push-T task, Scope-WM reduces peak GPU memory usage and planning time to $18.1\%$ and $14.3\%$ of those of dense DINO-WM, respectively, corresponding to a $6.97\times$ planning speedup, while maintaining competitive task performance. Further evaluations across five diverse visual planning tasks demonstrate the general applicability of Scope-WM. Code is available at https://github.com/ChunZheng2022/Scope-WM.
Introduction
Dense visual world-model planning is costly because it propagates all visual tokens across long horizons and many candidate sequences. Scope-WM allocates computation across token selection, background dynamics, and trajectory search to improve efficiency while maintaining competitive task performance.
Introduction
- Dense planning repeatedly propagates all visual tokens over long horizons for many sampled action sequences, increasing latency and peak memory.The cost grows with both visual-token count and rollout population, hindering real-time deployment.Source: Introduction · p. 1
- Sparse world-model methods can discard planning-relevant information or require specialized modules and multistage training pipelines.Efficient sparse imagination must also update remaining tokens cheaply and reduce trajectory-optimization costs.Source: Introduction · pp. 1–2
- Scope-WM scopes computation across prediction-relevant token selection, lightweight background updates, and trajectory search focused on high-quality action sequences.The framework frames efficient planning as a three-level computation-allocation problem.Source: Introduction · p. 2
- DRS distills prediction relevance into an action-conditioned selector, while FDBU updates background tokens from foreground states and their changes.The pipeline uses one-off selector distillation followed by joint training of the sparse predictor and FDBU.Source: Introduction · p. 2
- EB-CEM reuses high-quality early action sequences during subsequent low-budget MPC planning, alongside the token- and dynamics-level savings.Scope-WM reports improved planning efficiency while maintaining competitive task performance.Source: Introduction · p. 2
Related Work
Prior work reduces visual computation or improves latent planning, but sparse world-model planning requires action-dependent selection whose errors can accumulate over rollouts. Scope-WM combines prediction-relevance distillation, pooled foreground context for background updates, and persistent reuse of early trajectories during later replanning.
Visual world models and pretrained representations
- Visual world models moved from pixel prediction to compact latent dynamics and imagined trajectories, while later transformer- and diffusion-based methods advanced latent sequence modeling.DINO-WM predicts future patch-level DINOv2 features; Scope-WM retains this pretrained patch-level formulation and selectively allocates computation.Source: Related Work · p. 2
Sparse visual computation and imagination
- Vision Transformer sparsification methods target static recognition, whereas world-model planning needs state- and action-dependent token selection across multi-step rollouts.Sparse Imagination uses random patch dropping; DDP-WM uses sparse primary dynamics and foreground-conditioned background correction.Source: Related Work · p. 2
- Scope-WM uses DRS to distill prediction relevance and FDBU to replace pairwise foreground–background interactions with pooled foreground-state and foreground-delta context.After selector distillation, the sparse predictor and background updater are optimized in one joint stage.Source: Related Work · p. 2
Sampling-based planning
- CEM cost scales with candidate rollouts; EB-CEM assigns more samples to the initial search, then reuses high-quality trajectories from a persistent bank during lower-budget replanning.This sample-allocation strategy is complementary to iCEM's temporally correlated noise, warm starts, and elite reuse.Source: Related Work · p. 3
Method
Scope-WM reduces visual world-model planning costs by allocating computation across token selection, background dynamics, and trajectory search. Its method combines a distilled action-conditioned selector, lightweight foreground-conditioned background updates, and reuse of promising action sequences across MPC steps.
Method
- Scope-WM builds on DINO-WM and reduces dense rollout cost by sparsifying token dynamics, updating background tokens lightly, and reducing later search budgets.The three components are DRS, FDBU, and EB-CEM.Source: Method · p. 3, Figure 2 · p. 4
Distilled Relevance Selection
- DRS selects prediction-relevant tokens conditioned on state and action, using a lightweight token-wise MLP rather than static or visual-change-only selection.Its relevance targets come from token contributions to a trained dense teacher’s prediction loss; the selector is distilled once and fixed during sparse-model training.Source: Distilled Relevance Selection · p. 3
Foreground-Delta Background Update
- FDBU applies full dynamics prediction to selected foreground tokens and updates background tokens with a gated residual conditioned on pooled foreground states and changes.This avoids leaving the background unchanged or using costly attention-based foreground–background updates; full-grid loss jointly supervises foreground prediction and background updates.Source: Foreground-Delta Background Update · p. 4
Elite-Bank CEM
- EB-CEM leaves the planning objective and world model unchanged, reusing high-quality action sequences from the initial MPC search and reallocating rollout budget across MPC steps to refine promising plans while retaining global exploration.Source: Elite-Bank CEM · p. 5
Experiments
Scope-WM is evaluated on five visual planning tasks, with Push-T used for end-to-end efficiency and component analyses. Its sparse settings reduce computation while retaining strong performance on most tasks, and ablations examine the roles of token selection, background updates, and planning strategy.
Setup
- The evaluation covers five tasks spanning navigation, rigid-body manipulation, and deformable or multi-body dynamics, using SR or CD according to task.All tasks use CEM-based MPC; efficiency is measured by transition-model FLOPs, cumulative planning time, and peak GPU memory.Source: Setup · p. 5
Main Results
- At K = 32, Scope-WM retains 88% SR on Push-T while reducing planning time to 14.3% and peak memory to 18.1% of dense DINO-WM, a 6.97× planning speedup.At K = 98, it reaches 94% SR with planning time and peak memory at 38.6% and 38.4% of dense DINO-WM, respectively.Source: Main Results · p. 5, Table 1 · p. 6
- With K = 32, Scope-WM matches DINO-WM on PointMaze, remains within 2 and 4 percentage points on Push-T and Wall, and reaches 0.25 CD on Granular versus 0.26.With K = 98, it reaches 100% SR on PointMaze, 94% on Push-T, and 0.22 CD on Granular; Rope is weaker at 0.71 CD versus 0.41 for dense DINO-WM.Source: Main Results · pp. 5–6, Table 2 · p. 6
Ablation Study
- Under the 32-token setting, random selection lowers SR from 88% to 76%, removing FDBU lowers it to 56%, and standard CEM instead of EB-CEM lowers it to 82%.The complete Scope-WM has the highest success rate among these component variants.Source: Table 3 · p. 6, Ablation Study · p. 6
- With random selection, FDBU raises SR from 72% to 76%; with DRS, FDBU raises it from 56% to 88%, showing stronger complementarity under DRS.The comparison uses the same 32-token budget and EB-CEM planner.Source: Table 4 · p. 6, Ablation Study · p. 6
- As the full-dynamics token budget falls, Push-T planning time and peak memory decrease while SR stays between 88% and 94%; at K = 32, costs are 14.3% and 18.1% of dense levels.Figure 3 plots normalized planning time and peak memory as bars, with success rate as a line across token budgets.Source: Ablation Study · p. 6, Figure 3 · p. 7
- EB-CEM reaches 88% SR in 34.56 s, versus 60.96 s for standard CEM with 200 samples; on dense DINO-WM, EB-CEM instead lowers SR from 90% to 84% and raises time from 240.96 to 267.36 s.The reported comparison separates sparse rollout savings from the planner's sample-budget strategy.Source: Ablation Study · pp. 6–7, Table 5 · p. 7
- At 32 tokens, DRS focuses full dynamics on prediction-relevant regions, FDBU produces background errors comparable to dense updating, and EB-CEM reuses promising early trajectories.The qualitative analysis reports that FDBU uses foreground dynamics context without dense token-to-token interactions for background propagation.Source: Ablation Study · p. 7
Conclusion
Scope-WM allocates computation across token selection, latent dynamics, and trajectory optimization. On Push-T, it reports faster planning and lower peak GPU memory than dense DINO-WM.
Conclusion
- Scope-WM combines DRS token selection, FDBU background updates, and EB-CEM trajectory optimization, reducing rollout cost and subsequent replanning budgets.The stated training pipeline uses one-off selector distillation followed by joint sparse world-model training.Source: Conclusion · p. 7
- Figure 4 illustrates prediction-relevant computation, background updates, and trajectory reuse; Table 6 examines sample allocation and elite reuse on Push-T.The table caption defines early and later as initial and subsequent MPC steps; it does not provide the comparison outcome here.Source: Figure 4 · p. 7, Table 6 · p. 7
- On Push-T, Scope-WM achieves a 6.97× planning speedup, with planning time at 14.3% and peak GPU memory at 18.1% of dense DINO-WM.The paper reports only a modest change in task performance for this aggressive configuration.Source: Conclusion · p. 7
Supplementary Material
The supplement provides implementation and evaluation details, including the training protocol, shared model settings, and DRS distillation procedure.
A.1 Training Protocol and Checkpoint Selection
- Training uses at most 100 epochs, selects checkpoints by validation planning performance, and trains the sparse predictor and FDBU jointly after freezing the distilled selector.The selector is trained from gradient-times-input relevance targets exported from a trained dense DINO-WM teacher; FDBU has no separate training stage.Source: §A.1 · p. 10
A.2 Model and Optimization Settings
- The frozen DINOv2 ViT-S/14 encoder produces 196 patch tokens from 224 × 224 observations; the decoder is disabled during training and planning.The decoder is used only for qualitative visualization.Source: §A.2 · p. 10
A.3 DRS Distillation
- Across five tasks, DRS uses the same target-export procedure, selector architecture, and distillation configuration, with up to 3,200 trajectory windows per task.The selector takes globally contextualized patch tokens together with action and proprioceptive embeddings.Source: §A.3 · p. 10, Table S2 · p. 10
A.4 Task-Specific Training Settings
Task-specific settings define the data splits, observation histories, and goal construction; the planning configuration specifies CEM variants and task objectives.
A.4 Task-Specific Training Settings
- PointMaze, Wall, Rope, and Granular use 90/10 trajectory splits with seed 42, while Push-T uses its predefined training and validation directories.PointMaze and Push-T use three history frames; Wall, Rope, and Granular use one, with task-specific frameskip settings.Source: §A.4 · p. 10
A.5 Planning Configuration
- Standard CEM uses 100 candidates, 10 elites, and 10 iterations; EB-CEM starts with 300 candidates and 30 elites, then uses 100 candidates and 10 elites.EB-CEM saves 30 candidates after the initial search and samples 70% of later populations around banked elites, with the remainder from the current CEM Gaussian.Source: §A.5 · p. 10, Table S4 · p. 11
- Planning evaluates the final predicted latent state; PointMaze uses only visual-latent cost, while Wall, Push-T, Rope, and Granular use equal visual and proprioceptive weights.Goals are random environment states for PointMaze, Wall, Rope, and Granular, while Push-T uses a five-step validation-data segment.Source: §A.5 · p. 11, Table S3 · p. 11
B.1 Datasets and Goal Construction
The supplement specifies task datasets and goals, evaluation instances and criteria, efficiency measurements, and the hardware used for experiments.
B.1 Datasets and Goal Construction
- PointMaze and Wall use procedurally sampled initial–goal pairs, while Push-T goals replay recorded actions over a five-step validation segment; Rope and Granular goals apply geometric transformations.Rope goals use random in-plane rotation and translation; Granular goals use random scale and translation.Source: §B.1 · pp. 11–12
B.2 Random Seeds and Number of Runs
- PointMaze, Wall, and Push-T use 50 fixed evaluation instances each, while Rope and Granular use 10; reported Scope-WM results come from one selected checkpoint.Results are not averaged across independently trained models.Source: §B.2 · p. 12
B.3 Metrics and Efficiency Measurement
- Success thresholds are task-specific for PointMaze, Wall, and Push-T; Rope and Granular report symmetric Chamfer Distance between final and goal particle sets in three dimensions.Push-T requires both position distance below 20 and orientation error below π/9.Source: §B.3 · p. 12
- Efficiency is measured with transition-model FLOPs, cumulative CEM sub-planner time, and peak CUDA memory; the local Push-T comparison uses the same hardware and evaluation instances.Planning time excludes environment stepping, metric computation, decoding, and visualization.Source: §B.3 · p. 12
C Hardware and Software Environment
- PointMaze, Wall, and Push-T experiments use eight V100 GPUs for training and one 32-GB V100 for planning; Rope and Granular use one RTX 2080 Ti.The deformable-task GPU has 22,528 MiB of memory.Source: C Hardware and Software Environment · p. 12
D.1 Efficiency Across Rigid-Body Tasks
Scope-WM reduces planning time across PointMaze, Wall, and Push-T, while its peak-memory benefit varies with rollout-activation and search-population costs.
D.1 Efficiency Across Rigid-Body Tasks
- Efficiency measurements use the same initial–goal pair and a forced full-length planning run, with planning time summed across CEM sub-planner calls at all MPC steps.Task success rates come from the complete evaluation sets reported in the main paper.Source: §D.1 · p. 12
- 5.84×, 3.03×, and 6.97× planning-time speedups on PointMaze, Wall, and Push-T, respectively, show consistent reductions across the three tasks.Peak-memory savings are substantial on PointMaze and Push-T, but Wall has a modest absolute memory increase.Source: §D.1 · p. 12
- Memory savings are largest when dense rollout activations dominate; on Wall, the larger initial EB-CEM population outweighs smaller token-sparsity savings.The dense baseline already has a small memory footprint on Wall.Source: §D.1 · p. 12
E Additional Qualitative Results
On Push-T, decoded Scope-WM latent rollouts follow the overall motion of the agent and T-shaped object under the planned actions; decoding is for visualization only.
E Additional Qualitative Results
- The decoded Scope-WM rollout follows the overall motion of the agent and T-shaped object across successive steps under the same planned action sequence.The decoder is used only for visualization and is not loaded during planning.Source: E Additional Qualitative Results · p. 12
- Table S5 covers end-to-end efficiency on three rigid-body planning tasks, with time and peak memory measured in one fixed full-length run and success rate over 50 instances.Memory ratios are normalized per task, and peak memory includes the larger initial EB-CEM population.Source: Table S5 · p. 13
- Figure S1 places environment executions above decoded latent rollouts, with steps progressing left to right and the goal in the highlighted final column.The lower row uses the dense DINO-WM decoder to visualize Scope-WM latent states.Source: Figure S1 · p. 13
F Additional Token-Budget Analysis
Changing the full-dynamics token budget after training substantially degrades Push-T planning performance, so Fast and Faster use separate budget-specific training configurations.
F Additional Token-Budget Analysis
- Changing K after training substantially degrades planning performance in both directions for the Fast and Faster checkpoints.Table S6 evaluates each checkpoint under both token budgets using the same 50 fixed instances.Source: F Additional Token-Budget Analysis · p. 13, Table S6 · p. 13
- Fast and Faster are trained separately for their respective planning budgets rather than changing K only at planning time.The models use K = 98 and K = 32, respectively.Source: F Additional Token-Budget Analysis · p. 13
- K is part of the learned sparse-dynamics configuration because changing it alters the foreground–background partition handled by the sparse predictor and FDBU.The altered computation pattern mismatches the one observed during training.Source: F Additional Token-Budget Analysis · p. 13