PaperlayerView exact revision on arXiv

Source-linked AI summary

FutureWorlds: Learning Robotic World Models from Alternative Futures

Hao Wu, Shengju Qian, Weiyan Wang, Fan Xu, Fan Zhang, Yuanpeng He, Qingsong Wen, Yuxuan Liang

arXiv:2610.01019v1cs.CVcs.RO

TL;DR

Robotic world models need to learn from alternative futures, but similar candidates limit useful quality comparisons and diverging trajectories require maintained histories. FutureWorlds combines diverse search, candidate-specific bounded memory, and MemSPO’s reward-based updates. It reduces 32-frame LPIPS across three datasets, while beam search incurs latency and closed-loop planning remains future work.

  • Problem

    Similar candidate futures limit informative quality comparisons, while diverging trajectories require individual histories to be maintained for history-consistent learning.

    Source: §1 · pp. 1–2

  • Method

    FutureWorlds combines diverse beam search, candidate-specific bounded memory, and MemSPO, which converts video trajectory rewards into group-relative advantages for history-matched world-model updates.

    Source: §1 · p. 2

  • Results

    LPIPS decreases by 14.78% on RT-1, 20.84% on BridgeV2, and 9.12% on RoboCasa for 32-frame predictions versus each dataset’s strongest baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Three-dataset evaluations support gains in visual quality, object consistency, and motion prediction, with improved generation after 200 MemSPO updates and prediction beyond the training horizon.

    Source: §1 · p. 2, §5 · p. 9

    Autoregressive beam search incurs latency, and validating benefits in closed-loop planning and robot control remains future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Robotic world models predict action-conditioned future scenes, providing a foundation for understanding action outcomes. However, turning alternative predictions into useful learning signals remains challenging: similar candidates limit informative quality comparisons, while diverging trajectories require persistent maintenance of their individual histories. We introduce FutureWorlds, a framework that unifies candidate construction, history maintenance, and learning from relative quality. Built on a multimodal discrete autoregressive model, FutureWorlds uses diverse beam search during reinforcement learning to construct candidate futures that balance confidence and diversity. Candidate-specific bounded memory preserves scene states and ensures that generation and policy scoring use matching histories. We further propose MemSPO (Memory-Conditioned Search-Guided Policy Optimization), which converts video trajectory rewards into group-relative advantages to optimize the world model. On RT-1, BridgeV2, and RoboCasa, FutureWorlds reduces LPIPS for 32-frame predictions by 14.78%, 20.84%, and 9.12%, respectively, relative to the strongest baseline on each dataset. Under fixed evaluation configurations, only 200 MemSPO updates further improve generation quality and support continued prediction beyond the training horizon. Memory ablations, decoding sensitivity analysis, and optical-flow evaluation show that these gains extend beyond visual quality to more accurate motion prediction and more consistent object states. Project page and code: https://github.com/Alexander-wu/FutureWorlds.

1 INTRODUCTION

Robotic world models can generate alternative action-conditioned futures, but useful learning depends on informative candidate differences and comparisons that preserve each future’s history. FutureWorlds combines diverse search, candidate-specific memory, and reward-based learning, improving prediction quality across three datasets.

Learning from Alternative Futures

  • Alternative futures vary in robot motion, object states, and prediction quality, offering a basis for learning by comparing predictions with recorded observations.The motivation draws on action-conditioned world models used for action-outcome prediction, visual planning, behavior learning, and policy evaluation.Source: §1 · p. 1
  • Autoregressive models trained on recorded histories face a mismatch with self-generated inference histories, and token-level likelihood does not directly optimize complete generated sequences.More candidates alone do not guarantee useful comparisons; informative differences and history-consistent scoring are also needed.Source: §1 · p. 1
  • Candidate construction must balance model confidence and distinct predictions, while bounded memory preserves each diverging branch’s history for sustained generation and matching policy scoring.Independent sampling does not coordinate candidate properties, and conventional beam search can concentrate on similar sequences.Source: §1 · p. 2
  • FutureWorlds uses diverse beam search, bounded candidate-specific memory, and MemSPO to learn from relative trajectory rewards while matching histories for generation and policy scoring.MemSPO converts video trajectory rewards into group-relative advantages for world-model updates.Source: §1 · p. 2

Experimental Evaluation

  • LPIPS falls by 14.78% on RT-1, 20.84% on BridgeV2, and 9.12% on RoboCasa versus the strongest baseline on each dataset for 32-frame predictions.The evaluation also examines generation quality, long-horizon prediction, memory, decoding robustness, and motion consistency.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans action-conditioned world models, long-horizon memory, and search-guided reinforcement learning. FutureWorlds uses discrete autoregression to compare futures under shared conditions, preserve candidate-specific histories, and update video predictions from relative quality.

Robotic World Models

Memory for Long-Horizon World Modeling

  • Prior memory methods retain segment history, attention sinks, short- and long-term context, compressed frames, geometry-grounded memory, or retrieved observations; FutureWorlds keeps bounded, candidate-specific histories and matching scoring contexts.This preserves distinct conditioning as alternative futures diverge.Source: §2 · p. 3

Search-Guided Reinforcement Learning for World Models

  • Sequence-metric policy gradients, decoded-prediction rewards, inverse-dynamics rewards, and search-based methods inform prior work; FutureWorlds instead searches futures under fixed actions and updates the video predictor from their relative quality.MemSPO combines history-consistent scoring with a search-guided surrogate based on GRPO group-relative advantages and clipped updates.Source: §2 · p. 3

3 METHOD

FutureWorlds initializes a multimodal autoregressive model, then constructs alternative futures, maintains bounded histories for each candidate, and optimizes the model from relative trajectory quality. MemSPO uses diverse search and history-matched policy scoring to turn video rewards into updates.

3 METHOD

  • The method combines supervised initialization with post-training search, separate candidate histories, and learning from relative trajectory quality; generation and policy scoring use matching histories.True future frames enter reward computation only during post-training.Source: §3 · p. 3, Figure 2 · p. 4

3.1 PROBLEM FORMULATION

  • The world model predicts discrete future-frame tokens autoregressively from initial visual conditions, preceding tokens and actions, and optional language; the factorization supports candidate expansion and trajectory scoring.The visual decoder maps predicted tokens back to video.Source: §3.1 · pp. 3–4

3.2 MULTIMODAL AUTOREGRESSIVE WORLD MODEL

  • The multimodal model encodes the initial scene with more context tokens and later changes with fewer dynamic tokens, using discrete representations and a visual decoder to reconstruct frames.Its tokenizer remains frozen during world-model supervised fine-tuning and reinforcement learning.Source: §3.2 · p. 4

3.3 CANDIDATE-SPECIFIC PERSISTENT MEMORY

  • Each candidate’s bounded memory retains the initial scene, a state anchor, and recent predictions; branches inherit and reorder their own memories, while policy scoring uses the same retained history as generation.Older rolling history is dropped when the window fills; true future frames are reserved for reward computation.Source: §3.3 · p. 5

3.4 MEMORY-CONDITIONED SEARCH-GUIDED POLICY OPTIMIZATION

  • MemSPO combines diverse search, trajectory rewards, group-relative advantages, and a clipped policy surrogate to update the world model from candidate quality differences.The method uses fresh candidates for each update, while beam selection changes their distribution, so the search-guided surrogate has no unbiased on-policy gradient guarantee.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

The experiments examine prediction quality, post-training, motion and dynamics, and memory maintenance. Across RT-1, BridgeV2, and RoboCasa, FutureWorlds improves 32-frame visual metrics, while further evaluations assess post-training gains, motion, resource use, and retained history.

4.1 EXPERIMENTAL SETUP

  • The evaluation compares 32-frame predictions on 128 held-out trajectories across RT-1, BridgeV2, and RoboCasa using PSNR, SSIM, and LPIPS.The setup includes video-generation, action-conditioned world-model, RL post-training, and adapted video-foundation-model baselines; training starts from SFT and uses diverse beam search.Source: §4.1 · p. 6

4.2 VIDEO GENERATION QUALITY (RQ1)

  • LPIPS falls by 14.78% on RT-1, 20.84% on BridgeV2, and 9.12% on RoboCasa versus the strongest baseline per dataset; FutureWorlds leads all three metrics on every dataset.The comparison evaluates 32-frame predictions on 128 trajectories per dataset.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 POST-TRAINING EFFECTIVENESS AND EFFICIENCY (RQ2)

  • After 200 RL updates, MemSPO leads all three metrics at 10, 20, and 32 frames on 128 matched trajectories per dataset, with decoding and memory fixed.At 32 frames, LPIPS drops by 9.98%, 1.58%, and 2.17% versus GRPO on RT-1, BridgeV2, and RoboCasa, respectively.Source: §4.3 · p. 8, Table 2 · p. 8
  • Across five decoding settings, MemSPO gains persist; on 32 monitoring clips, 200 updates reduce LPIPS by 4.02% on RT-1 and 3.45% on BridgeV2 versus SFT.With checkpoints and beam width fixed, reductions remain 3.03–4.32% on RT-1 and 2.60–3.84% on BridgeV2 across tested temperature and top-p settings.Source: §4.3 · p. 8, Figure 4 · p. 8
  • FutureWorlds leads seven models in BridgeV2 LPIPS with 0.257B inference parameters and 1.89 GiB peak memory, while beam decoding takes 26.76 seconds per rollout.The resource measurements use three 32-frame rollouts at batch size one with native configurations and cached language features.Source: Figure 5 · p. 8, §4.3 · p. 9

4.4 MOTION AND DYNAMIC PREDICTION (RQ3)

  • Flow error falls by 12.40% on RT-1, 4.57% on BridgeV2, and 16.68% on RoboCasa versus SFT; flow-error and magnitude-gap gains support closer motion agreement, not merely larger movements.The evaluation uses 128 trajectories per dataset and reports that FutureWorlds leads all three motion diagnostics.Source: §4.4 · p. 9, Figure 6 · p. 9

4.5 MEMORY MAINTENANCE (RQ4)

  • With weights and decoding fixed, full memory leads PSNR, SSIM, and MSE at 32 frames across datasets, reducing MSE by 40.83% on RT-1, 57.69% on BridgeV2, and 29.22% on RoboCasa versus no historical memory.Recent history supplies most gains, while the initial anchor adds smaller benefits; no variant is retrained.Source: §4.5 · p. 9, Figure 7 · p. 9

5 CONCLUSION AND FUTURE WORK

FutureWorlds learns from alternative futures through coordinated candidate construction, history maintenance, and quality comparison. The conclusion reports gains in visual quality, object consistency, and motion prediction, while identifying decoding speed and closed-loop validation as future work.

5 CONCLUSION AND FUTURE WORK

  • Across three datasets, evaluations support gains in visual quality, object consistency, and motion prediction; BridgeV2 comparisons show better quality with fewer parameters and lower measured memory use, but autoregressive beam search incurs latency.Future work focuses on faster decoding and validating benefits in closed-loop planning and robot control.Source: §5 · p. 9

AI USE STATEMENT

The authors report using generative AI tools across manuscript preparation, coding, experimental protocols, result inspection, mathematical review, and figure and table preparation.

AI USE STATEMENT

  • The authors state that reported numerical results came from experimental outputs and that they take responsibility for verifying AI-assisted work and the final artifacts.The listed assistance also includes drafting, translation, editing, code implementation and debugging, protocol refinement, and preparation of figures and tables.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement points to the paper’s appendices and project link for implementation details, evaluation settings, and reproduction instructions.

REPRODUCIBILITY STATEMENT

  • Appendix B documents training and evaluation protocols, Appendices E.3, F, and G give comparison settings and results, and the project link provides code and reproduction instructions.The authors state that they plan to release trained checkpoints.Source: REPRODUCIBILITY STATEMENT · p. 10

A THEORETICAL ANALYSIS

The analysis characterizes MemSPO’s reward update for fixed candidate histories through pairwise reward and score-gradient differences. Candidate collapse constrains the reward-driven update, while greater dispersion alone does not ensure a larger or better-directed one.

A.1 PAIRWISE STRUCTURE OF THE REWARD GRADIENT

  • Each candidate pair contributes through differences in reward and score gradients, and either vanishing difference removes its contribution.The identity requires no independence assumption and applies to beam-search candidates; history matching gives the scores their intended interpretation, not a separate performance guarantee.Source: §A.1 · p. 15

A.2 A BOUND UNDER CANDIDATE COLLAPSE

  • At fixed reward scale, candidate score-gradient collapse limits the reward-driven update, but greater dispersion alone guarantees neither a larger nor better-directed update.Pairwise contributions can cancel, and connecting visual diversity to score-gradient dispersion requires empirical analysis.Source: §A.2 · pp. 15–16

B.1 PRETRAINED INITIALIZATION, SFT, AND RL

  • RT-1 reuses its pretrained tokenizer, while BridgeV2 and RoboCasa use adapted tokenizers that remain frozen during world-model SFT and RL.SFT uses recorded histories; MemSPO then uses ten-frame predicted rollouts and one update per fresh candidate set.Source: §B.1 · p. 16

B.2 TRAINING ALGORITHM

  • The training algorithm preserves branch ancestry and bounded histories, then scores candidates under their saved generation-time conditions before one optimizer step.Search penalties select candidates, while unmodified policy probabilities score them.Source: §B.2 · p. 17

B.3 FLEXIBLE REWARD DESIGN

  • MemSPO’s reward module can use whole-frame reconstruction or add local-appearance and frame-difference terms while retaining the same group-relative update.The two reward designs emphasize complementary prediction qualities; ground-truth futures enter reward computation only.Source: §B.3 · p. 18

B.4 BASELINE ADAPTATION AND EVALUATION

The evaluation fixes matched prediction settings and examines learning curves, decoding sensitivity, and motion diagnostics. MemSPO’s LPIPS gains persist across the tested decoding settings on RT-1 and BridgeV2.

B.4 BASELINE ADAPTATION AND EVALUATION

  • The main evaluation uses 32-frame predictions on 128 matched trajectories per dataset, with recorded actions conditioning generation and no ground-truth future refreshes.FutureWorlds uses beam width four and persistent memory; its main-table checkpoints use 500, 200, and 400 MemSPO updates for RT-1, BridgeV2, and RoboCasa.Source: §B.4 · p. 19

B.5 LEARNING-CURVE AND QUALITATIVE EVALUATION

  • Learning curves use 32 fixed monitoring clips per dataset, while selected frame-32 examples illustrate prediction quality rather than measured robot task success.The monitoring configurations share inputs and decoding settings; update counts do not imply equal compute.Source: §B.5 · p. 20

B.6 DECODING SENSITIVITY

  • 3.03–4.32% on RT-1 and 2.60–3.84% on BridgeV2: MemSPO reduces mean LPIPS under all five tested decoding settings.Each setting is compared with its own matched SFT reference, so the gains do not depend solely on the default decoding parameters.Source: §B.6 · p. 20, Table 3 · p. 20

B.7 MOTION AND DYNAMIC EVALUATION

  • Motion diagnostics compare predicted and reference flow, flow magnitude, and frame features over 32-frame rollouts; they supplement rather than replace pixel metrics.Lower Flow and Gap and higher DINO indicate closer agreement with the recorded future.Source: §B.7 · p. 20

C SHORT-HORIZON EVALUATION

This block covers short-horizon evaluation, policy feedback, selected visual and temporal comparisons, and candidate diversity. The examples illustrate prediction behavior, while the candidate diagnostics distinguish broader quality variation from improved average quality.

C SHORT-HORIZON EVALUATION

  • The ten-frame comparison uses the same model checkpoints and 128 trajectories per reported dataset, with scores averaged over frames and trajectories.Baseline scores use the first ten frames of their 32-frame rollouts; FutureWorlds uses its existing ten-frame evaluations.Source: C SHORT-HORIZON EVALUATION · p. 21, Table 4 · p. 21

D POLICY FEEDBACK IN A LEARNED WORLD MODEL

E ADDITIONAL QUALITATIVE COMPARISONS

E.1 TEMPORAL COMPARISONS

  • Aligned five-time-point sequences expose differences in motion timing, object displacement, and appearance; selected RT-1 and RoboCasa examples show closer displacement or arm-configuration tracking.These selected examples illustrate prediction behavior, not measured task success.Source: §E.1 · p. 25, Figure 12 · p. 25, Figure 14 · p. 27

E.2 CANDIDATE DIVERSITY AND RELATIVE LEARNING SIGNALS

  • Across 16 initial-batch inputs, diverse search yields 30.32% mean pairwise token disagreement and an 8.66×10−3 mean within-group reward range, versus 0.28% and 6.44×10−5 for ordinary search.Diverse search also has lower mean reward (−0.14353 versus −0.14041), so these diagnostics indicate reduced redundancy and broader quality variation, not an immediate increase in average quality.Source: §E.2 · p. 28, Figure 15 · p. 28

E.3 CANDIDATE CONSTRUCTION AND FINAL POST-TRAINING QUALITY

With matched post-training and evaluation settings, MemSPO improves PSNR and LPIPS over ordinary-beam post-training at every tested horizon on all three datasets. The comparison supports the usefulness of less redundant candidate construction but does not isolate the diversity coefficient.

E.3 CANDIDATE CONSTRUCTION AND FINAL POST-TRAINING QUALITY

  • The matched comparison uses 200 post-training updates and the same base reward, optimizer settings, reference policy, visual tokenizer, memory, and history-matched scoring.Ordinary beam search returns four complete trajectories from eight global paths; MemSPO uses four groups of two paths and returns one trajectory per group.Source: §E.3 · p. 29
  • The BridgeV2 values in this table use internally matched GPU evaluations because CPU/GPU numerical differences can alter beam decisions and subsequent rollouts.The comparison does not establish statistical significance or isolate the diversity coefficient from the different beam allocation.Source: §E.3 · p. 29
  • 7.37%, 0.20%, and 1.51%: at 32 frames, MemSPO reduces LPIPS versus ordinary-beam post-training on RT-1, BridgeV2, and RoboCasa, respectively.MemSPO improves PSNR and LPIPS at every tested horizon on all three datasets, though ordinary beam has lower BridgeV2 MAE at 32 frames (0.042731 versus 0.042887).Source: §E.3 · p. 29, Table 5 · p. 29

F MEMORY ABLATION: PROTOCOL AND FULL RESULTS

The memory ablation finds substantial gains from recent history and smaller, horizon- and metric-dependent gains from a persistent anchor. The resource comparison favors FutureWorlds on model size and allocated memory, but not inference speed.

F MEMORY ABLATION: PROTOCOL AND FULL RESULTS

  • The memory comparison fixes the 200-update checkpoint, beam width of four, 128 held-out trajectories per dataset, and 32-frame generation; variants differ only in retained state history.No ground-truth future frames refresh rollouts, and the 10- and 20-frame results use prefixes of the same generated sequence.Source: F MEMORY ABLATION: PROTOCOL AND FULL… · p. 30
  • The full model retains an initial state anchor plus six recent state frames, while the no-anchor variant keeps six recent frames and the no-history variant only the latest frame.All variants retain the same fixed initial scene encoding, and state frames retain their corresponding action tokens.Source: F MEMORY ABLATION: PROTOCOL AND FULL… · p. 30
  • At 32 frames, the full model reduces MSE by 40.83%, 57.69%, and 29.22% versus the history-removed variant on RT-1, BridgeV2, and RoboCasa, respectively.The full model also achieves the best PSNR and SSIM at this horizon; the reported means do not establish statistical significance.Source: Table 6 · p. 30, F MEMORY ABLATION: PROTOCOL AND FULL… · p. 30
  • Recent history yields the largest improvement over removing historical memory on every dataset; the initial anchor adds smaller, horizon- and metric-dependent gains.On RoboCasa, removing the anchor slightly lowers MSE at 10 and 20 frames, while the full model retains higher PSNR and SSIM.Source: F MEMORY ABLATION: PROTOCOL AND FULL… · p. 30

G INFERENCE RESOURCE MEASUREMENTS

  • Resource measurements use one H20 per model, batch size one, three fixed BridgeV2 cases, and 32 output frames; plotted quality uses all 128 trajectories.Latency follows one full warmup and three timed runs; peak memory is the maximum across those runs.Source: G INFERENCE RESOURCE MEASUREMENTS · p. 31, Table 7 · p. 31
  • Inference parameter totals include the prediction backbone, adapters, and visual codec, while excluding offline language encoders and unused task, reward, or critic modules.FutureWorlds totals 256,549,712 inference parameters, with 127,101,696 in the backbone and 129,448,016 in the visual codec.Source: G INFERENCE RESOURCE MEASUREMENTS · p. 31

G.1 NATIVE PRECISION, DECODING, AND CACHES

  • Native precision, internal resolution, decoding, and cache choices differ across models, so the benchmark preserves each model’s evaluated pipeline rather than enforcing shared settings.FutureWorlds reuses its key–value cache within each frame’s beam search, while histories and predicted-state caches are rebuilt for each rollout.Source: §G.1 · p. 32, Table 8 · p. 32
  • FutureWorlds has the fewest inference parameters and lowest observed allocated memory among seven implementations, but is slower than iVideoGPT and several diffusion pipelines.The comparison supports a quality–model-size–memory tradeoff, not universal inference-speed superiority; it uses three timed cases and native settings, and reports no FLOPs.Source: §G.1 · p. 32

Related papers

Loading 2610.01019v1…