Source-linked AI summary
World Model for Robot Learning: A Comprehensive Survey
Bohan Hou, Gen Li, Jindou Jia, Tuo An, Xinying Guo, Sicong Leng, Haoran Geng, Yanjie Ze, Tatsuya Harada, Philip Torr, Oier Mees, Marc Pollefeys, Zhuang Liu, Jiajun Wu, Pieter Abbeel, Jitendra Malik, Yilun Du, Jianfei Yang
TL;DR
World-model research for robot learning is fragmented across architectures, roles, and embodied domains. This survey organizes the field around predictive models’ relationships to policies, simulators, video generation, benchmarks, and applications, finding practical utility across multiple design paradigms while highlighting benchmark sensitivity and persistent deployment challenges.
Problem
World-model literature for robot learning remains fragmented across architectures, functional roles, and embodied application domains, limiting a unified understanding of the field.
Method
The survey develops a robot-learning-centered taxonomy of predictive world models by examining their roles in policy learning, planning, simulation, evaluation, data generation, and video modeling.
Results
Embodied world models show practical utility across manipulation benchmarks and multiple architectural paradigms, but performance varies with benchmark, embodiment, action space, task composition, and protocol.
Takeaways & Limitations
World modeling is not tied to one implementation, while long-horizon consistency and function-oriented evaluation remain central considerations for embodied control.
Takeaways & Limitations
Reliable deployment remains limited by action-dependent dynamics, training and inference efficiency, non-visual sensory integration, and the lack of standardized functional evaluation.
Abstract
from arXiv · showhide
World models, which are predictive representations of how environments evolve under actions, have become a central component of robot learning. They support policy learning, planning, simulation, evaluation, data generation, and have advanced rapidly with the rise of foundation models and large-scale video generation. However, the literature remains fragmented across architectures, functional roles, and embodied application domains. To address this gap, we present a comprehensive review of world models from a robot-learning perspective. We examine how world models are coupled with robot policies, how they serve as learned simulators for reinforcement learning and evaluation, and how robotic video world models have progressed from imagination-based generation to controllable, structured, and foundation-scale formulations. We further connect these ideas to navigation and autonomous driving, and summarize representative datasets, benchmarks, and evaluation protocols. Overall, this survey systematically reviews the rapidly growing literature on world models for robot learning, clarifies key paradigms and applications, and highlights major challenges and future directions for predictive modeling in embodied agents. To facilitate continued access to newly emerging works, benchmarks, and resources, we will maintain and regularly update the accompanying GitHub repository alongside this survey.
1 Introduction
This survey adopts a robot-learning-centered view of world models as predictive models supporting policy learning, planning, simulation, evaluation, and data generation. It organizes the field by architectural paradigms, functional roles, embodied domains, and resources while tracing tighter integration between prediction and action generation.
- Background: World models predict future states or state sequences under current states and actions, linking predictive modeling to intervention, planning, and control.The survey traces this concept from cognitive science, control theory, and classical robot planning to robot learning.
- Robot-learning perspective: The survey defines actionable world models by their utility for robotic action, covering policy learning, planning, simulation, evaluation, and data generation.It distinguishes robotic world models from generic perceptual predictors because predictive quality matters insofar as it supports action.
- Simulator roles: World models are also increasingly used as learned simulators for validation, post-training, reinforcement learning, and evaluation.Their simulator roles expand beyond rollout-based validation toward broader learned-environment uses.
- Policy integration: Recent robot-policy research progresses from decoupled video-generation-plus-IDM pipelines toward single-backbone, MoE/MoT, unified VLA, and latent world-modeling designs.This progression reflects tighter integration between predictive modeling and action generation.
- Survey contributions: The survey contributes a fine-grained taxonomy of world-model architectures and functional roles, alongside coverage of VLA policies, video generation, benchmarks, datasets, and open challenges.Its organization spans policy coupling, simulator applications, robotic video world models, navigation, autonomous driving, and evaluation resources.
2 Background
The survey defines a world model as an embodiment-centered predictive model of agent-environment dynamics under robot-relevant actions, emphasizing actionable futures for decision making. It distinguishes predominantly visual world models from robot policies, which map observations and instructions to future action trajectories.
- World models: A world model predicts how an embodied system evolves under actions, typically by forecasting next or future states from current states or observations.The definition is intentionally agnostic to the state space, provided predicted futures support downstream embodied decision making.
- Action conditioning: Low-level motor commands and high-level language instructions are both treated as actions that condition predictions of future embodied states or observations.Language actions specify what the future should be realized, while physical actions are concrete commands executed by the agent.
- World models: World models are defined functionally by supporting control, planning, simulation, evaluation, or data generation rather than merely producing plausible future predictions.Their defining property is action-sensitive prediction that supports embodied decision making.
- Visual world models: Because scalable embodied state representations commonly use visual observation streams, the survey focuses primarily on visual world models instantiated as future video generation.Video-space models preserve richer spatial, temporal, and interaction details but must also maintain temporal coherence, action consistency, physical plausibility, and long-horizon stability.
- Robot policies: Robot policies map current observations, optionally with language instructions, to future action trajectories, with imitation learning from expert demonstrations as the survey’s focus.Contemporary policies divide into specialized visuomotor policies and generalist Vision-Language-Action models.
3 World Model for Policy
World-model-based policies increasingly introduce predictive structure into action generation, progressing from decoupled future prediction and action inference toward unified and latent-space formulations. These approaches use anticipated observations, representations, or compact predictive variables to make control less reactive and reduce ambiguity in action generation.
- Overview: The field progresses from decoupled predict-then-act pipelines toward unified and internalized predictive control, without establishing any predictive substrate as universally superior.The survey organizes methods architecturally according to how predictive generation interacts with action production.
- Overview: World models provide predictive representations of future observations or dynamics that condition action generation beyond the current observation.This predictive structure can provide useful inductive bias when robot action data are limited and large-scale predictive pretraining is available.
- Decoupled designs: Decoupled methods first generate task-conditioned future observations or latent trajectories with a world model, then infer executable actions using the current observation and predicted future.UniPi exemplifies this paradigm by generating future video trajectories and using an inverse dynamics model to derive actions from adjacent frames.
- Unified architectures: Unified architectures embed world modeling directly into policies, combining predictive foresight with modality-specialized action mechanisms or jointly trained internal world models.Video diffusion can provide foresight while MoE/MoT mechanisms translate predictions into actions, and unified VLA models may use image, latent, semantic, or expert-based foresight.
- Latent-space formulations: Latent-space approaches internalize future dynamics without explicit image or video decoding by aligning predictive representations or learning compact control conditions.This non-pixel route treats world modeling as latent or abstract prediction when those variables offer a direct interface to action.
4 World Model as Simulator
World models can function as learned environments for reinforcement-based policy improvement, replacing costly physical interaction with imagined rollouts. They also support candidate-action selection and offline policy evaluation by predicting and scoring imagined consequences.
- Simulator paradigm: World models support two simulator roles: generating imagined transitions for reinforcement learning and scoring candidate-action consequences for decision-time validation.The reinforcement-learning role replaces physical interaction, while the validation role supports action selection through imagined outcomes.
- Reinforcement training: In reinforcement training, VLA policies roll out trajectories in learned environments, receive optional rewards and termination signals, and improve by maximizing expected return.This paradigm aims to replace expensive physical interaction with reinforcement learning inside a learned simulator.
- Reinforcement training: Recent methods broaden compatibility with VLA architectures through pixel-space imagination, GRPO-style optimization, flow-based action heads, compositional dynamics, and scaling to pretrained VLA adaptation.Representative methods include WMPO, ProphRL, RISE, and GigaBrain-0.5M∗.
- Reinforcement training: A second paradigm refines the simulator itself, using policy rollouts to improve world-model fidelity, action-following precision, and rollout reliability for subsequent policy updates.This shifts emphasis from reinforcement learning in a world model toward reinforcement learning with a continuously improved world model.
- Decision-time validation: For evaluation and decision making, world models predict candidate futures to rank and refine actions, compare policies, detect likely failures, and enable offline assessment and safety probing.GPC uses predictive look-ahead for candidate ranking, while WorldEval and Veo-based evaluation study scalable policy assessment and OOD testing.
- Decision-time validation: Latent-space models such as V-JEPA 2 and V-JEPA 2.1 perform prediction and planning in embedding space, with V-JEPA 2.1 supporting zero-shot robot planning with image goals.These methods avoid explicit pixel-space future generation for candidate-action ranking.
5 World Model for Robotic Video Generation
Robotic video world models predict visually plausible futures that must also remain temporally coherent, action-consistent, and actionable. The field is progressing from imagination-based supervision toward controllable, structurally grounded, and foundation-scale interactive world modeling.
- Core requirements: Robotic video generation predicts robot–environment evolution from observations, task specifications, and candidate actions, requiring visual plausibility, temporal coherence, and action consistency.Task or language conditioning is treated as a high-level action specifying the future before low-level control is grounded.
- Imagination for policy learning: Imagination-based methods synthesize future executions as supervision for policy learning, expanding training beyond the narrow support of collected demonstrations.Generated futures can support synthetic data generation, high-level demonstration surrogates, long-horizon visual planning, and executable action recovery.
- Imagination for policy learning: Digital-twin approaches connect generated videos to physical execution by reconstructing manipulable or physical world models and grounding motion into robot actions.DreMa combines Gaussian Splatting with physics simulation, while PhysWorld uses object-centric residual reinforcement learning to connect predicted motion with robot actions.
- Explicit controllability: Controllable world models increasingly evaluate fidelity through action faithfulness, controllable interaction, and usefulness for closed-loop decision making rather than realism alone.IRASim uses frame-level action conditioning, while RoboEnvision targets semantic and temporal consistency during long-horizon multi-task manipulation.
- Structured interaction: Richer structural priors, including trajectories, geometry, viewpoints, identity cues, and multimodal embodied signals, improve control and consistency across views and contacts.The surveyed methods use intermediate interaction structure to preserve scene relationships and support downstream embodied learning.
- Foundation-scale world models: Foundation-scale methods transform pretrained video diffusion models into interactive world models, increasingly positioning robotic video generation as a foundation layer for robotic interaction.Examples include Vid2World, Genie Envisioner, and DreamDojo.
6 World Model for Other Applications
World models support embodied navigation by imagining action-conditioned futures, planning states, and value signals under partial observability. In autonomous driving, they increasingly unify perception, prediction, planning, and simulation through future-evolving scene representations and action-conditioned imagination.
- Embodied navigation: World models enable navigation agents to reason about unseen spaces by imagining action-conditioned observations, future-aware planning states, and value-like signals before acting.This shifts navigation beyond purely reactive next-step decisions under severe partial observability.
- Embodied navigation: Pathdreamer, VISTA, and VISTAv2 develop look-ahead visual imagination for unseen viewpoints, instruction-conditioned navigation, and online value-map planning.Pathdreamer predicts future 360◦ RGB, depth, and semantic observations; VISTAv2 rolls out egocentric futures under candidate actions.
- Embodied navigation: Controllable navigation video models scale future prediction through NWM, sparse future generation in SparseVideoNav, and action-conditioned adaptation of Internet-scale video diffusion in EgoWM.SparseVideoNav targets faster deployment by replacing dense long-horizon rollout with sparse future generation.
- Autonomous driving: Driving world models unify perception, prediction, planning, and simulation using image-, multi-view-, occupancy-, or latent-space representations for downstream planning and driving decisions.Driving places stronger demands on long-horizon forecasting, multi-agent interaction, structured geometry, and safety-critical planning than robotic manipulation.
- Autonomous driving: Drive-WM, UniDWM, and DriveWorld-VLA advance planning-oriented driving intelligence through multiview future rollouts, structure- and dynamics-aware latent states, and action-conditioned imagination.Drive-WM uses image-based rewards to select safer trajectories, while DriveWorld-VLA uses latent world states as the planner’s decision state.
7 Benchmarks, Datasets, and Results
The section presents a layered evaluation framework for embodied world models, spanning open-loop prediction, closed-loop decision utility, and diagnostic physical grounding. Benchmark results show strong utility across multiple architectural paradigms, but persistent sensitivity to horizons, embodiments, protocols, and cross-benchmark transfer.
- Open-loop evaluation: Open-loop benchmarks test whether action-conditioned predictions remain semantically correct, temporally coherent, and responsive to commanded behavior over time.RBench emphasizes structural consistency, physical plausibility, and action completeness, while EWMBench factorizes embodied evaluation dimensions.
- Closed-loop evaluation: Closed-loop benchmarks evaluate world models as simulators, policy evaluators, or planning substrates whose predictions influence interactive decision making.WorldArena includes synthetic data generation, policy evaluation, and action planning; World-in-World tests iterative prediction and replanning through embodied task success.
- Layered evaluation framework: The evaluation framework separates open-loop future generation, closed-loop planning and policy utility, and diagnostics of physical grounding, controllability, and executability.Together, these categories indicate that realistic generation alone cannot capture embodied world-model quality.
- Benchmark results: Competitive downstream performance appears across decoupled, shared-backbone, unified, mixture-based, and latent predictive designs, rather than one architectural paradigm.The results indicate that photorealistic video generation is not necessary for effective embodied control.
- Limitations and reporting: Long-horizon robustness, cross-benchmark generalization, and standardized reporting remain major challenges because performance varies with embodiment, action space, task composition, and protocol.RoboTwin, CALVIN, and SIMPLER-style results are less suitable for strict ranking because their settings and protocols differ.
8 Challenges and Future Directions
Reliable deployment of robot-learning world models remains constrained by causal action-conditioning gaps, high computational cost, incomplete physical sensing, and missing functional evaluation standards. Future directions include more efficient predictive representations, multimodal interaction modeling, control-theoretic and symbolic abstractions, and function-aware benchmarks.
- Causal Action Conditioning: World models can generate plausible futures that are not causally faithful to the pending robot action, weakening policy alignment.Weak action conditioning arises when objectives rely mainly on observation history and task intent; WorldVLA couples future-state prediction with action generation to reduce this mismatch.
- Efficiency: World-model-based policies impose substantial training and inference costs, especially when jointly predicting videos and actions or iteratively denoising diffusion-based video predictions.Lightweight adapters, latent-space models such as LeWorldModel, and training-only world modeling such as Fast-WAM offer efficiency-oriented alternatives.
- Multimodal Physical Interaction: Visual and proprioceptive inputs miss physical properties such as friction, stiffness, and contact stability, while asynchronous tactile signals risk being overwhelmed by high-dimensional visual features.Haptic sensing and force feedback provide ground-truth interaction signals, but multimodal fusion must balance divergent frequencies and dimensions.
- Control and Structured Representations: World models need to reconcile expressive learned dynamics with formal control guarantees and reduce the computational burden of iterative rollouts for real-time planning.Potential directions include integrating learned dynamics with Lyapunov stability or robust control, alongside symbolic representations that support more stable long-horizon reasoning.
- Evaluation: Embodied world models require standardized evaluation focused on functional utility, including action sensitivity, causal consistency, long-horizon consistency, and control utility rather than visual plausibility alone.Proposed metrics include task success, policy-ranking fidelity, and executability-oriented diagnostics.