PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding World or Predicting Future? A Comprehensive Survey of World Models

Jingtao Ding, Yunke Zhang, Yu Shang, Jie Feng, Yuheng Zhang, Zefang Zong, Yuan Yuan, Hongyuan Su, Nian Li, Jinghua Piao, Yucheng Deng, Nicholas Sukiennik, Chen Gao, Fengli Xu, Yong Li

arXiv:2411.14499v4cs.CLcs.AIcs.LG

TL;DR

World-model research needs a systematic account of whether models understand the world or predict its future, amid advances in multimodal and video-generation systems. This survey categorizes world models by implicit representation and future prediction, reviews their applications, and identifies challenges including weak physical-law generalization. [p00,p01,p07,p08]

  • Problem

    Recent multimodal and video-generation models intensify questions about whether they constitute comprehensive world models and motivate a systematic review of the field.

  • Method

    The survey categorizes world models by implicit representations and future predictions, then organizes research across applications including generative games, autonomous driving, robotics, and social simulacra.

  • Results

    The survey synthesizes progress, applications, and future directions, with evidence that data-driven scaling alone is insufficient for robust physical-law generalization.

  • Takeaways & Limitations

    World models are studied as tools for understanding world mechanisms and predicting future states to support simulation and decision-making.

  • Takeaways & Limitations

    Video-generation world models remain limited in causal reasoning, restricting active prediction of how actions alter events.

Abstract

from arXiv · show

The concept of world models has garnered significant attention due to advancements in multimodal large language models such as GPT-4 and video generation models such as Sora, which are central to the pursuit of artificial general intelligence. This survey offers a comprehensive review of the literature on world models. Generally, world models are regarded as tools for either understanding the present state of the world or predicting its future dynamics. This review presents a systematic categorization of world models, emphasizing two primary functions: (1) constructing internal representations to understand the mechanisms of the world, and (2) predicting future states to simulate and guide decision-making. Initially, we examine the current progress in these two categories. We then explore the application of world models in key domains, including generative games, autonomous driving, robotics, and social simulacra, with a focus on how each domain utilizes these aspects. Finally, we outline key challenges and provide insights into potential future research directions. We summarize the representative papers along with their code repositories in https://github.com/tsinghua-fib-lab/World-Model.

1 INTRODUCTION

World models are debated as systems for either understanding the world’s underlying mechanisms or predicting future states for decision-making. This survey organizes the field around these two functions and examines their applications, prior surveys, and future directions.

  • World models are commonly defined from two perspectives: understanding the world and predicting its future.The first emphasizes abstracting external mechanisms, while the second adds envisioning possible states to inform decisions.
  • Applications impose distinct requirements, including real-time road perception and evolution forecasting in autonomous driving and dynamics understanding for robotics.Robotic uses include navigation, object detection, and task planning.
  • Existing surveys mainly cover domain-specific applications or transitions from multimodal models to world models, leaving the constitution of world models insufficiently examined.This survey addresses that gap through a systematic categorization.
  • The survey categorizes world models by constructing implicit representations of external mechanisms and predicting future external states.These categories respectively support internalized world knowledge and predictive or simulative capabilities from visual perceptions.
  • The paper reviews the two categories, surveys applications, and outlines open problems and future directions.Its application coverage includes generative games, autonomous driving, robots, and social simulacra.

2 BACKGROUND AND CATEGORIZATION

World models have evolved from internal representations and transition models toward systems that also predict future states and support interactive simulation. The survey traces these developments across cognitive foundations, model-based reinforcement learning, multimodal architectures, video generation, and applications.

  • The survey organizes world-model research into two branches: internal representation and future prediction.This distinction captures efforts to understand environmental dynamics and simulate possible future scenarios.
  • Pre Deep Learning Era: Early AI and reinforcement-learning approaches built structured or explicit environment models to represent dynamics and simulate action sequences before execution.These models used frames, transition models, tabular methods, or simple parametric functions.
  • Model-based Reinforcement Learning: Ha and Schmidhuber’s 2018 work popularized recurrent implicit world models that learn latent representations inspired by human mental models.The approach abstracts the external world into simplified elements and relationships.
  • Self-supervised Learning: JEPA combines an encoder and predictor in a self-supervised framework that learns world representations without extensive labeled data.The predictor evaluates sensory information for action assessment and response selection.
  • Large Language Models: Large language and video-generation models extend world modeling through latent world knowledge and generated visual sequences capturing temporal dynamics, spatial consistency, and physical laws.Recent systems such as Cosmos and Genie 3 are associated with physics-law adherence and real-time controllable simulation.
  • Interactive 3D Environments: Interactive 3D scene generation creates explorable virtual worlds from limited inputs, including single-image generation of interactive scenes.This paradigm emphasizes spatial consistency and user interaction.
  • Applications: World models have expanded across autonomous driving, robotics, navigation, and other application domains since 2023.Examples include GAIA-1 and Drive-WM for traffic dynamics and DayDreamer for robotics.
  • Cognitive mental models parallel AI world models by representing structural relationships and enabling predictive simulation of alternative scenarios.These representations preserve essential structure rather than exhaustive detail.

3 IMPLICIT REPRESENTATION OF THE EXTERNAL WORLD

World models support decision-making by representing environment dynamics for hypothetical action evaluation and policy generation. The survey covers model-based reinforcement learning and language-based world models, including direct action generation and modular planning.

  • Decision-making role: World models represent environments as latent variables, enabling hypothetical actions without affecting the real environment and reducing trial-and-error costs.They provide a foundation for optimized policy generation by supporting environment understanding.
  • Model-based reinforcement learning: In model-based reinforcement learning, the world model comprises state-transition dynamics and a reward function, with transition learning supporting policy optimization.The key learning task is typically estimating transition dynamics from interaction trajectories.
  • Model-based reinforcement learning: World-model learning commonly uses supervised objectives over trajectories, including one-step prediction error and probabilistic transition modeling for aleatoric uncertainty.Parameterized transitions can be learned from simulation data collected through real-environment interactions.
  • Policy generation: Policy generation uses model predictive control, Monte Carlo Tree Search, or related trajectory-optimization methods to select actions under learned dynamics.MPC plans action sequences over a planning horizon, while MCTS searches state trees using value estimates.
  • Policy generation: TD-MPC2 integrates trajectory optimization in the latent space of an implicit world model and scales to large agents across multiple continuous-control domains.The survey reports strong performance across diverse continuous-control tasks and training with hundreds of millions of parameters.
  • Language-based world models: Language-based world models use LLMs or MLLMs for direct action generation, environmental understanding, and modular planning with external verifiers or planners.Examples include navigation, robot manipulation, multi-expert visual-language navigation, and PDDL-based planning.
  • Language-based world models: Web-navigation agents use world models to predict action outcomes or combine global and local knowledge, addressing irreversible actions, blind trial-and-error, and hallucinatory actions.WebDreamer uses a specialized LLM to simulate actions and achieves competitive performance with higher efficiency than tree-search methods.

4 FUTURE PREDICTION OF THE PHYSICAL WORLD

Video world models extend world modeling from static or discrete future states toward continuous, multimodal, and interactive simulations of embodied environments. The survey highlights progress in dynamic environment generation while identifying causal reasoning, physical fidelity, and long-term prediction as unresolved challenges.

  • Video World Models: Video world models generate visual sequences from past observations and potential actions to simulate evolving future states.They focus on visual frames that represent environmental dynamics over time.
  • Video World Models: Sora produces temporally consistent videos up to one minute long, but its limited causal reasoning restricts active action-outcome prediction.It also inconsistently reproduces complex physical laws involving forces, fluids, and light.
  • Capabilities: Effective video world models require long-term predictive ability, multimodal integration, interactivity, and support for diverse environments.These capabilities target temporally coherent, realistic, controllable, and varied simulations.
  • Embodied Environments: Embodied-environment research is shifting from visual simulation toward interactive environments incorporating spatial and physical interactions.This transition provides platforms for developing agents that interact with complex real-world environments.
  • Embodied Environments: Embodied world models are categorized as indoor, outdoor, or dynamic environments, with most work still focused on static settings and growing interest in future-world prediction.Dynamic first-person simulations can provide flexible feedback for training embodied agents.
  • Dynamic Environments: Generative dynamic environments create flexible, real-time scenarios from multimodal conditions, allowing agents to train across varied manipulation and navigation settings.UniSim uses spatial movements, textual commands, camera parameters, and multimodal data to generate robot-manipulation videos.
  • Dynamic Environments: Physical constraints such as camera trajectories, normal maps, depth maps, keypoints, and prior geometric predictions improve realism and physical adherence in dynamic simulations.The survey presents Aether, TesserAct, Roboscape, and Deepverse as representative approaches.

5 APPLICATION DOMAINS

Gaming environments provide controlled settings for evaluating world models because they expose explicit rules and action-consequence relationships. Applications emphasize interactive gameplay, temporally consistent states, and generalization across diverse scenarios.

  • Gaming Environments: Games enable precise evaluation of world models through well-defined rules and observable action-consequence relationships.They provide controlled yet complex environments where physics, causality, and interaction can be tested.
  • Gaming Environments: World models shift game development toward generative systems that create content, adapt to player behavior, and support emergent gameplay.This contrasts with manually coded rules, pre-designed assets, and scripted interactions.
  • Capability Dimensions: GameNGen demonstrates real-time interactive gameplay at 20 frames per second while maintaining stable sessions.Interactivity is identified as a fundamental capability for gaming world models.
  • Capability Dimensions: Gaming world models must preserve numerical and spatial consistency so gameplay mechanics, scores, states, and scene transitions remain coherent.MineWorld addresses consistency through visual-action autoregressive transformers that model game states and action-state relationships.
  • Capability Dimensions: Generalization across diverse environments remains challenging, with approaches using scene-generalizable control, generative priors, and exploration driven by world-model uncertainty.These methods target adaptation to new games and environments without environment-specific rewards.

5.2 Embodied Intelligence

Embodied world models help robots understand unfamiliar environments, generate synthetic training data, imagine future observations for action generation, and simulate environments for policy evaluation. These capabilities support anticipatory control, long-term performance, and reduced simulation-to-reality gaps.

  • Understanding the Environment: Robots need implicit environmental representations because policies successful in structured settings can fail when unfamiliar features or dynamics are not explicitly modeled.Learning these representations is presented as a crucial first step toward robotic intelligence.
  • Applications: Embodied world models support synthetic robot-data generation, imagined-future action guidance, and environment simulation for policy evaluation.These three application areas organize the survey’s discussion of world models in robotics.
  • Predicting Future States: Generative video models guide action generation by learning environmental dynamics from visual data and predicting future observations or trajectories.Examples include diffusion- and transformer-based systems, inverse dynamics models, and latent-space planning with model predictive control.
  • Policy Evaluation: World-model simulators show high correlation between policy evaluation in learned environments and real environments, indicating captured world-transition dynamics.IRASim and Roboscape generate videos from initial frames and trajectories for evaluation.
  • Implications: Generative, vision-centric world modeling supports anticipatory robotic control and simulation, improving robots’ reasoning about future states and long-term task performance.The survey presents these capabilities as promising foundations for embodied intelligence.
  • From Simulation to Real World: World models can narrow the simulation-to-reality gap through generalized representations of real-world dynamics, enabling rapid adaptation across morphologies and environments.DayDreamer learns locomotion in real environments within hours, while SWIM reports task generalization with less than 30 minutes of interaction.

5.3 Urban Intelligence

Urban intelligence applications use world models to understand scenes and predict future states across autonomous driving, logistics, aerial vehicles, and urban analytics. Progress is strongest in scene understanding and simulation, while geographically accurate image generation and prediction remain limited.

  • Autonomous Driving: Autonomous driving world models align with a four-stage pipeline of perception, prediction, planning, and control, with perception and prediction supporting scene understanding.The survey organizes driving applications around implicit representation and future prediction.
  • Autonomous Driving: Earlier autonomous-driving systems separated perception and prediction into modules trained on distinct tasks and datasets, using images, point clouds, and other inputs.Perception produced abstract scene representations before downstream prediction and planning.
  • Autonomous Driving: Driving systems evolved from point-cloud and image-based deep learning toward transformer-based perception, multimodal reasoning, and traffic-participant behavior prediction.RNNs, CNNs, and transformers encode historical scene information to forecast future behaviors.
  • World Simulators: Diffusion-based video world models generate realistic camera perceptions for traffic scenarios, while other systems forecast future 3D occupancy grids.Drive-WM adds closed-loop planning control, and Vista targets higher resolution and longer prediction duration.
  • Autonomous Logistics: Autonomous logistics uses world models for navigation in complex environments and human-robot interactions through physical simulators or interactive controllable video models.Miniature mobile logistics vehicles use internet videos for environmental understanding and policy generalization.
  • Aerial Vehicles and Urban Analytics: Low-altitude aerial-vehicle research mainly addresses scene comprehension and navigation, while prediction applications remain nascent.Urban analytics uses multimodal models for scene recognition, human mobility, and GPS-conditioned image generation.
  • Urban Analytics: Existing image-generation models can relate GPS signals to urban scenery but still struggle to distinguish cultural styles and scenic characteristics across locations.The survey characterizes urban-analytics applications overall as relatively limited.

5.4 Societal Intelligence

Social simulacra operationalize societal intelligence by modeling societies explicitly and by enabling agents to form implicit world models through interaction. LLM-driven systems support investigations of social, strategic, economic, and collective phenomena.

  • Societal Intelligence: Societal intelligence involves collective sensing, reasoning about possible futures, and coordination toward shared goals among individuals, institutions, and surroundings.Social simulacra provide an in-silico setting for studying these interactions.
  • Mirroring Society: LLM-driven simulacra reproduce stylized social facts and generate credible predictions, extending beyond rule-based or reinforcement-learning systems’ simpler dynamics.AI Town illustrates believable individual behavior and emergent group dynamics resembling real communities.
  • Two Perspectives: The survey studies social simulacra as explicit models that mirror societies and as environments where agents develop implicit representations guiding decisions and social behavior.These two perspectives structure the societal-intelligence analysis.
  • Social Networks: LLM agents can reproduce information diffusion and spontaneously form network structures that mirror adaptive and communicative aspects of human societies.These findings concern social-network dynamics within simulated environments.
  • Large-Scale Platforms: Large-scale platforms such as AgentSociety support studies of polarization, policy interventions, and other phenomena across social, strategic, and economic domains.Several later platforms extend this multi-domain simulation vision.
  • Implicit World Models: Through interaction, agents accumulate experiences, store memories, and transform them into implicit world models that support reasoning about other agents and the broader environment.Structured beliefs, reflection, debate, and resource-focused dialogue are cited as mechanisms for these representations.

5.5 Functions of World Models

World models receive external commands or actions and model dynamic environmental state transitions. They operate either as cloud-based environments that generate data or as edge-side agent brains that convert observations into actions.

  • Core Functions: World models broadly serve as cloud-based environments or edge-side agent brains.The distinction concerns whether modeling primarily supports generated environments and data or deployed agent behavior.
  • Cloud-Based Environments: Cloud-based world models commonly use video-generation systems to synthesize high-quality video from text or action trajectories.These systems provide generated environmental data at scale.
  • Edge-Side Agent Brains: A two-stage edge-side approach processes visual observations first and then converts them into executable actions.This design links world-model perception to downstream action execution.

6 OPEN PROBLEMS AND FUTURE DIRECTIONS

The survey identifies open problems spanning physical fidelity, social behavior, benchmarking, embodiment, efficiency, safety, and accountability. It presents hybrid physics-based approaches, richer evaluation protocols, and system-level safeguards as directions for future work.

  • 6.1 Physical Rules and Counterfactual Simulation: Hybrid world models combine explicit simulators or physical priors with generative models to improve physically plausible simulation.Examples couple physics engines with rendering or diffusion refinement.
  • 6.1 Physical Rules and Counterfactual Simulation: Data-driven scaling alone may not recover robust physical laws for unseen counterfactual scenarios.Diagnostic studies report failures on out-of-distribution, combinatorial, and physics-reasoning tests.
  • 6.2 Enriching the Social Dimension: Realistic world models must represent human behavior and social interaction alongside physical elements.Evaluating generated human behavior remains difficult because realism assessment largely relies on subjective human judgment.
  • 6.3 Benchmarking World Models: Benchmarking lacks a canonical task or metric because world models pursue divergent goals across heterogeneous methods and domains.Recent testbeds separately assess perception, dynamics, physical fidelity, spatial reasoning, controllability, and embodied-process errors.
  • 6.3 Benchmarking World Models: Future benchmarks should become more diverse and realistic while evaluation protocols become standardized for stronger comparability and robustness assessment.The survey also points toward self-reinforcing loops linking generative world models with embodied agents.
  • 6.5 Simulation Efficiency: Simulation efficiency is constrained by autoregressive generation and the computational cost of large physical simulations.Suggested directions include model distillation, coordinated large-and-small models, request scheduling, and learned physical approximations.
  • 6.6 Ethical and Safety Concerns: World models introduce risks of harmful scenario simulation, misinformation, disinformation, and deepfake misuse.The survey calls for safeguards, detection methods, watermarking, and legal solutions.

7 CONCLUSION

The survey frames world models around understanding the external world and predicting its future states. It synthesizes research across core capabilities and applications, then identifies open problems and future directions.

  • The paper presents a comprehensive survey organized around implicit representations and future predictions of the external world.
  • It reviews world models for decision-making, learned world knowledge, video generation, and embodied environments.
  • It covers applications including generative games, robotics, autonomous driving, and social simulacra while highlighting unresolved challenges.

A RELATED SURVEY

The related-survey comparison positions this paper as a comprehensive overview of the systematic definition and capabilities of world models.

  • The paper compares itself with existing surveys through Table S1.
  • Its scope emphasizes a comprehensive overview of world models.
  • The survey focuses on systematic definition and capabilities.

B FIGURES AND TABLES

The supplementary figures and tables include a figure describing the development of the robotic world model.

  • Figure S1 concerns the development of the robotic world model.
  • Figure S1 provides a visual account of robotic world-model development.

C UPDATE HISTORY

The survey’s update history records a published ACM Computing Survey version and subsequent revisions to its history, roadmap, application-domain organization, and recent-paper coverage.

  • C UPDATE HISTORY: The survey published an ACM Computing Survey version on 2025.09.09.
  • C UPDATE HISTORY: On 2025.11.10, Section 2 was rewritten to cover history and current development.
  • C UPDATE HISTORY: The revision summarized the deep-learning-era roadmap of world models in Figure 2 and reorganized application domains in Section 5.
  • C UPDATE HISTORY: The update also incorporated recent papers and included Figure S2 on world models and social simulacra.
Loading 2411.14499v4…