PaperlayerView exact revision on arXiv

Source-linked AI summary

Simulating the Visual World with Artificial Intelligence: A Roadmap

Jingtong Yue, Ziqi Huang, Zhaoxi Chen, Xintao Wang, Pengfei Wan, Ziwei Liu

arXiv:2511.08585v4cs.AIcs.CV

TL;DR

Video-generation systems are increasingly considered pathways to world models, but the concept and taxonomy remain unclear and some capabilities are underexplored. This survey defines a physical world model, decomposes video generation into latent simulation and rendering, and traces progress through four generations, while identifying planning-scale gaps.

  • Problem

    World-model definitions remain ambiguous and no established taxonomy organizes capabilities and developmental stages, while real-time integration, controllable video-to-world pipelines, and comprehensive evaluation metrics remain underexplored.

    Source: §1.1 · p. 2

  • Method

    The survey defines a physical world model as next-scene prediction from environmental states and contextual priors, decomposes video generation into latent world-state transitions and rendering, and proposes a four-generation taxonomy.

    Source: §1.1 · p. 3, §2 · p. 5

  • Results

    The survey describes autonomous-driving models combining navigation signals as moving toward planning, while the Genie series progresses toward interactive open-world simulation.

    Source: §4.2.4 · p. 18, §4.2.5 · p. 19

  • Takeaways & Limitations

    Video generation offers a direct, comprehensive pathway for building world models, with visual streams carrying temporal dynamics and causal relationships relevant to prediction and planning.

    Source: §1.1 · p. 1

    Microscopic and macroscopic planning remain largely underexplored, despite early work on mesoscopic planning.

    Source: §3 · p. 9

Abstract

from arXiv · show

The landscape of video generation is shifting, from a focus on generating visually appealing clips to building virtual environments that support interaction and maintain physical plausibility. These developments point toward the emergence of video foundation models that function not only as visual generators but also as implicit world models, models that simulate the physical dynamics, agent-environment interactions, and task planning that govern real or imagined worlds. This survey provides a systematic overview of this evolution, conceptualizing modern video foundation models as the combination of two core components: an implicit world model and a video renderer. The world model encodes structured knowledge about the world, including physical laws, interaction dynamics, and agent behavior. It serves as a latent simulation engine that enables coherent visual reasoning, long-term temporal consistency, and goal-driven planning. The video renderer transforms this latent simulation into realistic visual observations, effectively producing videos as a "window" into the simulated world. We trace the progression of video generation through four generations, in which the core capabilities advance step by step, ultimately culminating in a world model, built upon a video generation model, that embodies intrinsic physical plausibility, real-time multimodal interaction, and planning capabilities spanning multiple spatiotemporal scales. For each generation, we define its core characteristics, highlight representative works, and examine their application domains such as robotics, autonomous driving, and interactive gaming. Finally, we discuss open challenges and design principles for next-generation world models, including the role of agent intelligence in shaping and evaluating these systems. An up-to-date list of related works is maintained at this link.

1 Introduction

The survey frames video generation as a pathway toward physical world models and proposes a capability-based account of that progression. It distinguishes external physical simulation from internal mental modeling and defines its scope around video-based systems for representing and predicting environments.

1.1 Motivation

  • Ambiguous definitions and the lack of a capability taxonomy leave progress and gaps in areas such as real-time integration, controllable video-to-world pipelines, and evaluation metrics unclear.The survey aims to organize existing knowledge and guide future research.Source: §1.1 · pp. 2–3
  • A physical world model predicts next scenes from environmental states and contextual priors, and the survey organizes video-generation progress into four capability-based generations.The generations emphasize faithfulness, interactiveness, planning, and stochasticity, respectively.Source: §1.1 · p. 3

1.2 Position

  • The survey treats physical world models as objective simulations of external dynamics, distinct from mental models of an agent’s internal states, intentions, and preferences.Physical models emphasize physical laws and objective dynamics; mental models emphasize perception, intention, and counterfactual reasoning.Source: §1.2 · pp. 3–4

1.3 Scope

  • The survey focuses on video-generation methods that capture environmental dynamics, spatial structure, and semantics to support tasks including visual planning, counterfactual reasoning, and generalization.The broader uses of powerful world-knowledge representations remain open to exploration.Source: §1.3 · p. 4

2 Problem Definition and Taxonomy

The section frames a physical world model as a simulation of world dynamics under external conditioning, and decomposes video generation into latent state transitions and a renderer. It organizes development into four generations and defines navigation modes through three criteria for external guidance.

2 Problem Definition and Taxonomy

  • The world model is described as a digital cousin that simulates causal and spatiotemporal dynamics, including agent behavior, interactions, and goals, under external conditioning.Inputs can include text, observations, audio, navigation modes, and spatial conditions; video generation produces observable outputs.Source: §2 · pp. 4–5, Figure 4 · p. 5
  • The world model comprises latent states and a transition function that captures world knowledge, while a renderer converts resulting states into perceptual outputs.In practice, the internal state is implicit and generation remains a monolithic mapping from input to video.Source: §2 · p. 5
  • Training on large-scale multi-view and multi-temporal data instills an objective physical prior, while inference from partial conditioning signals aligns with a POMDP.The section relates the learned transition to environment dynamics in a fully observable MDP during training.Source: §2 · p. 5

2.1 Taxonomy of Four Generations from Video Generation to World Model

  • The four-generation taxonomy is an evolutionary ladder: superficial faithfulness progresses to interactiveness, complex real-time planning, and modeling rare events across multiple scales.Generation 1 supports short videos and basic spatial control; later generations add richer interaction, planning, and stochasticity.Source: §2.1 · pp. 5–6

2.2 Definition of Navigation Mode

  • A navigation mode is a structured external guidance interface that must be temporally ordered, content-independent, and support spatial reasoning across a generated sequence.The three criteria distinguish navigation modes from spatial conditions and provide a way to assess planning and interactiveness.Source: §2.2 · p. 6

3 Roadmap

The roadmap treats video generation as an implicit world model paired with a renderer, and organizes its development into four generations. Faithfulness, interactiveness, and planning deepen across those generations, culminating in stochastic, multi-scale simulation.

Roadmap

  • Video generation combines an implicit world model that simulates and predicts world states with a renderer that turns those states into video.Figure 5 organizes capabilities across generations under faithfulness, interactiveness, and planning.Source: §3 · p. 7, Figure 5 · p. 8

Generation 1

  • Generation 1 emphasizes short-video superficial faithfulness and low-level interaction, but has no obvious planning ability.Long faithfulness is defined as maintaining realistic, physically consistent generation for at least five seconds at 24 FPS.Source: §3 · p. 7

Generation 2

  • Generation 2 centers on semantic and navigational interaction, with flexible control, coherent motion, subject-specific commands, and early short-horizon task planning.Its environments are typically relatively static or simplified, while longer sequences improve temporal coherence and video–text consistency.Source: §3 · pp. 7–8

Generation 3

  • Generation 3 extends planning to complex, adaptive sequences and describes models that internalize physical laws while supporting real-time, frame-level interaction.The roadmap characterizes its planning as spanning dozens or hundreds of motion steps and multiple interacting entities.Source: §3 · pp. 8–9

Generation 4

  • Generation 4 adds stochasticity-aware planning for low-probability events and planning across arbitrary spatial and temporal scales.Its description retains Generation 3 physical faithfulness and adds global, multimodal interaction; microscopic and macroscopic planning remain underexplored.Source: §3 · p. 9

Illustrative scenario

  • The coffee-making example illustrates the progression from short clips to envisioned multi-step completion with plausible incidents.Across the example, the generations progressively expand prediction and planning ability.Source: §3 · pp. 9–10

4 Methods: A Hierarchical Taxonomy

The taxonomy treats faithfulness, interactiveness, and planning as capabilities that co-evolve, while applications span all four generations. It groups applications under Generation 2 because many recent works and open-source systems are at that stage.

Capability development

  • Faithfulness, interactiveness, and planning advance in parallel, although each generation emphasizes one capability as its dominant focus.The taxonomy describes visual realism, controllability, and long-horizon reasoning as continuously refined together.Source: §4 · p. 10

Application domains

  • General scenes, robotics, autonomous driving, and gaming overlap across generations rather than belonging exclusively to one stage.The section organizes these applications under Generation 2 because most recent works and open-source systems lie there.Source: §4 · p. 10
  • Later generations extend the same application domains toward higher-level planning, real-time interaction, and stochastic simulation.The underlying modeling principles remain consistent as capability and control increase.Source: §4 · p. 10

4.1 Generation 1 - Faithfulness: Accurate Simulation of the Real World

Generation 1 encompasses video foundation models and early spatially or navigationally conditioned world models, emphasizing superficial visual faithfulness and basic interaction. Its limited 3D dynamics constrain physical consistency, object interaction, and task-oriented planning.

4.1 Generation 1 - Faithfulness: Accurate Simulation of the Real World

  • Generation 1 focuses on superficial faithfulness and basic interaction, including pixel-level quality, video duration, and video–text consistency.The section groups its methods into video foundation models, spatial world models, and navigational world models.Source: §4.1 · p. 10

4.1.1 Foundation Models with Visual-Centric World Knowledge

  • Video foundation models provide short clips with basic text–video consistency and limited control over simple object motion, but lack strong 3D dynamics and task-oriented planning.The survey focuses on pretrained video-generation methods that produce video from text and image inputs without additional conditioning signals or modalities.Source: §4.1.1 · pp. 11–13

4.1.2 Spatial World Model

  • Spatial world models condition generation on geometry, 3D, or physics priors, trading cross-domain adaptability in some cases for stronger spatial consistency or physical plausibility.Conditions include sketches, depth maps, point clouds, and physical signals; human poses count as spatial conditions because they are scene-bound.Source: §4.1.2 · p. 13

4.1.3 Navigation World Model

  • Navigational world models use scene-independent navigation inputs for motion or camera control, with domain-specific work spanning robotics, driving, and gaming.Examples include point-based subject dragging, trajectory-driven navigation, camera viewpoint control, and interactive gameplay videos.Source: §4.1.3 · p. 14

4.2 Generation 2 - Interactiveness: Controllability and Interactive Dynamics

Generation 2 advances video world models through richer spatial and navigation-mode controls, supporting more interactive and flexible scene simulation. The section traces this shift across foundation models, general scenes, robotics, autonomous driving, and gaming.

4.2 Generation 2 - Interactiveness: Controllability and Interactive Dynamics

  • Generation 2 combines spatial conditions and navigation modes, improving video fidelity, spatiotemporal and text-video consistency, control flexibility, and 3D and semantic interaction.Real-time control remains more limited for spatially conditioned methods, while navigation-mode approaches offer greater flexibility and transferability.Source: §4.2 · p. 14, §4.2.1 · p. 15

4.2.1 Foundation Models with Dynamic-Aware World Knowledge

  • Foundation models use large-scale pretraining, higher-capacity architectures, and richer conditioning; Cosmos integrates control, memory, and planning for goal-aligned navigation and simulation.Video models include DiT and hybrid diffusion/autoregressive designs; Self Forcing trains with full autoregressive rollout for efficient video-level supervision and semi-real-time generation.Source: §4.2.1 · p. 15

4.2.2 Video Generation as World Model in General Scene

  • General-scene methods condition generation on spatial priors or navigation inputs, including geometry, 3D and physical priors, actions, trajectories, camera motion, and instructions.SketchVideo uses sparse keyframe sketches; GEN3C uses a global 3D cache for structurally faithful camera control; AETHER [153] unifies action-conditioned prediction and visual planning.Source: §4.2.2 · pp. 15–16

4.2.3 Video Generation as World Model in Robotics

  • Robotics world models use action, instruction, and goal navigation, with goal navigation covering extended planning horizons and textual or image-based goals.COMBO simulates outcomes for multi-agent cooperation, UniPi extracts actions from generated future trajectories, and UniSim supports semi-real-time video-level interaction.Source: §4.2.3 · pp. 16–17

4.2.4 Video Generation as World Model in Autonomous Driving

  • Autonomous-driving models condition video on layouts, trajectories, vehicle actions, and instructions; hybrid methods combine multiple controls and are evolving toward planning capabilities.Real-time interaction is largely absent, layout annotations can hinder deployment, and multi-step instruction stability and robustness remain underexplored.Source: §4.2.4 · p. 18

4.2.5 Video Generation as World Model in Gaming

  • Gaming world models are comparatively mature in real-time interaction, but current systems remain limited in planning, viewpoint generalization, physical consistency, and semantic representation.Genie generates videos from varied inputs modulated by real-time actions; Genie 2 demonstrates emergent object manipulation, physical reasoning, and prediction of other agents’ behavior.Source: §4.2.5 · pp. 18–19

4.3 Generation 3 - Planning: Modeling the Future Evolution of Complex Systems

Generation 3 makes planning a central world-model capability, bridging physical simulation and agent-centered reasoning. The survey identifies real-time, navigation-aware generation as an emergence criterion while noting limits in scalability, action richness, and temporal consistency.

4.3 Generation 3 - Planning: Modeling the Future Evolution of Complex Systems

  • Generation 3 centers on planning, spanning objective physical evolution and subjective simulation of trajectories, intentions, or counterfactual futures from an agent perspective.The survey frames this as the transition toward integrating a mental world model with physical simulation.Source: §4.3 · p. 19
  • Real-time inference at ≥24 FPS and navigation-aware generation mark the Generation 2–3 boundary, even before fully consistent physics or complex multi-step planning.Examples include NWM imagining robot trajectories from one image, V-JEPA 2 [37] supporting zero-shot robot planning, and Genie 3 interacting at 24 FPS and 720 p with minute-scale visual memory.Source: §4.3 · p. 19
  • Current models remain limited in scalability, action richness, and long-duration temporal consistency, despite planning’s role as a turn toward agentic and mental world modeling.The survey characterizes this turn as unifying objective and subjective dimensions of intelligent simulation.Source: §4.3 · p. 19

4.4 Generation 4 - Stochasticity: Modeling Outlier and Low-Probability Events

Generation 4 extends world modeling to stochastic, low-probability events, including rare high-impact outcomes. The survey also envisions multimodal sensing and planning across larger and smaller temporal and spatial scales, while noting that these capabilities remain nascent or out of reach.

4.4 Generation 4 - Stochasticity: Modeling Outlier and Low-Probability Events

  • The survey argues that synchronized audio could add cues for events and behaviors, enriching agents’ reasoning, prediction, and action under uncertainty.Audio–visual world models are presented as likely to develop alongside higher-level agent intelligence.Source: §4.4 · p. 20
  • Generation 4 proposes learning everyday event probabilities before simulating extremely rare, high-impact events and their coherent long-term consequences.Examples range from traffic accidents and weather to financial crises, volcanic eruptions, genetic mutations, and asteroid collisions.Source: §4.4 · pp. 19–20
  • Advanced Generation 4 models are envisioned to plan at macroscopic and microscopic scales, but long-horizon forecasting is nascent and fine-grained biological fluctuations remain beyond current models.Macroscopic planning may compress ten years of system evolution into an hour or ten minutes; microscale modeling is constrained by temporal resolution and reasoning capabilities.Source: §4.4 · p. 20

4.5 Insight: Beyond Generation

The survey envisions world models that simulate diverse possible outcomes across time and space, while also extending beyond Earth-centric settings. It points to audiovisual simulation and applications across science, medicine, robotics, and autonomous driving.

4.5 Insight: Beyond Generation

  • Repeated stochastic rollouts from an initial state could generate diverse plausible outcomes and provide downstream agents with training trajectories and interaction scenarios.The envisioned model simulates everything, everywhere, and at every time.Source: §4.5 · p. 20
  • Vision-and-audio generation points toward unified audiovisual world models jointly simulating sound, narration, and perception with physics-based consistency.Source: §4.5 · p. 20
  • World models could generalize beyond Earth-centric environments to arbitrary physical laws, enabling cosmic simulation and autonomous satellite testing.The passage frames this capability as a future possibility via fine-tuning or zero-shot generalization.Source: §4.5 · p. 20
  • Proposed applications span molecular and physical simulation, synthetic training data and virtual testbeds, drug testing, and protein-structure prediction.Source: Figure 8 · p. 21

5 Future World Model: Everything, Everywhere, Anytime Simulation

The survey presents two complementary future directions: high-fidelity simulation and generative models that create diverse worlds under their own laws. It anticipates applications across research and problem-solving, with models serving as both scientific laboratories and virtual sandboxes.

5.1 Two Complementary Development Directions

  • Precision simulators aim to reproduce real-world dynamics and stochasticity closely enough to test hypotheses and interventions in silico before real-world deployment.The authors envision simulations becoming indistinguishable from actual physical observations.Source: §5.1 · p. 21
  • Latent-dynamics models such as PlaNet and the Dreamer Series [344] support decision-making through compact transitions, imagined actions, and long-horizon planning.This direction prioritizes predictive internal representations over pixel-level realism.Source: §5.1 · p. 21
  • Generative engines could create diverse virtual worlds from one initial state, with each world governed by consistent physical or abstract laws.Repeated inferences from the same initial condition may yield divergent but plausible outcomes.Source: §5.1 · p. 21
  • The two directions are complementary: world models could support faithful study of the universe and exploration of imagined worlds.Source: §5.1 · p. 21

5.2 Applications and Societal Impact

  • Potential applications include robotics training data, autonomous-driving failure-case simulation, wildlife and climate forecasting, and cosmic or asteroid-impact scenarios.The survey also names biology, physics, astronomy, medicine, and chemistry as disciplines that could be affected.Source: §5.2 · pp. 21–22

5.3 Conclusion and Outlook

  • The conclusion casts integrated accurate simulation and generative, zero-shot capabilities as a scientific laboratory and virtual sandbox for exploring complex systems.Source: §5.3 · p. 22
Loading 2511.08585v4…