Source-linked AI summary
Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond
Zheng Zhu, Xiaofeng Wang, Wangbo Zhao, Chen Min, Bohan Li, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, Yang You, Zhaoxiang Zhang, Dawei Zhao, Liang Xiao, Jian Zhao, Jiwen Lu, Guan Huang
TL;DR
World models are pursued as a route toward AGI, and recent surveys have covered a narrower range of applications. This survey reviews world models for video generation, autonomous driving, and agents, and offers broader coverage while examining their challenges and future directions.
Problem
Method
The survey reviews world-model advances and applications in video generation, autonomous driving, and autonomous agents, and assesses challenges and future directions.
Source: §1 · p. 2Results
Takeaways & Limitations
The survey aims to offer newcomers a reference and stimulate critical thinking and discussion among established researchers.
Source: §1 · p. 2World models need to extrapolate beyond training data to imagine rare accidents or abnormal driving behaviors.
Source: §5.1.1 · p. 19
Abstract
from arXiv · showhide
General world models represent a crucial pathway toward achieving Artificial General Intelligence (AGI), serving as the cornerstone for various applications ranging from virtual environments to decision-making systems. Recently, the emergence of the Sora model has attained significant attention due to its remarkable simulation capabilities, which exhibits an incipient comprehension of physical laws. In this survey, we embark on a comprehensive exploration of the latest advancements in world models. Our analysis navigates through the forefront of generative methodologies in video generation, where world models stand as pivotal constructs facilitating the synthesis of highly realistic visual content. Additionally, we scrutinize the burgeoning field of autonomous-driving world models, meticulously delineating their indispensable role in reshaping transportation and urban mobility. Furthermore, we delve into the intricacies inherent in world models deployed within autonomous agents, shedding light on their profound significance in enabling intelligent interactions within dynamic environmental contexts. At last, we examine challenges and limitations of world models, and discuss their potential future directions. We hope this survey can serve as a foundational reference for the research community and inspire continued innovation. This survey will be regularly updated at: https://github.com/GigaAI-research/General-World-Models-Survey.
1 INTRODUCTION
The survey presents world models as generative systems for predicting and understanding the world, with applications in video generation, autonomous driving, and autonomous agents. It reviews their advances, applications, challenges, and future directions, and aims to cover a broader range of uses than recent surveys.
1 INTRODUCTION
- Video-generation world models create and edit video, while driving models use scenarios and learned policies for action prediction and end-to-end driving.Agent world models support interactions in virtual settings such as games and physical settings such as robots.Source: §1 · p. 1, Figure 1 · p. 3
- Generative models can produce controlled training data that reduces the need for extensive real-world collection, while world-model understanding helps generated video respect physical constraints.The survey describes this relationship as symbiotic: each informs and enriches the other.Source: §1 · pp. 1–2
- Driving world models predict future scenarios despite real-world sample complexity, with newer generative approaches extending beyond earlier work focused on simulation environments.The survey describes applications including generating driving actions and training driving-policy networks.Source: §1 · p. 2
- World models support agent learning and planning: game agents predict future states, while robotic approaches learn manipulation policies, simulate interactions, or propose plans.The survey also discusses JEPA, which predicts outputs in a higher-level representation space.Source: §1 · p. 2
- The survey offers broader coverage than recent surveys [40], [41], [42],, spanning video generation and decision-making applications such as autonomous driving and robotics.It also assesses challenges and prospective research directions.Source: §1 · p. 2
2 VIDEO GENERATION AS A GENERAL WORLD MODEL
The section frames video generation as a task that requires models to understand and simulate physical mechanisms. It outlines the survey’s review of generation technologies and recent models, then discusses Sora as a major breakthrough.
2 VIDEO GENERATION AS A GENERAL WORLD MODEL
- Video generation aims to create realistic videos by understanding and simulating mechanisms in the physical world.Source: §2 · p. 3
- The section first introduces video-generation technologies, then reviews advanced models that have emerged in recent years.Source: §2 · p. 3
- The section concludes by discussing Sora, which the survey considers the largest breakthrough in video generation.Source: §2 · p. 3
2.1 Technologies behind Video Generation
This section surveys text-to-video generation technologies, visual foundations, text encoders, and generation methods. It describes how these components support the synthesis of videos from text prompts.
2.1 Technologies behind Video Generation
- The survey focuses on text-to-video generation and introduces visual foundations, text encoders, and generation techniques.Source: §2.1 · p. 3
2.1.1 Visual Foundation Models
- Visual foundations include convolutional and Transformer-based models; video extensions use 3D convolutions or spatial-temporal attention, and DiT is Sora’s main architecture.U-Net uses low- and high-level backbone features for pixel-wise prediction and is also used in image generation.Source: §2.1.1 · p. 3
2.1.2 Text Encoders
- Text encoders extract prompt embeddings; CLIP aligns image and text but may struggle with long, detailed prompts, while language models better understand them.Source: §2.1.2 · p. 4
2.1.3 Generation Techniques
- GANs train a generator against a discriminator, while diffusion methods learn to generate images by reversing a gradual Gaussian-noising process.In the described GAN update, the discriminator learns to distinguish real and generated data before the generator is trained to make generated samples approach real data.Source: §2.1.3 · p. 4
- Diffusion denoising reverses the noise process, and a denoising model learns to predict the noise added at each timestep.Source: §2.1.3 · p. 4
- Autoregressive models predict each token from earlier tokens and maximize current-token likelihood; masked modeling reconstructs randomly hidden tokens and also supports image and video generation.The survey also describes LVM as scaling to 420 billion training tokens and 3 billion parameters.Source: §2.1.3 · pp. 4–5
2.2 Advanced Video Generation Models
The survey reviews text-to-video approaches aimed at semantic alignment and cross-frame consistency, tracing a shift from pre-2021 GAN dominance to later autoregressive, diffusion, and masked models. It closes with dataset needs and evaluation comparability.
2.2.1 GAN-based Methods
- GAN text-to-video designs condition generation on captions or paragraphs and use multi-level discriminators or story/context encoders to promote text alignment and consistency.TGANs-C uses video-, frame-, and motion-level discriminators; TFGAN uses text-derived filters, while StoryGAN conditions each frame on sentence and paragraph representations.Source: §2.2.1 · p. 5
2.2.2 Diffusion-based Methods
- Diffusion methods span cascaded spatial-temporal upsampling, staged data curation, efficient spatial-temporal Transformers, and memory-based long-video extension.StreamingT2V extends short generated clips autoregressively and supports videos exceeding 1,200 frames.Source: §2.2.2 · p. 5
2.2.3 Autoregressive Modeling-based Methods
- Autoregressive methods predict latent video tokens sequentially, with sparse axis-wise attention and pretrained, keyframe-first hierarchical generation addressing computation and text alignment.GODIVA disentangles attention across temporal, row, and column dimensions; CogVideo generates key frames before recursively interpolating frames.Source: §2.2.3 · pp. 5–6
2.2.4 Masked Modeling-based Methods
- Masked modeling replaces sequential decoding with parallel token refinement; MAGVIT uses masked bidirectional reconstruction, while WorldDreamer uses windowed attention and generates faster than diffusion and autoregressive methods.MAGVIT-v2 improves the video tokenizer and supports joint image-video tokenization; VideoPoet integrates it into a large language model.Source: §2.2.4 · p. 6
2.2.5 Datasets and Evaluation Metrics
- Video-text alignment and caption richness matter for generation quality, while seed sensitivity and differing evaluation datasets make comparisons across papers difficult.The survey notes human preference annotations and comprehensive benchmarks as potential routes toward fairer evaluation.Source: §2.2.5 · pp. 6–7
2.3 Towards World Models: Sora
The section analyzes Sora, a closed-source text-to-video model, mainly through OpenAI’s technical report and outlines its likely latent-diffusion framework and data strategy. It presents Sora’s action-consistent examples and training and scaling approaches as possible inspiration for world-model research, while noting that its training data remain undisclosed.
2.3 Towards World Models: Sora
- Sora is a closed-source text-to-video model whose analysis here relies mainly on OpenAI’s technical report and may differ from its actual implementation.Source: §2.3 · p. 7
2.3.1 Framework
- Sora is described as a diffusion model combining video compression, latent-space Transformer diffusion, and language-conditioned generation.The proposed framework uses spatial-only or spatial-temporal compression, adds spatial-temporal and text cross-attention to a DiT-like model, and trains it to predict latent noise.Source: §2.3.1 · p. 7
2.3.2 Training Data
- Sora’s proposed data strategy uses a trained video captioner to recaption training videos and GPT-4 to expand simple user prompts at inference.The survey says data quality affects generation performance and that recaptioning improves training-data quality.Source: §2.3.2 · pp. 7–8
2.3.3 Towards World Models
- Persistent painterly strokes and burger bite marks are presented as evidence of action-result prediction; the survey also identifies training and inference strategies and scaling as potential world-model inspirations.The sources and scale of Sora’s training data are undisclosed; the suggestion that game videos were included is speculation.Source: §2.3.3 · p. 8
3 WORLD MODELS FOR AUTONOMOUS DRIVING
Autonomous-driving world models address uncertainty by predicting future changes in driving environments, with approaches spanning end-to-end driving and neural simulation. The survey covers methods for 2D and 3D scene generation, planning, and pre-training.
3.1 End-to-end Driving
- MILE exceeds prior cutting-edge methods by 31% in driving score on CARLA, including new town and weather conditions.It learns in 3D Bird’s Eye View space from an offline urban-driving dataset.Source: §3.1 · pp. 9–10
- SEM2 targets sample efficiency and robustness to input permutations; TrafficBots forecasts multi-agent actions, while Think2Drive reaches expert-level CARLA v2 performance after three days of single-GPU training.TrafficBots is evaluated on Waymo motion prediction; Think2Drive introduces the CornerCase Repository benchmark.Source: §3.1 · p. 10
- End-to-end driving methods still primarily emphasize simulation environments.The survey identifies this as a significant limitation despite advances using reinforcement learning.Source: §3.1 · p. 10
3.2.1 2D Scene Generation
- GAIA-1 predicts video, text, and action tokens, while DriveDreamer conditions generation on structured traffic information such as HD-Maps and 3D boxes.GAIA-1 was trained on 4,700 hours of London driving videos; DriveDreamer reports precise, controllable video generation on nuScenes.Source: §3.2.1 · pp. 10–11
- DriveDreamer-2 uses an LLM to translate queries into agent trajectories and HD-Maps, while Panacea and DrivingDiffusion target coherent multi-view video generation.Panacea uses decomposed 4D attention; DrivingDiffusion adds post-processing and a temporal sliding window for longer videos.Source: §3.2.1 · p. 11
- Drive-WM evaluates planner-sampled trajectories using generated future scenarios and an image-based reward, and SubjectDrive finds data diversity pivotal to scaling generated videos for perception models.Real-world driving-dataset tests support Drive-WM’s multi-view video generation capability.Source: §3.2.1 · p. 12
3.2.2 3D Scene Generation
- 3D scene methods forecast future LiDAR or occupancy representations, with LidarDM additionally generating layout-aware, physically plausible LiDAR videos.Copilot4D predicts point clouds, OccWorld forecasts occupancy and ego tokens, and MUVO predicts RGB, occupancy, and LiDAR scenes from camera and LiDAR inputs.Source: §3.2.2 · p. 12
- World-model-based 4D pre-training targets downstream driving tasks, while DriveWorld separately addresses aleatoric and epistemic uncertainty through temporal and spatial modules.UniWorld predicts future occupancy; ViDAR predicts future LiDAR point clouds using latent rendering and differentiable ray-casting.Source: §3.2.2 · pp. 12–13
4 WORLD MODELS FOR AUTONOMOUS AGENTS
Autonomous agents must plan in complex, stochastic environments where explicit human modeling is difficult. The section introduces world models as a way to learn predictive environment models and describes the section’s coverage of agent frameworks, tasks, and benchmarks.
4 WORLD MODELS FOR AUTONOMOUS AGENTS
- Planning in unknown, complex environments is difficult to specify from human experience, while model-free reinforcement learning can require prohibitively many environment interactions.Agents otherwise accumulate experience and learn behavior directly through interaction.Source: §4 · p. 13
- World Models showed that an RNN can learn environment dynamics from agent experience and support policy learning in imagination.The survey notes that this can greatly reduce the cost of trials and failures.Source: §4 · p. 13
- The section covers world-model agent frameworks and structures, applications in games and robotics, and commonly used performance benchmarks.Source: §4 · p. 13
4.1 General Framework of an Agent based on World Model
The agent framework centers on a world model that predicts future environment states and interacts with sensing, acting, and planning components. The section surveys key components and widely used model structures.
4.1.1 Key Components
- The framework decomposes an agent into sensors, actors, planning, and a world model.Sensors provide environmental information, actors execute actions, and planning selects action sequences toward goals.Source: §4.1.1 · p. 13
- The world model updates its state from component information and answers queries through tell-and-ask interfaces.Given current perceptions and actions, it predicts future states to guide planning.Source: §4.1.1 · p. 14
4.1.2 Widely-used Model Structure
- RSSM, JEPA, and Transformer-based models are widely used structures, with Transformers gaining attention since 2022.RSSM predicts in latent space with stochastic and deterministic state parts; JEPA predicts in higher-level representations rather than generating pixels directly.Source: §4.1.2 · p. 14, Figure 6 · p. 14
4.2 Agents for Different Tasks
World models support agents across games, robotics, navigation, and other tasks by enabling prediction and learning in imagined environments. The surveyed work spans sample-efficient game learning, real-world and visually robust robotics, and broader task generalization.
4.2.1 Game Agent
- Under a 100K-interaction constraint, SimPLe performs much better on Atari than previous model-free reinforcement learning methods.It uses a stochastic video-prediction model; world-model predictions let agents learn in imagination with fewer environment interactions.Source: §4.2.1 · p. 15
- DreamerV2 learns Atari behavior from imagined sequences and reaches human-comparable performance on Atari 200M.Its RSSM uses discrete categorical variables rather than continuous latent representations.Source: §4.2.1 · p. 15
- Genie learns interactive 2D worlds from unlabeled internet videos and generates action-conditioned video sequences.It can create environments from image or text prompts, although its video quality and frame rate still need improvement.Source: §4.2.1 · p. 15
4.2.2 Robotics
- In robotics, real-world interaction is costly and dynamics are stochastic, making sample efficiency especially important; unlike games, robot control also uses continuous actions.DreamerV2 has been applied online to four real robots, and the survey reports that it can learn skills quickly in real environments.Source: §4.2.2 · pp. 15–16
- Reconstruction-free latent learning addresses RSSM’s sensitivity to visual distractions, and DreamingV2 outperforms DreamerV2 on five simulated robot tasks.DreamerPro and Dr.G also use reconstruction-free approaches and demonstrate robustness to visual distractions in DMC environments with complex backgrounds.Source: §4.2.2 · p. 16
4.2.3 Diverse Environments and Tasks
- DreamerV3 masters multiple domains with fixed hyperparameters, while SWIM reports about twice the success of prior approaches with under 30 minutes of real-world interaction data.Other directions include prediction of future panoramic views for navigation and action-conditioned simulation of real-world interactions.Source: §4.2.3 · pp. 16–17
4.3 Commonly Used Benchmarks
The survey describes Atari100k and robotics benchmarks used to evaluate world-model-based agents, while noting that results across robotic studies are difficult to align. Reported Atari results are mixed across games despite the 100K-interaction limit.
Atari100k
- Atari100k uses 26 games and allows up to 100K interactions per game; Normalized Human Score compares agent scores with professional-human and random-policy scores.Table 5 collects scores reported for world-model-based game agents.Source: §4.3 · pp. 17–18, Table 5 · p. 17
- World-model-based agents outperform human players in about half of 26 Atari100k games with 100K interactions, but perform much worse in Alien, Amidar, and Seaquest.The survey suggests complex dynamics, insufficient interactions, or image-reconstruction methods missing important details as possible explanations.Source: Table 5 · p. 17, §4.3 · p. 18
Robotics benchmarks
- DMC supports visual-input control tasks built on MuJoCo, while DMC Remaster adds seven visual factors to test visual robustness.RoboSuite provides standardized robot-learning tasks and programmatic environment generation.Source: §4.3 · p. 18
- Robotic results are difficult to align because studies use different tasks and interaction constraints; Table 6 reports episode returns for selected agents and tasks.The cited evaluation varies latent-space discreteness and whether image reconstruction is used.Source: Table 6 · p. 17, §4.3 · p. 18
5 DISCUSSION
The survey argues that realistic video generation alone does not make a world model: models must reason about counterfactuals, physical laws, and situations beyond their training data. It identifies efficiency and evaluation gaps, and proposes 3D simulation and embodied intelligence as future directions.
5.1 General World Models
- Video generation is one manifestation of world models, but does not by itself address their core challenges.The discussion distinguishes visual realism from reliable world simulation.Source: §5.1.1 · p. 18, §5.1 · p. 18
5.1.1 Challenges
- World models should infer outcomes for unseen decisions through counterfactual reasoning, yet this remains challenging for current AI systems.The discussion gives unfamiliar traffic accidents and new robot environments as examples where simulating alternative actions could inform safer responses.Source: §5.1.1 · p. 18, Figure 7 · p. 19
- Sora’s realism does not ensure adherence to physical laws, and video-text training alone is described as insufficient for complex phenomena such as fluids.The survey suggests combining Sora with physics-driven simulators, which follow physical properties even if they may be less realistic.Source: §5.1.1 · p. 18
- World models need to extrapolate to rare events, while autoregressive video generation can be slow and generation-quality metrics do not measure predictive rationality.The discussion cites human-centric evaluation as a way to assess whether generated videos match user expectations or human reasoning.Source: §5.1.1 · p. 19
5.1.2 Future Perspectives
- Future world models should represent 3D spatial relationships, depth, and volume, and can serve as simulators for training embodied agents’ decisions.The survey cites Drive-WM as a preliminary attempt at embodied intelligence in autonomous driving.Source: §5.1.2 · p. 19
5.2 World Models for Autonomous Driving
The discussion describes autonomous-driving world models as immature relative to human drivers, with persistent controllability, 3D consistency, and data challenges. It also outlines future directions for driving simulators and autonomous agents, including environment understanding, task generalization, and real-world application.
5.2 World Models for Autonomous Driving
- Autonomous-driving world models remain rudimentary compared with a skilled human driver’s mental world model, despite the survey’s expectation that they will underpin future driving systems.The discussion identifies action controllability, 3D consistency, and data limitations as persistent challenges.Source: §5.2 · p. 19
5.2.1 Challenges
- Driving models achieve only coarse action control, and generation degrades for unconventional maneuvers such as high-speed turns or U-turns.The survey connects this limitation to normal-data distributions and the difficulty of learning continuous action representations from limited samples.Source: §5.2.1 · p. 19
- Video generation can appear realistic without 3D consistency; point-cloud or occupancy-grid conditioning is proposed to improve it, while scarce, non-diverse driving data remains a separate constraint.The survey cites fixed sensor positions, privacy and commercial competition, and long-tail data collection as contributors to data limitations.Source: §5.2.1 · p. 20
5.2.2 Future Perspectives
- Future driving models are expected to support closed-loop decision training and realistic simulators with scene-layout, lighting, and object control.The survey says current driving world models remain far from the end-to-end foundation-model goal and proposes integrating them with prior 3D simulation methods.Source: §5.2.2 · p. 20
5.3.1 Challenges
- Agents still need better environment understanding and generalization to unfamiliar tasks; the survey points to multimodal perception, internet data, and common-sense abstraction as directions.It notes that many current robots remain specialized for particular functions and argues that big-data learning alone is inefficient and poorly generalizable.Source: §5.3.1 · pp. 20–21
5.3.2 Future Perspectives
- Future agent research should explore world-model integration with large language models and test Dreamer-like methods on physical robots, where observation errors and control precision add uncertainty.The survey describes real-world robotic application of these methods as largely unexplored.Source: §5.3.2 · p. 21
5.4 Ethical and Safety Concerns
World models raise safety and ethical concerns involving prediction reliability and fairness, hyper-realistic misinformation, and privacy in training data.
5.4 Ethical and Safety Concerns
- Reliable, fair predictions are critical for decision-making, particularly autonomous driving, where prediction reliability is essential for safety.The survey calls for accountability measures to validate accuracy and fairness and address potential biases.Source: §5.4 · p. 21
- Hyper-realistic generated videos can spread emotionally manipulative misinformation, distort public opinion, and make reality harder to distinguish from falsehood.The passage highlights fabricated videos of politicians and misinformation reaching education.Source: §5.4 · p. 21
- Video-generation models raise privacy concerns because training videos may contain personal data, making disclosure of their sources important.The survey also calls for laws and policies clarifying privacy protections and requirements.Source: §5.4 · p. 21
6 CONCLUSION
The survey reviews general world models and their applications, highlighting Sora's simulation capabilities and nascent understanding of physical principles. It also assesses current challenges and limitations while considering future development and application.
6 CONCLUSION
- The survey reviews world models' current innovations and applications in video generation, autonomous driving, and autonomous agents.It frames these areas within a broader review of world models' importance to AGI and applications from virtual environments to decision-making systems.Source: §6 · p. 21
- Sora is highlighted for its simulation capabilities and nascent understanding of physical principles as a significant milestone in world models.Source: §6 · p. 21
- The survey evaluates current methodologies' challenges and limitations, including complexity, ethical considerations, and scalability, and considers future development.It aims to inspire novel solutions for world models and their applications in AGI.Source: §6 · p. 21