World Models for Autonomous Driving: An Initial Survey

• Current machine-learning systems struggle with patterns people resolve easily, while human decisions draw on perception, memory, and direct observation –.Source: §I · p. 1
• World-model ideas draw on control theory and mental models –, while neural networks and RNNs later enabled nonlinear dynamics and temporal prediction –[17].Source: §I · p. 1
• Pearl’s hierarchy moves from association (SEEING), through intervention (DOING), to counterfactual causal reasoning (IMAGINING), an arduous goal for learnable models,.Source: §I · p. 2
• Ha and Schmidhuber’s 2018 formulation used MDNs and RNNs to extract spatial and temporal environmental patterns and predict future scenarios [31],.Source: §I · p. 2
• In driving, data scarcity—particularly for BEV labeling—motivates generating scenarios from historical data and training systems in simulated environments [31], [33], [34].Source: §I · p. 2
• The survey maps world models’ foundations, methodological advances, practical applications, challenges, and future research directions for autonomous driving.Source: §I · p. 2

II. DEVELOPMENT OF WORLD MODELS

• Perception encodes environmental inputs, memory retains and replays experience, control selects actions, and the world model estimates missing state information and predicts future states.The control module uses model predictions to choose actions toward goals such as minimizing costs or maximizing rewards.Source: A. Architectural Foundations of World… · p. 3

A. Architectural Foundations of World Models

• Latent dynamical models compact high-dimensional observations into states that support parallel forward predictions and representation of uncertain possible futures.The section gives ambiguity about a car’s direction at an intersection as an example of real-world uncertainty.Source: A. Architectural Foundations of World… · pp. 3–4
• Design strategies include temperature-based uncertainty, Top-k sampling, and transformer architectures to balance prediction precision and flexibility.The section warns that highest-probability predictions can become repetitive in long-term forecasts, while excessive randomness can produce implausible futures.Source: A. Architectural Foundations of World… · p. 4
• RSSM combines stochastic and deterministic state components: deterministic-only dynamics limit future diversity, while stochastic-only dynamics challenge information retention.The hybrid aims to balance predictive stability with adaptive capacity.Source: A. Architectural Foundations of World… · p. 4, Figure 3 · p. 4
• RSSM models observation and state transitions over observation-action sequences, using a shared GRU to encode prior states and actions as a deterministic state.Its training objective maximizes the evidence lower bound (ELBO).Source: A. Architectural Foundations of World… · pp. 4–5
• JEPA predicts in representation space using dual encoders for inputs and targets and a latent variable for prediction.It emphasizes abstract outcomes rather than direct, detailed predictions.Source: A. Architectural Foundations of World… · p. 5, Figure 4 · p. 5
• JEPA’s energy measures the distance between target and predicted representations, while a latent-magnitude penalty controls model complexity.The text also describes higher-order optimization and a variational approximation as possible training approaches.Source: A. Architectural Foundations of World… · p. 5

B. Broad Spectrum Applications

• Four of the top five Atari 100k positions belong to world models; EfficientZero reached human-comparable play after two hours, while DreamerV3 autonomously mined Minecraft diamonds.The review attributes EfficientZero's result to improved sampling efficiency and DreamerV3's to symlog predictions.Source: B. Broad Spectrum Applications · p. 6
• I-JEPA learns semantic image representations without handcrafted augmentations, while A-JEPA achieved state-of-the-art results on multiple audio and speech classification tasks.A-JEPA outperformed models using externally supervised pre-training.Source: B. Broad Spectrum Applications · p. 6
• LEXA outperformed previous unsupervised methods across 40 robot tasks through joint explorer-achiever training in imagination, while L3P improved learning speed and test-time generalization in three manipulation environments.Pathdreamer improved navigation success using 3D point clouds, and SafeDreamer demonstrated the feasibility of high-performance, low-cost safety applications.Source: B. Broad Spectrum Applications · p. 6
• DayDreamer demonstrated rapid real-world robot learning, contrasting with traditional methods and highlighting world models' potential to accelerate learning.The passage describes this as an example of rapid training capabilities.Source: B. Broad Spectrum Applications · p. 6
• SORA generates coherent high-definition videos and maintains 3D spatial consistency, whereas Genie enables interactive environment generation and user-directed world manipulation.The review notes that SORA faces challenges in physical-interaction simulation and that Genie’s video quality is less advanced.Source: B. Broad Spectrum Applications · p. 6

III. WORLD MODELS IN AUTONOMOUS DRIVING

• Autonomous-driving applications of world models remain nascent and are concentrated mainly on scenario generation and planning and control.The review describes these areas as ripe for further exploration and innovation.Source: §III · p. 6
• The chronology runs from 2018's MDN/RNN world model to 2024 systems using RSSM, transformers, diffusion, JEPA, and parallelizable state-space models.The listed models include the 2018 World Models entry and 2024 WorldDreamer, S4WM, V-JEPA, and others.Source: §III · p. 7
• Recent entries span controllable driving-video generation, multi-view end-to-end modeling, multimodal motion prediction, and world models designed for safe reinforcement learning.The review's application table lists DriveDreamer, Drive-WM, TrafficBots, and Safe DreamerV3 among these examples.Source: §III · p. 7

A. Driving Scenario Generation

• High data-collection and annotation costs, legal constraints, and safety considerations complicate autonomous-driving data acquisition; scenario generation can enrich training data with rare and intricate situations.The review describes self-supervised learning on unlabeled data as a cost-effective way to extract useful information.Source: A. Driving Scenario Generation · p. 8
• GAIA-1 [66] generates diverse future driving videos from video, text, and action inputs, including prompted scenarios beyond its training set that the review associates with counterfactual reasoning.It uses an autoregressive transformer to predict image tokens and can generate scenarios involving changed weather, traffic, or vehicle actions.Source: A. Driving Scenario Generation · p. 8
• DriveDreamer [64] uses nuScenes data, HD maps, and 3D boxes to control scenario generation more precisely and generate future actions alongside predictive scenarios.The review says these inputs support deeper understanding and improved video-generation quality.Source: A. Driving Scenario Generation · p. 9
• ADriver-I combines an MLLM that autoregressively produces control signals with a video latent diffusion model that predicts subsequent video outputs.Repeated prediction cycles are described as enabling infinite driving in the predictive world.Source: A. Driving Scenario Generation · p. 9
• WorldDreamer [79] treats world modeling as unsupervised visual-sequence modeling and uses local spatiotemporal patch attention to learn dynamic visual signals and accelerate convergence.Although it is a general-purpose video-generation model, the review reports strong autonomous-driving video generation performance.Source: A. Driving Scenario Generation · p. 9
• MUVO [76] predicts videos, point clouds, and 3D occupancy grids from LiDAR and visual inputs; OccWorld [78] and Think2Drive [83] use occupancy information to model surroundings and plan actions.The review describes these developments as a shift toward multimodal world models for scenario generation.Source: A. Driving Scenario Generation · p. 9

B. Planning and Control

• MILE raised CARLA Driving Score from 46 to 61 in inexperienced test scenarios, compared with an expert data score of 88.The passage reports this as a significant improvement over state-of-the-art models.Source: B. Planning and Control · p. 9
• MILE jointly learns driving behavior and dynamics from offline CARLA datasets, using imagined future environments to compensate for missing perception and plan without high-definition maps.The review describes its future predictions as long-temporal and highly diversified; decoding predicted future states supports stable driving across scenarios.Source: B. Planning and Control · pp. 9–10
• SEM2 [60] filters task-relevant latent features and balances scenario sampling to address task-irrelevant information and imbalanced training data; it outperformed DreamerV2 after CARLA training and testing.Its sampler evenly adds examples from different scenarios in each training batch to support generalization and corner-case handling.Source: B. Planning and Control · p. 10
• Drive-WM [68] jointly generates multi-view frames and predicts intermediate views, while UniWorld [70] pretrains on unlabeled image-LiDAR pairs and improves nuScenes IoU for motion prediction and semantic scene completion.Drive-WM selects trajectories by sampling candidates and scoring them with an image-based reward function.Source: B. Planning and Control · p. 10
• TrafficBots [69] predicts individual agents' actions from a BEV perspective, learning agent-specific personalities conditioned on destinations; it runs faster and scales to more agents, though it may trail state-of-the-art open-loop strategies.The review presents it as evidence of the potential of closed-loop action-prediction strategies.Source: B. Planning and Control · p. 10

C. Conclusion

• World models in autonomous driving span scenario generation and control, with varied tasks and input-output mechanisms in this nascent area.Scenario generation includes predicted videos, information completion, and 3D occupancy prediction; control includes sensor-based driving and prompt-based vehicle control.Source: C. Conclusion · p. 10
• Lower FID and FVD scores indicate higher similarity to nuScenes scenes, and the comparison reports better video quality for Adriver-I than DriveWM and DriveDreamer.The text identifies FID and FVD as video-quality metrics for scenario generation using nuScenes.Source: C. Conclusion · p. 10
• Direct horizontal comparison is often infeasible across other world models because tasks and performance metrics differ, although they report state-of-the-art results against traditional methods on their respective tasks.Source: C. Conclusion · p. 10

IV. CHALLENGES AND FUTURE PERSPECTIVES

• The paper frames autonomous driving as a frontier of innovation with potential to redefine vehicular mobility.Source: §IV · p. 10
• The paper acknowledges that this promising landscape still faces challenges.Source: §IV · p. 10
• Addressing those hurdles and exploring future perspectives requires attention to technical complexities and broader societal implications.Source: §IV · p. 10

Long-Term Scalable Memory Integration

• Long-term memory remains constrained by vanishing gradients, catastrophic forgetting, and the scalability and speed of transformers on lengthy sequences.TRANSDREAMER and S4WM explore alternative architectures; future directions include larger networks, external memory, and iterative learning.Source: A. Technical and Computational… · pp. 10–11
• 500 steps: S4WM maintains high-quality generation over sequences up to this length, surpassing traditional architectures; performance declines beyond 1,000 steps.Source: A. Technical and Computational… · p. 11

Simulation-to-Real-World Generalization

• Simulation-to-real-world generalization is limited by mismatches in physical properties, sensor noise, and unforeseen events; proposed directions include more accurate simulation, robust models, sensory fusion, and learning from diverse data.Source: A. Technical and Computational… · p. 11

Theory and Hardware Breakthroughs

• World models currently perform better on generative than pure predictive tasks such as motion prediction, while restoring lost multimodal detail challenges memory and computation.The paper calls for further theoretical and structural refinement, lossless multimodal collection and processing, and stronger hardware computation.Source: A. Technical and Computational… · p. 11

Decision-Making Accountability

• Autonomous decision-making raises accountability concerns, and the paper calls for explainable AI principles in world-model development to make vehicle decisions interpretable.It describes explanation of ethical, logical, and practical considerations as part of this effort.Source: B. Ethical and Safety Challenges · p. 11

Privacy and Data Integrity

• Extensive driving datasets raise privacy and data-security concerns because collected data can include passenger information, origins, destinations, travel times, and routes.Source: B. Ethical and Safety Challenges · p. 11
• The proposed data response includes governance policies, data minimization, cybersecurity measures, transparency, and user controls over personal information.Examples include encryption, secure storage, audits, and options for access, correction, and deletion.Source: B. Ethical and Safety Challenges · p. 12

Responsibility and Criterion

• As world models support or take over driving tasks, responsibility is redistributed across involved people and organizations, prompting calls for new research, policies, and clearer liability rules.The paper also proposes ethics-design research and legal rules that balance culpability gaps with corrective justice.Source: B. Ethical and Safety Challenges · p. 12

a) Bridging Human Intuition and AI Precision

• World models could support cognitive co-piloting by combining human drivers’ intuitive decisions with AI precision and interpreting social cues, predicting pedestrian behavior, and adapting to passenger feedback.The cited examples include navigating four-way-stop conventions and adapting driving style to passenger comfort.Source: C. Future Perspectives · p. 12

b) Harmonizing Vehicles with the Urban Ecosystem

• World models could help vehicles optimize routes and driving patterns for efficiency, safety, and environmental impact, including emissions, congestion, and energy conservation.Source: C. Future Perspectives · p. 12

• Over 60% of respondents expect urban system automation to reduce transportation pollution and vehicle collisions, while over 70% anticipate less traffic noise and over 80% expect better roadworks and transport management.The survey reflects respondents’ expectations about urban automation; the passage presents world models as a direction that could facilitate greater vehicle and traffic-system automation.Source: C. Future Perspectives · p. 12

• The survey highlights world models’ role in advancing vehicle autonomy through enhanced prediction, simulation, and decision-making.Source: §V · p. 12
• Long-term memory integration, simulation-to-real-world generalization, and ethical considerations remain challenges for deploying world models in real-world applications.Source: §V · p. 12
• Addressing these challenges calls for a multidisciplinary approach combining AI research, ethical frameworks, and innovative computational solutions.The conclusion also calls for continued research and collaboration across fields, with attention to societal impacts and safety standards.Source: §V · pp. 12–13