PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of World Models for Autonomous Driving

Tuo Feng, Wenguan Wang, Yi Yang

arXiv:2501.11260v4cs.ROcs.CV

TL;DR

Autonomous driving requires robust handling of multimodal, dynamic, and difficult traffic scenes, while existing surveys leave aspects of world-model organization and planning–prediction interaction insufficiently covered. This paper systematically reviews world models through a taxonomy spanning future-world generation, behavior planning, and interactive prediction, then synthesizes training paradigms, applications, and open challenges. It concludes that multimodal fusion, self-supervised and generative methods, and tightly coupled prediction and planning are advancing robustness and adaptability, while short prediction horizons remain a limitation for rule-based planning.

  • Problem

    Autonomous driving must integrate heterogeneous sensory data and handle dynamic, long-tail traffic conditions, while prior reviews often lack a clear taxonomy covering planning–prediction interaction.

  • Method

    The paper systematically reviews world models for autonomous driving across future prediction, behavior planning, and interactive prediction, while organizing methods, applications, and research directions.

  • Results

    Recent advances unify multimodal sensing, use self-supervised and generative methods for robustness in rare conditions, and tightly couple motion prediction with planning.

  • Takeaways & Limitations

    World models provide a technical roadmap for more adaptable autonomous-driving systems by connecting environmental understanding, prediction, and decision-making.

  • Takeaways & Limitations

    Rule-based planners have an inherently short prediction horizon, limiting their ability to anticipate farther-future outcomes.

Abstract

from arXiv · show

Recent breakthroughs in autonomous driving have been propelled by advances in robust world modeling, fundamentally transforming how vehicles interpret dynamic scenes and execute safe decision-making. World models have emerged as a linchpin technology, offering high-fidelity representations of the driving environment that integrate multi-sensor data, semantic cues, and temporal dynamics. This paper systematically reviews recent advances in world models for autonomous driving, proposing a three-tiered taxonomy: (i) Generation of Future Physical World, covering Image-, BEV-, OG-, and PC-based generation methods that enhance scene evolution modeling through diffusion models and 4D occupancy forecasting; (ii) Behavior Planning for Intelligent Agents, combining rule-driven and learning-based paradigms with cost map optimization and reinforcement learning for trajectory generation in complex traffic conditions; (ii) Interaction between Prediction and Planning, achieving multi-agent collaborative decision-making through latent space diffusion and memory-augmented architectures. The study further analyzes training paradigms, including self-supervised learning, multimodal pretraining, and generative data augmentation, while evaluating world models' performance in scene understanding and motion prediction tasks. Future research must address key challenges in self-supervised representation learning, multimodal fusion, and advanced simulation to advance the practical deployment of world models in complex urban environments. Overall, the comprehensive analysis provides a technical roadmap for harnessing the transformative potential of world models in advancing safe and reliable autonomous driving solutions.

1 Introduction

The introduction motivates world models as a foundation for robust autonomous driving and surveys their taxonomy, methods, applications, evaluation, and research challenges. It emphasizes future-world generation, behavior planning, and prediction–planning interaction as central areas for improving safety and adaptability.

  • Motivation: Autonomous driving must fuse heterogeneous sensors, anticipate motion, and remain responsive and safe under degraded sensing, long-tail scenarios, and aggressive traffic.The introduction identifies heavy rain, poor road markings, construction zones, severe weather, and erratic behavior as representative challenges.
  • Motivation: World models compress multi-sensor observations into latent states and roll them forward under hypothetical actions, allowing vehicles to rehearse possible futures.They provide an internal representation that supports downstream physical-world prediction and planning.
  • Survey Scope: The survey addresses gaps in prior reviews by offering a structured taxonomy that includes future prediction, behavior planning, and interaction between planning and prediction.Earlier surveys often focused on general world models or simulation and lacked a clear treatment of planning–prediction interaction.
  • Survey Taxonomy: Future physical-world prediction is organized around Image-, BEV-, OG-, and PC-based generation, including diffusion-based generation and 4D occupancy forecasting.The review examines how these methods achieve geometric and semantic fidelity.
  • Survey Taxonomy: Behavior planning is reviewed through rule-based and learning-based approaches, with reported improvements in robustness and collision avoidance.The surveyed planning methods produce safe, efficient paths under uncertain driving conditions.
  • Survey Taxonomy: Interactive models jointly address future prediction and agent behavior, linking predictive insights with collaborative optimization and improved real-world adaptability.The survey also identifies self-supervised learning, unified multimodal embeddings, physics-aware simulation, and latency-aware architectures as frontier challenges.

2 Background

World models for autonomous driving encode multimodal observations into latent representations that model scene evolution and ego–environment dynamics. The section frames future-world generation and trajectory planning as coupled core tasks requiring safety, adaptability, and uncertainty handling.

  • Definition: World models encode physical environments into compact latent representations that capture geometry, semantics, and causal context.They learn these representations from sensor observations and use them for prediction and planning.
  • Definition: Self-supervised compression and time-aware prediction let agents roll forward hidden states under actions before acting in the real world.The differentiable pipeline supports trajectory rehearsal and what-if roll-outs.
  • Autonomous-driving variant: Autonomous-driving world models fuse synchronized cameras, LiDAR, radar, and HD maps into a unified latent scene graph.This unification connects perception and prediction within one representation.
  • Core tasks: The formal driving task predicts the next scene and ego trajectory from historical multi-view images and LiDAR points while modeling their coupled dynamics.The formulation accounts for the joint evolution of the environment and ego vehicle.
  • Core tasks: Future-world generation forecasts dynamic entities amid interactions, stochastic behavior, and uncertainty, while planning seeks feasible trajectories under safety and traffic constraints.Planning must also handle dynamic obstacles and real-time adaptability.

3.1 Generation of Future Physical World

Future-physical-world generation is organized into image, BEV, occupancy-grid, and point-cloud tracks, spanning photorealistic frames to 4D representations and LiDAR sweeps. The surveyed literature trends toward controllable, multimodal, temporally coherent, and increasingly efficient world models.

  • Overview: Future-scene generators span image, BEV, occupancy-grid, and point-cloud representations, enabled by controllable diffusion and long-horizon forecasting.These tracks progress from photorealistic frames and map layouts to 4D voxels and future LiDAR sweeps.
  • Image-based Generation: Image-centric frameworks remain limited to 2D outputs lacking spatial-temporal coherence, motivating 4D methods such as DriveDreamer4D.DriveDreamer4D uses world-model priors to generate spatial-temporally coherent 4D driving videos.
  • Image-based Generation: Image-based methods evolve from 2D, task-specific synthesis toward multimodal, multi-view, multi-agent, and long-horizon generation.The Dreamer series illustrates this progression toward general-purpose models and open natural-language conditions.
  • Image-based Generation: Diffusion methods improve controllability, temporal coherence, spatial resolution, and per-frame detail through geometric, trajectory, and multimodal conditioning.Representative conditioning signals include BEV layouts, text, optical flow, ego trajectories, and cross-sensor features.
  • BEV-based Generation: BEV methods progress from probabilistic monocular maps toward object-centric slots, 4D occupancy fields, trajectory-conditioned latents, and self-supervised occupancy learning.Lightweight multiscale designs and editable BEV sketches support more generative and potentially closed-loop models.
  • Synthesis: Across modalities, the field converges on controllable, efficient, multimodal world models and increasingly integrated closed-loop simulation systems.This evolution provides high-fidelity, scalable synthetic environments for autonomous driving.
  • OG-based Generation: Occupancy-grid generation shifts from deterministic voxel fusion toward stochastic tokenization and Gaussian splats, reducing memory cost while adding planning and language-grounded reasoning.Efficiency mechanisms and controllable priors push occupancy generation toward real-time, scalable simulation.
  • PC-based Generation: Point-cloud generation uses diffusion models, GANs, VQ-VAE, and NeRF to address LiDAR sparsity and irregularity while preserving geometric fidelity.Some methods model physical sensor constraints, whereas others use neural rendering for novel-view synthesis in dynamic scenarios.

3.2 Behavior Planning for Intelligent Agents

Behavior planning converts dynamic scene understanding into safe, comfortable, goal-directed trajectories using rule-based, learning-based, and hybrid approaches. The surveyed methods improve adaptability and interaction handling but remain constrained by coverage, interpretability, compute, and safety-certification challenges.

  • Trajectory generation: Planners sample kinematically feasible motions and rank them using semantic predictions, traffic rules, dynamics, and comfort metrics.This produces paths that adapt smoothly to changing conditions while meeting safety requirements.
  • Learning-based planning: Data-driven planners use multimodal sensor streams and learned semantic, interaction, and rule cues to produce adaptable trajectories beyond brittle heuristics.Compared with rule-based planning, they can generalize to new scenes when supplied with sufficiently diverse data.
  • Open challenges: Key limitations include out-of-distribution degradation, opaque latent policies, missing formal safety bounds, and high compute or sampling costs.These issues constrain scaling learning-based planning from research prototypes toward trustworthy deployment.
  • Cost-volume planning: Cost-volume planners explicitly pair trajectory samplers with cost maps and execute the lowest-cost candidate, yielding interpretable safety-oriented plans.Their effectiveness decreases when the sampler cannot cover the dense combinatorial space of urban traffic.
  • Learning-based planning: Learning-based planning spans RL and MPC, LLM-based, and cost-volume architectures that respectively combine latent dynamics, token prediction, or learned occupancy costs.These paradigms incorporate uncertainty, safety, interaction, or interpretable trajectory ranking in different ways.
  • Rule-based planning: Rule-based planners remain easy to audit and can balance update rate, smoothness, and global optimality, but finer grids or larger graphs increase complexity.Hybrid designs combine rapid collision avoidance with higher-level sampling or MPC to preserve broader optimality.

3.3 Interaction between Planning and Prediction

World-model interaction has progressed from static open-loop replay to autoregressive and controllable closed loops that respond to ego actions. Controllable systems add editable world dynamics and support more rigorous, interactive safety validation.

  • Controllable closed-loop regime: Controllable closed-loop systems add editable 4D worlds, occupancy control, and feedback signals so agents can interact with testable simulators.The field has consequently moved from passive log replay toward interactive, high-fidelity simulation.
  • Open-loop regime: Open-loop generators replay logged or pre-sampled futures without responding to online control inputs, breaking the causal link between actions and observations.Open-loop metrics can therefore overstate safety and performance when evaluated actions never influence the future.
  • Uncontrollable closed-loop regime: Autoregressive world models condition multi-sensor future generation on ego actions, unifying prediction and planning in a single generative loop.They provide diverse data, expose planners to downstream consequences, and capture uncertainty more richly than fixed-log corpora.
  • Uncontrollable closed-loop regime: Uncontrollable closed-loop models retain opaque latent physics, preventing users from freely editing rules, injecting rare events, or verifying safety guarantees.Compounding distribution drift further threatens reliability and rigorous safety-critical evaluation.
  • Controllable closed-loop regime: High-fidelity simulators can inject hazards, swap policies, tune metrics, and reveal covariate-shift failures before deployment.However, existing platforms still face limited geographic coverage, imperfect real-world visual fidelity, GPU cost, and regulatory hurdles.
  • Implications: The review recommends complementing or replacing open-loop metrics with closed-loop scores and using generative simulators to synthesize corner cases on demand.It also highlights curricula alternating logged supervision with interactive roll-outs and shared trajectory-policy losses.

4 Data and Training Paradigms

Training paradigms combine self-supervision, multimodal and foundation pretraining, and generative data augmentation to reduce annotation demands and broaden world-model capabilities. The main constraints are fidelity, compute, calibration, temporal forecasting, and safety evaluation.

  • Self-supervised learning: Self-supervised methods learn from images, LiDAR, rendered views, occupancy cues, tokenized multimodal data, or future point-cloud pseudo-labels without human annotation.These approaches reduce labeling costs while linking visual and geometric representations.
  • Pretraining: Large-scale multimodal pretraining produces unified backbones that transfer across detection, tracking, mapping, occupancy, trajectory prediction, and planning.Foundation generative models can scale to hundreds of millions of frames and synthesize controllable multi-camera scenes, but training cost and safety evaluation remain open issues.
  • Self-supervised learning: Self-supervised occupancy models still trail fully supervised 3D/4D baselines in accuracy, incur heavy compute and memory costs, and rarely forecast label-free 4D occupancy.The review calls for faster rendering and tighter planner integration to test whether saved labels translate into safety gains.
  • Pretraining: Hybrid systems are envisioned to combine dense LiDAR self-supervision with the semantic richness and controllability of vision-centric foundation models.Open challenges include scalable temporal memory, principled safety validation, and energy-efficient training.
  • Data generation: Structured BEV, occupancy, and point-cloud generation supplies scene, intention, semantic, and geometry labels while filtering appearance noise.These outputs support training and geometry-heavy tasks such as mapping and motion forecasting.
  • Data generation: Generative pipelines offer coverage, fidelity, and efficiency by synthesizing corner cases, preserving multimodal consistency, and refreshing data without new drives.Diffusion and Transformer models can generate action-conditioned scenes for perception and closed-loop policy training.
  • Open challenges: Synthetic curricula remain limited by GPU-intensive high-resolution diffusion, precise camera-calibration requirements, and unresolved safety assessment.These constraints complicate reliance on purely synthetic training data.

5 Application Areas and Tasks

World models support scene understanding, motion prediction, simulation, and end-to-end driving by maintaining spatiotemporal representations and rolling them forward under actions. Their benefits include richer geometry, longer-horizon forecasting, controllable simulation, and integrated control, alongside substantial compute and reliability challenges.

  • Scene understanding: Scene-understanding models fuse camera and LiDAR streams into continuously refreshed spatiotemporal or BEV representations that support detection, mapping, tracking, and semantics.Unified memory states and BEV lattices reduce reliance on task-specific heads and manual fusion heuristics.
  • Scene understanding: Dense 3D occupancy representations provide planners with free-space priors that remain topologically consistent over seconds rather than frames.Neural volumetric representations can also supply visibility or risk gradients for downstream reasoning.
  • Open challenges: Application-level limitations include GPU and memory demands, uncertain safety certificates, diffusion-rollout uncertainty calibration, token limits, and long-horizon coherence.These issues constrain reliability when imagined futures drift far from training data.
  • Motion prediction: Future-state models roll latent worlds forward with diffusion or 4D occupancy generation, producing probability fields and collision-risk queries for planners.Action-controllable occupancy roll-outs can bypass separate mapping stages, while tokenized video models extend horizons beyond a minute.
  • Motion prediction: Embedding multi-agent personalities into world models yields socially plausible manoeuvres without brittle lane-change heuristics.The unified latent can integrate map topology and social behavior for richer interaction modeling.
  • Simulation: Generative simulators produce controllable multimodal replicas with pixel-level realism, cross-view depth consistency, and dense physical states for verification.Mixing synthetic clips into training corpora has measurably improved BEV detectors, while interactive platforms expose covariate-shift failures before deployment.
  • End-to-end driving: End-to-end world-model policies integrate perception, prediction, and control, generate waypoint plans, evaluate imagined futures, and follow language directives.Reported benefits include lower collision rates, improved rule compliance and comfort, reduced real-world data requirements, and exposed intermediate occupancy representations.

6 Performance Comparison

The evaluation compares world models across 4D scene generation, point-cloud forecasting, occupancy forecasting, and motion planning using established datasets and task-specific metrics. Results indicate improvements from DynamicCity, DFIT-OccWorld, occupancy forecasting models, and richer supervision, while some comparisons depend on privileged information.

  • 4D Scene Generation: DynamicCity consistently achieves higher perceptual quality and stronger fidelity-diversity trade-offs than OccSora across both benchmarks and evaluation spaces.The comparison uses IS, FID, KID, Precision, and Recall on CarlaSC and Occ3D-Waymo in 2D and 3D spaces.
  • Point Cloud Forecasting: DFIT-OccWorld-O reaches a mean Chamfer Distance of 0.70m2, outperforming ViDAR across every forecast horizon.DFIT-OccWorld-V follows at 0.76m2, with the performance gap widening as lead time grows.
  • 4D Occupancy Forecasting: I2-World-O and T3Former-O attain state-of-the-art 4D occupancy forecasting results on Occ3D-nuScenes for 1s, 2s, and 3s horizons.The camera-only T3Former-F variant also remains highly competitive, indicating that occupancy forecasting can perform strongly without LiDAR input.
  • Motion Planning: T3Former-O achieves 1.00m average L2 and 0.30 average collision, reducing 3s collisions by approximately 62% in the 3D-occupancy regime.FSDrive reports 0.28m/0.10% but relies on privileged ego-status supervision, limiting direct comparability with fully perception-driven planners.

7 Future Research Directions

Future research directions focus on reducing label dependence, strengthening multimodal and physics-aware world models, improving simulation realism, and enabling efficient deployment. The proposed trajectory combines self-supervision, universal sensor representations, interactive simulation, diffusion generation, and resource-aware architectures.

  • Overview: The survey identifies self-supervised learning, multimodal fusion, advanced simulation, and efficient architectures as four frontier directions.These directions aim to reduce label dependence, improve perception and planning, increase simulation realism, and support resource-aware deployment.
  • Reducing Label Dependency: Cross-modal reconstruction and physics-based consistency losses are proposed to refine depth, motion, and semantics without hand-drawn boxes.Generative objectives and lightweight distillation are intended to reduce annotation budgets while keeping models suitable for automotive hardware.
  • Exploring Unlabeled Data Potential: Universal embeddings are envisioned to ingest cameras, LiDAR, radar, event, and thermal sensors without hand-tuned adapters.The proposed direction aims to support a continuously refreshed memory for control, lifelong adaptation, and robustness under sensor degradation.
  • Advanced Simulation: Physics-aware and photorealistic simulators are proposed to expose corner-case failures, support auto-curricula, and accelerate validation before road tests.The surveyed directions combine differentiable physics, multimodal rendering, fleet telemetry, and scalable scenario generation.
  • Diffusion-based Generation: Diffusion world models are progressing toward unified, interactive generators spanning volumetric 4D roll-outs, richer conditioning, efficient latents, and editable scenarios.The survey connects these capabilities with self-improving simulators intended to shorten validation cycles and reduce data costs.
  • Efficient Architectures: Lean backbones that fuse sensing, prediction, and control are intended to respect power, bandwidth, and privacy limits while sustaining real-time performance.Capacity-elastic layers and shared feature representations are proposed for modest vehicle GPUs.

8 Conclusion

The survey presents world models as increasingly central to autonomous driving because they integrate multimodal sensing, self-supervised learning, generative modeling, and coupled prediction-planning. These advances support synthetic long-tail scenarios and more adaptive navigation, although reliable large-scale deployment remains the broader objective.

  • Conclusion: World models increasingly integrate perception, prediction, and decision-making for autonomous driving.Recent systems combine information from cameras, LiDAR, and other sensors while reducing dependence on annotated datasets.
  • Conclusion: Diffusion-based generative methods facilitate diverse synthetic data for long-tail scenarios and enhance robustness in rare or extreme conditions.The conclusion links generative modeling with improved coverage of uncommon driving situations.
  • Conclusion: New frameworks tightly couple motion prediction with planning, moving toward closed-loop paradigms for safer and more adaptive navigation.The stated direction connects prediction and planning rather than treating them as isolated components.
  • Conclusion: World models are positioned to become more integral to reliable, large-scale deployment as sensing technologies evolve and cross-domain datasets proliferate.This is the paper's concluding deployment outlook rather than a reported benchmark result.
Loading 2501.11260v4…