Source-linked AI summary
DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, Jiwen Lu
TL;DR
Existing autonomous-driving world models have largely emphasized gaming or simulated environments, leaving real-world driving scenarios underrepresented. DriveDreamer constructs a real-world-driven world model with a diffusion-based environment representation and two-stage training, and experiments show controllable traffic-consistent video generation plus reasonable driving-policy generation.
Problem
Prior world-model research predominantly focuses on gaming or simulated environments, limiting representation of real-world driving scenarios.
Method
DriveDreamer uses the Autonomous-driving Diffusion Model and two-stage training to learn structured traffic constraints before predicting future driving states.
Results
DriveDreamer controllably generates diverse driving videos aligned with traffic constraints and achieves an average L2 trajectory error of 0.29m in open-loop nuScenes evaluation.
Takeaways & Limitations
DriveDreamer extends world modeling to real-world driving scenarios while supporting video generation, future action prediction, and interaction with the learned driving world.
Abstract
from arXiv · showhide
World models, especially in autonomous driving, are trending and drawing extensive attention due to their capacity for comprehending driving environments. The established world model holds immense potential for the generation of high-quality driving videos, and driving policies for safe maneuvering. However, a critical limitation in relevant research lies in its predominant focus on gaming environments or simulated settings, thereby lacking the representation of real-world driving scenarios. Therefore, we introduce DriveDreamer, a pioneering world model entirely derived from real-world driving scenarios. Regarding that modeling the world in intricate driving scenes entails an overwhelming search space, we propose harnessing the powerful diffusion model to construct a comprehensive representation of the complex environment. Furthermore, we introduce a two-stage training pipeline. In the initial phase, DriveDreamer acquires a deep understanding of structured traffic constraints, while the subsequent stage equips it with the ability to anticipate future states. The proposed DriveDreamer is the first world model established from real-world driving scenarios. We instantiate DriveDreamer on the challenging nuScenes benchmark, and extensive experiments verify that DriveDreamer empowers precise, controllable video generation that faithfully captures the structural constraints of real-world traffic scenarios. Additionally, DriveDreamer enables the generation of realistic and reasonable driving policies, opening avenues for interaction and practical applications.
1. Introduction
DriveDreamer addresses the limited representation of real-world driving scenarios in prior world-model research by building a world model from real driving videos and driver behaviors. Its diffusion-based Auto-DM and two-stage training pipeline model traffic structure, future states, and controllable driving behavior.
- World models can generate diverse driving videos and support predictive, end-to-end autonomous-driving capabilities.They can also provide data for training driving perception approaches, including long-tail scenarios.
- Prior research largely focused on gaming or simulated environments, leaving real-world driving scenarios insufficiently represented.
- DriveDreamer pioneers a world model constructed from real-world driving videos and human driver behaviors.
- Auto-DM uses traffic structural information as intermediate conditions, improving sampling efficiency and comprehension of dynamic and static scene elements.The model is trained within a two-stage pipeline designed for complex real-world driving scenes.
- The two stages first learn traffic structural information and then train video prediction, enabling controllable videos and future driving-policy generation.The paper reports promising open-loop planning results on nuScenes.
2. Related Work
Related work spans diffusion-based generation, video prediction, and world models for model-based imitation learning. DriveDreamer builds on these directions while targeting real-world driving with structured traffic conditions, text prompts, and driving actions.
- Diffusion models generate samples by learning to reverse a progressively added noise process and support controllable generation from inputs such as depth, segmentation, and layout.
- Video prediction anticipates future visual changes from historical observations, using architectures including VAEs, autoregressive models, flow-based models, GANs, and diffusion models.
- Existing driving video-prediction work links driving actions to pixels, whereas DriveDreamer combines structured traffic conditions, text prompts, and driving actions.
- World models learn environment dynamics and rendering functionality to make future states predictable for agents, often using VAEs and LSTMs.
- Prior world-model advances include disentangled visual dynamics, BEV semantic segmentation, and reinforcement-learning-based BEV representations, but relevant research has predominantly used gaming or simulated environments.
3. DriveDreamer
DriveDreamer combines ActionFormer and Auto-DM to predict future traffic structure and generate controllable driving videos from real-world scene inputs. Its two-stage training first learns structured traffic constraints and video generation, then enables future-state prediction and environmental interaction.
- Overall Framework: DriveDreamer predicts future road structural features with ActionFormer and supplies them as conditions to Auto-DM for future driving-video generation.The framework starts from a reference frame, HDMap, and 3D boxes; text prompts can adjust scenario style.
- First-stage Training: The first training stage progresses from single-frame structured-condition generation to multi-frame video generation, building understanding of traffic structure and motion transitions.Step 1 uses single-frame image supervision, while Step 2 uses multiple structured frames supervised by driving videos and adds temporal attention.
- Auto-DM: Auto-DM incorporates structured traffic information as conditional inputs to reduce the difficulty of modeling real-world driving scenes in pixel space.Spatially aligned conditions are integrated with visual features, while 3D-box position conditions are embedded and fused through gated self-attention.
- Auto-DM: Auto-DM uses text cross-attention to influence scene attributes and temporal attention to improve coherence across generated video frames.Its diffusion process predicts added noise and optimizes the model using a noise-prediction loss.
- Second-stage Training: The second stage uses video prediction to establish future-state prediction and interaction, with DriveDreamer acquiring structural-traffic understanding, future-state prediction, and world-model interaction.ActionFormer addresses unavailable future traffic structure by iteratively predicting future structural conditions from driving actions.
4. Experiment
Experiments on nuScenes show that DriveDreamer generates structurally consistent, controllable driving imagery and videos, while supporting synthetic-data augmentation and driving-action prediction.
- Dataset and setup: 700 training and 150 validation videos from nuScenes provide approximately 1M frames from six surround-view cameras at 12Hz.HDMap annotations are obtained for first-stage training, while ego-car yaw and velocity provide second-stage action inputs.
- Controllable generation: DriveDreamer generates images and videos aligned with structured traffic conditions and can vary weather and time through text prompts.The generated images also exhibit multi-view consistency with the supplied HDMaps and 3D boxes.
- Synthetic data augmentation: 0.7 and 3.0 mAP improvements for FCOS3D and BEVFusion, respectively, result when synthetic data is added to training without synthetic data.The comparison concerns 3D detection performance using generated images alongside real images.
- Controllable generation: DriveDreamer generates distinct videos under different driving actions, including left and right turns, enabling control of the vehicle trajectory.Action-conditioned generation is evaluated against DriveGAN and a concatenation baseline on nuScenes validation.
- Generation quality: After first-stage training, DriveDreamer produces higher-quality action-conditioned videos, while ActionFormer improves quality by iteratively updating future structured information.Without first-stage training, DriveDreamer already achieves superior FID and FVD scores to DriveGAN.
- Driving action generation: 0.29m average L2 trajectory error and a 21% relative reduction in reported average collision rate demonstrate reasonable future-action prediction in open-loop nuScenes evaluation.DriveDreamer surpasses the multi-modality method VAD on average L2 trajectory error.
5. Discussion and Conclusion
DriveDreamer extends world modeling to real-world autonomous-driving scenarios by using diffusion modeling to represent complex environments and generate videos and driving policies.
- Discussion and Conclusion: DriveDreamer focuses on real-world driving scenarios rather than the gaming or simulated environments emphasized by prior world-model research.The conclusion frames real-world representation as important for accurate modeling and decision-making in autonomous driving.
- Discussion and Conclusion: The model demonstrates complex-environment comprehension, high-quality driving-video generation, and realistic driving-policy formulation.These capabilities are presented as the basis for extending world modeling in autonomous driving.
6. Implementation Details
DriveDreamer uses specialized condition encoders and attention mechanisms to support structured, multi-view video generation, action prediction, and synthetic-data augmentation for 3D detection.
- Condition encoders: DriveDreamer embeds reference images, HDMaps, 3D boxes, and driving actions with dedicated condition encoders.Spatially aligned inputs use convolutional downsampling to match diffusion-noise dimensions.
- Multi-view generation: Multi-view video generation stacks view-wise and frame-wise attention layers to produce view-consistent and frame-consistent videos.View-wise attention links adjacent views, while frame-wise attention processes temporal relationships.
- Action prediction: Action prediction concatenates pooled multi-scale UNet features with encoded driving-action features before MLP layers learn future actions.The driving-action features represent velocity and yaw angle; trajectory evaluation also uses commands, acceleration, and past trajectories.
- Synthetic data training: DriveDreamer generates 768 × 448 high-fidelity images that are resized for training off-the-shelf 3D detectors.The synthetic images are produced from nuScenes data and used to augment 3D detection training.
7. Visualizations
DriveDreamer generates diverse driving videos conditioned on structured traffic information, text prompts, and driving actions, while also predicting plausible future actions in complex scenarios.
- Structured and text-conditioned generation: DriveDreamer generates diverse videos adhering to HDMaps and 3D boxes while text prompts vary weather and time of day.The visualizations highlight controllable changes in driving-scene style under fixed structural traffic conditions.
- Action-conditioned generation: Different driving actions, including left and right turns, produce distinct future driving videos from an initial frame and structural information.The action-conditioned examples demonstrate interaction with the driving scene.
- Future action prediction: Predicted future driving actions align with real-world scenarios and remain sensible at intersections, traffic lights, and turns.The predictions are compared with corresponding ground-truth driving videos.