Source-linked AI summary
DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, Xingang Wang
TL;DR
Customized driving-video generation remains challenging despite advances in autonomous-driving world models. DriveDreamer-2 translates user queries into trajectories, generates regulation-adhering HDMaps, and uses UniMVM for coherent multi-view video generation. It generates uncommon customized scenarios and reports FID 11.2 and FVD 55.7, with relative improvements of approximately 30% and 50%.
Problem
Existing autonomous-driving world models face challenges in generating customized driving videos from user-friendly inputs.
Method
DriveDreamer-2 finetunes an LLM to convert user prompts into agent trajectories, conditions an HDMap generator on them, and uses UniMVM for video generation.
Results
FID 11.2 and FVD 55.7 represent relative improvements of approximately 30% and 50% over previous best-performing methods.
Takeaways & Limitations
DriveDreamer-2 generates diverse user-customized videos, including abrupt vehicle cut-ins, and its generated videos enhance training for detection and tracking.
Abstract
from arXiv · showhide
World models have demonstrated superiority in autonomous driving, particularly in the generation of multi-view driving videos. However, significant challenges still exist in generating customized driving videos. In this paper, we propose DriveDreamer-2, which builds upon the framework of DriveDreamer and incorporates a Large Language Model (LLM) to generate user-defined driving videos. Specifically, an LLM interface is initially incorporated to convert a user's query into agent trajectories. Subsequently, a HDMap, adhering to traffic regulations, is generated based on the trajectories. Ultimately, we propose the Unified Multi-View Model to enhance temporal and spatial coherence in the generated driving videos. DriveDreamer-2 is the first world model to generate customized driving videos, it can generate uncommon driving videos (e.g., vehicles abruptly cut in) in a user-friendly manner. Besides, experimental results demonstrate that the generated videos enhance the training of driving perception methods (e.g., 3D detection and tracking). Furthermore, video generation quality of DriveDreamer-2 surpasses other state-of-the-art methods, showcasing FID and FVD scores of 11.2 and 55.7, representing relative improvements of 30% and 50%.
1 Introduction
DriveDreamer-2 addresses customized driving-video generation by translating user text into traffic conditions and generating coherent multi-view videos. It produces diverse scenarios, improves video-generation metrics, and supports perception training.
- DriveDreamer-2 is presented as the first world model to generate diverse driving videos in a user-friendly manner.
- User-friendly text prompts are converted into foreground agent trajectories and background HDMaps for customized traffic simulation.The traffic simulation separates foreground trajectories from background lane boundaries, lane dividers, and pedestrian crossings.
- An LLM is finetuned with a trajectory-generation function library to transform user inputs into diverse traffic scenarios.
- FID and FVD improve by ∼30% and ∼50%, respectively, compared with previous best-performing methods, while generated videos enhance perception-method training.Reported perception improvements are ∼4% for detection and ∼8% for tracking.
- UniMVM integrates intra-view and cross-view spatial consistency to improve temporal and spatial coherence in generated driving videos.
2 Related Works
Related work spans autonomous-driving world models, video-generation methods, and language-guided traffic simulation. DriveDreamer-2 combines these directions through a finetuned LLM and a functional library for user-friendly text-to-traffic simulation.
- Autonomous-driving world models address complex dynamic environments using approaches ranging from BEV semantic modeling to diffusion and language-model integrations.
- Video-generation research has progressed from VAEs, flow-based models, and GANs toward diffusion models and language-based visual-dynamics modeling.
- Traffic simulation: LCTGen requires detailed descriptions such as agent speed and orientation, while CTG and CTG++ use manually designed or GPT-4-generated loss functions.
- Traffic simulation: DriveDreamer-2 finetunes an LLM with a functional library to enable user-friendly text-to-traffic simulation without intricate loss design or complex prompts.
3 DriveDreamer-2
DriveDreamer-2 combines user-prompt trajectory generation, trajectory-conditioned HDMap synthesis, and unified multi-view video generation. Its design supports diverse traffic scenarios while maintaining traffic constraints and multi-view coherence.
- Customized Traffic Simulation: A finetuned LLM translates user prompts into customized agent trajectories through a trajectory-generation function library.The library is built from Text-to-Python-Script pairs and supports maneuvers including cut-ins and U-turns.
- Customized Traffic Simulation: The HDMap generator uses generated trajectories as conditions to synthesize background road structures that satisfy traffic constraints.It models trajectory-to-HDMap generation with a diffusion-based conditional image generator.
- Customized Traffic Simulation: The generated BEV HDMaps represent lane boundaries, lane dividers, and pedestrian crossings, which are post-processed into image-coordinate conditions.Binarization and skeleton extraction convert map elements into pixel-coordinate controls for UniMVM.
- UniMVM: UniMVM is designed to improve temporal and spatial coherence without compromising generation speed and diversity.Unlike sequential adjacent-view expansion, it avoids independent multi-view generation and does not introduce cross-view parameters.
- UniMVM: UniMVM unifies multiple views into a complete patch and uses masking to support several conditioned and unconditioned video-generation tasks.Masking can enable first-frame future prediction, front-view-conditioned generation, or generation without image input.
4 Experiment
Experiments evaluate DriveDreamer-2 on customized video generation, generation quality, downstream perception augmentation, and component ablations. The results include user-defined scenarios, multi-view consistency gains from UniMVM, and comparisons on nuScenes.
- Experiment Details: 700 training videos and 150 validation videos from nuScenes provide approximately 1 million frames captured by six surround-view cameras.Each video contains approximately 20 seconds of footage at 12Hz.
- User-Customized Driving Video Generation: DriveDreamer-2 generates customized videos in which the ego car changes lanes or an unexpected pedestrian crosses at night.The generated videos are conditioned on user text prompts describing the desired driving scenario.
- Quality Evaluation of Generated Videos: The generation-quality comparison uses nuScenes validation videos under no-image, video, and first-frame multi-view image conditions.The evaluation compares DriveDreamer-2 with several driving video generation approaches.
- Downstream Tasks: Synthetic videos improve downstream 3D object detection and multi-object tracking when generated videos are combined with real training videos.The experiments use structured conditions from nuScenes and retrain StreamPETR for downstream tasks.
- Ablation Studies: UniMVM produces higher multi-view consistency than generation without UniMVM.The ablation compares generations with and without the proposed unified multi-view model.
- Ablation Studies: SVD achieves 17.2 FID and 94.6 FVD in the backbone ablation, while its introduction yields an almost 70% improvement in FVD.The authors attribute a slight FID decrease to the cross-view module potentially disrupting SVD’s prior.
5 Discussion and Conclusion
DriveDreamer-2 extends DriveDreamer with an LLM, an HDMap generator, and UniMVM to produce user-customized, temporally and spatially coherent driving videos. It generates uncommon scenarios, supports perception training, and achieves FID 11.2 and FVD 55.7.
- Conclusion: DriveDreamer-2 converts user queries into agent trajectories, generates HDMaps from those trajectories, and uses UniMVM to improve temporal and spatial coherence.The structured conditions are then used for video generation.
- Conclusion: DriveDreamer-2 generates uncommon driving videos and improves training for driving perception methods.The supported examples include abrupt vehicle maneuvers, while the downstream applications include detection and tracking.
- Conclusion: FID 11.2 and FVD 55.7 represent approximately 30% and 50% relative improvements over previous best-performing methods.These scores are reported for multi-view driving video generation.
6 Implement Details
Implementation details cover trajectory-function generation, HDMap projection into image coordinates, diffusion-based UniMVM training, downstream-task augmentation, and long-video generation settings.
- Function Library: The cut-in trajectory function initializes an agent position, checks distances to other agents, and perturbs speed and orientation to avoid collisions and increase diversity.The function library is designed to expand to additional user-requested scenes.
- Function Library: The trajectory generator returns cutting-in trajectories and velocities, with randomized starting positions, speeds, and orientations.The implementation includes separate handling for ego and non-ego agents.
- BEV HDMap Post-Process: BEV HDMaps are binarized, skeletonized, and projected into pixel coordinates before serving as UniMVM conditions.Lane boundaries, lane dividers, and pedestrian crossings are extracted as distinct map features.
- Training UniMVM: UniMVM training uses diffusion denoising conditioned on HDMap features, box conditions, and text-prompt embeddings.The denoiser is trained with denoising score matching under EDM preconditioning.
- Evaluation Details: FID and FVD evaluation uses 150 nuScenes validation videos at 4Hz with 8 frames per clip.The official UCF FVD evaluation code is used.
7 Visualization
Visualization shows consistent multi-view and temporal behavior across occlusion, weather, pedestrian crossing, lane changing, and vehicle cut-in scenarios. Long videos are generated by chaining clips through successive frame conditioning.
- Consistency and Scene Understanding: DriveDreamer-2 maintains consistency across views and frames, allowing an occluded car to reappear in subsequent frames.The visualization highlights this behavior with yellow rectangles.
- Consistency and Scene Understanding: In a rainy scene, truck windshield wipers continuously clear the windshield while agent behavior changes according to the weather.The example is presented as evidence of scene understanding.
- Long-Video Generation: A 29-frame multi-view video depicts a pedestrian crossing on a rainy day, the ego car changing lanes, and a vehicle cutting in from the left.Each scenario is generated from a text prompt, with later clips conditioned on the preceding clip’s last frame.
- Long-Video Generation: Long videos are produced by generating an initial clip without image conditioning and using its last frame as the initial condition for subsequent clips.This procedure supports direct text-prompt generation of extended multi-view sequences.