Source-linked AI summary
DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation
Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Xueyang Zhang, Yida Wang, Guan Huang, Xinze Chen, Boyuan Wang, Youyi Zhang, Wenjun Mei, Xingang Wang
TL;DR
Closed-loop driving evaluation needs arbitrary-view scene data, while current reconstruction methods are limited by forward-driving training distributions and world models produce only 2D video. DriveDreamer4D uses structured world-model-generated trajectory videos and temporally aligned real and synthetic training data, improving novel-view rendering quality and spatiotemporal coherence.
Problem
NeRF and 3DGS are constrained by forward-driving training distributions, while driving world models generate 2D videos without the spatial-temporal coherence needed for complex dynamic scenes.
Source: §1 · p. 2Method
DriveDreamer4D uses structured conditions to generate novel-trajectory videos with controlled traffic motion, then trains 4DGS by merging temporally aligned real and synthetic data with perceptual regularization.
Source: §1 · p. 2Results
FID improves relatively by 32.1% versus PVG, 46.4% versus S3Gaussian, and 16.3% versus Deformable-GS; NTA-IoU increases by 22.6%, 43.5%, and 15.6%, respectively.
Source: §1 · p. 2Takeaways & Limitations
The framework provides diverse simulation viewpoints for lane changes, acceleration, and deceleration scenarios.
Source: §1 · p. 2Novel-trajectory depth maps are not used as 4DGS constraints because LiDAR points collected on the original trajectory cannot provide complete depth for new perspectives.
Source: §3.2.2 · p. 5
Abstract
from arXiv · showhide
Closed-loop simulation is essential for advancing end-to-end autonomous driving systems. Contemporary sensor simulation methods, such as NeRF and 3DGS, rely predominantly on conditions closely aligned with training data distributions, which are largely confined to forward-driving scenarios. Consequently, these methods face limitations when rendering complex maneuvers (e.g., lane change, acceleration, deceleration). Recent advancements in autonomous-driving world models have demonstrated the potential to generate diverse driving videos. However, these approaches remain constrained to 2D video generation, inherently lacking the spatiotemporal coherence required to capture intricacies of dynamic driving environments. In this paper, we introduce DriveDreamer4D, which enhances 4D driving scene representation leveraging world model priors. Specifically, we utilize the world model as a data machine to synthesize novel trajectory videos, where structured conditions are explicitly leveraged to control the spatial-temporal consistency of traffic elements. Besides, the cousin data training strategy is proposed to facilitate merging real and synthetic data for optimizing 4DGS. To our knowledge, DriveDreamer4D is the first to utilize video generation models for improving 4D reconstruction in driving scenarios. Experimental results reveal that DriveDreamer4D significantly enhances generation quality under novel trajectory views, achieving a relative improvement in FID by 32.1%, 46.4%, and 16.3% compared to PVG, S3Gaussian, and Deformable-GS. Moreover, DriveDreamer4D markedly enhances the spatiotemporal coherence of driving agents, which is verified by a comprehensive user study and the relative increases of 22.6%, 43.5%, and 15.6% in the NTA-IoU metric.
1. Introduction
DriveDreamer4D uses autonomous-driving world-model priors to generate structured novel-trajectory videos and combine synthetic and real data for 4D scene representation. It targets the limits of existing methods on complex maneuvers and arbitrary-view closed-loop evaluation.
1. Introduction
- Closed-loop evaluation needs sensor data from arbitrary viewpoints, but existing NeRF and 3DGS approaches struggle with complex maneuvers and world models remain limited to 2D video.The paper frames 4D driving-scene representation as necessary for reconstructing dynamic environments.Source: §1 · p. 2
- The Novel Trajectory Generation Module creates structured traffic conditions to regulate foreground and background motion and keep synthesized data aligned with vehicle maneuvers.The conditions undergo view projection synchronized with maneuvers to preserve spatiotemporal constraints.Source: §1 · p. 2
- The Cousin Data Training Strategy merges temporally aligned real and synthetic data for 4DGS training and incorporates regularization for perceptual coherence.Source: §1 · p. 2
- On novel trajectory viewpoints, DriveDreamer4D improves FID relatively by 32.1% versus PVG [9], 46.4% versus S3Gaussian [28], and 16.3% versus Deformable-GS.Source: §1 · p. 2
- NTA-IoU increases by 22.6%, 43.5%, and 15.6%, respectively, while a user study reports an average win rate above 80% against three baselines.Source: §1 · p. 2
- DriveDreamer4D is presented as the first framework to use video-generation models to improve 4D reconstruction in autonomous driving.Source: §1 · p. 2
2. Related Work
Prior driving-scene representations, world models, and diffusion-based methods provide relevant tools but leave limitations in training-distribution coverage, spatial-temporal coherence, or modeling full 4D driving complexity.
2.1. Driving Scene Representation
- NeRF and 3DGS represent scenes using volumetric MLPs or 3D Gaussians, while driving-scene variants remain limited by training data concentrated in forward-driving conditions.Source: §2.1 · pp. 2–3
2.2. World Models
- Autonomous-driving world models generate action-conditioned driving videos, but their 2D outputs lack the spatial-temporal coherence needed for dynamic driving environments.Source: §2.2 · p. 3
2.3. Diffusion Prior for 3D Representation
- Diffusion-prior methods broaden training viewpoints but mainly address sparse images or static backgrounds, leaving full 4D driving-environment complexity unresolved [78] [20] [14].Source: §2.3 · p. 3
3. Method
DriveDreamer4D combines a temporal 4D Gaussian scene representation with controllable driving-world-model video generation, then trains on temporally aligned original and generated trajectory data.
3.1.1. 4D Driving Scene Representation
- 4DGS represents each scene with 3D Gaussians and a temporal field that offsets their parameters over time, rendered differentiably and optimized with RGB, depth, and SSIM losses.Source: §3.1.1 · p. 3
3.1.2. World Models for Driving Video Generation
- Driving world models use diffusion to predict future video from structured or action conditions, which can include reference images, speed, steering, layouts, camera poses, and text.Source: §3.1.2 · pp. 3–4
3.2. DriveDreamer4D
- DriveDreamer4D alters trajectory actions, generates videos conditioned on the initial frame and new-view 3D boxes and HDMap, then pairs them with original videos to train 4DGS.Source: Figure 2 · p. 4, §3.2 · p. 4
3.2.1. Novel Trajectory Video Generation
- NTGM proposes text-generated or custom trajectories, checks drivable-area and collision safety, then projects road structure and 3D boxes into new views to condition video generation.Source: §3.2.1 · p. 5
3.2.2. Cousin Data Training Strategy
- CDTS batches temporally aligned original and generated data; generated-view training omits depth supervision because original-route LiDAR misses newly visible areas, and adds a perceptual-coherence regularizer.Source: §3.2.2 · p. 5
- Table 1 compares NTA-IoU and NTL-IoU across lane-change, acceleration, and deceleration views.Source: Table 1 · p. 6
4. Experiments
The experiments evaluate DriveDreamer4D on novel-trajectory views using coherence and image-quality measures, comparisons with three 4DGS baselines, and ablations of its training strategy. Results cover lane changes, acceleration, and deceleration, with ablations examining loss weights, depth supervision, and cousin-data training.
4.2. Comparison with Different 4DGS Baselines
- FID improved 32.1%, 46.4%, and 16.3% over PVG [9], S3Gaussian [28], and Deformable-GS, respectively, on lane-change novel-trajectory views.FID comparisons focus on lane changes because acceleration and deceleration views have distributional similarities that limit the metric’s discriminative capability.Source: Table 2 · p. 6, §4.2 · p. 8
4.1. Experiment Setup
- The evaluation uses eight dynamic Waymo scenes and introduces NTA-IoU and NTL-IoU to assess foreground-agent and background-lane coherence in novel views.PSNR and SSIM are inapplicable because corresponding sensor data are unavailable for novel trajectories; the study also assesses FID and asks participants to choose preferred renderings.Source: §4.1 · pp. 6–8
4.2. Comparison with Different 4DGS Baselines
- NTA-IoU rose 22.6%, 43.5%, and 15.6% for PVG [9], S3Gaussian [28], and Deformable-GS, respectively, while NTL-IoU rose 1.8%, 3.7%, and 1.6%.The comparisons cover complex maneuvers including lane changes, acceleration, and deceleration.Source: §4.2 · p. 8
- Lane-change comparisons show sharper vehicle contours and fewer background speckles and ghosting artifacts than baseline renderings.The qualitative comparisons also describe blurred or ghosted vehicles and blurred lane markings in baseline results.Source: §4.2 · p. 8
4.3. Ablation Studies
- Temporal-aligned cousin pairs achieved FID 76.20 and NTA-IoU 0.423; adding perceptual regularization improved these to 71.52 and 0.428.The ablation reports approximately 10% improvement in FID and approximately 2% increase in NTA-IoU for CDTS; depth loss is excluded because LiDAR depth maps are incomplete due to occlusions.Source: §4.3 · p. 8
5. Discussion and Conclusion
DriveDreamer4D uses world-model priors to improve 4D driving-scene representations by generating novel-trajectory videos that complement real driving data. Experiments report improved rendering fidelity and spatiotemporal coherence, supporting its potential as a foundation for closed-loop simulation of dynamic driving scenes.
5. Discussion and Conclusion
- DriveDreamer4D generates novel-trajectory videos to complement real driving data and uses structured conditions to maintain spatial-temporal consistency across traffic elements.The conclusion describes the framework as addressing dependence on forward-driving training distributions and difficulty modeling complex maneuvers.Source: §5 · p. 9
- Experiments show improved rendering fidelity and spatiotemporal coherence across scene components for diverse simulation viewpoints.The paper identifies high-fidelity reconstruction of dynamic driving scenes as a potential foundation for closed-loop simulation.Source: §5 · p. 9
6. Baselines
The paper evaluates DriveDreamer4D with PVG [9], S3Gaussian [28], and Deformable-GS, which represent dynamic urban scenes using distinct temporal Gaussian models.
6. Baselines
- PVG [9] uses time-varying Periodic Vibration Gaussians with optimizable parameters and self-supervised static-dynamic decomposition based on lifespan.Its parameters include vibration directions, lifespan, and the moment of highest opacity.Source: §6 · p. 13
- S3Gaussian [28] combines explicit 3D Gaussians with a spatial-temporal field network, while Deformable-GS predicts Gaussian-parameter offsets from a canonical space.Both methods use self-supervised decomposition of dynamic and static elements; Deformable-GS uses a deformation network for scene dynamics.Source: §6 · p. 13
7. Implementation Details
The implementation trains a controllable front-view video-generation model on Waymo data and evaluates selected dynamic validation scenes. User-study comparisons cover lane-change, acceleration, and deceleration trajectories across three baselines.
7. Implementation Details
- Qualitative speed-change comparisons report improved novel-trajectory rendering quality over PVG [9], S3Gaussian [28], and Deformable-GS [74].The supplementary implementation description focuses on front-view generation rather than multiview video generation.Source: §7 · p. 13, Figure 5 · p. 14
- The controllable video-generation model is trained on 798 Waymo videos divided into approximately 64K clips of 40 frames, at 960 × 640 resolution.Training uses 3D bounding boxes, HDMaps, and text as control conditions, with SVD initialization and 50K iterations.Source: §7 · pp. 13–14
- Eight Waymo validation scenes were selected for significant motion dynamics, and the user study created 72 comparison videos across three trajectories and three baselines.The baseline and DriveDreamer4D renderings were randomly assigned to the left or right side for each comparison.Source: §7 · pp. 14–15
8. Visualization
The visualizations assess CDTS through a PVG ablation and compare novel-trajectory rendering under speed changes.
8. Visualization
- The visualization section includes a CDTS ablation and qualitative comparisons for speed-change scenarios.Source: §8 · p. 15
- DriveDreamer4D renders vehicle positions more precisely than baseline methods on novel trajectories, whether or not CDTS is applied.Adding CDTS further improves dynamic-vehicle reconstruction, especially at the edges.Source: §8 · p. 15
- Under speed changes, baseline methods show inaccurate positional shifts, while DriveDreamer4D improves spatial consistency and rendering quality.The baseline comparisons are shown in rows 1, 3, and 5 of Fig. 5.Source: §8 · p. 15