Source-linked AI summary
DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning
Yang Zhou, Xiaofeng Wang, Hao Shao, Letian Wang, Guosheng Zhao, Jiangnan Shao, Jiagang Zhu, Tingdong Yu, Zheng Zhu, Guan Huang, Steven L. Waslander
TL;DR
Existing world-action models often lack explicit geometric grounding, limiting how structured spatial cues support embodied planning. DriveDreamer-Policy unifies depth generation, future-video imagination, and motion planning through an LLM with modular generative experts. It achieves strong planning and world-generation results on Navsim, including 88.7 EPDMS on Navsim v2 and improved future-video quality.
Problem
Existing world-action models often represent the world through images, videos, or latent rollouts without explicit geometry, limiting structured spatial cues for planning.
Method
DriveDreamer-Policy uses an LLM to produce query-based world and action embeddings that condition modular depth, video, and action generators in a causal depth→video→action pathway.
Results
The model achieves strong performance across Navsim planning and world-generation tasks, including 88.7 EPDMS on Navsim v2 and an FVD of 53.59.
Takeaways & Limitations
Explicit depth provides complementary geometric cues that improve future-video coherence and planning robustness when combined with video imagination.
Abstract
from arXiv · showhide
Recently, world-action models (WAM) have emerged to bridge vision-language-action (VLA) models and world models, unifying their reasoning and instruction-following capabilities and spatio-temporal world modeling. However, existing WAM approaches often focus on modeling 2D appearance or latent representations, with limited geometric grounding-an essential element for embodied systems operating in the physical world. We present DriveDreamer-Policy, a unified driving world-action model that integrates depth generation, future video generation, and motion planning within a single modular architecture. The model employs a large language model to process language instructions, multi-view images, and actions, followed by three lightweight generators that produce depth, future video, and actions. By learning a geometry-aware world representation and using it to guide both future prediction and planning within a unified framework, the proposed model produces more coherent imagined futures and more informed driving actions, while maintaining modularity and controllable latency. Experiments on the Navsim v1 and v2 benchmarks demonstrate that DriveDreamer-Policy achieves strong performance on both closed-loop planning and world generation tasks. In particular, our model reaches 89.2 PDMS on Navsim v1 and 88.7 EPDMS on Navsim v2, outperforming existing world-model-based approaches while producing higher-quality future video and depth predictions. Ablation studies further show that explicit depth learning provides complementary benefits to video imagination and improves planning robustness.
1 GigaAI 2 University of Toronto 3 CUHK MMLab * Corresponding Author
The project website is provided at drivedreamer-policy.github.io.
- The project website is available at drivedreamer-policy.github.io.
1. Introduction
Existing driving world-action models unify future-world generation and planning but often lack explicit geometric grounding. DriveDreamer-Policy addresses this gap with depth-guided, modular generation and planning, achieving strong Navsim results and improved video quality.
- Existing world-action models unify future-world generation with planning but commonly use image/video representations without explicit geometric grounding.This can limit the usefulness of imagined futures for planning and leave geometric safety cues underrepresented.
- Autonomous driving requires modeling evolving 3D geometry for occlusion reasoning, distance estimation, and physically consistent motion.Depth is compact, directly tied to geometry, and can scaffold future video generation and planning.
- DriveDreamer-Policy jointly generates depth-based scene geometry, action-conditioned future videos, and future trajectories using LLM-produced world and action embeddings.Its causal depth→video→action pathway allows later stages to consume upstream geometric and imagined-future context.
- 88.7 EPDMS on Navsim v2 and 53.59 FVD demonstrate improved planning performance and future-video quality over previous methods.The reported improvements are +2.6 EPDMS and -32.36 FVD, respectively.
2. Related Works
Driving research has progressed from video generation and language-guided action systems toward unified world-action models. DriveDreamer-Policy adds diffusion video generation and explicit depth-based geometry to this line.
- Driving generative models use sensor data to produce future videos for scalable data synthesis and simulation.
- VLM-based driving systems evolved from scene interpretation and maneuver suggestions toward direct action generation.
- DriveDreamer-Policy replaces image-token world prediction with a diffusion video-generation head and adds depth-based 3D representation for geometric grounding.
3. Methodology
DriveDreamer-Policy uses an LLM to transform language, multi-view observations, and action context into query-based conditions for modular depth, video, and action generators. A causal depth→video→action interface integrates geometry, temporal imagination, and planning.
- 3.1. Preliminaries: The model applies conditional flow matching to train generative experts that transport noise toward conditioned target samples.At inference, it samples noise and integrates the induced ODE backward to obtain a conditioned sample.
- 3.2.1. World Understanding: The pipeline encodes language instructions, multi-view RGB images, and current action context, then processes them with an LLM and learned world/action queries.The resulting embeddings condition three modular experts for depth, future video, and future action generation.
- 3.2.1. World Understanding: A causal query ordering lets depth queries precede video queries, while action queries attend to both depth and video context.This creates a single-pass information flow without extra synchronization or iteration.
- 3.2.2. World and Action Prediction: The depth generator produces monocular depth as an explicit 3D scaffold for downstream video imagination and action planning.Depth supplies cues such as occlusions, object boundaries, free space, and distance to collision.
- 3.2.2. World and Action Prediction: The pixel-space depth diffusion transformer uses RGB input and LLM depth embeddings to preserve global structure and fine-grained geometric details.Its depth embeddings can be queried on demand and serve as upstream geometric features for video and action queries.
- 3.2.2. World and Action Prediction: The video generator conditions noisy future-video latents on LLM world-video embeddings that summarize language, perception, action context, and upstream depth cues.
- 3.2.2. World and Action Prediction: The action generator maps a noise trajectory to feasible future actions using an LLM action embedding that aggregates instruction, observation, geometry, and imagination cues.Trajectory states use position and continuous heading components (x, y, cos θ, sin θ) to avoid angular wrap-around.
- 3.3. Training Details: Training uses a single-stage joint multi-task objective combining depth, video, and trajectory prediction losses.The depth loss weight is 0.1, while the remaining default loss weights are 1.0.
4. Experiments
Experiments evaluate DriveDreamer-Policy on Navsim planning and world-generation tasks, using standard planning metrics, generative-quality metrics, baselines, and ablations. The model achieves strong planning results, while depth and video learning jointly improve planning, future-video generation, and qualitative trajectory behavior.
- Datasets and Metrics: Navsim uses synchronized surround-view driving logs, with PDMS for v1 and EPDMS for v2 closed-loop planning evaluation.The navtrain and navtest splits contain 100k and 12k samples, respectively, sampled at 2Hz.
- Planning Performance: 89.2 PDMS on Navsim v1 and 88.7 EPDMS on Navsim v2 outperform all considered comparison methods.The comparisons include vision-based end-to-end, vision-language-action, and world-model-based planners.
- World Generation: DriveDreamer-Policy improves both video and depth generation quality against the evaluated world-model baselines.Video is compared with PWM, while depth is compared with zero-shot and fine-tuned PPD variants; the video improvement over PWM is 32.36 on FVD.
- Ablation Studies: Joint depth-video-action training produces the largest planning gains, whereas depth-only or video-only world training yields smaller improvements.The ablation compares action-only, depth+action, video+action, and depth+video+action variants under identical training budgets.
- Ablation Studies: Joint depth learning improves future-video accuracy by providing a 3D scaffold for coherent prediction.The comparison uses video-only and depth+video variants with the same training data and compute budget, with video queries causally conditioned on depth queries in the joint variant.
- Ablation Studies: Increasing the query budget generally improves planning and world-generation performance by providing higher-capacity slots for relevant context.The default uses 64 depth, 64 video, and 8 action query tokens, compared with smaller 32, 32, and 4-token budgets.
- Qualitative Analysis: Qualitative visualizations show depth-conditioned imagination supporting safer, more human-like trajectories and reducing collision risk, off-road drift, and late maneuver corrections.Across representative scenarios, world-conditioned variants maintain clearer safety margins, recover correct maneuvers earlier, and more closely match expert paths.
5. Conclusion
DriveDreamer-Policy unifies depth generation, future video imagination, and motion planning in a modular framework. Depth and video provide complementary cues that improve planning robustness while supporting strong planning and world-generation performance.
- DriveDreamer-Policy jointly performs depth generation, future video imagination, and motion planning within one unified driving world-action framework.The architecture combines an LLM with modular generative experts connected through a compact query interface.
- Depth provides an explicit geometric scaffold, while future video contributes predicted dynamics for action prediction.The model organizes information flow in a depth→video→action manner.
- Depth and video provide complementary cues that improve planning robustness across planning and world-generation tasks.