Source-linked AI summary
DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving
Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, Xinggang Wang
TL;DR
End-to-end autonomous driving needs diverse plausible actions without the heavy denoising cost of robotic diffusion policies in dynamic traffic scenes. DiffusionDrive uses anchored truncated diffusion and a cascade decoder, achieving 88.1 PDMS on NAVSIM navtest while running at 45 FPS.
Problem
Vanilla diffusion policies require many denoising steps, while driving scenes demand diverse plausible trajectories at real-time speed.
Method
DiffusionDrive learns denoising from an anchored Gaussian distribution using a truncated diffusion policy and an efficient cascade diffusion decoder.
Results
88.1 PDMS on NAVSIM navtest was achieved with an aligned ResNet-34 backbone, alongside 45 FPS inference on an NVIDIA 4090.
Takeaways & Limitations
DiffusionDrive generates diverse planning trajectories while combining planning quality, running efficiency, and mode diversity at real-time speeds.
Abstract
from arXiv · showhide
Recently, the diffusion model has emerged as a powerful generative technique for robotic policy learning, capable of modeling multi-mode action distributions. Leveraging its capability for end-to-end autonomous driving is a promising direction. However, the numerous denoising steps in the robotic diffusion policy and the more dynamic, open-world nature of traffic scenes pose substantial challenges for generating diverse driving actions at a real-time speed. To address these challenges, we propose a novel truncated diffusion policy that incorporates prior multi-mode anchors and truncates the diffusion schedule, enabling the model to learn denoising from anchored Gaussian distribution to the multi-mode driving action distribution. Additionally, we design an efficient cascade diffusion decoder for enhanced interaction with conditional scene context. The proposed model, DiffusionDrive, demonstrates 10$\times$ reduction in denoising steps compared to vanilla diffusion policy, delivering superior diversity and quality in just 2 steps. On the planning-oriented NAVSIM dataset, with the aligned ResNet-34 backbone, DiffusionDrive achieves 88.1 PDMS without bells and whistles, setting a new record, while running at a real-time speed of 45 FPS on an NVIDIA 4090. Qualitative results on challenging scenarios further confirm that DiffusionDrive can robustly generate diverse plausible driving actions. Code and model will be available at https://github.com/hustvl/DiffusionDrive.
1. Introduction
End-to-end driving planners must represent diverse driving decisions while meeting real-time constraints. DiffusionDrive addresses these challenges with anchored truncated diffusion and a cascade decoder, achieving strong planning quality and efficiency.
- Mainstream end-to-end planners typically regress a single-mode trajectory, while anchor vocabularies broaden behavior coverage but discretize the continuous action space.
- 20 denoising steps in vanilla diffusion introduce heavy inference overhead, limiting real-time deployment in dynamic traffic scenes.
- Anchored Gaussian distributions add limited noise around prior driving anchors, allowing truncated diffusion to model diverse action distributions.
- The cascade diffusion decoder interacts with structured queries, BEV features, and PV features through sparse deformable attention while iteratively refining trajectories.
- 88.1 PDMS on NAVSIM navtest was achieved with an aligned ResNet-34 backbone, while DiffusionDrive ran at 45 FPS on an NVIDIA 4090.
- DiffusionDrive is reported to outperform prior state-of-the-art methods, generate diverse plausible trajectories, and maintain real-time performance without bells and whistles.
2. Related Work
Prior work spans single-trajectory planning, fixed anchor vocabularies, robotic diffusion policies, and diffusion models for traffic simulation. DiffusionDrive extends diffusion-based planning to real-time end-to-end autonomous driving through truncated diffusion and an efficient decoder.
- Single-trajectory planners improve planning performance, while VADv2 shifts toward multi-mode planning by scoring and sampling from a large fixed vocabulary of anchor trajectories.
- Traffic-simulation diffusion models sample trajectories from Gaussian noise but generally rely on abstract perception ground truth rather than end-to-end sensor inputs.
- DiffusionDrive targets real-time end-to-end autonomous driving by combining truncated diffusion with an efficient diffusion decoder.
- Figure 3 contrasts vanilla diffusion with a schedule truncated around anchor trajectories and conditional reconstruction of ground-truth trajectories.
- Robotic diffusion policies capture multi-mode action distributions, but directly applying vanilla diffusion to driving requires both real-time efficiency and plausible trajectories in dynamic open-world scenes.
- The proposed approach introduces an explicit driving prior to guide diffusion toward more accurate and efficient generation for end-to-end autonomous driving.
3. Method
DiffusionDrive formulates end-to-end trajectory prediction as a truncated, multi-mode diffusion process and combines anchored initialization with a cascade decoder conditioned on scene features. The design targets mode collapse and denoising overhead while supporting flexible trajectory sampling and real-time planning.
- Task formulation: End-to-end driving predicts a sequence of ego-vehicle waypoints over a defined planning horizon.Each waypoint specifies the ego vehicle’s (x_t, y_t) location in its current coordinate system.
- Investigation: Vanilla diffusion improves planning quality over deterministic Transfuser but exhibits mode collapse when multiple random noises converge to similar trajectories.The paper quantifies diversity using a score based on mean IoU, where higher mIoU indicates less diversity.
- Investigation: 20 denoising steps reduce inference speed from 60 to 7 FPS, creating substantial computational overhead for online driving.This overhead motivates truncating the diffusion process for real-time use.
- Truncated Diffusion: The truncated diffusion policy starts from noisy trajectories around K-Means-derived prior anchors rather than standard Gaussian noise.Training diffuses the anchors using a truncated schedule with T_trunc much smaller than the full diffusion horizon.
- Truncated Diffusion: Training assigns the closest noisy anchor to the ground-truth trajectory as positive and optimizes trajectory reconstruction together with classification.The objective combines L1 reconstruction and BCE classification losses, balanced by λ.
- Truncated Diffusion: At inference, noisy anchored trajectories are progressively refined with DDIM updates, while the number of sampled trajectories can vary with computational resources.The model is trained with N_anchor trajectories but supports an arbitrary N_infer at inference.
- Architecture: The diffusion decoder uses deformable attention over BEV or PV features and perception-derived agent/map queries, with cascade refinement across denoising steps.DiffusionDrive can integrate existing perception modules and different sensor inputs, and its architecture is evaluated through cascade-decoder ablations.
4. Experiment
Experiments evaluate DiffusionDrive on NAVSIM and nuScenes, showing strong planning quality, mode diversity, efficiency, and flexible trajectory generation. Ablations examine the diffusion decoder, denoising steps, cascade stages, sampled noises, and cross-dataset performance.
- 4.1. Dataset: NAVSIM emphasizes challenging driving scenarios and evaluates planning with non-reactive simulations and closed-loop metrics, including PDMS.PDMS combines collision avoidance, drivable-area compliance, time-to-collision, comfort, and ego progress.
- 4.3. Quantitative Comparison: 88.1 PDMS is achieved on NAVSIM with the aligned ResNet-34 backbone, exceeding VADv2 by 7.2 PDMS while using 20 instead of 8192 anchors.The 20-anchor configuration represents a 400× reduction in anchors.
- 4.4. Efficiency and Diversity: 2 denoising steps reduce TransfuserDP’s 20 steps while DiffusionDrive improves PDMS by 3.5 and mode diversity by 64%, yielding a 6× FPS speedup.The final model reaches 74% mode diversity score D and enables real-time, high-quality, multi-mode planning.
- 4.5. Ablation Study: The diffusion decoder reduces parameters by 39% and improves planning quality by 2.4 PDMS relative to TransfuserTD.Spatial cross-attention is vital for accurate planning, while cascade refinement further improves performance.
- 4.5. Ablation Study: One denoising step already provides good planning quality, while additional steps improve quality and inference flexibility for complex environments.Increasing cascade stages improves quality but saturates at four stages and increases parameters and inference time.
- 4.5. Ablation Study: Sampling more noises from the anchored Gaussian distribution covers more potential actions and improves planning quality; on nuScenes, DiffusionDrive reduces SparseDrive’s average L2 error by 0.04 m.It also achieves the lowest average collision rate and runs 1.8× faster than VAD with lower L2 error and collision rate.
5. Conclusion
DiffusionDrive combines truncated diffusion with a cascade diffusion decoder for end-to-end autonomous driving. It generates diverse planning trajectories from an anchored Gaussian distribution at real-time speeds, with experiments validating planning quality, efficiency, and mode diversity.
- 5. Conclusion: DiffusionDrive combines a truncated diffusion policy with an efficient cascade diffusion decoder for generative end-to-end autonomous driving.The model denoises a variable number of samples from an anchored Gaussian distribution to produce diverse planning trajectories.
A. Further Implementation Detail
The supplementary implementation details describe the datasets, perception inputs, training recipes, and diffusion configurations used for NAVSIM and nuScenes.
- Further Implementation Detail: Additional implementation details are provided for both NAVSIM and nuScenes experiments.
- NAVSIM Dataset: The NAVSIM implementation uses ImageNet-initialized ResNet-34 features, LiDAR, auxiliary perception tasks, object queries, and BEV features.The diffusion decoder receives object queries and BEV features as inputs.
- nuScenes Dataset: The nuScenes implementation follows SparseDrive’s two-stage recipe and replaces its planning module with the proposed diffusion decoder.Stage one uses perception tasks, while stage two trains on nuScenes.
B. Further Ablation Study
Further ablations test driving-prior choices and anchor-source generalization. Results support clustered prior anchors and show that the anchored Gaussian distribution transfers across datasets.
- Driving Priors: The driving-prior ablation compares anchored Gaussian distributions with extrapolated trajectories and single-anchor inference.The reported results demonstrate the superiority of the proposed prior-anchor formulation.
- Anchor-Source Generalization: Training on CARLA with NAVSIM-clustered anchors yields superior results, supporting generalization of the anchored Gaussian distribution across datasets.The authors frame the distribution as covering potential multi-mode driving action space rather than relying on train/validation information leakage.
C. Further Qualitative Comparison
Qualitative comparisons on challenging NAVSIM navtest scenarios show that DiffusionDrive generates trajectories that adapt to traffic conditions while retaining diverse, plausible driving behaviors.
- The qualitative comparisons cover going straight, turning left, and turning right scenarios on the NAVSIM navtest split.
- Going straight: Top-1 trajectories resemble ground truth in straight-driving scenarios, while top-10 trajectories support robust lane changes and traffic-light-aware stopping.The diverse trajectories enable reasonable lane changes and stopping at the stop line.
- Turning left: Denoised trajectories for left turns dynamically adjust to traffic conditions, with top-10 trajectories performing robust lane changes.
- Turning right: Top-1 right-turn trajectories follow the leading vehicle like ground truth, whereas top-10 trajectories tend to overtake it.