Source-linked AI summary
DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving
Anqing Jiang, Yu Gao, Zhigang Sun, Yiru Wang, Jijun Wang, Jinghao Chai, Qian Cao, Yuweng Heng, Hao Jiang, Yunda Dong, Zongzheng Zhang, Xianda Guo, Hao Sun, Hao Zhao
TL;DR
End-to-end autonomous driving methods face challenges in BEV computation, action diversity, and decision quality in complex scenarios. Diff-VLA addresses these issues with VLM guidance, hybrid sparse-dense perception, and diffusion planning, achieving 45.0 EPDMS on navsim-v2’s private test set. The paper reports a modular training limitation, with joint end-to-end training left unexplored.
Problem
Existing end-to-end driving methods face expensive BEV computation, limited action diversity, and sub-optimal decisions in complex real-world scenarios.
Method
Diff-VLA combines VLM guidance, sparse-dense hybrid perception, and diffusion-based planning for multimodal driving behavior.
Results
45.0 EPDMS is achieved on the navsim-v2 private test set.
Takeaways & Limitations
The framework provides a closed-loop driving approach evaluated with reactive traffic participants and realistic synthetic multi-view camera images.
Takeaways & Limitations
The VLM, sparse perception, dense perception, and planning head are trained separately, leaving single-stage end-to-end training unexplored.
Abstract
from arXiv · showhide
Research interest in end-to-end autonomous driving has surged owing to its fully differentiable design integrating modular tasks, i.e. perception, prediction and planing, which enables optimization in pursuit of the ultimate goal. Despite the great potential of the end-to-end paradigm, existing methods suffer from several aspects including expensive BEV (bird's eye view) computation, action diversity, and sub-optimal decision in complex real-world scenarios. To address these challenges, we propose a novel hybrid sparse-dense diffusion policy, empowered by a Vision-Language Model (VLM), called Diff-VLA. We explore the sparse diffusion representation for efficient multi-modal driving behavior. Moreover, we rethink the effectiveness of VLM driving decision and improve the trajectory generation guidance through deep interaction across agent, map instances and VLM output. Our method shows superior performance in Autonomous Grand Challenge 2025 which contains challenging real and reactive synthetic scenarios. Our methods achieves 45.0 PDMS.
1. Introduction
Diff-VLA combines VLM guidance, hybrid sparse-dense perception, and diffusion planning to address multimodal driving behavior and closed-loop robustness. On navsim-v2, it achieves 45.0 EPDMS on the private test set.
- Motivation: Existing end-to-end methods often regress single-mode trajectories, while diffusion methods model multimodal action distributions.
- Approach: The framework integrates VLM guidance, sparse-dense hybrid perception, and diffusion-based planning.The VLM produces trajectories and high-level commands that guide the diffusion planner, while perception supplies dense and sparse scene information.
- Results: 45.0 EPDMS is achieved on the navsim-v2 private test set.navsim-v2 assesses robustness and generalization in closed loop using reactive traffic participants and realistic synthetic multi-view images.
2. Perception
The perception system combines sparse explicit object and map representations with dense BEV features, integrating both into trajectory planning. This design supports scene understanding, collision detection, and drivable-area checks.
- Dual perception: The perception system uses parallel sparse and dense modules whose outputs are integrated into a subsequent trajectory head.The sparse branch produces 3D bounding boxes and map vectors, while the dense branch produces BEV feature vectors.
- Dual perception: The sparse branch extracts 3D objects and online maps, whereas the dense branch constructs a BEV feature space through projection.
- Representation: The dual design combines implicit agent and environment features with explicit object and map information.It is intended to overcome limitations of relying only on projection-based or sampling-based BEV construction.
- Planning support: Explicit object and map outputs enable collision detection and drivable-area checks beyond feature-based trajectory selection.
- Training: Perception training is staged: sparse detection and map training precedes joint dense-branch and trajectory-head training.All perception branches use the VoV-99 backbone.
3. VLM
The VLM Command Guidance module processes multi-view images and navigation instructions to generate high-level driving decisions. These decisions are encoded and fused with external driving signals to guide diffusion-based trajectory planning.
- Architecture: The VLM Command Guidance module uses multi-image encoding and multi-view prompting for scene understanding.
- Architecture: Senna-VLM comprises a Vision Encoder, Driving Vision Adaptor, Text Encoder, and Large Language Model.The Vision Encoder extracts features from multi-view image sequences, and the adaptor compresses them into image tokens.
- Decision generation: Image and text tokens are jointly provided to the LLM, which generates high-level driving decisions.Text tokens encode user instructions and navigation commands.
- Implementation: The implementation uses ViT-L/14 from CLIP as the Vision Encoder and Vicuna-v1.5-7B as the LLM.
- Command guidance: The module decomposes decisions into lateral and longitudinal control, one-hot encodes them, and combines them with external driving signals.A command encoder converts the resulting commands into semantic guidance for the diffusion planner.
4. Planning
The planning module discretizes ego-vehicle trajectories into learned vocabularies, initializes diffusion anchors with Gaussian noise, and conditions trajectory generation on sparse and dense perception. Its decoder predicts waypoints and classification scores.
- Trajectory representation: Trajectory vocabularies discretize the ego vehicle’s action space into clustered waypoint sequences with a planning horizon of Th = 4.The vocabularies are built by applying k-means clustering to trajectories sampled from the NavTrain split.
- Diffusion initialization: Gaussian noise is added to trajectory vocabularies to create Nanchor = 32 trajectory anchors for the diffusion process.
- Guidance: The diffusion planner uses explicit sparse-branch guidance and implicit dense-branch guidance as the condition z = {zi, ze}.
- Prediction: The diffusion trajectory decoder predicts target waypoints and classification scores for trajectory candidates.The trajectory head is trained with trajectory classification and regression losses.
5. Post processing
The post-processing analysis identifies higher collision rates under high-speed conditions and attributes them to trajectory-distribution deviation. A 2% y-axis deceleration was added to prevent collisions without materially changing other evaluation scores.
- High-speed conditions produce slightly higher collision rates than other scenarios.
- The authors attribute the increased collision rate to deviation between predicted and actual trajectory distributions.
- A 2% y-axis deceleration effectively prevents collisions without significant differences in other evaluation scores.
6. Experiments
Experiments train the solution in stages, using separate or jointly trained components and specified hyperparameters. Results for the proposed VLA architecture are reported in NAVSIM2.
- Training procedure: Training is divided into two stages, with the VLA guidance and hybrid perception modules trained first and the parallel motion planner trained in stage two.
- Training details: Stage-one and stage-two training use AdamW with cosine learning-rate decay and different batch sizes, learning rates, and epoch counts.The sparse branch uses batch size 128, learning rate 10^-4, and 100 epochs; VLM training uses batch size 192, learning rate 2×10^-5, and 1 epoch; stage two uses batch size 256, initial learning rate 10^-4, and 100 epochs.
- Results: The proposed VLA architecture’s experimental results are reported in Table 2 for NAVSIM2.
7. Limitation
The main limitation is that time constraints prevented single-stage end-to-end training of all modules. Joint training remains unexplored and could improve performance.
- Time constraints required separate training of the VLM, sparse perception, dense perception, and planning head.
- Single-stage end-to-end training of all modules remains unexplored.
- The authors identify potential performance improvements from combining all modules in one training stage.
8. Conclusion
The conclusion presents a VLA that combines end-to-end autonomous driving with VLM guidance, hybrid perception, and a diffusion planner. The proposed framework reaches a metric of 45.0 in Table 2.
- The proposed VLA combines end-to-end autonomous driving with VLM guidance, hybrid perception, and a diffusion planner.
- The framework integrates VLM guidance, hybrid perception, and diffusion planning within one proposed architecture.
- 45.0 is the proposed framework’s reported metric in the last row of Table 2.