Source-linked AI summary
UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
Yongkang Li, Lijun Zhou, Sixu Yan, Bencheng Liao, Tianyi Yan, Kaixin Xiong, Long Chen, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye, Wenyu Liu, Haiyang Sun, Xinggang Wang
TL;DR
Existing autonomous-driving VLA models face a trade-off between spatial perception and semantic reasoning. UniDriveVLA addresses this with decoupled experts, sparse perception, and masked coordination, achieving state-of-the-art open- and closed-loop performance alongside broad task coverage.
Problem
Existing VLA systems trade spatial perception against semantic reasoning because 2D VLMs provide limited spatial perception, while 3D representations can impair native reasoning.
Method
UniDriveVLA uses a Mixture-of-Transformers architecture with dedicated understanding, perception, and action experts, combined with sparse perception and masked joint attention.
Results
UniDriveVLA achieves state-of-the-art performance in open-loop nuScenes and closed-loop Bench2Drive evaluations, with a best Driving Score of 78.37 among methods trained without PDM-Lite.
Takeaways & Limitations
The model demonstrates broad applicability across autonomous-driving perception, prediction, understanding, and planning tasks while preserving semantic reasoning capability.
Takeaways & Limitations
The paper reports that UniDriveVLA does not match the foundation model’s general multimodal capability.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have recently emerged in autonomous driving, with the promise of leveraging rich world knowledge to improve the cognitive capabilities of driving systems. However, adapting such models for driving tasks currently faces a critical dilemma between spatial perception and semantic reasoning. Consequently, existing VLA systems are forced into suboptimal compromises: directly adopting 2D Vision-Language Models yields limited spatial perception, whereas enhancing them with 3D spatial representations often impairs the native reasoning capacity of VLMs. We argue that this dilemma largely stems from the coupled optimization of spatial perception and semantic reasoning within shared model parameters. To overcome this, we propose UniDriveVLA, a Unified Driving Vision-Language-Action model based on Mixture-of-Transformers that addresses the perception-reasoning conflict via expert decoupling. Specifically, it comprises three experts for driving understanding, scene perception, and action planning, which are coordinated through masked joint attention. In addition, we combine a sparse perception paradigm with a three-stage progressive training strategy to improve spatial perception while maintaining semantic reasoning capability. Extensive experiments show that UniDriveVLA achieves state-of-the-art performance in open-loop evaluation on nuScenes and closed-loop evaluation on Bench2Drive. Moreover, it demonstrates strong performance across a broad range of perception, prediction, and understanding tasks, including 3D detection, online mapping, motion forecasting, and driving-oriented VQA, highlighting its broad applicability as a unified model for autonomous driving. Code and model have been released at https://github.com/xiaomi-research/unidrivevla
1 Introduction
Existing driving VLA systems face a perception–reasoning dilemma: 2D VLMs offer semantic reasoning but limited spatial perception, while 3D enhancements may impair reasoning. UniDriveVLA addresses this conflict through expert decoupling, sparse perception, and progressive training, achieving strong benchmark performance and broad task coverage.
- 2D VLM-based VLAs provide semantic reasoning but insufficient spatial perception, whereas 3D spatial enhancements can improve perception while weakening native reasoning.
- Shared-parameter optimization creates representation interference between spatial perception and semantic reasoning, undermining VLM reasoning capacity.
- UniDriveVLA decouples understanding, perception, and action through dedicated Mixture-of-Transformers experts with masked joint attention for controlled communication.
- Its sparse perception paradigm extracts spatial priors from 2D VLM features, while three-stage joint training improves perception and planning while preserving reasoning ability.
- UniDriveVLA achieves state-of-the-art open-loop nuScenes and closed-loop Bench2Drive performance, with broad applicability across perception, prediction, and understanding tasks.
2 Related Work
Related work includes dual-system and unified VLA approaches, spatial enhancements using 3D representations or tokens, and Mixture-of-Transformers methods for multimodal modeling. UniDriveVLA extends these directions by combining expert decoupling with sparse spatial perception for driving.
- Dual-system approaches combine VLMs with end-to-end driving models, using low-frequency trajectories or high-level instructions to guide driving.
- Existing VLA methods improve driving through unified architectures or reinforcement-learning fine-tuning, but their spatial perception remains limited for demanding driving tasks.
- Spatial enhancements include BEV encoders, 3D Q-Formers, agent tokens, and occupancy tokens injected into language-model decoders.
- Mixture-of-Transformers integrates modality-decoupled experts with decoupled attention, while later work applies joint attention to multimodal understanding and generation.
- UniDriveVLA extends Mixture-of-Transformers to autonomous driving by combining expert decoupling with sparse spatial perception to unify perception, reasoning, and planning.
- Sparse perception replaces dense 3D or BEV representations with query-driven aggregation of multi-view image features, including sparse object-query methods.
3 Methodology
UniDriveVLA addresses the perception–reasoning conflict in autonomous driving by separating understanding, perception, and action into specialized experts while retaining controlled information exchange. Its sparse perception design and progressive training strategy support spatial, planning, and semantic capabilities within one model.
- UniDriveVLA Architecture: UniDriveVLA uses a Mixture-of-Transformers architecture that assigns driving understanding, scene perception, and action planning to specialized experts.Expert-specific pathways decouple heterogeneous tokens before cross-expert interaction.
- UniDriveVLA Architecture: The model constructs understanding, sparse perception, and action token groups from multi-view observations, historical trajectories, and navigation commands.A vision-language backbone produces understanding tokens, a sparse perception module extracts spatial priors, and an action encoder supports trajectory modeling.
- UniDriveVLA Architecture: Masked Joint Attention coordinates the three expert groups while preserving semantic reasoning through controlled visibility patterns.Understanding tokens remain causally masked, while perception and action tokens aggregate appropriate semantic and spatial context.
- Joint Training: UniDriveVLA jointly optimizes language modeling, structured perception tasks, and flow-matching-based trajectory generation within one framework.The understanding, perception, and action branches receive task-specific objectives.
- Sparse Perception: Sparse perception derives spatial information from multi-scale 2D visual features and jointly models detection, mapping, ego status, motion, and occupancy.The perception expert covers heterogeneous but mutually supportive outputs, with occupancy represented as an auxiliary latent branch.
- Progressive Training: A three-stage progressive training strategy anchors semantic reasoning, introduces perception and planning supervision, and specializes perception and action experts.The stages use multimodal pre-training, controlled joint optimization with Low-Rank Adaptation, and expert fine-tuning with motion prediction.
4 Experiments
UniDriveVLA is evaluated across closed-loop and open-loop driving, perception, understanding, and ablation settings. Results show strong planning and multi-task capability, while general visual performance remains below strong general-purpose VLMs.
- Bench2Drive: 78.37 Driving Score and 198.86 Efficiency are the best among methods trained without PDM-Lite on Bench2Drive, with a 51.82% Success Rate.It also achieves the best Merging score of 38.75% and Overtaking score of 80.00%, with a competitive mean score of 51.53%.
- nuScenes Planning: UniDriveVLA-Large achieves the lowest average L2 error under both ST-P3 and UniAD without ego-state inputs on nuScenes.The model remains competitive with ego-state inputs against recent VLA methods and end-to-end driving baselines.
- nuScenes Perception: UniDriveVLA-Large reaches 0.407 detection mAP, 0.460 NDS, and 0.535 map mAP on the nuScenes validation set.Motion prediction remains behind specialized baselines, while the model retains multi-task perception capability within a unified VLA framework.
- Ablation Study: Adding ego-state, detection, occupancy, and motion components progressively changes planning accuracy and collision performance, with detection reducing collision rate to 0.10 and occupancy reaching 0.53 best L2 error.The ablation evaluates how perception and planning components contribute within the framework.
- Understanding and General VQA: UniDriveVLA performs strongly on DriveBench and retains general visual capability, scoring 49.9 on RealWorldQA, 76.3 on AI2D, and 76.3 on ChartQA.It does not match strong general-purpose VLMs on the general multimodal benchmarks.
- Ablation Study: Mixture-of-Transformers consistently improves understanding, perception, and planning over a shared-weight decoder, especially trajectory prediction and collision reduction.The comparison supports expert decoupling as a way to mitigate the perception–reasoning conflict.
5 Conclusion
UniDriveVLA unifies understanding, perception, and planning through expert-decoupled Mixture-of-Transformers architecture, sparse perception, and progressive training. Experiments demonstrate its effectiveness as a unified autonomous-driving model, with potential extension to robotic manipulation.
- Framework: UniDriveVLA unifies understanding, perception, and planning while addressing representation interference through dedicated Understanding, Perception, and Action experts.Controlled cross-expert interaction reduces optimization conflict within a single model.
- Training and Perception: Sparse perception extracts spatial priors from multi-scale 2D visual features, while three-stage progressive training improves perception and planning while preserving semantic reasoning.
- Conclusion: Experiments across perception, understanding, and planning benchmarks demonstrate UniDriveVLA’s effectiveness as a unified autonomous-driving model.The conclusion frames the model as broadly applicable within autonomous driving.
- Future Scope: The decoupled VLA design may extend to robotic manipulation scenarios requiring structured spatial perception and semantic reasoning.