Source-linked AI summary
DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
Zhe Liu, Runhui Huang, Rui Yang, Siming Yan, Zining Wang, Lu Hou, Di Lin, Xiang Bai, Hengshuang Zhao
TL;DR
Prior VLA-based approaches can struggle to provide reliable outputs because they lack fine-grained intermediate 3D perception and prediction outputs, compromising interpretability and safety assurances. DrivePI integrates LiDAR and camera imagery with language in a unified model that jointly optimizes spatial understanding, occupancy, flow, planning, and scene understanding. With a 0.5B-parameter MLLM backbone, DrivePI outperforms existing VA models in 3D occupancy and occupancy flow while maintaining comparable interaction capabilities to VLA frameworks.
Problem
Prior VLA-based approaches can struggle to provide reliable outputs because they lack fine-grained intermediate 3D perception and prediction outputs, compromising interpretability and safety assurances.
Source: §1 · p. 2Method
DrivePI integrates LiDAR and camera imagery with language in a unified model that jointly optimizes spatial understanding, occupancy, flow, planning, and scene understanding.
Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 4Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Although multi-modal large language models (MLLMs) have shown strong capabilities across diverse domains, their application in generating fine-grained 3D perception and prediction outputs in autonomous driving remains underexplored. In this paper, we propose DrivePI, a novel spatial-aware 4D MLLM that serves as a unified Vision-Language-Action (VLA) framework that is also compatible with vision-action (VA) models. Our method jointly performs spatial understanding, 3D perception (i.e., 3D occupancy), prediction (i.e., occupancy flow), and planning (i.e., action outputs) in parallel through end-to-end optimization. To obtain both precise geometric information and rich visual appearance, our approach integrates point clouds, multi-view images, and language instructions within a unified MLLM architecture. We further develop a data engine to generate text-occupancy and text-flow QA pairs for 4D spatial understanding. Remarkably, with only a 0.5B Qwen2.5 model as MLLM backbone, DrivePI as a single unified model matches or exceeds both existing VLA models and specialized VA models. Specifically, compared to VLA models, DrivePI outperforms OpenDriveVLA-7B by 2.5% mean accuracy on nuScenes-QA and reduces collision rate by 70% over ORION (from 0.37% to 0.11%) on nuScenes. Against specialized VA models, DrivePI surpasses FB-OCC by 10.3 RayIoU for 3D occupancy on OpenOcc, reduces the mAVE from 0.591 to 0.509 for occupancy flow on OpenOcc, and achieves 32% lower L2 error than VAD (from 0.72m to 0.49m) for planning on nuScenes. Code will be available at https://github.com/happinesslz/DrivePI
1. Introduction
DrivePI addresses the gap between VA models’ spatial perception and VLA models’ language interaction with one spatial-aware 4D MLLM. It jointly supports language understanding, fine-grained perception and prediction, and driving tasks.
1. Introduction
- VA models offer spatial perception but limited language interaction, while VLA approaches can lack fine-grained 3D outputs needed for reliable, interpretable, and safety-conscious driving.DrivePI is proposed to combine these complementary strengths.Source: §1 · pp. 1–2
- DrivePI combines LiDAR with camera imagery, generates intermediate occupancy and flow representations, and jointly optimizes perception, prediction, planning, and scene understanding.The authors state that these intermediate representations support interpretability and safety assurances.Source: §1 · p. 2
- DrivePI unifies linguistic spatial understanding with fine-grained 3D perception in a framework bridging VA and VLA approaches.It outputs 3D occupancy and flow to capture spatial-temporal dynamics.Source: §1 · p. 2
- With a 0.5B-parameter MLLM backbone, DrivePI outperforms existing VA models in 3D occupancy and occupancy flow while maintaining comparable interaction capabilities to VLA frameworks.The introduction also describes three benchmarks for static occupancy, dynamic flow, and trajectory planning.Source: §1 · p. 2
2. Related Works
Prior work explores MLLM spatial reasoning, end-to-end autonomous driving, and VLA methods, but the described approaches leave a gap between coarse spatial understanding and fine-grained driving outputs. DrivePI is presented as combining those levels.
2. Related Works
- Spatial MLLM work includes 3D spatial benchmarks, metric 3D boxes, and unified grounding tasks, but is described as mainly coarse-grained or object-level.Examples include VSI-Bench, Gemini Robotics-ER [1], and Seed1.5-VL [12].Source: §2 · p. 2
- End-to-end VA systems such as UniAD and VAD perform perception, prediction, and planning, but lack natural-language interaction.VLA approaches integrate language reasoning and planning; OpenDriveVLA [71] uses BEV tokens processed by an MLLM to generate trajectories.Source: §2 · p. 3
- DrivePI is presented as unifying textual interpretability with fine-grained 3D occupancy and trajectory predictions.Figure 2 depicts the pipeline from image and LiDAR features through vision and text tokens to specialized output heads.Source: §2 · p. 3, Figure 2 · p. 3
3. Methods
DrivePI combines language-based scene understanding with explicit occupancy, flow, and planning outputs, integrating LiDAR and multi-view imagery in a jointly optimized MLLM pipeline.
3. Methods
- DrivePI unifies coarse-grained linguistic understanding with fine-grained spatial perception, combining complementary capabilities of VA models and VLA frameworks.Its fine-grained outputs include 3D occupancy and occupancy flow.Source: §3 · p. 3
3.1. Overview
- LiDAR and multi-view images become compact BEV features, which a spatial projector maps to vision tokens before the MLLM’s four task-specific heads.The projector uses cross-attention over patch features rather than simple pooling to preserve fine-grained spatial information.Source: §3.1 · pp. 3–4
3.2. Coarse-grained Spatial Understanding
- The data engine merges front- and back-view captions, creates occupancy and flow QA from ground truth, and generates planning QA from future ego-vehicle trajectories.Occupancy and flow questions address occupied positions, object categories, and velocity; planning QA pairs request high-level commands and suggested trajectories.Source: §3.2 · p. 4, Figure 3 · p. 5
3.3. Fine-grained Spatial Learning
- Three fine-grained heads handle volumetric scene understanding, temporal dynamics, and trajectory planning, using projected vision tokens arranged as a spatial feature map.The heads are the 3D occupancy, occupancy flow, and action diffusion heads.Source: §3.3 · p. 4
3.4. Loss Function
- The total loss weights text, occupancy, occupancy-flow, and action-diffusion losses, with all tasks optimized jointly end to end.The four balancing weights correspond respectively to text scene understanding, 3D occupancy perception, occupancy flow prediction, and trajectory planning.Source: §3.4 · pp. 4–5
4. Experiments
Across five experimental sections, DrivePI is evaluated on language understanding, 3D occupancy, occupancy flow, and planning, with quantitative comparisons, ablations, and qualitative visualizations. Results include strong benchmark comparisons and evidence that text and vision tasks can be unified in one model.
4.1. Datasets and Evaluation Metrics
- Experiments use nuScenes for planning and text understanding, OpenOcc for occupancy and flow, and Occ3D for additional occupancy evaluation.nuScenes-QA has 377k training QA pairs; the generated training set includes 84k scene descriptions, 560k 4D spatial QA pairs, and 24k planning QA pairs.Source: §4.1 · p. 5
4.2. Implementation Details
- Training uses Qwen2.5-0.5B and two one-epoch stages: projector alignment, followed by joint optimization of the projector, MLLM, and task heads with the vision encoder frozen.Experiments use eight NVIDIA L40S GPUs.Source: §4.2 · p. 5
4.3. Main Results
- On OpenOcc, DrivePI reports OccScore 49.3%, RayIoU 49.3%, and mAVE 0.509, beating FB-OCC by 10.3 RayIoU and reducing mAVE from 0.591 to 0.509.It also exceeds ALOcc-Flow-3D by 6.3% in OccScore and 7.4% in RayIoU, and reduces mAVE by 0.047.Source: Table 1 · p. 6, §4.3 · p. 6
- On nuScenes planning, DrivePI reduces collision rate by 70% versus ORION, from 0.37% to 0.11%, and achieves 32% lower L2 error than VAD without ego status, from 0.72m to 0.49m.With ego status, it reports L2 error 0.40 and collision rate 0.11%.Source: Table 2 · p. 6, §4.3 · p. 6
- DrivePI’s nuScenes-QA accuracy of 60.7% beats OpenDriveVLA-7B by 2.5%, while its Occ3D RayIoU is 46.0%, 4.8% above OPUS.The Occ3D comparison uses a model trained exclusively on that occupancy task.Source: §4.3 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7
4.4. Ablation Study
- Combining text and vision heads improves RayIoU by 1.8%, mAVE by 0.18, and L2 error by 0.52 over vision-only, while retaining 60.7% text accuracy.The vision-only setting reports 47.5% RayIoU, 0.69 mAVE, and 1.02 L2 error with 0.39 collision rate; text-only accuracy is 61.2%.Source: Table 5 · p. 7, §4.4 · p. 7
- Scaling occupancy QA pairs from 28K to 560K raises occupancy-category accuracy to 44.9%, and adding other QA pairs improves occupancy-status and category performance by 1.6% and 0.6%.The reported occupancy status accuracy at 560K occupancy QA pairs is 14%; the jointly trained 0.5B model improves action-status and L2 performance despite lower QA accuracy.Source: §4.4 · pp. 7–8, Table 6 · p. 8
4.5. Visualization
- Visualizations show detailed scene descriptions alongside fine-grained occupancy, flow, action, and trajectory outputs, with linguistic object categories aligned to occupancy predictions.The paper also reports similar consistency between coarse-grained descriptions and predicted trajectories.Source: §4.5 · p. 8
5. Conclusion
DrivePI combines coarse-grained text spatial understanding with fine-grained spatial perception in a VLA framework. The conclusion highlights its compact backbone and notes two limitations for future work.
5. Conclusion
- With a 0.5B-parameter MLLM, DrivePI outperforms existing VA models in 3D occupancy and occupancy flow while maintaining comparable interactive capabilities to VLA frameworks.The conclusion describes the model as efficient despite its compact backbone.Source: §5 · p. 9
- DrivePI combines coarse-grained spatial understanding in text with fine-grained spatial perception, inheriting advantages of VA models and VLA frameworks.Source: §5 · p. 9
- The authors identify simple multi-task loss balancing and the absence of reinforcement learning for complex planning as limitations.They suggest both warrant future investigation.Source: §5 · p. 9
6. More Ablation Studies
The ablations examine loss-weight trade-offs and learned importance weights over MLLM hidden states. Higher occupancy and flow loss weights favor those tasks, while deeper layers tend to receive larger learned weights.
6. More Ablation Studies
- Higher occupancy and flow loss weights improve those tasks but slightly degrade planning L2 error and text understanding in QA.Occupancy and flow losses account for over 60% of combined loss magnitude; the final implementation uses a weight of 1.0 for each.Source: §6 · p. 12, Table 7 · p. 12
- Deeper MLLM layers tend to receive larger learned importance weights, indicating that their higher-level features are more crucial to DrivePI's effectiveness.The analysis includes 25 hidden states for the 0.5B model and 37 for the 3B model, including input embeddings.Source: §6 · p. 12, Table 8 · p. 12, Table 9 · p. 12
7. Details of Coarse-grained Spatial Understanding
The data engine creates multi-stage captions and instruction QA pairs for coarse-grained spatial understanding. Its prompts cover occupancy, flow, action, and trajectory prediction.
7. Details of Coarse-grained Spatial Understanding
- Front- and back-view captions are generated individually, then merged and refined into a coherent final caption.Table 10 presents an example produced by the multi-stage data engine.Source: §7 · p. 12, Table 10 · p. 13
- The example occupancy prompt uses a 200 × 200 horizontal grid and 16 height bins, alongside prompts for safe action and a future six-frame ego trajectory.Source: §7 · p. 13
- Instruction QA pairs cover occupancy, flow, action, and trajectory prediction; occupancy and flow questions use ground truth to ask about location occupancy, category, and velocity.Source: §7 · p. 13
8. Details of Fine-grained Spatial Learning
The fine-grained heads predict 3D occupancy, occupancy flow, and trajectories. Their designs use task-specific prediction and loss procedures.
8. Details of Fine-grained Spatial Learning
- The occupancy head reshapes spatial features into height bins and uses an MLP to predict occupancy categories, with focal, geometric, semantic, and Lovász losses.Source: §8 · pp. 13–14
- The occupancy-flow head predicts velocity from occupancy and weights dynamic flow loss at 1.0 versus 0.01 for static flow to address distribution imbalance.Source: §8 · p. 14
- The action diffusion head predicts trajectories through denoising and directly generates them without SparseDrive plan-query initialization; training uses L1 loss.Source: §8 · p. 14
9. More Visualization
The supplemental visualizations show DrivePI’s coarse and fine outputs across stationary, straight-driving, and turning scenarios. They report consistent coarse/fine understanding on the straight case and accurate description in a challenging low-light turn.
9. More Visualization
- Table 11’s visualization reports accurate occupancy/flow predictions in congested scenes and appropriate actions for stationary waiting scenarios.The table presents coarse-grained and fine-grained results.Source: §9 · p. 14, Table 11 · p. 15
- For Table 12’s straight-driving case, DrivePI shows consistency between coarse and fine understanding; the example includes a busy street, a car-flow estimate, and a six-frame trajectory.The action decision is to go straight.Source: §9 · pp. 14, 16, Table 12 · p. 16
- The turning example predicts a left action and six-frame trajectory, while the section reports accurate scene description in a challenging nighttime, low-light case.Table 13 presents the turning visualization.Source: §9 · pp. 14, 17, Table 13 · p. 17