Source-linked AI summary
VLP: Vision Language Planning for Autonomous Driving
Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, Liu Ren
TL;DR
Autonomous driving still faces limited reasoning, weak generalization, and long-tail challenges despite progress in vision-only systems. VLP integrates language models through ALP and SLP to strengthen BEV source memory and planning context, achieving improved planning, broader task performance, and generalization to new cities and long-tail cases.
Problem
Vision-only autonomous driving methods still face limited reasoning, low out-of-domain generalization, and long-tail scenarios that affect safe motion planning.
Method
VLP integrates language-model common-sense capabilities into vision-based ADS through ALP for BEV reasoning and SLP for self-driving decision-making.
Results
VLP improves perception, prediction, and planning across driving tasks and shows robustness in new cities and long-tail cases.
Takeaways & Limitations
VLP extends ADS beyond the training environment toward safer and more reliable autonomous driving.
Abstract
from arXiv · showhide
Autonomous driving is a complex and challenging task that aims at safe motion planning through scene understanding and reasoning. While vision-only autonomous driving methods have recently achieved notable performance, through enhanced scene understanding, several key issues, including lack of reasoning, low generalization performance and long-tail scenarios, still need to be addressed. In this paper, we present VLP, a novel Vision-Language-Planning framework that exploits language models to bridge the gap between linguistic understanding and autonomous driving. VLP enhances autonomous driving systems by strengthening both the source memory foundation and the self-driving car's contextual understanding. VLP achieves state-of-the-art end-to-end planning performance on the challenging NuScenes dataset by achieving 35.9\% and 60.5\% reduction in terms of average L2 error and collision rates, respectively, compared to the previous best method. Moreover, VLP shows improved performance in challenging long-tail scenarios and strong generalization capabilities when faced with new urban environments.
1. Introduction
Vision-only autonomous driving systems struggle with reasoning, out-of-domain generalization, and long-tail scenarios. VLP integrates language-model capabilities through ALP and SLP to improve BEV reasoning, planning decisions, and generalization.
- Autonomous driving requires scene understanding and reasoning for safe motion planning, but conventional modular pipelines can suffer compounding errors and safety concerns.
- Vision-based ADS use BEV representations for downstream tasks but struggle in new cities and long-tail scenarios.
- Language models provide common-sense and generalization capabilities, yet their reasoning ability remains underexplored in real-world autonomous driving.
- VLP integrates language-model common sense into vision-based ADS through ALP for BEV reasoning and SLP for decision-making.
- ALP improves local BEV semantic representation and reasoning, while SLP strengthens the planning query's ability to acquire information for final decisions.
- VLP reports consistent improvements across planning, tracking, forecasting, and other driving tasks, including zero-shot generalization across distinct cities.
2. Related Work
Related work covers end-to-end autonomous driving, vision-language models, and embodied language models. These lines of work motivate integrating language understanding with autonomous driving while distinguishing VLP's target domain.
- End-to-End Autonomous Driving: Early end-to-end systems planned motion without explicit intermediate tasks, whereas modular end-to-end frameworks add interpretable perception, prediction, and planning components.
- Vision-Language Models: Large language models and vision-language models combine broad language or visual-text understanding with common-sense capabilities and zero-shot generalization.
- Embodied Language Models: Embodied language-model research has mainly focused on robotics, with relatively few efforts applying embodied language models to autonomous driving.
3. Methodology
VLP equips vision-based autonomous driving with language-model-guided local BEV reasoning and global planning-query refinement. ALP aligns agent-level BEV features with language-derived expectations, while SLP aligns ego-vehicle query features with textual planning information during training without inference overhead.
- Framework overview: VLP uses ALP and SLP to enhance BEV reasoning and self-driving decision-making with language models and contrastive learning.ALP targets local details through agent-wise learning, whereas SLP targets global context through sample-wise learning.
- Agent-centric Learning Paradigm: ALP aligns camera-derived BEV features with a true bird’s-eye-view representation to refine local semantic details for downstream driving decisions.It crops ego-vehicle, foreground-object, and lane regions using 3D boxes and panoptic scene masks, then applies language-based expectation alignment.
- Agent-centric Learning Paradigm: ALP forms an agent-wise similarity matrix between normalized BEV features and language-derived expectation features, then averages cross-entropy losses across both modes.The ground-truth similarity matrix is a unit matrix, matching each agent BEV feature to its corresponding expected feature.
- SDC-centric Learning Paradigm: SLP aligns the ego-vehicle query feature with a language feature encoding the planning ground truth, including high-level driving commands and future trajectories.The sample-wise contrastive process refines the ego query using linguistic information associated with the intended plan.
- Training objective: The overall VLP loss combines BEV encoder reasoning loss with decoder decision-making loss spanning downstream perception, prediction, planning, and SLP.The encoder loss equals the ALP loss, while the decoder loss includes downstream task losses and the SLP loss.
- Inference: VLP discards the language-model head during inference, adding no parameters or computations to the original autonomous driving system.The language model and adaptation components support training, while the deployed system retains the original inference cost.
4. Experiments
Experiments on nuScenes evaluate VLP across planning, perception, prediction, generalization, and prompt ablations. VLP improves open-loop planning, multiple downstream tasks, new-city transfer, and long-tail performance over vision-only baselines.
- Experimental setup: Experiments use six camera images from nuScenes, comparing VLP-enriched UniAD and VAD with their vision-only baselines under matched training settings.nuScenes contains 1,000 scenes from Boston and Singapore.
- Perception and prediction: VLP improves multi-object tracking, mapping, motion forecasting, occupancy prediction, 3D object detection, and vectorized scene segmentation across the evaluated driving tasks.For motion forecasting, SLP reduces minADE, minFDE, and MR while increasing EPA, with further gains from the full VLP.
- New-city generalization: 15.1% and 18.5% reductions in average L2 error and collision rate are achieved in Boston, while Singapore shows 19.2% and 48.7% reductions, respectively, for VLP-VAD versus VAD.The evaluation trains in one city and tests in the other.
- Long-tail generalization: 3.7% and 7.4% increases in AMOTA and Recall are observed for UniAD on long-tail tracking, while VAD mAP increases by 15.9%.The long-tail evaluation covers multi-object tracking and 3D detection.
- Ablations: Ablations report that both SLP and ALP are important for enhanced planning, while all ground-truth prompt information components contribute to improved planning performance.The prompt-format study examines the effect of different ground-truth information components.
5. Conclusion
The paper presents VLP as a language-prompt-guided approach for improving perception, prediction, and planning in autonomous driving. Experiments report adaptability to new cities and long-tail cases, while the evaluation remains limited to nuScenes and camera inputs.
- Conclusion: VLP combines self-driving-car-centric SLP and agent-wise ALP, guided by language prompts, to improve perception, prediction, and planning in ADS.The approach is intended to support safer and more reliable autonomous driving in real-world conditions.
- Conclusion: VLP demonstrates robustness to new cities and long-tail cases, extending ADS capabilities beyond the training environment.The conclusion presents this as adaptability supported by the reported generalization experiments.
- Limitations: The experiments are confined to the nuScenes dataset and camera modality, with broader datasets and sensor modalities left for future work.This is the paper’s stated limitation and scope boundary.
6. Experiment Setup
The experimental setup builds on UniAD and VAD, whose architectures use BEV extraction followed by transformer decoder modules for perception, prediction, and planning. Detailed training configurations are reported separately for both model families.
- Model components: UniAD uses a BEV extractor followed by five transformer decoder-based modules for tracking, mapping, motion forecasting, occupancy prediction, and planning.These modules cover the P3-oriented downstream tasks evaluated for UniAD.
- Model components: VAD uses a BEV extractor followed by four transformer decoder-based modules for 3D detection, scene segmentation, motion forecasting, and planning.The architecture differs from UniAD in its downstream task decomposition.
- Hyperparameters: Detailed training configurations for UniAD-based and VAD-based models are provided in Table 11.The passage identifies the location of the reported hyperparameters but does not list them.
7. Ablation Study
The ablations show that VLP consistently improves downstream performance across language-model choices, prompt settings, and its ALP and SLP components. The largest emphasis remains trajectory planning, while ALP and SLP together also improve long-tail generalization and other tasks.
- Prompt Information: Completed ground-truth prompt information consistently yields the strongest downstream performance for both UniAD and VAD.VLP-equipped models outperform models without VLP regardless of the included ground-truth information.
- Different LMs: Different pretrained language models consistently improve decision-making over two baselines, supporting VLP's design across language-model choices.The selected language model balances computational efficiency and effectiveness; LLAMA integration is left for future work.
- ALP and SLP: Both SLP and ALP consistently improve all tasks, with SLP emphasizing final planning through the ego-car query and ALP operating on the shared BEV map.When combined, the components yield a +5.9% improvement over the baseline, despite smaller gains in scene segmentation and occupancy prediction.
8. Long-tail Generalization for 3D Object Detection
VLP components improve 3D object-detection generalization on long-tail cases. SLP provides a noticeable baseline improvement, while combining SLP and ALP further enhances generalization.
- Long-tail Generalization for 3D Object Detection: SLP improves long-tail 3D object-detection generalization over the baseline, and combining SLP with ALP enhances it further.Long-tail classes include construction vehicles, buses, motorcycles, bicycles, and trailers, which comprise approximately 6% of nuScenes.
9. Visualization
Qualitative comparisons in Figures 3–9 show VLP against the baseline, with green arrows marking areas where VLP performs better. The visualizations depict more efficient and safer self-driving navigation with VLP.
- Visualization: Figures 3–9 use green arrows to highlight regions where VLP outperforms the baseline in qualitative comparisons.The visual comparisons illustrate more efficient and safer navigation with VLP.
10. Why human-like?
VLP aims to make visual representations more human-like by using contextual cognitive semantics rather than plain numerical representations. Designed prompts and pretrained language models support reasoning and decision-making in an enriched feature space retained during inference.
- Why human-like?: VLP imbues the visual system with a more human-like feature space by incorporating designed prompts and pretrained language models into reasoning and decision-making.The enriched feature space and robust capabilities are retained during inference.