Source-linked AI summary
DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, Hang Zhao
TL;DR
Autonomous-driving systems need better understanding of complex, long-tail scenarios, while existing pipelines and VLMs have complementary limitations. DriveVLM uses staged VLM reasoning, and DriveVLM-Dual combines it with traditional perception and planning; experiments and production-vehicle deployment report effectiveness in complex driving environments.
Problem
Existing autonomous-driving pipelines struggle with rare objects and decision-level interactions, while VLMs have spatial-reasoning and onboard-computation limitations.
Method
DriveVLM applies VLM-based scene description, scene analysis, and hierarchical planning, while DriveVLM-Dual optionally integrates traditional 3D perception and planning modules.
Results
DriveVLM and DriveVLM-Dual surpass reported state-of-the-art methods on public and SUP-AD benchmarks, with DriveVLM-Dual also validated through production-vehicle testing.
Takeaways & Limitations
The systems provide a combined VLM and traditional-pipeline approach for scene understanding and planning in intricate and dynamic driving scenarios.
Takeaways & Limitations
Existing driving-language datasets lack sufficient data on critical safety scenarios in self-driving systems.
Abstract
from arXiv · showhide
A primary hurdle of autonomous driving in urban environments is understanding complex and long-tail scenarios, such as challenging road conditions and delicate human behaviors. We introduce DriveVLM, an autonomous driving system leveraging Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities. DriveVLM integrates a unique combination of reasoning modules for scene description, scene analysis, and hierarchical planning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, we propose DriveVLM-Dual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driving pipeline. Experiments on both the nuScenes dataset and our SUP-AD dataset demonstrate the efficacy of DriveVLM and DriveVLM-Dual in handling complex and unpredictable driving conditions. Finally, we deploy the DriveVLM-Dual on a production vehicle, verifying it is effective in real-world autonomous driving environments.
1 Introduction
DriveVLM addresses difficult autonomous-driving scene understanding with VLM-based reasoning, while DriveVLM-Dual combines this approach with traditional modules for spatial reasoning and real-time planning. The paper also introduces SUP-AD and evaluation metrics, reports strong benchmark performance, and validates deployment in a production vehicle.
- Motivation: Existing systems struggle with rare objects, unique attributes, and decision-level interactions because their components focus largely on familiar-object tracking and trajectory-level actions.The paper identifies scene understanding as a central hurdle in complex, unpredictable scenarios such as adverse weather and intricate road layouts.
- DriveVLM: DriveVLM uses scene description, scene analysis, and hierarchical planning to address complex autonomous-driving scene understanding.The modules progress from linguistically describing the environment and critical objects to analyzing their influence and producing step-by-step plans.
- Results: Experiments on nuScenes and SUP-AD report superior performance, while DriveVLM-Dual exceeds state-of-the-art end-to-end motion-planning methods.The reported gains are especially notable in few-shot scenarios and complex or unpredictable driving conditions.
- DriveVLM-Dual: DriveVLM-Dual combines DriveVLM with a traditional autonomous-driving pipeline to improve spatial reasoning and real-time planning.The hybrid can integrate 3D perception and planning modules, including object detectors, occupancy networks, and motion planners.
- Evaluation resources: The paper defines the scene understanding and planning task, proposes evaluation metrics, and constructs the SUP-AD dataset through data mining and annotation.These resources assess scene analysis and meta-action planning capabilities.
- Deployment: DriveVLM-Dual was deployed and tested on a production vehicle, verifying effectiveness in real-world autonomous-driving environments.The deployment included strategies for accelerating VLM inference in real driving scenarios.
2 Related Works
Related work spans VLM-based robotics, learning-based motion planning, and language-annotated driving datasets. The paper positions these datasets as useful for natural-language driving tasks while noting insufficient coverage of critical safety scenarios.
- Vision-Language Models: VLMs have been applied in robotics to produce high-level instructions or low-level robot actions.Examples include MiniGPT-4, LLaVA, and Qwen-VL among broader large VLM research.
- Learning-based Planning: Learning-based planning methods use reinforcement learning or imitation learning to map raw sensory inputs directly to control actions.Other approaches improve interpretability by constructing dense cost maps from learned models.
- Driving Caption Datasets: Driving-caption datasets connect language with driving objectives through object references, free-form captions, question answering, and action explanations.The cited datasets include Refer-KITTI, Talk2Car, NuPrompt, nuScenes-QA, BDD-X, and BDD-OIA.
- Positioning: Figure 1 contrasts DriveVLM’s VLM reasoning pipeline with DriveVLM-Dual’s addition of traditional 3D perception and trajectory planning.The dual design targets spatial reasoning and real-time trajectory planning.
- Driving Caption Datasets: Existing language-annotated driving datasets lack sufficient data on critical safety scenarios in self-driving systems.This limitation bounds their coverage of safety-relevant driving conditions.
3 DriveVLM
DriveVLM uses VLM chain-of-thought reasoning to progress from scene description and critical-object analysis to hierarchical driving plans. DriveVLM-Dual supplements this reasoning with 3D perception and high-frequency trajectory refinement for more accurate spatial understanding and real-time planning.
- DriveVLM: DriveVLM processes image sequences through scene description, scene analysis, and hierarchical planning modules.Its architecture combines a vision transformer encoder, an attention-based extractor, and a large language model.
- Scene Description: The scene description module characterizes driving environments and identifies critical objects likely to influence the current scenario.Environment conditions include weather, time, road, and lane information; critical objects include categories and approximate image bounding boxes.
- Scene Analysis: DriveVLM analyzes critical objects through static attributes, motion states, or particular behaviors, then predicts each object’s influence on the ego vehicle.Only the characteristics applicable to a given critical object need to be analyzed.
- Hierarchical Planning: Hierarchical planning combines the scene summary with route, ego pose, and velocity to generate meta-actions, decision descriptions, and trajectory waypoints.Decision descriptions specify an action, interacting subject, and duration; waypoints represent the future vehicle path at fixed intervals.
4 Task and Dataset
The paper defines Scene Understanding for Planning as a multi-output task covering scene description, analysis, decisions, and trajectories, and introduces evaluation procedures and the SUP-AD dataset. SUP-AD targets long-tail objects and challenging scenarios through mining, keyframe selection, and annotation.
- 4.1 Task Definition: The Scene Understanding for Planning task takes multi-view surrounding-camera videos and optionally 3D perception results as input.Its outputs include scene description, scene analysis, meta-actions, decision description, and trajectory waypoints.
- 4.2 Evaluation: Evaluation measures both interpretation of the driving scene and the decisions made.Scene description and analysis use structured comparison with human annotations, while meta-actions are compared with annotated sequences using dynamic programming.
- 4.3 Dataset Construction: The SUP-AD construction pipeline mines long-tail objects and challenging scenarios, selects a keyframe, and annotates scene understanding and planning information.Long-tail examples include unusual vehicles, road debris, and animals; challenging scenarios are selected using variance in recorded driving maneuvers.
- Keyframe Selection: Keyframes in challenging scenes are selected 0.5s to 1s before a required speed or direction change to provide reaction time for decision-making.Scenes without behavioral changes use a frame relevant to the current driving scenario.
- Scene Annotation: Scene annotations cover description, analysis, and planning, while trajectory waypoints can be auto-labeled from vehicle IMU recordings.The annotation tool supports video replay, bounding-box drawing with language descriptions, and selection from action and decision candidates.
5 Experiments
Experiments evaluate DriveVLM and DriveVLM-Dual on SUP-AD and nuScenes, including benchmark comparisons, ablations, and qualitative driving scenarios. The systems show strong scene-understanding and planning performance, with DriveVLM-Dual achieving state-of-the-art nuScenes planning results.
- 5.1 Settings: DriveVLM is evaluated on the SUP-AD test set, while DriveVLM-Dual is evaluated on nuScenes planning.The experiments also include module ablations, traditional-pipeline comparisons, and qualitative results.
- 5.2 Main Results: DriveVLM with Qwen-VL achieves the best SUP-AD performance among the compared vision-language models.GPT-4V uses in-context learning and its extraneous scene-description information is often classified as hallucination under the evaluation metric.
- 5.2 Main Results: DriveVLM-Dual achieves the best performance on the nuScenes planning task when cooperating with VAD.The result indicates that the system performs well on ordinary scenarios as well as complex-scene understanding.
- 5.3 Ablation Study: Critical object analysis and 3D perception are ablated as design choices in the nuScenes validation experiments.Critical object analysis prioritizes important driving-environment elements, while 3D perception provides refined surroundings understanding and precise predictions.
- 5.3 Ablation Study: DriveVLM-Dual adapts well to different traditional autonomous driving pipelines, with both tested variants achieving nearly identical performance.A standalone MLP method shows a notable performance gap compared with VAD.
- 5.4 Qualitative Results: Qualitative examples show DriveVLM recognizing cyclist interactions, traffic-police gestures, and a nearby tricycle when making driving decisions.Figure 4 visualizes planned future trajectories for the next 3 seconds.
6 Onboard Deployment and Testing
The onboard deployment combines a high-frequency end-to-end driving system with DriveVLM on separate OrinX processors. Deployment experiments address model size, visual encoding, token compression, temporal context, and speculative sampling for real-time inference.
- 6 Onboard Deployment and Testing: DriveVLM-Dual runs on an autonomous vehicle with the end-to-end driving system and DriveVLM operating asynchronously on two OrinX processors.DriveVLM is optimized to an average inference speed of 410 ms on OrinX.
- 6 Onboard Deployment and Testing: Vehicle hardware limitations motivate using LLMs with fewer than 4 billion parameters, with wide-and-shallow Qwen models outperforming narrow-and-deep models in OrinX inference speed.The comparison targets real-time inference under limited memory and bandwidth.
- 6 Onboard Deployment and Testing: SigLIP-L-384 with position-embedding interpolation is selected to provide high-resolution input for real-time inference.The design is compared with alternative strategies for scaling the ViT input resolution.
- 6 Onboard Deployment and Testing: LDPNetv2 reduces image tokens by 75% without compromising performance, improving the trade-off between computational cost and speed.Replacing average pooling with a convolution layer further enhances performance.
- 6 Onboard Deployment and Testing: A short-term memory bank fuses current visual features with historical frames to provide temporal context for assessing object-motion changes.The approach also uses spatiotemporal pooling and SE blocks for weighted temporal fusion.
- 6 Onboard Deployment and Testing: 2.7 × decode-latency speedup from Eagle exceeds Medusa’s 2.17 × speedup in the OrinX speculative-sampling comparison.Speculative sampling accelerates prediction generation without substantial accuracy loss.
7 Conclusion
The paper concludes that DriveVLM advances VLM-based interpretation of complex driving environments, while DriveVLM-Dual addresses VLM spatial-reasoning and computational challenges through traditional autonomous-driving components. The systems surpass state-of-the-art methods on public and proposed benchmarks and are verified through production-vehicle deployment.
- 7 Conclusion: DriveVLM uses VLMs to interpret complex driving environments, and DriveVLM-Dual synergizes existing 3D perception and planning approaches.The dual design addresses spatial reasoning and computational challenges inherent in VLMs.
- 7 Conclusion: DriveVLM and DriveVLM-Dual surpass state-of-the-art methods on public and proposed benchmarks, especially in intricate and dynamic scenarios.The conclusion also reports onboard deployment and testing on a production vehicle.
- 7 Conclusion: The paper defines a scene-understanding-for-planning task with evaluation metrics, a dataset, and a construction protocol.These components support benchmark evaluation of the proposed driving systems.
A SUP-AD Dataset
SUP-AD contains 1,000 driving video clips spanning more than 40 scenario categories, with manually annotated meta-action sequences representing complete future driving strategies. Its examples cover diverse hazards and interaction conditions requiring varied speed, turning, and lane-control responses.
- Meta-action statistics: Most scenes contain two or three meta-actions, and the distributions of first, second, and third actions are reported.These statistics indicate varied action sequences across the dataset.
- Meta-action annotation: Meta-actions are manually annotated from future driving strategies and structured to remain consistent with the ego vehicle’s future trajectory.The 17 categories comprise speed-control, turning, and lane-control actions.
- Dataset scope: 1,000 video clips span more than 40 driving-scenario categories in SUP-AD.The dataset includes conditions such as mixed traffic, road construction, narrow roads, animals, weather, and unusual obstacles.
- Scenario examples: The dataset includes long-tail situations such as fallen bicycles, slow-moving herds, collapsed trees, narrow bridges, barrier gates, and snowfall.The examples require lane changes, cautious or slow driving, immediate deceleration, uninterrupted crossing, stopping, or following tire tracks.
B Evaluation Method
DriveVLM evaluation focuses on whether an autonomous driving system can interpret driving scenes accurately and make logical, suitable decisions. The evaluation therefore covers scene description/analysis and meta-actions.
- Evaluation scope: Evaluation covers two components: scene description/analysis and meta-actions.These components assess scene interpretation and driving-decision representation.
B.1 Scene Description/Analysis Evaluation
Scene description evaluation addresses the subjectivity of textual descriptions by using GPT-4 to compare generated descriptions with manually annotated ground truth. The process extracts and scores matched information while penalizing hallucinated content.
- GPT-4 evaluation: GPT-4 evaluates generated scene descriptions against manually annotated ground truth because equivalent scenes can have different valid textual descriptions.The method is designed for both structured and unstructured descriptions.
- Evaluation content: Ground-truth evaluation covers environmental conditions and event summaries of critical objects and their influence.Environmental conditions include weather, time, road, and lane descriptions.
- Scoring procedure: The evaluator extracts unique key information, compares it between descriptions, and aggregates the resulting scores into a scene description score.The scoring prompt specifies the comparison inputs, criteria, and structured output format.
B.2 Meta-action Evaluation
Meta-action sequences are evaluated with dynamic programming that accounts for both matched-action quantity and sequence order. The method reduces penalties for conservative actions and recognizes alternative sequences with similar driving semantics.
- Dynamic-programming evaluation: Dynamic programming compares model and ground-truth meta-action sequences while considering matched quantity and sequential arrangement.The approach resembles longest-common-subsequence matching and stores scores in a two-dimensional matrix.
- Action weighting: Conservative actions such as “Slow Down”, “Wait”, and “Go Straight Slowly” receive reduced mismatch penalties.These actions modify driving assertiveness or caution without changing the basic semantic essence of a decision as strongly as turning or lane-changing actions.
- Semantic equivalence: The evaluator can treat different meta-action sequences as semantically equivalent when they represent valid alternatives for the same maneuver.The paper illustrates this with two overtaking sequences differing in acceleration intensity.
- Dynamic-programming evaluation: The dynamic-programming matrix tracks optimal scores across reference and model actions, including missing and redundant-action transitions.The final action score is normalized by the length of the selected reference sequence.
C Qualitative Results
Qualitative visualizations show DriveVLM recognizing unconventional vehicles, fallen trees, traffic-police gestures, and roundabout environments while producing corresponding driving decisions or trajectories.
- DriveVLM recognizes a slowly moving vehicle ahead and decides to change lanes for overtaking.
- DriveVLM identifies unconventional vehicle types and a fallen tree, demonstrating recognition of long-tail objects.The fallen tree is localized precisely, enabling an appropriate detour trajectory.
- DriveVLM accurately captures a traffic police officer’s hand gesture signaling vehicles to proceed.
- DriveVLM recognizes a roundabout road environment and generates a planned trajectory with a curved path.