DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

Motivation

• Existing systems struggle with rare objects, unique attributes, and decision-level interactions because their components focus largely on familiar-object tracking and trajectory-level actions.The paper identifies scene understanding as a central hurdle in complex, unpredictable scenarios such as adverse weather and intricate road layouts.Source: §1 · p. 1

DriveVLM

• DriveVLM uses scene description, scene analysis, and hierarchical planning to address complex autonomous-driving scene understanding.The modules progress from linguistically describing the environment and critical objects to analyzing their influence and producing step-by-step plans.Source: §1 · pp. 1–2

Results

• Experiments on nuScenes and SUP-AD report superior performance, while DriveVLM-Dual exceeds state-of-the-art end-to-end motion-planning methods.The reported gains are especially notable in few-shot scenarios and complex or unpredictable driving conditions.Source: §1 · p. 2

DriveVLM-Dual

• DriveVLM-Dual combines DriveVLM with a traditional autonomous-driving pipeline to improve spatial reasoning and real-time planning.The hybrid can integrate 3D perception and planning modules, including object detectors, occupancy networks, and motion planners.Source: §1 · p. 2

Evaluation resources

• The paper defines the scene understanding and planning task, proposes evaluation metrics, and constructs the SUP-AD dataset through data mining and annotation.These resources assess scene analysis and meta-action planning capabilities.Source: §1 · p. 2

Deployment

• DriveVLM-Dual was deployed and tested on a production vehicle, verifying effectiveness in real-world autonomous-driving environments.The deployment included strategies for accelerating VLM inference in real driving scenarios.Source: §1 · p. 2

Vision-Language Models

• VLMs have been applied in robotics to produce high-level instructions or low-level robot actions.Examples include MiniGPT-4, LLaVA, and Qwen-VL among broader large VLM research.Source: §2 · p. 2

Learning-based Planning

• Learning-based planning methods use reinforcement learning or imitation learning to map raw sensory inputs directly to control actions.Other approaches improve interpretability by constructing dense cost maps from learned models.Source: §2 · p. 2

Driving Caption Datasets

• Driving-caption datasets connect language with driving objectives through object references, free-form captions, question answering, and action explanations.The cited datasets include Refer-KITTI, Talk2Car, NuPrompt, nuScenes-QA, BDD-X, and BDD-OIA.Source: §2 · p. 2

Positioning

• Figure 1 contrasts DriveVLM’s VLM reasoning pipeline with DriveVLM-Dual’s addition of traditional 3D perception and trajectory planning.The dual design targets spatial reasoning and real-time trajectory planning.Source: Figure 1 · p. 3

Driving Caption Datasets

• Existing language-annotated driving datasets lack sufficient data on critical safety scenarios in self-driving systems.This limitation bounds their coverage of safety-relevant driving conditions.Source: §2 · p. 3

DriveVLM

• DriveVLM processes image sequences through scene description, scene analysis, and hierarchical planning modules.Its architecture combines a vision transformer encoder, an attention-based extractor, and a large language model.Source: §3.1 · p. 3

Scene Description

• The scene description module characterizes driving environments and identifies critical objects likely to influence the current scenario.Environment conditions include weather, time, road, and lane information; critical objects include categories and approximate image bounding boxes.Source: §3.2 · pp. 3–4

Scene Analysis

• DriveVLM analyzes critical objects through static attributes, motion states, or particular behaviors, then predicts each object’s influence on the ego vehicle.Only the characteristics applicable to a given critical object need to be analyzed.Source: §3.3 · p. 4

Hierarchical Planning

• Hierarchical planning combines the scene summary with route, ego pose, and velocity to generate meta-actions, decision descriptions, and trajectory waypoints.Decision descriptions specify an action, interacting subject, and duration; waypoints represent the future vehicle path at fixed intervals.Source: §3.4 · p. 4

4.1 Task Definition

• The Scene Understanding for Planning task takes multi-view surrounding-camera videos and optionally 3D perception results as input.Its outputs include scene description, scene analysis, meta-actions, decision description, and trajectory waypoints.Source: §4.1 · pp. 5–6

4.2 Evaluation

• Evaluation measures both interpretation of the driving scene and the decisions made.Scene description and analysis use structured comparison with human annotations, while meta-actions are compared with annotated sequences using dynamic programming.Source: §4.2 · p. 6

4.3 Dataset Construction

• The SUP-AD construction pipeline mines long-tail objects and challenging scenarios, selects a keyframe, and annotates scene understanding and planning information.Long-tail examples include unusual vehicles, road debris, and animals; challenging scenarios are selected using variance in recorded driving maneuvers.Source: §4.3 · pp. 6–7

Keyframe Selection

• Keyframes in challenging scenes are selected 0.5s to 1s before a required speed or direction change to provide reaction time for decision-making.Scenes without behavioral changes use a frame relevant to the current driving scenario.Source: §4.3 · p. 7

Scene Annotation

• Scene annotations cover description, analysis, and planning, while trajectory waypoints can be auto-labeled from vehicle IMU recordings.The annotation tool supports video replay, bounding-box drawing with language descriptions, and selection from action and decision candidates.Source: §4.3 · p. 7

5.1 Settings

• DriveVLM is evaluated on the SUP-AD test set, while DriveVLM-Dual is evaluated on nuScenes planning.The experiments also include module ablations, traditional-pipeline comparisons, and qualitative results.Source: §5.1 · p. 7

5.2 Main Results

• DriveVLM with Qwen-VL achieves the best SUP-AD performance among the compared vision-language models.GPT-4V uses in-context learning and its extraneous scene-description information is often classified as hallucination under the evaluation metric.Source: §5.2 · p. 8, Table 1 · p. 8
• DriveVLM-Dual achieves the best performance on the nuScenes planning task when cooperating with VAD.The result indicates that the system performs well on ordinary scenarios as well as complex-scene understanding.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 Ablation Study

• Critical object analysis and 3D perception are ablated as design choices in the nuScenes validation experiments.Critical object analysis prioritizes important driving-environment elements, while 3D perception provides refined surroundings understanding and precise predictions.Source: §5.3 · pp. 8–9, Table 3 · p. 9
• DriveVLM-Dual adapts well to different traditional autonomous driving pipelines, with both tested variants achieving nearly identical performance.A standalone MLP method shows a notable performance gap compared with VAD.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 Qualitative Results

• Qualitative examples show DriveVLM recognizing cyclist interactions, traffic-police gestures, and a nearby tricycle when making driving decisions.Figure 4 visualizes planned future trajectories for the next 3 seconds.Source: §5.4 · p. 9, Figure 4 · p. 9

6 Onboard Deployment and Testing

• DriveVLM-Dual runs on an autonomous vehicle with the end-to-end driving system and DriveVLM operating asynchronously on two OrinX processors.DriveVLM is optimized to an average inference speed of 410 ms on OrinX.Source: §6 · p. 10
• Vehicle hardware limitations motivate using LLMs with fewer than 4 billion parameters, with wide-and-shallow Qwen models outperforming narrow-and-deep models in OrinX inference speed.The comparison targets real-time inference under limited memory and bandwidth.Source: §6 · p. 10
• SigLIP-L-384 with position-embedding interpolation is selected to provide high-resolution input for real-time inference.The design is compared with alternative strategies for scaling the ViT input resolution.Source: Table 7 · p. 10, §6 · p. 11
• LDPNetv2 reduces image tokens by 75% without compromising performance, improving the trade-off between computational cost and speed.Replacing average pooling with a convolution layer further enhances performance.Source: §6 · p. 11, Table 8 · p. 11
• A short-term memory bank fuses current visual features with historical frames to provide temporal context for assessing object-motion changes.The approach also uses spatiotemporal pooling and SE blocks for weighted temporal fusion.Source: §6 · p. 11
• 2.7 × decode-latency speedup from Eagle exceeds Medusa’s 2.17 × speedup in the OrinX speculative-sampling comparison.Speculative sampling accelerates prediction generation without substantial accuracy loss.Source: §6 · p. 11, Table 9 · p. 11

7 Conclusion

• DriveVLM uses VLMs to interpret complex driving environments, and DriveVLM-Dual synergizes existing 3D perception and planning approaches.The dual design addresses spatial reasoning and computational challenges inherent in VLMs.Source: §7 · pp. 11–12
• DriveVLM and DriveVLM-Dual surpass state-of-the-art methods on public and proposed benchmarks, especially in intricate and dynamic scenarios.The conclusion also reports onboard deployment and testing on a production vehicle.Source: §7 · p. 12
• The paper defines a scene-understanding-for-planning task with evaluation metrics, a dataset, and a construction protocol.These components support benchmark evaluation of the proposed driving systems.Source: §7 · p. 12

Meta-action statistics

• Most scenes contain two or three meta-actions, and the distributions of first, second, and third actions are reported.These statistics indicate varied action sequences across the dataset.Source: §A.1 · p. 17, Figure 5 · p. 18, Figure 6 · p. 18

Meta-action annotation

• Meta-actions are manually annotated from future driving strategies and structured to remain consistent with the ego vehicle’s future trajectory.The 17 categories comprise speed-control, turning, and lane-control actions.Source: §A.1 · p. 17

Dataset scope

• 1,000 video clips span more than 40 driving-scenario categories in SUP-AD.The dataset includes conditions such as mixed traffic, road construction, narrow roads, animals, weather, and unusual obstacles.Source: §A.2 · pp. 17, 19, 20

Scenario examples

• The dataset includes long-tail situations such as fallen bicycles, slow-moving herds, collapsed trees, narrow bridges, barrier gates, and snowfall.The examples require lane changes, cautious or slow driving, immediate deceleration, uninterrupted crossing, stopping, or following tire tracks.Source: §A.3 · p. 20, Figure 8 · p. 20, Figure 9 · p. 20

Evaluation scope

• Evaluation covers two components: scene description/analysis and meta-actions.These components assess scene interpretation and driving-decision representation.Source: B Evaluation Method · p. 22

GPT-4 evaluation

• GPT-4 evaluates generated scene descriptions against manually annotated ground truth because equivalent scenes can have different valid textual descriptions.The method is designed for both structured and unstructured descriptions.Source: §B.1 · p. 22, Table 10 · p. 23

Evaluation content

• Ground-truth evaluation covers environmental conditions and event summaries of critical objects and their influence.Environmental conditions include weather, time, road, and lane descriptions.Source: §B.1 · p. 22

Scoring procedure

• The evaluator extracts unique key information, compares it between descriptions, and aggregates the resulting scores into a scene description score.The scoring prompt specifies the comparison inputs, criteria, and structured output format.Source: §B.1 · pp. 22–23

Dynamic-programming evaluation

• Dynamic programming compares model and ground-truth meta-action sequences while considering matched quantity and sequential arrangement.The approach resembles longest-common-subsequence matching and stores scores in a two-dimensional matrix.Source: §B.2 · pp. 23–24

Action weighting

• Conservative actions such as “Slow Down”, “Wait”, and “Go Straight Slowly” receive reduced mismatch penalties.These actions modify driving assertiveness or caution without changing the basic semantic essence of a decision as strongly as turning or lane-changing actions.Source: §B.2 · pp. 23–24

Semantic equivalence

• The evaluator can treat different meta-action sequences as semantically equivalent when they represent valid alternatives for the same maneuver.The paper illustrates this with two overtaking sequences differing in acceleration intensity.Source: §B.2 · p. 24, Table 11 · p. 24

Dynamic-programming evaluation

• The dynamic-programming matrix tracks optimal scores across reference and model actions, including missing and redundant-action transitions.The final action score is normalized by the length of the selected reference sequence.Source: §B.2 · pp. 24–25

• DriveVLM recognizes a slowly moving vehicle ahead and decides to change lanes for overtaking.Source: C Qualitative Results · p. 25, Figure 14 · p. 26
• DriveVLM identifies unconventional vehicle types and a fallen tree, demonstrating recognition of long-tail objects.The fallen tree is localized precisely, enabling an appropriate detour trajectory.Source: C Qualitative Results · p. 25, Figure 15 · p. 27, Figure 16 · p. 28
• DriveVLM accurately captures a traffic police officer’s hand gesture signaling vehicles to proceed.Source: Figure 17 · p. 29
• DriveVLM recognizes a roundabout road environment and generates a planned trajectory with a curved path.Source: Figure 18 · p. 30