Source-linked AI summary
Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives
Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, Liang Pan
TL;DR
The paper examines whether VLM driving explanations are truly grounded in visual cues rather than general knowledge or textual cues. It introduces DriveBench to test reliability across clean, corrupted, and text-only conditions, finding that plausible unsupported responses and evaluation biases can overstate reliability. The authors therefore emphasize balanced, context-aware datasets and metrics that assess visual grounding, reasoning, and safety.
Problem
Whether VLM driving decisions are reliably grounded in visual cues remains underexplored despite growing use of VLMs for interpretable autonomous-driving interactions.
Method
DriveBench evaluates VLM reliability across four driving tasks, 15 corruption types, 19,200 images, and 20,498 question-answer pairs, including text-only prompts.
Results
VLMs frequently generate plausible unsupported responses under severe visual degradation, while dataset imbalance and suboptimal evaluation protocols can overestimate their reliability.
Takeaways & Limitations
Reliable assessment requires well-balanced, context-aware datasets and advanced metrics that evaluate visual grounding, contextual reasoning, and safety.
Takeaways & Limitations
The findings rely exclusively on the DriveLM dataset, which may limit generalizability to other driving benchmarks or real-world settings.
Abstract
from arXiv · showhide
Recent advancements in Vision-Language Models (VLMs) have sparked interest in their use for autonomous driving, particularly in generating interpretable driving decisions through natural language. However, the assumption that VLMs inherently provide visually grounded, reliable, and interpretable explanations for driving remains largely unexamined. To address this gap, we introduce DriveBench, a benchmark dataset designed to evaluate VLM reliability across 17 settings (clean, corrupted, and text-only inputs), encompassing 19,200 frames, 20,498 question-answer pairs, three question types, four mainstream driving tasks, and a total of 12 popular VLMs. Our findings reveal that VLMs often generate plausible responses derived from general knowledge or textual cues rather than true visual grounding, especially under degraded or missing visual inputs. This behavior, concealed by dataset imbalances and insufficient evaluation metrics, poses significant risks in safety-critical scenarios like autonomous driving. We further observe that VLMs struggle with multi-modal reasoning and display heightened sensitivity to input corruptions, leading to inconsistencies in performance. To address these challenges, we propose refined evaluation metrics that prioritize robust visual grounding and multi-modal understanding. Additionally, we highlight the potential of leveraging VLMs' awareness of corruptions to enhance their reliability, offering a roadmap for developing more trustworthy and interpretable decision-making systems in real-world autonomous driving contexts. The benchmark toolkit is publicly accessible.
1. Introduction
This section questions whether VLM driving explanations are genuinely grounded in visual cues and evaluates their reliability, data biases, and metrics. It reports that plausible answers can persist without visual information, while existing datasets and evaluation protocols can conceal this failure.
- The study asks whether VLM-generated driving decisions reflect sensory information or general knowledge and fabricated responses from textual cues.
- Comparable performance under clean and text-only inputs can result from imbalanced datasets and suboptimal evaluation protocols rather than inherent robustness.
- VLMs often produce plausible yet fabricated responses under degraded visual conditions, including scenarios without visual cues, making reliability difficult to detect.
- VLMs show some corruption awareness but generally acknowledge visual degradation only when explicitly prompted, limiting autonomous reliability assessment.
- Traditional language metrics and GPT scores may miss reasoning, context, and safety requirements in autonomous-driving tasks, motivating tailored evaluation metrics.
2. Related Work
Prior work uses VLMs for language-mediated driving decisions and multimodal interaction, but their robustness in complex real-world tasks remains largely untested. The related-work discussion also highlights visual-input variation and behavior-distribution imbalance in existing benchmarks.
- VLM research in autonomous driving spans end-to-end frameworks and natural-language interactions intended to make decisions more interpretable.
- Existing VLM driving systems interact with environments through multimodal visual and language inputs, extending earlier language-based simulator approaches.
- Robustness and reliability across complex driving tasks and different visual inputs remain largely untested, particularly under visual corruptions and anomalies.
- DriveLM-nuScenes behavior distributions are dominated by “Going Ahead” actions, creating a dataset imbalance relevant to evaluating driving models.
3. DriveBench: Driving with VLMs
DriveBench is constructed to evaluate VLM reliability in autonomous driving across representative tasks, corrupted inputs, and multiple evaluation settings. Its design targets visual grounding, dataset bias, and robustness to realistic corruption scenarios.
- Benchmark construction: DriveBench builds on driving-with-language benchmarks and covers perception, prediction, planning, and behavior tasks.The tasks address object status and surroundings, future object movement, safe navigation actions, and future ego-vehicle steering and speed.
- Dataset construction: Dataset selection removes samples highly dependent on temporal context or difficult visual interpretation to retain cases with sufficient single-frame cues.The strategy prioritizes instances that GPT-4o can correctly interpret and separately analyzes other GPT-4o failure cases.
- Failure analysis: VLM failure cases include misreading object coordinates, orientation, relative frame position, and examples requiring multiple frames, while corruption evaluation compares MCQ accuracy against clean inputs.Figure 8 reports that human performance largely decreases after corruption whereas most VLMs remain unchanged.
4. Experiments
Experiments show that VLMs can retain apparently strong performance under corruption or text-only inputs by relying on textual cues and general knowledge rather than visual grounding. Corruption-aware prompting exposes model uncertainty, while dataset and metric choices can conceal these reliability problems.
- 4.2.1. Corruption Resilience: GPT-4o retains approximately 95% of its clean-image performance under text-only conditions for open-ended questions.This result indicates that strong performance can persist even when visual cues are absent.
- 4.2.1. Corruption Resilience: Text-only performance is likely driven by general knowledge and textual cues, enabling plausible answers without visual evidence.Models can infer an object’s motion from references to camera identity and image position rather than directly using visual information.
- 4.2.2. Corruption Awareness: Explicit corruption references reduce accuracy for some models, with LLaVA-NeXT7B declining by approximately 19.62%.Corruption prompts also increase uncertainty and can lead models to acknowledge that the image is unreliable.
- 4.2.1. Corruption Resilience: LLaVA-1.5 consistently answers “Going ahead” across corruption scenarios, indicating responses based on general knowledge rather than visual inputs.This uniform output explains why performance can remain unchanged despite altered visual conditions.
- 4.2.2. Corruption Awareness: Advanced VLMs rely predominantly on text cues under visual degradation, whereas less advanced models depend more strongly on learned general-knowledge priors.The distinction persists even though advanced models show some awareness of corruption.
5. Conclusion
The work identifies visual-grounding reliability as a central challenge for VLM deployment in autonomous driving. VLMs can generate plausible but unsupported responses under severe visual degradation, while dataset imbalance and evaluation protocols further complicate reliability assessment.
- VLMs frequently generate plausible yet unsupported responses under severe visual degradation.This casts doubt on their reliability in critical decision-making tasks.
A. Benchmark Setup
This section details the procedures and protocols used to establish DriveBench.
- The section explains the procedures used to establish DriveBench.
- It specifies the protocols used in benchmark establishment.
- The discussion focuses on how DriveBench was established.
A.1. Benchmark Construction
DriveBench is constructed from balanced DriveLM keyframes and evaluates driving models across curated questions and realistic visual corruptions. Its setup also includes structured GPT-based evaluation information and robustness distributions.
- Benchmark Construction: 200 keyframes are subsampled from DriveLM’s training dataset to balance ground-truth distributions and reduce majority-case bias.Each keyframe contains multiple questions related to different driving tasks.
- Evaluation Protocol: GPT evaluators can revise scores and provide more accurate evaluations when given more contextual information about driving scenarios.The evaluation prompt includes the question, ground-truth description or answer, and model answer, with criteria covering action accuracy, reasoning, confidence, safety, and clarity.
- Corruption Design: 15 corruption types cover weather and lighting, external disturbances, sensor failures, motion blur, and data transmission errors.The corruptions are generated with high-fidelity image-processing algorithms developed in previous work.
- Corruption Design: The benchmark includes five weather and lighting corruptions: Brightness, Dark, Snow, Fog, and Rain.
- Corruption Design: External disturbances simulate camera-lens occlusion by external objects or stains, while sensor failures simulate camera and frame problems.The listed sensor-failure corruptions are Camera Crash, Frame Lost, and Saturate.
- Question Distribution: The curated benchmark contains 1,461 questions, including perception, planning, behavior, and prediction questions with task-specific distributions.Each keyframe has two perception questions, four planning VQA questions, and one behavior multiple-choice question; prediction questions are not available for every keyframe.
B. Benchmark Study
The benchmark study examines dataset distributions in representative driving-with-language datasets and finds that severe action imbalance can make model behavior and accuracy misleading.
- Dataset Context: Representative driving-with-language datasets are presented as resources supporting development of driving-with-language models.
- Dataset Imbalance: Highly imbalanced driving data can cause fine-tuned VLMs to memorize majority behaviors and answer with them despite absent visual cues.In DriveLM-nuScenes, going straight constitutes the majority of vehicle-behavior choices.
- Dataset Imbalance: BDD-X shows similar imbalance, with most car actions labeled Stop or Going Ahead.Random guessing can achieve high accuracy because models may rely on common knowledge and general cases.
C. Additional Implementation Details
This section documents implementation details, including the benchmark’s model descriptions, action-distribution visualization, and shared prompting setup.
- Overview: The implementation details section introduces the benchmark’s model configurations and supporting reproducibility information.The section explicitly frames its purpose as providing implementation details to facilitate reproduction.
- VLM configurations: The candidate models include GPT-4o, Phi-3, Phi-3.5, LLaVA-1.5, LLaVA-NeXT, InterVL, and DriveLM-Agent.The passages describe these models’ multimodal, language, or autonomous-driving capabilities at a high level.
- Dataset distribution: Figure 19 visualizes BDD-X car actions occurring more than 80 times, highlighting stop and turn actions.Turning actions represent only a small portion of all actions in the reported distribution.
- Prompting: All candidate VLMs receive the same system prompt, and MCQ explanations are elicited to support GPT-based evaluation.The shared prompting procedure is intended to make model outputs comparable under the evaluation setup.
C.3. GPT Evaluations
The GPT evaluation setup supplies contextual descriptions and task-specific prompts to assess model answers, including scenarios with multiple camera inputs.
- GPT evaluation prompts: MCQ evaluation prompts use natural-language descriptions of critical objects to provide context from the driving environment.The descriptions are extracted from the current drive-with-language dataset.
- GPT evaluation prompts: The open-ended perception evaluation prompts use predicted and ground-truth information with detailed rubrics.The supplied visual descriptions and moving status of important objects are already included for these questions.
- Inference prompt: The inference prompt supports up to six camera images with normalized coordinates ranging from 0 to 1.The camera sequence includes front, side, and rear views.
C.4. Human Evaluations
Human evaluation samples keyframes while reducing repeated sampling across corruption conditions to limit overlap and information leakage.
- Procedures: Human evaluation subsamples 15 of 200 keyframes from the curated dataset.The evaluation interface focuses on multiple-choice questions and uses single-view images when appropriate.
- Procedures: Sampling probability is lowered for keyframes selected previously under another corruption to avoid overlap.The stated goal is to reduce information leakage between corruption conditions.
- Ethics: The study records no identifiable participant information and reports that no IRB review is required under the stated exemption.The passage identifies Exempt Research 2(i) as applicable.
D. Detailed Experiment Results
Detailed experiments compare GPT scores, accuracy, language metrics, spatial distributions, and corruption examples, revealing that conventional metrics can obscure unreliable or fabricated VLM behavior.
- GPT scores: GPT scores are reported across multiple task tables and form the primary basis for the main-paper observations.The discussion therefore emphasizes accuracy and language scores.
- Accuracy scores: 50% accuracy across all input types for LLaVA-1.5 indicates repeated selection of “Going Ahead” for perception MCQs.The same behavior appears in the prediction spatial distributions.
- ROUGE-L scores: ROUGE-L favors fine-tuned DriveLM outputs, while GPT-4o’s more detailed answers receive lower scores because of answer length.This demonstrates that language-score behavior differs from GPT-based evaluation behavior.
- Corruption robustness: Some VLMs show only subtle GPT-score changes under corruptions, and text-only scores can exceed scores with visual information.The radar graphs average performance across 1,261 questions spanning four tasks.
- Failure cases: Qwen2-VL72B can fabricate perception answers from coordinate text and receive the same GPT score as with a clean image.In planning, missing images can also yield higher scores despite failure to perceive a motorcycle or stop sign.
- Evaluation limitations: GPT evaluation can reward a more detailed clean-image explanation or a less safe prediction under degraded conditions.Examples include GPT-4o under bit errors and low-light planning scenarios.
- Implications: The study argues that reliability and safety evaluations should include contextual understanding, interpretability, and robustness beyond traditional performance metrics.These requirements are framed as important for benchmarks, testbeds, and metrics applied to real-world physical systems.
E.2. Potential Limitations
The study’s findings are constrained by dataset coverage, limited contextual information, language-focused evaluation, and a finite model and task scope.
- The experiments use only the DriveLM dataset, which may limit generalizability to other driving benchmarks and real-world settings.The authors attribute this restriction to the prohibitive computational cost of large-scale VLM inference and GPT-based evaluations.
- Limited visual descriptions of key objects may prevent GPT-based assessments from fully capturing the broader context needed for accurate, nuanced evaluation.The authors suggest richer temporal and spatial context as a way to improve evaluation fidelity.
- The study primarily evaluates language-based explanations from 12 VLMs across specific tasks, metrics, and settings, leaving generalization to action models and emerging scenarios unresolved.The authors identify trajectory prediction, manipulation control, newer architectures, and more diverse driving scenarios as directions for future investigation.