Vision Language Models in Autonomous Driving: A Survey and Outlook

• Current autonomous-driving systems struggle with complex dynamic environments, decision explanation, and human instructions, often missing details and context.Source: §I · p. 1
• VLMs combine visual and linguistic data to support traditional AD tasks and emerging applications such as zero-shot perception and language-guided navigation.Source: §I · p. 1
• The survey categorizes existing VLM studies by model types and application domains in autonomous driving.Source: §I · p. 2
• It consolidates mainstream vision-language tasks and their commonly used metrics for autonomous driving.Source: §I · p. 2
• The paper systematically summarizes classic and language-enhanced autonomous-driving datasets.Source: §I · p. 2
• It discusses VLM applications and technological advances, alongside benefits, challenges, and research gaps in autonomous driving.Source: §I · p. 2

• SAE levels range from Level 0 no automation to Level 5 full automation, with higher autonomy reducing human intervention and increasing environmental-understanding requirements.Source: A. Autonomous Driving · p. 2
• Modular autonomous driving separates perception, prediction, planning, and control into independently developed components.Source: A. Autonomous Driving · p. 2
• Perception processes sensor data from cameras, LiDAR, radar, or event-based cameras to extract traffic participants’ locations, sizes, velocities, and related details.Source: A. Autonomous Driving · p. 2
• Planning generates paths using global route search and local real-time adjustments based on vehicle state and environmental information.Source: A. Autonomous Driving · pp. 2–3
• End-to-end autonomous driving integrates separate components into one differentiable system mapping raw sensor inputs directly to control signals.Source: A. Autonomous Driving · p. 3
• Both modular and end-to-end systems face challenges in generalization, interpretability, causal confusion, and robustness.Source: A. Autonomous Driving · p. 3

• Transformers and BERT established scalable, context-aware language representations that support the development of large models.Source: B. Large Language Models · p. 3
• VLMs connect modalities through vision-text matching or vision-text fusion, with fused features supporting downstream tasks.Source: Figure 3 · p. 3, C. Vision-Language Models · p. 4
• Large language models typically contain billions of parameters and exhibit few-shot, zero-shot, and multi-step reasoning abilities.Source: B. Large Language Models · p. 3
• Vision-language models bridge language and computer vision by learning relationships between visual content and natural language.Source: C. Vision-Language Models · p. 3
• Autonomous-driving VLMs are categorized as Multimodal-to-Text, Multimodal-to-Vision, or Vision-to-Text according to their input-output modalities.Source: C. Vision-Language Models · pp. 3–4, Figure 4 · p. 4

• The survey organizes autonomous-driving vision-language tasks into object referring and tracking, open-vocabulary perception, scene understanding, language-guided navigation, and conditional data generation.Source: §III · p. 4

Object Referring and Tracking

• Object referring localizes one or more language-specified objects in 2D images or 3D spaces, generalizing conventional object detection.Source: A. Object Referring and Tracking · p. 4
• Referred object tracking uses language expressions to track specified objects across consecutive video frames, emphasizing referring consistency and robustness.Source: A. Object Referring and Tracking · p. 4, Figure 5 · p. 5
• Object referring is evaluated with mAP, while multiple-object referring and tracking adopts HOTA, MOTA, MOTP, and IDS.Source: A. Object Referring and Tracking · pp. 4–5

Open-Vocabulary 3D Object Detection

• Open-vocabulary 3D object detection identifies novel categories absent from training and assigns text-defined semantic labels to 3D bounding boxes.Source: B. Open-Vocabulary Traffic Environment… · p. 5

Open-Vocabulary 3D Semantic Segmentation

• Open-vocabulary 3D segmentation classifies point-cloud or mesh regions into semantically meaningful categories unseen during training.Source: B. Open-Vocabulary Traffic Environment… · p. 5

Open-Vocabulary Traffic Environment Perception

• OV-3D detection uses 3D mAP, whereas OV-3D segmentation uses mIOU and mean Accuracy for evaluation.Source: B. Open-Vocabulary Traffic Environment… · p. 5

Traffic Scene Understanding

• Traffic-scene VQA answers questions from images or video across perception, planning, spatial, temporal, and causal reasoning categories.Source: C. Traffic Scene Understanding · p. 5
• Captioning generates textual descriptions of scenes or objects and can cover scene description, importance ranking, and action explanation.Source: C. Traffic Scene Understanding · pp. 5–6
• Multiple-choice VQA uses Top-N Accuracy, while open-ended VQA and captioning commonly use BLEU, METEOR, ROUGE, and CIDEr.Source: C. Traffic Scene Understanding · p. 6
• Because these metrics have different focuses and limitations, studies often combine them and may evaluate reasoning chains or semantic similarity with language models.Source: C. Traffic Scene Understanding · p. 6

Task Description

• Language-guided navigation makes reasoned plans to reach specified locations from natural-language instructions in complex, variable, and uncertain outdoor scenes.Source: D. Language-Guided Navigation · p. 6
• The task commonly includes language-guided localization and path planning, with target-location understanding as a prerequisite for successful navigation.Source: D. Language-Guided Navigation · p. 6
• Existing studies often use Vision-Text Matching VLMs to locate instruction-specified destinations among landmarks in maps.Source: D. Language-Guided Navigation · p. 6

Evaluation Metrics

• Evaluation considers task completion, target-localization accuracy, and deviation between the target location and final arrival.Source: D. Language-Guided Navigation · p. 6

Conditional Autonomous Driving Data Generation

• Conditional driving-data generation produces targeted photorealistic synthetic data to diversify and purposefully augment autonomous-driving training databases.Source: E. Conditional Autonomous Driving Data… · pp. 6–7
• Generation systems can condition outputs on text prompts, BEV masks, action states, bounding boxes, or HD maps for single- or multiple-view images and videos.Source: E. Conditional Autonomous Driving Data… · p. 7

Evaluation Metrics

• FID and FVD evaluate conditional image and video generation, while CLIP score measures visual consistency across generated multiple street views.Source: E. Conditional Autonomous Driving Data… · p. 7

Perception and Understanding

• VLM-based perception research spans object referring, open-vocabulary 3D detection and segmentation, language-guided retrieval, driving scene understanding, and visual scene reasoning.Table I summarizes these task categories alongside related LLM and VLM types.Source: §IV · p. 7, Table I · p. 9

Open-Vocabulary Perception

• OpenScene and CLIP2Scene align image, text, and point-cloud representations to support zero-shot open-vocabulary 3D scene understanding.UP-VL extends this direction with unsupervised multimodal auto-labeling for class-agnostic 3D detection and tracking.Source: A. Perception and Understanding · pp. 7–8

Language-Guided Retrieval

• Language-guided retrieval uses semantic similarity, BEV features, language augmentation, and multiple imperfect descriptions for scene or vehicle retrieval.Applications include data selection, corner-case retrieval, and natural-language vehicle search.Source: A. Perception and Understanding · p. 8

Driving Scene Understanding

• Traffic VLM studies address image captioning, anomaly recognition, visual question answering, temporal reasoning, and traffic-domain knowledge injection through synthetic captions.NuScenes-QA and NuScenes-MQA provide question-answering benchmarks based on nuScenes, while traffic video-caption pairs support fine-tuning.Source: A. Perception and Understanding · pp. 8–10

Pedestrian Detection

• VLM-based pedestrian detection uses CLIP-derived pixel-wise semantic contexts and pseudo-labeling to address human-like object confusion and scarce border-case samples.The cited approach is designed as an extra-annotation-free method.Source: A. Perception and Understanding · p. 10

Research Status

• The surveyed perception and understanding applications remain in early stages despite covering increasingly diverse language-enhanced capabilities.The survey identifies these directions as promising areas for further work.Source: A. Perception and Understanding · p. 10

Language-Guided Navigation

• CLIP-driven cross-modal alignment broadens language-guided navigation from predefined locations to free and arbitrary instructions.This development also promotes language-augmented maps.Source: B. Navigation and Planning · p. 10
• Talk to the Vehicle maps semantic occupancy and natural-language encodings to local waypoints, which a planning module converts into an executable trajectory.The waypoint generator network performs the language-and-scene mapping.Source: B. Navigation and Planning · p. 10

Prediction and Planning

• GPT-driver reformulates motion planning as language modeling, while CoverNet-T jointly encodes text-based scene descriptions and rasterized scene images for planning-related prediction.These approaches leverage language models or multimodal representations for motion planning and trajectory prediction.Source: B. Navigation and Planning · p. 10

Decision Making

• Decision-making studies use LLMs to assist or emulate drivers and to handle complex scenarios requiring human commonsense understanding.Closed-loop systems commonly add memory modules for driving scenarios, experiences, and decision information.Source: C. Decision-Making and Control · p. 11
• LanguageMPC uses an LLM for complex-scene decisions, while Drive as You Speak enables direct driver–vehicle communication through an orchestrated LLM framework.Drive as You Speak stores past scenario experiences, decision cues, and reasoning processes in a vector database.Source: C. Decision-Making and Control · p. 11

Unified Representations

• BEVGPT unifies scenario prediction, decision-making, and motion planning through a single LLM with BEV input, while DwLLMs generates driving actions from object-level scene vectors.DwLLMs uses two-stage pre-training and fine-tuning to understand driving scenarios and produce actions.Source: C. Decision-Making and Control · p. 11

Vision-LLM Integration

• Most existing decision-making and control studies rely on LLMs alone, but visual-LLM connectors can link them with perception for mid-to-mid or end-to-end AD.The survey presents connector design tailored to AD as an open research direction.Source: C. Decision-Making and Control · p. 11

End-to-End AD System

• End-to-end AD integrates separate driving components into a differentiable system that maps raw sensor data directly to plans or control signals.This structure aligns naturally with multimodal-to-text VLMs.Source: D. End-to-End Autonomous Driving · p. 11

End2End AD System

• DriveGPT4, ADAPT, DriveMLM, and VLP demonstrate end-to-end systems that combine sensor inputs with questions, rules, commands, or context to produce controls and explanations.ADAPT continuously outputs control signals, narration, and reasoning descriptions, while DriveMLM supports closed-loop driving in Town05Long.Source: D. End-to-End Autonomous Driving · p. 11

Open Challenges

• Autoregressive large VLMs and LLMs introduce inherent limitations, leaving many end-to-end AD issues requiring further consideration and resolution.The survey therefore treats current demonstrations as feasibility evidence rather than a completed solution.Source: D. End-to-End Autonomous Driving · p. 11

Conditional Data Generation

• Conditional generation methods produce large-scale, high-quality autonomous-driving data from controllable visual or textual conditions, supporting data-driven AD development.DrivingDiffusion demonstrates conditional multi-view video generation with layout and text control.Source: Figure 8 · p. 11, E. Data Generation · p. 12
• DriveGAN controls generated vehicle behavior by disentangling action-dependent and action-independent scene features for high-fidelity neural simulation and data generation.BEVControl uses sketch-style BEV layouts and text prompts to generate street-view multi-view images.Source: E. Data Generation · p. 12

World Models

• World-model approaches learn structured driving-environment representations from video, action, and text to generate predictable or realistic future scenes.DriveDreamer, GAIA-1, and ADriver-I respectively model real-world scenarios, decode token sequences with video diffusion, or fuse vision-action pairs.Source: E. Data Generation · p. 12

• Autonomous-driving datasets underpin the development of safe and efficient perception, prediction, and planning systems.Source: A. Autonomous Driving Datasets · p. 12
• Mainstream datasets cover tasks including object detection, tracking, and segmentation across multiple data modalities.Source: A. Autonomous Driving Datasets · p. 12
• Language-enhanced datasets extend visual data with natural-language questions, instructions, and descriptions for richer scene comprehension and human–vehicle interaction.Source: B. Language-Enhanced AD Datasets · p. 12, §V · p. 12
• Table III defines dataset task abbreviations including SS, OT, REID, MP, AE, VSR, VLN, VQA, DM, IC, IR, and related categories.Source: Table III · p. 13
• Language-oriented datasets support tracking, referring, visual question answering, spatial reasoning, navigation, and related autonomous-driving tasks.Examples include CityFlow-NL, Refer-KITTI, NuPrompt, Talk2Car, Talk2BEV, and language-guided navigation datasets.Source: B. Language-Enhanced AD Datasets · pp. 12–13

Advantages of large VLMs in AD

• Large VLMs extend classic autonomous-driving modules—including perception, navigation, planning, and control—and offer a new perspective on end-to-end driving.Source: §VI · pp. 13–14
• Zero-shot generalization can improve recognition of corner cases and unknown objects, while scene–text fusion supports open-ended driving-related question answering.Source: §VI · p. 14
• Large VLMs support language-guided localization, navigation, and planning by aligning cross-modal features and interpreting detailed instructions.Source: §VI · p. 14
• Their high-level reasoning, common-sense knowledge, and traffic-rule understanding offer potential for high-level decision-making and control.Source: §VI · p. 14
• Large VLMs can generate realistic, controllable, high-quality synthetic autonomous-driving data with implicit environmental modeling.Source: §VI · p. 14

Autonomous Driving Foundation Model

• Current foundation-model research motivates Autonomous Driving Foundation Models pretrained on diverse datasets for interpretability, reasoning, forecasting, introspection, and multiple driving tasks.Source: §VI · p. 14

Multi-Modality Adaptation

• Directly converting sensor data into text can lose context and environmental information and makes motion planning, control, and decision-making dependent on perception quality.Source: §VI · p. 14

Public Data Availability and Formatting

• Existing autonomous-driving datasets are not yet optimal for direct LLM adaptation, and instruction-tuning formats and large-scale traffic image–text pairs remain underinvestigated.Source: §VI · p. 14

Computation Demands and Deployment Latency

• Large VLMs impose substantial computation and latency demands because billion-scale models make fine-tuning and inference resource-intensive.Source: §VII · p. 15
• On-vehicle deployment offers low latency and local data storage, whereas cloud deployment offers centralized computation but introduces latency, privacy, and network-dependency risks.Source: §VII · p. 15
• A hybrid architecture could place critical real-time decisions on vehicles and data analytics or model updates in the cloud, but software and hardware requirements remain open questions.Source: §VII · p. 15

Temporal Scene Understanding

• Image-level VLMs are insufficient for temporal scene understanding because driving dynamics and accident causality require video information.Source: §VII · p. 15
• Potential remedies include training video-language models or adding temporal adapters while converting video into image-language processing formats.Source: §VII · p. 15

Ethical and Societal Concern in Driving Safety

• Unfiltered pretraining data can produce toxic, biased, or harmful content, while language-model hallucinations challenge driving-system safety and reliability.Source: §VII · p. 15

Data Privacy and Security

• Sensitive driving data raises collection, usage, storage, and privacy concerns, while vehicle and adversarial attacks can manipulate systems or cause decision misjudgments.Source: §VII · p. 15

Conclusion

• The survey synthesizes challenges spanning computation and latency, temporal understanding, ethical and societal safety, and data privacy and security.Source: §VII · p. 15, §VIII · p. 16