Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future

Tianshuai Hu; Xiaolu Liu; Song Wang; Yiyao Zhu; Ao Liang; Lingdong Kong; Guoyang Zhao; Zeying Gong; Jun Cen; Zhiyu Huang; Xiaoshuai Hao; Linfeng Li; Hang Song; Xiangtai Li; Jun Ma; Shaojie Shen; Jianke Zhu; Dacheng Tao; Ziwei Liu; Junwei Liang

Source-linked AI summary

Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future

Tianshuai Hu, Xiaolu Liu, Song Wang, Yiyao Zhu, Ao Liang, Lingdong Kong, Guoyang Zhao, Zeying Gong, Jun Cen, Zhiyu Huang, Xiaoshuai Hao, Linfeng Li, Hang Song, Xiangtai Li, Jun Ma, Shaojie Shen, Jianke Zhu, Dacheng Tao, Ziwei Liu, Junwei Liang

arXiv:2512.16760v2Dec 18, 2025cs.RO

TL;DR

Autonomous-driving research is moving beyond cascaded perception–decision–action pipelines and opaque vision-action policies toward vision-language-action systems. This survey characterizes that landscape through architectural taxonomies, representative benchmarks, and open challenges, with cited benchmark examples showing gains associated with language grounding.

Problem
Modular pipelines propagate perception errors, while vision-action models have limited interpretability and weak generalization under distribution shifts and long-tailed events.
Method
The paper organizes autonomous-driving VLA methods into End-to-End and Dual-System paradigms and distinguishes their action-generation and guidance mechanisms.
Results
Language-grounded systems achieve 0.31m L2 and 0.09 collision rate on nuScenes, 99.1 NC and 87.6 EP on NAVSIM, and an 85.94 driving score on Bench2Drive.
Takeaways & Limitations
The survey establishes a structured foundation for analyzing VLA architectures, datasets, benchmarks, robustness, interpretability, and instruction fidelity in autonomous driving.
Takeaways & Limitations
VA models remain widely deployed despite limitations in interpretability and generalization, motivating continued attention to these challenges in VLA research.

Abstract

from arXiv · show

Autonomous driving has long relied on modular "Perception-Decision-Action" pipelines, where hand-crafted interfaces and rule-based components often break down in complex or long-tailed scenarios. Their cascaded design further propagates perception errors, degrading downstream planning and control. Vision-Action (VA) models address some limitations by learning direct mappings from visual inputs to actions, but they remain opaque, sensitive to distribution shifts, and lack structured reasoning or instruction-following capabilities. Recent progress in Large Language Models (LLMs) and multimodal learning has motivated the emergence of Vision-Language-Action (VLA) frameworks, which integrate perception with language-grounded decision making. By unifying visual understanding, linguistic reasoning, and actionable outputs, VLAs offer a pathway toward more interpretable, generalizable, and human-aligned driving policies. This work provides a structured characterization of the emerging VLA landscape for autonomous driving. We trace the evolution from early VA approaches to modern VLA frameworks and organize existing methods into two principal paradigms: End-to-End VLA, which integrates perception, reasoning, and planning within a single model, and Dual-System VLA, which separates slow deliberation (via VLMs) from fast, safety-critical execution (via planners). Within these paradigms, we further distinguish subclasses such as textual vs. numerical action generators and explicit vs. implicit guidance mechanisms. We also summarize representative datasets and benchmarks for evaluating VLA-based driving systems and highlight key challenges and open directions, including robustness, interpretability, and instruction fidelity. Overall, this work aims to establish a coherent foundation for advancing human-compatible autonomous driving systems.

1 Introduction

This work reviews the shift from modular and vision-action driving systems toward vision-language-action models, providing a structured roadmap of their architectures, evaluation resources, and open challenges.

Motivation: Modular driving pipelines can struggle in complex, dynamic, and long-tailed scenarios because cascaded perception errors propagate into downstream reasoning and control.The paper motivates alternatives by identifying limitations of hand-crafted interfaces, rules, and sequential processing.
From VA to VLA: VA models directly map sensory observations to control commands or trajectory waypoints, but remain opaque, fragile under distribution shift, and limited in reasoning and instruction-following.These limitations include weak interpretability, poor long-tail generalization, and insufficient contextual deliberation.
From VA to VLA: VLA models couple multimodal inputs with a VLM backbone and action head to support language-grounded decision making, reasoning, and executable driving actions.Representative systems introduced language-guided closed-loop driving, structured reasoning, and natural-language rationales.
Contributions: The paper characterizes VLA models for autonomous driving by tracing their evolution, organizing architectures, synthesizing evaluation resources, and identifying deployment challenges.Its scope is domain-specific to autonomous driving and emphasizes historical continuity and architectural depth.
Architectural taxonomy: The taxonomy separates End-to-End VLA models from Dual-System designs, which divide high-level VLM deliberation from low-level trajectory execution by planners or controllers.The paper also discusses distinctions such as textual versus numerical action generation and explicit versus implicit guidance.
Evaluation and outlook: The paper reviews datasets and open- and closed-loop benchmarks, then highlights robustness, interpretability, reasoning, instruction fidelity, and human-AI interaction as future priorities.Its stated aim is to support safer, more reliable, and human-compatible autonomous driving research.

2 Preliminary Foundations

VLA frameworks map heterogeneous sensory, scene, language, and vehicle-state inputs through a VLM backbone to executable driving actions. Their foundations include alternative action heads and action-space representations, spanning textual or numerical outputs and discrete, continuous, or direct-control formulations.

VLA Framework Formulation: VLA systems combine multimodal inputs with a VLM backbone and action-generation head to produce executable driving actions.Inputs may include sensor observations, latent scene representations, language instructions, and proprioceptive states.
Input Modalities: The input space includes camera images, LiDAR point clouds, BEV or occupancy representations, textual task instructions, and ego-vehicle state.These signals describe both the external environment and the vehicle’s current dynamics.
VLM Backbone: The VLM backbone fuses modalities into a shared latent representation and either directly generates actions or provides guidance to another action expert.A vision encoder, language decoder, and alignment mechanism connect visual features with language embeddings.
Action Heads: Action heads comprise Language, Regression, Trajectory Selection, and Trajectory Generation designs, differing in output formulation and generation mechanism.Language heads emit textual commands or action tokens, regression heads predict continuous values, selection heads score candidate paths, and generation heads synthesize trajectories probabilistically.
Action-Space Representations: Action spaces represent driving outputs as future waypoints, continuous motion functions, or low-level control commands.Waypoint actions specify spatial coordinates over a prediction horizon; continuous actions use speed and curvature profiles; direct control uses steering, throttle, and braking signals.

3 Vision-Action Models

Vision-Action models replace modular autonomous-driving stacks with direct sensory-to-action learning, spanning end-to-end and world-model approaches. The section organizes these methods by outputs, supervision, representations, and forecasting modalities while identifying interpretability, generalization, reasoning, and language limitations.

3.1 End-to-End and World Models: VA models directly map sensory observations to actions or trajectories, and are broadly divided into end-to-end models and world models.End-to-end models implicitly couple perception, prediction, and planning, whereas world models predict action-conditioned future dynamics.
3.1 End-to-End Models: End-to-end models comprise action-only networks and perception-action models, with the latter using perception supervision to constrain trajectory prediction.Perception-action methods generally use dense BEV or sparse query representations.
3.1.1 Action-Only Models: Imitation learning is efficient and avoids reward engineering, but distribution shift and causal confusion can impair reliability in rare-event scenarios.Reinforcement learning offers greater behavioral flexibility and is often combined with supervised pretraining or privileged-information distillation.
3.1.2 Perception-Action Models: Dense BEV models support geometry-aware planning but require substantial computation, while sparse-query models reduce latency at the cost of long-horizon planning and safety constraints.The supplied methods include vectorized, occupancy-enhanced, generative, diffusion, and sparse-query planning designs.
3.2 World Models: World models are categorized as image-based, occupancy-based, or latent-based according to prediction modality and representation granularity.Occupancy models forecast spatiotemporal geometry and semantics, whereas latent models forecast compressed future dynamics while avoiding pixel-generation overhead.
3.3 Limitations of VA Compared to VLA: VA models remain structurally limited by weak interpretability, generalization, explicit reasoning, and language understanding compared with VLA models.The supplied passages describe VLA models as providing language explanations, broader pretrained knowledge, step-wise reasoning, and instruction integration.

4 Vision-Language-Action Models

VLA models extend vision-action driving by coupling visual perception with multimodal reasoning and actionable outputs. The review summarizes representative systems and organizes their inputs, language components, and action outputs.

4 Vision-Language-Action Models: VLA models couple visual perception with large vision-language models to support multimodal reasoning and executable driving actions.The review frames them as promising for rare, ambiguous, and long-tailed driving scenarios.
4 Vision-Language-Action Models: Table 3 summarizes representative VLA systems by their input modalities, vision and language components, datasets, and action outputs.The supplied table passage identifies the summary scope but does not provide individual system comparisons.

C EVA-CLIP [61]

This subsection lists representative VLA systems alongside their language backbones, action outputs, and publication venues.

C EVA-CLIP [61]: The listed systems include RDA-Driver, DriveLM, DriveGPT4, DriVLMe, VLAAD, and OccLLaMA.The entries span ECCV’24, RA-L’24, IROS’24, WACVW’24, and arXiv’24 venues.
C EVA-CLIP [61]: Representative systems pair LLaMA-family or GPT-4 language models with description, control, and trajectory outputs.The listed combinations include Desc.+Ctrl., Desc.+Traj., Meta.+Traj., Ctrl.+Traj., and Ctrl.

V BLIP-2 [132]

The supplied passages list numerous VLA systems and their model, venue, and action-output configurations, including textual and numerical action-generator categories. They do not provide a dedicated BLIP-2 description or comparative evaluation.

V BLIP-2 [132]: The supplied entries span multiple vision-language backbones and venues, but they do not establish a comparative winner or performance ranking.The passages enumerate systems rather than report benchmark values.
V BLIP-2 [132]: The table fragment distinguishes textual and numerical action generators among representative VLA systems.The listed outputs include descriptions, controls, trajectories, and metadata in different combinations.

C CLIP [210]

This subsection lists additional representative systems using CLIP-related visual backbones and language models for control and trajectory generation.

C CLIP [210]: The listed systems combine Vicuna-1.5, Qwen2, Qwen, and Tinyllama with control and trajectory outputs.The entries include GEN and REG categories alongside Ctrl.+Traj. outputs.
C CLIP [210]: Representative entries include SimLingo, DriveGPT4-V2, and AutoVLA.The listed publication venues are CVPR’25 and NeurIPS’25.

C CLIP [210] SigLIP [314]

This subsection lists later systems using Qwen2.5-VL, PaliGemma, LLaMA, InternVL3, and related components for trajectory and control generation.

C CLIP [210] SigLIP [314]: The listed systems use Qwen2.5-VL, PaliGemma, LLaMA, InternVL3, and CVQ-related components for trajectory and control outputs.The entries include Traj., Ctrl., Desc.+Traj., and Ctrl.+Traj. output types.
C CLIP [210] SigLIP [314]: Representative systems include DriveMoE, DSDrive, OccVLA, VDRive, and ReflectDrive.These entries are listed as arXiv’25 works, while the component listing also includes SimLingo-related material.

N LLaDA-V [303] LLaDA-V [303] GEN Traj. 46 E3AD [240] arXiv’25

The supplied entries enumerate VLA-related models with backbone, generator, action-output, and publication metadata.

The entries cover models using varied backbones and output types, including trajectories, controls, descriptions, and combined outputs.
The entries include works associated with dual-system categories such as explicit action guidance and implicit representation transfer.

N D VAD [107]

The paper organizes autonomous-driving VLA methods into End-to-End and Dual-System paradigms, with End-to-End models distinguished by textual or numerical outputs and Dual-System models by explicit or implicit guidance.

End-to-End VLA: End-to-End VLA models directly map multimodal sensory inputs and language to actions within a single architecture.
End-to-End VLA: End-to-End outputs divide into textual action generators for language-space reasoning and numerical action generators for trajectories, waypoints, or controls.
Textual Action Generator: 80K-clip corner-case pretraining significantly improves trajectory accuracy and closed-loop stability, while benchmarking reveals interpretability–numerical-precision trade-offs.
Textual Action Generator: Textual generators offer interpretability and reasoning but must bridge discrete language tokens with continuous control, which can limit precision or destabilize trajectories.
Numerical Action Generator: Numerical generators produce planner- and actuator-compatible outputs but can sacrifice interpretability, require substantial supervised data, and incur quantization artifacts.
Dual-System VLA: Dual-System VLA separates slow VLM reasoning from fast trajectory generation and control, and categorizes methods as explicit guidance or implicit representation transfer.
Implicit Representations Transfer: Implicit transfer reduces inference cost and runtime dependence on large VLMs but may sacrifice interpretability or oversimplify reasoning signals through distillation.

5 Datasets & Benchmark

VLA driving research relies on diverse multimodal datasets and evaluation protocols spanning trajectory prediction, language, and interactive closed-loop performance. Benchmarks report benefits from language grounding, while annotation assumptions and cross-benchmark metrics shape what systems can demonstrate.

Datasets: VLA datasets extend sensory driving logs with language instructions, question-answer pairs, or rationales aligned with visual observations and actions.This supports joint perception-language-action learning and reasoning-oriented evaluation.
Datasets: QA-style annotations dominate VLA dataset extensions, but their embedded assumptions motivate more diverse perspectives, planning horizons, and evaluation protocols.
nuScenes Benchmark: 0.31m L2 and 0.09 collision rate: Drive-R1 outperforms UniAD’s 0.69m L2 and 0.12 collision rate on nuScenes open-loop planning.The passage attributes the result to supervised chain-of-thought alignment with reinforcement-learning fine-tuning.
WOD-E2E Benchmark: WOD-E2E evaluates long-tail, safety-critical end-to-end driving with routing commands, ego status, multi-camera views, and human-preference-based Rater Feedback Score.Reported results show balanced RFS and ADE for Poutine and dVLM-AD, while AutoVLA visualization is presented separately.
Bench2Drive Benchmark: 85.94 driving score: SimLingo leads the reported Bench2Drive results by aligning natural-language instructions with control sequences through action dreaming.The benchmark emphasizes language-guided reasoning in interactive driving scenarios.

6 Challenges & Future Directions

VLA systems face unresolved deployment challenges involving efficiency, domain adaptation, data quality, trustworthy reasoning, temporal coherence, and evaluation. Future directions emphasize world models, richer sensor fusion, socially aware learning, continual adaptation, standardized safety testing, and human-centered personalization.

Model Architecture and System Efficiency: Sub-50ms inference remains unmet for safety-critical deployment because high-resolution, multi-view VLA processing creates substantial memory and latency costs.Streaming token compression and adaptive visual encoders are identified as promising directions.
Model Architecture and System Efficiency: Generic VLMs are not optimized for driving-specific spatial reasoning, physics, multisensor fusion, traffic rules, or rare high-stakes edge cases.Dedicated driving foundation models remain a missing cornerstone for scalable and dependable VLA systems.
Data and Generalization: Paired vision-action-language triplets are expensive to collect, while synthetic data retains sim-to-real discrepancies in noise, lighting, and other-agent behavior.Data efficiency and distribution-shift mitigation remain open challenges.
Core Capabilities and Trustworthiness: Generated rationales may be unfaithful explanations, allowing confident language hallucinations to justify incorrect decisions and leaving perception-action-explanation grounding unresolved.
Core Capabilities and Trustworthiness: Limited context windows and short-term conditioning can fragment temporal reasoning, producing inconsistent decisions in multi-agent or highly dynamic scenes.
Future Directions: Future systems should combine predictive world models, tighter multimodal sensor fusion, socially aware knowledge grounding, continual onboard learning, and evaluations for instruction errors and hallucinations.Personalization must remain balanced against safety and regulatory requirements.

7 Conclusion

The paper frames VLA models as a shift toward autonomous-driving agents that combine perception, high-level reasoning, and natural-language interaction. It organizes architectures, datasets, and benchmarks while identifying robustness, continuous-control alignment, and faithful safety evaluation as remaining priorities.

VLA models couple perception with high-level reasoning and natural-language understanding, while the paper organizes their architectures, datasets, and benchmarks.
Key unresolved challenges include aligning symbolic reasoning with continuous control, robust long-tail behavior, and evaluation of instruction following and safety.The paper points to efficient architectures, multimodal fusion, world-model planning, and human-centered testing as future directions.