Source-linked AI summary
A Survey on Vision-Language-Action Models for Autonomous Driving
Sicong Jiang, Zilin Huang, Kangan Qian, Ziang Luo, Tianze Zhu, Yang Zhong, Yihong Tang, Menglin Kong, Yunlong Wang, Siwen Jiao, Hao Ye, Zihao Sheng, Xin Zhao, Tuopu Wen, Zheng Fu, Sikai Chen, Kun Jiang, Diange Yang, Seongjin Choi, Lijun Sun
TL;DR
Autonomous-driving VLA research is rapidly expanding, but its literature remains fragmented and prior VLM systems are not tightly connected to control. This survey formalizes VLA4AD architectures, traces their evolution, and consolidates models, datasets, benchmarks, and evaluation protocols. It presents VLA4AD as a progression toward integrated, interpretable, instruction-following autonomous driving while identifying robustness, efficiency, and verification challenges.
Problem
Existing autonomous-driving research lacks a comprehensive survey of the emerging VLA paradigm, while VLM systems remain loosely coupled to low-level control.
Method
The survey formalizes shared VLA4AD architecture blocks, traces four development waves, compares over twenty representative models, and consolidates datasets, benchmarks, and evaluation protocols.
Results
The survey unifies VLA4AD methods under a taxonomy spanning inputs, architectural components, outputs, and the progression from explainers to reasoning-augmented control.
Takeaways & Limitations
VLA4AD provides a coherent reference for studying transparent, instruction-following, and socially aligned autonomous vehicles.
Takeaways & Limitations
VLA4AD still faces sub-30 Hz reasoning requirements, formal-verification needs, long-tail generalization challenges, and sim-to-real transfer difficulties.
Abstract
from arXiv · showhide
The rapid progress of multimodal large language models (MLLM) has paved the way for Vision-Language-Action (VLA) paradigms, which integrate visual perception, natural language understanding, and control within a single policy. Researchers in autonomous driving are actively adapting these methods to the vehicle domain. Such models promise autonomous vehicles that can interpret high-level instructions, reason about complex traffic scenes, and make their own decisions. However, the literature remains fragmented and is rapidly expanding. This survey offers the first comprehensive overview of VLA for Autonomous Driving (VLA4AD). We (i) formalize the architectural building blocks shared across recent work, (ii) trace the evolution from early explainer to reasoning-centric VLA models, and (iii) compare over 20 representative models according to VLA's progress in the autonomous driving domain. We also consolidate existing datasets and benchmarks, highlighting protocols that jointly measure driving safety, accuracy, and explanation quality. Finally, we detail open challenges - robustness, real-time efficiency, and formal verification - and outline future directions of VLA4AD. This survey provides a concise yet complete reference for advancing interpretable socially aligned autonomous vehicles. Github repo is available at \href{https://github.com/JohnsonJiang1996/Awesome-VLA4AD}{SicongJiang/Awesome-VLA4AD}.
1. Introduction
Autonomous driving must combine complex scene understanding, contextual reasoning, and safe real-time action, yet existing stacks remain brittle or only loosely connected to control. VLA4AD is introduced as a comprehensive survey of models that integrate perception, language, and action.
- Motivation: Classic autonomous-driving pipelines divide perception, prediction, planning, and control, but remain brittle at module boundaries and in long-tail scenarios.These scenarios often require high-level reasoning or nuanced human interaction.
- Motivation: VLM-augmented systems improve semantic perception and explanations but remain passive, with language loosely coupled to low-level control.They may hallucinate hazards or misinterpret colloquial instructions.
- VLA4AD: VLA models fuse camera streams, natural-language instructions, and low-level actuation within a single policy.They can follow free-form commands, verbalize rationales, and use commonsense priors in rare situations.
- Survey scope: The survey frames VLA4AD as integrating perception, reasoning, and action for interpretable and robust closed-loop control.This contrasts with end-to-end driving and perception-centric VLM4AD paradigms.
- Survey scope: VLA4AD is presented as the first comprehensive survey addressing this rapidly emerging autonomous-driving paradigm.The survey organizes architectures, representative models, datasets, benchmarks, training paradigms, evaluation protocols, and open challenges.
2. Development of Autonomous Driving
Autonomous-driving research has progressed from modular pipelines to end-to-end learning, language-enhanced perception, and action-aware VLA systems. Each transition addresses prior integration or reasoning limits, while current methods still face generalization, verification, and control-alignment challenges.
- Modular stacks: Traditional autonomous driving factorized perception, prediction, planning, and control into separately engineered modules.This modularity supported component-level engineering and testing.
- End-to-end learning: End-to-end policies map raw sensor streams directly to control commands, bypassing hand-crafted modular pipelines.They narrow the gap between sensing and control but face sparse planning-level data and an unstructured solution space.
- End-to-end learning: Closed-loop evaluations show diminishing returns beyond certain data volumes and substantial performance variance across scenario types.These findings indicate that pure data scaling is insufficient for Level 4+ autonomy.
- VLM4AD: VLM-based driving adds natural-language reasoning and commonsense priors, but large foundation models can suffer from poor spatial awareness, ambiguous numerical outputs, and high latency.Follow-up work targets spatial grounding, latency reduction, and hallucination mitigation.
- VLM4AD: VLM methods remain perception-centric because their plans lack tight closed-loop-control integration and their explanations provide no formal safety guarantees.Aligning VLM outputs with the action space remains an additional challenge.
- VLA4AD: VLA addresses this gap with an explicit action head that unifies perception, reasoning, and control within one policy.Recent systems combine textual and trajectory outputs, memory, safety checks, and multimodal diffusion planning.
3. Architecture Paradigm of VLA4AD
VLA4AD architectures combine multimodal sensing and language inputs with visual encoding, language processing, and action decoding. Their outputs range from low-level controls to trajectories and explanations, reflecting increasing demands for stable, interpretable, context-aware driving.
- Inputs: VLA4AD uses multimodal sensor streams and linguistic inputs to represent both the external environment and high-level driver intent.Inputs include visual data, complementary sensors, and commands, queries, descriptions, or dialogue.
- Inputs: Language inputs have progressed from direct navigation commands to environmental queries, task-level specifications, multi-turn dialogue, reasoning chains, and tool-augmented interfaces.Spoken language is also being explored for embodied interaction.
- Core architecture: The core pipeline integrates visual perception, language understanding, and action generation.Visual encoders produce latent representations, language processors encode instructions, and action decoders emit controls or trajectories.
- Action outputs: Action decoders may autoregressively predict action tokens or waypoints, sample continuous controls with diffusion, or use flow-matching and policy-gradient experts.Hierarchical controllers can let a language planner dispatch sub-goal sketches to PID or MPC stacks.
- Action outputs: Output formats have evolved from raw steering, throttle, and braking signals toward trajectory- or waypoint-level predictions.Trajectory representations support more stable, interpretable, and longer-horizon planning through MPC or downstream planners.
- Action outputs: Typical VLA4AD models produce driving decisions at varying abstraction levels and may also generate language-grounded explanations.This reflects an ambition to drive robustly, explainably, and contextually.
4. Progress of VLA4AD Paradigm
VLA4AD has evolved through four stages: explanatory models, modular VLA, unified end-to-end VLA, and reasoning-centric VLA. Across these stages, language moves from passive narration toward planning, control, memory, and long-horizon reasoning.
- Evolution overview: The survey traces four successive stages: Explanatory Language Models, Modular VLA4AD, End-to-end VLA4AD, and Reasoning-centric VLA4AD.These stages are presented as successive responses to limitations of earlier approaches.
- Explanatory Language Models: Early explainer systems paired frozen vision models with LLM decoders to describe scenes or recommend maneuvers without directly producing control.DriveGPT-4 generated textual descriptions or high-level maneuver labels from a front-camera image.
- Modular VLA4AD: Modular VLA systems made language an active planning component by converting instructions into interpretable intermediate representations such as waypoints.OpenDriveVLA fused camera and LiDAR inputs with textual route instructions.
- End-to-end VLA4AD: Unified end-to-end VLA models map sensors and optional text commands directly to trajectories or control signals in one forward pass.EMMA jointly performs object detection and motion planning using a shared representation.
- End-to-end VLA4AD: End-to-end VLA models are reactive and effective at sensorimotor mapping but can struggle with long-horizon reasoning and fine-grained explanations.These limitations motivate the reasoning-centric wave.
- Reasoning-centric VLA4AD: Reasoning-centric systems add memory, chain-of-thought action alignment, and interactive language reasoning to the control loop.ORION summarizes minutes of history, while Impromptu VLA aligns annotated reasoning steps with actions.
- Evolution overview: Overall, language progresses from passive explanation to active perception, planning, control, reasoning, and dialogue.The survey characterizes this as a progressively tighter loop between seeing, speaking, and acting.
5. Datasets and Benchmarks
VLA4AD datasets and benchmarks combine realistic driving logs, safety-critical corner cases, structured reasoning annotations, and action-level human-preference evaluation. Together, they support training and assessment across perception, reasoning, control, and explanation.
- Model taxonomy: Table 1 organizes representative VLA4AD models by sensor inputs, language integration, action outputs, evaluation data or environments, and core contributions.Its output categories include low-level control, future trajectories, and multi-task outputs.
- Reasoning benchmarks: 600 k video–text pairs in Reason2Drive annotate chain-of-thought questions across perception, prediction, and action while scoring logical consistency.Its consistency metric penalizes incoherent multi-step answers.
- Corner-case data: 80k 30s clips in Impromptu VLA pair expert trajectories and instructions with captions and time-stamped QA for corner-case traffic.The corpus covers dense crowds, ambulances, and adverse weather and provides an open evaluation server.
- Action evaluation: 2.6 k driving scenarios and 16.2 k vision-language QA pairs in DriveAction use action-level labels to score human-preferred driving decisions.This benchmark addresses the gap left by perception-only suites.
- Dataset coverage: The dataset ecosystem spans sensor-rich realism, safety-critical corner cases, structured language reasoning, and human-aligned action evaluation.BDD-X and nuScenes provide realistic data; Bench2Drive and Impromptu VLA target corner cases; Reason2Drive, DriveLM, NuInteract, and DriveAction support language-conditioned reasoning and actions.
6. Training and Evaluation Strategies
VLA4AD training typically combines imitation learning, simulation-based reinforcement learning, multimodal pretraining, and efficiency-oriented compression. Evaluation must jointly assess driving control, language fidelity, robustness, and their coupling.
- Training paradigms: Most VLA4AD systems use pre-training followed by targeted refinement through imitation learning, simulation, reinforcement learning, or rule-based constraints.This pipeline addresses safe controller learning and retention of a faithful language interface.
- Imitation Learning: Imitation learning reproduces expert controls or trajectories with ℓ2 or cross-entropy losses but provides limited supervision for rare hazards.DAgger-style noisy roll-outs and corner-case augmentation are common remedies, yet distribution drift can still cascade when perception or language grounding fails.
- Reinforcement Learning: Reinforcement learning usually builds on an imitation-learning warm start and optimizes route progress, collision avoidance, and traffic-rule compliance in simulation.SafeAuto adds logical traffic rules as constraints or penalties, while LangCoop refines vehicle-to-vehicle coordination.
- Multi-stage Training: Four-stage curricula pretrain general vision-language priors, align them with driving data, refine rare-event behavior, and compress models for deployment.Compression uses LoRA adapters, sparse mixture-of-experts routing, or teacher–student distillation.
- Evaluation Protocols: Evaluation covers closed-loop driving, open-loop prediction, language competence, and robustness, reflecting the dual requirement to drive safely and communicate faithfully.Reported measures include route success, infractions, trajectory error, collision rate, command following, reason-chain consistency, perturbation robustness, and latency.
- Evaluation gap: Current evaluation suites cover control, reasoning, and language facets largely in isolation, motivating a unified test that measures their coupling.CARLA and Bench2Drive emphasize control, whereas NuInteract, Reason2Drive, and DriveLM emphasize reasoning.
7. Open Challenges
VLA4AD remains constrained by robustness, real-time computation, data scarcity, incomplete multimodal fusion, social coordination, and domain adaptation. The survey also identifies a missing evaluation and regulatory framework spanning control and explanations.
- Robustness & Reliability: Language reasoning introduces hallucinated hazards and slang misinterpretation, while sensor corruption and the lack of formal verification remain major reliability concerns.Logic-based safety vetoes are an initial step, but socially compliant policies remain largely unexplored.
- Real-time Performance: Running vision transformers and LLMs at ≥30 Hz on automotive hardware remains difficult as model sizes reach billions of parameters.Token reduction, quantisation, event-triggered reasoning, distillation, and sparse mixture-of-experts are proposed efficiency directions.
- Data & Annotation Bottlenecks: Tri-modal supervision is scarce and costly, with limited coverage of non-English dialects, traffic slang, and legally binding phrasings.Impromptu VLA required 80k manually labelled clips, while synthetic augmentation only partially addresses coverage gaps.
- Multimodal Alignment: A principled, temporally consistent fusion of camera, LiDAR, radar, HD-map, and temporal-state information is still missing.Existing approaches include BEV point-cloud projection, 3-D token adapters, language summaries, and retrieval of textual map rules.
- Multi-agent Social Complexity: Dense multi-agent traffic creates unresolved protocol, trust, authentication, and security challenges for exchanging intent through a constrained traffic language.Cryptographic vehicle-to-vehicle communication and gesture-to-text grounding remain early research threads.
- Domain Adaptation & Evaluation: Sim-to-real transfer, cross-region generalization, continual learning, long-tail coverage, and a regulatory test for control and explanations remain unresolved.Benchmarks such as Bench2Drive cover only a fraction of long-tail conditions.
8. Future Directions
Future VLA4AD research aims to move from prototypes toward scalable, cooperative, socially aware, and verifiable driving platforms. Proposed directions span foundation-scale models, symbolic safety, fleet learning, standardized traffic language, and cross-modal social intelligence.
- Foundation–scale Driving Models: Foundation-scale driving models would pretrain on dash-cams, LiDAR, HD-maps, and textual road rules for data-efficient downstream adaptation.Realizing this vision requires masked multimodal objectives and architectures that process panoramic video with free text.
- Neuro-symbolic Safety Kernels: Neuro-symbolic safety kernels could have neural VLA policies emit structured action programs or chain-of-thought plans for symbolic verification.This direction seeks to bridge end-to-end flexibility with certifiability.
- Fleet-scale Continual Learning: Fleet-scale continual learning could aggregate concise language descriptions of novel hazards into curriculum updates shared across deployed vehicles.Cloud agents could answer real-time queries from uncertain vehicles and bootstrap knowledge across the fleet.
- Standardised Traffic Language: A constrained, ontology-driven traffic language could standardize intent messages such as “I-yield-to-you” and “Obstacle-ahead” for wide-area coordination.Mixture-of-experts routing and token reduction could help keep vehicle-to-vehicle communication bandwidth low.
- Cross-modal Social Intelligence: Cross-modal social intelligence would align gestures, voice, and signage with explicit human-readable vehicle responses such as lights, displays, or honks.Retrieval-augmented planners offer one route for grounding non-verbal cues in live perception, symbolic rules, and context.
9. Conclusion
This survey unifies VLA4AD methods through a taxonomy of modalities, architectures, and outputs, while tracing four waves that progressively connect perception, language understanding, and control. It compares training workflows and datasets, and identifies deployment challenges including throughput, verification, long-tail robustness, and sim-to-real transfer.
- The survey introduces a taxonomy covering input modalities, core architectural components, and output formats for VLA4AD methods.
- VLA4AD evolves through four waves: Pre-VLA Explainers, Modular VLA4AD, End-to-End VLA4AD, and Reasoning-Augmented VLA4AD.
- The review compares training from multimodal pre-training and modality alignment through corner-case augmentation and efficient compression.
- Rich multi-sensor and language-grounded datasets are identified as critical for advancing VLA4AD capabilities.
- Open challenges include sub-30 Hz reasoning throughput, formal verification, long-tail generalization, and seamless sim-to-real transfer.