PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model

Xingcheng Zhou, Xuyuan Han, Feng Yang, Yunpu Ma, Volker Tresp, Alois Knoll

arXiv:2503.23463v2cs.CV

TL;DR

OpenDriveVLA targets limitations in end-to-end driving models, including weak complex-semantic understanding, limited spatial grounding, and insufficient modeling of agent interactions. It combines structured 2D and 3D visual tokens, ego states, language commands, hierarchical vision-language alignment, and agent–environment–ego interaction modeling in an autoregressive framework. On nuScenes, the paper reports state-of-the-art performance in open-loop planning and driving-related question answering, while noting inference latency and limited planning-evaluation scope as limitations.

  • Problem

    Existing driving models face limited long-tail generalization, weak complex-semantic understanding, and insufficient explicit modeling of 3D spatial layout and agent interactions.

  • Method

    OpenDriveVLA aligns structured 2D and 3D visual tokens with language embeddings and incorporates ego states, commands, and agent–environment–ego interaction modeling for trajectory generation.

  • Results

    OpenDriveVLA achieves state-of-the-art performance across nuScenes open-loop trajectory planning and driving-related question answering tasks.

  • Takeaways & Limitations

    The results support the feasibility of a scalable vision-language-driven approach for end-to-end autonomous driving action systems.

  • Takeaways & Limitations

    Autoregressive decoding introduces inference latency, and the current planning evaluation is limited in scope.

Abstract

from arXiv · show

We present OpenDriveVLA, a Vision Language Action model designed for end-to-end autonomous driving, built upon open-source large language models. OpenDriveVLA generates spatially grounded driving actions by leveraging multimodal inputs, including 2D and 3D instance-aware visual representations, ego vehicle states, and language commands. To bridge the modality gap between driving visual representations and language embeddings, we introduce a hierarchical vision language alignment process, projecting both 2D and 3D structured visual tokens into a unified semantic space. Furthermore, we incorporate structured agent environment ego interaction modeling into the autoregressive decoding process, enabling the model to capture fine-grained spatial dependencies and behavior-aware dynamics critical for reliable trajectory planning. Extensive experiments on the nuScenes dataset demonstrate that OpenDriveVLA achieves state-of-the-art results across open-loop trajectory planning and driving-related question answering tasks. Qualitative analyses further illustrate its capability to follow high-level driving commands and generate trajectories under challenging scenarios, highlighting its potential for next-generation end-to-end autonomous driving.

I. INTRODUCTION

OpenDriveVLA addresses semantic, spatial, and interaction-reasoning challenges in end-to-end autonomous driving by combining structured visual representations with language-guided action generation. Its design targets spatially grounded, behavior-aware driving actions within a unified autoregressive framework.

  • Existing end-to-end systems jointly optimize perception, prediction, and planning but still struggle with long-tail generalization, complex semantics, and rigid task reasoning.
  • Agent–environment–ego interaction modeling is added as an auxiliary autoregressive training objective to learn behaviorally grounded and safety-aware actions.
  • OpenDriveVLA integrates spatially grounded multimodal reasoning and trajectory generation using 2D and 3D instance-aware representations, ego states, and high-level commands.
  • Hierarchical alignment projects structured 2D and 3D visual features into a unified semantic space for spatially grounded actions.
  • Unlike approaches using language heads or high-level decisions separately, OpenDriveVLA is presented as a fully differentiable 3D spatial-aware driving VLA.

III. OPENDRIVEVLA

OpenDriveVLA converts multi-view visual inputs into structured scene, agent, and map tokens, then aligns them with language representations for trajectory planning. Its visual-centric design explicitly models driving-relevant objects, spatial layout, and environmental structure.

  • The pipeline extracts tokenized environmental representations from multi-view images, aligns them into the textual domain, and trains end-to-end to predict future ego trajectories.
  • A visual-centric query module focuses token selection on driving-relevant objects and map elements through 3D vision tasks.
  • Multi-view features are lifted into BEV space and processed by scene, agent, and map query modules that encode complementary environmental semantics.
  • The resulting environment tokens represent global scene context, dynamic agents, and static structures such as lane boundaries and drivable areas.

B. Stage 1 - Hierarchical Vision-Language Alignment

OpenDriveVLA aligns structured visual representations with language and models agent–environment–ego interactions to support spatially grounded, behavior-aware driving decisions.

  • Hierarchical Vision-Language Alignment: Hierarchical alignment maps instance-aware agent, scene, and map visual tokens into the language space using token-specific projectors.Agent tokens are matched with captions describing appearance and 3D positions, while scene and map tokens capture holistic context and static structure.
  • Driving Instruction Tuning: Driving instruction tuning injects perception, motion prediction, attention, action reasoning, and decision-making knowledge through curated instruction-response pairs.The multimodal input combines environmental visual tokens, ego state, and a driving query; the model autoregressively generates the target response.
  • Agent-Env-Ego Interaction Modeling: Agent–environment–ego interaction modeling predicts each detected agent’s future waypoint sequence conditioned on scene context, map structure, and ego state.This auxiliary stage captures multi-agent dynamics and supports interaction-aware, spatially grounded motion sequences.
  • Agent-Env-Ego Interaction Modeling: The interaction objective provides spatial priors that bridge high-level semantic reasoning and physically grounded motion planning.The paper links these priors to scene-aware trajectory generation and decision-making in complex traffic scenarios.

E. Stage 3 - End-to-end Trajectory Planning Tuning

Stage 3 trains OpenDriveVLA to generate ego trajectories as discrete waypoint tokens, with the visual encoder, projectors, and language model optimized in an end-to-end pipeline.

  • Trajectory Representation: OpenDriveVLA represents the ego vehicle’s future trajectory as a short-horizon sequence of 2D waypoints.Each waypoint contains the ego vehicle’s (x_t, y_t) coordinates at a future time step.
  • Autoregressive Planning: The waypoint sequence is tokenized into discrete textual tokens and generated autoregressively as a causal sequence prediction task.This converts trajectory planning into language-model token generation.
  • End-to-end Optimization: The 3D visual encoder, cross-modality projectors, and LLM are jointly optimized end-to-end during training, while the 2D encoder remains frozen.At inference, generated trajectory tokens are decoded back into numerical waypoints.

IV. EXPERIMENTS

Experiments evaluate OpenDriveVLA on nuScenes using open-loop trajectory planning and driving visual question answering, with comparisons across planning settings and multiple QA datasets.

  • Training Datasets: OpenDriveVLA is trained from curated data spanning TOD3Cap, nuCaption, nuScenesQA, nuX, and GPT-Driver, and evaluated on standard nuScenes training and validation splits.The validation set is reserved exclusively for performance evaluation.
  • Evaluation Protocol: Evaluation covers L2 displacement at 1, 2, and 3 seconds, average collision rate, and standard NLG metrics for driving VQA.Planning is reported under ST-P3 and UniAD settings, while QA uses nuCaption, nuScenesQA, and nuX.
  • Open-Loop Trajectory Planning: 0.33m average L2 error is achieved by both 3B and 7B models under ST-P3 metrics, outperforming prior autoregressive language models.OpenDriveVLA-7B also records 0.66m average L2 error under UniAD metrics, while the 0.5B model outperforms prior models.
  • Driving Question Answering: OpenDriveVLA reaches best-in-class performance across three driving VQA datasets and outperforms prior language-enhanced and general-purpose multimodal baselines on most metrics.Its spatially grounded alignment is associated with advantages on object- and status-related nuScenesQA questions, while 0.5B surpasses 7B on Nu-X.
  • Command Following: Qualitative evaluation shows OpenDriveVLA adapting its plan from keeping forward to turning right while maintaining context-aware and environment-consistent behavior.The same visualization includes QA predictions and agent motion predictions after interaction modeling.

E. Ablation Study

The ablation study examines input modalities, multi-stage training, and command following for OpenDriveVLA. Visual inputs, additional context, and successive training stages improve planning, while qualitative results show command-responsive behavior.

  • Evaluation Scope: The study combines quantitative ablations with qualitative assessments of diverse driving-command following and driving-related question answering.
  • Effect of Input Modalities: Visual inputs significantly boost overall trajectory-planning performance, while textual commands and historical information further improve predictions.Ego-state features are particularly important on the nuScenes open-loop benchmark.
  • Effect of Multi-Stage Training Strategy: Each additional multi-stage training phase consistently improves performance, with the largest collision-rate reductions occurring after hierarchical alignment and agent-environment-ego interaction modeling.
  • Effect of Driving Command: OpenDriveVLA adapts its trajectory to different high-level commands at an intersection while maintaining context-aware and environment-consistent behavior.The qualitative comparison considers keep-forward and turn-right instructions, with the right turn as ground truth.

A. Model Details

OpenDriveVLA uses structured 2D and 3D visual tokens from a vision-centric perception encoder, combining dynamic agents, static maps, and global scene context for language alignment and planning. Its language backbone receives these tokens alongside ego state and command information.

  • Vision Encoder: Vision-centric pretraining uses 3D detection, tracking, and BEV panoptic segmentation to provide structured, instance-level supervision for driving.This approach is motivated by the ambiguity and imprecision of general language-guided visual grounding in complex traffic scenes.
  • Vision Encoder: The visual encoder processes multi-view images into multi-scale 2D features and aggregates them into a BEV representation using a query-based architecture.
  • Structural Token Extraction: Three query-based token extractors encode complementary scene elements: global context, dynamic agents, and static map structure.Together they form V_env = {V_scene, V_agent, V_map}, a compact interface for downstream alignment and planning.
  • Agent Tokens: Agent tokens represent individual objects’ locations, categories, and motion trajectories, with low-confidence predictions filtered to improve efficiency and reduce hallucination.
  • Map and Scene Tokens: Map tokens encode static elements such as lane dividers, road boundaries, and drivable areas, while scene tokens capture weather, lighting, layout, and traffic flow.
  • Language Backbone: OpenDriveVLA evaluates Qwen2.5-Instruct backbones with 0.5B, 3B, and 7B parameters within the LLaVA NeXT framework.
  • Structured Input: Extended special tokens organize scene, track, map, ego, and command inputs into discrete semantic segments for structured multimodal reasoning.Projected visual tokens replace scene, track, and map placeholders, while ego state and command are formatted as text.

B. Prompting Techniques

The prompting design frames OpenDriveVLA as a driving model responsible for perception, reasoning, trajectory planning, and optional question answering. It specifies coordinate conventions, safety objectives, output structure, and interaction behavior.

  • 1) System Prompt:: The system prompt is prepended during training and inference to maintain consistent task framing and instruction following.
  • 1) System Prompt:: The system prompt assigns OpenDriveVLA responsibility for safe trajectory planning and interpretable decision-making.
  • 1) System Prompt:: The prompt uses coordinates with the ego vehicle at (0,0), the X-axis pointing right, and the Y-axis pointing forward, with distances in meters.
  • 1) System Prompt:: The model is instructed to analyze visual data, identify road users and hazards, predict motion, and assess critical objects.
  • 1) System Prompt:: Trajectory planning is defined as a safe, feasible 3-second route consisting of six waypoints.
  • 1) System Prompt:: Outputs prioritize a trajectory formatted as six coordinate pairs, with an optional user-question response enclosed in separate answer markers.

2) Prompts for Hierarchical Feature Alignment:

OpenDriveVLA uses prompts to align visual tokens with language descriptions and to train driving question answering, interaction modeling, and trajectory planning. These prompts preserve structured scene, agent, map, ego, and temporal context.

  • 2) Prompts for Hierarchical Feature Alignment:: Separate captioning prompts instruct the LLM to describe scene, agent, and map visual-token segments for hierarchical feature alignment.
  • 2) Prompts for Hierarchical Feature Alignment:: Agent-token prompts request both an object caption and its BEV coordinate.
  • 2) Prompts for Hierarchical Feature Alignment:: Map- and scene-token prompts request captions for the corresponding structured visual inputs.
  • 3) Prompts for Driving Question Answering:: Driving question-answering prompts combine scene, object tracking, map, ego-state, and historical-trajectory information before the question.
  • 4) Prompts for Agent-Env-Ego Interaction:: The agent-environment-ego prompt supplies structured visual context, historical ego trajectory, and a target object whose future motion must be predicted.
  • 5) Prompts for Trajectory Planning Tuning:: Trajectory-planning tuning supervises the model to generate a 3-second driving plan from structured multimodal context and a mission command.

C. Training and Inference Details

OpenDriveVLA uses a multi-stage training pipeline with curated multimodal nuScenes-derived datasets, followed by resource-constrained inference evaluation.

  • Training and Inference Details: Training uses mixed-precision bf16 and gradient checkpointing, with the 2D vision backbone frozen during final end-to-end training.The 0.5B variant is trained on four NVIDIA H100 GPUs with per-GPU batch size 1 and takes approximately two days.
  • Training and Inference Details: Inference evaluates 0.5B, 3B, and 7B OpenDriveVLA variants under bf16 precision on one NVIDIA A100 GPU.The NuScenes trajectory validation set contains 6019 samples.
  • Training Data: The multi-stage training data covers object captioning, visual question answering, scene description, and decision reasoning.The overview tables organize task types, annotations, data sources, and sample counts.
  • Training Data: TOD3Cap provides 2.3M verified descriptions for over 64K objects across 850 nuScenes scenes, enabling fine-grained 3D object-language alignment.Its captions cover appearance, motion, context, and inter-object relations.
  • Training Data: nuScenes-QA contains 460K question–answer pairs over 34K multimodal driving scenes, covering existence, counting, attributes, spatial relations, and comparisons.The benchmark uses synchronized images and LiDAR.
  • Training Data: nuCaption supplies image-text and LiDAR-text pairs with global and viewpoint-specific descriptions of traffic layout, object interactions, and potential risks.The dataset supports 3D captioning and scene-level reasoning.
  • Training Data: nuX provides factual and causal natural-language explanations grounded in perception, prediction, and planning outputs for interpretable driving.Visualization examples illustrate the annotations used in hierarchical feature alignment training.

VIII. RESULTS AND DISCUSSIONS

Qualitative results show OpenDriveVLA reasoning about driving questions, predicting agent motion, and producing smoother open-loop plans than UniAD in a narrow-road scenario.

  • Driving Question Answering: OpenDriveVLA answers driving-related questions across nuScenes-QA, nuCaption, and nuX using perception, commonsense, and contextual reasoning.Figures 8 and 9 provide representative responses from these three datasets.
  • Driving Question Answering: The model explains ego behavior near a speed bump but hallucinates that no pedestrians appear in the right-front view despite one being present.This example combines coherent causal narration with a scene-level grounding error.
  • Driving Question Answering: In a complex intersection, OpenDriveVLA identifies the ego maneuver, localizes the turn, and grounds static and dynamic scene elements.The described elements include traffic lights, pedestrians near a crosswalk, and parked vehicles.
  • Agent Motion Prediction: Stage 2.5 jointly reasons over agent trajectories, the environment, and ego state for agent motion prediction.Figures 10 and 11 show qualitative predictions on the nuScenes validation set.
  • Agent Motion Prediction: One night-time intersection example predicts one vehicle consistently with lane orientation but incorrectly curves another vehicle right where a left turn better fits the road geometry.The example highlights both context-sensitive prediction and an error under low-light conditions.
  • Agent Motion Prediction: A curved-road example produces distinct, plausible trajectories for front- and rear-view vehicles under partial observability.The predictions reflect motion uncertainty and agent-specific intention.
  • Open-Loop Planning: In a narrow road with parked vehicles, OpenDriveVLA generates smoother, more consistent trajectories than UniAD, which produces unstable zigzagging plans.The OpenDriveVLA trajectory better follows the intended path while maintaining safe clearance.

2) Qualitative Results of Driving Instruction Following.:

OpenDriveVLA follows varied driving instructions and performs competitively across model sizes, while the authors identify data, reasoning, latency, and evaluation constraints.

  • 2) Qualitative Results of Driving Instruction Following.:: OpenDriveVLA adapts trajectory plans to left-turn, right-turn, and keep-forward instructions across intersection scenarios.The qualitative examples report safe, scene-aligned, collision-free behavior under changed commands.
  • 1) Results Regarding Model Size.:: Larger model size generally improves performance, but OpenDriveVLA-7B does not consistently outperform the 0.5B and 3B variants across benchmarks.On NuX, the 0.5B model achieves higher CIDEr and ROUGE-L than 7B, while planning differences are relatively small.
  • 1) Results Regarding Model Size.:: Limited and insufficiently diverse domain-specific training data may prevent the 7B model from fully exploiting its representation capacity.The authors connect this observation to the scarcity of large open-source vision-language driving datasets.
  • 1) Results Regarding Model Size.:: Larger autoregressive models may rely more on language priors, weakening visual grounding in fine-grained spatial or factual reasoning tasks.The passage contrasts this with stronger vision-language coupling in smaller models under constrained supervision.
  • 1) Results Regarding Model Size.:: Larger models are more sensitive to optimization settings and may overfit dominant training patterns when dataset diversity is limited.The stated consequence is reduced robustness and generalization during inference.
  • 1) Results Regarding Model Size.:: Model size alone is not a robust performance indicator; data quality, data quantity, and multi-stage training also matter in this domain.This is the authors’ summary of the model-scaling results.
  • 2) Current Limitations:: OpenDriveVLA uses implicit reasoning without inference-time chain-of-thought, incurs autoregressive latency, and is evaluated only in open-loop planning.The authors state that open-loop evaluation may overestimate robustness because it omits interactive ego-agent feedback.
  • 3) Extension to Closed-Loop Planning:: Closed-loop evaluation is constrained because established planning benchmarks lack the vision-language annotations central to domain-specific LLM-based autonomous-driving models.nuScenes-derived datasets provide multimodal captions and question–answer supervision for future extensions.
Loading 2503.23463v2…