Source-linked AI summary
Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models
Haohan Chi, Huan-ang Gao, Ziming Liu, Jianing Liu, Chenyu Liu, Jinwei Li, Kaisen Yang, Yangcheng Yu, Zeda Wang, Wenyi Li, Leichen Wang, Xingtao Hu, Hao Sun, Hang Zhao, Hao Zhao
TL;DR
Autonomous-driving VLA models lack large, targeted data for unstructured scenarios, limiting evidence about their capabilities in these settings. Impromptu VLA builds a curated dataset and taxonomy with planning-oriented annotations, then evaluates models across driving benchmarks and task-specific diagnostics. The dataset improves closed-loop safety and scores, open-loop trajectory prediction, and measured perception, prediction, and planning capabilities.
Problem
Existing autonomous-driving datasets insufficiently cover large-scale, diverse, meticulously annotated unstructured scenarios.
Method
The paper distills over two million clips from eight public sources into approximately 80,000 verified clips organized by four scenario types and enriched with multi-task Q&A and trajectories.
Results
Models trained with Impromptu VLA show gains on NeuroNCAP safety and driving scores, nuScenes trajectory prediction, and validation-suite perception, prediction, and planning evaluations.
Takeaways & Limitations
The dataset provides a benchmark and diagnostic resource for studying autonomous-driving model behavior in diverse, challenging unstructured road scenarios.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models for autonomous driving show promise but falter in unstructured corner case scenarios, largely due to a scarcity of targeted benchmarks. To address this, we introduce Impromptu VLA. Our core contribution is the Impromptu VLA Dataset: over 80,000 meticulously curated video clips, distilled from over 2M source clips sourced from 8 open-source large-scale datasets. This dataset is built upon our novel taxonomy of four challenging unstructured categories and features rich, planning-oriented question-answering annotations and action trajectories. Crucially, experiments demonstrate that VLAs trained with our dataset achieve substantial performance gains on established benchmarks--improving closed-loop NeuroNCAP scores and collision rates, and reaching near state-of-the-art L2 accuracy in open-loop nuScenes trajectory prediction. Furthermore, our Q&A suite serves as an effective diagnostic, revealing clear VLM improvements in perception, prediction, and planning. Our code, data and models are available at https://github.com/ahydchh/Impromptu-VLA.
1 Introduction
Impromptu VLA targets the scarcity of specialized data for unstructured driving scenarios with a large, richly annotated dataset and systematic curation pipeline. Training with it improves established driving benchmarks and supports diagnosis of perception, prediction, and planning capabilities.
- Existing driving datasets predominantly cover structured traffic, leaving limited large-scale, annotated coverage of ambiguous roads, unusual obstacles, temporary rules, and hazardous conditions.
- Impromptu VLA distills over two million clips from eight public sources into approximately 80,000 verified clips spanning four challenging unstructured scenario types.
- The dataset combines a systematic taxonomy, VLM-based curation, multi-task Q&A annotations, and human verification for training advanced VLMs.
- Training with Impromptu VLA improves standard driving benchmarks, including closed-loop NeuroNCAP and open-loop nuScenes trajectory prediction.
- 2.15/5.00 NeuroNCAP score versus 1.77/5.00 and 65.5% versus 72.5% collision rate are reported for the enhanced and baseline 3B models, respectively.
Impromptu VLA Dataset: Learning to Drive on Unstructured Roads
The dataset is designed to replace a monolithic view of unstructured driving with a granular taxonomy of specific challenges. Its collection focuses on scenarios that test current autonomous-driving systems.
- Impromptu VLA addresses the lack of sufficient large-scale, diverse, meticulously annotated datasets focused on unstructured scenarios.
- The taxonomy seeks a granular understanding of unstructured environments and targets scenarios that genuinely test current autonomous-driving systems.
1.Dataset Collection
The dataset pipeline standardizes and filters multi-source driving clips, classifies unstructured scenarios, generates planning-oriented annotations, and verifies labels with humans. The resulting resource supports perception and planning evaluation, including NeuroNCAP analysis.
- Data Processing and Annotation: Figure 3 presents sequential raw-data collection, taxonomy definition, frequency alignment, keyclip selection, annotation generation, and human verification.
- Taxonomy Definition: Approximately 10% of aggregated clips were sampled for open-ended Qwen2.5-VL 72B descriptive analysis before automated filtering and semantic clustering.
- Scenario Taxonomy: The four categories cover unclear boundaries, temporary traffic-rule changes, unconventional dynamic obstacles, and challenging road or environmental conditions.
- Keyclip Selection: Sequences were standardized to 2 Hz, aligned with NAVSIM using 1.5 seconds of past context and 5 seconds of future context, then filtered through local packs of up to 15 seconds.
- Scene Classification: Qwen2.5-VL 72B used Chain-of-Thought prompting to extract scene, roadway, and object information before assigning each keyclip to a category.
- Annotation and Verification: Multi-task annotations include scene descriptions, traffic-signal detection, and other task-specific labels, while human reviewers accepted, rejected, or corrected generated annotations.
- Dataset Utility: Planning-oriented Q&A pairs connect visual inputs, textual outputs, and action-trajectory predictions for training and standardized evaluation.
3 Experiments
The experiments test whether Impromptu VLA improves established closed-loop and open-loop driving benchmarks and whether its validation suite diagnoses perception, prediction, and planning gains.
- Benchmark Evaluation: The study compares Base+Impromptu+nuScenes against Base+nuScenes using closed-loop NeuroNCAP and open-loop nuScenes trajectory prediction.NeuroNCAP measures collision rates and NeuroNCAP scores, while open-loop evaluation uses L2 trajectory error at future horizons.
- Benchmark Evaluation: Improvement in open-loop trajectory accuracy is consistent across the 1s, 2s, and 3s prediction horizons on nuScenes.The adapted 3B/7B models reach a competitive range with EMMA+, whose reported average L2 is 0.29m.
- Diagnostic Evaluation: The validation suite compares base and fine-tuned VLMs on planning-oriented Q&A tasks covering perception, prediction, reasoning for planning, and planned trajectories.Table 4 reports accuracy for perception, dynamic-object prediction, meta-planning, and planning tasks across 3B and 7B models.
- Diagnostic Evaluation: Fine-tuning on Impromptu VLA improves performance across the validation suite’s crucial autonomous-driving aspects.The reported aspects include perception, prediction, planning reasoning, and the planned trajectory.
4 Related Works
Related work spans vision-language models for multimodal autonomous driving, specialized techniques and datasets, and end-to-end driving benchmarks for imitation learning and simulation.
- Vision-Language Models: Vision-language models extend language models with visual understanding and have been applied to autonomous driving for multimodal reasoning.Prior work includes simulation-based LVLM planners and large-scale pretraining aimed at improving generalization.
- Vision-Language Models: Driving-specific Q&A datasets and benchmarks have been proposed to align model training with downstream planning tasks.These works build on the broader VLM paradigm for autonomous driving.
- Specialized Techniques and Datasets: Specialized autonomous-driving research addresses perception, simulation, mapping, and prediction through techniques such as realistic neural-radiance-field simulation.Mars is described as an instance-aware, modular, realistic simulator for generating and testing complex scenarios.
- End-to-End Benchmarks: End-to-end benchmarks are categorized into large-scale imitation-learning datasets and simulation benchmarks.Real-world road-network datasets support annotated perception, prediction, and planning tasks.
5 Conclusion
The paper concludes that Impromptu VLA provides a curated, richly annotated resource for unstructured driving and improves benchmark performance and diagnostic capabilities.
- Conclusion: Impromptu VLA contains approximately 80,000 clips with multi-task Q&A annotations and action trajectories for unstructured driving scenarios.The dataset is designed to address data scarcity in unstructured environments.
- Conclusion: Training with Impromptu VLA improves closed-loop NeuroNCAP safety and driving scores and open-loop nuScenes trajectory prediction accuracy.The conclusion reports gains across both evaluation settings.
- Conclusion: The validation suite reveals model advancements in perception, prediction, and planning on diverse unstructured road scenarios.The suite functions as a diagnostic tool for these capabilities.
- Limitation: Annotation generation primarily relies on Qwen2.5-VL, which may introduce model-specific biases despite human verification.The authors identify this reliance as a limitation while noting the dataset’s demonstrated utility.
Supplementary Material
The supplementary document adds technical details about training implementation and the construction of the unstructured-driving dataset.
- Supplementary Material: The supplement details training data formats and hyperparameter settings for Qwen2.5-VL model variants.It also describes the construction process for the unstructured-driving dataset.
- Supplementary Material: The dataset-construction discussion introduces three key phases for building the unstructured-driving data resource.The excerpt indicates that the supplement elaborates on these phases after covering implementation specifics.
A Implementation Details
This section describes the training processes and data format used for nuScenes fine-tuning, alongside general Qwen2.5-VL hyperparameter settings.
- The implementation details cover training processes used in the experiments.
- The nuScenes dataset has a specific data format for fine-tuning and evaluation.
- The experiments use general hyperparameter settings for Qwen2.5-VL model variants.
A.1 Data format for nuScenes Fine-tuning and Evaluation
The nuScenes-specific setup defines a data format for fine-tuning and evaluation that differs from the dataset’s general trajectory Q&A format.
- The nuScenes-specific data format is used for both fine-tuning and evaluation.
- The setup supports the open-loop and closed-loop nuScenes experiments.
- Its trajectory format resembles the Impromptu VLA End-to-End Trajectory Prediction Q&A format but includes distinctions specific to nuScenes.The related Impromptu VLA format uses past 1.5s and future 5s trajectories.
A.2 Hyperparameters
The training hyperparameters were selected empirically and applied consistently across the pipeline, without thorough task-specific optimization.
- The hyperparameters were established for training on the Impromptu VLA Dataset and subsequently on nuScenes data.
- The settings were based on general observation and common practice rather than a thorough optimization process.The authors attribute this choice to the extensive search space.
- Further exhaustive tuning could yield better results.
- The same empirical settings were consistently applied across all experiments.
B VLM-based Unstructured Scene Identification
The pipeline identifies unconventional driving scenes using a VLM-based classifier prompted to distinguish challenging cases from routine conditions. The prompt was refined against human annotations before broader dataset curation.
- B VLM-based Unstructured Scene Identification: Rich textual scene descriptions were generated before isolating genuinely unstructured or unconventional driving conditions.
- B VLM-based Unstructured Scene Identification: The categorization prompt asks a VLM to label scenes as Conventional or Unconventional based on predictability, challenge, and required attention.
- B VLM-based Unstructured Scene Identification: Unconventional scenarios are defined as deviations from typical driving that increase complexity or risk relative to standard conditions.
- B VLM-based Unstructured Scene Identification: The VLM classification step was used to focus the dataset on true corner cases.
- B VLM-based Unstructured Scene Identification: Prompt refinement compared VLM classifications with consensus labels from two human annotators on approximately 1000 validation descriptions.
C Q&A Examples
The Impromptu VLA dataset uses diverse question-answering tasks to represent driving scenes, planning decisions, object motion, trajectories, and traffic signals. Task availability varies with the source clip’s annotations.
- Scene understanding: The QA format includes scene descriptions and task-specific questions built from front-facing images and detailed information from source datasets.Examples cover environmental context, traffic participants, weather, road conditions, and traffic-light status.
- Annotation coverage: The availability of QA tasks for each clip depends on the richness of annotations in its source dataset.Missing object-detection ground truth can prevent VRU or actor-specific motion-intention annotations even when scene-description or ego-trajectory QA remains possible.
- Planning tasks: Meta-action planning asks the model to produce separate speed and path plans for the next three seconds.The permitted speed outputs are KEEP, ACCELERATE, DECELERATE, and STOP; path outputs include STRAIGHT, lane changes, and turns.
- Planning tasks: Planning-explanation QA pairs require explanations linking an immediate driving decision to scene evidence such as construction cones and a digger.The example maintains speed and goes straight because road work suggests caution, while the commanded turn may occur later.
- Prediction tasks: End-to-end trajectory prediction uses multi-view images and recent vehicle status to predict future waypoints over a five-second horizon.The example formats predictions as BEV [x, y] locations sampled at 0.5-second intervals.
- Perception and prediction tasks: Other QA tasks include traffic-signal detection and motion-intention prediction for dynamic objects, with outputs expressed as categorical speed and path decisions.Traffic-signal detection uses Red, Green, Yellow, or None; motion-intention prediction assigns speed and path labels to detected objects.