Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

• Existing driving datasets predominantly cover structured traffic, leaving limited large-scale, annotated coverage of ambiguous roads, unusual obstacles, temporary rules, and hazardous conditions.Source: §1 · p. 2
• Impromptu VLA distills over two million clips from eight public sources into approximately 80,000 verified clips spanning four challenging unstructured scenario types.Source: §1 · p. 2
• The dataset combines a systematic taxonomy, VLM-based curation, multi-task Q&A annotations, and human verification for training advanced VLMs.Source: §1 · p. 2
• Training with Impromptu VLA improves standard driving benchmarks, including closed-loop NeuroNCAP and open-loop nuScenes trajectory prediction.Source: §1 · pp. 2–3
• 2.15/5.00 NeuroNCAP score versus 1.77/5.00 and 65.5% versus 72.5% collision rate are reported for the enhanced and baseline 3B models, respectively.Source: §1 · p. 2

• Impromptu VLA addresses the lack of sufficient large-scale, diverse, meticulously annotated datasets focused on unstructured scenarios.Source: §2.1 · p. 3
• The taxonomy seeks a granular understanding of unstructured environments and targets scenarios that genuinely test current autonomous-driving systems.Source: §2.2 · p. 3

Data Processing and Annotation

• Figure 3 presents sequential raw-data collection, taxonomy definition, frequency alignment, keyclip selection, annotation generation, and human verification.Source: Figure 3 · p. 4, §2.3 · p. 5

Taxonomy Definition

• Approximately 10% of aggregated clips were sampled for open-ended Qwen2.5-VL 72B descriptive analysis before automated filtering and semantic clustering.Source: 1.Dataset Collection · p. 4

Scenario Taxonomy

• The four categories cover unclear boundaries, temporary traffic-rule changes, unconventional dynamic obstacles, and challenging road or environmental conditions.Source: 1.Dataset Collection · pp. 4–5

Keyclip Selection

• Sequences were standardized to 2 Hz, aligned with NAVSIM using 1.5 seconds of past context and 5 seconds of future context, then filtered through local packs of up to 15 seconds.Source: §2.3 · p. 5

Scene Classification

• Qwen2.5-VL 72B used Chain-of-Thought prompting to extract scene, roadway, and object information before assigning each keyclip to a category.Source: §2.3 · p. 5

Annotation and Verification

• Multi-task annotations include scene descriptions, traffic-signal detection, and other task-specific labels, while human reviewers accepted, rejected, or corrected generated annotations.Source: §2.3 · p. 5

Dataset Utility

• Planning-oriented Q&A pairs connect visual inputs, textual outputs, and action-trajectory predictions for training and standardized evaluation.Source: §2.4 · p. 6

Benchmark Evaluation

• The study compares Base+Impromptu+nuScenes against Base+nuScenes using closed-loop NeuroNCAP and open-loop nuScenes trajectory prediction.NeuroNCAP measures collision rates and NeuroNCAP scores, while open-loop evaluation uses L2 trajectory error at future horizons.Source: §3.1 · pp. 6–7
• Improvement in open-loop trajectory accuracy is consistent across the 1s, 2s, and 3s prediction horizons on nuScenes.The adapted 3B/7B models reach a competitive range with EMMA+, whose reported average L2 is 0.29m.Source: §3.1 · p. 7

Diagnostic Evaluation

• The validation suite compares base and fine-tuned VLMs on planning-oriented Q&A tasks covering perception, prediction, reasoning for planning, and planned trajectories.Table 4 reports accuracy for perception, dynamic-object prediction, meta-planning, and planning tasks across 3B and 7B models.Source: §3.2 · p. 8, Table 4 · p. 8
• Fine-tuning on Impromptu VLA improves performance across the validation suite’s crucial autonomous-driving aspects.The reported aspects include perception, prediction, planning reasoning, and the planned trajectory.Source: §3.2 · p. 8

Vision-Language Models

• Vision-language models extend language models with visual understanding and have been applied to autonomous driving for multimodal reasoning.Prior work includes simulation-based LVLM planners and large-scale pretraining aimed at improving generalization.Source: §4 · p. 8
• Driving-specific Q&A datasets and benchmarks have been proposed to align model training with downstream planning tasks.These works build on the broader VLM paradigm for autonomous driving.Source: §4 · pp. 8–9

Specialized Techniques and Datasets

• Specialized autonomous-driving research addresses perception, simulation, mapping, and prediction through techniques such as realistic neural-radiance-field simulation.Mars is described as an instance-aware, modular, realistic simulator for generating and testing complex scenarios.Source: §4 · p. 9

End-to-End Benchmarks

• End-to-end benchmarks are categorized into large-scale imitation-learning datasets and simulation benchmarks.Real-world road-network datasets support annotated perception, prediction, and planning tasks.Source: §4 · p. 9

Conclusion

• Impromptu VLA contains approximately 80,000 clips with multi-task Q&A annotations and action trajectories for unstructured driving scenarios.The dataset is designed to address data scarcity in unstructured environments.Source: §5 · p. 9
• Training with Impromptu VLA improves closed-loop NeuroNCAP safety and driving scores and open-loop nuScenes trajectory prediction accuracy.The conclusion reports gains across both evaluation settings.Source: §5 · p. 10
• The validation suite reveals model advancements in perception, prediction, and planning on diverse unstructured road scenarios.The suite functions as a diagnostic tool for these capabilities.Source: §5 · p. 10

Limitation

• Annotation generation primarily relies on Qwen2.5-VL, which may introduce model-specific biases despite human verification.The authors identify this reliance as a limitation while noting the dataset’s demonstrated utility.Source: §5 · p. 10

Supplementary Material

• The supplement details training data formats and hyperparameter settings for Qwen2.5-VL model variants.It also describes the construction process for the unstructured-driving dataset.Source: Supplementary Material · p. 17
• The dataset-construction discussion introduces three key phases for building the unstructured-driving data resource.The excerpt indicates that the supplement elaborates on these phases after covering implementation specifics.Source: Supplementary Material · p. 17

• The implementation details cover training processes used in the experiments.Source: A Implementation Details · p. 17
• The nuScenes dataset has a specific data format for fine-tuning and evaluation.Source: A Implementation Details · p. 17
• The experiments use general hyperparameter settings for Qwen2.5-VL model variants.Source: A Implementation Details · p. 17

• The nuScenes-specific data format is used for both fine-tuning and evaluation.Source: §A.1 · p. 17
• The setup supports the open-loop and closed-loop nuScenes experiments.Source: §A.1 · p. 17
• Its trajectory format resembles the Impromptu VLA End-to-End Trajectory Prediction Q&A format but includes distinctions specific to nuScenes.The related Impromptu VLA format uses past 1.5s and future 5s trajectories.Source: §A.1 · p. 17

• The hyperparameters were established for training on the Impromptu VLA Dataset and subsequently on nuScenes data.Source: §A.2 · p. 17
• The settings were based on general observation and common practice rather than a thorough optimization process.The authors attribute this choice to the extensive search space.Source: §A.2 · p. 17, Table 5 · p. 17
• Further exhaustive tuning could yield better results.Source: §A.2 · p. 17
• The same empirical settings were consistently applied across all experiments.Source: Table 5 · p. 17

B VLM-based Unstructured Scene Identification

• Rich textual scene descriptions were generated before isolating genuinely unstructured or unconventional driving conditions.Source: B VLM-based Unstructured Scene… · p. 17
• The categorization prompt asks a VLM to label scenes as Conventional or Unconventional based on predictability, challenge, and required attention.Source: B VLM-based Unstructured Scene… · p. 18
• Unconventional scenarios are defined as deviations from typical driving that increase complexity or risk relative to standard conditions.Source: B VLM-based Unstructured Scene… · p. 18
• The VLM classification step was used to focus the dataset on true corner cases.Source: B VLM-based Unstructured Scene… · p. 18
• Prompt refinement compared VLM classifications with consensus labels from two human annotators on approximately 1000 validation descriptions.Source: B VLM-based Unstructured Scene… · p. 18

Scene understanding

• The QA format includes scene descriptions and task-specific questions built from front-facing images and detailed information from source datasets.Examples cover environmental context, traffic participants, weather, road conditions, and traffic-light status.Source: C Q&A Examples · pp. 18, 19, 21

Annotation coverage

• The availability of QA tasks for each clip depends on the richness of annotations in its source dataset.Missing object-detection ground truth can prevent VRU or actor-specific motion-intention annotations even when scene-description or ego-trajectory QA remains possible.Source: C Q&A Examples · p. 19

Planning tasks

• Meta-action planning asks the model to produce separate speed and path plans for the next three seconds.The permitted speed outputs are KEEP, ACCELERATE, DECELERATE, and STOP; path outputs include STRAIGHT, lane changes, and turns.Source: C Q&A Examples · p. 20
• Planning-explanation QA pairs require explanations linking an immediate driving decision to scene evidence such as construction cones and a digger.The example maintains speed and goes straight because road work suggests caution, while the commanded turn may occur later.Source: C Q&A Examples · p. 20

Prediction tasks

• End-to-end trajectory prediction uses multi-view images and recent vehicle status to predict future waypoints over a five-second horizon.The example formats predictions as BEV [x, y] locations sampled at 0.5-second intervals.Source: C Q&A Examples · p. 20

Perception and prediction tasks

• Other QA tasks include traffic-signal detection and motion-intention prediction for dynamic objects, with outputs expressed as categorical speed and path decisions.Traffic-signal detection uses Red, Green, Yellow, or None; motion-intention prediction assigns speed and path labels to detected objects.Source: C Q&A Examples · p. 20