Source-linked AI summary
OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, Jose M. Alvarez
TL;DR
Autonomous-driving VLMs need to extend strong 2D reasoning to 3D geometric and spatial understanding, but existing supervision and evaluation remain limited. OmniDrive addresses this with counterfactual synthetic Q&A data and two agent frameworks, and reports significant improvements on DriveLM Q&A and nuScenes open-loop planning. The framework comparison further indicates that migrating 2D VLMs to 3D is more straightforward than integrating traditional 3D perception stacks into VLMs.
Problem
Autonomous-driving VLMs lack a holistic framework for extending 2D reasoning to 3D geometric and spatial understanding, while expert actions provide sparse supervision of underlying reasoning.
Method
OmniDrive generates large-scale synthetic 3D driving Q&A data through counterfactual reasoning, checklist- and prompt-based GPT-4 annotation, human-in-the-loop validation, and Omni-L/Omni-Q agent frameworks.
Results
OmniDrive-pretrained models show significant improvements on the DriveLM Q&A benchmark and nuScenes open-loop planning, while the framework comparison favors migrating 2D VLMs to 3D as the more straightforward approach.
Takeaways & Limitations
Counterfactual data connects planning trajectories with language-based reasoning and provides insights into designing effective autonomous-driving LLM agents.
Takeaways & Limitations
Open-loop evaluation can exhibit implicit ego-status bias, overly simple planning scenarios, and easy overfitting to expert trajectories.
Abstract
from arXiv · showhide
The advances in vision-language models (VLMs) have led to a growing interest in autonomous driving to leverage their strong reasoning capabilities. However, extending these capabilities from 2D to full 3D understanding is crucial for real-world applications. To address this challenge, we propose OmniDrive, a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning. This approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions. Our counterfactual-based synthetic data annotation process generates large-scale, high-quality datasets, providing denser supervision signals that bridge planning trajectories and language-based reasoning. Futher, we explore two advanced OmniDrive-Agent frameworks, namely Omni-L and Omni-Q, to assess the importance of vision-language alignment versus 3D perception, revealing critical insights into designing effective LLM-agents. Significant improvements on the DriveLM Q\&A benchmark and nuScenes open-loop planning demonstrate the effectiveness of our dataset and methods.
1. Introduction
OmniDrive addresses the gap between strong 2D VLM reasoning and the 3D geometric and spatial understanding required for autonomous driving. It introduces counterfactual data and compares two agent designs, with improvements reported on DriveLM Q&A and nuScenes open-loop planning.
- Motivation: OmniDrive targets the missing holistic framework for extending VLM reasoning from 2D understanding to 3D geometric and spatial understanding in autonomous driving.Prior planning benchmarks rely on expert trajectories, while expert actions provide sparse supervision of reasoning and decision-making.
- OmniDrive dataset: Counterfactual reasoning evaluates potential scenarios and outcomes, creating a denser connection between planning trajectories and language-based reasoning.The approach is motivated by human drivers considering alternative actions when making safer decisions.
- OmniDrive dataset: The dataset pipeline combines simulated trajectories, rule-based checklists, GPT-4 prompts, and human-in-the-loop validation for scalable, reliable, and interpretable Q&A generation.These components support structured 3D scene representation and assessment of object attention, trajectory outcomes, and traffic-rule compliance.
- Omni-L/Q agents: Omni-L uses an MLP projector to align existing 2D VLMs with 3D space, whereas Omni-Q combines a BEV perception architecture with Q-Former design.The comparison investigates vision-language alignment versus integration with traditional 3D perception stacks.
- Results: Models pretrained on OmniDrive show significant improvements on the DriveLM Q&A benchmark and nuScenes open-loop planning.The findings demonstrate the reported effectiveness and quality of the dataset.
- Results: The Omni-L and Omni-Q comparison provides insights into designing effective LLM agents, with the paper reporting that migrating 2D VLMs to 3D is more straightforward.This conclusion follows the comparison of the two framework paradigms across counterfactual reasoning and open-loop planning.
2. OmniDrive
OmniDrive is a counterfactual synthetic dataset on nuScenes covering 3D perception, reasoning, and planning. Its generation pipeline selects representative keyframes, structures scene information around trajectories, and uses checklist-guided GPT-4 generation with quality control.
- Dataset: OmniDrive provides synthetic nuScenes Q&A pairs covering 3D perception, reasoning, and planning.The dataset is designed for captioning, open-loop planning, and counterfactual reasoning tasks.
- Planning-oriented key-frame selection: Semantic key-frame selection compresses redundant driving data by clustering CLIP embeddings and selecting 20% of cluster centers.The selected frames support checklist design, prompt iteration, and broader scenario coverage.
- Q&A generation: Human-in-the-loop checklist and prompt design supports quality assurance while large-scale iteration extends the verified generation process.The pipeline integrates semantic key-frame selection, counterfactual checklist and prompt design, and human quality checks.
- Counterfactual representation: Directly presenting GPT-4 with extensive 3D scene information makes it difficult to understand vehicle status and relationships between traffic elements.The difficulty increases as the number of traffic elements grows.
- Counterfactual representation: The method centers scene representation on simulated trajectories and converts them into high-level decisions for counterfactual reasoning.Rule design assigns objects and lanes and determines lane-changing behavior before GPT-4 assesses safety and traffic-rule compliance.
- Q&A generation: The pipeline uses captions, boxes, simulated and expert trajectories, and four response types to generate structured Q&A examples.The response types include scene description, attention, counterfactual reasoning, and decision-making or planning.
3. OmniDrive-agent
OmniDrive-agent explores two designs for extending vision-language models to 3D driving: language-aligned multi-view features in Omni-L and 3D perception-centered representations in Omni-Q.
- Shared design: Both frameworks share a visual encoder and projector-to-LLM pipeline, but their projectors prioritize different objectives: language alignment in Omni-L versus 3D perception in Omni-Q.Multi-view image features are combined with positional encoding before projection and text generation.
- Omni-Q: Omni-Q combines Q-Former-style interaction with query-based 3D perception, using carrier queries for language alignment and perception queries for foreground prediction.Carrier queries are projected to LLM-token dimensions, while perception queries predict foreground categories and coordinates.
- Omni-Q: Omni-Q lets carrier queries exploit geometric priors from 3D positional encoding and representations learned through 3D perception supervision.This design uses the carrier queries as the bridge between visual representations and language.
- Omni-L: Omni-L extends LLaVA by flattening multi-view features, adding 3D positional encoding, and using an MLP for vision-language alignment.Its position-encoding weights are initialized to zero for training stability.
- Training: Both models use 2D pretraining followed by 3D finetuning on driving tasks such as motion planning and counterfactual reasoning.Training uses text-generation loss without contrastive or matching losses.
4. Experiment
Experiments evaluate OmniDrive across driving benchmarks, counterfactual reasoning, and architecture variants. Results show benefits from OmniDrive pretraining and different trade-offs between language ability and 3D perception supervision.
- Implementation details: Omni-L uses EVA-02-L as its vision encoder and follows LLaVA v1.5 training data and strategies during 2D pretraining.The reported finetuning uses AdamW, batch size 16, and a cosine annealing policy.
- Dataset and metrics: The experiments evaluate captioning with CIDEr, open-loop planning with collision and road-boundary intersection rates, and counterfactual reasoning tasks.DriveLM additionally uses BLEU, ROUGE L, accuracy, ChatGPT Score, Match Score, and a weighted final score.
- Open-loop planning: Training with OmniDrive Q&A data without ego status reduces collision rate from 3.22% to 1.90% and intersection rate from 3.90% to 3.29%.The authors also report that Omni-L performs significantly better than Omni-Q without ego status, while ego status can encourage overfitting.
- DriveLM results: OmniDrive pretraining raises the average DriveLM score from 53% by 3% when training otherwise uses only DriveLM data.The dataset remains quality-controlled through counterfactual checklists and human-in-the-loop validation despite highly automated annotations.
- Counterfactual reasoning: Omni-Q records 32.3% collision-detection precision and 72.6% recall, benefiting from 3D perception supervision in tasks such as collision detection.Omni-L performs better on average in the reported counterfactual reasoning results, including 72.1% precision and 58.0% recall in safety tasks.
- Architecture analysis: Omni-L achieves counterfactual AP of 53.7%, AR of 63.0%, Language CIDEr of 73.2, collision rate of 1.90%, and intersection rate of 3.29%.These results are reported alongside stronger language ability and open-loop planning metrics than Omni-Q.
- Architecture analysis: Omni-L’s counterfactual AP is 53.7%, AR is 63.0%, and Language CIDEr is 73.2, compared with Omni-Q’s 52.3%, 59.6%, and 68.6.The table analysis reports a positive correlation between language ability and performance across the evaluated tasks.
5. Related works
End-to-end autonomous-driving research spans open-loop and closed-loop settings, with open-loop benchmarks relying on log-replayed real-world data and facing overfitting concerns. VLM-based driving agents and language-rich datasets extend evaluation toward reasoning, perception, and planning.
- Open-loop autonomous driving trains and evaluates on log-replayed real-world datasets, whereas closed-loop driving is the other main development path.
- Open-loop benchmarks can overfit ego status and expert trajectories, limiting how reliably they assess driving capability.
- VLMs align language with visual and other modalities for tasks including visual question answering, captioning, and open-world detection.
- Driving LLM-agent methods use language models for explanations, control prediction, graph-based VQA, and chain-of-thought reasoning across scene analysis, prediction, and planning.
- Existing autonomous-driving datasets cover perception, tracking, planning, steering, motion, and language annotations, but comprehensive language annotations remain needed for Drive LLM evaluation.
6. Conclusion
OmniDrive introduces a counterfactual-based 3D driving Q&A pipeline and evaluates Omni-L and Omni-Q for LLM-agent design. The framework improves benchmark performance and highlights 3D vision-language alignment, while its counterfactual simulation does not yet model reactions from other agents.
- OmniDrive uses counterfactual-based 3D driving Q&A to generate scalable, high-quality data for end-to-end autonomous driving with LLM-agents.
- Pre-trained models on OmniDrive improve performance on the DriveLM QA benchmark and nuScenes open-loop planning.
- Omni-L and Omni-Q provide insights into LLM-agent design by examining vision-language alignment and 3D environmental understanding.
- Counterfactual outcome simulation does not yet account for reactions from other agents, motivating future use of closed-loop planning results.