Source-linked AI summary
Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving
Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, Li Zhang
TL;DR
Autonomous-driving research lacks datasets with annotated reasoning chains that explain decisions. Reason2Drive provides a 600K+ video-text benchmark, a chain-based evaluation protocol, and a VLM framework using object-level perceptual elements; experiments report improved reasoning and downstream planning support.
Problem
Research is hindered by the scarcity of datasets that explain autonomous-driving decisions through chained reasoning labels.
Method
Reason2Drive combines multi-dataset object-centric annotations, an aggregated chain-based metric, and VLM components for object-level feature extraction and perceptual prediction.
Results
Reason2Drive comprises 600K+ video-text pairs, and the proposed method substantially improves reasoning accuracy while supporting downstream planning and generalizing to unseen scenarios.
Takeaways & Limitations
The benchmark and method provide a basis for studying interpretable chain-based reasoning and object-level perception in autonomous-driving VLMs.
Abstract
from arXiv · showhide
Large vision-language models (VLMs) have garnered increasing interest in autonomous driving areas, due to their advanced capabilities in complex reasoning tasks essential for highly autonomous vehicle behavior. Despite their potential, research in autonomous systems is hindered by the lack of datasets with annotated reasoning chains that explain the decision-making processes in driving. To bridge this gap, we present Reason2Drive, a benchmark dataset with over 600K video-text pairs, aimed at facilitating the study of interpretable reasoning in complex driving environments. We distinctly characterize the autonomous driving process as a sequential combination of perception, prediction, and reasoning steps, and the question-answer pairs are automatically collected from a diverse range of open-source outdoor driving datasets, including nuScenes, Waymo and ONCE. Moreover, we introduce a novel aggregated evaluation metric to assess chain-based reasoning performance in autonomous systems, addressing the semantic ambiguities of existing metrics such as BLEU and CIDEr. Based on the proposed benchmark, we conduct experiments to assess various existing VLMs, revealing insights into their reasoning capabilities. Additionally, we develop an efficient approach to empower VLMs to leverage object-level perceptual elements in both feature extraction and prediction, further enhancing their reasoning accuracy. The code and dataset will be released.
1 Introduction
Autonomous-driving research lacks interpretable, chain-based reasoning data and evaluation, while existing approaches can obscure decision logic. Reason2Drive addresses these gaps with a large reasoning benchmark, ADRScore, and a VLM enhancement framework.
- End-to-end systems can obscure decision logic and complicate failure diagnosis, whereas VLMs may improve interpretability and generalization.
- Existing autonomous-driving VLM work primarily adapts question-answering tasks, leaving how to exploit VLMs’ reasoning abilities underexplored.
- Existing datasets often reduce driving to closed-form answers, lacking chained labels that explain complex perception, prediction, and reasoning processes.
- Over 600K video-text pairs form Reason2Drive, built from nuScenes, Waymo, and ONCE with object- and scenario-level annotations and reasoning chains.GPT-4 and manual instructions support verification and enrichment.
- ADRScore evaluates chain-based reasoning by addressing ambiguities in BLEU and CIDEr, which do not assess whether reasoning steps support final conclusions.
- Experiments identify weak use of perceptual priors and inaccurate perceptual outputs, motivating a prior tokenizer and instructed vision decoder that improve reasoning and downstream planning.The paper reports substantial reasoning improvements and generalization to unseen scenarios.
2 Related Work
Related multimodal models align visual and language features and support grounded visual question answering, but driving datasets largely emphasize perception or tracking rather than whole-scenario reasoning. Reason2Drive is positioned to address this limitation.
- Multimodal models such as BLIP-2, LLaVA, and MiniGPT-4 align visual and language representations for multimodal comprehension.
- Reason2Drive’s schema presents automated dataset construction alongside perception, prediction, and reasoning instances augmented with GPT-4.
- Video-LLaMA and ImageBind-LLM integrate multiple modalities, while Kosmos-2 and Shikra support instruction-based object detection and grounded visual question answering.
- Existing driving datasets primarily target tracking or scene perception, with limited annotations for analyzing complex reasoning across an entire scenario.
3 Reason2Drive Dataset
Reason2Drive is a large, diverse benchmark that organizes autonomous-driving understanding into perception, prediction, and chain-based reasoning across object- and scenario-level tasks. It also introduces ADRScore to evaluate reasoning-chain quality, including perceptual grounding and visual elements.
- Dataset characteristics: Its diversity includes object attributes, locations, motion, ego-vehicle relationships, global scene understanding, and longer step-by-step question-answer pairs.
- Dataset construction: The benchmark is constructed from public datasets including nuScenes, Waymo, and ONCE, using an object-centric database, question templates, and GPT-4 verification or enrichment.
- Task organization: The dataset represents driving through perception, prediction, and reasoning tasks at both object and scenario levels.Perception identifies objects, prediction infers future object states, and reasoning analyzes current and predicted states step by step.
- Dataset characteristics: Reason2Drive is presented as the largest dataset to date, with extensive long-text chain-based reasoning references.
- Dataset analysis: The dataset has a balanced task distribution, with perception, prediction, and reasoning questions comprising 39%, 34%, and 27%, respectively.Multi-object tasks constitute the majority, while single-object and scenario-level questions are similar in quantity and ego-vehicle questions are fewest.
- Benchmark protocol: ADRScore evaluates reasoning chains by measuring alignment, redundancy, and missing steps, while its visual adaptation uses geometric similarity for perceptual elements.The protocol addresses limitations of BLEU, CIDEr, and METEOR by considering reasoning relationships and perceived results.
4 Methodology
The framework augments VLMs with components that tokenize video and perceptual priors, align them with text, and generate both reasoning answers and precise perception results. Its training jointly optimizes text generation and perception outputs while using staged pre-training and fine-tuning.
- The framework adds a prior tokenizer and instructed vision decoder to help VLMs use object-level perceptual elements for interpretable visual reasoning.The components are designed to use available perception inputs rather than perform detection.
- Video frames and perceptual priors are tokenized, then a Q-former aligns their non-text features with the textual domain before language generation.The vision encoder processes video frames, while the prior tokenizer extracts visual-prior embeddings.
- The prior tokenizer extracts region-level features and positional embeddings from visual priors to preserve locations and motions in a shared embedding.Region features are obtained with RoIAlign and fused with positional encodings of geometry locations and motions.
- The instructed vision decoder uses <LOC> and <MOT> tokens to request perception outputs alongside the LLM’s textual reasoning.Textual features associated with these tokens are combined with visual features to decode locations and motions.
- Training combines autoregressive text-generation loss with perception loss, and uses pre-training followed by efficient LoRA fine-tuning.The perception loss encourages accurate locations and motions through binary cross-entropy and MSE components.
5 Experiments
Experiments benchmark models on Reason2Drive using ADRScore and captioning metrics, compare task combinations and architectural components, and evaluate predicted visual elements and control signals. The proposed method outperforms other models across metrics, while ablations identify reasoning data and perception modules as important contributors.
- The benchmark uses nuScenes, Waymo, and ONCE, with segment-disjoint training and validation splits and five cropped input frames.Seventy percent of segments are used for training and thirty percent for validation.
- The proposed method outperforms other models comprehensively across ADRScore and traditional caption-based metrics.ADRScore reveals larger performance gaps than traditional metrics, which show minimal differences among models with varying reasoning capabilities.
- Task contributions: Training on reasoning tasks contributes most strongly, while perception and prediction tasks provide additional improvements for visual reasoning.The comparison evaluates models trained on different combinations of task types.
- Ablation study: Visual features from multiple frames improve ADRScore-S by 1.5%, while region-level features and positional embeddings improve it by 2.4% and 1.4%.These results come from ablations of visual input and perception priors.
- Ablation study: Pre-training and textual embedding contribute 3.5% and 2.9% in ADRScore-S for the instructed vision decoder.The ablation compares different instructed vision decoder settings.
- The experiments also assess predicted visual elements and downstream control-signal prediction on the benchmark.Trajectory accuracy receives particular emphasis in the visual-element evaluation.
6 Conclusion
Reason2Drive targets interpretable reasoning in autonomous driving by combining a large, diverse benchmark with chain-based evaluation and model enhancements.
- Reason2Drive provides 600K+ video-text pairs for interpretable reasoning in complex driving scenarios.The benchmark is described as exceeding existing datasets in scale, sources, and task diversity.
- The benchmark introduces a chain-based reasoning evaluation protocol to address semantic ambiguities in existing metrics.
- Experiments evaluate various VLMs and propose an efficient method for using object-level perceptual elements in encoding and prediction.
- Reason2Drive contains words spanning perception, prediction, and reasoning tasks, including “moving”, “distance”, and “risk”.
A.2 Detailed sub-tasks in Reason2Drive
Reason2Drive organizes autonomous driving data into perception, prediction, and reasoning groups, while using prompts and human instructions to augment question-answer pairs.
- The dataset covers 15 perception, 14 prediction, and 6 reasoning sub-tasks.
- GPT prompts use exemplar question-answer pairs and human-written instructions to generate augmented question-answer pairs.The real question-answer pairs are supplied in the user content during augmentation.
B.1 Architecture
The architecture combines a frozen ViT-G/14 visual encoder with either encoder-decoder or decoder-only language models.
- The frozen visual encoder uses ViT-G/14 from EVA-CLIP and extracts features from the transformer’s second-last layer.The final ViT layer is removed.
- The language-model options are FlanT5-XL for encoder-decoder modeling and Vicuna for decoder-only modeling.
B.2 Training loss
Training combines language modeling with an auxiliary perception objective, while baseline outputs are formatted for fair perceptual evaluation.
- Training objective: The model uses language-modeling loss Ltxt to generate text from Q-former modality features.
- Training objective: An auxiliary perception loss Lper combines binary cross-entropy classification and regression losses.
- Training objective: λreg is set to 0.25 to balance the regression loss in practice.
- Baseline evaluation: Baseline models receive perception priors as text and produce textual perceptual results for ADRScore-S evaluation.
C.1 Ablation of visual encoders
Visual encoder choice affects VLM performance, particularly on the strict reasoning metric. Additional evaluations examine metric validity, cross-dataset generalization, qualitative successes, and failure cases.
- C.1 Ablation of visual encoders: Visual encoder choice inevitably influences VLM performance, especially on the strict reasoning metric.The ablation compares CLIP ViT-L/14 with EVA-CLIP ViT-G/14.
- Metric validation: GPT-4 validation indicates that the proposed reasoning metric remains rational and is not dependent on GPT migration.The authors also describe the metric as more interpretable than the comparison approach.
- Generalization: When trained only on nuScenes and tested on Waymo and ONCE, the method experiences limited performance drops, suggesting cross-source generalization.The authors attribute this generalization to the language model’s world knowledge.
- Qualitative results: Qualitative results show stronger performance than InstructBLIP on object planning, potential-risk recognition, and reasoning across task levels.These successful cases support the method’s interpretable and chain-based reasoning behavior.
- Failure cases: Complex scenes remain challenging because ego-vehicle displacement can confuse motion status and distant risk objects may be missed.The paper identifies distance-feature enhancement and dynamic ego-displacement encoding as future mitigation directions.