DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding

I. INTRODUCTION

• Existing driving VQA work often prioritizes final-answer accuracy, leaving intermediate reasoning quality and logical consistency less directly assessed.Step-by-step reasoning for autonomous driving must integrate visual and contextual information.Source: §I · p. 1
• DriveLMM-o1 combines a step-by-step reasoning benchmark with multiview images, LiDAR point clouds, and temporal information for diverse real-world driving scenarios.The dataset is designed to evaluate perception, prediction, and planning and to support multimodal VQA.Source: §I · pp. 1–2
• The benchmark introduces a driving-specific metric for evaluating the logical coherence and accuracy of model-generated explanations.Its examples connect scene understanding and visual-spatial cues to stepwise inference and answer generation.Source: §I · p. 2, Figure 1 · p. 2
• The authors evaluate existing open- and closed-source models and report that their fine-tuned model outperforms all models in reasoning score and final accuracy.The proposed model is fine-tuned on the step-by-step reasoning dataset.Source: §I · p. 2

II. RELATED WORK

• The related-work section covers VQA methods for autonomous driving and general visual reasoning, focusing on their contributions and limitations relative to this study.Source: §II · p. 2

A. Visual Reasoning

• Visual reasoning approaches include staged perception-and-verification and multistage reasoning, while short-answer training may limit generalization to tasks requiring detailed explanations.The cited work motivates structured visual reasoning for autonomous driving.Source: A. Visual Reasoning · p. 2

B. Visual Question Answering in Autonomous Driving

• Table I compares driving-QA datasets by frames, QA pairs, reasoning availability, modalities, image views, annotation methods, and data sources; its statistics are for training sets.The caption identifies DriveLMM-o1 as combining multiview images and LiDAR point clouds with manually curated step-by-step reasoning.Source: Table I · p. 3
• Autonomous-driving VQA benchmarks address multimodal scenes, action rationalization, and reasoning, but many emphasize end-task performance or lack multimodal diversity and structured justifications.DriveLMM-o1 pairs multiview images and LiDAR point clouds with explicit reasoning steps, and evaluates both reasoning coherence and final-answer accuracy.Source: B. Visual Question Answering in… · pp. 2–3

III. AUTONOMOUS DRIVING REASONING BENCHMARK

• DriveLMM-o1 evaluates step-by-step reasoning across perception, prediction, and planning in intricate autonomous-driving scenarios.Source: §III · p. 3
• The benchmark assesses both the logical explanations and the accuracy of models’ final predictions.Source: §III · p. 3
• Compared with existing datasets, DriveLMM-o1 uses manually curated step-by-step reasoning; the comparison also considers frames, QA pairs, input modalities, image views, annotation methods, and data sources.The passage notes that existing datasets may lack step-by-step reasoning, dataset size, or modalities such as multiview images and LiDAR point clouds.Source: §III · p. 3

A. Benchmark Development

• Ten standardized questions span perception, prediction, and planning, grounded in NuScenes keyframes and relevant objects.The template is applied to keyframes and objects to generate diverse, balanced questions.Source: Question Set Curation · p. 3
• GPT-4o generates structured initial reasoning chains and answers from the finalized question set.The responses describe observations, entity relationships, and final outcomes.Source: Automated Generation · p. 4
• Annotators check each generated sample for logical consistency and factual correctness, discard mismatched questions or irrelevant objects, and revise retained reasoning and answers.Corrections improve completeness and logical flow while aligning answers with driving context.Source: Manual Correction and Verification · p. 4

B. Evaluation Methodology

• Existing reasoning evaluators do not address autonomous-driving VQA’s real-world scene understanding and multimodal reasoning requirements.ROSCOE [25] and ReCeval [26] use reference-free evaluation, while ReasonEval [27] targets mathematical reasoning.Source: B. Evaluation Methodology · p. 4
• The benchmark extends borrowed reasoning metrics with driving-specific attributes for safety-critical reasoning assessment.The metric set includes Faithfulness-Step, Informativeness-Step, Repetition-Token, Hallucination, Semantic Coverage-Step, Commonsense, and Missing Step.Source: B. Evaluation Methodology · p. 4, Table II · p. 4
• GPT-4o scores generated steps and answers against ground truth using a structured rubric, and final reasoning score averages the metrics.Final-answer accuracy is also computed for multiple-choice questions.Source: B. Evaluation Methodology · p. 4

IV. PROPOSED MODEL

• The model fine-tunes InternVL2.5-8B on the dataset and integrates stitched multiview images with stepwise reasoning for perception, prediction, and planning.The vision encoder represents spatial and semantic information while the language model processes textual questions.Source: §IV · p. 4, Large Multimodal Model Finetuning · p. 4
• Stitching front, side, and rear camera views into one image provides broader spatial context and lets the model consider multiple viewpoints simultaneously.The design also reduces the need for independent feature extraction across views.Source: Multiview Representation for Scene… · p. 4
• Table III reports that DriveLMM-o1 performs better than other open-source models in reasoning ability and final-answer accuracy.The table reports driving-specific metrics alongside overall reasoning score and final-answer accuracy.Source: Table III · p. 5
• Dynamic image patching handles variable-resolution inputs, while LoRA adapts attention layers with few trainable parameters.The training strategy freezes the vision encoder and most language-model layers; patching tiles and aggregates image embeddings.Source: Large Multimodal Model Finetuning · p. 5

V. EXPERIMENTS

• The experiments benchmark DriveLMM-o1 against open- and closed-source models with quantitative and qualitative analyses.The section describes the training data and experimental setup before evaluating the model.Source: §V · p. 5

A. Training Dataset

• The training dataset draws on 1,962 urban and highway driving scenes and follows the benchmark methodology without rigorous verification..Source: A. Training Dataset · p. 5
• The training set contains 18,507 reasoning-based Q&A pairs and over 55,000 reasoning steps for perception, prediction, and planning.The steps are intended to train structured justifications before final answers.Source: A. Training Dataset · p. 5

• InternVL2.5 8B is the baseline, fine-tuned with a rank-16 LoRA adapter for one epoch.Only 0.49% of total parameters are trainable; the vision encoder and all LLM layers remain frozen.Source: B. Experimental Setup · p. 5
• Dynamic image patching divides high-resolution images into smaller tiles while preserving spatial context and accommodating larger fields of view.Source: B. Experimental Setup · p. 5
• Training uses four NVIDIA A6000 GPUs.Source: B. Experimental Setup · p. 5

• DriveLMM-o1 scores 73.01 in risk assessment accuracy, 81.56 in traffic rule adherence, and 75.39 in scene awareness and object understanding.Its relevance score is 79.42 and missing details score is 74.49.Source: Quantitative Comaprisons · pp. 5, 7
• Final answer accuracy is 62.36 for DriveLMM-o1 versus 54.87 for InternVL2.5-8B, and its overall reasoning score is 75.24; it leads open-source models.Evaluations are zero-shot and compare outputs with ground-truth annotations.Source: Quantitative Comaprisons · pp. 5, 7
• In zero-shot evaluation, LlamaV-o1 and LLaVA-CoT score 63.13 and 61.41, respectively, in overall reasoning, below DriveLMM-o1.The passage notes they were not fine-tuned on this dataset.Source: Quantitative Comaprisons · p. 7
• In qualitative examples, DriveLMM-o1 identifies the ego vehicle’s position, localizes a relevant object, and predicts motion where InternVL2.5-8B gives generic or incorrect judgments.Figure 3 compares reasoning and final answers from the baseline and fine-tuned model against ground truth.Source: Figure 3 · p. 6, Qualitative Comparison · p. 7
• Against LlamaV-o1, DriveLMM-o1 incorporates safety elements and traffic signs and interprets multiview images relative to the ego vehicle.The comparison describes LlamaV-o1 as overlooking safety cues and traffic signs and failing to understand the multiview spatial setting.Source: Figure 4 · p. 6, Qualitative Comparison · p. 7

• The paper introduces a dataset and benchmark for evaluating reasoning abilities across real-world scenarios, plus a metric assessing explanation coherence and accuracy.Source: §VI · p. 7
• The model trained on the step-by-step reasoning dataset surpasses all open-source models in both reasoning scores and final accuracy.Source: §VI · p. 7