Source-linked AI summary
NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models
Sung-Yeon Park, Can Cui, Yunsheng Ma, Ahmadreza Moradipari, Rohit Gupta, Kyungtae Han, Ziran Wang
TL;DR
Driving-scene MLLMs face challenges with multi-view inputs and contextual scene reasoning, while existing driving VQA datasets and benchmarks leave gaps in coverage. NuPlanQA provides a multi-view dataset and evaluation benchmark, alongside BEV-LLM, which integrates BEV features; BEV-LLM outperforms open-source models in six of nine subtasks.
Problem
Driving VQA resources have limited scenario diversity and scene coverage, while contextual traffic understanding requires finer-grained assessment of spatial relations and ego-centric reasoning.
Source: §1 · pp. 1–2Method
NuPlanQA provides 1M multi-view QA pairs and a nine-subtask evaluation benchmark, and BEV-LLM integrates BEV features with MLLMs.
Source: §1 · p. 3, §2.1 · p. 3Results
BEV-LLM outperforms open-source models in six of nine subtasks, while existing MLLMs struggle with traffic-light understanding, spatial relations, and ego-centric reasoning.
Source: §3.2 · p. 6, §5 · p. 8Takeaways & Limitations
The nine-subtask benchmark supports task-wise comparisons across MLLMs and facilitates targeted skill enhancement.
Source: §5 · p. 8Spatial-relations recognition depends on visual input and temporal reasoning across frames, a short-term reasoning challenge for MLLMs typically trained on longer video clips.
Source: §3.2 · p. 7
Abstract
from arXiv · showhide
Recent advances in multi-modal large language models (MLLMs) have demonstrated strong performance across various domains; however, their ability to comprehend driving scenes remains less proven. The complexity of driving scenarios, which includes multi-view information, poses significant challenges for existing MLLMs. In this paper, we introduce NuPlanQA-Eval, a multi-view, multi-modal evaluation benchmark for driving scene understanding. To further support generalization to multi-view driving scenarios, we also propose NuPlanQA-1M, a large-scale dataset comprising 1M real-world visual question-answering (VQA) pairs. For context-aware analysis of traffic scenes, we categorize our dataset into nine subtasks across three core skills: Road Environment Perception, Spatial Relations Recognition, and Ego-Centric Reasoning. Furthermore, we present BEV-LLM, integrating Bird's-Eye-View (BEV) features from multi-view images into MLLMs. Our evaluation results reveal key challenges that existing MLLMs face in driving scene-specific perception and spatial reasoning from ego-centric perspectives. In contrast, BEV-LLM demonstrates remarkable adaptability to this domain, outperforming other models in six of the nine subtasks. These findings highlight how BEV integration enhances multi-view MLLMs while also identifying key areas that require further refinement for effective adaptation to driving scenes. To facilitate further research, we publicly release NuPlanQA at https://github.com/sungyeonparkk/NuPlanQA.
1. Introduction
Driving-scene VQA resources leave gaps in scenario diversity, broad scene understanding, multi-view coverage, and reliable evaluation. NuPlanQA addresses these gaps with a large real-world dataset, a nine-subtask benchmark, and a multi-view BEV-LLM baseline.
1. Introduction
- Existing driving VQA datasets often rely on limited resources and instance-level annotations, restricting scenario diversity and broader road-scene comprehension.Fixed-format QA also limits rich, free-form reasoning responses.Source: §1 · p. 1
- Front-view-only free-form datasets are generally small, while simulated traffic scenes leave a gap to real-world conditions.The paper identifies these constraints as barriers to adapting MLLMs to diverse real driving scenarios.Source: §1 · p. 2
- Traffic-scene understanding requires identifying context such as which traffic light applies to the ego vehicle and whether traffic is stopped or turning.Broad skill categories do not isolate these weaknesses, motivating finer-grained task groupings.Source: §1 · p. 2
- Text-based metrics can miss traffic-scene spatial, temporal, motion, road-rule, and causal details, and a dedicated driving-scene MLLM benchmark remains largely unexplored.The paper notes that semantically similar responses can still be contextually incorrect.Source: §1 · p. 2
- NuPlanQA-1M provides 1M real-world multi-view QA pairs, while NuPlanQA-Eval offers 8K multiple-choice items across nine subtasks and three core skills.BEV-LLM integrates BEV features with visual and language modalities as a multi-view VQA baseline.Source: §1 · p. 3
2. Methodology
The methodology structures driving-scene questions into a hierarchy of nine tasks, builds NuPlanQA from multi-view nuPlan data, and compares it with existing datasets. It also introduces BEV-LLM, which refines and fuses multi-view visual features with temporally enriched BEV context.
2.1. Overview
- The nine subtasks progress from static road cues to spatial relations and then ego-centric maneuver, assessment, and recommendation tasks.The three groups are Road Environment Perception, Spatial Relations Recognition, and Ego-Centric Reasoning.Source: §2.1 · p. 3
2.2. NuPlanQA
- NuPlanQA uses nuPlan’s four-city, 4.3M-frame corpus; sampled multi-view images, past frames, and annotations support GPT-4o generation of hierarchical QA pairs.Approximately 1M pairs go to training, while the multiple-choice evaluation set is human-reviewed and divided into train, validation, and held-out test subsets.Source: §2.2 · pp. 3–4
2.3. Dataset Comparison
- NuPlanQA-1M is larger and broader than prior driving VQA datasets, while front-view-only alternatives remain limited in perspective and task coverage.NuPlanQA includes more than 90% free-form QA pairs, excluding short-answer items.Source: Table 1 · p. 4, §2.3 · p. 5
2.4. BEV-LLM Framework
- BEV-LLM uses LLaMA-3.2-Vision features, temporal self-attention over historical BEV embeddings, and cross-attention to fuse projected BEV context into visual features.Multi-layer features are concatenated and projected into the language model’s embedding space, where they are combined with text and vehicle information.Source: §2.4 · pp. 5–6
3. Experiments
On NuPlanQA-Eval, BEV-LLM leads open-source models on six of nine subtasks with multi-frame inputs, while traffic-light and spatial reasoning remain difficult for MLLMs. Ablations report gains from BEV features and multi-view, multi-frame inputs.
3.1. Settings
- BEV-LLM training begins with three past front-view frames, approximately 1.5 seconds of video, before BEV-Fusion is added for continued training.The initial pretraining adapts LLaMA-3.2-Vision-11B to multi-timestep images without the BEV-Fusion module.Source: §3.1 · p. 6
- Evaluation compares MLLMs across nine subtasks using accuracy on multiple-choice questions, with structured multi-view images supplied to models other than the multi-view-trained BEV-LLM baseline.The four-option format gives random guessing a baseline accuracy of 25%.Source: §3.1 · p. 6
3.2. Main Results
- 61.1% accuracy with multi-frame inputs and 58.1% with single-frame inputs: BEV-LLM leads open-source models on traffic-light detection.Seven of nine models recorded their lowest score on this subtask; identifying which signal to prioritize remains challenging.Source: §3.2 · pp. 6–7, Table 2 · p. 7
- 78.7% average accuracy and leadership on six of nine subtasks: BEV-LLM outperforms open-source models with multi-frame inputs.It also surpasses the second-best model by an average of 6.2% across the three ego-centric reasoning tasks.Source: §3.2 · p. 6, Table 2 · p. 7
- Spatial relations recognition is a particular weakness: it relies on visual input and temporal reasoning, while most MLLMs struggle with short-term spatial reasoning.GPT-4o and Gemini-1.5-Pro score 6.9% and 4.5% below their respective total averages in this skill.Source: §3.2 · p. 7
3.3. Ablation Study
- 11.3%, 13.3%, and 14.9% accuracy improvements across the three skills: adding BEV-Fusion improves the baseline under the same multi-view, multi-frame setting.The authors attribute stronger gains in spatial relations and ego-centric reasoning to BEV features' more stable spatial representation across frames.Source: Table 3 · p. 7, §3.3 · p. 8
- 13.2% average improvement over the baseline with multi-view, multi-frame inputs exceeds the 5.9% gain with single-view, single-frame inputs.Multi-frame inputs yield an approximate 5.3% gain over single-frame inputs under the same settings.Source: Table 3 · p. 7, §3.3 · p. 8
4. Discussion
The discussion highlights short-term, driver-perspective reasoning and diverse driving scenarios as challenges for MLLMs. It also describes how BEV features support multi-view understanding and how integration choices may affect adoption.
Reasoning on Driving Scenes
- General MLLMs often lack driver-perspective recognition, while driving decisions require instantaneous reasoning within less than 2–3 seconds.The discussion points to denser frame sampling and architectural adjustments for short-term prediction.Source: §4 · p. 8
- Language datasets still lack diverse corner cases and complex environments; the discussion calls for developing existing diverse-scenario datasets into language-oriented datasets.Source: §4 · p. 8
Multi-view Understanding of MLLMs
- Combining image and BEV features supports stable multi-view fusion that can overcome occlusions and extend field of view.The discussion also says BEV projection simplifies RADAR and LiDAR fusion, and that encoder and adapter choices matter for integrating BEV features into different LLMs.Source: §4 · p. 8
5. Conclusion
The paper presents NuPlanQA-Eval and NuPlanQA-1M, and reports that BEV-LLM leads open-source models overall. It identifies model backbones, fusion approaches, and dataset diversity as avenues for further work.
5. Conclusion
- NuPlanQA-Eval and NuPlanQA-1M provide an MLLM evaluation benchmark and a large-scale multi-view VQA dataset, respectively.Source: §5 · p. 8
- Most evaluated MLLMs struggle with traffic-light detection and color recognition, spatial relations, and ego-centric reasoning.Source: §5 · p. 8
- BEV-LLM achieves the highest overall score among open-source models at 78.7%, and the paper identifies other backbones, fusion approaches, and more diverse datasets as future directions.The authors note that BEV-LLM uses LLaMA and propose further experiments integrating BEV inputs into other MLLMs.Source: §5 · p. 8