NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario

Tianwen Qian; Jingjing Chen; Linhai Zhuo; Yang Jiao; Yu-Gang Jiang

Source-linked AI summary

NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario

Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, Yu-Gang Jiang

arXiv:2305.14836v2May 24, 2023cs.CV

TL;DR

Existing VQA benchmarks do not adequately represent autonomous-driving scenes, which combine multi-modal, multi-frame, outdoor visual data. The paper introduces NuScenes-QA, generating questions from annotated scene graphs and manual templates, and reports a balanced 34K-scene, 460K-pair benchmark whose baseline results expose substantial challenges.

Problem
Existing VQA benchmarks inadequately address autonomous-driving questions involving multi-modal, multi-frame, outdoor scenes and moving foreground objects.
Method
NuScenes-QA uses annotated 3D labels and bounding boxes to build scene graphs, manually designed templates to specify questions, and programmatic instantiation with filtering.
Results
NuScenes-QA contains 34K scenes and 460K question-answer pairs, while the best baseline reaches 60.4% accuracy compared with 84.3% for GroundTruth.
Takeaways & Limitations
The benchmark provides a foundation for evaluating and improving models that understand and reason over complex autonomous-driving visual data.
Takeaways & Limitations
Counting remains the hardest question type, with the best baseline achieving 23.2% accuracy.

Abstract

from arXiv · show

We introduce a novel visual question answering (VQA) task in the context of autonomous driving, aiming to answer natural language questions based on street-view clues. Compared to traditional VQA tasks, VQA in autonomous driving scenario presents more challenges. Firstly, the raw visual data are multi-modal, including images and point clouds captured by camera and LiDAR, respectively. Secondly, the data are multi-frame due to the continuous, real-time acquisition. Thirdly, the outdoor scenes exhibit both moving foreground and static background. Existing VQA benchmarks fail to adequately address these complexities. To bridge this gap, we propose NuScenes-QA, the first benchmark for VQA in the autonomous driving scenario, encompassing 34K visual scenes and 460K question-answer pairs. Specifically, we leverage existing 3D detection annotations to generate scene graphs and design question templates manually. Subsequently, the question-answer pairs are generated programmatically based on these templates. Comprehensive statistics prove that our NuScenes-QA is a balanced large-scale benchmark with diverse question formats. Built upon it, we develop a series of baselines that employ advanced 3D detection and VQA techniques. Our extensive experiments highlight the challenges posed by this new task. Codes and dataset are available at https://github.com/qiantianwen/NuScenes-QA.

Introduction

NuScenes-QA addresses the mismatch between existing VQA benchmarks and autonomous-driving scenes by introducing a benchmark for multi-modal, multi-frame, outdoor visual reasoning. It combines scene-graph-based question generation with manually designed templates and establishes baselines for this task.

Motivation: Existing VQA models struggle with autonomous-driving questions requiring accurate object identification, status recognition, and complementary visual information.The scenario involves multi-modal, multi-frame, outdoor data and moving foreground objects against static backgrounds.
Contribution: NuScenes-QA introduces a VQA benchmark specifically designed for autonomous-driving scenarios and their distinctive visual-data characteristics.The benchmark targets complex visual understanding and reasoning in multi-modal, multi-frame, outdoor scenes.
Dataset: 34K scenes and 460K question-answer pairs comprise the NuScenes-QA dataset.The dataset includes 377K training pairs and 83K testing pairs generated from annotated nuScenes scenes.
Construction: Scene graphs from annotated objects, attributes, and 3D spatial relations are combined with manually designed templates to generate question-answer pairs automatically.Template parameters are instantiated and answers are inferred from the scene graph.
Evaluation: The paper establishes baseline models, evaluates existing techniques, and conducts ablations to support future research on autonomous-driving VQA.The contribution evaluates models’ ability to understand and reason over complex visual data.

Related Works

Related VQA work spans image-based and video-based benchmarks, with conventional systems combining visual feature extraction, question processing, and joint vision-language embeddings.

Existing VQA Benchmarks: Image-based VQA benchmarks include VQA2.0, CLEVR, and GQA, while TGIF-QA and TVQA provide video-based evaluation.These datasets represent established image and video settings for visual question answering.
VQA Methods: Earlier image-based VQA systems commonly extract image features with CNNs, process questions with RNNs, and decode from joint vision-language embeddings.The visual and linguistic representations are combined through concatenation or related operations before decoding.

3D Visual Question Answering

3D-QA targets questions about point-cloud scenes and spatial relations, but existing benchmarks do not cover the multi-modal, multi-frame, outdoor complexity of autonomous driving. NuScenes-QA addresses this gap through structured scene-graph and template-based generation.

3D Visual Question Answering: 3D Visual Question Answering answers questions about 3D scenes represented by point clouds, emphasizing geometric structure and object spatial relations.Existing examples include manually annotated and semi-automatically generated question-answer datasets based on ScanNet.
Limitations of Existing 3D-QA: Current 3D-QA datasets and models remain limited for autonomous-driving scenarios involving multi-modalities, multiple frames, and outdoor scenes.The cited 3D-QA datasets are based on indoor point-cloud data.
Autonomous-Driving Language Tasks: NuScenes-QA differs from language-guided driving benchmarks by requiring high-level question understanding and reasoning over both images and point clouds.This contrasts with datasets focused on object referral or tracking from language descriptions.
Dataset Construction: Its data-construction flow generates scene graphs from annotated labels and 3D boxes, instantiates manually designed question templates, and filters the resulting pairs.The flow adapts automated generation ideas from structured VQA benchmarks to autonomous-driving data.

NuScenes-QA Dataset

NuScenes-QA constructs autonomous-driving VQA data by converting annotated scenes into relational graphs and instantiating manually designed templates. The resulting benchmark contains 34,149 scenes and 459,941 question-answer pairs with balanced, diverse question distributions.

Scene Graph Construction: NuScenes-QA treats each annotated nuScenes keyframe as a scene and represents annotated objects and attributes as scene-graph nodes.Relationships are derived from 3D bounding-box annotations.
Scene Graph Construction: Six spatial relations—front, back, front left, front right, back left, and back right—are computed from object-center angles relative to the ego-car’s forward direction.The forward direction is defined as 0° with counterclockwise angles positive.
Question Template Design: Manually designed templates encode question type, expected answer, and reasoning requirements, including zero-hop and one-hop questions across five categories.The 66 templates cover existence, counting, query-object, query-status, and comparison questions.
Q&A Pair Generation and Filtering: Question-answer pairs are generated by assigning template parameters through depth-first search, deducing graph-based answers, and filtering ill-posed or degenerate instances.Post-processing also removes unsuitable expressions, invalid attribute-object combinations, and counting answers greater than 10.
Dataset Statistics: 459,941 question-answer pairs span 34,149 visual scenes, with 376,604 training questions and 83,337 testing questions.The dataset includes 28,130 training scenes and 6,019 testing scenes.
Dataset Statistics: Question lengths range from 5 to 35 words, while answer and question-category distributions are balanced to reduce answer biases and language shortcuts.The statistics are reported for NuScenes-QA’s training split.

Method

The baseline combines multimodal, multi-frame scene inputs with image and point-cloud feature extraction, object-region embeddings, and a question-answering head. It uses BEV representations and cross-modal interaction to predict answers, with established detection and VQA components evaluated on the test set.

Task Definition: The task selects an answer from a predefined answer space by maximizing P(a | S, Q) for visual scene S and question Q.NuScenes-QA scenes may include multi-view images, point clouds, and preceding frames; single-modality or single-frame inputs are also possible.
Framework: The baseline has three components: a feature-extraction backbone, a region-proposal module for object embeddings, and a QA head for answer prediction.The framework processes multimodal inputs into BEV features before object-feature extraction and question answering.
Object Embedding: Detected 3D bounding boxes define rotated BEV regions whose enclosed features are pooled into object embeddings.A cross-product procedure identifies pixels inside each rotated rectangle before mean pooling.
Question Embedding: Question tokens are initialized with pretrained GloVe embeddings and encoded by a single-layer bidirectional LSTM into a question embedding.Each word feature concatenates the forward and backward hidden states.
Visual Feature Extraction: Image and point-cloud streams produce multimodal BEV feature maps, using image depth lifting and point-cloud voxelization with sparse convolution.The image branch uses ResNet with FPN, while the point-cloud branch applies 3D sparse convolution to voxel grids.
Answer Head and Training: MCAN models language and visual context with self-attention and cross-attention, then projects fused features to the answer space through MLP layers.Object embeddings are extracted offline using a pretrained 3D detector, and the answer head is trained with cross-entropy loss.

Experiments

Experiments evaluate modality, QA-head, bounding-box, cropping, and pooling choices across NuScenes-QA. Results show complementary camera-LiDAR information, strong effects from QA-head design, and persistent difficulty in counting.

Results: 53.4% accuracy for Q-Only is 7% below MSMDFusion+MCAN, indicating that language shortcuts cannot replace visual information.The authors report that rich visual information is necessary for strong performance.
Results: 60.4% accuracy is achieved by MSMDFusion, outperforming camera-only and LiDAR-only settings through complementary camera-LiDAR information.The reported single-modality accuracies are 57.9% for CenterPoint and 59.5% for BEVDet; GroundTruth reaches 84.3%.
Results: 59.5% overall accuracy for CenterPoint+MCAN is 1.4% higher than CenterPoint+BUTD, showing that QA-head choice substantially affects performance.The experiments compare BUTD and MCAN while holding the detection backbone constant.
Results: 23.2% accuracy is the best baseline result for counting, making it the most difficult question type in the benchmark.The paper identifies counting modules as a possible direction for future improvement.
Ablation Studies: 13.5% accuracy improvement occurs when ground-truth box features are added, whereas detected boxes decrease performance by 0.6%.The authors attribute the decrease to noise in immature detected boxes and note that point-cloud XYZ already expresses position well.
Ablation Studies: 58.9% accuracy for Max Pooling is 0.6% lower than Mean Pooling, while circumscribed-box cropping is slightly inferior to rotated-box cropping.The authors relate these results to structure-focused questions and oversmoothing from large circumscribed regions around elongated objects.

Conclusion

NuScenes-QA is presented as a large-scale multimodal VQA benchmark for autonomous driving, generated from visual scene graphs and question templates. Baseline models and comprehensive experiments establish a foundation for future research.

NuScenes-QA applies visual question answering to autonomous driving scenarios.
The benchmark is automatically generated from visual scene graphs and question templates, containing 34K scenes and 460K question-answer pairs.
A series of baseline models and comprehensive experiments provide a foundation for future research.

Dataset Comparison

NuScenes-QA is distinguished from other 3D VQA datasets by its larger scale, multimodal data, and autonomous-driving content. The authors position it as a research resource for 3D visual question answering.

34K visual scenes and 460K question-answer pairs make NuScenes-QA the largest 3D VQA dataset described in the comparison.It averages 13.5 question-answer pairs per scene, whereas other listed datasets typically contain fewer than 1000 scenes.
NuScenes-QA stands out from other 3D VQA datasets in scale, data modality, and content.
The dataset is presented as an important research resource for advancing 3D visual question answering.

Question Templates

NuScenes-QA uses manually designed templates to programmatically generate questions from scene-graph information. The templates cover five question types and distinguish zero-hop from one-hop reasoning.

Manually designed templates generate questions programmatically using status, object, and relation slots.For example, a template combines object attributes with a spatial relation to form a question about bicycles and a moving car.
The benchmark contains five question types: existence, counting, query-object, query-status, and comparison.
Zero-hop questions require no reasoning between objects, whereas one-hop questions require one step of spatial reasoning.

Question Distribution of the First Four Words

The first-four-word distribution indicates balanced questions with diverse visual semantics, including object categories and status attributes. Visualized examples further expose multimodal fusion and scene-complexity challenges, while the dataset remains limited in semantic and linguistic coverage.

Question Distribution of the First Four Words: The first-four-word distribution is balanced and spans diverse visual semantics.Questions require object categories such as pedestrians and motorcycles as well as statuses such as moving or parked.
Visualization Examples: Different coordinate systems, background noise, and dense object layouts make spatial fusion, foreground recognition, and reasoning difficult.
Visualization Examples: Different baseline predictions show that multimodal fusion models usually outperform single-modal models.The comparison is presented as evidence of complementarity between point clouds and images.
Discussion: The questions focus on foreground objects and spatial relationships while omitting background connections, richer semantic relations, and sufficiently diverse language.The authors attribute limited language diversity partly to template-based generation.