Source-linked AI summary
ScanQA: 3D Question Answering for Spatial Scene Understanding
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, Motoaki Kawanabe
TL;DR
Existing 2D question-answering models struggle with 3D spatial relations, occlusion, and object identification. This paper introduces 3D-QA, the ScanQA dataset, and a model that combines textual and 3D scene information; the baseline outperforms 2D-based VQA baselines on most evaluation measures.
Problem
2D image-based question answering struggles to represent relative directions, distances, occlusion, and consistent object identity in 3D scenes.
Method
The paper introduces 3D-QA, requiring answers and related 3D object localization, and develops ScanQA using textual and 3D scene encoding.
Results
The ScanQA baseline outperforms counterpart 2D-based VQA baselines on most evaluation measures, including exact match and image captioning metrics.
Takeaways & Limitations
The dataset and task support spatial understanding of embodied 3D scenes through object-grounded question answering.
Abstract
from arXiv · showhide
We propose a new 3D spatial understanding task of 3D Question Answering (3D-QA). In the 3D-QA task, models receive visual information from the entire 3D scene of the rich RGB-D indoor scan and answer the given textual questions about the 3D scene. Unlike the 2D-question answering of VQA, the conventional 2D-QA models suffer from problems with spatial understanding of object alignment and directions and fail the object identification from the textual questions in 3D-QA. We propose a baseline model for 3D-QA, named ScanQA model, where the model learns a fused descriptor from 3D object proposals and encoded sentence embeddings. This learned descriptor correlates the language expressions with the underlying geometric features of the 3D scan and facilitates the regression of 3D bounding boxes to determine described objects in textual questions and outputs correct answers. We collected human-edited question-answer pairs with free-form answers that are grounded to 3D objects in each 3D scene. Our new ScanQA dataset contains over 40K question-answer pairs from the 800 indoor scenes drawn from the ScanNet dataset. To the best of our knowledge, the proposed 3D-QA task is the first large-scale effort to perform object-grounded question-answering in 3D environments.
1. Introduction
The paper introduces 3D question answering, which combines whole-scene spatial understanding with textual question answering and localization of the relevant 3D objects. It addresses limitations of 2D approaches and contributes the ScanQA dataset and baseline model.
- 2D image-based question answering struggles with relative 3D directions, distances, occlusion, and tracking identical objects across images.
- Existing 3D question-answering datasets remain limited in size and question variety because they often rely on template-based collections.
- 3D-QA uses the entire 3D scene and a linguistic question to predict an answer phrase together with corresponding 3D bounding boxes.
- The ScanQA dataset is built from RGB-D indoor scans, human-edited questions, free-form answers, and object annotations derived from ScanNet.
- The proposed ScanQA model combines textual and 3D scene encoding, and it is evaluated against 2D-QA, 3D localization, and question-answering baselines.
2. Related work
Related work spans 2D visual question answering, 3D localization and captioning, and embodied question answering. ScanQA is positioned as spatial comprehension of an entire 3D scene with object identification.
- ScanQA differs by treating question answering as spatial comprehension of the entire 3D scene and by including object identification.
- VQA answers questions about 2D images using methods that model relationships among visual objects and, in some cases, video or 360° imagery.
- Unlike ScanRefer, ScanQA questions may require localization of multiple objects because several objects can appear in one question.
- Embodied question answering combines visual question answering with navigation for agents operating in 3D scenes.
3. ScanQA Dataset
The ScanQA dataset defines object-grounded question answering over complete 3D scenes and uses an automated-to-human collection process. Its questions are filtered, edited, answered, and annotated with relevant objects.
- 3.1. 3D-QA Task: 3D-QA requires models to answer questions from all information in a 3D scene while specifying related objects with 3D bounding boxes.
- 3.1. 3D-QA Task: Questions may reference multiple objects, so the task permits answers associated with multiple 3D bounding boxes.
- 3.2. Question-Answer Collection: The dataset pipeline generates seed questions from ScanRefer referring expressions, then filters, edits, and collects answers through human workers.
- 3.2. Question-Answer Collection: Workers evaluated seed questions as valid, too easy, unanswerable, or unclear, with each question assessed by at least three workers.
- 3.3. Dataset Statistics: 41,363 questions and 58,191 answers were collected, including 32,337 unique questions and 16,999 unique answers.
- 3.3. Dataset Statistics: The dataset includes two test sets, with and without object annotations, and multiple answers for validation and test questions.
4. ScanQA Model
ScanQA combines 3D scene representations with language to answer questions and localize question-related objects in indoor scans.
- 3D-QA takes a point cloud and question as input, aiming to generate an answer that semantically matches the true answer.
- 3D & language encoder: The 3D and language encoder transforms questions into contextualized word representations and point clouds into object proposals.
- 3D & language encoder: A biLSTM and nonlinear projection produce contextualized word features, while the scene-processing pathway detects objects from the scan.
- 3D & language fusion: Transformer encoder and decoder layers combine object features with language information to form question-aware object proposal representations.
- Object localization & QA: The model estimates object-box localization likelihoods and predicts object categories associated with the question before producing answers.
5. Experiments
Experiments compare ScanQA with 2D and 3D baselines, evaluate answer quality and localization, and analyze the effects of model components and input features.
- Experimental setup: The baselines include 2D image QA, 3D detection followed by MCAN, and ScanRefer-based pipelines that separate or jointly learn localization and QA.
- Quantitative analysis: ScanQA significantly outperformed VoteNet+MCAN, ScanRefer+MCAN (pipeline), and ScanRefer+MCAN (end-to-end) across data splits and evaluation metrics.
- Quantitative analysis: ScanQA exceeded RandomImage+MCAN, while OracleImage+MCAN performed better or more competitively, highlighting the importance of accurate object identification.
- Ablation studies: RGB values were ineffective for object localization and QA, while multiview image features slightly helped localization but not QA.
- Ablation studies: Geometry, preprocessed multiview image features, and normals produced the strongest feature combination, although its effect was limited.
- Qualitative results: Qualitative examples show ScanQA correctly localizing target objects and answering questions when the pipeline baseline localized an incorrect object.
6. Conclusion
The paper introduces 3D-QA, the ScanQA dataset, and a baseline model for answering questions about complete 3D scenes with object localization.
- ScanQA contains 41,363 questions and 32,337 unique answers from 800 ScanNet-derived scenes, and its baseline outperforms 2D VQA baselines on most measures.
Supplementary Material
The supplementary material provides additional experimental details, quantitative and qualitative results, and an annotation-website snapshot.
- The supplement covers experimental details, additional quantitative and qualitative results, and a snapshot of the annotation website.
A. Additional Experimental Details
Additional experiments compare ScanQA with image-based baselines and extend localization from one object to multiple objects. The comparisons use real, mesh, and top-down image inputs.
- Image-Based Baselines: Additional 2D-QA baselines use three real or mesh images sampled or rendered from the environment for each question.RandomImage and OracleImage distinguish randomly sampled and target-informed image selection.
- Multiple-Object ScanQA: ScanQA was extended to localize and label multiple objects when a question refers to more than one object.The extension normalizes proposal scores with a sigmoid function and trains localization and classification with binary cross-entropy loss.
- Image-Based Baselines: The image comparisons include real images, mesh images, and TopDownImage inputs for answering the same scene question.Figures show examples for the question “What color is the bathroom door?” across these image settings.
B.1. Object Localization Results
Object localization and question answering are evaluated against ScanRefer-based baselines, with ablations examining modules, input features, and model parameters. The reported analyses emphasize shared learning and feature choices for ScanQA.
- Object Localization Results: ScanQA, ScanRefer + MCAN (pipeline), and ScanRefer + MCAN (end-to-end) are compared using Acc@0.25 and Acc@0.5 localization accuracy.A positive prediction requires intersection-over-union with the ground truth above the specified threshold.
- Object Localization Results: Shared end-to-end learning of QA, object localization, and object classification effectively predicted the target object for a question.This conclusion is reported from the object-localization results in Table 6.
- Question Answering Results: ScanQA single and multiple models outperformed the listed image-based baselines across all evaluation metrics and splits, except for EM@1 in one test split.RandomImage + Oscar (real) exceeded ScanQA on EM@1 for the test without objects split but not on the other evaluation metrics.
- Ablation Studies: RGB values were effective for QA, while multiview image features were effective for both object localization and QA.The authors relate the broader feature use to predicting multiple objects.
- Performance by Parameter: L = 1 and hidden size d = 128 were suitable for the test splits among the evaluated parameter settings.The parameter studies varied the number of layers L and hidden size d.
- Accuracy for Question Types: Place questions received lower performance scores than other question types.The authors attribute this partly to multiple valid phrasings and longer answers, for which captioning metrics are more suitable than exact matching.
C. Additional Qualitative Analysis
Qualitative analyses visualize localization and QA behavior, while annotation and dataset analyses describe question diversity and the MTurk collection process. The examples contrast ScanQA with an end-to-end ScanRefer-based baseline.
- Qualitative Comparison: Qualitative evaluation compares ScanQA with ScanRefer + MCAN (end-to-end) using scene visualizations, predicted bounding boxes, and ground-truth annotations.The analysis examines both object localization and QA correctness.
- Qualitative Comparison: ScanQA correctly resolved several spatial references that led the baseline to localize nearby but incorrect objects.Examples include identifying a bathrobe on the wall, a backpack on a chair, and an object in a corner.
- Qualitative Comparison: ScanQA correctly interpreted the positional relationship between an ottoman, chair, and pillow, whereas the baseline localized a lamp and answered its color.The example links relational understanding with selecting the queried object.
- MTurk Annotation Details: MTurk workers used an interactive 3D modeling viewer that supported rotating and zooming during QA annotation.The website supported question editing and answer collection.
- MTurk Annotation Details: Auto-generated questions were filtered with rules and worker judgments, and underspecified questions were rewritten before answers were collected.At least two of three workers had to judge a question valid for it to remain.
- Question and Dataset Analysis: Human (re)annotation produced questions with varied meanings, many unique answers, and a fat-tailed distribution of question lengths.Examples range from “How many chairs?” to a longer relational description involving chairs and a red bag.
- Question and Dataset Analysis: The dataset includes question types classified as object, color, object nature, place, number, and other.Detailed accuracy is reported for valid, test with object, and test without object splits.