3D Question Answering

1 INTRODUCTION

• The introduced 3DQA task answers questions from color point clouds and requires both appearance understanding and comprehension of 3D geometry.The introduction highlights questions about object orientation and spatial relationships as examples of the added geometry demands.Source: §1 · pp. 1–2
• ScanQA is a fully human-annotated collection of natural, free-form, open-ended QA in free-perspective scans, with 10,062 pairs across 806 ScanNet scenes.It includes questions about appearance, geometry, spatial relationships, and comparisons, and provides annotator confidence.Source: §1 · pp. 1–2
• 3DQA-TR performed comparatively better than representative VQA methods on 3DQA questions using images from ScanNet videos.The authors attribute the result to the need for spatial information and to the method's use of both geometry and appearance.Source: §1 · p. 1
• 3DQA-TR uses separate encoders for appearance and geometry, then fuses those features with the question in 3D-Linguistic BERT to predict answers.The geometry encoder models object features and spatial relationships; the appearance encoder is pretrained on a synthetic dataset tailored to appearance extraction.Source: §1 · pp. 1–2

2.1 VQA

• VQA models commonly combine image and language encoders with fusion; transformer examples include LXMERT, VilBERT, and VL-BERT.The related work notes that VQA extensions span areas such as video understanding and embodied question answering, but remain limited to 2D images.Source: §2.1 · p. 2
• The related work identifies direct 3D-scene question answering as an unaddressed extension of VQA beyond image-based settings.It distinguishes direct point-cloud input from multi-view or RGB-D QA.Source: §2.1 · p. 2

2.2 3D Vision and Language.

• Prior point-cloud language work includes localization, captioning, and relationship grounding; localization and captioning methods are restricted to objects in limited categories.ScanRefer is described as requiring multi-view ScanNet photos, whereas this work takes point clouds as input.Source: §2.2 · pp. 2–3
• MP3D-EQA uses 2.5D RGB-D frames for navigation and templated questions, while 3DQA covers a wider range of objects and spatial and visual concepts.The related work also contrasts MP3D-EQA's restricted viewpoints and question forms with free-view point-cloud input and free-form questions.Source: §2.2 · p. 3

3 3DQA TASK AND SCANQA DATASET

• ScanQA introduces 3D question answering: answering natural, free-form, open-ended questions about real-world 3D scenes represented as point clouds with color channels; its dataset is built on ScanNet.ScanNet is an indoor-scene dataset of 1,613 scans from 806 scenes.Source: §3 · p. 3, §3.1 · p. 3

3.1 Dataset Collection

• Annotators explored colored 3D scans with adjustable viewpoints, camera settings, and transparency; collection procedures promoted diverse questions and filtered duplicates and easy queries.Questions and answers were crowdsourced through Amazon Mechanical Turk, with syntax checking and manual correction.Source: §3.1 · p. 3

3.2.1 Question and Answer Statistics

• 12.51-word questions average more than VQA’s 5.1; 25.51% of ScanQA answers contain two or more words, versus VQA’s 9.65%.Simple yes/no answers can still require fine-grained object recognition and aggregating placement information across multiple objects.Source: §3.2.1 · pp. 3–4

3.2.2 Question Distribution Based on the First Three Words

• ScanQA questions span appearance, spatial concepts, placement, navigation, and aggregation across more than two objects.Examples include reachability, arrangement, and determining which object is closer.Source: §3.2.2 · p. 4

3.2.3 Answer Distributions over Different Type of Questions

• Only a few question types are typically answerable with “yes” or “no”; most have diverse answers, although “can you” questions show a roughly three-to-one imbalance favoring “yes.”The authors note that some questions may be answerable without extensive scene comprehension because of this bias.Source: §3.2.3 · p. 4

3.2.4 Comparison with VQA Dataset

• Compared with image VQA, ScanQA emphasizes spatial concepts, placement, and spatial comparisons, while 3D scans provide information across views and can reduce single-view occlusion.Questions may also require viewpoint selection, navigation, or aggregating information across multiple objects; some clues cannot be recovered from the shown 2D views alone.Source: §3.2.4 · pp. 5–6

3.2.5 Comparison with 3D Captioning Dataset

• In a human study, participants given scene captions performed significantly worse than participants given 3D scenes, indicating captions are insufficient for 3DQA.Table 2 compares exact-match performance under the two input settings; its caption attributes the captions to ScanRefer [27].Source: §3.2.5 · p. 6, Table 2 · p. 7
• Unlike 3D captions restricted to limited object classes, ScanQA uses free-form questions that can cover fine-grained objects and entire scenes; its word distributions differ significantly from ScanRefer.The reported tests include Kolmogorov-Smirnov, paired t-test, and Anderson-Darling results.Source: §3.2.5 · pp. 6–7

4.1 3DQA-TR Framework Overview

• 3DQA-TR predicts answers by classifying over candidate answers using scene and question inputs.Its three jointly trainable parts are the appearance encoder, geometry encoder, and 3D-L BERT.Source: §4.1 · p. 7

4.2 Geometry Encoder

• The geometry encoder detects objects, extracts per-object geometry features, and encodes their locations, scales, and bounds to represent spatial relationships.It derives object features by tracing detected-object indices back to scene and detector representations; the box vector is normalized by scan dimensions for scale invariance.Source: §4.2 · p. 7

4.3 Appearance Encoder

• The separate appearance encoder extracts color features for detected objects and is pretrained on synthetic color-question-answer pairs.The synthetic questions concern object colors and are generated without overlap with ScanQA.Source: §4.3 · pp. 8–9

4.4 3D-Linguistic BERT

• 3D-L BERT combines appearance, geometry, and linguistic elements, using intra-modal aggregation and inter-modal alignment to predict answers.Question tokens can selectively attend to geometry or appearance clues, while element-specific scaling parameters are jointly learned.Source: §4.4 · p. 9

5 EXPERIMENTS

• Experiments compare 3DQA-TR with state-of-the-art 2D VQA approaches, examine human performance under different inputs, and ablate major components.The experiments report quantitative and qualitative results.Source: §5 · p. 9

5.1 Data Preparation and Evaluation Metric

• ScanQA is split by disjoint scenes into 8,551 training, 609 validation, and 902 test questions; evaluation uses EM and METEOR.The test set is divided into Yes/No, Color, Number, and Other answer types; METEOR is included because over one quarter of answers exceed one word.Source: §5.1 · p. 9

5.2 Training

• The model initializes 3D-L BERT from pretrained BERT-base and pretrains its detector and appearance network before jointly fine-tuning the framework.The detector is pretrained on 3D object detection, and the appearance network on synthetic color-related questions; results are reported on the test split.Source: §5.2 · p. 9

5.3 Main Results

• 3DQA-TR outperforms LXMERT, VILBERT, and 12-in-1 in both EM and METEOR, especially on spatial questions, despite viewpoint-matched image inputs for the baselines.The comparison uses ScanNet video frames selected to match the ground-truth answer viewpoint and direction; the authors note images remain insufficient for 3DQA questions.Source: §5.3 · p. 10
• Human EM with both questions and 3D scenes exceeds 83%, while the Question+Image to Question+Scene performance gap exceeds 20.0% on spatial-related questions.The human study compares question-only, question-plus-image, and question-plus-scene settings.Source: §5.3 · pp. 10–11
• In qualitative examples, 2D baselines misjudge a comparison of two beds’ sizes, while 3DQA-TR answers correctly; Fig. 10 marks exact-match answers in green.Examples span spatial, placement, viewpoint/navigation, and aggregation questions.Source: §5.3 · p. 11, Figure 10 · p. 11
• 3DQA-TR exceeds related QA methods by more than 12.28% on spatial questions and 12.5% on aggregation, and remains ahead of Image+Q baselines adapted with 3D detectors.Table 4 covers multi-view images, panoramic images, panoramic video, spatial reasoning VQA, and 3D-detector adaptations.Source: §5.3 · p. 12

5.4.1 Component Validation

• Removing both scene encoders drops overall performance from 42.35% to 27.94%; appearance and geometry each improve on question-only input in their relevant question types.Without appearance, Color EM falls from 35.85% to 13.20%; without geometry, Color EM rises from 11.32% to 30.19% versus question-only, while spatial subsets benefit from geometry.Source: §5.4.1 · p. 12, Table 6 · p. 13

5.4.2 Appearance Questions Pre-training

• Appearance pretraining improves overall EM from 40.80% to 42.35% and Color EM from 15.09% to 35.85% versus training from scratch.The appearance encoder is pretrained on generated color-question-answer data; its accuracy on that dataset is 45.03%, above the 25.74% most-frequent-answer lower bound.Source: §5.4.2 · p. 12, Table 6 · p. 13

5.4.3 Element Design

• Combining geometry and appearance into one element lowers Color EM from 35.85% to 33.96% and spatial EM from 40.35% to 35.67%.The ablation concatenates geometry features, spatial embeddings, and appearance features into one element rather than separating them.Source: §5.4.3 · p. 12, Table 6 · p. 13

5.4.4 Spatial Embedding

• Removing spatial embeddings lowers overall EM from 42.35% to 38.02% and spatial EM for spatial, placement, and viewpoint questions.The reported EM pairs are spatial 40.35% to 28.65%, placement 41.67% to 31.94%, and viewpoint 31.86% to 24.78%; overall METEOR falls from 25.71% to 23.35%.Source: §5.4.4 · pp. 12–13, Table 6 · p. 13

5.4.5 Different Detector

• Replacing Group-Free with Vote produces comparable overall and subclass performance under both EM and METEOR.The authors report this comparison as evidence of generalizability to different detectors.Source: §5.4.5 · p. 13, Table 6 · p. 13

6 CONCLUSION

• 3DQA-TR uses separate appearance and geometry encoders, then combines their outputs with question embeddings in a multimodal 3D-Linguistic BERT to predict answers.The framework is end-to-end.Source: §6 · p. 13
• ScanQA contains 10,062 question-answer pairs from 806 scenes to support the 3DQA task.Source: §6 · p. 13
• Experiments and analyses demonstrate 3DQA-TR's superiority over existing VQA baselines.Source: §6 · p. 13

7 LIMITATIONS

• ScanQA is less diverse than large-scale image VQA datasets because it is built on ScanNet's 806 indoor scenes.Source: §7 · p. 13
• The authors say limited dataset diversity impairs generalizability and robustness; humans are scarce in ScanQA, and the model tends to answer questions about them incorrectly.Source: §7 · p. 13
• The authors say ScanQA requires common sense about entity names and reasoning, while how far the model can go beyond perception and recognition remains unclear.Source: §7 · p. 13

8.1 ScanQA Dataset Collection

• The annotation interface showed annotators instructions and sample questions selected to encourage interesting, diverse questions; price and working-hours information was pixelated at annotators' request.Source: §8.1 · p. 16

8.2.1 Comparison with VQA

• Question types are classified by question-word appearances into spatial, spatial comparison to the average, placement, viewpoint and navigation, or aggregation categories.Source: §8.2.1 · p. 16

8.3 Details on Human Accuracy Collection

• Human accuracy was collected through AMT from people who were not ScanQA annotators, using question-only, question-plus-image, question-plus-scene, and question-plus-caption settings.The caption setting provided 10 referring expressions from ScanRefer.Source: §8.3 · p. 16

9 ADDITIONAL IMPLEMENTATION DETAILS

• The candidate answer vocabulary contains 166 answers for both training and testing.Source: §9 · p. 16
• Training uses four 2080Ti GPUs with a total batch size of 64 and the AdamW optimizer.Source: §9 · p. 16
• The initial learning rate is 1.0e-8 and is updated with a cyclical learning-rate policy.Source: §9 · p. 16

• For “can you” questions, “yes” answers are roughly three times as common as “no” and other answers, reflecting real-world data bias.Source: §10 · p. 16
• This prior can make some questions answerable without extensive scene comprehension and may further affect framework generalization.Source: §10 · p. 16
• Existing approaches to answer bias are generally task- and dataset-specific, making their application to 3DQA nontrivial; the authors are investigating extra annotations and more robust models.Source: §10 · p. 16
• Supplemental AMT figures show annotation prompts and answer-collection interfaces for Question, Question+Image, and caption-only settings without frames or scenes.Source: Figure 11 · p. 17, Figure 12 · p. 17, Figure 13 · p. 17
• The authors say the work offers insights into spatial understanding, navigation, and aggregation while motivating further study of the bias issue.Source: §10 · p. 17