Source-linked AI summary
SQA3D: Situated Question Answering in 3D Scenes
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, Siyuan Huang
TL;DR
Embodied agents must understand a textual description of their position and orientation in a 3D scene before answering situated questions. SQA3D introduces a benchmark and dataset spanning diverse situations and reasoning abilities, but current models remain far below humans, while the benchmark is limited to mostly static indoor ScanNet environments.
Problem
Embodied agents still struggle to generalize across situations and tasks requiring knowledge-intensive reasoning from dynamic egocentric views.
Method
SQA3D combines textual situation understanding with situated question answering over 3D scene contexts, using crowdsourced situations and diverse reasoning questions.
Results
47.2% for the best model versus 90.06% for amateur human testers shows that current multimodal models remain substantially behind humans on SQA3D.
Takeaways & Limitations
SQA3D provides a benchmark for developing stronger situation understanding and situated reasoning in embodied scene understanding.
Takeaways & Limitations
SQA3D mainly covers static indoor household scenes from ScanNet, excluding outdoor, warehouse, and interactive articulated-object scenarios.
Abstract
from arXiv · showhide
We propose a new task to benchmark scene understanding of embodied agents: Situated Question Answering in 3D Scenes (SQA3D). Given a scene context (e.g., 3D scan), SQA3D requires the tested agent to first understand its situation (position, orientation, etc.) in the 3D scene as described by text, then reason about its surrounding environment and answer a question under that situation. Based upon 650 scenes from ScanNet, we provide a dataset centered around 6.8k unique situations, along with 20.4k descriptions and 33.4k diverse reasoning questions for these situations. These questions examine a wide spectrum of reasoning capabilities for an intelligent agent, ranging from spatial relation comprehension to commonsense understanding, navigation, and multi-hop reasoning. SQA3D imposes a significant challenge to current multi-modal especially 3D reasoning models. We evaluate various state-of-the-art approaches and find that the best one only achieves an overall score of 47.20%, while amateur human participants can reach 90.06%. We believe SQA3D could facilitate future embodied AI research with stronger situation understanding and reasoning capability.
1 INTRODUCTION
SQA3D targets embodied scene understanding by requiring agents to interpret a situated, first-person perspective before answering reasoning-intensive questions. The benchmark combines diverse 3D scenes, situations, and questions, exposing substantial gaps between current models and humans.
- Embodied scene understanding requires agents to understand their situation and surroundings from a dynamic egocentric view before perceiving, reasoning, and acting.
- SQA3D combines situation understanding—imagining arbitrary positions and orientations—with situated reasoning about the environment for complex action planning.
- SQA3D provides 20.4k situation descriptions for 6.8k unique situations and 33.4k questions collected from 650 ScanNet scenes.
- The questions cover spatial relations, navigation, common-sense reasoning, and multi-hop reasoning from an imagined first-person view.
- 47.2% of the best model versus 90.06% for amateur human testers reveals a large performance gap on SQA3D.
- Analysis indicates that current multimodal models struggle with situation understanding, 3D representations, spatial understanding, and accurate scene captions.
2 THE SQA3D DATASET
SQA3D formulates situated question answering over a scene context, textual situation, and question, with optional location prediction. Its dataset retains suitable ScanNet scenes and uses a staged annotation process to manage dense scene interaction.
- Each SQA3D instance is a triplet ⟨S, s, q⟩ comprising scene context, a situation, and a question.
- The textual situation description stxt specifies an agent’s position and orientation, with rotation applied before translation in the scene coordinate system.
- The task retrieves the correct answer from a candidate answer set and may additionally predict the ground-truth position and orientation.
- 650 ScanNet scenes were retained after excluding scenes whose object richness, layout, or volume made situation and question collection infeasible.
- The annotation workload is divided into Situation Identification, Question Preparation, and Answer Collection & Human Study.
I. Situation Identification. We ask the workers to pick 5 situations by changing the location ⟨spos, srot⟩of a virtual avatar
SQA3D collects and curates natural situation descriptions and situated questions through staged human annotation, then reports dataset statistics and evaluation procedures. The resulting questions are diverse and designed to depend on the agent’s current situation.
- I. Situation Identification: Workers select diverse avatar locations and orientations, write uniquely identifying situation descriptions, and manually curate them for diversity and low ambiguity.
- II. Question Preparation: Question writers are trained to create questions requiring situated reasoning, while situation-independent questions are removed or penalized.
- III. Answer Collection & Human Study: Additional workers answer the collected questions, and the study records their responses using the annotation interface.
- Curation: Curation applies grammar checks and removes or rewrites low-effort and template-like descriptions and questions.
- Statistics: SQA3D is described as the largest grounded 3D scene-understanding dataset with human-annotated question-answer pairs, featuring natural and diverse annotations.
- Dataset splits and evaluation metric: The dataset uses train, validation, and test splits, 706 top-K answer candidates, and exact match evaluation.
3 MODELS FOR SQA3D
SQA3D models combine scene context, situation descriptions, and questions through 3D, video/image, or zero-shot architectures. The approaches adapt existing QA systems or convert 3D scenes into text for language-model prompting.
- 3D model: The 3D model adapts ScanQA to map 3D scans, situation descriptions, and questions to answers.It uses object-centric 3D features, language encoders, and cross-attention transformer blocks.
- 3D model: The 3D architecture optionally predicts the agent’s situation location, represented by position and orientation.An auxiliary localization head predicts ⟨spos, srot⟩.
- Auxiliary task: Situation prediction is trained with auxiliary mean-square-error losses for position and orientation alongside answer prediction.The combined objective is L = Lans + αLpos + βLrot.
- Video and image-based model: Video and image models prepend the situation description to the question while using egocentric video clips or bird’s-eye-view pictures as scene context.This treats the situation description as context for standard video or visual QA models.
- Zero-shot model: Zero-shot models prompt GPT-3 or Unified QA with a 3D scene caption, situation description, and question.A 3D captioning model converts the scene into text before prompting the language model.
4 EXPERIMENTS
Experiments benchmark SQA3D across 3D scans, egocentric videos, and bird’s-eye-view images, then analyze question types, situation understanding, scene representations, zero-shot models, localization, and human performance. The results show that situation understanding and suitable 3D representations remain central challenges.
- 4.1 Setup: The benchmark evaluates models using 3D scans, egocentric videos, and bird’s-eye-view pictures, with test results reported after validation-set tuning.The 3D scans and videos come from ScanNet, while videos are down-sampled for computation.
- Question types: Models generally perform better on “Is” and “Can” questions than “What” questions, while scene context matters most for “What” and “Which” categories.Binary-answer categories have fewer candidates, whereas “What” and “Which” show the largest blind-test gaps.
- Situation understanding and reasoning: Removing situation descriptions worsens results, while auxiliary situation-prediction tasks improve overall performance, especially on challenging “What” questions.“How” questions are an exception, plausibly because many involve counting and can benefit from question-only guessing or memorization.
- Representations of 3D scenes: 3D-scan models outperform the tested video- and image-based models, suggesting scans better support holistic scene reasoning in this benchmark.The comparison may also reflect ScanQA’s task specialization and reduced vision bias relative to general-purpose systems.
- Qualitative results: Better situation understanding produces more reasonable attention over relevant objects and correlates with more robust answer prediction.Qualitative visualizations compare predicted answers with transformer attention over VoteNet-generated object boxes.
- Zero-shot vs. training from scratch: GPT-3 and Unified QA generally fail to produce reasonable results, although GPT-3 with ScanRefer captions reaches 39.67% on “What” questions.Changing the 3D-caption source from ScanRefer to ReferIt3D consistently affects performance.
- Human vs. machine: 47.2% of the best model versus 90.06% for amateur humans shows a large machine–human performance gap.The cited comparison is for ScanQA + auxiliary tasks versus amateur human participants.
5 RELATED WORK
SQA3D extends embodied AI, grounded 3D understanding, and multimodal QA by requiring reasoning from an imagined egocentric situation. Unlike much prior 3D work, it combines first-person scene understanding with broad knowledge-based questions.
- Embodied AI: Embodied AI studies intelligence through physical interaction with virtual or photorealistic environments and tasks such as navigation and manipulation.These tasks examine whether intelligence emerges through acting in environments.
- Grounded 3D understanding: Prior grounded 3D understanding commonly connects language to 3D visual data through detection, segmentation, and related grounding tasks.The cited work establishes 3D data as an immersive representation for linking language and the visual world.
- Grounded 3D understanding: Most related 3D tasks assume passive third-person observation, whereas SQA3D requires problem-solving from an egocentric viewpoint.This first-person framing introduces challenges and opportunities for embodied-AI tasks.
- Multi-modal question answering: Multimodal QA has expanded from visual question answering toward video, commonsense, and factual knowledge settings.SQA3D builds on this broader QA trajectory while situating questions in 3D scenes.
6 CONCLUSION
SQA3D benchmarks embodied scene understanding by combining situation understanding with situated reasoning across diverse 3D-scene questions. Its results identify substantial machine–human gaps and emphasize the importance of suitable 3D representations and situation understanding.
- 6 CONCLUSION: SQA3D contains 20.4k situation descriptions and 33.4k questions spanning navigation, common sense, and multi-hop reasoning.The benchmark is designed to preserve relatively large scale while including diverse situations and questions.
- 6 CONCLUSION: The best machine-learning models remain substantially behind humans, while the findings highlight better 3D representations and situation understanding as important directions.The conclusion frames SQA3D as a benchmark for advancing embodied scene understanding methods.
A.1 DATA COLLECTION WEB UI
The dataset collection interface combines interactive 3D scene exploration with staged instructions and quality controls, followed by grammatical cleaning and answer-distribution balancing.
- Web UI: The collection UIs provide a 3D scene viewer, clickable objects or tags, and instruction sets guiding each annotation task.Users can drag, rotate, and zoom the scene while highlighting objects through meshes or sidebar tags.
- Quality control: Detailed tutorials, clear rejection and bonus criteria, and qualification tests are used to improve annotation quality.Tutorials include examples and animated demonstrations across the collection stages.
- Post-processing: The dataset is grammatically corrected using rule-based procedures and the ML-based GECToR tool, with thresholds adjusted through human review.The correction pipeline combines automated methods with judgment over corrected samples.
- Post-processing: Questions are re-sampled by prefix and answer type to reduce question-answer bias and produce a more balanced answer distribution.This balancing procedure follows practices established in prior visual question-answering datasets.
- Crowdsourcing: The MTurk setup specifies eligible regions, approval requirements, per-task rewards, response monitoring, and task lifetimes.Five responses were collected for each task in all three stages.
B DATASET DETAILS
The dataset details describe balancing, scene representations, model inputs, zero-shot prompting, and implementation adaptations for baseline systems.
- Statistics: Prefix-based balancing prevents any single answer from dominating each question type, but may not eliminate n-gram answer-pattern bias.The authors identify collecting more questions with infrequent answers as future work.
- Statistics: Most situation descriptions and questions contain 10-20 words.The length distributions are summarized with separate histograms for situation descriptions and questions.
- Scene representations: Egocentric videos are formed by taking every twentieth ScanNet frame, while BEV images are rendered from a top-down camera and resized for model input.The rendering camera is placed seven scene-radius units from the bounding-sphere center, and inputs are resized to 224 × 224.
- Model inputs: ClipBERT samples 8 two-frame clips, while MCAN and other baselines use adapted image, video, or 3D-scene input pipelines.The ScanQA, MCAN, and ClipBERT implementations are modified to accommodate SQA3D’s situation and question inputs.
- Zero-shot models: Zero-shot evaluation uses situation-question prompts, with GPT-3 receiving a 1-shot example and Unified QALarge using greedy sampling.The prompt format places the situation description before the question and answer field.
D ADDITIONAL EMPIRICAL RESULTS
Additional qualitative results and failure modes are provided to illustrate model behavior and errors in SQA3D.
- Qualitative analysis: The paper provides additional qualitative examples and a failure-mode analysis as supplementary empirical evidence.The failure-mode figure highlights errors caused by not attending to relevant objects.
E LIMITATION AND POTENTIAL IMPACT
The paper discusses SQA3D’s household-assistant relevance and examples while bounding its scope to mostly static indoor scenes and non-interactive reasoning.
- Limitations: SQA3D’s mostly indoor household scenes do not cover outdoor or warehouse environments, limiting direct application to those settings.The authors specifically mention autonomous driving and warehouse robots as affected application domains.
- Limitations: Because ScanNet scenes are static, SQA3D does not evaluate interaction with articulated objects such as drawers.The authors characterize the benchmark as limited to hovering over 3D scenes and non-interactive scenarios.
- Potential impact: Situation and QA annotations could support household assistant robots by linking natural-language descriptions and instructions to scene locations.The paper identifies commonsense reasoning as part of this description-location connection.
- Examples: The examples require reasoning over spatial relations, relative directions, object attributes, and navigation actions from situated viewpoints.Questions include identifying an object above or to the left and deciding which way to go.