SQA3D: Situated Question Answering in 3D Scenes

• Unlike mostly third-person 3D grounding tasks focused on spatial understanding, SQA3D uses an imagined first-person view; unlike embodied QA, it uses a simplified QA-only protocol to enable larger-scale collection and more knowledge-intensive questions.The paper frames embodied scene understanding as interpreting an agent’s situation and surroundings from a dynamic egocentric view before perceiving, reasoning, and acting.Source: §1 · p. 2
• SQA3D asks agents to localize a text-described position and orientation in a 3D scene, then answer questions requiring situated reasoning from that perspective.The benchmark brings together situation understanding—imagining and interpreting surroundings from arbitrary situations—and situated reasoning about the environment based on the agent’s current situation.Source: §1 · p. 2
• The dataset contains 20.4k descriptions of 6.8k situations from 650 ScanNet scenes and 33.4k questions spanning spatial relations, navigation, commonsense, and multi-hop reasoning.Situation descriptions were crowdsourced, with participants instructed to select diverse locations and orientations.Source: §1 · pp. 2–3
• 47.2% for the best model versus 90.06% for amateur human testers; challenge-alleviation settings improve models, indicating that situation understanding remains a key difficulty.The paper also reports that caption-based GPT-3 and Unified QA can be bottlenecked by limited spatial understanding and inaccurate captions.Source: §1 · p. 3
• The authors hypothesize that proper 3D representations and better situation understanding are crucial for embodied scene understanding.They examine state-of-the-art multimodal reasoning models and report that they continue to struggle on SQA3D.Source: §1 · p. 3

2 THE SQA3D DATASET

• Each SQA3D instance is a scene, a text-described agent situation specifying position and orientation, and a question answered from a fixed answer set.The task optionally includes predicting the situation’s ground-truth location and orientation, which could help address situation-understanding challenges.Source: §2 · p. 3

2.1 DATA FORMATION

• 650 ScanNet scenes were retained after manual categorization by object and layout richness and space volume.Source: §2.1 · p. 3
• To manage dense scene exploration and annotation work, collection was divided into situation identification, question preparation, and answer collection with a human study.Participants were recruited on AMT and used an interactive web-based interface.Source: §2.1 · p. 4, Figure 3 · p. 4

I. Situation Identification. We ask the workers to pick 5 situations by changing the location ⟨spos, srot⟩of a virtual avatar

• Workers choose five situations by changing a virtual avatar’s location and orientation, then write descriptions that uniquely identify each situation in the scene.Examples and bonuses encourage natural descriptions involving human activities; collected situations are manually curated for diversity and low ambiguity, with more added when needed to cover scene areas.Source: §I · p. 4
• Workers prepare questions for each scene and situation description, and responses that do not depend on the current situation are removed and penalized.Workers receive tutoring and examples to encourage substantial situated reasoning; additional workers answer the questions using the same interface to maintain consistency with question collection.Source: §I · p. 4

2.2 CURATION, DATA STATISTICS, AND METRICS

• 20.4k descriptions and 33.4k diverse, challenging questions across 6.8k unique situations remain after language cleanup, low-effort removal, and answer-distribution balancing.The authors also describe SQA3D as the largest dataset of grounded 3D scene understanding with human-annotated question-answering pairs; its crowd-sourced descriptions include human activities, and its questions are more diverse and balanced than counterparts.Source: §2.2 · pp. 4–5
• The dataset is split into train, validation, and test sets, with ScanNet validation scenes divided to form the latter two; evaluation uses exact-match accuracy over 706 top-K answer candidates.The authors exclude answers appearing very few times and state that exact match suffices to measure differences among baseline models.Source: §2.2 · p. 5

3D model

• The modified ScanQA model uses VoteNet object-centric features, LSTM encoders for the question and situation text, and cross-attention to predict answers.The object features attend successively to situation and question tokens before fusion; an optional head predicts the agent’s position and orientation, and object detection is also trained.Source: §3 · pp. 5–6

Auxiliary task

• Situation understanding is encouraged through auxiliary predictions of situation position and orientation, trained with MSE losses alongside the answer loss.The total objective is L = Lans + αLpos + βLrot, with α and β balancing the auxiliary losses.Source: §3 · p. 6

Video and Image-based model

• For egocentric-video and BEV inputs, ClipBERT and MCAN receive the situation text prepended to the question as context.The models are borrowed from video QA and VQA, respectively, with most default hyperparameters retained.Source: §3 · p. 6, Figure 6 · p. 6

Zero-shot model

• Zero-shot GPT-3 and Unified QA are prompted with a 3D scene caption, situation text, and question to generate an answer.Few-shot examples improved GPT-3 results, and minor post-processing was used to ensure answer quality.Source: §3 · p. 6, Figure 6 · p. 6

4.1 SETUP

• The setup compares 3D scans, egocentric video, and BEV images, tunes models on validation data, and reports test results.ScanNet supplies the scans and videos; videos and captions are down-sampled, and BEV images are rendered from above each scene.Source: §4.1 · pp. 6–7

4.2 QUANTITATIVE RESULTS

• 47.2% for ScanQA + aux. task versus 90.06% for amateur humans; all machine learning models fall behind human participants.Participants received only a limited number of examples before solving the SQA3D problems.Source: Table 3 · p. 7, §4.2 · p. 8
• Models generally do better on “Is” and “Can” than “What” questions, while the blind-test gap is largest for “What” and “Which,” indicating greater need for visual information.The authors attribute some of the pattern to binary-answer questions having fewer answer candidates.Source: §4.2 · p. 7
• Removing situation descriptions worsens results, while auxiliary situation-prediction tasks improve overall performance, especially on “What” questions; “How” questions are an exception.The authors suggest that many “How” questions involve counting, which may be guessable from the question when situation understanding is imperfect.Source: §4.2 · p. 7
• ScanQA’s 3D scans outperform MCAN and ClipBERT using BEV pictures or egocentric video, though generalist-specialist differences and vision bias may also contribute to the gap.BEV and video models also perform worse than the blind test; the authors suggest their visual inputs may exacerbate vision bias.Source: §4.2 · pp. 7–8

4.3 QUALITATIVE RESULTS

• In Figure 7, correct predictions attend to objects relevant to the situation and question, and identifying the right instance matters when objects share a category.For the first question, only ScanQA + aux. task predicts correctly and attends to the right chair; ScanQA attends to the wrong instance.Source: Figure 7 · p. 8, §4.3 · pp. 8–9

4.2 QUANTITATIVE RESULTS

• Unified QA Large and GPT-3 fail to deliver reasonable overall results, although GPT-3 + ScanRefer reaches 39.67% on “What” questions, above the best ScanQA variant.Changing the caption source from ScanRefer to ReferIt3D consistently affects model performance, which the authors cite as a possible bottleneck.Source: §4.2 · p. 8

4.4 ADDITIONAL TASK FOR LOCALIZATION

• The localization task evaluates predicted position with Acc@0.5m and Acc@1.0m, and orientation with Acc@15° and Acc@30°; evaluation uses x,y position and z-axis rotation.The paper reports random-prediction results as a reference.Source: §4.4 · p. 9

Embodied AI

• SQA3D targets embodied scene understanding, motivated by embodied agents’ poor generalization from training tasks to novel, more complex tests.Prior embodied AI work includes robotic navigation (Das et al., 2018; Anderson et al., 2018; Savva et al., 2019; Chen et al., 2019; Wijmans et al., 2019b; Qi et al., 2020; Deitke et al., 2022) and vision-based manipulation (Kolve et al., 2017; Puig et al., 2018; Xie et al., 2019; Shridhar et al., 2020a;b; 2022); models have used multimodal fusion and from-scratch training (Perez et al., 2018; Fried et al., 2018) as well as pretrained models (Pashevich et al., 2021; Suglia et al., 2021).Source: §5 · p. 9

Grounded 3D understanding

• Unlike prior 3D grounding and reasoning tasks that mostly use a passive, third-person perspective, SQA3D requires egocentric problem-solving.Earlier work grounded words with 3D detection and segmentation (Dai et al., 2017; Chang et al., 2017), while later work addressed sentence-level semantics (Chen et al., 2020; Chen et al., 2021) and complex 3D visual reasoning (Azuma et al., 2022; Ye et al., 2021; Yan et al., 2021).Source: §5 · pp. 9–10

Multi-modal question answering

• SQA3D extends multimodal QA toward embodied scene understanding and is compatible with 3D scans, egocentric video, or BEV pictures.The work builds on QA research spanning VQA, common-sense and factual knowledge (Marino et al., 2019), and video and 3D QA (Lei et al., 2018; Jia et al., 2020; 2022; Grunde-McLaughlin et al., 2021; Wu et al., 2021; Datta et al., 2022; Ye et al., 2021; Azuma et al., 2022; Yan et al., 2021).Source: §5 · p. 10

6 CONCLUSION

• SQA3D benchmarks embodied scene understanding by combining situation understanding and situated reasoning.The dataset includes 20.4k situation descriptions and 33.4k questions.Source: §6 · p. 10
• Its diverse questions probe embodied agents’ navigation, common sense, and multi-hop reasoning.The benchmark preserves relatively large scale while curating diverse situations and interesting questions.Source: §6 · p. 10
• The best ML model still trails human performance significantly, and findings suggest proper 3D representations and better situation understanding are crucial.The authors hope SQA3D will foster research toward better embodied scene understanding and more intelligent embodied agents.Source: §6 · p. 10

A.1 DATA COLLECTION WEB UI

• The collection interface combined an interactive 3D scene viewer, clickable objects or tags, task instructions, scene and object-only views, and multiple responses per scene.The interfaces were developed based on Chen et al. (2020).Source: §A.1 · p. 16
• Tutorials with examples and animations, clear rejection and bonus criteria, and a qualification test were used to support high-quality annotation.The authors report that these tutorials and instruction sets were helpful for collecting high-quality data.Source: §A.1 · p. 16

A.2 DATA POST-PROCESSING

• Grammatical cleaning combined rule-based correction with GECToR, with the correction threshold adjusted using human judgment of corrected samples.GECToR is attributed to Omelianchuk et al. (2020).Source: §A.2 · p. 16
• To reduce question-answer bias, questions were resampled by prefix and answer type to seek a more balanced answer distribution.The authors follow Antol et al. (2015) and Marino et al. (2019), and provide pre- and post-balancing answer distributions in Section B.1.Source: §A.2 · p. 16

A.3 MORE MTURK DETAILS

• MTurk eligibility required a 95% approval rate and over 1,000 completed tasks, relaxed to 90% approval for Stage III; access was enabled in 12 listed countries and regions.The listed regions were US, DE, GB, AU, CA, SG, NZ, NO, SE, FI, DK, and IE.Source: §A.3 · p. 16
• Participants received $0.5 per Stage I and II task and $0.2 per Stage III QA task, with possible quality-based bonuses; each task received five responses.Response quality was monitored, with bonuses or rejections sent daily.Source: §A.3 · p. 16
• Task lifetimes were 10 days for Stage I and 20 days for Stages II and III, although most tasks were completed in under seven days.Source: §A.3 · p. 16

B.1 MORE STATISTICS

• Prefix-based balancing removes dominance by any single answer within question types, but may not prevent models from exploiting n-gram patterns.The authors suggest collecting more questions with less-frequent answers as future work.Source: §B.1 · p. 16, Figure 12 · p. 19, Figure 13 · p. 20
• Most situation descriptions and questions are 10–20 words long.The appendix provides histograms for both lengths.Source: §B.1 · p. 17, §C.3 · p. 20

B.2 DETAILS ON EGOCENTRIC VIDEO AND BEV IMAGE

• Egocentric videos use every 20th ScanNet frame resized to 224 × 224; BEV images are rendered at 1920 × 1080 and resized to 224 × 224 for MCAN.The BEV camera is placed seven scene-bounding-sphere radii from its center.Source: §B.2 · p. 17

C.1 INPUT PIPELINE

• ClipBERT receives eight randomly sampled clips of two frames each, with frames resized to 1000 × 1000; ScanQA follows its original input pipeline.MCAN images are transformed to fit the ImageNet-pretrained encoder.Source: §C.1 · p. 17

C.4 ADDITIONAL DETAILS ON SCANQA/MCAN/CLIPBERT

• ScanQA is adapted by separating situation and question inputs, using two self- and cross-attention blocks, and concatenating language with visual features.The reference branch is discarded, and language-classification supervision becomes regression; blind and no-situation ablations respectively drop VoteNet features or replace situation text with unknown tokens.Source: §C.4 · p. 21
• MCAN uses RelViT’s raw-image implementation with situation text prepended to the question; ClipBERT uses its official repository and does the same.MCAN’s learning-rate decay is canceled; ClipBERT uses the MSR-VTT QA configuration, changes evaluated question types, and disables mixed precision due to instability.Source: §C.4 · p. 21

• Figures 15 and 16 provide additional qualitative results and failure modes.Source: D ADDITIONAL EMPIRICAL RESULTS · p. 21, Figure 15 · p. 22

Limitations

• SQA3D covers mostly indoor ScanNet household scenes, excluding outdoor and warehouse environments, and its static scans limit evaluation to non-interactive situation understanding and reasoning.The authors note that this may limit relevance to autonomous driving and warehouse robots, as well as tasks involving interaction with articulated objects such as drawers.Source: E LIMITATION AND POTENTIAL IMPACT · p. 22

Societal impact

• SQA3D’s situation and QA annotations could support household-assistant robots, while its released annotation interfaces and metadata invite customized QA datasets and 3D-language research.Situation annotations connect natural-language descriptions to scene locations and require commonsense reasoning; the QA tasks benchmark embodied-agent capabilities in household domains.Source: E LIMITATION AND POTENTIAL IMPACT · p. 22