Multi-modal Situated Reasoning in 3D Scenes

1 Introduction

• Situation awareness links agents’ multimodal local context with global environment status to support reasoning and planning in 3D scenes.The motivating contexts include text descriptions, images, and point clouds.Source: §1 · p. 1
• Situated modeling in 3D remains underexplored because prior simulated data has constrained diversity and complexity, while real-scene collection such as SQA3D is prohibitively expensive.These constraints restrict the scale of available situational data for model learning and evaluation.Source: §1 · p. 1
• 251K situated QA pairs from diverse real-world scenes are collected with a scalable automated pipeline, alongside navigation data for evaluating transfer from reasoning to action.The pipeline uses ScanNet, 3RScan, and ARKitScenes.Source: §1 · p. 2
• MSQA and MSNN evaluate situated reasoning and navigation with interleaved text, images, and scene point clouds to address ambiguity in text-only situation descriptions.MSQA covers questions such as object existence, counting, attributes, and spatial relationships; MSNN focuses on the immediate next action.Source: §1 · p. 2
• Existing vision-language models show limitations in modeling complex situations and fully using interleaved inputs, while MSR3D achieves superior results on both benchmarks.Data-scaling and cross-domain-transfer experiments demonstrate the efficacy of pre-training on MSQA.Source: §1 · pp. 2–3

2 Related Work

• Existing 3D vision-language tasks cover grounding, captioning, and question answering, but situated understanding remains absent from many tasks and contributes to a gap with embodied AI.Earlier situated-reasoning work often used simulated environments and simple questions; SQA3D introduced real-world scenes with a focus on spatial reasoning and scene understanding.Source: §2 · p. 3
• LLM-assisted data generation offers a route to scarce 3D vision-language data, but data quality remains a concern despite improved prompting and post-processing methods.Prior work has not resolved quality concerns or how generated data compares with human annotations; this paper also conducts a human study of its approach.Source: §2 · p. 3
• Text descriptions can leave object references ambiguous, while ego-view observations require bridging a modality gap; interleaved multimodal input is proposed as a general way to delineate situations.The format requires both vision-language grounding and multimodal situated reasoning.Source: §2 · p. 3

3 Multi-modal Situated Reasoning Dataset

• MSQA contains 251K multi-modal situated QA pairs from ScanNet, 3RScan, and ARKitScenes, each combining a scene point cloud, situated description, interleaved question, and answer.Table 1 compares MSQA with existing 3D vision-language datasets and defines its “situated” and “multi-modal” criteria.Source: §3 · p. 4, Table 1 · p. 5

3.1 Data Collection

• The LLM-based collection pipeline samples locations and viewpoints, builds situated scene graphs, and generates questions across nine scopes, including attributes, counting, spatial relations, and navigation actions.Situation sampling includes standable, sittable, and object-reachable locations; scene-graph relations are adjusted for the sampled situation and viewpoint.Source: §3.1 · p. 4, QA pairs generation · pp. 4–5
• Refinement checks scene-graph attributes and relations, manually reviews generated QA pairs, uses regular expressions to detect and correct errors, and balances answer distributions.These procedures yield the 251K pairs across ScanNet, 3RScan, and ARKitScenes.Source: QA pairs generation · p. 5

3.2 Data Quality Control

• In a 100-instance human study, MSQA matched SQA3D in average quality across three aspects and matched or exceeded its share of data scoring at least 4.Figure 4 presents question-type distribution, average quality scores, and proportions of high-scoring data compared with SQA3D.Source: §3.2 · p. 5, Figure 4 · p. 5

• MSQA evaluates answers to text-image interleaved questions grounded in a 3D scene, using a GPT-based correctness metric because responses are open-ended.The metric reports correctness score C over N test samples; each s_i rating ranges from 1 to 5, with higher scores better.Source: §4 · p. 6
• MSNN tests prediction of the best immediate next action from the agent’s current situation, textual goal, and overall scene, separating situated understanding from long-horizon planning.The current situation includes location, orientation, and text description.Source: §4 · p. 6
• MSNN data generation samples starting situations and goals, plans shortest paths with A* on floor plans, and derives the ground-truth next action from the trajectory direction.The dataset contains 34K samples across 378 ScanNet scenes.Source: §4 · p. 6

5.1 Model Settings

• MSR3D extends LEO for interleaved multimodal input and models situation by translating and rotating point clouds conditioned on the agent’s situation.Zero-shot LLMs and VLMs receive textual scene descriptions because they cannot directly handle 3D point clouds.Source: §5.1 · p. 7

5.2 Evaluation Results

• Evaluation uses average correctness across both tasks and varies input modality, scene representation, and model setting; MSNN also varies pretraining data.Source: §5.2 · p. 7

5.2.1 Multi-modal Situated Question Answering (MSQA)

• Zero-shot models struggle with spatial and navigation reasoning, while situation modeling improves those categories over LEO; blind MSR3D drops sharply.MSR3D (T+I) also drops significantly when images appear in questions.Source: §5.2.1 · pp. 7–8
• Textual scene descriptions underperform point clouds on attributes, spatial relations, and navigation, but do better below three targets and worse at three or more.The authors attribute the higher-count weakness to target objects being truncated by limited context length.Source: §5.2.1 · p. 7
• Removing the situation component causes a notable performance drop, especially on navigation questions.Source: §5.2.1 · pp. 7–8

5.2.2 Multi-modal Situated Next-step Navigation (MSNN)

• MSR3D (T) beats LEO (T) by +8.56% in navigation action prediction, while MSQA pretraining significantly improves both LEO and MSR3D on MSNN.Removing situation information also causes a great performance drop in MSR3D.Source: §5.2.2 · p. 9

5.3 Additional Analysis

• Scaling QA pairs, situations, or scenes yields consistent improvement on MSQA.The authors identify this as evidence of a scaling effect and potential for further scaling.Source: §5.3 · p. 9
• In-domain training performs best on each scene subset, while ARKitScenes training transfers less well; complex-scene training may benefit cross-domain generalization.Source: Cross-domain transfer · p. 9

6 Conclusion

• MSQA contains 251K multimodal situated question-answer pairs, and MSNN evaluates situated reasoning and embodied navigation.The conclusion reports that comprehensive experiments highlight the value of the dataset and benchmarks.Source: §6 · p. 9
• LLM-generated data can remain unnatural because of rule-based scene graphs and LLM biases, including descriptions that select distant objects improbable for humans.The authors call for human feedback to better align generated data with human preference.Source: §6 · p. 10
• The pipeline has not fully used available 3D assets, and prompting LLMs for large-scale QA generation is expensive.The authors suggest expanding scene coverage and training an LLM tailored to generate QA pairs as future directions.Source: §6 · p. 10
• The authors argue that situational-awareness evaluation should extend beyond question answering and action prediction, with object grounding as one example.Table 6 organizes cross-domain transfer experiments by source domain and target domain; its caption does not state outcomes.Source: Table 6 · p. 9, §6 · p. 10

A.1 Situation Sampling

• For standing situations, the location is sampled from the floor area and the orientation from an angle.The supplied description does not provide a legible angle range.Source: §A.1 · p. 16
• For sitting situations, the location is sampled from a sitable area, and orientation follows the backrest’s normal direction.Source: §A.1 · p. 16
• For interaction with large objects, the method parses an interactive part, samples a nearby floor location, and uses that part’s normal for orientation.Cabinets and refrigerators are given as examples of large objects.Source: §A.1 · p. 16
• For interaction with small objects, the method samples a nearby floor location and orients the agent toward the object’s center.Bags and trash cans are examples of small objects.Source: §A.1 · p. 16
• Situation examples include the presumed agent’s location, orientation, action description, and location description; image placeholders are replaced by corresponding images.Figure 8 illustrates these situation components and placeholders.Source: Figure 8 · p. 16

A.2.1 HOI Examples

• The pipeline generates varied human-object interaction descriptions with LLMs; it produced 4,210 descriptions for 421 objects.Table 7 provides examples of these descriptions for interactions with large and small objects.Source: §A.2.1 · p. 16, Table 7 · p. 16

A.2.2 Object Attribute Detection

• GPT-4V detects seven object-attribute types from cropped, deblurred multi-view images, producing concise and descriptive attributes.The attribute types are color, 3D shape, material, usage, texture, structure, and state.Source: §A.2.2 · p. 17, Table 8 · p. 17

A.3 LLM-assisted Data Generation

• ChatGPT is prompted to generate question-answer pairs and their categories, with a demonstration provided as an example.Figure 10 shows prompts, and Figure 11 shows a human demonstration.Source: §A.3 · p. 17, Figure 10 · p. 18, Figure 11 · p. 19

A.3.1 Refinement

• Refinement manually corrects counting, existence, and non-existence errors, checks other types, and filters unreasonable data points.Corrections are made according to situated scene graphs; Table 9 summarizes error statistics and Table 10 gives refinement examples.Source: Table 9 · p. 17, §A.3.1 · pp. 18, 20, Table 10 · p. 19

A.3.2 Data Balance

• Existence questions are balanced by adding no-responses, including responses about objects in the scene or in the wild.After adding 600 in-the-wild objects, the yes/no ratio nears 1:1.Source: §A.3.2 · p. 20, Figure 12 · p. 20

A.3.3 Situated Scene Description

• The appendix points to Table 11 for examples of situated scene descriptions.Figures 13 and 14 show the data visualizer and 3D viewer used in the human study.Source: §A.3.3 · p. 20, Table 11 · p. 20, Figure 13 · p. 21

A.4 Human Study

• The human study scores question quality, answer correctness, and situation quality; questions should be spatially aware and unambiguous, and situations should locate the agent uniquely.The study interface displays each instance’s location and orientation in a 3D viewer.Source: §A.4 · p. 21

A.6 Dataset Statistics

• MSQA statistics include question distributions by HOI type and data source, as well as scene quantity.Figure 16 and Table 12 present the dataset statistics, with the table giving fine-grained statistics.Source: §A.6 · p. 22, Figure 16 · p. 22, Table 12 · p. 23

B.1.1 Prompts for GPT-score

• GPT-score prompts include scoring examples intended to improve robustness and align scores with human evaluation.Figure 17 shows the prompt for LLM-assisted scoring.Source: §B.1.1 · p. 22, Figure 17 · p. 23

B.1.2 Correlation of GPT-scores and Human Scores

• On 200 manually scored responses, GPT-score correlates more closely with human scores than Exact Match or caption metrics.The compared metrics include Exact Match, Refined Exact Match, BLEU-4, METEOR, and ROUGE; Table 13 reports Pearson coefficients.Source: Table 13 · p. 23, §B.1.2 · p. 24, Table 14 · p. 24

B.2 MSNN Data Generation Details

• The MSNN pipeline samples a start and an object-interaction goal, plans an A* route through passable floor areas, and labels the next action by required orientation adjustment.The four possible actions are moving forward, turning left, moving backward, and turning right.Source: §B.2 · p. 24
• Start situations include an agent location, orientation, and text-image interleaved descriptions, while goals target navigating to and interacting with a sampled scene object.Source: §B.2 · p. 24

C.1.1 Overall Structure

• MSR3D extends LEO for text-image interleaving and represents 3D scenes as object tokens derived from Mask3D proposals, PointNet++ features, and a spatial transformer.Source: §C.1.1 · pp. 24–25
• Agent location and orientation normalize scene objects to place the agent at the origin facing the positive x direction, while OpenCLIP encodes instruction images.Source: §C.1.1 · p. 25

C.1.2 Model Training

• MSR3D uses autoregressive prefix language modeling to predict response tokens, with LoRA tuning and selected modules trained while the LLM and PointNet++ encoder are frozen for pretraining and fine-tuning.Source: §C.1.2 · p. 26

C.1.3 Choice of Situation Modeling

• On SQA3D with ground-truth object proposals, transformation-based situation modeling performs best among the tested methods and is selected for MSR3D.Source: §C.1.3 · p. 27, Table 17 · p. 27

C.1.4 Prompt Details

• The shared system prompt frames MSR3D as a 3D-scene visual assistant that uses scene objects and, when needed, an ego-view image to answer instructions.Source: §C.1.4 · p. 27

C.1.5 Scene Encoder Details

• A Spatial Transformer combines object-pair distances and angles with attention weights to encode spatial relations; the implementation uses three layers and eight heads.Source: §C.1.5 · pp. 27–28

C.2.1 Prompts for MSQA

• The MSQA GPT-4o prompts are provided, while GPT-3.5 prompts replace images with corresponding class labels.Source: §C.2.1 · p. 28

C.2.2 Prompts for MSNN

• For MSNN, the GPT-4o prompts are provided, and GPT-3.5 prompts likewise replace images with corresponding class labels.Source: §C.2.2 · p. 28

D.1 Scaling Effect on MSNN

• MSNN performance improves as the MSQA training scale increases, supporting the dataset’s effectiveness and scalability.Source: §D.1 · p. 28

D.2 More Qualitative Results and Failure Cases in MSQA

• GPT-4o can struggle with spatial reasoning despite accurate object coordinates and sizes, and the examples show perception and reasoning errors across both models.Failure cases include inaccurate object names or counts, spatial relations, and colors.Source: §D.2 · p. 28, Figure 26 · p. 33

D.3 Additional analyses for situation component

• Removing in-the-wild object-existence questions amplifies the measured impact of situation, supporting the hypothesis that those questions can be answered without situation understanding.Source: §D.3 · pp. 28–29, Table 18 · p. 29
• For spatial questions, directional answers and object-referential relations show notable differences across settings, supporting the authors’ hypothesis that situation matters for these subsets.Source: §D.3 · p. 29, Table 19 · p. 29