Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model

1. Introduction

• Existing 3D LLM methods are limited to text or numerical outputs and struggle to locate objects from complex or abstract concepts.The paper notes these methods primarily rely on spatial relationships in scenes.Source: §1 · p. 2
• A direct [SEG]-token adaptation can struggle with sparse, unstructured point clouds, particularly for small objects in large scenes.The paper presents this challenge as motivation for a hierarchical decoder.Source: §1 · p. 2
• Reason3D groups point features into superpoints, aligns them with textual instructions, and uses a coarse location prior to refine the final object mask.The LLM produces [LOC] and [SEG] embeddings: [LOC] guides a region mask, which serves as a prior alongside [SEG].Source: §1 · p. 2
• Reason3D processes point clouds and language to produce textual outputs and detailed 3D segmentation masks for reasoning, searching, referring, and question answering.The paper also introduces 3D reasoning segmentation and a dataset for evaluating it.Source: §1 · p. 2

2. Related Work

• Point-cloud segmentation research includes point- and voxel-based methods, while open-vocabulary approaches and complex-instruction interaction represent related directions.The paper frames advanced interaction with 3D systems as an emerging need.Source: §2 · pp. 2–3
• 3D referring methods use language and spatial cues to identify objects, but the paper says they do not fully explore deeper reasoning.Examples include TGNN, X-RefSeg3D, and 3D-STMN [52].Source: §2 · p. 3
• Multimodal LLM research integrates visual data for question answering and reasoning; LISA [26] uses a segmentation token decoded into a mask.Source: §2 · p. 3
• Prior 3D LLM approaches are described as limited to bounding-box grounding or textual responses, whereas Reason3D supports fine-grained segmentation of searched objects.The passage contrasts Reason3D with Point-LLM [56], 3D-LLM [19], and LL3DA [10].Source: §2 · p. 3
• Figure 2 pairs a Matterport3D sample with the answer “pool table” and a ScanNetV2 sample with the answer “fireplace.”Source: Figure 2 · p. 3

3. 3D Reasoning Segmentation

• 3D reasoning segmentation maps a point cloud and complex textual instruction to a 3D mask, including queries that require world knowledge beyond simple object descriptions.The example asks for an object typically found in a living room that can seat multiple people.Source: §3 · p. 3
• The authors collected Matterport3D and ScanNetv2 scans with complex text instructions and detailed masks, yielding 1,339 training samples and 1,145 validation samples.Source: §3 · p. 3
• Hierarchical searching extends reasoning segmentation to locating an object within a specified place in a larger scene, such as finding an object to sit on inside a bedroom.Source: §3 · p. 3
• Figure 3 depicts a pipeline in which point features are pooled into superpoints, aligned with a learnable query and text in a frozen LLM, then decoded through [LOC] and [SEG] into a mask.The [LOC]-based coarse location acts as a prior for final mask prediction.Source: Figure 3 · p. 4

4.1. Alignment between LLMs and Point Cloud

• The framework extracts point-cloud features, pools them into superpoints, and aligns them with a decoder-only LLM using an interactor and learnable query.The point encoder and LLM are frozen while the interactor is updated.Source: §4.1 · p. 4
• For large scenes, superpoint pooling reduces N point features to M aggregated features; the paper cites a Matterport3D scene with about one million points as a motivating scale.The interactor maps superpoint features and a learnable query to Q′, which is fed with text instructions to the frozen LLM.Source: §4.1 · p. 4

4.2. Hierarchical Mask Decoder

• The hierarchical mask decoder uses a location prompt to predict a coarse region mask, then uses that mask as a prior alongside the segmentation prompt to produce the final object mask.The two prompts are derived from instruction-conditioned LLM embeddings.Source: §4.2 · p. 5

4.3. Training Reason3D

• Training combines an autoregressive cross-entropy loss for text generation with mask losses based on binary cross-entropy and Dice for region- and object-level superpoints.The region-level ground truth is defined using a distance threshold or, for hierarchical searching, points within the specified room.Source: §4.3 · p. 5

5.1. Experimental Setting

• The experimental setting uses ScanRefer and Sr3D for expressive referring segmentation, ScanQA for question answering, and a Reason3D dataset built from ScanNetV2 and Matterport3D for reasoning segmentation.The model uses a pretrained Sparse 3D UNet and FlanT5; evaluation includes mIoU and accuracy at specified IoU thresholds.Source: §5.1 · pp. 5–6

5.2. 3D Reasoning Segmentation Results

• Reason3D significantly outperforms previous methods in 3D reasoning segmentation, including open-vocabulary and two-stage approaches, and adapts better than 3D-STMN to indirect queries.The task requires spatial understanding, reasoning, and contextual comprehension; 3D-STMN excels in direct referencing but struggles with indirect queries.Source: §5.2 · p. 6

5.3. 3D Hierarchical Searching Results

• On Matterport3D hierarchical searching, the full Reason3D model outperforms the two-stage baseline, while HMD’s coarse-region token improves performance as point-cloud complexity increases.The region-segmentation baseline improves over Reason3D-base but faces optimization challenges in its two-stage training pipeline.Source: §5.3 · pp. 6–7, Table 2 · p. 6

5.4. 3D Referring Expression Segmentation Results

• On ScanRefer validation, Reason3D significantly outperforms 3D-STMN overall and vastly outperforms the LLM-based 3D-LLM among compared referring-segmentation approaches.The comparison also includes 3D grounding methods that predict bounding boxes, with points inside boxes extracted as mask predictions.Source: §5.4 · p. 7, Table 3 · p. 7

5.5. Ablation Study

• Removing region supervision drops accuracy to 14.27% at 0.25 IoU; probability-based masks outperform hard thresholding, and summation outperforms concatenation for combining features with location priors.These ablations evaluate HMD designs on Matterport3D with 3–4 rooms.Source: §5.5 · p. 7, Table 4 · p. 7
• Superpoint pooling reduces training complexity, average pooling performs better, and combining Dice and BCE losses gives more favorable outcomes than either loss alone.The reported superpoint and loss ablations are on ScanNetV2 for 3D reasoning segmentation.Source: §5.5 · p. 7, Table 5 · p. 8

5.6. Visualization Results

• Figure 4 visualizes predicted masks for 3D reasoning segmentation queries, with purple regions indicating the masks.The examples pair each textual query with its input point cloud.Source: §5.6 · p. 7, Figure 4 · p. 8

6. Conclusion

• Reason3D generates textual responses and segmentation predictions, and introduces 3D reasoning segmentation for interpreting implicit instructions in 3D scenes.Source: §6 · p. 8
• The hierarchical mask decoder first identifies a broad region likely to contain the target, then uses it as a feature prior to refine the segmentation mask.Experiments on ScanNetV2 and Matterport3D report performance across reasoning segmentation, hierarchical searching, referring segmentation, and question answering.Source: §6 · p. 8

• Section A provides additional details about the dataset.Source: Large Language Model · p. 12
• Section B elaborates on implementation and presents further results.Source: Large Language Model · p. 12
• Sections C to E discuss potential failure cases, limitations, and broader impact.Source: Large Language Model · p. 12

A.1. Dataset Annotation

• ScanNetV2 and Matterport3D supply room type and instance-object tags that are incorporated into GPT-4 prompts for dataset annotation.Each scene pairs a textual query with a binary segmentation mask identifying the target object.Source: §A.1 · p. 12

A.2. Dataset Statistics

• The collected dataset uses official training and validation splits, with 934 training and 837 validation samples from Matterport3D, and 405 training and 308 validation samples from ScanNetV2.Source: §A.2 · p. 12

A.3. 3D Hierarchical Searching Extension

• Hierarchical searching specifies a target room and expands its search space with neighboring rooms; this extension uses a diverse Matterport3D subset because ScanNet contains only single-room scenes.Source: §A.3 · p. 12

B.1. Implementation Details

• Training uses two NVIDIA RTX A6000 GPUs, batch size 16, AdamW, a 1,000-step linear learning-rate warm-up, and cosine decay.Inference uses batch size 1.Source: §B.1 · p. 12

B.2. Used Datasets

• The training data spans ScanRefer and Nr3D for referring segmentation, ScanQA for question answering, and Reason3D data from ScanNetV2 and Matterport3D for reasoning segmentation.The Matterport3D scene encoder is not fine-tuned.Source: §B.2 · p. 12

B.3. Training

• Training first freezes a fully trained sparse 3D UNet backbone, then pre-trains the entire network on other tasks before task-specific fine-tuning.Source: §B.3 · p. 12

B.4. Instruction Template

• Task templates use scene and task-specific text placeholders, with [LOC] for the target room and [SEG] for the object mask in hierarchical searching.The question-answering template requests an answer and a related segmentation mask.Source: §B.4 · pp. 13–14

B.5. 3D Question Answering Results

• Reason3D significantly improves ScanQA validation metrics over baseline methods and 3D-LLM, and also visualizes related segmentation masks.Table 8 reports better results than all baseline models.Source: Table 8 · p. 14, §B.5 · p. 14

B.6. 3D Visual Grounding Results.

• For visual grounding, Reason3D derives 3D boxes from predicted masks after DB-SCAN noise removal, and evaluates them with Acc@0.25 and Acc@0.5.The metrics measure the percentage of predicted boxes whose IoU with ground truth exceeds 0.25 or 0.5.Source: §B.6 · p. 14
• Reason3D outperforms recent 3D visual grounding methods and significantly surpasses LLM-based methods such as 3D-LLM [19].Table 9 reports ScanRefer validation results; Reason3D uses no 3D box annotation for training.Source: §B.6 · p. 14, Table 9 · p. 14

B.7. More Visualization Results.

• Qualitative referring-segmentation examples show correct predictions with multiple similar objects or long query sentences.The examples compare Reason3D predictions with 3D-STMN [52].Source: §B.7 · p. 14

C. Failure Case.

• Reported failure cases include small objects, similar objects or point-cloud structures, and questions requiring complex world knowledge.Similar structures include mirrors and sensor-induced fragments.Source: C. Failure Case · p. 14

D. Limitations

• Reason3D struggles with large-scale scenes, including identifying an object within a 30-room Matterport house.The paper suggests refining the hierarchical decoder or integrating adaptive multi-scale processing.Source: D. Limitations · p. 14
• Queries involving absent objects may challenge the model, and its performance on multi-object or multi-category tasks remains untested.The implementation is designed primarily for single-object queries.Source: D. Limitations · p. 17

E. Broader Impact

• Reason3D's natural-language customization may increase misuse potential, and dataset or pre-trained-model biases could influence performance.The paper frames these as broader-impact concerns.Source: E. Broader Impact · p. 17