Source-linked AI summary
Segment Any 3D Object with Language
Seungjun Lee, Yuyang Zhao, Gim Hee Lee
TL;DR
Open-vocabulary 3D instance segmentation must recognize unseen objects from free-form language without relying on class annotations, but prior mask-generation strategies lose semantic or geometric information. SOLE predicts semantic-related masks directly from 3D point clouds using multimodal fusion and three association-based supervision signals, outperforming prior methods across three benchmarks and approaching fully supervised performance.
Problem
Method
SOLE directly predicts semantic-related masks from 3D point clouds through multimodal fusion and mask-visual, mask-caption, and mask-entity associations.
Source: §1 · pp. 2–3Results
SOLE outperforms previous methods by a large margin on ScanNetv2, ScanNet200, and Replica, with results close to the fully supervised counterpart.
Source: §1 · p. 3, §5 · p. 14Takeaways & Limitations
SOLE demonstrates versatility for free-form language instructions, including questions, attribute descriptions, and functional descriptions.
Source: §1 · p. 3
Abstract
from arXiv · showhide
In this paper, we investigate Open-Vocabulary 3D Instance Segmentation (OV-3DIS) with free-form language instructions. Earlier works that rely on only annotated base categories for training suffer from limited generalization to unseen novel categories. Recent works mitigate poor generalizability to novel categories by generating class-agnostic masks or projecting generalized masks from 2D to 3D, but disregard semantic or geometry information, leading to sub-optimal performance. Instead, generating generalizable but semantic-related masks directly from 3D point clouds would result in superior outcomes. In this paper, we introduce Segment any 3D Object with LanguagE (SOLE), which is a semantic and geometric-aware visual-language learning framework with strong generalizability by generating semantic-related masks directly from 3D point clouds. Specifically, we propose a multimodal fusion network to incorporate multimodal semantics in both backbone and decoder. In addition, to align the 3D segmentation model with various language instructions and enhance the mask quality, we introduce three types of multimodal associations as supervision. Our SOLE outperforms previous methods by a large margin on ScanNetv2, ScanNet200, and Replica benchmarks, and the results are even close to the fully-supervised counterpart despite the absence of class annotations in the training. Furthermore, extensive qualitative results demonstrate the versatility of our SOLE to language instructions.
1 Introduction
SOLE targets open-vocabulary 3D instance segmentation with free-form language by predicting semantic-related masks directly from 3D point clouds. Its multimodal fusion network and three multimodal associations improve language alignment, mask quality, and benchmark performance.
1 Introduction
- Earlier closed-set and base-category methods generalize poorly to unseen categories, while 2D-projection approaches overlook semantic or geometric information during mask generation.SOLE addresses these limitations by generating semantic-related masks directly from 3D point clouds.Source: §1 · p. 2
- Three multimodal associations—mask-visual, mask-caption, and mask-entity—align the 3D segmentation model with language and enrich mask prediction.The associations are designed to improve both mask quality and responsiveness to language instructions.Source: §1 · p. 3
- SOLE outperforms previous methods by a large margin on ScanNetv2, ScanNet200, and Replica, with results close to the fully supervised counterpart.The reported setting does not use class annotations during training.Source: §1 · p. 3
- SOLE responds to diverse free-form language inputs, including questions, attribute descriptions, and functional descriptions.Qualitative results demonstrate this versatility across language instructions.Source: §1 · p. 3
- SOLE directly predicts semantic-related 3D masks from point clouds while incorporating multimodal information for more generalizable segments.Its fusion network adds point-wise CLIP features to the backbone and language-domain features through a cross-modality decoder.Source: §1 · pp. 2–3
2 Related Work
Related work progresses from closed-set 3D instance segmentation toward open-vocabulary recognition by transferring 2D multimodal knowledge into 3D. Existing approaches still face limited semantic generalization and mask quality, motivating SOLE’s multimodal visual-language framework.
Closed-Set 3D Instance Segmentation
- Closed-set 3D instance segmentation assumes identical training and testing categories and therefore cannot handle unseen categories effectively.This limitation restricts application in real-world scenes despite advances in mask-based decoding.Source: §2 · p. 3
Open-Vocabulary 2D Segmentation
- Open-vocabulary 2D segmentation commonly transfers image-level multimodal embeddings to pixel- or mask-level tasks using large-scale vision-language models.Unified frameworks also extend open-vocabulary segmentation across instance, panoptic, and referring tasks.Source: §2 · pp. 3–4
Open-Vocabulary 3D Scene Understanding
- Open-vocabulary 3D methods lift 2D foundation-model knowledge to point clouds because 3D multimodal foundation models are unavailable.Many semantic-segmentation methods construct task-agnostic point-wise representations, while instance methods use base/novel splits, class-agnostic masks, or direct 2D prediction lifting.Source: §2 · p. 4
- Prior open-vocabulary 3D instance-segmentation methods remain limited by poor semantic generalization and low-quality mask prediction.SOLE addresses these limitations with a visual-language framework combining a multimodal network and multimodal associations.Source: §2 · p. 4
3 Method
SOLE targets OV-3DIS with free-form language by combining 3D and CLIP-derived features, textual decoding, and three multimodal associations for mask supervision.
3 Method
- SOLE defines OV-3DIS as detecting and segmenting instances from 3D point clouds without ground-truth class annotations, using text prompts at inference.Source: §3 · p. 4
3.1 Backbone Feature Ensemble
- The backbone concatenates learned 3D features with projected per-point 2D CLIP features across aligned resolutions before decoding.Source: §3.1 · pp. 5–6
3.2 Cross Modality Decoder (CMD)
- The cross-modality decoder fuses CLIP-combined point features with textual features in attention layers, enabling responses to descriptions, questions, and functional attributes.Source: Figure 2 · p. 5, §3.2 · pp. 6–7
3.3 Vision-Language Learning
- SOLE supervises masks with mask-visual, mask-caption, and mask-entity associations derived hierarchically from pooled CLIP features, generated captions, and noun phrases.Source: Figure 3 · p. 6, §3.3 · pp. 7–8
3.4 Training and Inference
- Soft multimodal matching combines the three association losses with mask losses, while inference fuses mask and pooled CLIP classification probabilities using a soft geometric mean with τ=0.667.Source: §3.4 · pp. 8–9
4 Experiments
SOLE is evaluated across closed-set and open-set 3D instance segmentation benchmarks, with ablations testing its multimodal fusion and association components. It achieves strong cross-dataset performance while supporting free-form language queries.
4.1 Experimental Setting
- Experiments use ScanNetv2, ScanNet200, and Replica for closed-set and hierarchical or cross-domain open-set 3D instance segmentation.Evaluation reports average precision at 25% and 50% IoU thresholds and average AP from 50% to 95% IoU.Source: §4.1 · pp. 9–11
4.2 Comparison with Previous Methods
- SOLE outperforms prior methods across ScanNetv2, ScanNet200, and Replica, including base, novel, and cross-domain open-set settings.It achieves competitive performance with fully supervised models and improves AP by 11.6% over OpenMask3D on ScanNet200→Replica.Source: Table 1 · p. 10, §4.2 · pp. 11–12, Table 3 · p. 12
4.3 Ablation Studies and Analysis
- Combining projected 2D CLIP features with 3D backbone features preserves semantic generalization and geometry detection, while CMD improves AP by 1.6%.Using either feature stream alone is sub-optimal, whereas their combination yields optimal results.Source: §4.3 · pp. 12–13, Table 5 · p. 13
- Mask-entity association is strongest for category alignment, while mask-visual and mask-caption associations improve free-form instruction recognition despite reducing some benchmark mask-accuracy metrics.With the instruction “I wanna see outside,” adding the latter associations enables segmentation of the related object.Source: §4.3 · pp. 13–14, Table 6 · p. 13, Figure 5 · p. 14
- Qualitative results show that SOLE responds to visual questions, attribute descriptions, and functional descriptions.These results demonstrate versatility across free-form language queries rather than only category-name instructions.Source: §4.3 · p. 14, Figure 4 · p. 14
5 Conclusion
SOLE is a multimodal framework for open-vocabulary 3D instance segmentation with free-form language instructions, outperforming prior methods and approaching fully supervised performance. The appendix adds implementation details, visual-grounding evaluation, CLIP analysis, and further qualitative results.
5 Conclusion
- SOLE outperforms previous methods by a large margin on three benchmarks while achieving competitive performance with the fully supervised counterpart.Source: §5 · p. 14
- The framework aligns 3D segmentation with varied free-form language instructions through multimodal fusion and three supervised multimodal associations.Source: §5 · p. 14
Appendix
- The appendix reports additional implementation details, 3D visual-grounding evaluation, CLIP-feature analysis, and qualitative mask-caption and segmentation results.Source: Appendix · p. 15
A Implementation Details
The implementation uses transformer-based mask prediction with object-query initialization and spatial post-processing, then generates mask captions and entities from CLIP features.
A Implementation Details
- Transformer-based mask prediction initializes masks from 150 farthest-sampled object queries, while DBSCAN splits non-contiguous masks into spatially contiguous clusters.The maximum distance is 0.95 and the neighborhood-point number is 1.Source: A Implementation Details · p. 15
- DeCap generates captions from pooled CLIP mask features, after which noun phrases extracted with TextBlob and spaCy form mask-entity associations.Captions are inserted into the prompt “a {} in a scene.” for alignment with the data.Source: A Implementation Details · p. 15
B 3D Visual Grounding
SOLE is evaluated for 3D visual grounding by converting language-conditioned instance masks into bounding boxes and comparing matching accuracy against specialist and generalist baselines.
B 3D Visual Grounding
- For each ScanRefer validation description, SOLE retrieves an instance mask whose bounding box is evaluated using ACC@25 and ACC@50.Source: B 3D Visual Grounding · p. 15
- 4% higher ACC@25 than 3D-LLM and 22.6% versus 27.4% ACC@50 against a fully supervised specialist demonstrate strong grounding performance with only mask annotations.SOLE ignores ScanRefer text descriptions during training and uses only ScanNetv2 mask annotations.Source: B 3D Visual Grounding · p. 16, Table 7 · p. 16
- The comparison includes five specialist grounding models and one generalist 3D-LLM baseline, with most baselines trained or fine-tuned on ScanRefer.Source: B 3D Visual Grounding · p. 16
C Analysis of CLIP Visual Feature
The CLIP-feature analysis examines their role in SOLE’s backbone and inference ensembles, including feature clustering and classification-probability combination.
C Analysis of CLIP Visual Feature
- CLIP visual features are analyzed for their role in both backbone feature ensembling and inference ensembling.Source: C Analysis of CLIP Visual Feature · p. 16
- K-means clustering visualizes different backbone features as color-coded clusters.Source: Figure 6 · p. 17
- Classification-probability ensemble analysis reports results on ScanNetv2 with 2cm voxel size.Source: Table 8 · p. 17
C.1 Backbone Feature Ensemble
SOLE combines 3D backbone features with projected CLIP visual features because each provides complementary information for semantic generalization and instance segmentation.
C.1 Backbone Feature Ensemble
- Combining 3D backbone and projected CLIP features improves semantic generalization and segmentation compared with using either feature alone.Projected CLIP features provide semantic information but weak instance detection, whereas 3D backbone features identify instances but degrade semantic generalization.Source: §C.1 · p. 17
- Projected CLIP features cluster objects by shared semantics, while 3D backbone features separate instances within categories.The visualizations show chairs forming one semantic cluster with projected CLIP features but three instance clusters with 3D backbone features.Source: §C.1 · p. 17
- The combined features detect the trash can semantically and separate chairs into six clusters.Source: §C.1 · p. 17
C.2 Inference Ensemble
SOLE ensembles 3D segmentation and CLIP predictions during inference, using a soft geometric mean that dynamically fuses their class probabilities.
C.2 Inference Ensemble
- SOLE pools per-point CLIP features within each predicted mask and combines the resulting CLIP and mask predictions for classification.Source: §C.2 · pp. 17–18
- The soft geometric mean achieves the best results among the evaluated probability-ensemble methods.The comparison includes no ensemble, the 3D model prediction alone, and a hard geometric mean.Source: §C.2 · p. 18
- SOLE remains competitive without CLIP prediction ensembling, supporting the generalization of its multimodal fusion network.Source: §C.2 · p. 18
D Qualitative Results
SOLE produces qualitative segmentations for diverse free-form language instructions and generates captions that capture object semantics and spatial information.
D Qualitative Results
- Generated mask captions contain semantic information about the corresponding 3D object and its location.Source: D Qualitative Results · p. 18
- Because captions may contain multiple nouns, SOLE aggregates noun phrases with attention in the mask-entity association.Source: D Qualitative Results · p. 19
- SOLE responds effectively to various free-form language instructions in qualitative segmentation examples.The reported examples include visual questions, attribute descriptions, and functional descriptions.Source: D Qualitative Results · p. 19