Weakly Supervised 3D Open-vocabulary Segmentation

Kunhao Liu; Fangneng Zhan; Jiahui Zhang; Muyu Xu; Yingchen Yu; Abdulmotaleb El Saddik; Christian Theobalt; Eric Xing; Shijian Lu

Source-linked AI summary

Weakly Supervised 3D Open-vocabulary Segmentation

Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, Shijian Lu

arXiv:2305.14093v4May 23, 2023cs.CV

TL;DR

Open-vocabulary 3D segmentation lacks diverse training datasets, while distilling from existing 2D models can weaken open-vocabulary behavior. The paper distills CLIP and DINO into a NeRF using only scene text descriptions and no manual segmentation annotations, and reports accurate boundaries, long-tail recognition, and performance surpassing fully supervised models in certain scenes.

Problem
Open-vocabulary 3D segmentation lacks large-scale, diverse datasets, and 2D-model distillation can compromise open-vocabulary behavior through close-vocabulary fine-tuning.
Method
The method weakly supervises NeRF with scene text descriptions while distilling CLIP multimodal knowledge and DINO boundary information through Selection Volume, RDA, and FDA losses.
Results
The method produces accurate boundaries, recognizes long-tail classes, and surpasses fully supervised segmentation models in certain scenes.
Takeaways & Limitations
The results suggest that 3D open-vocabulary segmentation can be learned from 2D images and text-image pairs without manual segmentation annotations.
Takeaways & Limitations
The method requires text labels before training and retraining for new text labels, and it struggles with complex scenes and large objects with unobvious textures.

Abstract

from arXiv · show

Open-vocabulary segmentation of 3D scenes is a fundamental function of human perception and thus a crucial objective in computer vision research. However, this task is heavily impeded by the lack of large-scale and diverse 3D open-vocabulary segmentation datasets for training robust and generalizable models. Distilling knowledge from pre-trained 2D open-vocabulary segmentation models helps but it compromises the open-vocabulary feature as the 2D models are mostly finetuned with close-vocabulary datasets. We tackle the challenges in 3D open-vocabulary segmentation by exploiting pre-trained foundation models CLIP and DINO in a weakly supervised manner. Specifically, given only the open-vocabulary text descriptions of the objects in a scene, we distill the open-vocabulary multimodal knowledge and object reasoning capability of CLIP and DINO into a neural radiance field (NeRF), which effectively lifts 2D features into view-consistent 3D segmentation. A notable aspect of our approach is that it does not require any manual segmentation annotations for either the foundation models or the distillation process. Extensive experiments show that our method even outperforms fully supervised models trained with segmentation annotations in certain scenes, suggesting that 3D open-vocabulary segmentation can be effectively learned from 2D images and text-image pairs. Code is available at \url{https://github.com/Kunhao-Liu/3D-OVS}.

1 Introduction

Open-vocabulary 3D segmentation is important for understanding diverse scene semantics but is limited by scarce datasets and the closed-vocabulary bias of many 2D models. This work distills CLIP and DINO knowledge into NeRF using only scene text descriptions, achieving annotation-free segmentation with accurate boundaries.

Motivation: Open-vocabulary 3D segmentation must handle diverse regions, long-tail objects, and natural-language references in real-world scenes.The task supports applications including robot navigation, object localization, autonomous driving, 3D editing, and augmented or virtual reality.
Challenge: Limited large-scale, diverse 3D segmentation datasets make open-vocabulary model training difficult.Existing datasets such as ScanNet focus on restricted scenes and limited object classes.
Challenge: Distilling from 2D open-vocabulary models avoids requiring 3D datasets but can compromise open-vocabulary behavior because those models are often fine-tuned on close-vocabulary data.The limitation is especially relevant for long-tail text labels.
Approach: The method distills CLIP multimodal knowledge and DINO object reasoning into NeRF using only open-vocabulary scene text descriptions.It uses CLIP’s Internet-scale text-image knowledge and DINO’s scene-layout and object-boundary information without segmentation annotations.
Approach: A hierarchical patch scheme and Selection Volume align CLIP image-level and pixel-level features without fine-tuning.Relevancy-Distribution Alignment addresses CLIP ambiguities, while Feature-Distribution Alignment distills DINO boundary information and re-balances similarity-based weights.
Outcome: The resulting system performs weakly supervised open-vocabulary segmentation with accurate boundaries and no manual segmentation annotations.Experiments surpass fully supervised segmentation models in certain scenes, and the contributions include recognition of long-tail classes with limited input data.

2 Related Work

Prior work commonly distills large pretrained models or 2D features for open-vocabulary segmentation, while 3D segmentation methods remain constrained by limited class variety and generalization to unseen classes. Foundation models provide general knowledge for downstream adaptation across language and vision tasks.

Open-vocabulary Segmentation: 2D open-vocabulary segmentation methods commonly distill image-text contrastive or diffusion models but often fine-tune on close-vocabulary datasets.This differs from the massive datasets used for initial foundation-model pretraining.
3D Scenes Segmentation: Traditional 3D segmentation methods use point clouds or voxels with limited class variety, restricting generalization to unseen classes.
Foundation Models: Foundation models are treated as a powerful paradigm because they capture general knowledge and adapt to varied downstream tasks.The cited background covers training paradigms in both natural language processing and computer vision.

3 Method

The method lifts CLIP’s open-vocabulary semantics and DINO’s scene-structure information into NeRF using weak supervision from multi-view images and class text descriptions. It adapts image-level CLIP features to pixels, selects suitable feature scales in 3D, and aligns relevancy and feature distributions for segmentation.

The method segments reconstructed NeRF scenes from multi-view images and open-vocabulary class descriptions, assigning each 3D point a class label.
Multi-scale and multi-spatial image patches adapt CLIP’s image-level features to pixel-level features while preserving surrounding-context invariance.Features from varying patch sizes and pixel positions are averaged within each scale.
A 3D Selection Volume chooses the feature scale most appropriate for each 3D point’s object size instead of simply averaging scales.Each point produces a selection vector over the available scales, which is normalized with Softmax.
Class relevancy maps are independently normalized within each view, then aligned with ray-level segmentation probabilities using Jensen-Shannon divergence to reduce CLIP ambiguities.The alignment helps identify image regions corresponding to each class text description.
Feature-Distribution Alignment reweights correlation losses toward DINO-similar locations and downweights dissimilar locations to mitigate instability from diverse distributions.The method distinguishes similar and distant locations using a DINO-feature similarity threshold.

4 Experiments

Experiments evaluate the method on a 10-scene dataset with long-tail objects, qualitative comparisons, ablations, and reduced-input settings. The method recognizes long-tail classes, produces accurate boundaries, and remains robust with substantially less input.

Dataset: The evaluation uses 10 scenes containing long-tail objects in varied poses and backgrounds, with manually annotated test-view masks.
Baselines: The method is benchmarked against FFD, Semantic-NeRF, and LERF, including variants based on LSeg, ODISE, and OV-Seg.
Results: Qualitative and quantitative comparisons show the method outperforming competing techniques, including approaches relying on extensive segmentation annotations.
Ablations: The RDA and FDA losses are both crucial, because removing either causes severe performance degradation and produces segmentation errors.
Limited input: Halving input views or using one CLIP feature scale causes less than 1% degradation, while using 10% of views reduces mIoU by 9% and Accuracy by 1%.With one scale from 10% of views—one-thirtieth of the full input—degradation remains limited to 10%.
Limited input: Under extreme input reduction, the method still outperforms LERF despite LERF using all input views and feature scales.

5 Limitations

The method has two main limitations: it requires text labels before training and struggles with complex scenes and objects whose shapes or textures do not fit its patch-based assumptions.

New text labels require retraining, unlike LERF, which supports querying arbitrary objects without rerunning optimization.
NeRF’s continuous implicit representation makes accurate open-vocabulary object boundaries difficult to infer.The authors identify object-level discrete representations as a promising direction for future work.
Without segmentation maps during training, the method fails to segment complex indoor scenes with high precision.Its patch-based CLIP distillation favors compact objects with aspect ratios near 1 and struggles with large complex shapes or unobvious textures.

6 Conclusion

The method distills CLIP and DINO into a reconstructed NeRF using Selection Volume, Relevancy-Distribution Alignment, and Feature-Distribution Alignment to achieve weakly supervised open-vocabulary 3D segmentation.

The method distills CLIP and DINO foundation-model knowledge into a reconstructed NeRF in a weakly supervised manner.CLIP supplies open-vocabulary multimodal knowledge, while DINO contributes scene-layout information for object boundaries.
A Selection Volume aligns image-level CLIP features with pixel-level features, while Relevancy-Distribution Alignment addresses CLIP feature ambiguities.
Feature-Distribution Alignment leverages DINO scene-layout information to extract accurate object boundaries for 3D segmentation.
The approach recognizes long-tail classes and produces precise segmentation maps with limited input data, without requiring manual segmentation annotations during distillation.

A.3 Training

Training reconstructs scenes with TensoRF-style settings, then trains segmentation components in staged phases using sampled rays and patches.

Segmentation training runs for 15k iterations, first training the selection volume and CLIP branch, then fine-tuning shared and RGB components.The first phase lasts 5k iterations and the second lasts 10k iterations.
The optimization uses Adam with separate learning rates for volume and MLP branches, followed by lower-rate fine-tuning and learning-rate decay.
Training precomputes multiscale CLIP features, computes DINO features on sampled patches, and uses batches of 4096 rays or eight 256×256 patches.

A.4 Dataset

The dataset contains 10 smartphone-captured scenes with long-tail text labels and manually annotated views reserved for evaluation.

The dataset contains 10 scenes captured with smartphones, with 20–30 images per scene at 4032 × 3024 resolution.
Five views per scene receive manually annotated segmentation maps for ground-truth evaluation.
The text labels include many long-tail classes and may add general background terms such as wall or desktop.
The supplementary evaluations include human body and human head datasets, as shown in Figure 8.

B More Ablations

Ablations show that the Selection Volume and FDA loss contribute to performance, while dissimilar DINO features can impair correlation-loss stability.

Removing the Selection Volume lowers both mIoU and Accuracy compared with the full model.The ablation replaces learned scale selection with simple averaging of multiscale CLIP features.
Dissimilar DINO features can impair correlation-loss stability, although the dissimilar component encourages different segmentation predictions.

C More Evaluations

Additional evaluations test the method across difficult scenes, low-quality indoor imagery, multiple same-class instances, and rephrased text prompts. The method generally segments more objects and labels than LERF, preserves boundaries, and remains reliable under prompt changes, while similar-appearance labels can still be confused.

Evaluation settings: The evaluations cover low-quality indoor images, complex LERF scenes, multiple same-class instances, and rephrased text prompts.The experiments compare against LERF where applicable and use varied linguistic descriptions, including translations, names, and actions.
Indoor scenes with low-quality images: Our method segments more labels than LERF on indoor datasets with unrealistically rendered, limited-resolution images.The images have resolutions of 640 × 480 or 1024 × 768, and similar-appearance labels can still be confused.
Complex scenes: In complex scenes, our method segments more objects with more precise boundaries than LERF despite complex geometry and varied object sizes.LERF cannot segment most objects because of CLIP-feature ambiguities.
Complex scenes: All three apples are assigned to the same class with accurate boundaries in a scene containing multiple instances of one class.FDA preserves similar DINO features across same-class instances, while RDA assigns them the same text label.
Segmentation with different text prompts: Rephrased text prompts still produce reliable segmentation across different languages, names, and actions.Examples include Portuguese translations, alternative object names, and action-based descriptions.

D More Results

The paper provides additional qualitative visualizations across diverse objects and settings. These examples complement the quantitative results reported in Table 6.

Additional visualizations: Additional visualizations cover beds, sofas, lawns, rooms, benches, tables, office desks, blue sofas, snacks, and covered desks.The visualizations are presented in Figures 12–21.
Additional visualizations: The reported examples include both generic and attribute-qualified objects, such as sofas and blue sofas, and desks and covered desks.These visualizations illustrate segmentation across varied object descriptions.
Quantitative results: Quantitative results for the additional experiments are listed in Table 6.The passage identifies Table 6 as the location of the quantitative results accompanying the visualizations.