PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-Grounded Indoor 3D Semantic Segmentation in the Wild

David Rozenberszki, Or Litany, Angela Dai

arXiv:2204.07761v2cs.CV

TL;DR

Existing 3D semantic segmentation benchmarks use limited vocabularies that do not capture real-scene diversity. The paper introduces ScanNet200 and language-grounded pre-training that aligns 3D features with CLIP text embeddings, yielding consistent improvements on large-vocabulary and limited-annotation settings. The authors report that this approach supports more robust 3D feature learning under imbalanced and limited data.

  • Problem

    Existing 3D semantic segmentation benchmarks contain limited class vocabularies, while real scenes include substantially more diverse categories and naturally imbalanced observations.

  • Method

    The paper introduces a 200-class ScanNet benchmark and language-grounded pre-training that maps learned 3D features to pre-trained CLIP text embeddings with a contrastive loss.

  • Results

    The approach consistently improves over strong 3D pre-training baselines on ScanNet200 and limited-annotation scenarios.

  • Takeaways & Limitations

    Anchoring 3D features to richly structured text embeddings provides a basis for more robust 3D semantic scene understanding under imbalanced and limited data.

  • Takeaways & Limitations

    Infrequently observed objects are often small and geometrically low-resolution, leaving substantial room to improve tail-category recognition; color images may provide higher-resolution signals.

Abstract

from arXiv · show

Recent advances in 3D semantic segmentation with deep neural networks have shown remarkable success, with rapid performance increase on available datasets. However, current 3D semantic segmentation benchmarks contain only a small number of categories -- less than 30 for ScanNet and SemanticKITTI, for instance, which are not enough to reflect the diversity of real environments (e.g., semantic image understanding covers hundreds to thousands of classes). Thus, we propose to study a larger vocabulary for 3D semantic segmentation with a new extended benchmark on ScanNet data with 200 class categories, an order of magnitude more than previously studied. This large number of class categories also induces a large natural class imbalance, both of which are challenging for existing 3D semantic segmentation methods. To learn more robust 3D features in this context, we propose a language-driven pre-training method to encourage learned 3D features that might have limited training examples to lie close to their pre-trained text embeddings. Extensive experiments show that our approach consistently outperforms state-of-the-art 3D pre-training for 3D semantic segmentation on our proposed benchmark (+9% relative mIoU), including limited-data scenarios with +25% relative mIoU using only 5% annotations.

1 Introduction

The paper introduces ScanNet200, a 200-class indoor 3D semantic segmentation benchmark designed to reflect real-scene diversity and natural class imbalance. It also proposes language-grounded pre-training that aligns 3D features with CLIP text embeddings for robust learning under imbalanced and limited observations.

  • Existing 3D benchmarks such as ScanNet and SemanticKITTI evaluate only 20 and 28 classes, respectively, limiting their representation of real-scene diversity.
  • ScanNet200 extends ScanNet to 200 classes, adding finer-grained categories and previously unaddressed objects while preserving naturally imbalanced class frequencies.Walls and floors occur more often than nightstands, which occur more often than fire extinguishers.
  • The benchmark evaluates both fully dense annotations and limited annotation scenarios with sparse labels per training scene.The limited-annotation setting reflects the expense of 3D data annotation.
  • Language-grounded pre-training maps learned 3D features to pre-trained CLIP text embeddings using a contrastive loss.The goal is to construct a richly structured 3D feature space from category embeddings.
  • The proposed representation learning approach is intended to improve robustness when 3D observations are imbalanced or limited.

2 Related Work

Prior work developed point-based, contrastive, data-efficient, text-based, and imbalance-mitigation methods for 3D understanding. This paper differs by using supervised multimodal 3D representation learning guided by text features for a substantially larger vocabulary.

  • 3D Semantic Segmentation: Point-based methods improved memory efficiency for 3D semantic segmentation by operating directly on unstructured point clouds.
  • 3D Representation Learning: Unsupervised contrastive pre-training methods explored augmentation and limited-data strategies for 3D scene understanding.
  • 3D Representation Learning: The proposed approach uses supervised multimodal 3D representation learning guided by text-encoded features to cover more class categories than prior 3D pre-training methods.
  • 3D Representation Learning: Prior 3D methods also used text embeddings for zero-shot point-cloud segmentation and classification, while this work uses text guidance for representation learning.
  • 3D Scene Understanding Benchmarks: ScanNet provides large-scale reconstructed RGB-D scans, but its benchmark evaluates only 20 classes despite hundreds of raw annotated labels.
  • Class Imbalance: Image-understanding research commonly addresses class imbalance through oversampling, undersampling, frequency-based sampling, augmentation, and loss re-balancing.
  • Language-Grounded 3D Feature Learning: Figure 2 depicts language-guided pre-training that maps 3D features to text anchors using a contrastive objective.

3 Method

The method combines language-grounded 3D pre-training with class-balancing strategies for ScanNet200's long-tailed 200-class segmentation setting.

  • Language-Grounded 3D Feature Learning: CLIP text embeddings anchor learned 3D features so geometrically represented classes lie near their corresponding semantic text encodings.A contrastive objective aligns matching 3D and text features while separating non-matching categories.
  • Language-Grounded 3D Feature Learning: A sparse 3D convolutional U-Net processes sparse voxelized RGB scans and produces features matched to the CLIP text representation dimension.The implementation uses a MinkowskiEngine-based MinkUNet34 with feature dimension D = 512.
  • Language-Grounded 3D Feature Learning: The pre-training objective combines positive alignment and sampled negative separation, weighted by λ, using cosine-based feature comparisons.Negative sampling encourages cluster separation across categories, including classes that rarely co-occur in scenes.
  • Class Re-balancing: Instance-based augmentation inserts rare-category objects into physically valid scene locations to reduce context overfitting and rebalance rare classes.Candidate placements that collide with existing objects are discarded, and accepted placements update the scene height map.
  • Class Re-balancing: Fine-tuning addresses residual long-tail imbalance with a class-balanced focal loss that emphasizes difficult-to-classify examples.The authors report improved performance over standard focal loss and direct category-balanced cross entropy when category imbalance is explicitly considered.

4 ScanNet200 Benchmark

ScanNet200 expands indoor 3D semantic segmentation from 20 to 200 classes and evaluates performance under natural imbalance and limited annotations.

  • Benchmark construction: 200 classes form ScanNet200, an order of magnitude more than the original ScanNet benchmark.The benchmark follows ScanNet’s original train/validation/test split.
  • Benchmark construction: 550 unique classes were reduced from 607 raw labels, and the 200 most represented categories were selected.This selection enforces at least 10 samples for every category.
  • Class imbalance: The 200 categories are divided into 66 head, 68 common, and 66 tail classes by labeled surface-point frequency.Reporting these splits helps characterize performance across naturally imbalanced categories.
  • Evaluation settings: Limited-annotation experiments provide 5%, 10%, and 50% of annotated surface points while retaining complete scene geometry.The setting emulates annotations beginning with one randomly sampled point per object and adding points by farthest-point sampling.
  • Evaluation settings: ScanNet200 also supports 3D instance-segmentation evaluation using mAP at IoU 25%, 50%, and averaged from 50% to 95%.The average uses 5% overlap increments.
  • Evaluation metrics: Semantic segmentation is primarily evaluated with category-level mIoU, alongside precision and recall across global and frequency-based class splits.mIoU is defined as tp/(tp+fp+fn).

5 Experiments

Experiments compare language-grounded pre-training and balancing techniques against established baselines on ScanNet200, including limited-annotation and instance-segmentation settings. Results show structured representations and consistent gains, while tail categories remain constrained by limited geometric resolution.

  • Experimental setup: The method and baselines use the same 80M-parameter sparse 3D U-Net backbone implemented with MinkowskiNet.This controls the backbone across comparisons.
  • Semantic segmentation: Over 5% mIoU improvement over training from scratch is achieved, more than double the improvement of CSC.The comparison covers frequent and infrequently seen categories.
  • Limited annotation: +8 mIoU is obtained on tail categories over CSC with 5% of annotations available.The approach consistently improves and recovers training-from-scratch performance in this low-annotation regime.
  • Qualitative analysis: Qualitative results show more consistent and accurate predictions for less frequently seen categories such as dish rack and telephone.Comparisons include training from scratch, class-balanced focal loss, and CSC.
  • Balancing methods: Class-balanced focal loss improves over standard cross entropy, original focal loss, and class-balanced cross entropy.The improvement is also consistent with a smaller 3D backbone, particularly for tail categories.
  • Balancing methods: Instance sampling consistently provides a small improvement across common and rare class categories.Its effect is evaluated during training as a data-balancing component.
  • Language-grounded pre-training: Language-grounded pre-training alone improves over all baselines, while adding focal loss and instance sampling produces consistent further gains.The learned representation space is more structured when anchored to text embeddings.
  • Instance segmentation: Language-grounded pre-training improves 3D instance segmentation over training from scratch and CSC.The same pre-trained backbones are fine-tuned for the downstream task.

6 Conclusion

The paper presents ScanNet200 and language-grounded pre-training for 3D semantic feature learning under imbalanced and limited data. The approach shows consistent improvements on ScanNet200 and limited-annotation settings, motivating further multimodal 3D perception research.

  • Conclusion: ScanNet200 extends 3D semantic segmentation to an order of magnitude more classes, while language-grounded pre-training addresses imbalanced and limited data.The method anchors learned features to richly structured CLIP text embeddings.
  • Conclusion: The approach demonstrates consistent improvements over strong baselines on ScanNet200 and under limited annotation scenarios.The authors frame this as a step toward 3D semantic scene understanding in the wild.

Appendix

The appendix adds downstream instance-segmentation results, further ScanNet200 analyses, ablations, and per-class IoU breakdowns.

  • Supplementary analyses: Additional results evaluate 3D instance segmentation and the extended ScanNet200 benchmark.These analyses appear in the supplemental material.
  • Supplementary analyses: Ablations examine the pretrained language model and distance metric used in the point-to-language contrastive objective.The appendix also reports per-class IoU scores.

7 Additional Ablations

Additional ablations show that the language-grounded approach remains effective with smaller backbones, point-based baselines, and alternative pretraining choices. CLIP text embeddings and cosine distance perform especially well.

  • Generalization across backbone sizes: The approach consistently improves performance over state of the art with a smaller 20M-parameter 3D backbone.The evaluation uses a 20M-parameter 3D backbone and a 96-dimensional PCA projection of CLIP text embeddings.
  • Comparison with point-based baselines: Language-guided pretraining outperforms RandLA-Net and SCF-Net on the ScanNet200 large-vocabulary task.The comparison uses point-based models processing complete ScanNet scenes.
  • Effect of contrastive distance metric: Cosine distance produces the best performance, while ℓ1 and ℓ2 distances are more difficult to optimize for text-feature alignment.The ablation compares cosine, ℓ1, and ℓ2 distance metrics.
  • Effect of the pre-trained language model: CLIP produces the best results among the evaluated language models because its multimodal training yields rich embedding structure.The alternatives include BERT and GPT2, which are trained on large text corpora rather than image-text data.

8 Implementation Details

The implementation combines language-grounded pretraining with fine-tuning, tail-class instance sampling, and augmentation. The method also examines lighting effects and the contribution of voxel color information.

  • Training parameters: Pretraining uses momentum SGD for 400 epochs with batch size 8 and fixed sampling and loss parameters.The learning rate starts at 0.05, decays at epochs 150 and 250, and uses momentum 0.9.
  • Training parameters: Fine-tuning uses the same optimizer and batch size for 170 epochs after pretraining.The initial learning rate is 0.05 and is decayed during training.
  • Instance Sampling: Instance sampling targets the 66 least-frequent ScanNet200 classes during fine-tuning.The sampling probability is computed from inverse log frequencies of the training-set histogram.
  • Instance Sampling: Instance-based augmentation places sampled objects on scene geometry with random orientations and combines this process with color and geometry augmentation.Object centers are sampled in the scene, with height determined by the maximum scene geometry at each location.
  • Instance Sampling: The method does not explicitly correct lighting differences during instance augmentation, while removing voxel color changes final performance by only 1% mIoU.The authors hypothesize that lighting inconsistencies have limited-to-negligible effects at 2cm resolution.

9 3D Instance Segmentation Results

The paper evaluates qualitative 3D semantic instance segmentation on ScanNet scenes against training from scratch and CSC.

  • 3D Instance Segmentation Results: Figure 8 visualizes downstream 3D instance segmentation results compared with training from scratch and CSC.The passage identifies the qualitative comparison but does not report a numerical outcome.

10 Breakdown of Class IoU scores

The paper reports per-category IoU scores and qualitative instance-segmentation visualizations for the ScanNet200 benchmark, including comparisons with other approaches.

  • Breakdown of Class IoU scores: Table 6 reports per-category IoU scores for 3D semantic segmentation on ScanNet200.The table compares the proposed method with other state-of-the-art approaches.
  • Breakdown of Class IoU scores: Figure 8 provides qualitative 3D semantic instance segmentation results on ScanNet scenes.The visualizations compare the proposed approach with training from scratch and CSC.
Loading 2204.07761v2…