PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts

Ji Hou, Benjamin Graham, Matthias Nießner, Saining Xie

arXiv:2012.09165v3cs.CV

TL;DR

3D scene understanding is constrained by the cost of collecting scenes and obtaining point-level annotations. The paper introduces Contrastive Scene Contexts, a pre-training method combining point correspondences with spatial context, and studies data-efficient learning across scarce-data benchmarks. On ScanNet, using 0.1% of point labels reaches 89% of full-annotation baseline instance-segmentation performance and 96% of semantic-segmentation performance.

  • Problem

    3D scene understanding needs data, but scanning scenes and acquiring labels such as instance masks are costly and labor-intensive.

  • Method

    The paper proposes unsupervised contrastive pre-training that integrates scene contexts and evaluates fine-tuning and active labeling in limited-reconstruction and limited-annotation settings.

  • Results

    0.1% of ScanNet point labels reaches 89% of full-annotation baseline instance-segmentation performance and 96% of semantic-segmentation performance.

  • Takeaways & Limitations

    The results suggest that exhaustive 3D point-cloud labeling might not be necessary and indicate opportunities for larger-scale 3D data collection and representation learning.

  • Takeaways & Limitations

    A reported comparison uses matched points sampled per scene rather than from the whole mini-batch, although the authors verified that this nuance does not influence the conclusion.

Abstract

from arXiv · show

The rapid progress in 3D scene understanding has come with growing demand for data; however, collecting and annotating 3D scenes (e.g. point clouds) are notoriously hard. For example, the number of scenes (e.g. indoor rooms) that can be accessed and scanned might be limited; even given sufficient data, acquiring 3D labels (e.g. instance masks) requires intensive human labor. In this paper, we explore data-efficient learning for 3D point cloud. As a first step towards this direction, we propose Contrastive Scene Contexts, a 3D pre-training method that makes use of both point-level correspondences and spatial contexts in a scene. Our method achieves state-of-the-art results on a suite of benchmarks where training data or labels are scarce. Our study reveals that exhaustive labelling of 3D point clouds might be unnecessary; and remarkably, on ScanNet, even using 0.1% of point labels, we still achieve 89% (instance segmentation) and 96% (semantic segmentation) of the baseline performance that uses full annotations.

1. Introduction

The paper frames data-efficient 3D scene understanding as learning with limited scenes or supervision, motivated by costly 3D data collection and annotation. It studies benchmark settings, representation learning, and contrastive pre-training to improve performance with scarce data or labels.

  • Motivation: 3D scene collection and annotation remain expensive because scanning real environments is limited and semantic labeling requires labor-intensive pipelines.The bottleneck includes both the number of indoor scenes that can be scanned and the effort required for labels such as bounding boxes and instance masks.
  • Motivation: Data-efficient 3D scene understanding addresses learning with limited data or supervision while seeking better performance and reduced annotation labor.The paper identifies both immediate performance improvement and longer-term reduction of human annotation effort as motivations.
  • Benchmark settings: The proposed benchmark suite evaluates limited scene reconstructions and limited annotations across object detection, semantic segmentation, and instance segmentation.Limited reconstructions constrain the number of scanned scenes, whereas limited annotations constrain the labeled points within each scene.
  • Approach: The paper studies representation learning through pre-training on richer data followed by fine-tuning on smaller 3D target sets.This direction is motivated by the success of representation learning for data-efficient learning in 2D.
  • Approach: The new 3D pre-training method integrates scene contexts into contrastive learning, addressing point-level approaches that disregard spatial configurations.The paper reports that the method transfers to complex tasks such as instance segmentation and outperforms state-of-the-art results.
  • Results: 0.1% of ScanNet point labels recovers 89% of baseline instance-segmentation performance and 96% of baseline semantic-segmentation performance.The baseline uses full annotations, and the result supports the paper’s observation that exhaustive point-cloud labeling might not be necessary.
  • Results: The pre-trained network consistently improves supervised fine-tuning across multiple tasks and datasets, while clustered point features enable an active labeling strategy.These findings cover both limited-reconstruction and limited-annotation settings.

2. Related Work

Prior 3D scene-understanding work spans detection and segmentation, while data-efficient learning uses limited examples or labels. This paper positions representation learning as a promising route for reducing 3D data and annotation demands.

  • 3D Scene Understanding: 3D scene-understanding research has expanded from synthetic single-object classification to large-scale real-world detection, semantic segmentation, and instance segmentation.The literature includes diverse datasets, algorithms, and sparse convolutional networks for point-cloud learning.
  • Data-Efficient Learning: Data-efficient learning concerns training with limited examples or labels, for which representation learning and transfer learning can make downstream prediction less data-hungry.The paper notes successful validation in natural language processing and, to some extent, 2D image classification.

3. Contrastive Scene Contexts for Pre-training

Contrastive Scene Contexts extends point-level contrastive pre-training by organizing negatives according to spatial context around each anchor point. The method uses relative angles and distances to partition scenes, improving transfer to instance segmentation.

  • PointContrast minimizes distances for matched point pairs and separates unmatched points, but does not model spatial configurations and scene contexts.Contrastive Scene Contexts addresses this limitation by integrating spatial context into the objective.
  • Contrastive Scene Contexts partitions each scene into spatial regions and computes contrastive learning losses separately within those regions.The partition losses are aggregated into a final pre-training loss.
  • For each anchor point, other points are assigned to partitions using their relative angles and distances.The partitioning is relative to the anchor point; 2 or 4 partitions use angles, while 8 or more also use distances.
  • Analysis: Using scene context partitions enables contrastive learning to utilize more sampled points for better performance on S3DIS instance segmentation.The analysis varies both the number of partitions and sampled points during pre-training.
  • Results: 2.9% absolute improvement over PointContrast pre-training raises the improvement over the train-from-scratch baseline to 4.1% on instance segmentation.The evaluation reports state-of-the-art results in terms of mAP@0.5 using a simple bottom-up clustering mechanism with voting loss.

4. Data-Efficient 3D Scene Understanding

The paper defines data-efficient 3D scene understanding through limited scene reconstructions and limited annotations, and evaluates these settings across diverse downstream tasks. Its benchmarks model bottlenecks in scanning scenes and labeling points or boxes, while pre-training supports downstream learning.

  • The benchmarks cover two complementary settings: limited scene reconstructions and limited annotations.Limited reconstructions constrain the number of scanned scenes, whereas limited annotations constrain labeling budgets within scenes.
  • Limited-annotation evaluation includes semantic segmentation, instance segmentation, and object detection.For segmentation, the annotation budget is measured by labeled points; for detection, it is measured by labeled bounding boxes.
  • Pre-training produces a U-Net feature extractor and associated weights for fine-tuning or optional active selection of points to annotate.The baselines train supervised models from scratch.
  • Limited Annotations (LA): The limited-annotation ScanNet benchmark evaluates configurations with 20, 50, 100, or 200 labeled points per scene.The point-label budget is intended to represent a practical constraint on annotating full rooms.
  • Limited Reconstructions (LR): The limited-reconstruction benchmark simulates scarce scanning by randomly sampling subsets of the ScanNetV2 training set.This setting reflects situations where entering many homes or obtaining detailed scans is unrealistic.

5. Experimental Results

Experiments evaluate data-efficient 3D scene understanding under limited annotations and limited scene reconstructions. Contrastive Scene Contexts improves performance across segmentation and detection tasks, with especially strong results when labels or scenes are scarce.

  • Experimental setup: The evaluation covers ScanNet-LA and ScanNet-LR, measuring semantic segmentation, instance segmentation, and object detection against training-from-scratch baselines.Limited annotations include point labels and bounding boxes; limited reconstructions vary the number of training scenes.
  • Limited annotations: More than 10% absolute improvement in mAP@0.5 and mIoU is obtained with active labeling alone using only 20 point labels per scene.The active strategy selects points using the pre-trained model and outperforms random and raw-input k-means sampling.
  • Limited annotations: Joint active labeling and fine-tuning produces a further, smaller performance boost over either strategy alone.The reported sequence is to actively sample points, then fine-tune with the pre-trained weights.
  • Limited annotations: 50.4% mAP@0.5 for instance segmentation and 69.0% mIoU for semantic segmentation are achieved using around 0.1% of point labels.These results recover 89% and 96%, respectively, of baseline performance using 100% of annotations.
  • Limited scene reconstructions: 50.6% mAP@0.5 and 64.6% mIoU are achieved using only 20% of the total scenes in the limited-reconstruction setting.Fine-tuning with pre-trained weights maintains a clear gap over training from scratch; object detection also improves, including by more than 16% mAP@0.5 with 20% training data.
  • Comparison to PointContrast: Contrastive Scene Contexts outperforms PointContrast in almost every benchmark setting, including both limited annotations and limited reconstructions.On S3DIS, it reaches 72.2% mIoU and exceeds the supervised pre-training result reported for PointContrast.

6. Conclusion

The paper presents Contrastive Scene Contexts as an unsupervised pre-training method for data-efficient 3D scene understanding. Its results show that extremely few data or annotations can support competitive performance and may enable broader 3D data collection and learning.

  • Conclusion: Contrastive Scene Contexts integrates scene contexts into a contrastive learning framework for data-efficient 3D scene understanding.The method is described as a novel unsupervised pre-training algorithm.
  • Conclusion: Competitive performance is achieved with extremely few data or annotations by leveraging representation learning.The conclusion frames this as a possibility demonstrated by the reported results.
  • Conclusion: The findings may open opportunities in interactive 3D data collection, unsupervised 3D representation learning, and large-scale 3D scene understanding.These are presented as potential directions rather than established outcomes.

A. Implementation Details

The implementation uses corresponding partial frames for contrastive pre-training, defines PointInfoNCE over matched point features, and applies feature-based clustering for active labeling and instance grouping.

  • Data preprocessing: Partial frames are subsampled every 25 frames and paired by computing world-coordinate overlaps under a 2.5cm threshold.Points with matching transformed coordinates are stored as correspondences.
  • PointInfoNCE loss: PointInfoNCE uses matched point pairs as query-positive examples and features from neighboring matched points as negatives.Non-matched points are discarded, and a subset of matched pairs is sampled during training.
  • Active labeling: Active labeling clusters pre-trained 96-dimensional voxel features and point coordinates to select k representative points for annotation.Voxel features are back-projected to the raw point cloud using nearest-neighbor search before clustering.
  • Instance segmentation: Instance segmentation groups neighboring points shifted by learned directional and distance vectors when they share a semantic label.The implementation uses a 3cm neighborhood threshold and performs clustering only at test time.

B. More Visualizations

Additional visualizations show semantic segmentation and object detection predictions produced with extremely scarce training annotations.

  • More visualizations: The supplemental visualizations include ScanNet semantic segmentation trained with 20 labeled points per scene and object detection trained with one bounding-box annotation per scene.The predictions are shown on the ScanNet validation set.

C. Per-Category Results

The paper supplements its data-efficient benchmarks with per-category results across instance segmentation, semantic segmentation, and object detection tasks on multiple datasets.

  • Per-category instance segmentation results are reported on ScanNet-LA using 200 labelled points for training.The results appear in Table 7.
  • Per-category semantic segmentation results are reported on ScanNet-LA.The results appear in Table 8.
  • Per-category object detection results are reported for the Limited Bounding Boxes Annotations setting.The results are identified as appearing in Table ??.
  • Supplementary full-data results cover instance and semantic segmentation on S3DIS, object detection on SUN-RGBD, and instance segmentation and object detection on ScanNet.These results are reported in Tables 9–13.

D. Different Backbones.

Experiments examine backbone generality and provide task-specific results across semantic segmentation, instance segmentation, and object detection benchmarks, including limited-annotation settings.

  • Different Backbones: The method uses SR-UNet-34 as its backbone and also evaluates SR-UNet-18A and PointNet++ to test backbone agnosticism.The authors state that the performance gap with 3D-MPA comes from additional head modules rather than the pre-training method.
  • Different Backbones: With 20 labelled points per scene, the pre-trained model generates high-quality ScanNet-LA semantic segmentation predictions under active labeling.This result is shown in Figure 8.
  • Different Backbones: ScanNet-LA instance segmentation is evaluated with mAP@0.5 over 18 classes using 200 labelled points per scene for training.These are the metrics and setting for Table 7.
  • Different Backbones: ScanNet-LA semantic segmentation is evaluated with mean IoU over 20 classes using 200 labelled points per scene for training.These are the metrics and setting for Table 8.
  • Different Backbones: ScanNetV2 validation results include instance segmentation and object detection over 18 classes.Both tasks use IoU-thresholded or category-level metrics specified in Tables 12 and 13.
  • Different Backbones: Pre-training with different backbones is evaluated using 100% of available training data.The authors expect larger deltas with smaller training sets.

E. ScanNet Benchmark

The paper reports validation and test-set comparisons on ScanNet, including direct comparison with PointContrast and performance exceeding 3D-MPA without its special proposal module.

  • Validation results directly compare the proposed method with PointContrast.Both methods evaluate on the validation set.
  • The proposed method significantly outperforms 3D-MPA on the ScanNet benchmark test set.The comparison does not use 3D-MPA’s special proposal module.
Loading 2012.09165v3…