PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Pretraining of 3D Features on any Point-Cloud

Zaiwei Zhang, Rohit Girdhar, Armand Joulin, Ishan Misra

arXiv:2101.02691v1cs.CV

TL;DR

3D recognition lacks large annotated datasets because 3D acquisition and labeling are difficult. The paper introduces DepthContrast, a self-supervised method for diverse unregistered 3D data and multiple representations, and reports state-of-the-art performance across nine benchmarks with improved label efficiency.

  • Problem

    3D recognition lacks large annotated datasets because 3D data is difficult to acquire and time-consuming to label.

  • Method

    DepthContrast uses self-supervised instance discrimination on single-view depth maps and jointly pretrains point-cloud and voxel representations without registered point clouds or correspondences.

  • Results

    DepthContrast improves performance across nine downstream tasks, achieves state-of-the-art results on detection and segmentation benchmarks, and outperforms training from scratch on four detection datasets.

  • Takeaways & Limitations

    Single-view depth scans can learn powerful 3D feature representations, and the pretrained models are label efficient, including for detection settings with few labeled samples.

  • Takeaways & Limitations

    The method treats points in a depth map as a set and does not rely on any specific ordering of the points.

Abstract

from arXiv · show

Pretraining on large labeled datasets is a prerequisite to achieve good performance in many computer vision tasks like 2D object recognition, video classification etc. However, pretraining is not widely used for 3D recognition tasks where state-of-the-art methods train models from scratch. A primary reason is the lack of large annotated datasets because 3D data is both difficult to acquire and time consuming to label. We present a simple self-supervised pertaining method that can work with any 3D data - single or multiview, indoor or outdoor, acquired by varied sensors, without 3D registration. We pretrain standard point cloud and voxel based model architectures, and show that joint pretraining further improves performance. We evaluate our models on 9 benchmarks for object detection, semantic segmentation, and object classification, where they achieve state-of-the-art results and can outperform supervised pretraining. We set a new state-of-the-art for object detection on ScanNet (69.0% mAP) and SUNRGBD (63.5% mAP). Our pretrained models are label efficient and improve performance for classes with few examples.

1. Introduction

3D recognition lacks large annotated datasets because acquisition, registration, and labeling are difficult. DepthContrast addresses this gap with self-supervised learning across diverse 3D inputs and representations, improving downstream performance and label efficiency.

  • Motivation: 3D annotations are difficult and time-consuming because scene reconstruction requires registration, while segmentation and detection require extensive point-level or group-level labeling.A single ScanNet scene takes around 22 minutes to annotate.
  • Motivation: Single-view depth maps provide abundant raw 3D data, but pretraining must handle absent supervision, missing alignments, and differing point-cloud or voxel representations.These constraints exclude methods relying on multiview correspondences or a single representation.
  • Approach: DepthContrast applies instance discrimination to depth maps, avoiding registered point clouds and correspondences by treating each depth map as an instance.The method can jointly learn features for different 3D representations by processing them as augmentations.
  • Approach: Joint training of point and voxel representations is important for learning good representations, while naive contrastive learning may not suffice.The paper explicitly identifies joint training across input formats as a key design insight.
  • Scope: DepthContrast applies across model architectures, indoor and outdoor data, and single- and multiview 3D data, including high-capacity architectures that otherwise overfit.The contribution is designed for broad 3D input and architecture settings rather than one fixed data format.
  • Results: The method improves nine downstream tasks, achieves new state-of-the-art results on ScanNet and SUNRGBD detection, and produces efficient few-shot learners.The introduction reports improvements across nine tasks and state-of-the-art detection on two benchmarks.

2. Related Work

Prior self-supervised learning methods largely focus on images or single 3D objects, while scene-level 3D methods can require registered point clouds and cross-view correspondences. DepthContrast extends contrastive instance discrimination to multiple 3D input formats without those registration requirements.

  • Self-supervised learning: Self-supervised learning has used clustering, GANs, pretext tasks, and contrastive instance discrimination, with recent methods showing viability for 2D recognition.DepthContrast builds on contrastive learning and extends instance discrimination to 3D input formats.
  • Self-supervised learning: Multimodal contrastive methods align representations across modalities, and DepthContrast follows this direction for multiple 3D input formats using a momentum encoder.The paper positions its method as an extension of prior instance discrimination work to 3D formats.
  • Self-supervised learning for 3D: A prior scene-level 3D method requires complete scene reconstruction and point-wise correspondences from registered depth maps, limiting it to registered static scenes.DepthContrast is contrasted with this registration-dependent setting.

3. Approach

DepthContrast pretrains 3D representations from unprocessed depth maps using instance discrimination across augmented views and input formats. It jointly trains point-cloud and voxel encoders so the learned features can transfer across architectures and downstream settings.

  • Instance Discrimination: DepthContrast uses unprocessed single-view or multi-view depth maps and does not require 3D registration or camera calibration.The method is designed for varied 3D data rather than registered scene reconstructions.
  • Instance Discrimination: Each depth map is augmented, encoded into unit-norm global features, and trained with instance discrimination so matching augmentations are similar while other scenes are dissimilar.Features are obtained by pooling over 3D spatial coordinates; a momentum encoder and queue provide many negatives.
  • Multiple 3D Input Formats: DepthContrast jointly learns point-cloud and voxel representations by applying a single objective within and across input formats.Across-format training aligns features produced by format-specific encoders with different architectures.
  • Model Architecture: The framework uses point coordinates for point inputs and voxel occupancy grids with RGB values for voxel inputs derived from the same augmented 3D data.The encoders differ in input representation while operating on the same underlying augmented sample.
  • Data Augmentation for 3D: Random point sampling, flips, rotations, cuboid cropping, and cuboid dropping provide augmentations that encourage local geometric features.Cuboid cropping uses random scales and aspect ratios, while dropping erases cuboids from the scene.
  • Training Details: The method uses 130K negatives, a momentum value of 0.9, 128-dimensional features, and temperature 0.1 for contrastive pretraining.A two-layer projection MLP and L2 normalization are applied after the network encoder.

4. Experiments

DepthContrast is evaluated across diverse 3D tasks and datasets, where it improves transfer, label efficiency, scaling behavior, and detection performance. Joint point-voxel pretraining further strengthens representations, while naive within-format discrimination can be insufficient.

  • DepthContrast is evaluated across classification, semantic segmentation, and object detection using diverse 3D datasets, sensors, and settings.
  • Pretraining with Point Input Format: DepthContrast outperforms scratch training on all four detection datasets and improves S3DIS performance by 12.1% mAP with only 200 labeled training samples.
  • Using More Pretraining Data: Combining Redwood-vid and ScanNet-vid triples pretraining data, and performance improves with both model capacity and sample count across all four detection datasets.
  • State-of-the-art Detection Frameworks: DepthContrast pretraining achieves state-of-the-art detection performance on SUNRGBD and ScanNet, with larger gains at mAP evaluated at IoU=0.5.
  • Label Efficiency of Pretrained Models: With 20% finetuning data, pretrained models gain over 10% mAP, and with 50% data they match scratch training using the full dataset.
  • Pretraining with Multiple Input Formats: Joint point-voxel training improves both input formats, while the combined within- and across-format loss gives the best transfer performance and raises voxel performance 4% over within-format training.

5. Analysis

The analysis examines augmentation, view structure, and outdoor LiDAR transfer. DepthContrast benefits from task-specific augmentations and remains effective across single-view, multi-view, indoor, and outdoor data.

  • Data Augmentation: The proposed augmentations improve ModelNet classification accuracy by 5% and SUNRGBD detection by 2% mAP over the compared supervised-learning augmentations.
  • Single-view or Multi-view 3D Data: ScanNet-vid, ScanNet, and Redwood-vid pretraining produce similar transfer performance, indicating robustness to single-view, multi-view, and different data distributions.
  • Generalization to Outdoor LiDAR Data: Using Waymo single-view scans for pretraining outperforms scratch training on KITTI cyclist detection for both point and voxel inputs, especially with fewer labeled samples.
  • Generalization to Outdoor LiDAR Data: With Spconv-UNet and 5% labeled data, outdoor LiDAR pretraining achieves a 20% gain, supporting transfer across input formats, sensors, and scene types.

6. Conclusion

DepthContrast is presented as an easy-to-implement self-supervised method spanning architectures, input formats, and 3D settings, with state-of-the-art detection and segmentation performance.

  • 6. Conclusion: DepthContrast works across model architectures, input data formats, indoor and outdoor 3D, and single- or multi-view 3D data.It is designed to leverage large-scale 3D data without requiring multi-view information.
  • 6. Conclusion: The method achieves state-of-the-art performance on detection and segmentation benchmarks and outperforms prior work on detection.The conclusion attributes its performance to joint training across multiple input formats and a generalizable augmentation scheme.

Supplemental Material

The supplemental material provides additional architecture, training, fine-tuning, and evaluation details for the models and experiments.

  • Supplemental Material: Appendix A provides model architecture details.The appendix covers the architectures used in the paper's experiments.
  • Supplemental Material: Appendix B reports PointNet++ and UNet training and fine-tuning hyperparameters, along with additional results.These details include experiments involving the PointNet++ and UNet models.
  • Supplemental Material: Appendix C reports PointnetMSG and Spconv-UNet hyperparameters and additional KITTI detection results.The appendix includes results for other KITTI detection categories.

A. Architecture Details

The supplemental architecture details describe PointNet++ and PointnetMSG point-cloud backbones alongside UNet and Spconv-UNet voxel or sparse-convolution architectures used across evaluations.

  • A. Architecture Details: PointNet++ uses four set abstraction layers and two feature up-sampling layers.Set abstraction layers specify output points, receptive-field radius, and MLP channel sizes; up-sampling interpolates input-point features.
  • A. Architecture Details: PointNet++ 2×, 3×, and 4× increase each set-abstraction MLP feature size by factors of 2, 3, and 4.The remaining model is adjusted when PointNet++ is used in VoteNet and H3DNet.
  • A. Architecture Details: PointnetMSG processes LiDAR point clouds with four multi-scale set abstraction layers and four feature up-sampling layers.Each abstraction layer uses two scales, each with a ball-region radius and an MLP.
  • A. Architecture Details: UNet contains four encoding and four decoding residual blocks using sparse convolution or deconvolution operations.Encoding and decoding begin with stride-2 operations, followed by stride-1 sparse convolution layers.
  • A. Architecture Details: Spconv-UNet contains four sparse encoding blocks and four sparse decoding upblocks for the Part A2 detection model.The architecture uses sparse convolution and sparse residual blocks.
  • A. Architecture Details: Pretraining uses SGD with momentum 0.9 for 1000 epochs, although 400 epochs already produce good results.The cosine learning rate decreases from 0.12 to 0.00012, and the longer schedule only slightly improves the model.

B.2. Experimental Details for PointNet++

The supplemental experiments specify fine-tuning protocols for PointNet++-based evaluations and show label-efficient detection improvements over training from scratch across the reported benchmarks.

  • B.2. Experimental Details for PointNet++: VoteNet fine-tuning uses Adam with a 0.001 learning rate, scheduled reductions at 80, 120, and 160 epochs, and 180 total epochs.S3DIS uses 360 epochs because its training set contains only 200 instances; batch sizes are 8 or 16 depending on the dataset.
  • B.2. Experimental Details for PointNet++: H3DNet fine-tuning loads only the pretrained PointNet++ backbone and uses one backbone network instead of the original four.A 3× PointNet++ backbone reproduces the prior reported results under this setup.
  • B.2. Experimental Details for PointNet++: ModelNet transfer learning evaluates linear classifiers on fixed pretrained features and also evaluates full fine-tuning.The full-fine-tuning setup extracts per-point features, max-pools them, and applies one linear classification layer.
  • B.2. Experimental Details for PointNet++: DepthContrast pretraining is label efficient on Matterport3D and S3DIS detection tasks.The PointNet++ model is pretrained on ScanNet-vid for these evaluations.
  • B.2. Experimental Details for PointNet++: DepthContrast results are consistently better than training from scratch across all detection benchmarks used in the paper.The supplemental material extends the label-efficiency evaluation beyond ScanNet and SUNRGBD to Matterport3D and S3DIS.

B.3. Experimental Details for UNet

The UNet evaluation tests DepthContrast pretraining for scene segmentation across ScanNet, S3DIS, and Synthia, with reported implementation settings and consistent gains as training data varies.

  • B.3. Experimental Details for UNet: The ScanNet evaluation uses 4cm voxels instead of the default 2cm because of memory constraints, changing scratch-training results relative to prior work.The authors still report improvements when pretraining and fine-tuning use the same dataset.
  • B.3. Experimental Details for UNet: DepthContrast-pretrained UNet models are fine-tuned for scene segmentation on S3DIS, ScanNet, and Synthia.The evaluation uses 8 V100 GPUs and trains for 20,000 iterations across the three tasks.
  • B.3. Experimental Details for UNet: DepthContrast provides a consistent performance boost across different percentages of training data used for S3DIS scene segmentation.The pretrained UNet is trained on ScanNet-vid and evaluated through a data-efficiency plot on S3DIS.

C. Experimental Details for Lidar Data

The LiDAR experiments evaluate DepthContrast pretraining for PointRCNN and Part A2 under varying labeled-data fractions, with KITTI label-efficiency figures covering car and pedestrian detection and joint-loss comparisons.

  • C. Experimental Details for Lidar Data: The LiDAR experiments use PointRCNN and Part A2 configurations to compare training from scratch with fine-tuning on labeled-data splits.The evaluated fractions are 5%, 10%, 20%, 50%, and 100%; full-data training required parameter changes because of overfitting in cyclist and pedestrian classes.
  • C. Experimental Details for Lidar Data: The UNet model is pretrained on ScanNet-vid with DepthContrast for the S3DIS scene-segmentation label-efficiency evaluation.This voxel-input evaluation uses the UNet architecture.
  • C. Experimental Details for Lidar Data: Figures 10 and 11 evaluate KITTI car and pedestrian detection label efficiency at moderate difficulty on the validation split.Figure 12 compares within-format and joint training losses for pedestrian detection at moderate difficulty.
  • C. Experimental Details for Lidar Data: The LiDAR training uses AdamW, model-specific learning rates, batch sizes, 80 epochs, and 10× learning-rate drops at 35 and 45 epochs.PointRCNN uses batch size 24 and Part A2 uses batch size 16, with the data augmentation and processing pipelines from the cited configuration.

C.2. Results on KITTI

The KITTI experiments extend label-efficiency results beyond the cyclist class to car and pedestrian detection, showing stronger transfer with fewer labeled training instances and joint loss training.

  • C.2. Results on KITTI: The KITTI results include car and pedestrian detection in addition to the cyclist results shown in the main paper.The appendix presents results for the remainder classes.
  • C.2. Results on KITTI: DepthContrast improves KITTI car detection with fewer training instances, especially when using 5% of labeled data.The reported evaluation uses the validation split at moderate difficulty; other difficulty levels show a similar pattern.
  • C.2. Results on KITTI: Pretraining gives PointRCNN consistent gains over scratch for KITTI pedestrian detection and gives Part A2 a significant boost with fewer training instances.The comparison is reported at moderate difficulty on the KITTI validation split.
  • C.2. Results on KITTI: Joint loss training provides more performance gain with fewer training instances for KITTI pedestrian detection.The result supports the benefit of joint training across different pretraining data and architectures.
Loading 2101.02691v1…