Self-Supervised Pretraining of 3D Features on any Point-Cloud

1. Introduction

• 3D pretraining is constrained by costly scene reconstruction and annotation, scarce labeled datasets, absent large-scale self-supervision, unregistered views, and differing point-cloud and voxel representations.Annotating a ScanNet scene takes around 22 minutes [18].Source: §1 · p. 1
• DepthContrast treats each depth map as an instance for contrastive learning, avoiding registered point clouds or correspondences, even for views from the same scene.It adapts Instance Discrimination by Wu et al. to depth maps.Source: §1 · p. 2
• Single-view 3D depth scans can support powerful self-supervised feature learning despite the scarcity of annotated 3D data.Consumer-grade sensors make large quantities of raw single-view depth maps increasingly accessible.Source: §1 · pp. 1–2
• Jointly training point and voxel representations as augmentations is important for good features; naively applying contrastive learning may not work well.Each representation is processed by its associated network [97].Source: §1 · p. 2
• The method applies across architectures, indoor and outdoor data, and single- and multiview inputs, and enables pretraining high-capacity models that otherwise overfit detection and segmentation.Source: §1 · p. 2
• The method improves performance across nine downstream tasks, sets new state-of-the-art results on ScanNet and SUNRGBD detection, and acts as an efficient few-shot learner.The introduction does not provide the detection scores or quantify few-shot efficiency.Source: §1 · p. 2

Self-supervised learning for images

• Building on contrastive instance discrimination [64], the method extends Wu et al. to multiple 3D input formats following Tian et al. [97], using a momentum encoder [36] rather than a memory bank.Recent advances have also made self-supervised pretraining a viable alternative to supervised pretraining for 2D recognition [9] [13] [30] [36] [37] [49] [59] [98].Source: §2 · p. 2

Self-supervised learning for 3D data

• Unlike Xie et al.’s scene method, which requires registered static scenes and point-wise cross-view correspondences, this approach learns from single-view 3D data and also supports multiview data.Many prior 3D self-supervised methods focus on single-object representations for reconstruction, classification, or part segmentation [1] [2] [25] [35] [44] [77] [103].Source: §2 · p. 2

Representations of 3D scenes

• Because point clouds are convenient sensor inputs but difficult for direct convolution, the work jointly pretrains PointNet++ [71] on points and a sparse-convolution U-Net [17] on voxels.Point clouds and voxels are common 3D representations, alongside meshes and efforts to combine multiple representations [79] [100] [118].Source: §2 · p. 2

3D transfer tasks and datasets

• Transfer evaluation covers ModelNet shape classification, indoor scene segmentation and detection, and outdoor detection across synthetic-object, indoor-scene, and self-driving datasets.The paper uses ModelNet for shape classification and evaluates indoor detection [67] [68] [118], scene segmentation [17] [71] [95] [105], and outdoor detection [15] [50] [79] [80] [81].Source: §2 · p. 2

3.1. Instance Discrimination

• Two augmented versions of each scene are trained to have similar global features, while features from other scenes are pushed apart using a contrastive loss.The method assumes no specific point ordering and requires neither carefully sampled overlapping multi-view inputs nor object-centric depth maps; it can use single-view depth from varied sensors without 3D calibration.Source: §3.1 · p. 3
• Using 130K negatives for contrastive learning, negatives come from a queue, enabling a large negative set without increasing training batch size; the momentum encoder supplies features of the other augmentation.The momentum encoder uses momentum 0.9; projected features are 128-dimensional with temperature 0.1. Training uses SGD with momentum 0.9 for 1000 epochs, batch size 1024, and a cosine learning-rate schedule from 0.12 to 0.00012.Source: §3.1 · p. 3, §3.5 · p. 4

3.2. Extension to Multiple 3D Input Formats

• A joint objective performs instance discrimination within and across formats, aligning point and voxel features from the same 3D data despite different encoder architectures.When formats match, the objective reduces to within-format discrimination; across formats, it aligns representations from format-specific networks.Source: §3.2 · p. 3

3.3. Model Architecture

• Point inputs use PointNet++ on 20K XYZ points, while voxel inputs use a sparse-convolution U-Net on 5 cm occupancy grids with RGB; both produce global 256-dimensional features.Each encoder pools its final spatial features, then applies a two-layer MLP and L2 normalization; the point network uses 512 channels in its final two upsampling layers.Source: §3.3 · pp. 3–4

3.4. Data Augmentation for 3D

• Augmentations combine random sampling, flips, and rotations with random cuboid crops and dropped patches to encourage learning local geometric features.For voxel inputs, the method also uses voxel-specific augmentations. The Table 1 caption reports consistent gains over scratch across eight transfer tasks, using two pretraining datasets, and notes additional LiDAR evaluation in § 5.3.Source: §3.4 · p. 4, Table 1 · p. 4

4.1. Transfer Datasets and Tasks

• Transfer evaluations span classification, semantic segmentation, and object detection across varied datasets, sensors, views, and indoor/outdoor settings; detection and segmentation use local features, while classification uses global features.The experiments cover full-scene and object-centric datasets as well as real and synthetic data.Source: §4 · p. 4, §4.1 · p. 5

4.2. Pretraining with Point Input Format

• DepthContrast outperforms scratch training on all four detection datasets, adds 12.1% mAP on S3DIS, and beats detection-specific supervised pretraining on SUNRGBD and Matterport3D.S3DIS has only 200 labeled training samples.Source: §4.2 · p. 5, Table 2 · p. 5

4.2.1 Training Higher Capacity Models

• Pretraining reduces the overfitting that limits larger PointNet++ models trained from scratch, allowing transfer performance to improve or remain stable as model capacity grows.The effect is especially relevant to small datasets such as S3DIS, where scratch-trained larger models do not improve.Source: §4.2.1 · p. 5, Figure 3 · p. 6

4.2.2 Using More Pretraining Data

• Tripling pretraining data with Redwood-vid and ScanNet-vid improves detection across all four datasets as data and model capacity grow, with larger gains on S3DIS.Redwood-vid contributes over 23 million depth scans; the study uses 370K scans from 2,500 video sequences because the dataset lacks camera extrinsics for registration.Source: §4.2.2 · pp. 5–6, Figure 3 · p. 6

4.2.3 State-of-the-art Detection Frameworks

• DepthContrast pretraining with PointNet++ 3× sets state-of-the-art detection performance on ScanNet and SUNRGBD using H3DNet and VoteNet.The gains are larger at the stricter IoU=0.5 threshold, indicating improved localization.Source: Table 3 · p. 5, §4.2.3 · p. 6

4.2.4 Label Efficiency of Pretrained Models

• With 50% of labeled samples, pretrained models match scratch training on the full ScanNet and SUNRGBD datasets; with 20%, they gain over 10% mAP.Pretraining also improves SUNRGBD tail-class AP by 4–8%, for classes with fewer labeled instances.Source: §4.2.4 · pp. 6–7, Figure 4 · p. 6

4.3. Pretraining with Multiple Input Formats

• Joint within- and across-format pretraining gives the best transfer performance, while Random Cuboid and Random Drop improve point-model transfer over standard supervised-learning augmentation.Across-format loss improves both point and voxel models; the combined loss improves voxel transfer by 4% over within-format loss, whereas within-format pretraining alone does not consistently help voxel models.Source: §4.3 · p. 7, Table 4 · p. 7, Table 5 · p. 7

5.1. Importance of Data Augmentation

• 5% accuracy on ModelNet classification and 2% mAP on SUNRGBD detection are gains from the proposed augmentations over widely used VoteNet augmentations.The authors report consistent gains from the improved augmentations across all downstream tasks from § 4.Source: §5.1 · pp. 7–8

5.2. Impact of Single-view or Multi-view 3D Data

• ScanNet-vid, ScanNet, and Redwood-vid pretraining yield similar transfer performance, and every DepthContrast model beats scratch, indicating robustness to view count, preprocessing, and pretraining data distribution.ScanNet and ScanNet-vid differ in 3D preprocessing, while Redwood-vid is single-view only.Source: §5.2 · p. 8, Table 6 · p. 8

5.3. Generalization to Outdoor LiDAR data

• On KITTI cyclist mAP, outdoor-pretrained point and voxel models beat scratch; Spconv-Unet gains 20% with 5% labeled data, especially in low-label fine-tuning.The models are pretrained on 79K single-view Waymo scans and jointly optimized with within- and across-format losses; random cuboid augmentation is adapted to the scene’s full depth range.Source: Figure 5 · p. 8, §5.3 · p. 8

• DepthContrast achieves state-of-the-art performance on detection and segmentation benchmarks, outperforming all prior work on detection.Source: §6 · p. 8
• DepthContrast pretrains high-capacity 3D recognition models across architectures, input formats, indoor/outdoor settings, and single- or multi-view data.It can leverage large-scale 3D data that may not have multi-view information.Source: §6 · p. 8
• The method’s design insights include training jointly with multiple input data formats and using a generalizable data augmentation scheme.Source: §6 · p. 8

PointNet++

• PointNet++ uses four set-abstraction layers and two feature-propagation layers, with wider 2×, 3×, and 4× variants for higher capacity.Set-abstraction layers are designed in [67], and feature propagation interpolates input-point features to output points as designed in [71]. Width is increased in the MLPs of the four abstraction layers; feature dimensions are adjusted for VoteNet and H3DNet.Source: A. Architecture Details · p. 14

PointnetMSG

• For LiDAR detection, PointnetMSG has four multi-scale set-abstraction layers and four feature up-sampling layers, and its learned backbone is evaluated in PointRCNN on KITTI.Each abstraction layer uses two ball-region radii and two MLPs, following the architecture design in [80].Source: A. Architecture Details · p. 14

UNet

• The UNet used for scene segmentation has four encoding and four decoding resblocks, using sparse convolutions and sparse resblocks designed in [17].Each resblock begins with a stride-2, kernel-2 sparse convolution or deconvolution, followed by N stride-1, kernel-3 sparse convolution layers; the backbone is applied to different scene-segmentation tasks.Source: A. Architecture Details · p. 14, Figure 6 · p. 14

Spconv-UNet

• For LiDAR Part A2 detection, Spconv-UNet uses four sparse encoding blocks and four decoding upblocks, with a backbone fine-tuned for KITTI detection.It uses sparse convolutions and sparse resblocks designed in [81], and adopts the Part A2 architecture design.Source: A. Architecture Details · pp. 14–15, Figure 7 · p. 15

B.1. Pretraining Details

• Pretraining used SGD with momentum 0.9, cosine learning-rate decay from 0.12 to 0.00012, 1,000 epochs, and a batch size of 1,024.The authors report that 400 epochs already gives good results; training for 1,000 epochs only slightly improves the model.Source: §B.1 · p. 15

B.2. Experimental Details for PointNet++

• VoteNet fine-tuning follows the original configurations, matches scratch-training settings, and loads only the pretrained PointNet++ backbone.Training uses Adam at a 0.001 base learning rate for 180 epochs, except 360 for S3DIS; the rate drops 10× at epochs 80, 120, and 160. Batch sizes are 8 for ScanNet, Matterport3D, and S3DIS, and 16 for SUNRGBD.Source: §B.2 · p. 15
• For H3DNet fine-tuning, the experiments use one backbone instead of the original four and retain the original learning-rate configuration.A 3× PointNet++ backbone reproduces the paper’s reported results; scratch training uses the same configuration, and fine-tuning loads only the pretrained backbone.Source: §B.2 · p. 15

B.2.1 Modelnet Classification

• ModelNet transfer is evaluated by training linear classifiers on fixed features and measuring classification accuracy.Source: §B.2.1 · p. 15
• On ModelNet classification, pretrained PointNet++ provides consistent improvements over scratch in both linear probing and full fine-tuning.Table 9 compares the two evaluation protocols for models pretrained with DepthContrast on ScanNet-vid.Source: §B.2.1 · p. 15, Table 9 · p. 15
• For full ModelNet fine-tuning, PointNet++ point features are max-pooled before a linear classifier predicts class labels.Training uses SGD with momentum for 28 epochs, an initial learning rate of 0.01, and 10× drops at epochs 8, 16, and 24; linear probing fixes pretrained weights and trains only the last layer.Source: §B.2.1 · p. 15

B.2.2 Label efficiency of PointNet++

• Across detection benchmarks, pretrained PointNet++ results are consistently better than scratch; Figure 8 shows label-efficiency plots for Matterport3D and S3DIS.The evaluation follows the main paper’s § 4.2.4 settings and uses PointNet++ pretrained with DepthContrast on ScanNet-vid.Source: §B.2.2 · p. 15, Figure 8 · p. 16

B.3. Experimental Details for UNet

• UNet fine-tuning uses SGD with momentum, initial learning rate 0.1, polynomial scheduling with power 0.9, and weight decay 0.0001.Source: §B.3 · p. 16
• Voxel sizes are 5 cm for S3DIS and Synthia and 4 cm for ScanNet, with the original data augmentations from [17].Source: §B.3 · p. 16
• Training uses batch sizes of 48 for S3DIS and 56 for ScanNet and Synthia, with eight V100 GPUs and 20,000 iterations for each task.Source: §B.3 · p. 16
• Scratch training and fine-tuning use the same configuration, and fine-tuning loads only the pretrained UNet backbone.Source: §B.3 · p. 16

B.3.1 ScanNet Segmentation

• Although pretraining and fine-tuning use the same ScanNet dataset, the approach still improves segmentation; memory constraints raised voxel size from 2 cm to 4 cm, changing scratch results versus [17].The improvement is shown in Table 10.Source: §B.3.1 · p. 16

B.3.2 Label efficiency of UNet

• DepthContrast-pretrained UNet, fine-tuned for S3DIS segmentation, consistently boosts performance across different training-data percentages.The model is pretrained on ScanNet-vid; the data-efficiency plot covers S3DIS scene segmentation.Source: §B.3.2 · p. 16

C. Experimental Details for Lidar Data

• DepthContrast is evaluated on LiDAR data using the experimental setup corresponding to Section 5.3 of the main paper.Source: C. Experimental Details for Lidar Data · p. 16

C.1. Fine-tuning Details

• Scratch-training results across dataset splits use the original PointRCNN [80] and Part A2 [81] configurations.Source: §C.1 · p. 16
• Label-efficiency figures cover S3DIS scene segmentation with a ScanNet-vid-pretrained voxel UNet and KITTI car detection at moderate difficulty on the validation split.Source: Figure 9 · p. 16, Figure 10 · p. 16
• The KITTI validation-split figures cover pedestrian detection at moderate difficulty, including a comparison of within-format and joint training loss.Source: Figure 11 · p. 17, Figure 12 · p. 17
• PointRCNN and Part A2 use AdamW [54], with initial learning rates of 0.01 and 0.003, respectively.Both use weight decay 0.01 and momentum 0.9; batch sizes are 24 and 16, and training lasts 80 epochs with 10× learning-rate drops at epochs 35 and 45. Both methods use the same augmentation and processing pipelines in.Source: §C.1 · p. 17
• Scratch training and fine-tuning share configurations for 5%, 10%, 20%, and 50% of labeled data; full-data training was adjusted to address overfitting on cyclist and pedestrian.For 100% data, the initial learning rate was doubled; PointRCNN training was shortened to 60 epochs with its first weight drop at epoch 30, without changing scratch-training performance.Source: §C.1 · p. 17

• The section presents KITTI results for classes beyond the cyclist class shown in the main paper.Source: §C.2 · p. 17
• With 5% labeled data, DepthContrast boosts KITTI car detection at moderate difficulty; other difficulty levels show a similar pattern.The evaluation reports label efficiency for car detection and notes the boost is especially pronounced with 5% of labeled data.Source: §C.2 · p. 17
• Pretraining consistently improves PointRCNN pedestrian detection over scratch, while Part A2 gains significantly with fewer training instances.Source: §C.2 · p. 17
• Joint-loss training improves KITTI pedestrian detection more with fewer training instances, with benefits holding across pretraining data and architectures.The comparison evaluates the full joint loss against the within-format-only loss.Source: §C.2 · p. 17