Source-linked AI summary
PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding
Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas J. Guibas, Or Litany
TL;DR
Transfer learning was well established in language and vision, but its usefulness for 3D point-cloud understanding remained uncertain despite the cost of 3D annotation. The paper evaluates unified unsupervised pre-training across high-level tasks and diverse datasets, finding improvements across six benchmarks and results similar to supervised pre-training.
Problem
The usefulness of transferable representations for high-level 3D point-cloud understanding is poorly established, despite the difficulty and expense of annotating 3D data.
Method
The paper pre-trains a shared sparse residual U-Net on ScanNet using contrastive objectives, then fine-tunes it across diverse segmentation and detection datasets.
Results
The learned representation improves performance across 6 different benchmarks spanning segmentation and detection, with results indicating generalization across domains.
Takeaways & Limitations
The findings support scaling 3D pre-training data over pursuing more detailed annotations and encourage further research on unsupervised 3D representation learning.
Takeaways & Limitations
Temporal learning for 4D networks is left for future work, so the reported pre-trained weights use a 3D network rather than an explicitly temporal model.
Abstract
from arXiv · showhide
Arguably one of the top success stories of deep learning is transfer learning. The finding that pre-training a network on a rich source set (eg., ImageNet) can help boost performance once fine-tuned on a usually much smaller target set, has been instrumental to many applications in language and vision. Yet, very little is known about its usefulness in 3D point cloud understanding. We see this as an opportunity considering the effort required for annotating data in 3D. In this work, we aim at facilitating research on 3D representation learning. Different from previous works, we focus on high-level scene understanding tasks. To this end, we select a suite of diverse datasets and tasks to measure the effect of unsupervised pre-training on a large source set of 3D scenes. Our findings are extremely encouraging: using a unified triplet of architecture, source dataset, and contrastive loss for pre-training, we achieve improvement over recent best results in segmentation and detection across 6 different benchmarks for indoor and outdoor, real and synthetic datasets -- demonstrating that the learned representation can generalize across domains. Furthermore, the improvement was similar to supervised pre-training, suggesting that future efforts should favor scaling data collection over more detailed annotation. We hope these findings will encourage more research on unsupervised pretext task design for 3D deep learning.
1 Introduction
The paper addresses limited evidence for transferable 3D representations by evaluating unsupervised pre-training for high-level scene understanding across diverse tasks and datasets. A unified architecture, source dataset, and contrastive objective improves results across six benchmarks and approaches supervised pre-training.
- Motivation: 3D scene understanding still commonly trains from scratch because large, high-quality data and unified backbone architectures remain limited.3D data is harder to collect and label, while sensing devices can introduce domain gaps.
- Approach: The study evaluates unsupervised pre-training with supervised fine-tuning across a large source dataset, shared backbone, two objectives, and diverse downstream tasks.The evaluation covers semantic segmentation and object detection across indoor, outdoor, real, and synthetic settings.
- Approach: ScanNet, a sparse residual U-Net, and contrastive losses provide the unified pre-training setup for transferring point-cloud representations.The evaluated objectives are Hardest-contrastive loss and PointInfoNCE.
- Implication: The results suggest a relatively small advantage for supervised pre-training, motivating greater emphasis on scaling pre-training data than on precise annotations.The authors connect this finding to the substantial labor required to annotate thousands of rooms.
- Contribution: The work evaluates transferability to high-level 3D scene understanding rather than restricting representation learning to single objects or low-level tasks.This directly expands the evaluation target identified as missing in prior work.
- Results: The unified unsupervised framework achieves new state-of-the-art performance across 6 different benchmarks.The benchmarks span segmentation and detection across diverse domains.
2 Related work
Prior 3D representation-learning work largely targets point-cloud processing challenges and low-level or single-object settings. The paper positions broader, transferable point-cloud representations as requiring more standardized architectures and more general pre-training data.
- Representation learning: 3D representation-learning methods have mainly focused on ShapeNet single-object CAD models, assuming synthetic features can transfer to real-world applications.The paper reassesses this assumption through supervised pre-training on ShapeNet and downstream fine-tuning.
- Point-cloud architectures: Point-cloud architecture design remains less mature than 2D vision because unordered sets, neighborhood aggregation, and discrete surface sampling create distinct processing challenges.The surveyed aggregation families include hierarchical, spatial CNN-like, spectral, and graph-based approaches.
- Point-cloud architectures: Sparse convolutional networks facilitate adopting common deep architectures for point-cloud processing.The paper highlights the Minkowski Engine as an extension to higher-dimensional sparse convolutional networks.
3 PointContrast Pre-training
PointContrast revisits 3D pre-training by moving beyond ShapeNet single-object data toward dense, scene-level point representations. It combines a fully convolutional architecture, point-level contrastive learning, and ScanNet pre-training for transfer across high-level tasks.
- Motivation: ShapeNet pre-training is limited by domain gaps because its synthetic, normalized, pose-aligned objects lack scene context.These differences make source and target distributions substantially different for real-world scene understanding.
- Evaluation: PointContrast improves performance across diverse high-level 3D understanding benchmarks compared with training from scratch.The summary includes semantic segmentation, detection, and classification tasks, with some results using only 1% of training data.
- PointContrast Pretext Task: The pretext pipeline generates two aligned views, computes point correspondences, applies random geometric transformations, and optimizes contrastive loss on matched points.The transformations include rotation, translation, and scaling, encouraging feature consistency across geometric changes.
- PointContrast Pretext Task: PointContrast repurposes fully convolutional geometric feature learning for high-level 3D understanding through point-level contrastive pre-training.The framework contrasts corresponding points across transformed views rather than relying only on instance-level or global representations.
- Contrastive Loss Design: PointInfoNCE is proposed alongside hardest-contrastive loss to exploit the large number of point-level negatives available in 3D scenes.The paper reports that hardest-contrastive training can be unstable and collapse with extended training, while PointInfoNCE is more robust against mode collapse.
- Architecture and Dataset: ScanNet supplies the pre-training data, while a 37.85M-parameter Sparse Residual U-Net serves as the shared backbone for pre-training and diverse fine-tuning tasks.The dataset is built from partial scans from different views, and the architecture is intended to support a unified design across tasks.
4 Fine-tuning on Downstream Tasks
PointContrast is evaluated by fine-tuning a shared ScanNet-pretrained backbone across diverse 3D tasks and datasets. Pre-training improves segmentation, detection, classification, and synthetic-domain transfer, with especially strong benefits when labels or training data are scarce.
- ShapeNet: Classification and Part Segmentation: PointContrast further improves ShapeNet part segmentation beyond the gains from replacing the backbone with SR-UNet, especially with limited labels.The reported table caption states that pre-training adds a significant gain and outperforms where labels are most limited.
- ShapeNet: Classification and Part Segmentation: A 4.0% improvement over the training-from-scratch baseline is achieved on ShapeNet classification with the hardest-negative objective.The gain is largest with less training data, and pre-training also improves underrepresented tail classes.
- S3DIS Segmentation: 2.7% mIoU with Hardest-Contrastive and 2.1% mIoU with PointInfoNCE improve S3DIS segmentation over the corresponding training-from-scratch setup.The improvement persists across both contrastive objectives.
- SUN RGB-D Detection: +3.1% mAP@0.5 closes the detection gap and establishes a new state of the art, despite the SR-UNet scratch baseline being 1.4% mAP@0.25 worse.The larger gain at the harder mAP@0.5 metric indicates stronger benefits for localization.
- Synthia4D Segmentation: +2.3% mIoU improves Synthia4D segmentation over training from scratch, with PointInfoNCE outperforming the hardest-contrastive loss.The models use 3D pre-training for individual frames; extending the approach to temporal 4D networks is left for future work.
- Analysis Experiments and Discussions: An mIoU of 71.2% for supervised pre-training is only 0.3% better than PointContrast unsupervised pre-training on the S3DIS transfer setup.The authors interpret this as evidence that the gap between supervised and unsupervised 3D representation learning is mostly closed.
5 Conclusions
PointContrast demonstrates that learned 3D point-cloud representations transfer to high-level understanding tasks, achieving state-of-the-art results across six benchmarks and generalizing across domains.
- State-of-the-art results were achieved across 6 different benchmarks.
- The evaluation measures transferability to high-level 3D understanding tasks.
- The learned representation generalizes across domains.
A Visualization of the SR-UNet Architecture
SR-UNet is the shared backbone used during pre-training and fine-tuning, with task-dependent choices about which network components are retained.
- SR-UNet serves as the shared backbone for pre-training and fine-tuning tasks.
- Segmentation and detection fine-tune both the encoder and decoder weights.
- Classification fine-tunes only the encoder network.
B Visualization of the ScanNet Point Cloud Pair Dataset
The ScanNet pre-training dataset consists of paired point-cloud views sampled from the same scenes, with overlapping points providing correspondence across views.
- Each dataset row contains a randomly sampled scene, while each column shows a different point-cloud pair from that scene.
- The paired views represent two partial scans, distinguished by different colors.
- At least 30% of the points overlap between the two views.
C ShapeNet Supervised Training Details
The ShapeNet supervised-training setup uses a sparse ResNet encoder with standard augmentations and a grid search over key training parameters, reaching 85.4% accuracy on the SHREC benchmark split.
- The sparse ResNet has the same structure as the SR-UNet encoder.
- Training used Adam with rotation, scaling, and translation augmentations.
- A grid search varied learning rate, weight decay, voxel size, and input-point count.
- 85.4% accuracy was achieved on the SHREC benchmark split.The authors report this as the best result known to them for that split.
- PointInfoNCE computes cross-entropy over an Ns by Ns feature-similarity matrix with matched points as positives.
- The shared backbone is updated through backpropagation after the contrastive loss is computed.
E S3DIS Segmentation Experimental Details
The S3DIS experiments use the Area 5 Test (Fold 1) split and identical hyperparameter settings across PointContrast variants.
- S3DIS uses the widely adopted Area 5 Test (Fold 1) split for training and testing.
- Training from scratch, Hardest-contrastive Pretrained, and PointInfoNCE Pretrained use the same hyperparameter settings.
- Models are trained for 10,000 iterations on 8 V100 GPUs with data parallelism and batch size 48.
- Optimization uses SGD with momentum, initial learning rate 0.8, polynomial scheduling with power 0.9, weight decay 0.0001, and 5cm voxels.
- Table 9 reports per-category IoU performance for the Stanford Area 5 Test (Fold 1).
F Synthia4D Segmentation Experimental Details
The Synthia4D segmentation experiments use only a 3D sparse ConvNet, omit temporal aggregation, and retain largely identical settings across PointContrast variants.
- Synthia4D uses only a 3D sparse ConvNet without temporal aggregation mechanisms such as 4D kernels and temporal CRF.
- Training from scratch, Hardest-contrastive Pretrained, and PointInfoNCE Pretrained use the same hyperparameter settings.
- The Synthia4D settings are mostly identical to those used in the S3DIS experiments.
G ScanNet Segmentation Experimental Details
The ScanNet experiments specify segmentation and detection configurations, including separate voxel sizes and training schedules, with results reported through per-category tables.
- Segmentation: ScanNet segmentation training runs for 15,000 iterations on 8 V100 GPUs with data parallelism and batch size 48.
- Segmentation: ScanNet segmentation uses SGD with momentum, initial learning rate 0.8, polynomial scheduling with power 0.9, weight decay 0.0001, and 2.5cm voxels.
- Detection: 3D detection follows VoteNet configurations after replacing its backbone with SR-UNet, using Adam with learning rate 0.001.
- Detection: Detection uses batch sizes 64 for SUN RGB-D and 32 for ScanNet, with 20,000 and 40,000 input points respectively.
I PointContrast vs FCGF for low- and high-level tasks
FCGF performs better on registration feature matching, whereas PointContrast performs better on segmentation, indicating different design requirements for low- and high-level 3D tasks.
- FCGF achieves a registration feature matching recall (FMR) of 0.958 but does not perform well for S3DIS segmentation.
- The PointContrast model that performs best for segmentation achieves lower FMR on the registration task.
- The authors conclude that low-level and high-level 3D tasks might require different design choices.
- Table 14 summarizes the comparison: FCGF has higher registration FMR, while PointContrast has higher segmentation mIoU.