Source-linked AI summary
Self-Supervised Learning for Domain Adaptation on Point-Clouds
Idan Achituve, Haggai Maron, Gal Chechik
TL;DR
The paper addresses the unresolved question of whether self-supervised learning can enable domain adaptation for 3D point clouds. It introduces deformation-based pretext tasks and Point Cloud Mixup within a shared multi-task architecture. Across classification and segmentation adaptations, the approach improves over existing and baseline methods and reaches state-of-the-art results.
Problem
SSL has been effective for domain adaptation in images and videos, but its applicability to domain adaptation for 3D point clouds remains unknown.
Method
The paper combines DefRec deformation-reconstruction pretext tasks and PCM point-cloud mixing in a multi-task architecture with a shared feature encoder and supervised and SSL heads.
Results
The approach achieves state-of-the-art domain-adaptation results, outperforming baselines in classification and improving segmentation performance across most adaptations.
Takeaways & Limitations
DefRec and PCM form a relatively simple model that improves domain adaptation for point-cloud classification and segmentation across several adaptation setups.
Abstract
from arXiv · showhide
Self-supervised learning (SSL) is a technique for learning useful representations from unlabeled data. It has been applied effectively to domain adaptation (DA) on images and videos. It is still unknown if and how it can be leveraged for domain adaptation in 3D perception problems. Here we describe the first study of SSL for DA on point clouds. We introduce a new family of pretext tasks, Deformation Reconstruction, inspired by the deformations encountered in sim-to-real transformations. In addition, we propose a novel training procedure for labeled point cloud data motivated by the MixUp method called Point cloud Mixup (PCM). Evaluations on domain adaptations datasets for classification and segmentation, demonstrate a large improvement over existing and baseline methods.
1. Introduction
This paper investigates whether self-supervised learning can support domain adaptation for 3D point clouds, where its applicability remains unclear. It introduces Deformation Reconstruction and Point Cloud Mixup, reporting improved classification and segmentation adaptation performance.
- SSL has proven effective for learning representations and adapting across image and video domains, but its applicability to 3D point-cloud domain adaptation remains unknown.
- The paper uses a multi-task architecture with a shared feature encoder, a supervised source-domain head, and an SSL head usable on both domains.
- Deformation Reconstruction (DefRec) deforms regions by dislocating points and trains the network to reconstruct their original locations, modeling distortions common in scanned point clouds.
- DefRec deformation strategies include volume-based selection in R3, feature-based semantic selection, and sampling-based selection schemes.
- Point Cloud Mixup (PCM) adapts MixUp by combining randomly sampled points from two point clouds and convexly combining their one-hot labels.
- DefRec and PCM yield large improvements in classification domain adaptation, while DefRec extends to segmentation and improves performance over baseline methods.
2. Related work
Related work spans point-cloud representation learning, self-supervised point-cloud tasks, domain adaptation for point clouds, SSL-based domain adaptation, and reconstruction or completion. The paper differentiates its approach by applying SSL domain adaptation to unordered point clouds with deformation-based pretext tasks.
- Deep learning on point clouds: Early point-cloud deep-learning methods used volumetric convolutions, while later models operated directly on point-cloud representations.
- Self-supervised learning for point clouds: Prior self-supervised point-cloud methods used voxel shuffling, space-filling sequence prediction, half-shape matching, clustering, prediction, and noisy-input reconstruction.
- Domain adaptation for point clouds: Point-cloud domain adaptation work includes PointDAN for jointly aligning local and global features, shared-space modules, and methods targeting LiDAR or segmentation data.
- Self-supervised learning for domain adaptation: Existing SSL domain-adaptation studies mainly target images using shared encoders, reconstruction, rotation, patch-location, Jigsaw, or synthetic-label pretext tasks.
- Deep learning of point cloud reconstruction and completion: Point-cloud reconstruction and completion studies primarily pursue high-quality shape recovery, whereas this paper draws on them to design reconstruction pretexts for domain adaptation.
3. Approach
The approach combines supervised learning with self-supervised Deformation Reconstruction for adapting point-cloud representations, optionally strengthened by Point Cloud Mixup. DefRec deforms regions and reconstructs their original points using several region-selection strategies.
- General pipeline: DefRec trains a shared encoder with supervised source-domain learning and self-supervised reconstruction of deformed point clouds.The two data flows are trained alternately and end-to-end, using labeled source data and unlabeled source or target data.
- Deformation reconstruction: DefRec deforms selected regions by relocating points, then reconstructs their original locations to learn representations of target-domain structure.The task is motivated by missing parts and other distortions encountered in scanned point clouds.
- Deformation reconstruction: The deformation family selects regions volumetrically, semantically from deep features, or through sampling-based protocols.Volume-based methods use spatial proximity; feature-based methods use point embeddings; sampling-based methods use three sampling schemes.
- Deformation reconstruction: DefRec applies a within-region Chamfer reconstruction loss between the original points and the reconstructed outputs.Restricting Chamfer distance to points inside the deformed region avoids burdening the computation.
- Training choice: Applying DefRec only to target samples produced better results in the reported experiments.This target-only configuration is used unless otherwise stated.
- Point Cloud Mixup: Point Cloud Mixup forms a new point cloud by combining randomly sampled points from two inputs and assigns their labels a convex combination.A Beta-distributed coefficient controls the proportions of points and the mixed one-hot labels.
- Point Cloud Mixup: PCM improves accuracy across baselines and is particularly beneficial when combined with DefRec.It is applied to source objects during training instead of the standard classification task.
4. Experiments
The experiments evaluate classification on PointDA-10 and segmentation on the newly constructed PointSegDA benchmark. These datasets contain domain shifts in source and target point-cloud distributions, shapes, poses, and sampling characteristics.
- Evaluation datasets: The evaluation covers point-cloud classification on PointDA-10 and segmentation on the newly introduced PointSegDA benchmark.PointSegDA was constructed from human-model meshes for segmentation evaluation.
- Classification benchmark: PointDA-10 contains ModelNet-10, ShapeNet-10, and ScanNet-10, which share ten object classes but differ in dataset characteristics.The subsets include clean CAD models, online CAD repositories, and RGB-D scans.
- Segmentation benchmark: PointSegDA uses ADOBE, FAUST, MIT, and SCAPE as domains sharing eight human-body-part classes.The domains differ in humans, poses, point distributions, discretizations, and deformable-object shapes.
- Experimental setup: The network processes 1024 points for PointDA-10 or 2048 points for PointSegDA, using DGCNN for feature extraction and supervised prediction.The SSL head receives the global feature vector concatenated with point-level backbone representations.
5. Results
DefRec and PCM outperform baseline methods across point-cloud domain-adaptation experiments, with benefits extending to segmentation. Analyses identify effective deformation choices and show that combining the components is advantageous.
- Classification accuracy: DefRec and DefRec + PCM outperform all baselines in 5 of 6 classification adaptations, while their six-task average is highest.DefRec + PCM improves by 5% over the best baseline and 5.5% over PointDAN.
- Classification accuracy: DefRec is more accurate on the ModelNet-to-ScanNet and ShapeNet-to-ScanNet sim-to-real adaptations.The paper associates this with semantic shape properties and missing-region generalization.
- Deformation analysis: Volume-based deformations achieve the highest average test accuracy, with object-part-scale regions outperforming global deformations.Accuracy is sensitive to deformation radius: small regions can be too easy, while large regions can be too difficult.
- Deformation analysis: Feature-based deformation remains robust across layers and point counts for sufficiently small regions, while regions larger than 300 points degrade performance.Layer 3 slightly outperforms layer 4 in the reported analysis.
- SSL versus augmentation: Using sample-based deformations as an SSL task is preferred over using them as data augmentation, with 11 of 18 comparisons below the y = x line.Five of those comparisons are sim-to-real adaptations.
- Ablation and segmentation: Combining DefRec and PCM significantly boosts overall performance, whereas applying DefRec to both source and target samples or reconstructing from global features often reduces performance.The segmentation experiments also show that DefRec, with or without PCM, achieves the highest results on most adaptations.
6. Conclusions
The paper addresses 3D point-cloud domain adaptation with DefRec and PCM, combining self-supervised deformation reconstruction with MixUp-inspired training. Across several adaptation setups, the combined approach achieves state-of-the-art results with a relatively simple architecture.
- Contributions: DefRec introduces self-supervised pretext tasks based on deformations encountered in real 3D point-cloud data.PCM is a MixUp-based training procedure applicable to classification and segmentation.
- Contributions: DefRec and PCM are complementary, and their combination forms a strong model with a relatively simple architecture.The paper reports benefits across several adaptation setups.
- Results: The combined method reaches new state-of-the-art results on the evaluated domain-adaptation setups.
A. PointSegDA dataset
PointSegDA is a segmentation domain-adaptation benchmark built from four human-mesh subsets with shared body-part classes but differing shapes, poses, and point distributions.
- Dataset construction: PointSegDA uses ADOBE, FAUST, MIT, and SCAPE as distinct domains derived from human-model triangular meshes.Each mesh is converted to a point cloud by extracting vertices and sampling 2048 points with farthest point sampling.
- Domain shifts: The four domains differ in body pose, shape, and point distribution, creating segmentation domain shifts.Point labels are obtained from the mesh annotations.
B.1. PointDA-10 dataset
The PointDA-10 experiments use aligned, sampled point clouds with a DGCNN-based architecture and specified training and deformation configurations. The setup includes classification, SSL, and PCM components for evaluating domain adaptation.
- Data processing: PointDA-10 preprocessing aligns point clouds with the positive Z axis and samples 1024 points using farthest point sampling.ModelNet and ScanNet samples are split into training and validation portions as described in the procedure.
- Network architecture: The network uses a DGCNN feature extractor with four point-cloud convolution layers followed by a 1024-dimensional feature layer and max pooling.The architecture includes a spatial transformation network and classification heads.
- Training procedure: Training alternates source and target batches, uses batch size 32 per domain, ADAM, cosine annealing, and grid search over learning rate and weight decay.The domains are balanced by undersampling the larger domain in each epoch.
- Evaluation: The reported PointDA-10 comparisons include PCM-equipped baselines and are evaluated with test accuracy averaged over three runs.DefRec training requires 6–9 hours on a 16 GB Nvidia V100 GPU, depending on the dataset.
- Deformation configurations: The implementation evaluates volume-, feature-, and sample-based deformation families with their respective region-selection schemes.Volume-based methods vary voxel partitions or sphere radii; feature-based methods vary encoder layers and nearest-neighbor counts.
- Segmentation configuration: Segmentation uses a DGCNN-based head with global and point features, four 1D convolution layers, and eight output classes.Segmentation training uses batch size 16 per domain, 200 epochs, and auxiliary-task-weight search.
C. Additional experiments
Additional experiments show that Point Cloud Mixup improves several baseline methods, while DefRec combined with PCM remains superior. A mixed-deformation protocol offers comparable results and improves selected adaptations.
- C.1. PCM on baselines: PCM boosts RS, DAE-Global, and DAE-Point, but provides less benefit for DANN and PointDAN.
- C.1. PCM on baselines: DefRec combined with PCM remains superior despite PCM's weaker effect on some baselines.The authors speculate that PCM may hurt PointDAN because its discrepancy loss uses two classification heads.
- C.2. Combining deformation strategies: Applying all deformation types with equal probability avoids selecting one deformation type and its specific hyperparameters beforehand.Each deformation is selected with probability 1/3 per batch, using hyperparameters informed by source-validation sensitivity analysis.
- C.2. Combining deformation strategies: The equal-weight deformation protocol is comparable to the original protocol and raises accuracy by 2% on ModelNet to ShapeNet and 3% on ModelNet to ScanNet when applying PCM.
D. Estimating target perplexity
The paper evaluates source–target representation alignment using target log perplexity under source-fitted class models. DefRec + PCM produces more similar representations and lower class-balanced perplexity than PointDAN in both adaptations.
- Target perplexity: A DA solution should align source and target distributions while preserving discriminative representations.
- Target perplexity: Target log perplexity measures the average number of bits required to encode target test representations under a model fitted to source representations.
- Target perplexity: Each class-conditional target representation distribution is modeled with a Gaussian fitted by maximum likelihood from source samples of that class.The class prior is based on the proportion of source samples in each class.
- Target perplexity: Class-balanced perplexity averages log perplexity across classes, preventing well-represented classes from dominating the alignment assessment.
- Evaluation setup: The perplexity estimates are computed after reducing representations to 2D with t-SNE because covariance estimation in the original space is degenerate with relatively few samples.The t-SNE plots show one of ten random seeds, while reported values use the mean and standard error across ten seeds.
- Findings: DefRec + PCM produces more similar source and target representations and lower class-balanced perplexity than PointDAN in both ModelNet-to-ScanNet and ModelNet-to-ShapeNet.The authors interpret the lower class-balanced perplexity as better learning of under-represented classes.
E. Shape reconstruction
DefRec reconstructs point-cloud regions after localized deformations, learning to identify the altered region and restore a plausible object shape. Reconstruction quality varies by object region and can differ from the exact ground truth.
- E. Shape reconstruction: DefRec reconstructs deformed shapes by reversing localized point-cloud deformations.The illustrated volume-based variant divides the input space into 3 × 3 × 3 voxels and selects one voxel uniformly at random.
- E. Shape reconstruction: Across simulated and real domains, DefRec learns to recognize the deformed region and reconstruct it while preserving the object's overall shape.
- E. Shape reconstruction: Some reconstructions are plausible but not fully consistent with the ground truth, including examples such as Monitor and Lamp.The network may reconstruct the object in a different but still plausible manner.
- E. Shape reconstruction: Reconstruction succeeds for some regions, such as a chair's top rail and table legs, but fails for others, such as the chair's seat.