Source-linked AI summary
CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding
Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, Ranga Rodrigo
TL;DR
Large-scale point-cloud annotation is laborious, creating a need for self-supervised learning that avoids human labels. CrossPoint learns transferable representations by jointly aligning point clouds with rendered 2D images and enforcing invariance across point-cloud transformations, outperforming prior unsupervised methods across diverse downstream tasks.
Problem
Annotating large-scale point-cloud datasets is laborious because point clouds have an irregular structure, motivating self-supervised learning without human labels.
Method
CrossPoint jointly learns intra-modal point-cloud invariance and cross-modal correspondence between augmented point clouds and their corresponding rendered 2D images.
Results
CrossPoint outperforms previous unsupervised methods across object classification, few-shot learning, and part segmentation on diverse synthetic and real-world datasets.
Takeaways & Limitations
Joint intra-modal and cross-modal learning produces more generic and transferable point-cloud features for downstream 3D understanding.
Takeaways & Limitations
CrossPoint fails to generalize to CIFAR-FS because rendered point-cloud images differ considerably from real-world CIFAR-FS images.
Abstract
from arXiv · showhide
Manual annotation of large-scale point cloud dataset for varying tasks such as 3D object classification, segmentation and detection is often laborious owing to the irregular structure of point clouds. Self-supervised learning, which operates without any human labeling, is a promising approach to address this issue. We observe in the real world that humans are capable of mapping the visual concepts learnt from 2D images to understand the 3D world. Encouraged by this insight, we propose CrossPoint, a simple cross-modal contrastive learning approach to learn transferable 3D point cloud representations. It enables a 3D-2D correspondence of objects by maximizing agreement between point clouds and the corresponding rendered 2D image in the invariant space, while encouraging invariance to transformations in the point cloud modality. Our joint training objective combines the feature correspondences within and across modalities, thus ensembles a rich learning signal from both 3D point cloud and 2D image modalities in a self-supervised fashion. Experimental results show that our approach outperforms the previous unsupervised learning methods on a diverse range of downstream tasks including 3D object classification and segmentation. Further, the ablation studies validate the potency of our approach for a better point cloud understanding. Code and pretrained models are available at http://github.com/MohamedAfham/CrossPoint.
1. Introduction
CrossPoint addresses the difficulty of annotating large point-cloud datasets by learning transferable 3D representations through self-supervised intra-modal and cross-modal contrastive learning. It links point clouds with rendered 2D images, preserves transformation invariance, and improves performance across downstream tasks.
- Manual annotation of large-scale point-cloud datasets is laborious because of their irregular structure, motivating self-supervised representation learning.
- Existing point-cloud contrastive methods mainly impose invariance to 3D augmentations, leaving cross-modal learning comparatively unexplored for point-cloud understanding.
- CrossPoint aligns augmented point clouds with corresponding rendered 2D images in feature space while bringing augmented views of the same point cloud together.
- Its joint objective combines intra-modal and cross-modal losses, embedding image features near corresponding point-cloud prototypes and reducing bias toward a particular augmentation.
- CrossPoint is evaluated on classification, few-shot learning, and part segmentation across synthetic and real-world datasets, outperforming previous unsupervised methods.
- The approach also improves few-shot image classification after finetuning its pretrained image backbone with the RFS method.
2. Related Work
Prior point-cloud representation learning includes direct architectures, augmentation strategies, generative and reconstruction objectives, pretext tasks, and 3D contrastive learning. Cross-modal pretraining provides another route to transferable representations by combining complementary modalities, but CrossPoint specifically biases this process toward 3D point-cloud understanding.
- Representation Learning on Point Clouds: Point-cloud learning is challenging because point sets are irregular and models must preserve permutation invariance across points.
- Representation Learning on Point Clouds: Supervised point-cloud representation methods depend on annotated data, which is difficult to acquire, while cTree targets label-efficient few-shot learning.
- Self-Supervised Learning on Point Clouds: Self-supervised point-cloud methods use generative modeling, reconstruction, pretext tasks, and contrastive learning to learn semantic or invariant representations.
- Cross-Modal Learning: Cross-modal pretraining combines modalities to provide rich learning signals and transferable representations for downstream tasks such as recognition and detection.
- Cross-Modal Learning: Unlike related approaches that apply instance discrimination across both modalities, CrossPoint applies it to point clouds and uses images to improve point-cloud understanding.
3. Proposed Method
CrossPoint jointly learns point-cloud representations through intra-modal invariance to geometric augmentations and cross-modal alignment with rendered images. The combined objective produces a point-cloud backbone for downstream tasks while discarding the image branch after training.
- 3. Proposed Method: CrossPoint fuses intra-modal and cross-modal contrastive objectives into one end-to-end self-supervised training objective.The overall loss combines Limid and Lcmid.
- 3.2. Intra-Modal Instance Discrimination: The intra-modal branch enforces invariance to randomly composed point-cloud transformations, including rotation, scaling, translation, jittering, normalization, and elastic distortion.It applies contrastive instance discrimination to augmented versions of the same point cloud.
- 3.3. Cross-Modal Instance Discrimination: The image branch aligns rendered 2D image features with point-cloud prototype features through a cross-modal contrastive loss.The image feature is extracted and projected into the invariant space before alignment with the prototype.
- 3. Proposed Method: During downstream use, CrossPoint discards the image branch and retains only the point-cloud feature extractor as the backbone.The image modality contributes during pretraining, while downstream tasks use the learned point-cloud features.
- 3.3. Cross-Modal Instance Discrimination: The point-cloud prototype is computed as the mean of the projected features from two augmented views and matched to the corresponding image feature.This design makes the 3D-2D correspondence agnostic to point-cloud transformations.
4. Experiments
CrossPoint is evaluated with PointNet and DGCNN backbones across classification, few-shot learning, part segmentation, and image classification. Results show strong transfer across synthetic and real-world settings, while ablations support the joint objective and reveal an out-of-distribution limitation for images.
- Experimental setup: ShapeNet pretraining uses 2048-point clouds, corresponding rendered RGB images resized to 224 × 224, and modality-specific augmentations.Point-cloud augmentations are supplemented with random crop, color jittering, and horizontal flips for rendered images.
- Experimental setup: CrossPoint uses PointNet or DGCNN for point clouds, ResNet-50 for images, and 2-layer MLP projection heads producing 256-dimensional invariant-space features.Optimization uses Adam with weight decay 1×10^-4, initial learning rate 1 × 10^-3, and cosine annealing.
- Downstream tasks: CrossPoint is evaluated on 3D object classification, few-shot object classification, and 3D object part segmentation across synthetic and real-world datasets.The downstream evaluation is designed to test transferability across multiple tasks and dataset types.
- Downstream tasks: CrossPoint surpasses prior unsupervised methods on ModelNet40 linear classification with both PointNet and DGCNN backbones, including a 5.8% margin over DepthContrast.Classification uses a frozen pretrained feature extractor and a linear SVM; the reported comparison establishes a new self-supervised point-cloud benchmark.
- Downstream tasks: On ScanObjectNN, CrossPoint gains 1.3% with PointNet and 3.4% with DGCNN over previous state-of-the-art unsupervised methods, indicating transfer to out-of-distribution data.The results are reported as linear evaluation accuracy gains on the real-world ScanObjectNN dataset.
- Few-shot and ablation analysis: CrossPoint improves few-shot accuracy across reported ModelNet40 settings and most ScanObjectNN settings, while also supporting ShapeNetPart segmentation and image-backbone transfer analysis.DGCNN can underperform PointNet in some ModelNet40 few-shot settings, and the image branch fails to generalize directly to CIFAR-FS images despite fine-tuning gains over baseline.
- Ablations and analysis: The joint intra-modal and cross-modal objective outperforms individual objectives, yielding DGCNN accuracy gains of 1.2% on ModelNet40 and 0.7% on ScanObjectNN.Cross-modal learning marginally exceeds intra-modal learning, while joint learning produces better class discrimination and can reduce boundary imprecision in examples such as desk and table.
5. Conclusion
CrossPoint learns transferable point cloud representations through self-supervised joint intra-modal and cross-modal correspondence learning. Results across synthetic and real-world classification and part-segmentation datasets, together with ablations, support the efficacy and generality of this approach.
- CrossPoint is a self-supervised framework for learning transferable 3D point cloud representations.
- Joint intra-modal and cross-modal correspondence learning produces more generic and transferable point cloud features.
- Downstream experiments demonstrate efficacy on 3D object classification and part segmentation across synthetic and real-world datasets.
- A few-shot image classification experiment offers insight into cross-modal understanding for future research.