PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders

Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, Hongsheng Li

arXiv:2212.06785v1cs.CVcs.AI

TL;DR

Limited large-scale 3D datasets motivate transferring the knowledge of abundant 2D pre-trained models into 3D representation learning. I2P-MAE performs this transfer through masked point-cloud autoencoding with 2D-guided masking and 2D-semantic reconstruction, achieving strong results across downstream evaluations.

  • Problem

    Large-scale 3D datasets are scarce because point-cloud acquisition and annotation are expensive, limiting the data available for learning high-quality 3D features.

  • Method

    I2P-MAE uses 2D pre-trained models to guide self-supervised 3D masked autoencoding through 2D-guided masking and 2D-semantic reconstruction.

  • Results

    93.4% linear-SVM accuracy is achieved on ModelNet40 without fine-tuning, and fine-tuning reaches 90.11% on ScanObjectNN’s hardest split, +3.68% over Point-M2AE.

  • Takeaways & Limitations

    The reported experiments and conclusion indicate that image-to-point pre-training yields transferable 3D representations and alleviates demand for large-scale 3D datasets.

Abstract

from arXiv · show

Pre-training by numerous image data has become de-facto for robust 2D representations. In contrast, due to the expensive data acquisition and annotation, a paucity of large-scale 3D datasets severely hinders the learning for high-quality 3D features. In this paper, we propose an alternative to obtain superior 3D representations from 2D pre-trained models via Image-to-Point Masked Autoencoders, named as I2P-MAE. By self-supervised pre-training, we leverage the well learned 2D knowledge to guide 3D masked autoencoding, which reconstructs the masked point tokens with an encoder-decoder architecture. Specifically, we first utilize off-the-shelf 2D models to extract the multi-view visual features of the input point cloud, and then conduct two types of image-to-point learning schemes on top. For one, we introduce a 2D-guided masking strategy that maintains semantically important point tokens to be visible for the encoder. Compared to random masking, the network can better concentrate on significant 3D structures and recover the masked tokens from key spatial cues. For another, we enforce these visible tokens to reconstruct the corresponding multi-view 2D features after the decoder. This enables the network to effectively inherit high-level 2D semantics learned from rich image data for discriminative 3D modeling. Aided by our image-to-point pre-training, the frozen I2P-MAE, without any fine-tuning, achieves 93.4% accuracy for linear SVM on ModelNet40, competitive to the fully trained results of existing methods. By further fine-tuning on on ScanObjectNN's hardest split, I2P-MAE attains the state-of-the-art 90.11% accuracy, +3.68% to the second-best, demonstrating superior transferable capacity. Code will be available at https://github.com/ZrrSkywalker/I2P-MAE.

1. Introduction

I2P-MAE transfers knowledge from abundant 2D pre-trained models into 3D point-cloud masked autoencoding through image-to-point learning. Its 2D-guided masking and semantic reconstruction produce transferable 3D representations and strong downstream results.

  • Motivation: Large-scale 3D data remains scarce because acquisition and annotation are expensive, motivating transfer from abundant 2D pre-trained models.ShapeNet contains 50k point clouds across 55 categories, compared with 14 million ImageNet images and 400 million image-text pairs.
  • Framework: I2P-MAE uses an asymmetric encoder-decoder masked autoencoder and projects point clouds into multi-view depth maps to obtain 2D visual representations.The framework transfers 2D knowledge into 3D through self-supervised point-cloud pre-training.
  • Image-to-Point Learning: 2D-guided masking preserves point tokens with greater spatial semantics, helping the encoder focus on critical 3D structures and the decoder reconstruct masked tokens from informative cues.Semantic saliency maps are back-projected into 3D as a spatial saliency cloud that determines nonuniform token visibility.
  • Image-to-Point Learning: 2D-semantic reconstruction makes visible tokens reconstruct multi-view 2D features alongside masked 3D coordinates, combining spatial patterns with high-level semantics.The corresponding projected 2D representations from different views form the reconstruction targets.
  • Results: 93.4% linear-SVM accuracy on ModelNet40 is achieved without fine-tuning, while fine-tuning reaches 90.11% on ScanObjectNN’s hardest split, exceeding Point-M2AE by +3.68%.The reported experiments also indicate faster pre-training convergence and the significance of image-to-point pre-training.

2. Related Work

Prior work develops supervised and self-supervised 3D representation learning, masked autoencoding, and several forms of 2D-to-3D transfer. I2P-MAE extends masked autoencoding for point clouds by adding guidance from 2D pre-trained knowledge.

  • 3D Point Cloud Pre-training: Supervised point-cloud methods make progress through specialized architectures and local operators but learn from closed-set datasets, limiting general 3D representation learning.Self-supervised learning from unlabeled point clouds offers transferable initialization for downstream fine-tuning.
  • Masked Autoencoders: MAE uses an asymmetric encoder-decoder transformer in which the encoder processes visible image patches and the decoder reconstructs masked RGB patches.This paradigm provides the masked autoencoding foundation later adapted to point clouds.
  • Masked Autoencoders: M2AE modifies the transformer hierarchically for multi-scale 3D learning, whereas I2P-MAE adds 2D-guided masking and 2D-semantic reconstruction.The proposed designs provide 2D pre-trained knowledge as guidance for point-cloud masked autoencoding.
  • 2D-to-3D Learning: Existing 2D-to-3D methods upgrade 2D networks into 3D variants or project point clouds into images with parameter-efficient tuning; I2P-MAE follows a distinct transfer approach.The related methods include kernel inflation, modality-agnostic transformers, multi-view adapters, and point-to-pixel prompting.

3. Method

I2P-MAE transfers knowledge from off-the-shelf 2D models into point-cloud masked autoencoding through multi-view representations, saliency-guided masking, and semantic reconstruction. Its hierarchical encoder-decoder reconstructs masked 3D coordinates while learning 2D semantics from visible tokens.

  • 3.1. Basic 3D Architecture: I2P-MAE uses a token embedding module, hierarchical encoder-decoder transformer, and reconstruction heads for masked 3D coordinates.Point tokens are formed by FPS downsampling, k-NN grouping, and mini-PointNet feature aggregation; the encoder processes only visible tokens before the decoder reconstructs masked regions.
  • 3.2. 2D Pre-trained Representations: The method projects point clouds onto three orthogonal depth-map views and applies pretrained ResNet or ViT models to obtain multi-view visual features and saliency maps.Projection omits one coordinate per view, uses the omitted coordinate as pixel depth, and avoids offline rendering and projective transformation.
  • 3.3. Image-to-Point Learning Schemes: 2D-semantic reconstruction uses visible point coordinates to aggregate corresponding multi-view 2D features and reconstruct them after decoding.The design jointly reconstructs masked 3D coordinates and visible 2D concepts, combining low-level spatial patterns with high-level pretrained semantics.
  • 3.3. Image-to-Point Learning Schemes: 2D-guided masking back-projects multi-view saliency maps into a 3D saliency cloud and preferentially preserves tokens covering semantically significant structures.Softmax-normalized saliency scores define nonuniform visibility probabilities, helping the encoder focus on important geometries and providing informative cues for reconstruction.
  • 3.3. Image-to-Point Learning Schemes: The image-to-point operation back-projects multi-view 2D representations into 3D space using 3D point coordinates for aggregation.This operation supplies the spatial correspondence used by both saliency-guided masking and semantic reconstruction.

4. Experiments

Experiments evaluate I2P-MAE through transfer, fine-tuning, ablations, and limited-data pre-training. Results show strong classification performance and benefits from both image-to-point components.

  • Image-to-Point Pre-training: 93.4% linear SVM accuracy on ModelNet40 and 87.1% on ScanObjectNN were achieved without fine-tuning, exceeding the second-best results by +0.5% and +3.0%.The encoder features also converged faster during pre-training than Point-MAE and Point-M2AE.
  • Downstream Classification: 90% accuracy was reached on ScanObjectNN’s hardest PB-T50-RS split, with gains of +2.93%, +2.76%, and +3.68% across its three splits.The comparison uses the official three ScanObjectNN splits.
  • Ablation Study: +0.4% and +0.9% classification accuracy came from 2D-guided masking over random masking on the two downstream datasets.Masking the most important tokens instead harmed SVM results by -0.9% and -3.3%.
  • Ablation Study: +0.5% and +2.4% classification accuracy resulted from reconstructing 2D semantics from visible tokens.The best configuration reconstructs 2D targets from visible tokens and 3D coordinates from masked tokens.
  • Limited 3D Data: With 60% of ShapeNet pre-training data, I2P-MAE reached 93.1% on ModelNet40, exceeding fully trained Point-MAE at 91.0% and Point-M2AE at 92.9%.At 20% and 60% data, it also outperformed Point-M2AE by +1.3% and +1.0%, respectively.
  • Effectiveness of Pre-training: Pure 3D pre-training improved ScanObjectNN accuracy by +1.09%, while 2D-to-3D knowledge transfer added a further +2.59%.Similar improvements were observed on other downstream datasets.

5. Visualization

Visualization compares random masking with I2P-MAE’s spatially guided masking. The guided strategy preserves semantically important parts of the input 3D shapes.

  • Visualization: 2D semantic scores guide masking to preserve significant 3D geometries, including airplane bodies, guitar grips, chair frames, and headphone structures.Darker points indicate higher semantic scores in the spatial saliency cloud.

6. Conclusion

I2P-MAE transfers 2D knowledge into 3D masked point modeling through guided masking and semantic reconstruction. The framework targets stronger downstream representations with less reliance on large 3D datasets.

  • Conclusion: I2P-MAE combines 2D-guided masking with 2D-semantic reconstruction to transfer learned 2D knowledge into 3D domains.The framework is designed for self-supervised masked point modeling.
  • Conclusion: The paper identifies future directions including point token sampling, 2D-3D class-token contrast, 3D object detection, and visual grounding.These are proposed extensions beyond masking and reconstruction.

7. Appendix

Appendix experiments examine training settings, convergence, view aggregation, few-shot transfer, and fine-tuning feature pooling. Multiple design choices affect performance across downstream tasks.

  • Implementation: The appendix specifies 300-epoch fine-tuning with AdamW and cosine scheduling for shape classification and part segmentation.Classification uses batch size 32; segmentation uses batch size 16.
  • Learning Curves: Image-to-point pre-training accelerated fine-tuning convergence and improved final classification accuracy over training I2P-MAE from scratch.The comparison covers ModelNet40 and ScanObjectNN learning curves.
  • Image-to-Point Settings: Using fewer than three projected views harmed pre-training performance because occlusion limited the 2D models’ view of complete 3D shapes.The default uses three views along the x, y, and z axes.
  • Image-to-Point Settings: Averaging multi-view 2D saliency scores produced the best performance among average, maximum, and minimum aggregation.The same section also studies aggregation for 2D-semantic targets.
  • Fine-tuning Settings: Adding max and average pooled features performed best on ModelNet40, while concatenating them performed best on ScanObjectNN.The appendix compares Max Only, Ave Only, Add, and Concat feature summaries.
  • Few-shot Classification: +0.5% classification accuracy over Point-M2AE was obtained on the 10-way 20-shot ModelNet40 split.I2P-MAE remained competitive across the evaluated few-shot splits.
Loading 2212.06785v1…