Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training

Renrui Zhang; Ziyu Guo; Rongyao Fang; Bin Zhao; Dong Wang; Yu Qiao; Hongsheng Li; Peng Gao

Source-linked AI summary

Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training

Renrui Zhang, Ziyu Guo, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, Hongsheng Li, Peng Gao

arXiv:2205.14401v2May 28, 2022cs.CVcs.AI

TL;DR

Masked autoencoding had not been widely adopted for learning representations of irregular 3D point clouds. Point-M2AE addresses this gap with hierarchical multi-scale encoding, masking, and global-to-local reconstruction. It reports strong transfer results, including 92.9% frozen-encoder linear-SVM accuracy on ModelNet40 and gains across downstream tasks.

Problem
The central gap is adapting MAE-style masked autoencoding from grid-based images to irregular point clouds for 3D representation learning.
Method
Point-M2AE uses hierarchical multi-scale encoder-decoder architectures, consistent masking across scales, and global-to-local reconstruction for point-cloud pre-training.
Results
92.9% linear-SVM classification accuracy on ModelNet40 is achieved with a frozen encoder after ShapeNet pre-training, surpassing CrossPoint by +1.2%.
Takeaways & Limitations
The hierarchical pre-training scheme improves transfer across classification, few-shot classification, part segmentation, and 3D object detection tasks.

Abstract

from arXiv · show

Masked Autoencoders (MAE) have shown great potentials in self-supervised pre-training for language and 2D image transformers. However, it still remains an open question on how to exploit masked autoencoding for learning 3D representations of irregular point clouds. In this paper, we propose Point-M2AE, a strong Multi-scale MAE pre-training framework for hierarchical self-supervised learning of 3D point clouds. Unlike the standard transformer in MAE, we modify the encoder and decoder into pyramid architectures to progressively model spatial geometries and capture both fine-grained and high-level semantics of 3D shapes. For the encoder that downsamples point tokens by stages, we design a multi-scale masking strategy to generate consistent visible regions across scales, and adopt a local spatial self-attention mechanism during fine-tuning to focus on neighboring patterns. By multi-scale token propagation, the lightweight decoder gradually upsamples point tokens with complementary skip connections from the encoder, which further promotes the reconstruction from a global-to-local perspective. Extensive experiments demonstrate the state-of-the-art performance of Point-M2AE for 3D representation learning. With a frozen encoder after pre-training, Point-M2AE achieves 92.9% accuracy for linear SVM on ModelNet40, even surpassing some fully trained methods. By fine-tuning on downstream tasks, Point-M2AE achieves 86.43% accuracy on ScanObjectNN, +3.36% to the second-best, and largely benefits the few-shot classification, part segmentation and 3D object detection with the hierarchical pre-training scheme. Code is available at https://github.com/ZrrSkywalker/Point-M2AE.

1 Introduction

Point-M2AE adapts masked autoencoding to irregular point clouds through hierarchical, multi-scale encoding and reconstruction. It reports strong transfer performance across classification, segmentation, few-shot learning, and detection tasks.

Motivation: MAE-style masked autoencoding is extended from grid-based images to irregular point clouds as a 3D representation-learning framework.The paper asks whether masked autoencoding can serve as a powerful learner for irregular 3D point clouds.
Method: Point-M2AE uses hierarchical transformer encoders and decoders to learn multi-scale spatial geometries and local-global 3D shape relations.The architecture progressively encodes multi-scale features rather than directly reducing points to a single low-resolution representation.
Results: 92.9% classification accuracy is achieved for linear SVM on ModelNet40 with a frozen encoder, surpassing CrossPoint by +1.2%.The encoder is pre-trained on ShapeNet before evaluation.
Results: 86.43% (+3.36%) accuracy is achieved on ScanObjectNN, alongside gains in ModelNet40 classification, ShapeNetPart segmentation, and few-shot classification.The reported downstream results include 94.0% (+0.8%) on ModelNet40, 86.51% (+0.91%) instance mIoU on ShapeNetPart, and 95.0% (+2.7%) on 10-way 20-shot ModelNet40.
Results: Point-M2AE also improves ScanNetV2 detection by +1.3% AP25 and +1.3% AP50.The paper attributes this benefit to providing the detection backbone with hierarchical point-cloud understanding.
Method: Point-M2AE combines a U-Net-like transformer architecture with multi-scale masking that maintains consistent visible regions across scales.These design choices support hierarchical point-cloud pre-training.

2 Related Work

Prior masked-modeling methods established self-supervised pre-training in language and images, while point-cloud representation learning used reconstruction and contrastive pretexts. Masked autoencoding had not yet been widely adopted for 3D point clouds.

Pre-training by Masked Modeling: Masked modeling predicts masked input parts and has driven downstream gains in language and image pre-training.GPT and BERT apply masked modeling to language, while BEiT and MAE apply related objectives to images.
Pre-training by Masked Modeling: MAE reconstructs raw pixel values from masked image tokens using a standard vision transformer and a high mask ratio.Follow-up methods modify the encoder, learning objective, or reconstruction targets.
Pre-training by Masked Modeling: Masked autoencoding was not widely adopted for self-supervised pre-training of 3D point clouds.Point-BERT used discrete tokens for masked point modeling but relied heavily on contrastive learning, augmentation, and costly components.
Self-supervised Learning for Point Clouds: Point-cloud self-supervised learning commonly reconstructs transformed inputs using pretexts based on rotation, deformation, rearranged parts, or occlusion.Other approaches use contrastive learning between features of corresponding points from different views.

3 Method

Point-M2AE uses a hierarchical U-Net-like transformer with multi-scale masking, staged token processing, and a lightweight decoder to learn point-cloud geometries from global and local perspectives.

3.1 Multi-scale Masking: Point-M2AE represents point clouds across S scales using FPS downsampling and k-NN grouping, recording inclusion relations between scales.The number of points decreases progressively across scales.
3.2 Hierarchical Encoder: The encoder uses stages with self-attention, FFNs, and spatial token merging to downsample visible tokens while enlarging receptive fields.Deeper stages use higher feature dimensions to encode richer spatial semantics.
3.1 Multi-scale Masking: Multi-scale masking samples masks at the highest scale and recursively back-projects visible positions to earlier scales for consistent regions.Visible positions at each earlier scale are obtained from the k-nearest neighbors of visible positions at the next scale.
3.2 Hierarchical Encoder: During fine-tuning, multi-scale ball-query radii expand across stages, enabling local-to-global feature aggregation.The local spatial attention focuses shallower stages on nearby patterns while progressively enlarging attention scopes.
3.3 Hierarchical Decoder: The lightweight decoder progressively upsamples tokens by weighted interpolation, uses encoder skip connections, and preserves global attention for reconstruction.Decoder stages correspond to encoder scales and use complementary visible-token features; reconstruction predicts masked coordinates with an l2 Chamfer Distance loss.
3.3 Hierarchical Decoder: Point-M2AE reconstructs masked coordinates from higher-scale representations rather than raw points, using a challenging first-scale pretext task.The design avoids extra spatial noise and computational overhead from reconstructing masked raw points.

4 Experiments

Experiments evaluate Point-M2AE through linear evaluation, fine-tuning, few-shot classification, object detection, and ablations. Results show strong transfer performance and benefits from hierarchical modules, multi-scale masking, and pre-training.

Linear SVM: 92.9% accuracy on ModelNet40 with a frozen encoder demonstrates strong representation capacity in linear SVM evaluation.This result surpasses the second-best CrossPoint by +1.7% and exceeds Point-BERT by +5.5%.
Part Segmentation: 86.51% instance mIoU on ShapeNetPart establishes strong part-segmentation performance with a simple segmentation head.The result surpasses the second-best Point-BERT by +0.91%.
Few-shot Classification: Point-M2AE achieves the best performance across four few-shot ModelNet40 settings, surpassing Point-BERT by +2.2%, +2.0%, +1.3%, and +2.7%.The method also shows smaller deviations than other transformer-based methods.
3D Object Detection: Hierarchical pre-training boosts 3DETR-m on ScanNetV2 by +1.34% AP25 and +1.29% AP50 over training from scratch.The experiment evaluates Point-M2AE as a feature backbone for indoor 3D object detection.
Hierarchical Modules: Removing hierarchical encoder, decoder, or skip connections hurts accuracy, while skip connections provide complementary information to the decoder.The hierarchical encoder has a larger effect than the decoder in the reported ablation.
Masking Strategy: Random masks at each scale harm performance by -4.5% accuracy, whereas an 80% mask ratio provides the best pre-training setting.Multi-scale masking keeps visible regions consistent across scales and avoids fragmented regions.
With and Without Pre-training: Pre-training boosts downstream performance by +1.5%, +2.5%, +3.8%, and +1.1% across four datasets.The comparison uses randomly initialized networks trained from scratch as the no-pre-training condition.

5 Visualization

Visualizations illustrate how Point-M2AE represents point clouds at multiple scales and preserves fine-grained geometric information. Its masking strategy produces block-wise visible regions that remain consistent across scales.

Multi-scale Masking: Multi-scale representations capture different levels of geometric detail and semantics in point clouds.Figure 5 presents inputs, multi-scale representations, reconstructed coordinates, and masked point clouds for each row.
Multi-scale Masking: Consistent visible positions across scales produce block-wise masked regions that support hierarchical pre-training.The visualization contrasts multi-scale masking with fragmented regions that would expose different parts at different stages.
Fine-grained Information: Fine-grained structures such as thin branches, fingers, and plane engines are represented by the hierarchical architecture.The visualization compares multi-stage hierarchical and single-scale architectures through extracted features and reconstructed point clouds.

6 Conclusion

Point-M2AE is presented as a multi-scale masked autoencoder for hierarchical self-supervised pre-training on 3D point clouds. Experiments support its effectiveness as a 3D representation learner, while future work targets broader applications.

Conclusion: Point-M2AE encodes multi-scale point clouds hierarchically and reconstructs masked coordinates through global-to-local upsampling.The framework is designed to learn powerful 3D representations through self-supervised pre-training.
Conclusion: Extensive experiments demonstrate Point-M2AE’s superiority as a 3D representation learner.
Limitations and Future Work: Future work will apply Point-M2AE to wider 3D applications, including outdoor and open-world scene understanding.

Checklist

The checklist records affirmative responses about claim accuracy, limitations, reproducibility, training details, asset attribution, and ethics review. Items involving theoretical results and human-subject research are marked as applicable or affirmative as specified.

Claims and Scope: The authors affirm that the paper describes its contributions and scope and discusses the limitations of the work.
Ethics: The authors affirm discussion of potential negative societal impacts and conformity with ethics review guidelines.
Theoretical Results: The checklist marks assumptions and complete proofs as included for theoretical results.
Reproducibility: The authors affirm that code, data, reproduction instructions, and training details are provided.
Experimental Reporting: The checklist marks error bars and compute-resource reporting as included for experiments.
Assets: Existing assets are cited, while new assets are included through supplemental material or a URL.
Human Subjects: Human-subject and participant-risk items are marked not applicable.

7 Appendix

The appendix details Point-M2AE’s multi-scale masking, training and evaluation settings, and visualizes how hierarchical design choices affect learned features and downstream predictions.

Implementation Details: Point-M2AE is pre-trained on ShapeNet using 2,048 points per shape for 300 epochs with AdamW, cosine scheduling, and warm-up.The initial learning rate is 10^-4 and weight decay is 5×10^-2.
Shape Classification: Shape classification evaluates ModelNet40’s synthetic shapes and ScanObjectNN’s noisy real-world point clouds, including three ScanObjectNN evaluation splits.The PB-T50-RS split is identified as the most difficult for recognition.
Multi-scale Masking Pipeline: Multi-scale masking randomly masks points at the highest scale and back-projects visible positions into preceding scales.The multi-scale representation is first obtained with FPS and k-NN.
Learning Curves: Pre-training produces faster convergence and higher classification accuracy than training without pre-training on ModelNet40 and ScanObjectNN.The comparison uses both loss and accuracy curves.
Feature and Attention Visualization: Fine-tuning further clusters same-category features and separates different-category features, while local spatial attention restricts early-stage focus to neighboring structures.The local attention visualization contrasts spatially localized weights with attention scattered across the full shape.
Architecture Analysis: The multi-scale architecture predicts finer-grained part labels than the single-scale architecture, and the best explored configuration uses a 3-stage encoder with a 2-stage decoder.Using three decoder stages and reconstructing at the first scale adversely affects performance.