Source-linked AI summary
Masked Autoencoders for Point Cloud Self-supervised Learning
Yatian Pang, Wenxiao Wang, Francis E. H. Tay, Wei Liu, Yonghong Tian, Li Yuan
TL;DR
Point-cloud self-supervised learning needs an effective masked-autoencoding design that avoids location-information leakage and accommodates uneven information density. Point-MAE masks irregular point patches and reconstructs them with an asymmetric standard-Transformer autoencoder using shifted mask tokens. It generalizes across downstream tasks, achieving strong classification, few-shot, and segmentation results while surpassing dedicated supervised Transformer models.
Problem
Point-cloud masked autoencoding must address positional-information leakage, uneven information density, and the lack of a unified standard-Transformer architecture.
Method
Point-MAE randomly masks irregular point patches at a high ratio and reconstructs them with an asymmetric standard-Transformer autoencoder whose lightweight decoder receives shifted mask tokens.
Results
Point-MAE generalizes across object classification, few-shot learning, and part segmentation, outperforming other self-supervised methods and surpassing dedicated supervised Transformer models.
Takeaways & Limitations
The results support applying masked-autoencoding and unified architectures developed for languages and images to point-cloud representation learning.
Abstract
from arXiv · showhide
As a promising scheme of self-supervised learning, masked autoencoding has significantly advanced natural language processing and computer vision. Inspired by this, we propose a neat scheme of masked autoencoders for point cloud self-supervised learning, addressing the challenges posed by point cloud's properties, including leakage of location information and uneven information density. Concretely, we divide the input point cloud into irregular point patches and randomly mask them at a high ratio. Then, a standard Transformer based autoencoder, with an asymmetric design and a shifting mask tokens operation, learns high-level latent features from unmasked point patches, aiming to reconstruct the masked point patches. Extensive experiments show that our approach is efficient during pre-training and generalizes well on various downstream tasks. Specifically, our pre-trained models achieve 85.18% accuracy on ScanObjectNN and 94.04% accuracy on ModelNet40, outperforming all the other self-supervised learning methods. We show with our scheme, a simple architecture entirely based on standard Transformers can surpass dedicated Transformer models from supervised learning. Our approach also advances state-of-the-art accuracies by 1.5%-2.3% in the few-shot object classification. Furthermore, our work inspires the feasibility of applying unified architectures from languages and images to the point cloud.
1 Introduction
Point-MAE applies masked autoencoding to point clouds by masking irregular patches and reconstructing them with a Transformer autoencoder designed to avoid location leakage and handle uneven information density. It generalizes across downstream tasks, including classification, few-shot learning, and segmentation.
- Motivation: Masked autoencoding reconstructs masked content from unmasked inputs, encouraging encoders to learn high-level latent features.This scheme has achieved strong results in language and vision with Transformer backbones.
- Motivation: Point clouds share local-to-global structure with language and images, making masked autoencoding applicable after embedding point subsets into tokens.The approach is also motivated by the relatively small size of point-cloud datasets and Transformers’ large data demand.
- Challenges: Point-cloud pre-training faces three issues: limited unified Transformer architectures, positional-embedding leakage from mask tokens, and uneven information density.High-information regions such as sharp edges are harder to reconstruct than relatively flat surfaces.
- Point-MAE: 60%-80% random masking works well despite uneven point-cloud information density, indicating image-like rather than language-like density behavior.The result contrasts with the difficulty of reconstructing key local features observed at 60% masking.
- Point-MAE: Point-MAE divides point clouds into irregular patches, randomly masks them at a high ratio, and reconstructs masked patches in coordinate space with an asymmetric standard-Transformer autoencoder.The encoder processes only visible patches, while a lightweight decoder receives encoded and shifted mask tokens; shifting reduces computation and avoids early location leakage.
- Results: 85.18% accuracy is achieved on ScanObjectNN and 94.04% on ModelNet40, while few-shot ModelNet40 accuracy improves by 1.5%-2.3% and part segmentation improves by 1% mean IoU.The model outperforms other self-supervised methods and can surpass dedicated supervised Transformer models.
2 Related Work
Related work frames Point-MAE within self-supervised learning, masked autoencoding, and Transformer-based representation learning. Prior point-cloud methods use diverse pretext tasks, while Point-MAE aims for a neater masked-autoencoding scheme.
- Self-supervised Learning: Self-supervised learning generates supervision from data rather than human annotations, commonly through a pretext task that predicts one part from another.This reduces reliance on manually labeled data.
- SSL for NLP and Image: NLP methods such as BERT mask input tokens and predict original vocabularies, while image methods have increasingly adopted masked reconstruction alongside contrastive learning.MAE is cited as an image method that masks input patches and reconstructs masked content.
- SSL for Point Cloud: Point-cloud self-supervised learning uses diverse tasks, including instance discrimination, occlusion recovery, implicit-feature reconstruction, and discrete-token prediction.Point-BERT uses a BERT-style masking strategy assisted by a discrete variational autoencoder.
- Masked Autoencoding: Autoencoders encode inputs into high-level latent features and decode them to reconstruct the input, typically optimizing reconstruction similarity.Masked autoencoders introduce input noise through masking, placing them within denoising autoencoders.
- Transformer Backbones: Transformers model global dependencies through self-attention, but point-cloud Transformer backbones remain less developed than those in NLP and computer vision.Prior point-cloud designs include dedicated embedding layers and modified self-attention mechanisms.
3 Point-MAE
Point-MAE uses irregular point patches, high-ratio random masking, and an asymmetric Transformer autoencoder to reconstruct masked patches. Its design embeds visible patches with a lightweight PointNet and shifts mask tokens to the decoder to reduce computation and avoid early location leakage.
- Point Patches Generation: Point-MAE divides unordered point clouds into irregular patches using Farthest Point Sampling and K-Nearest Neighborhood.Patches may overlap, and each point uses coordinates normalized relative to its patch center.
- Masking: Randomly masking 60%-80% of point patches preserves complete information within each separately masked patch.Masked patches provide the reconstruction ground truth.
- Embedding: A lightweight PointNet with MLPs and max pooling embeds visible patches, while masked patches are replaced by shared learnable mask tokens.The PointNet embedding follows permutation invariance, unlike a naive linear embedding.
- Encoder-decoder: The asymmetric backbone encodes only visible tokens, then a smaller decoder combines encoded tokens with mask tokens and positional embeddings.The decoder outputs only decoded mask tokens for prediction, while separate positional embeddings are used for encoder and decoder.
- Encoder-decoder: Shifting mask tokens to the decoder reduces encoder input under high masking ratios and avoids early leakage of location information.The computational saving follows from the quadratic complexity of Transformers.
- Reconstruction Target: A fully connected prediction head reshapes decoder outputs into masked patch coordinates, with reconstruction trained using l_2 Chamfer Distance.Predicted and ground-truth masked patches have shape mn × k × 3.
4 Experiments
Point-MAE is pre-trained on ShapeNet and evaluated across reconstruction, object classification, few-shot learning, and part segmentation. The experiments show strong generalization, efficient pre-training, and sensitivity to masking design and mask-token placement.
- Experiments: Point-MAE is pre-trained on ShapeNet and evaluated on object classification, few-shot learning, part segmentation, and masking-strategy studies.The experiments also examine shifting mask tokens and use a 12-block encoder with a 4-block decoder.
- Pre-training Setup: 1.7× faster pre-training than Point-BERT, while reconstructions remain effective across masking ratios different from the 60% pre-training ratio.Figure 4 visualizes masked inputs and reconstructions on ShapeNet validation examples.
- Downstream Tasks: 85.18% accuracy on ScanObjectNN’s hardest PB-T50-RS setting, outperforming Point-BERT by 2.11% and improving the baseline by 7.94%.The model generalizes from clean ShapeNet pre-training to cluttered real-world indoor-scene objects without voting or test-time augmentation.
- Downstream Tasks: 94.04% accuracy on ModelNet40 with 8192 input points, while standard-Transformer Point-MAE surpasses other self-supervised and dedicated supervised Transformer models.With 1024 points, it achieves 93.8% accuracy, 2.4% above training from scratch and 0.6% above Point-BERT.
- Downstream Tasks: 1.5%-2.3% advances over state-of-the-art accuracy across four ModelNet40 few-shot settings, with smaller deviations.Results average 10 independent experiments for each n-way, m-shot setting.
- Downstream Tasks: 86.1% mIoU on ShapeNetPart, improving the baseline by 1% mIoU with a simple segmentation head.The head avoids propagation and DGCNN operations.
- Ablation Study: Random masking performs best at a 60%-80% ratio, whereas low ratios degrade performance and excessively high ratios cause a slight decline.Block masking can also work with a medium masking ratio despite making reconstruction harder.
- Ablation Study: Shifting mask tokens to the lightweight decoder improves fine-tuning accuracy to 93.19% versus 92.14% when mask tokens enter the encoder.Processing mask tokens in the encoder yields lower reconstruction loss but causes early location-information leakage.
5 Conclusions
The paper presents Point-MAE, a masked-autoencoder scheme for point-cloud self-supervised learning. It reports strong generalization across object classification, few-shot learning, and part segmentation, while using a standard Transformer architecture.
- Point-MAE is a novel masked-autoencoder scheme for point-cloud self-supervised learning.
- Point-MAE is verified across object classification, few-shot learning, and part segmentation tasks.
- Point-MAE outperforms all other self-supervised learning methods in the reported evaluations.
- A simple architecture entirely based on standard Transformers can surpass dedicated Transformer models from supervised learning.
- The approach supports the feasibility of applying unified architectures from languages and images to point clouds.