Source-linked AI summary
Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, Jiwen Lu
TL;DR
Point-cloud Transformers lack both sufficient annotated data and a pre-existing vocabulary for direct BERT-style pre-training. Point-BERT learns discrete geometric tokens with a dVAE and trains standard Transformers through masked point modeling, achieving strong classification and transfer performance. The authors report improved Transformer learning and generalization across several 3D tasks, while noting that pre-training plus fine-tuning is time-consuming.
Problem
Point-cloud Transformers lack sufficient annotated 3D data and a pre-defined vocabulary needed to directly apply BERT-style pre-training.
Method
Point-BERT uses a dVAE-based tokenizer to convert local point-cloud patches into discrete tokens, then pre-trains standard Transformers by predicting masked tokens.
Results
93.8% accuracy on ModelNet40 and 83.1% accuracy on the complicated setting of ScanObjectNN are achieved by a pure Transformer with fewer human priors, alongside strong transfer to new tasks and domains.
Takeaways & Limitations
BERT-style pre-training can substantially improve standard Transformers’ learning and generalization for 3D point-cloud tasks.
Takeaways & Limitations
The full pre-training plus fine-tuning procedure is time-consuming, and improving training efficiency remains future work.
Abstract
from arXiv · showhide
We present Point-BERT, a new paradigm for learning Transformers to generalize the concept of BERT to 3D point cloud. Inspired by BERT, we devise a Masked Point Modeling (MPM) task to pre-train point cloud Transformers. Specifically, we first divide a point cloud into several local point patches, and a point cloud Tokenizer with a discrete Variational AutoEncoder (dVAE) is designed to generate discrete point tokens containing meaningful local information. Then, we randomly mask out some patches of input point clouds and feed them into the backbone Transformers. The pre-training objective is to recover the original point tokens at the masked locations under the supervision of point tokens obtained by the Tokenizer. Extensive experiments demonstrate that the proposed BERT-style pre-training strategy significantly improves the performance of standard point cloud Transformers. Equipped with our pre-training strategy, we show that a pure Transformer architecture attains 93.8% accuracy on ModelNet40 and 83.1% accuracy on the hardest setting of ScanObjectNN, surpassing carefully designed point cloud models with much fewer hand-made designs. We also demonstrate that the representations learned by Point-BERT transfer well to new tasks and domains, where our models largely advance the state-of-the-art of few-shot point cloud classification task. The code and pre-trained models are available at https://github.com/lulutang0608/Point-BERT
1. Introduction
Point-BERT addresses limited labeled 3D data and the lack of a point-cloud vocabulary by combining discrete point-tokenization with BERT-style masked pre-training. Its standard Transformer architecture achieves strong classification and transfer results while using fewer hand-made priors.
- Standard point-cloud Transformers remain understudied and existing models introduce inductive biases through local feature aggregation and neighbor embedding.
- Limited annotated 3D data motivates self-supervised pre-training for scalable and generalizable point-cloud representation learning.
- Point-BERT uses local geometric patches as units and proposes a vocabulary of discrete tokens representing their geometric patterns.
- A dVAE-based tokenizer converts point clouds into discrete point tokens, enabling Transformers to perform masked point modeling.
- Point-BERT correctly predicts masked tokens and produces holistic reconstructions on synthetic ShapeNet data and unseen real ScanObjectNN scans.
- 93.8% accuracy on ModelNet40 and 83.1% accuracy on the complicated setting of ScanObjectNN are achieved by a pure Transformer with fewer human priors.
2. Related Work
Prior work established Transformers and BERT-style masking in language, vision, and point-cloud self-supervised learning, but point clouds remain difficult to tokenize for direct BERT-style pre-training.
- Self-supervised Learning (SSL): Self-supervised point-cloud methods use pretext tasks including orientation estimation, deformation reconstruction, geometric cues, spatial cues, contrastive learning, and occlusion reconstruction.
- Transformers: Transformers offer parallel computing, long-distance characteristics, and minimal inductive bias, but their point-cloud applications remain limited.
- BERT-style Pre-training: BERT pre-trains bidirectional representations from unlabeled text using masked language modeling that recovers randomly masked input tokens.
- BERT-style Pre-training: BEiT applies discrete visual tokenization and masked patch recovery, whereas point clouds are harder to tokenize because of their unstructured nature.
3. Point-BERT
Point-BERT extends BERT-style pre-training to 3D point clouds by tokenizing local patches, masking patch embeddings, and recovering discrete point tokens with auxiliary semantic learning.
- Point Tokenization: Point-BERT learns discrete point tokens with a dVAE-based Tokenizer so local geometric patterns can supervise Transformer pre-training.The Tokenizer converts point embeddings into discrete tokens and reconstructs corresponding sub-clouds.
- Transformer Backbone: Local point patches are centered, projected by a mini-PointNet, augmented with positional embeddings, and processed as a sequence by standard Transformers.Grouping points into patches avoids the computational cost of point-wise self-attention.
- Masked Point Modeling: Masked Point Modeling replaces selected patch embeddings with a shared mask embedding and trains the Transformer to recover tokens for the masked locations.The task uses informative dVAE tokens as surrogate supervision for missing geometric structures.
- Masked Point Modeling: Block-wise masking hides continuous local regions, making the model infer missing patches from the remaining point cloud context.The masking ratio controls the set of masked positions, while positional embeddings remain unchanged.
- Masked Point Modeling: Point Patch Mixing creates virtual samples by combining sub-cloud groups and requires prediction of tokens from the original sub-clouds.The auxiliary task increases pre-training difficulty without requiring alignment between patches.
- Masked Point Modeling: Contrastive learning complements MPM by encouraging high-level semantic representations alongside local geometric structure understanding.The combined objectives target both local geometric patterns and high-level semantic patterns.
4. Experiments
Experiments evaluate Point-BERT across pre-training, classification, few-shot learning, transfer to real-world data, part segmentation, and ablations. The results show consistent gains from masked point modeling and strong transfer across tasks and domains.
- Pre-training Setups: ShapeNet provides over 50,000 models for pre-training, with each model represented by 1,024 points divided into 64 patches.Each patch contains 32 points and is projected into point embeddings for the dVAE and Transformer.
- Pre-training Setups: During MPM pre-training, 25%–45% of point embeddings are masked while the Transformer predicts tokens from a fixed dVAE Tokenizer.The Transformer uses 12 layers, feature dimension 384, and six attention heads.
- Downstream Tasks: 93.8% accuracy is achieved on ModelNet40 with 8,192 input points, compared with 91.2% for the baseline and 92.2% for OcCo at 4,096 points.Point-BERT gains 1.8%/2.2% over training from scratch at 1,024/4,096 inputs.
- Downstream Tasks: Point-BERT achieves the best few-shot results, improving over the baseline by 6.8%, 3.0%, 6.4%, and 3.3% across four settings.It also improves over the OcCo-based method by 0.6%, 0.4%, 1.6%, and 0.3%.
- Downstream Tasks: Point-BERT improves vanilla Transformers by 7.57%, 7.57%, and 5.83% across the three ScanObjectNN variants.ScanObjectNN contains real-world scans with background and occlusions.
- Ablation Study: MPM improves performance by about 1.17%, while point patch mixing and MoCo each add a further 0.33% in the ablation study.Block-wise masking performs better than random masking at masking ratios of 25%–45%.
5. Conclusion and Discussions
Point-BERT introduces BERT-style pre-training for 3D point cloud Transformers and reports improved learning and generalization across several tasks. The authors note that the full pre-training and fine-tuning procedure remains time-consuming.
- Point-BERT uses BERT-style pre-training to learn low-level structural information and high-level semantic features.
- Comprehensive experiments show significant improvements in Transformer learning and generalization across several 3D point cloud tasks.
- The full pre-training and fine-tuning procedure is time-consuming, as with other Transformer pre-training methods.Improving training efficiency is identified as a future direction.
A. Discrete VAE
The discrete VAE uses a tokenizer and decoder for point-cloud reconstruction, optimizing reconstruction and distribution losses. Its training uses a discrete vocabulary with scheduled KL-divergence weighting and Gumbel-softmax temperature.
- The dVAE consists of a tokenizer and decoder, using DGCNN layers and FoldingNet for point-cloud reconstruction.The tokenizer and decoder each include four-layer DGCNN components; the decoder additionally uses FoldingNet.
- The dVAE jointly optimizes coarse- and fine-grained reconstruction using the ℓ1-form Chamfer Distance and KL-divergence losses.The KL-divergence regularizes predicted token distributions toward a uniform prior.
- The learnable vocabulary contains 8192 words represented by 256-dimensional vectors.
- The KL-divergence weight α increases from 0 to 0.1, while the Gumbel-softmax temperature decays from 1 to 0.0625 using cosine schedules.α is held at 0 initially before being increased over subsequent training.
B. Point-BERT
Point-BERT follows a standard Transformer backbone and adds task-specific heads for classification and segmentation. Its segmentation head upsamples sparse features and propagates them hierarchically to produce dense predictions.
- Point-BERT architecture: The backbone uses stacked standard Transformer blocks containing multi-head self-attention, FFNs, and LayerNorm.The experiments use 12 blocks, 6 attention heads per layer, and 384-dimensional features.
- Point-BERT architecture: Multi-head attention projects queries, keys, and values into multiple representation subspaces before concatenating head outputs.The final linear layer uses weights W_o.
- Classification head: The classification head combines the [CLS] feature with max-pooled node features before a two-layer MLP.
- Segmentation head: The segmentation head addresses single-resolution limitations through geometry-based feature upsampling and hierarchical feature propagation.
- Segmentation head: Geometry-based upsampling estimates dense-point features from nearby sparse-point features using distance-based weighted addition.Features from shallow and deep Transformer layers are mapped to different resolutions.
- Segmentation head: Hierarchical propagation updates features from coarse to fine resolutions with a lightweight DGCNN to obtain the final dense feature map.