PaperlayerView exact revision on arXiv

Source-linked AI summary

Masked Discrimination for Self-Supervised Learning on Point Clouds

Haotian Liu, Mu Cai, Yong Jae Lee

arXiv:2203.11183v2cs.CV

TL;DR

Point-cloud masked autoencoding has not matched its success in language and images, partly because reconstruction is vulnerable to point sampling variance and positional leakage. MaskPoint uses occupancy-based binary classification for masked point-cloud pretraining, achieving state-of-the-art results across downstream tasks while reducing pretraining time.

  • Problem

    Masked autoencoding has not produced comparable benefits for point-cloud understanding, while point-cloud annotation is difficult and expensive.

  • Method

    MaskPoint masks local point groups and trains a Transformer to classify real and fake query points using occupancy values as binary labels.

  • Results

    MaskPoint achieves state-of-the-art performance across shape classification, segmentation, detection, and few-shot classification, while reducing pretraining time compared with the prior Transformer-based method.

  • Takeaways & Limitations

    Discriminative masked point-cloud pretraining supports varied downstream tasks with a simpler binary objective and strong performance.

  • Takeaways & Limitations

    Mixing two point clouds can introduce unnecessary noise and domain shifts during pretraining and harm downstream performance.

Abstract

from arXiv · show

Masked autoencoding has achieved great success for self-supervised learning in the image and language domains. However, mask based pretraining has yet to show benefits for point cloud understanding, likely due to standard backbones like PointNet being unable to properly handle the training versus testing distribution mismatch introduced by masking during training. In this paper, we bridge this gap by proposing a discriminative mask pretraining Transformer framework, MaskPoint}, for point clouds. Our key idea is to represent the point cloud as discrete occupancy values (1 if part of the point cloud; 0 if not), and perform simple binary classification between masked object points and sampled noise points as the proxy task. In this way, our approach is robust to the point sampling variance in point clouds, and facilitates learning rich representations. We evaluate our pretrained models across several downstream tasks, including 3D shape classification, segmentation, and real-word object detection, and demonstrate state-of-the-art results while achieving a significant pretraining speedup (e.g., 4.1x on ScanNet) compared to the prior state-of-the-art Transformer baseline. Code is available at https://github.com/haotian-liu/MaskPoint.

1 Introduction

MaskPoint addresses the limited success of masked pretraining on point clouds by replacing reconstruction with discriminative masked-point classification. The resulting Transformer pretraining framework is reported to achieve strong downstream performance with substantially lower pretraining cost.

  • Masked autoencoding performs strongly in language and vision but has not achieved comparable performance for point cloud understanding.Point cloud annotations are especially difficult and expensive to obtain, motivating effective self-supervised learning.
  • PointNet-style local aggregation can create a training–testing distribution mismatch when masking drastically changes local neighborhoods.This mismatch is proposed as the primary reason masked autoencoding has worked poorly for point clouds.
  • Predicting masked point coordinates is unsuitable because positional encodings can leak the correct xyz locations and encourage shortcut learning.The decoder could use positional information rather than learn meaningful point-cloud features.
  • MaskPoint feeds visible point groups to a Transformer encoder and trains a decoder to classify masked object queries versus random 3D-space queries.The decoder is discarded after pretraining, while the encoder is reused for downstream tasks.
  • 4.1× faster pretraining on ScanNet is reported relative to Point-BERT, whose dVAE and auxiliary techniques add computational overhead.MaskPoint also processes only unmasked points in its Transformer encoder.
  • MaskPoint reports state-of-the-art results across classification, segmentation, detection, and few-shot classification tasks.The paper also reports that a standard Transformer can outperform sophisticatedly designed point-cloud backbones.

2 Related Work

Prior point-cloud self-supervision includes reconstruction, geometric and orientation pretexts, contrastive learning, and masked modeling adapted from language and vision. MaskPoint differs from concurrent masked approaches through its discriminative point-classification objective, designed to address sampling variance.

  • Transformers have been applied to point clouds through specialized attention mechanisms and detection architectures, but standard Transformers have remained comparatively weak.The paper positions self-supervised masked point classification as a way to improve standard Transformer representations.
  • Point-cloud self-supervised methods include deformation reconstruction, geometric structure prediction, orientation estimation, and contrastive learning across views or transformations.
  • Masking-based self-supervision trains models to predict masked content from surrounding context, with Transformers favored for token representations and long-range dependencies.
  • BERT and masked image modeling methods establish masked-token or masked-patch prediction as successful Transformer-based pretraining strategies.BEiT represents image patches as discrete tokens before masking and reconstruction.
  • MaskPoint’s key novelty over contemporary point-cloud masked approaches is a discriminative point-classification objective that addresses sampling variance.

3 Approach

MaskPoint learns semantic point-cloud representations by inferring object occupancy from sparse visible groups rather than reconstructing every missing point. Its pipeline uses masked-point and random-space queries, with filtering to reduce ambiguous negatives and patch embedding to control attention cost.

  • Motivation: A 90% masked airplane example motivates learning semantic structure from sparse visible points instead of precisely reconstructing every missing point.The intended representation includes object identity and the approximate locations of semantic parts.
  • Motivation: Discriminative point classification is less ambiguous than reconstruction because the model judges whether sampled 3D points belong to the object.The task remains difficult because it requires inferring shape from partial observations.
  • Architecture: MaskPoint partitions each cloud into masked and unmasked groups, encodes unmasked tokens with self-attention, and uses the latent representation to classify real and fake queries.
  • Architecture: The pretext task is reported to be difficult and meaningful enough to learn rich semantic point-cloud representations.
  • Discarding Ambiguous Points: Fake queries near object surfaces can receive ambiguous fake labels and cause vanishing gradients early in training.The method removes fake queries within a dynamically selected distance threshold of object points.
  • Architecture: Patch embedding converts point clouds into 3D point patches because feeding every point to quadratic-cost self-attention is unacceptable.

3D Point Patchification.

The method converts point clouds into local patch tokens, processes visible patches with a Transformer encoder, and uses a query-based decoder objective to learn representations robust to sampling variance.

  • 3D Point Patchification.: Farthest point sampling selects patch centers, and k-nearest neighbors around each center form local 3D point patches.Each patch is encoded with PointNet into a feature embedding, producing tokens for the Transformer.
  • 3D Point Patchification.: The encoder input combines a class token with patch features augmented by MLP positional embeddings.Transformer blocks produce contextual feature embeddings for the point patches.
  • 3D Point Patchification.: During decoding, real queries from masked points and fake queries sampled from 3D space are compared with encoder outputs through cross attention.The decoder performs cross attention between queries and encoder keys and values, without attention between different queries.
  • 3D Point Patchification.: A binary classification head distinguishes real from fake query points, using binary focal loss to address positive-negative imbalance.The pretrained patchification module and Transformer encoder initialize downstream-task models.
  • 3D Point Patchification.: Occupancy values replace Chamfer-distance reconstruction, reducing sensitivity to point sampling variance while training the model to recover point-cloud information.The point cloud is represented inside its tightest 3D bounding box with binary occupancy labels, and real and fake queries receive corresponding labels.

4 Experiments

Experiments evaluate MaskPoint across classification, segmentation, few-shot classification, detection, reconstruction, and ablations, showing strong downstream performance and design trends.

  • Object Classification: 2.4% OA improvement over training from scratch on ModelNet40, with gains over OcCo and Point-BERT pretraining.With 1k points, PointViT reaches 93.8% OA versus 93.7% for PointTransformer.
  • Object Classification: 7.4% OA improvement over training from scratch on ScanObjectNN’s hardest PB split, while achieving SOTA performance on all three splits.The method also gains 5.8% over OcCo and 1.5% over Point-BERT on PB.
  • Part Segmentation: 1.0%/0.9% gains in categorical/instance mIoU over PointViT and 0.3%/0.4% over Point-BERT on part segmentation.The dense discriminative objective supports scaling to dense prediction tasks.
  • Few-shot Classification: 7.2%/3.8%/4.6%/2.5% absolute gains over PointViT across four few-shot settings, with smaller gains over OcCo and Point-BERT.Each setting averages results over 10 sampled splits, and MaskPoint performs best in all settings.
  • Object Detection: +1.3 AP25 and +2.7 AP50 over the 3DETR train-from-scratch baseline on ScanNet detection.Increasing the encoder from 3 to 12 layers further adds +2.1 AP25 and +4.2 AP50 for MaskPoint; VoteNet-based results are not directly comparable.
  • Reconstruction and Ablations: 90% masking still allows reconstruction of the original point cloud’s overall shape without overfitting, while higher masking ratios generally improve downstream performance.Random masking is slightly better than block masking at higher ratios, and 256 decoder queries provide the reported balance between information and noise.

5 Conclusion

The paper presents MaskPoint, a discriminative masked point-cloud pretraining framework using occupancy values and reports strong downstream performance with reduced pretraining time. It also identifies learnable masking as a future direction.

  • MaskPoint uses occupancy values to form a simple binary pretraining objective for masked point clouds.The framework represents point-cloud occupancy and performs discriminative pretraining.
  • The framework supports downstream 3D shape classification, detection, and segmentation tasks.The conclusion reports strong performance across these task categories.
  • MaskPoint significantly reduces pretraining time compared with the prior Transformer-based state-of-the-art method.
  • Learning how to mask points instead of randomly masking local point groups is identified as an open direction.

ShapeNetPart

On ShapeNetPart, MaskPoint performs strongly on dense part segmentation, achieving the best reported class and instance mIoU with a PointViT backbone. The results exceed prior self-supervised and standard train-from-scratch backbones.

  • ShapeNetPart: 84.4% class mIoU and 86.0% instance mIoU are achieved with a PointViT backbone on ShapeNetPart.These are the highest class and instance mIoU values reported in the passage.
  • ShapeNetPart: MaskPoint outperforms prior self-supervised approaches including OcCo and Point-BERT on part segmentation.
  • ShapeNetPart: The method also outperforms train-from-scratch PointNet++ and DGCNN backbones.
  • ShapeNetPart: Dense discriminative pretraining supports good performance on dense prediction tasks such as part segmentation.

ScanNet

The ScanNet experiments visualize MaskPoint’s dense occupancy predictions in 3D space. Varying the threshold reveals a continuous occupancy probability distribution despite training on discrete sampled-point labels.

  • ScanNet: The occupancy visualizations include reconstruction results and closer views of the predicted distribution.
  • ScanNet: MaskPoint densely predicts occupancy probabilities throughout 3D space for ScanNet scenes.
  • ScanNet: Varying threshold τ displays points whose predicted occupancy probability exceeds the selected threshold.
  • ScanNet: The model outputs a continuous occupancy probability distribution despite training with discrete occupancy values from sampled points.

More reconstruction visualizations

The reconstruction visualizations show that MaskPoint models occupancy probabilities beyond the observed masked points. Its predictions reflect plausible higher and lower occupancy regions, while feature embeddings are compared across training settings.

  • More reconstruction visualizations: These predictions are consistent with the original point cloud’s regional occupancy pattern.
  • More reconstruction visualizations: The reconstructed probability distribution assigns low occupancy to the red region and high occupancy to the purple region.
  • More reconstruction visualizations: The model infers likely points in unseen regions using partial desk-top and desk-leg structure.
  • More reconstruction visualizations: The visualizations suggest that pretraining models a continuous occupancy probability distribution and captures scene structure.
  • More reconstruction visualizations: The t-SNE analysis compares encoder features from scratch training, ShapeNet pretraining, and ModelNet40 finetuning.

t-SNE visualizations

MaskPoint pretraining produces more distinguishable feature representations than training from scratch, and its qualitative ScanNetV2 detections localize annotated objects while also identifying an unannotated book.

  • Feature separability: MaskPoint pretraining makes features from different categories more distinguishable than training ModelNet40 from scratch.After finetuning on ModelNet40, projected features from different classes become clearly separable.
  • Feature separability: The resulting feature layout indicates a more compact and disjoint decision boundary.The passage connects this layout to applications such as mixup and uncertainty estimation.
  • 3D object detection: ScanNetV2 visualizations show green ground-truth and red predicted bounding boxes for qualitative 3D object detection.The model precisely localizes objects in examples including Fig. 7b and Fig. 7j.
  • 3D object detection: The detector also produces a bounding box for an object that was not annotated, such as a book in Fig. 7c.The passage presents this as evidence that the pretrained model can discover unannotated objects in the visualization.

B Additional Implementation Details

The implementation uses a PointViT Transformer backbone, task-specific heads, and multi-scale feature processing for classification, segmentation, and 3D object detection. Detailed settings specify the module dimensions, sampling resolutions, losses, and finetuning configurations across datasets.

  • Transformer backbone: PointViT is a 12-block Transformer whose blocks combine multi-head self-attention, feed-forward networks, LayerNorm, and MLP-based positional embeddings.The backbone uses hidden dimension 384 and six self-attention heads; the feed-forward expansion rate is specified in the detailed configuration.
  • Pretraining head: The pretraining point-classification head is a two-layer GELU MLP trained with binary focal loss.Binary focal loss balances information from positive and negative samples.
  • Dataset-specific settings: For ScanNet-Medium pretraining, the encoder uses three Transformer blocks with hidden dimension 256, four attention heads, and dropout rate 0.1.The feed-forward hidden dimension is 128, while LayerNorm and MLP positional embeddings remain enabled.
  • Classification: Classification concatenates the Transformer CLS token with a max-pooled feature before applying a three-layer MLP head.For ScanObjectNN, 10-percentile clipping is used to stabilize training.
  • Segmentation: Segmentation converts encoder outputs into a multi-scale feature pyramid, propagates features across scales, and applies a pointwise MLP at the highest resolution.Features are taken from Transformer layers 4, 8, and 12, with target cardinalities 2048, 512, 256, and 128.
Loading 2203.11183v2…