PaperlayerView exact revision on arXiv

Source-linked AI summary

Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?

Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, Kaisheng Ma

arXiv:2212.08320v2cs.CV

TL;DR

3D representation learning is constrained by limited labeled data and structural differences from images and language. ACT transfers pretrained image or language Transformers into 3D teachers through self-supervised autoencoding, then distills their latent features into masked point modeling. The resulting 3D learner shows strong generalization across downstream benchmarks, including ScanObjectNN.

  • Problem

    3D data are more difficult and expensive to collect and label than images or language, while point-cloud structure differs from those modalities.

  • Method

    ACT tunes pretrained image or language Transformers as 3D autoencoding teachers and uses their latent features as masked point-modeling targets.

  • Results

    ACT achieves strong generalization across downstream 3D tasks, including +10.4% average accuracy over a Transformer-from-scratch baseline across three ScanObjectNN variants.

  • Takeaways & Limitations

    Cross-modal knowledge transfer from pretrained Transformers can support self-supervised 3D representation learning through semantically enriched teacher targets.

Abstract

from arXiv · show

The success of deep learning heavily relies on large-scale data with comprehensive labels, which is more expensive and time-consuming to fetch in 3D compared to 2D images or natural languages. This promotes the potential of utilizing models pretrained with data more than 3D as teachers for cross-modal knowledge transferring. In this paper, we revisit masked modeling in a unified fashion of knowledge distillation, and we show that foundational Transformers pretrained with 2D images or natural languages can help self-supervised 3D representation learning through training Autoencoders as Cross-Modal Teachers (ACT). The pretrained Transformers are transferred as cross-modal 3D teachers using discrete variational autoencoding self-supervision, during which the Transformers are frozen with prompt tuning for better knowledge inheritance. The latent features encoded by the 3D teachers are used as the target of masked point modeling, wherein the dark knowledge is distilled to the 3D Transformer students as foundational geometry understanding. Our ACT pretrained 3D learner achieves state-of-the-art generalization capacity across various downstream benchmarks, e.g., 88.21% overall accuracy on ScanObjectNN. Codes have been released at https://github.com/RunpeiDong/ACT.

1 Xi’an Jiaotong University 2 Tsinghua University 3 MEGVII Technology‡

The listed affiliations include Xi’an Jiaotong University, Tsinghua University, and MEGVII Technology.

  • The authors are affiliated with Xi’an Jiaotong University, Tsinghua University, and MEGVII Technology.

1 INTRODUCTION

The introduction frames 3D representation learning as lagging behind 2D vision and NLP because of architectural, data, and pattern differences. ACT addresses these challenges by transferring knowledge from pretrained 2D or language Transformers through self-supervised 3D autoencoding and masked modeling.

  • 3D foundational visual computing lags behind 2D vision and NLP, motivating analysis of its distinctive challenges.
  • Data desert: 3D data are harder and more expensive to collect and label than images or free-form language, creating a data-scarcity problem.
  • Pattern difference: 3D point clouds are unstructured and sparse, while their irregular sampling differs from grid-based images, complicating tokenization and contrastive target construction.
  • ACT: ACT uses pretrained image or language Transformers as cross-modal teachers, tuning them as 3D autoencoders with prompt tuning to preserve pretrained knowledge.
  • ACT: The tuned Transformers provide semantically enriched masked-modeling targets for 3D point clouds without requiring image, language, or 3D downstream annotations during transfer.

2 RELATED WORKS

Prior work on 3D self-supervised learning includes reconstruction-based geometry tasks and cross-modal methods that use additional modalities, especially through contrastive feature alignment.

  • Classical 3D self-supervised methods use reconstruction-oriented pretexts such as part reordering, orientation estimation, reconstruction, flow consistency, deformation, and occlusion.
  • Large-scale datasets in images and NLP provide substantially more labels or words than typical 3D data, motivating cross-modal transfer.
  • Cross-modal 3D representation learning leverages image or language signals beyond point clouds, commonly through contrastive global-feature matching.

3 PRELIMINARIES

The preliminaries describe a Transformer-based point-cloud representation and masked modeling as teacher-guided reconstruction. ACT applies this framework by using pretrained cross-modal Transformers and their latent features as learning targets.

  • 3D point-cloud representations: ACT uses Transformer blocks to process local point-cloud patch embeddings, with point-feature extraction and learnable positional embeddings.
  • 3D point-cloud representations: The point-cloud representation prepends a learnable global [CLS] embedding and adds positional embeddings before Transformer processing.
  • ACT overview: Figure 1 summarizes ACT’s two-stage pipeline: prompt-tuned 3D autoencoding followed by teacher-targeted masked point modeling.
  • Masked modeling: Masked modeling corrupts token sequences with learned mask embeddings, then trains a student to predict teacher outputs and distill dark knowledge.
  • Masked modeling: ACT uses pretrained 2D or language Transformers as knowledgeable teachers for masked 3D modeling.

4 ACT: AUTOENCODERS AS CROSS-MODAL TEACHERS

ACT uses pretrained image or language Transformers as cross-modal 3D teachers through two-stage autoencoding and masked point modeling. Prompts adapt frozen Transformer blocks to 3D, while distilled latent features train a 3D Transformer student.

  • Overview: ACT addresses cross-modal structural differences by using a two-stage procedure that first adapts pretrained Transformers to 3D and then distills their representations.Stage I performs self-supervised prompt tuning for 3D autoencoding; Stage II uses the resulting autoencoder as a teacher for masked modeling.
  • Cross-Modal Embedding with Prompts: A DGCNN-style patch encoder produces point-cloud tokens that are processed by Transformer blocks with 3D positional embeddings.The Transformer backbone provides a modality-unified learner, while local geometric features are encoded before Transformer processing.
  • Cross-Modal Embedding with Prompts: Prompt tuning inserts learnable prompt embeddings into each layer while keeping the pretrained Transformer blocks frozen.This parameter-efficient strategy tunes the foundational Transformer while preserving pretrained knowledge as much as possible.
  • Point Cloud Autoencoding: The adapted Transformer becomes a 3D discrete variational autoencoder that reconstructs point clouds and produces latent features for teaching.A post-processing DGCNN extracts local geometric features, and FoldingNet reconstructs the input point cloud under an ELBO objective.
  • Masked Point Modeling as Cross-Modal Knowledge Distillation: Masked point modeling distills latent features from the pretrained 3D autoencoder teacher into a 3D Transformer student.The teacher combines the pretrained point-cloud encoder and post-processing components, while the student is a 3D Transformer trained through masked modeling.
  • Evaluation: Table 2 reports ScanObjectNN classification using overall accuracy, with separate results for no augmentation and simple point-cloud rotation.The table caption identifies these as Ours1 and Ours2 and distinguishes data augmentation used during fine-tuning.

5 EXPERIMENTS

ACT improves 3D transfer performance across object classification and scene segmentation, with gains over scratch, Point-MAE, and other self-supervised methods. Ablations show that masking, teacher choice, decoder depth, and tokenizer training strategy affect performance and reconstruction quality.

  • 3D real-world classification: +10.4% average accuracy: ACT improves over a scratch Transformer across three ScanObjectNN benchmarks under FULL tuning.With simple point-cloud rotation, the average improvement reaches +11.9%.
  • 3D real-world classification: +3.0% accuracy: ACT outperforms Point-MAE on ScanObjectNN PB_T50_RS, the most challenging benchmark.ACT reaches state-of-the-art performance among pure 3D Transformer methods on ScanObjectNN.
  • 3D scene segmentation: +2.5% mAcc and +1.2% mIoU: ACT improves over the scratch baseline on S3DIS semantic segmentation.ACT also exceeds Point-MAE by +1.2% mAcc and +0.4% mIoU, using only xyz inputs while matching or exceeding some xyz+rgb architectures.
  • 3D synthetic classification: +2.5% accuracy: ACT improves over the scratch FULL baseline on full-data ModelNet40 classification and is comparable or better than other SSL methods across protocols.In few-shot settings, ACT improves over the scratch FULL baseline by +9.0%, +4.7%, +8.7%, and +6.2% across four settings.
  • Decoder depth: Decoder depth has limited impact, with a two-block decoder achieving the highest results; zero-depth masked modeling is inferior.The zero-depth configuration exposes the encoder to all tokens, including masked ones.
  • Masking strategy and teacher choice: 85.12±0.54% average accuracy: BERT-B works as a cross-modal teacher, while larger teachers consistently perform better.Higher random-masking ratios improve results, whereas block masking favors lower ratios; zero masking is inferior.
  • 3D autoencoder training: Pretrained image Transformers substantially improve 3D autoencoder reconstruction, while prompt tuning or freezing outperforms full tuning.The ablation compares tokenizer training strategies using F-Score and Chamfer-distance metrics.

6 DISCUSSIONS

The experiments show that ACT’s semantic latent features are better suited to masked point modeling than discrete tokens, and that ACT also supports auxiliary feature distillation. Positional embeddings further help the pretrained image Transformer encode 3D inputs.

  • 6.1 IS A STRONGER TOKENIZER ALL YOU NEED?: Using Point-BERT discrete tokens as masked modeling targets gives the worst performance, even with pretrained 2D image Transformers.The results indicate that discrete tokens are unsuitable for semantically sparse point clouds regardless of tokenizer strength.
  • 6.1 IS A STRONGER TOKENIZER ALL YOU NEED?: ACT significantly improves performance by providing semantically rich encoded features as masked point modeling targets.These features are better suited to masked point modeling than discrete tokens.
  • 6.2 CAN ACT BE USED AS AN AUXILIARY KNOWLEDGE DISTILLATION METHOD?: ACT improves Point-MAE by +0.87% accuracy on ScanObjectNN when used for auxiliary intermediate-feature distillation.This demonstrates that ACT can function as a scalable knowledge-distillation method.
  • 6.3 HOW DOES THE 2D IMAGE TRANSFORMER UNDERSTAND 3D INPUTS?: Without positional embeddings, pretrained ViT still learns transferable 3D features, reaching 84.21±0.45% accuracy.The study also reports that 2D and 3D coordinate embeddings help the image Transformer encode 3D inputs into semantic feature space.
  • 6.3 HOW DOES THE 2D IMAGE TRANSFORMER UNDERSTAND 3D INPUTS?: Using all 3D coordinates for positional embeddings lets the 2D image Transformer exploit additional coordinate information for better feature encoding.The positional embedding is learned to fit the frozen image Transformer and enable encoding of 3D inputs in pretrained 2D feature space.

7 CONCLUSIONS

The paper presents ACT as a self-supervised framework that transfers knowledge from pretrained foundational Transformers to 3D students through feature-distillation masked modeling. Its semantic-enriched teacher features support strong generalization across downstream 3D tasks.

  • 7 CONCLUSIONS: ACT performs masked modeling as feature distillation from pretrained foundational Transformers to 3D Transformer students.It transfers pretrained Transformers into cross-modal 3D teachers through self-supervised 3D autoencoding.
  • 7 CONCLUSIONS: ACT uses semantic-enriched latent features from tuned 3D autoencoders as masked modeling targets for 3D Transformer representation learning.The framework is described as a general self-supervised learning approach.
  • 7 CONCLUSIONS: ACT shows remarkable generalization performance across various downstream 3D tasks.The conclusion presents cross-modal transfer as a potential route toward foundational modeling beyond 3D data.

B.1 SELF-SUPERVISED PRETRAINING SETUP

The pretraining setup uses ShapeNetCore point clouds to train a 3D autoencoder and Transformer-based masked point model. The autoencoder combines local geometry encoding, a pretrained 2D image Transformer, reconstruction, and variational distribution objectives.

  • Data: ShapeNetCore supplies approximately 51K 3D CAD models from 55 object categories for pretraining.Each model is sampled into 1,024 points, grouped into 64 local patches of 32 points using farthest point sampling and KNN.
  • 3D Autoencoder: The 3D autoencoder embeds KNN geometry patches with DGCNN and feeds them into a pretrained 2D image Transformer such as ViT or DeiT.The paper uses ViT-B by default when no specific description is given.
  • 3D Autoencoder: The autoencoder is trained with coarse- and fine-grained point-cloud reconstruction against the ground-truth point cloud.The reconstruction objective uses L1-style Chamfer Distance.
  • 3D Autoencoder: The overall dVAE objective combines coarse reconstruction, fine reconstruction, and KL-divergence distribution alignment.A uniform prior is used for discrete variational autoencoder training.
  • Masked Point Modeling: Masked point modeling uses Transformer encoder and decoder architectures with a lightweight PointNet patch embedding module.The encoder has 12 blocks and the decoder has 2 blocks, both using embedding dimension 384.
  • Downstream Evaluation: ModelNet40, ScanObjectNN, ShapeNetPart, S3DIS, and ScanNetV2 provide downstream evaluation settings spanning classification, segmentation, and detection.The setup includes clean CAD objects, real-world indoor objects, part segmentation, semantic segmentation, and scene-level scans.

C ADDITIONAL EXPERIMENTS

Additional experiments show that ACT transfers effectively to 3D detection, cross-modal classification comparisons, and part segmentation. The reported gains extend across detection accuracy, ScanObjectNN overall accuracy, and segmentation IoU.

  • 3D Object Detection: ACT improves ScanNetV2 3D object detection over the from-scratch baseline by +1.7% AP25 and +4.2% AP50.ACT also outperforms MaskPoint by a clear margin among self-supervised methods.
  • Comparison to Supervised Cross-Modal 3D Representation Learning Methods: ACT achieves 88.21% OA on ScanObjectNN PB_T50_RS using only a 22.1M pure 3D Transformer.P2P reports 87.4%/89.3% with 42.7M/195.8M image models, respectively.
  • 3D Part Segmentation: ACT improves the from-scratch ShapeNetPart baseline by 1.2% Cls. mIoU and 1.0% Ins. mIoU.It achieves top or second IoU performance in up to 12 of the 16 categories.

D VISUALIZATION

ACT’s 3D autoencoder reconstructs detailed local geometry, while ShapeNet pretraining and downstream fine-tuning produce discriminative feature manifolds across ModelNet40 and ScanObjectNN.

  • Reconstruction Results: ACT reconstructs high-quality object details, particularly local geometric structure in thin shelves and armchairs.On simpler objects such as rectangular tables, both ACT and Point-BERT reconstruct well.
  • t-SNE: After ShapeNet pretraining, ACT already yields discriminative features on ModelNet because the domain gap is relatively minor.
  • t-SNE: Fine-tuning produces discriminative features on both ModelNet40 and ScanObjectNN PB_T50_RS.
Loading 2212.08320v2…