PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining

Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, Li Yi

arXiv:2302.02318v2cs.CV

TL;DR

3D representation learning must balance contrastive models’ limited-data over-fitting against generative models’ weaker scaling capacity. RECON combines both through ensemble distillation and a reconstruction-guided contrastive architecture, achieving state-of-the-art transfer results while remaining mainly evaluated in 3D and dependent on its proposed block.

  • Problem

    Contrastive pretraining over-fits limited 3D data, while generative pretraining has inferior data-scaling capacity, motivating a unified approach.

  • Method

    RECON uses ensemble distillation from generative and single- or cross-modal contrastive teachers, with an encoder-decoder RECON-block transferring reconstruction guidance through cross attention and stop-gradient.

  • Results

    RECON achieves a new state-of-the-art in 3D representation learning, including average accuracy improvements of +9.2% on ScanObjectNN and +2.9% on ModelNet40.

  • Takeaways & Limitations

    RECON learns high-capacity, generalizable 3D representations by combining generative reconstruction with contrastive modeling and multimodal data.

  • Takeaways & Limitations

    The paper mainly explores RECON for 3D representation learning, and its architecture-agnostic extension remains future work.

Abstract

from arXiv · show

Mainstream 3D representation learning approaches are built upon contrastive or generative modeling pretext tasks, where great improvements in performance on various downstream tasks have been achieved. However, we find these two paradigms have different characteristics: (i) contrastive models are data-hungry that suffer from a representation over-fitting issue; (ii) generative models have a data filling issue that shows inferior data scaling capacity compared to contrastive models. This motivates us to learn 3D representations by sharing the merits of both paradigms, which is non-trivial due to the pattern difference between the two paradigms. In this paper, we propose Contrast with Reconstruct (ReCon) that unifies these two paradigms. ReCon is trained to learn from both generative modeling teachers and single/cross-modal contrastive teachers through ensemble distillation, where the generative student guides the contrastive student. An encoder-decoder style ReCon-block is proposed that transfers knowledge through cross attention with stop-gradient, which avoids pretraining over-fitting and pattern difference issues. ReCon achieves a new state-of-the-art in 3D representation learning, e.g., 91.26% accuracy on ScanObjectNN. Codes have been released at https://github.com/qizekun/ReCon.

1. Introduction

3D self-supervised representation learning faces limited-data challenges that affect contrastive and generative pretraining differently. RECON unifies both paradigms through ensemble distillation and reconstruction-guided contrastive learning, improving data efficiency and downstream transfer.

  • 3D vision faces a data desert because collecting 3D data is difficult compared with 2D vision and NLP.
  • Contrastive models over-fit limited pretraining data, whereas generative models provide improvements with approximately 25% of the data.The reported contrastive failure occurs below 90% of pretraining data.
  • Generative models have limited data-scaling capacity, while vanilla multi-task combinations remain below the generative baseline and retain representation over-fitting.
  • RECON uses ensemble distillation from generative and single- or cross-modal contrastive teachers, with a generative student guiding contrastive learning.
  • A cross-attention RECON-block with stop-gradient transfers reconstruction guidance to contrastive modeling while separating local and global learning patterns.The design is intended to address representation over-fitting and pattern differences between paradigms.
  • RECON improves downstream transfer by an average of +9.2% accuracy on ScanObjectNN and +2.9% on ModelNet40.

2. Related Works

Contrastive learning extracts semantics from invariant views, while generative masked representation learning learns structured knowledge by reconstructing masked inputs. The paper presents RECON-block pretraining as a related architecture combining local reconstruction and global contrastive learning.

  • Contrastive representation learning aligns and distinguishes representations from views sharing or lacking high-level semantics.
  • Generative masked representation learning encourages associations among local patches by reconstructing masked input data.
  • The RECON-block trains an encoder with local reconstruction and a decoder with global contrastive learning guided through cross attention.Stop-gradient is applied to cross-attention connections.

3. RECON: Contrast with Reconstruct

RECON unifies generative masked modeling and contrastive learning through ensemble distillation, using reconstruction guidance to train contrastive representations without task conflicts.

  • Unified distillation: Contrastive learning transfers invariance knowledge from transformed views or different modalities, using distance functions such as InfoNCE or feature correlation.The transformations may produce augmented views or modality-specific views sharing the same high-level concept.
  • RECON-block: RECON supports single-modal point-cloud learning and cross-modal learning with point clouds, RGB images, and language.In the cross-modal setting, views from other modalities can be encoded by different pretrained teacher networks.
  • Unified distillation: Generative masked modeling trains a student to reconstruct masked input data, with masking corruptions selecting tokens for masking and corresponding unmasked targets.The reconstruction distance may be ℓ2 distance, crossentropy, or Chamfer-Distance.
  • Unified distillation: RECON formulates representation learning as ensemble distillation from generative and contrastive teachers, with student-student assistance.Its overall objective combines reconstruction and contrastive learning while allowing the generative student to guide the contrastive student.
  • RECON-block: The RECON-block links a dense masked-modeling encoder to a sparse query-based contrastive decoder through layer-wise cross-attention with stop-gradient.Stop-gradient cuts misleading signals from global contrastive learning to local reconstruction, while the contrastive student receives generative guidance.

4. Experiments

Experiments evaluate RECON on transfer, ablation, and pretraining-loss analyses, showing gains across real-world, synthetic, few-shot, and zero-shot 3D recognition tasks.

  • Transfer learning: RECON improves ScanObjectNN accuracy by +11.3% averaged across three variants after reconstruction-guided contrastive pretraining.It outperforms Point-MAE by +5.6% across the three ScanObjectNN variants and generalizes across single-modal and cross-modal settings.
  • Transfer learning: 94.7% ModelNet40 classification accuracy is achieved under the FULL protocol, improving by 2.7% over the Transformer baseline.RECON also achieves the best performance under all reported few-shot protocols.
  • Transfer learning: 61.7% Top-1 accuracy is achieved on ModelNet40 zero-shot recognition with multiple text prompts.This surpasses PointCLIP and CLIP2Point by 41.5% and 12.3%, respectively.
  • Ablations: The masking-ratio and decoder-depth optima match Point-MAE, while ViT is the strongest tested 2D teacher.CLIP performs worse than ViT, possibly because pre-aligned text features reduce feature diversity.
  • Ablations: Without reconstruction guidance, the contrastive model performs poorly because it over-fits the limited 3D pretraining data.The cited ablation directly evaluates the importance of pretraining targets.
  • Ablations: Smooth ℓ1 distance outperforms InfoNCE in both contrastive-metric ablation tasks.The authors attribute this to frozen-teacher positive-only learning and the characteristics of ShapeNet data.
  • Pretraining analysis: Figure 4 compares pretraining ShapeNet test loss with ScanObjectNN fine-tuning accuracy across pretraining learning rates.Orange denotes pretraining test loss and blue denotes fine-tuning accuracy.

5. Discussions

RECON-CMC improves contrastive generalization and fine-tuning accuracy, while stop-gradient is necessary to prevent degraded representations. Attention visualizations suggest that local tokens and global queries learn complementary geometric information.

  • 90.63% versus 82.48%: RECON-CMC achieves higher ScanObjectNN fine-tuning accuracy than vanilla CMC.Its test contrastive loss also converges more stably to 0.034 versus 0.052.
  • -9.03% on ScanObjectNN and -4.4% on ModelNet40: removing stop-gradient seriously degrades RECON performance.The authors attribute this to noisy contrastive gradients disturbing generative guidance and valid semantic representation learning.
  • Local tokens focus on nearby point-cloud geometry, whereas global queries attend to broader areas.Figure 7 visualizes these complementary attention patterns using red and yellow as high-attention regions and blue and purple as low-attention regions.

6. Conclusions

The paper concludes that RECON combines generative masked modeling and contrastive learning for multimodal 3D representation learning. It reports improved data efficiency and generalization, state-of-the-art real-world 3D recognition, and decoupled local-global representations.

  • RECON combines generative masked modeling and contrastive modeling while scaling to multimodal 3D data.
  • RECON reports high-capacity data efficiency and generalization in pretraining and downstream representation transfer.
  • RECON achieves a new state-of-the-art on challenging real-world 3D object recognition.
  • The proposed RECON-block learns decoupled local and global representations, with reconstruction helping avoid contrastive over-fitting under limited 3D data.
  • Prior contrastive-generative representation learning has been explored in 2D vision, NLP, and vision-language learning.

B. Additional Implementation Details

The implementation uses multimodal ShapeNet pretraining with supervised single-modal, cross-modal, reconstruction, and ensemble-distillation losses. RECON transfers pooled local and global features through zero-shot and fine-tuning settings across model variants.

  • B.1. Loss Function: The multimodal minibatch contains paired point clouds, rendered images, and text descriptions, with unique text categories converted into fine-grained labels.
  • B.1. Loss Function: Single-modal contrastive learning treats same-label samples as positives and different-label samples as negatives using a supervised contrastive objective.
  • B.1. Loss Function: Cross-modal contrastive learning uses same-sample representations from paired modalities as positives and other minibatch pairs as negatives.Frozen image and language teachers receive no back-propagated gradient because of stop-gradient.
  • B.1. Loss Function: Masked point modeling reconstructs masked point clouds using an ℓ2 Chamfer-distance loss.
  • B.1. Loss Function: RECON combines reconstruction with single-modal or cross-modal contrastive losses through ensemble distillation and uses positive-only Smooth ℓ1 representation learning.
  • B.2. Experimental Details: ShapeNetCore supplies approximately 51K 3D CAD models from 55 categories, paired with rendered RGB images and templated text descriptions.Pretraining runs for 300 epochs with AdamW, batch size 128, and a cosine learning rate beginning at 5e-4 after 10 warmup epochs.
  • B.2. Experimental Details: Downstream transfer uses summed multimodal features for zero-shot classification and concatenated pooled local and global query features for fine-tuning.
  • B.2. Experimental Details: RECON includes Tiny, Small, and Base configurations, with smaller variants reducing channel dimension while retaining the same number of layers.

C. Additional Baselines

The additional baselines test whether performance gains arise from simply sharing or separating reconstruction and contrastive networks. RECON instead uses reconstruction as guidance for contrastive learning to disentangle the tasks while preserving both merits.

  • Vanilla Multi-task Learning Fusion: Vanilla multi-task learning shares one Transformer encoder between masked reconstruction input tokens and global contrastive tokens.It does not account for the pattern difference between the two tasks.
  • Two-Tower Network: The Two-Tower baseline uses separate Transformers for masked reconstruction and global contrastive learning, then concatenates both streams during fine-tuning.
  • RECON Comparison: RECON uses reconstruction as guidance for global contrastive learning rather than merely sharing or separating task towers.
  • RECON Comparison: RECON disentangles the two tasks while preserving both merits and achieves significantly better improvements than the additional baseline designs.
  • Additional Baselines: Table 9 reports overall accuracy without voting for the additional baseline study.

D.1. Additional Evaluations

Additional evaluations show that RECON transfers effectively across discriminative, geometric, and real-world zero-shot 3D tasks.

  • Linear SVM Evaluation: +6.0% ModelNet40 overall accuracy over Point-BERT in Linear SVM evaluation.RECON also exceeds PointM2AE by +0.5%.
  • 3D Part Segmentation: +1.4% Cls. mIoU and +1.7% Inst. mIoU over the from-scratch baseline on ShapeNetPart.
  • Zero-Shot Recognition on Real-World Dataset: 30.5% Top-1 accuracy on ScanObjectNN PB T50 RS, exceeding CLIP2Point by +7.2% and PointCLIP by +15.1%.

D.2. Additional Ablation Study

Ablations examine augmentation, paired data, prompting, learning rates, teacher freezing, deployment, and multimodal-training choices affecting RECON.

  • Data Augmentation: Rotation performs best on ScanObjectNN, while Scale&Translate performs best on ModelNet40.These augmentations are used by default for the respective datasets.
  • Paired Data Ablation: Less than 1% degradation occurs in fine-tuning tasks and 6.4% in zero-shot tasks when paired rendered images are shuffled within categories.
  • Pretraining Learning Rate Ablation on Contrastive Models: Contrastive models are sensitive to pretraining learning rate under limited data, with smaller rates improving generalization and training stability.RECON uses an adjusted smaller learning rate for contrastive models unless otherwise specified.
  • Ablation Study on Freezing Cross-Modal Teachers: Unfrozen teacher parameters cause serious downstream degradation for positive-pair contrastive learning, but have limited impact with InfoNCE.
  • Edge Device Deployments: RECON is reported as deployable across laptops, pads, smartphones, and single-chip microcomputers using CPU inference.The evaluation uses ONNX and 1K-point ModelNet40 clouds per sample.

E. Discussions on Cross-Modal Teachers and Multimodal Training

The ablation isolates contributions from single-modal contrastive learning, cross-modal data, and pretrained teachers in RECON’s downstream performance.

  • 89.73% overall accuracy on ScanObjectNN is achieved by RECON+SMC, exceeding Point-MAE by +1.31% and ACT by +0.72%.This configuration uses single-modal contrastive learning without other modality data or pretrained teachers.
  • 90.32% overall accuracy on ScanObjectNN is reached by RECON+CMC from scratch, adding +0.59% through cross-modal contrastive learning on multimodal data.

F. Limitations and Future Works

The paper identifies scope and architecture limitations while proposing extensions beyond 3D representation learning.

  • RECON is mainly explored for 3D representation learning, leaving multimodal settings such as 2D vision-language for future study.
  • The RECON-block remains architecture-specific, motivating future work toward an architecture-agnostic framework.
  • Potential future applications include large-scale multimodal learning, LLM-assisted multimodal understanding, and cross-modal generative modeling.
Loading 2302.02318v2…