PaperlayerView exact revision on arXiv

Source-linked AI summary

PointCLIP: Point Cloud Understanding by CLIP

Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, Hongsheng Li

arXiv:2112.02413v1cs.CVcs.AIcs.RO

TL;DR

3D recognition lacks established evidence that CLIP models pretrained on 2D image-text pairs can generalize to point clouds, especially for unseen categories. PointCLIP bridges the modalities with multi-view depth-map projections and an inter-view adapter, and reports improved few-shot performance plus gains from ensembling with classical 3D networks. The paper concludes that this provides an effective low-resource route to 3D point-cloud understanding, while broader 3D tasks remain unexplored.

  • Problem

    Whether CLIP pretrained on large-scale 2D image-text pairs can generalize to 3D point-cloud recognition and classify unseen 3D objects remains under explored.

  • Method

    PointCLIP projects point clouds into multi-view depth maps for CLIP encoding, aggregates view-wise predictions, and uses a lightweight inter-view adapter for few-shot adaptation.

  • Results

    PointCLIP improves with few-shot adapter fine-tuning and boosts classical 3D networks through ensembling, including 92.03% versus PointNet++’s 89.71% on 16-shot ModelNet40.

  • Takeaways & Limitations

    PointCLIP serves as a plug-and-play source of complementary 2D contrastive knowledge for effective 3D point-cloud understanding under limited training resources.

  • Takeaways & Limitations

    How to use CLIP’s knowledge for other 3D tasks remains under explored.

Abstract

from arXiv · show

Recently, zero-shot and few-shot learning via Contrastive Vision-Language Pre-training (CLIP) have shown inspirational performance on 2D visual recognition, which learns to match images with their corresponding texts in open-vocabulary settings. However, it remains under explored that whether CLIP, pre-trained by large-scale image-text pairs in 2D, can be generalized to 3D recognition. In this paper, we identify such a setting is feasible by proposing PointCLIP, which conducts alignment between CLIP-encoded point cloud and 3D category texts. Specifically, we encode a point cloud by projecting it into multi-view depth maps without rendering, and aggregate the view-wise zero-shot prediction to achieve knowledge transfer from 2D to 3D. On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D. By just fine-tuning the lightweight adapter in the few-shot settings, the performance of PointCLIP could be largely improved. In addition, we observe the complementary property between PointCLIP and classical 3D-supervised networks. By simple ensembling, PointCLIP boosts baseline's performance and even surpasses state-of-the-art models. Therefore, PointCLIP is a promising alternative for effective 3D point cloud understanding via CLIP under low resource cost and data regime. We conduct thorough experiments on widely-adopted ModelNet10, ModelNet40 and the challenging ScanObjectNN to demonstrate the effectiveness of PointCLIP. The code is released at https://github.com/ZrrSkywalker/PointCLIP.

1. Introduction

PointCLIP transfers CLIP’s 2D image-text knowledge to 3D point-cloud recognition through multi-view projections, enabling zero-shot classification and few-shot adaptation. Its inter-view adapter improves few-shot performance, while ensembling with classical 3D networks can surpass state-of-the-art results.

  • Motivation and contribution: Irregular, sparse point clouds and unseen object categories make direct 2D-to-3D transfer and conventional classifier reuse difficult.The paper frames recognizing newly captured, unlabeled categories as a central challenge for 3D understanding.
  • Method: PointCLIP projects point clouds onto multiple image planes as scatter depth maps, allowing CLIP to encode multi-view 3D representations without post-rendering.The online projection has marginal time and computation cost while preserving multi-view point-cloud information.
  • Method: An inter-view adapter aggregates and interacts across view features to extract global point-cloud representations for few-shot fine-tuning.The adapter is lightweight, and only it is fine-tuned while CLIP’s visual and textual encoders remain frozen.
  • Results: Comparable performance with some fully trained models is achieved by PointCLIP after few-shot fine-tuning, providing a performance–cost trade-off.The reported setting fine-tunes the lightweight adapter using only a few samples per class.
  • Results: 92.03% improves PointNet++’s 89.71% on 16-shot ModelNet40, while 94.08% improves CurveNet’s 93.84% through ensembling.PointCLIP itself obtains 87.20% in the PointNet++ example; ensembling two fully trained ModelNet40 models without PointCLIP instead causes performance loss.
  • Motivation and contribution: PointCLIP transfers 2D pre-trained knowledge into 3D point-cloud recognition for cross-modality zero-shot classification.It uses CLIP knowledge learned from image-text pairs to recognize 3D data without 3D training.

2. Related Work

Related work covers zero-shot learning, transfer learning, point-cloud neural networks, and projection-based representations. PointCLIP differs by performing direct zero-shot recognition without 3D training and by using lightweight point projections instead of costly rendering.

  • Zero-shot Learning in 3D: Prior 3D zero-shot methods divide data into seen and unseen samples, train on the former, and evaluate recognition on the latter.Related methods improve feature quality or use transductive unlabeled data, but remain based on 3D training subsets.
  • Zero-shot Learning in 3D: PointCLIP performs direct zero-shot recognition across whole point-cloud datasets without any 3D training.This distinguishes its setting from earlier approaches that train on part of the 3D samples.
  • Transfer Learning: Transfer learning uses knowledge from data-abundant domains to assist learning in data-scarce domains, including ImageNet and language-model pre-training.The paper situates PointCLIP within this broader paradigm of transferring pre-trained representations.
  • Deep Neural Networks for Point Cloud: Point-cloud networks include point-based models such as PointNet and PointNet++, which encode points and use max pooling for permutation invariance.Later point-based methods develop stronger local aggregators and architectural designs.
  • Deep Neural Networks for Point Cloud: Naively projecting raw points onto image planes produces depth maps with marginal time and computation costs, unlike mesh construction or post-rendering.The projection-based representation is presented as suitable for efficient end-to-end zero-shot recognition.

3. Method

PointCLIP transfers CLIP’s 2D visual-language knowledge to 3D point clouds by projecting them into multi-view depth maps, then supports few-shot adaptation and ensembling with classical 3D networks.

  • 3.1. A Revisit of CLIP: CLIP aligns visual and textual features for zero-shot recognition, motivating PointCLIP’s transfer of 2D pre-trained knowledge into 3D point cloud understanding.PointCLIP extends this framework to 3D recognition through cross-modality knowledge transfer.
  • 3.2. Point Cloud Understanding by CLIP: PointCLIP bridges unordered point clouds and grid-based images by directly projecting raw points onto multiple image planes to produce scatter depth maps.The projection uses raw points without rendering or color information, keeping time and computation costs marginal.
  • 3.2. Point Cloud Understanding by CLIP: For zero-shot classification, CLIP encodes each projected view and category prompt, computes view-wise logits, and combines them through weighted summation.The process is non-parametric on unseen 3D datasets and requires no 3D training.
  • 3.3. Inter-view Adapter for PointCLIP: The inter-view adapter concatenates multi-view features, extracts a compact global representation, and generates view-wise adapted features through residual fusion with CLIP features.This design combines cross-view information while injecting newly learned 3D few-shot knowledge into the pre-trained 2D representation.
  • 3.3. Inter-view Adapter for PointCLIP: Fine-tuning only the lightweight inter-view adapter improves PointCLIP from 20.18% to 87.20% on ModelNet40 with 16 samples per category.The adapter is used because fine-tuning the entire model with insufficient few-shot data could cause over-fitting.
  • 3.4. Multi-knowledge Ensembling: PointCLIP can be ensembled with fully trained 3D networks as a plug-and-play enhancement module using complementary 2D and 3D knowledge.The method is described as improving classical networks and achieving state-of-the-art performance in some cases.

4. Experiments

Experiments evaluate PointCLIP in zero-shot, few-shot, prompt, view, encoder, and ensembling settings across ModelNet and ScanObjectNN. PointCLIP transfers 2D CLIP knowledge to 3D and complements fully trained 3D networks.

  • 4.1. Zero-shot Classification: 20.18% is the best zero-shot ModelNet40 result among the listed prompt designs, produced by “point cloud depth map of a [CLASS].”The prompt comparison includes zero-shot and 16-shot results for five fixed templates and one learnable-token design.
  • 4.1. Zero-shot Classification: More than 6 projected views causes zero-shot performance decay, whereas the right view contributes most and top and bottom views contribute relatively less.The few-shot setting reverses the view-importance pattern: the left view becomes most informative, and 10 views outperform 6 views.
  • 4.2. Few-shot Classification: PointCLIP with its inter-view adapter surpasses PointNet, PointNet++, SimpleView, and CurveNet across the evaluated 1-, 2-, 4-, 8-, and 16-shot settings.On ModelNet40 with 1 shot, PointCLIP exceeds PointNet by 25.49% and CurveNet by 12.29%; the gap narrows with more samples because of the adapter’s limited fitting capacity.

5. Conclusion and Limitation

PointCLIP transfers CLIP’s 2D knowledge to 3D point-cloud recognition through multi-view projection and lightweight adapter fine-tuning. It also complements classical 3D networks, while broader 3D-task generalization remains open.

  • Multi-view projection transfers CLIP’s pre-trained 2D knowledge into the 3D domain.
  • A lightweight inter-view adapter aggregates multi-view representations and generates adapted features under few-shot settings.
  • PointCLIP can provide complementary information to classical 3D networks as a plug-and-play module, surpassing state-of-the-art performance.
  • Generalizing CLIP’s knowledge to other 3D tasks remains under explored and is identified as future work.

A. Datasets

The evaluation uses ModelNet10, ModelNet40, and ScanObjectNN, covering synthetic indoor-object benchmarks and a challenging point-cloud dataset.

  • ModelNet10 contains 4,899 synthetic meshed CAD models spanning 10 indoor categories.It includes 3,991 training and 908 testing models.
  • ModelNet40 contains 12,311 samples across 40 common categories.It includes 9,843 training and 2,468 testing samples.
  • ModelNet10 and ModelNet40 uniformly sample 1,024 points from each object as network input.

B. Implementation Details

Implementation varies projected views between zero-shot and few-shot settings, and trains the inter-view adapter with standardized optimization and augmentation choices.

  • Zero-shot and few-shot PointCLIP use different projected-view configurations, with the view number M selecting the first M views.Zero-shot emphasizes right views, whereas few-shot replaces left settings with right settings.
  • The inter-view adapter is fine-tuned for 1, 2, 4, 8, and 16 shots using batch size 32, learning rate 0.01, and 250 epochs.
  • Training uses SGD with momentum 0.9, cosine learning-rate decay, Smooth Loss, and dataset-specific augmentation.ModelNet10 and ModelNet40 use random scaling and translation; ScanObjectNN additionally uses jitter and random rotation.

C. Supplementary Ablations

Supplementary ablations examine adapter architecture, feature fusion, encoder choices, and fine-tuning modules, showing that inter-view global extraction and lightweight adaptation are important design choices.

  • Inter-view Adapter: The inter-view adapter uses three linear layers for global extraction and view-wise adapted-feature generation.
  • Inter-view Adapter: Removing or repositioning adapter modules hurts performance, especially when inter-view global-feature extraction is changed.
  • Adapted Features Fusion: The view-wise adapted feature is added to the original CLIP feature through a residual connection, with fusion ratio β controlling its proportion.On ModelNet40, β=0.6 performs better than other tested ratios, while overall variance is small.
  • Full Training Set: Fine-tuning only the lightweight inter-view adapter reaches 92.01% on the full ModelNet40 training set with visual encoders of varying size.
  • Fine-tuning Settings: Unfreezing only the textual encoder normally hurts performance, whereas training both encoders and all modules reaches 91.40% and 91.89%, respectively.

D. Visualizations

Figure 6 visualizes how ensembling PointCLIP with PointNet++ preserves correct predictions and rectifies some complementary errors.

  • PointCLIP and PointNet++ both predict four visualized samples correctly, so ensembling preserves those predictions.
  • For other samples, the models exhibit complementary errors, allowing the ensemble to rectify one model’s wrong prediction.
  • The visualizations demonstrate the importance of knowledge interaction between PointCLIP and PointNet++.
Loading 2112.02413v1…