PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training

Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson W. H. Lau, Wanli Ouyang, Wangmeng Zuo

arXiv:2210.01055v3cs.CV

TL;DR

Limited 3D-text data and an image-depth domain gap constrain transferring CLIP to point clouds. CLIP2Point addresses these challenges with contrastive image-depth pre-training and a dual-path adapter, achieving state-of-the-art results in zero-shot and few-shot classification.

  • Problem

    3D vision-language pre-training lacks sufficient training data, while depth maps differ from CLIP's image domain and exhibit varied distributions across views.

  • Method

    CLIP2Point pre-trains a depth encoder with cross-modality and intra-modality contrastive learning, then uses a Gated Dual-Path Adapter to combine CLIP and CLIP2Point for downstream tasks.

  • Results

    CLIP2Point achieves state-of-the-art results compared with 3D transfer-learning and pre-training methods, including zero-shot and few-shot classification.

  • Takeaways & Limitations

    CLIP2Point effectively transfers CLIP knowledge to 3D vision for zero-shot and few-shot point-cloud classification.

  • Takeaways & Limitations

    Performance and generalization are affected by pre-training data quantity and quality, including the limited realism of synthetic data for noisy real-world scenes.

Abstract

from arXiv · show

Pre-training across 3D vision and language remains under development because of limited training data. Recent works attempt to transfer vision-language pre-training models to 3D vision. PointCLIP converts point cloud data to multi-view depth maps, adopting CLIP for shape classification. However, its performance is restricted by the domain gap between rendered depth maps and images, as well as the diversity of depth distributions. To address this issue, we propose CLIP2Point, an image-depth pre-training method by contrastive learning to transfer CLIP to the 3D domain, and adapt it to point cloud classification. We introduce a new depth rendering setting that forms a better visual effect, and then render 52,460 pairs of images and depth maps from ShapeNet for pre-training. The pre-training scheme of CLIP2Point combines cross-modality learning to enforce the depth features for capturing expressive visual and textual features and intra-modality learning to enhance the invariance of depth aggregation. Additionally, we propose a novel Dual-Path Adapter (DPA) module, i.e., a dual-path structure with simplified adapters for few-shot learning. The dual-path structure allows the joint use of CLIP and CLIP2Point, and the simplified adapter can well fit few-shot tasks without post-search. Experimental results show that CLIP2Point is effective in transferring CLIP knowledge to 3D vision. Our CLIP2Point outperforms PointCLIP and other self-supervised 3D networks, achieving state-of-the-art results on zero-shot and few-shot classification.

1. Introduction

CLIP2Point addresses limited 3D vision-language pre-training and the image-depth domain gap by aligning depth features with CLIP through contrastive pre-training and lightweight adaptation. It reports state-of-the-art results across zero-shot, few-shot, and fully supervised point-cloud classification.

  • Motivation: 3D vision-language pre-training remains limited by insufficient training data, leaving many 3D networks trained from scratch on downstream datasets.CLIP uses more than 400M image-text pairs, whereas PointContrast uses 100k point-cloud pairs from 1,513 ScanNet scenes.
  • Motivation: PointCLIP transfers CLIP to point clouds through multi-view depth maps, but image-depth domain differences and varied depth distributions restrict performance.Point clouds are projected to multi-view depth maps, aggregated by the CLIP visual encoder, and compared with textual prompts.
  • Method: CLIP2Point uses cross-modality contrastive learning to align depth features with CLIP image features and intra-modality learning to make depth aggregation invariant to view changes.The CLIP image encoder is frozen while the depth encoder is trained for both alignment and view-invariant aggregation.
  • Method: The Gated Dual-Path Adapter jointly uses CLIP and CLIP2Point encoders with global-view aggregators and gated fusion for downstream representation learning.The dual paths combine complementary instance-level depth and category-level visual information.
  • Results: CLIP2Point significantly improves zero-shot classification and achieves state-of-the-art results on zero-shot, few-shot, and fully supervised point-cloud classification.Experiments evaluate ModelNet10, ModelNet40, and ScanObjectNN against 3D transfer-learning and pre-training networks.

2. Related Work

Related work covers 3D vision-language transfer, self-supervised representation learning, and parameter-efficient downstream adaptation. CLIP2Point builds on these directions by combining contrastive depth-image pre-training with lightweight adapter-based fine-tuning.

  • 3D Vision-Language Transfer: 3D vision-language pre-training is constrained by insufficient 3D-text data, motivating transfer from large-scale vision-language models.CLIP transfers cross-modal knowledge to visual concepts, while 3D-language pre-training remains restricted by limited paired data.
  • Self-Supervised Pre-training: Self-supervised 3D pre-training uses task-independent data and pretext objectives, including contrastive learning across point, instance, scene, and modality granularities.The paper positions its image-depth contrastive scheme within this broader self-supervised learning landscape.
  • Downstream Fine-Tuning: Full fine-tuning can overfit limited downstream data, whereas partial tuning and prompt tuning provide more data-efficient adaptation.These approaches update fewer parameters or use learnable tokens and task heads instead of tuning all pre-trained weights.
  • Downstream Fine-Tuning: CLIP2Point proposes a Gated Dual-Path Adapter for lightweight fine-tuning with CLIP textual prompts and supervised downstream learning.The adapter uses efficient parameter updates while retaining the pre-trained model components.

3. CLIP-Based Transfer Learning in 3D

CLIP2Point transfers CLIP knowledge to point-cloud classification by aligning rendered depth features with CLIP image features and stabilizing multi-view depth aggregation. It also introduces depth rendering changes to improve visual continuity and compatibility with CLIP.

  • 3.1. Review of CLIP and PointCLIP: PointCLIP applies CLIP to multi-view depth maps, but image-depth domain differences and varied depth distributions restrict performance.Point clouds are rendered into 2D depth maps, whose sparsity and disorder produce inconsistent depth distributions across views.
  • 3.2. Aligning with CLIP Visual Features: CLIP2Point trains a depth encoder to align rendered depth features with frozen CLIP image features, enabling CLIP textual prompts to classify depth features.Both encoders are initialized from CLIP’s visual encoder, but only the depth encoder is learnable during pre-training.
  • 3.2.1 Pre-Training Scheme: Cross-modality contrastive learning matches paired RGB images and depth maps, while intra-modality learning promotes invariant aggregation across depth views.The combined loss balances the intra-modality and cross-modality terms with a learnable parameter σ.
  • 3.2.2 Depth Rendering: The rendering procedure projects 3D points to view-specific pixels, handles collisions through depth aggregation, and expands points to neighboring pixels with dilation rate R = 2.The method motivates minimum-depth aggregation for perspective visualization and neighborhood expansion for sparse coverage.

4. Downstream Representation Learning

GDPA adapts CLIP2Point to downstream recognition by combining its depth encoder with CLIP’s visual encoder through efficient global-view aggregation and gated fusion. The design supports few-shot and fully supervised object-level learning.

  • 4.1. Gated Dual-Path Adapter: GDPA uses dual paths containing the pre-trained depth encoder and CLIP visual encoder because their instance-level and category-level knowledge can be complementary.The depth encoder can be adjusted toward an appropriate feature domain while the visual encoder focuses on category selection.
  • 4.1. Gated Dual-Path Adapter: Learnable global-view aggregators extract overall features from multiple views for each encoder, reducing parameters and avoiding post-search.Direct global-feature estimation avoids expanding aggregated features back to the multi-view feature dimension.
  • 4.1. Gated Dual-Path Adapter: A learnable gate adds weighted CLIP features to the depth features to control efficient fusion of the two modalities.The final global representation is used to calculate category logits through cosine similarity with textual features.
  • 4.2. Supervised Downstream Learning: GDPA supports both few-shot and fully supervised recognition, with cross-entropy supervision for object-level discrimination.The paper also states that scene-level tasks can be accessible when possible proposals are provided.

5. Experiments

Experiments evaluate CLIP2Point across zero-shot, few-shot, and fully supervised point-cloud classification, including synthetic and real-world datasets. Results show strong transfer performance, while ablations link gains to rendering, pre-training, intra-modality learning, and the dual-path design.

  • Datasets: 52,460 ShapeNet models yield 41,943 training and 10,517 validation RGB-depth pairs across ten views for pre-training.Images and depth maps are rendered at 224×224 resolution, using the farthest 1,024 points for depth rendering.
  • Overall results: CLIP2Point achieves state-of-the-art results on zero-shot, few-shot, and fully supervised classification across ModelNet10, ModelNet40, and ScanObjectNN.The evaluation compares against 3D transfer-learning and self-supervised pre-training networks.
  • Zero-shot classification: 36.40% and 29.20% accuracy gains occur on ModelNet10 and ModelNet40 after pre-training, while S-OBJ ONLY gains 20.31%.The improvement is smaller on S-OBJ BG and S-PB T50 RS, which the authors associate with background points and possible instability on real-world data.
  • Few-shot classification: 89.79% accuracy is reached in 16-shot classification with the pre-trained pipeline, approaching traditional supervised networks such as PointNet++.The pipeline without pre-training already outperforms other compared methods, partly due to the global-view aggregator.
  • Fully-supervised classification: CLIP2Point outperforms P2P (HorNet-L) with lower input requirements and fine-tunes only learnable adapters instead of all parameters.The authors report state-of-the-art fully supervised performance with low input and parameter requirements.
  • Ablation study: 11.09% lower zero-shot accuracy results when intra-modality learning is removed, supporting its role in view-invariant depth aggregation.Cross-modality-only pre-training reaches 38.29% accuracy, whereas intra-modality contrastive learning improves invariance across camera views.
  • Ablation study: The dual-path structure substantially improves performance, while extra inter-view-adapter weights make few-shot training easier to overfit.The authors attribute the dual-path gain to complementary CLIP and CLIP2Point knowledge and report information loss from some feature expansion and gathering operations.
  • Ablation study: Minimum-depth rendering performs better in both zero-shot and few-shot classification, while excessive dilation can blur object-corner depth maps.Feature visualizations also show chaotic representations without pre-training and better class separation after CLIP2Point pre-training.

6. Conclusion

CLIP2Point pre-trains a depth encoder to transfer CLIP knowledge to 3D vision using cross- and intra-modality contrastive learning. With ShapeNet image-depth data and a Gated Dual-Path Adapter, it achieves state-of-the-art results against 3D transfer and pre-training methods, while synthetic data limits generalization to realistic scenes.

  • Conclusion: CLIP2Point pre-trains a depth encoder to adapt CLIP knowledge to the 3D domain.The method uses cross-modality and intra-modality contrastive learning to align depth and image features while preserving multi-view depth-distribution invariance.
  • Conclusion: 52,560 images and corresponding depth maps are rendered from ShapeNet using a new depth-rendering setting for pre-training.
  • Conclusion: The Gated Dual-Path Adapter uses global-view aggregators and gated fusion to adapt pre-trained weights to downstream tasks.
  • Conclusion: CLIP2Point achieves state-of-the-art results compared with 3D transfer learning and pre-training methods.The reported comparison concerns the resulting point-cloud classification performance.
  • Conclusion: Performance and generalization are affected by the amount and quality of pre-training data, with synthetic data limited for noisy, complex real-world downstream tasks.The authors propose using more realistic data and extending the method to detection and segmentation.

A. Details of Loss Function

The loss design combines InfoNCE-based cross-modality and intra-modality objectives. Cross-modality learning aligns depth and image features, while intra-modality learning compares depth representations across views; the temperature coefficient is set to 0.7.

  • Details of Loss Function: The cross- and intra-modality losses are formulated as InfoNCE-based objectives.
  • Details of Loss Function: Cross-modality learning pulls corresponding depth and image features together relative to other feature pairs.
  • Details of Loss Function: Intra-modality learning compares depth features from different views while contrasting them against other depth representations.
  • Details of Loss Function: The similarity function uses a temperature coefficient τ = 0.7.

B. Complexity Analysis

The complexity analysis compares evaluation computation and training parameter sizes under fully supervised classification. CLIP2Point combines favorable computation with lightweight adaptation, but MAC comparisons involving Transformers omit point-cloud grouping and gathering costs.

  • Complexity Analysis: Evaluation computation costs and training parameter sizes are reported for fully supervised classification.
  • Complexity Analysis: CLIP2Point achieves higher accuracy than P2P with much lower computation cost.P2P's cost must be multiplied by 40 because it infers one view at a time.
  • Complexity Analysis: Tuning only lightweight adapters enables CLIP2Point to outperform Point-MAE while using fewer training parameters than full-tuning methods.
  • Complexity Analysis: Transformer MAC values are not directly comparable with 3D networks because grouping and gathering mechanisms are excluded.

C. Application on Scene-Level Tasks

CLIP2Point is applied to open-vocabulary 3D detection by classifying bounding boxes generated by 3D detectors. It outperforms the compared detection and cross-modal methods, although noisy point clouds in proposed boxes reduce its mAP relative to a detector trained specifically for 3D detection.

  • Application on Scene-Level Tasks: The open-vocabulary detection pipeline uses CLIP2Point to classify bounding boxes generated by 3D detectors.
  • Application on Scene-Level Tasks: CLIP2Point outperforms two 3D detection networks and two cross-modal pre-training methods on ScanNet open-vocabulary 3D detection.
  • Application on Scene-Level Tasks: Its mAP is relatively low compared with OV-3DETIC because proposed bounding boxes contain noisy point-cloud data and OV-3DETIC is specifically trained on 3D detection datasets.

D. Why Not Applying CLIP to 3D Networks?

Directly applying CLIP pre-training to 3D networks performs poorly because 2D and 3D representations differ in granularity and model scale. The paper identifies this transfer as promising future work while using a rendering-based alternative.

  • 20.83% accuracy is obtained on ModelNet40 zero-shot classification when CLIP2Point's depth encoder is replaced with a Point Transformer.This result motivates examining why direct transfer to 3D encoders performs poorly.
  • 2D encoders extract single-view features, whereas 3D encoders aggregate complete 3D objects, creating a feature-granularity mismatch.
  • The authors describe direct transfer of CLIP knowledge to 3D networks as promising future work rather than a resolved approach.
  • The rendering setup uses ten spherical views, with six orthogonal views for zero-shot evaluation and four additional corner views for pre-training and downstream learning.

F. Dataset Visualization

The dataset visualization presents rendered RGB images and corresponding depth maps across ShapeNet categories. Figures 7–9 organize these examples into category ranges, while Figure 6 shows multi-view airplane images.

  • Each ShapeNet category is represented by a rendered RGB image paired with a corresponding depth map.
  • Figure 6 visualizes multi-view RGB images for an airplane.
  • The visualizations pair named object examples with corresponding image and depth entries, including airplanes, furniture, tools, vehicles, and household objects.Examples include airplane/airplane-depth, bed/bed-depth, bowl/bowl-depth, keyboard/keyboard-depth, guitar/guitar-depth, and rocket/rocket-depth.
  • Figure 7 shows rendered RGB images for ShapeNet Categories 1–20.
  • Additional paired examples include keyboards, dishwashers, displays, earphones, faucets, guitars, helmets, jars, and knives.
  • Figure 8 shows rendered RGB images for ShapeNet Categories 21–40.
  • The remaining visualized pairs include pistols, pots, printers, controls, rifles, skateboards, sofas, stoves, and tables.
  • Figure 9 shows rendered RGB images for ShapeNet Categories 41–55.
Loading 2210.01055v3…