Source-linked AI summary
PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, Peng Gao
TL;DR
Pre-trained vision-language models have had limited application to point clouds beyond classification, leaving broader 3D open-world learning underexplored. PointCLIP V2 combines CLIP with GPT-3, using realistic visual projection and 3D-specific text prompting. It surpasses PointCLIP on three zero-shot classification datasets and extends to few-shot classification, zero-shot part segmentation, and zero-shot detection.
Problem
Existing vision-language-model applications to point clouds are limited and focus only on classification, while recognizing unseen 3D categories remains underexplored.
Method
PointCLIP V2 unifies CLIP and GPT-3 by generating realistic point-cloud projections and 3D-specific textual descriptions without 3D training.
Results
PointCLIP V2 surpasses PointCLIP by +42.90%, +40.44%, and +28.75% accuracy on ModelNet10, ModelNet40, and ScanObjectNN, respectively, and extends to several open-world 3D tasks.
Takeaways & Limitations
The framework supports unified 3D open-world learning across zero-shot and few-shot classification, zero-shot part segmentation, and zero-shot 3D object detection.
Takeaways & Limitations
Future work remains to adapt CLIP to wider open-world applications such as outdoor 3D detection and visual grounding.
Abstract
from arXiv · showhide
Large-scale pre-trained models have shown promising open-world performance for both vision and language tasks. However, their transferred capacity on 3D point clouds is still limited and only constrained to the classification task. In this paper, we first collaborate CLIP and GPT to be a unified 3D open-world learner, named as PointCLIP V2, which fully unleashes their potential for zero-shot 3D classification, segmentation, and detection. To better align 3D data with the pre-trained language knowledge, PointCLIP V2 contains two key designs. For the visual end, we prompt CLIP via a shape projection module to generate more realistic depth maps, narrowing the domain gap between projected point clouds with natural images. For the textual end, we prompt the GPT model to generate 3D-specific text as the input of CLIP's textual encoder. Without any training in 3D domains, our approach significantly surpasses PointCLIP by +42.90%, +40.44%, and +28.75% accuracy on three datasets for zero-shot 3D classification. On top of that, V2 can be extended to few-shot 3D classification, zero-shot 3D part segmentation, and 3D object detection in a simple manner, demonstrating our generalization ability for unified 3D open-world learning.
1. Introduction
PointCLIP V2 unifies CLIP and GPT for 3D open-world learning by improving both visual projection and textual prompting, extending beyond zero-shot classification to additional 3D tasks.
- Unseen 3D objects remain difficult to recognize because existing systems rely on defined and trained categories, while large-scale vision-language models have seen limited point-cloud use.
- PointCLIP enables zero-shot point-cloud classification without 3D training but uses sparse depth projections, performs unsatisfactorily, and relies only on CLIP rather than an LLM.
- PointCLIP V2 prompts CLIP with realistic projections and GPT-3 with 3D-oriented commands to reduce the domain gap among 2D, 3D, and language.
- Realistic projection converts point clouds into grid-based voxels and applies non-parametric 3D local filtering, producing denser points and smoother depth values.
- +42.90%, +40.44%, and +28.75% accuracy over PointCLIP are reported for zero-shot classification on ModelNet10, ModelNet40, and ScanObjectNN, respectively.
- PointCLIP V2 unifies CLIP and GPT for zero-shot and few-shot classification, zero-shot part segmentation, and zero-shot 3D object detection without 3D training.
2. Related Works
Related work covers 3D open-world learning, point-cloud projection, vision prompt learning, and GPT-3, positioning PointCLIP V2 as a CLIP-and-GPT approach with realistic projection.
- 3D Open-world Learning: Traditional 3D open-world methods require 3D training data, whereas PointCLIP performs zero-shot point-cloud recognition without training on 3D datasets.
- Projection for Point Clouds: Projection-based point-cloud analysis transfers 2D networks to 3D by projecting point clouds into images, but PointCLIP’s efficient perspective transformation has limited classification accuracy.
- Projection for Point Clouds: PointCLIP V2’s realistic projection has faster inference than other approaches and higher zero-shot performance than PointCLIP.
- Prompt Learning in Vision: Vision prompt learning narrows the gap between pre-training and downstream scenarios, and this paper prompts CLIP’s visual and textual encoders for zero-shot 3D prediction.
- GPT-3 Model: GPT-3 is a 175-billion-parameter autoregressive language model, and this work uses 3D-related commands to support open-world 3D tasks.
3. Methods
PointCLIP V2 aligns projected 3D point clouds with CLIP by combining realistic depth-map projection and GPT-generated 3D-specific text. The framework supports zero- and few-shot classification plus zero-shot segmentation and detection through task-specific extensions.
- Framework Overview: The framework uses CLIP’s visual and textual encoders, realistic projection from 3D point clouds to depth maps, and GPT-generated 3D-specific text.These components bridge the modal gap between depth maps and language for 3D open-world learning.
- Prompting CLIP with Realistic Projection: Projection converts point clouds into depth maps through quantization, densification, smoothing, and squeezing.The process uses grid-based voxels, local minimum pooling, Gaussian filtering, and depth-channel minimum pooling.
- Prompting GPT with 3D Command: GPT-3 receives caption, question-answering, paraphrase, and word-to-sentence commands to generate category-specific descriptions for CLIP’s textual encoder.The generated text emphasizes 3D shape characteristics and can incorporate appearance terms such as smoothness.
- 3D Zero-shot Classification: Zero-shot classification feeds multi-view projected depth maps and GPT-derived category text into CLIP, then aggregates view-wise visual-textual alignment.View weights α_i determine the importance of each view.
- Experimental Setting: Table 2 reports zero-shot 3D classification performance on ModelNet10, ModelNet40, and ScanObjectNN under different pre-training and model settings.The table distinguishes image-language “2D Pre-train” from training on 3D datasets.
- Unified Open-world Learning: Few-shot classification makes the smoothing module learnable, while segmentation uses dense CLIP features, GPT-generated part descriptions, and geometric back-projection into 3D.The framework also supports zero-shot detection and segmentation through simple modifications.
4. Experiments
The experiments evaluate PointCLIP V2 across zero-shot and few-shot classification, zero-shot segmentation, and zero-shot detection, including ablations and efficiency comparisons. V2 improves over PointCLIP across these settings while using no 3D training data for zero-shot evaluation.
- Zero-shot Classification: V2 achieves 73.13% and 64.22% accuracy on ModelNet10 and ModelNet40, surpassing PointCLIP by +42.90% and +40.44%, respectively.It also achieves 35.36% on the PB T50 RS split of ScanObjectNN.
- Zero-shot Classification: +25.11% zero-shot performance improvement results when all four GPT command types generate 3D-specific text.The realistic projection ablation also reports gains from densifying and smoothing, while minimum pooling performs best among tested alternatives.
- Few-shot Classification: +12% and +7% are V2’s 1-shot accuracy gains over PointCLIP on ModelNet40 and ScanObjectNN.At 16 shots, V2 reaches 89.55% accuracy on ModelNet40, approaching fully supervised PointNet.
- Zero-shot Part Segmentation: +17.4% overall mIoUI improvement over PointCLIP is achieved for zero-shot part segmentation on ShapeNetPart.The method also performs consistently better across object categories and captures fine-grained 3D patterns.
- Computation and Additional Ablations: V2 incurs similar computation overhead to PointCLIP while achieving superior zero-shot accuracy on ModelNet40.The comparison is made using one RTX A6000 with a ViT-B/32 backbone.
- Computation and Additional Ablations: Increasing the sampled point count improves zero-shot classification performance, while ViT-B/16 gives the best overall results among tested visual backbones.The point-count study uses a resampled ModelNet40 version containing 8192 points per cloud.
5. Conclusion
PointCLIP V2 is presented as a unified 3D open-world learner that combines realistic CLIP projection with GPT-generated 3D-specific descriptions. It extends beyond classification to few-shot classification, zero-shot part segmentation, and object detection, while outdoor detection and visual grounding remain future directions.
- 5. Conclusion: PointCLIP V2 combines a realistic projection module for CLIP with GPT-3-generated 3D-specific descriptions to improve visual-language alignment.The method is designed as a unified 3D open-world learner.
- 5. Conclusion: V2 generalizes beyond classification to 3D few-shot classification, zero-shot part segmentation, and object detection with promising performance.These extensions are identified as challenging 3D open-world tasks.
- 5. Conclusion: Outdoor 3D detection and visual grounding are identified as future applications for adapting CLIP to wider open-world settings.The paper does not report these applications as completed experiments.