Source-linked AI summary
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, Silvio Savarese
TL;DR
3D recognition is limited by scarce annotated data and fixed categories, while multimodal learning for 3D remains underexplored. ULIP aligns a 3D encoder with pre-trained image-text representations using synthesized triplets, achieving state-of-the-art standard and zero-shot classification on ModelNet40 and ScanObjectNN.
Problem
3D recognition is limited by small annotated datasets and pre-defined categories, and multimodal learning involving 3D remains insufficiently studied.
Method
ULIP aligns a 3D point-cloud encoder with pre-aligned image-text features from a pre-trained vision-language model using automatically synthesized ShapeNet55 triplets.
Results
ULIP achieves state-of-the-art performance in standard and zero-shot 3D classification on ModelNet40 and ScanObjectNN.
Takeaways & Limitations
ULIP improves multiple 3D backbones and supports cross-modal applications such as image-to-point-cloud retrieval.
Takeaways & Limitations
The method freezes the image and text encoders because updating them with limited data causes catastrophic forgetting and downstream performance drops.
Abstract
from arXiv · showhide
The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of images, texts, and 3D point clouds by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models are released at https://github.com/salesforce/ULIP.
1 Salesforce AI, Palo Alto, USA 2 Stanford University, Stanford, USA 3 UT Austin, Texas, USA
The ULIP project website provides access to the project materials.
- The project website is available at tycho-xue.github.io/ULIP/.
- The website is associated with the ULIP project.
- The site offers an online project resource for ULIP.
1. Introduction
ULIP addresses limited 3D data and categories by aligning language, images, and point clouds in a unified representation. It uses pre-trained vision-language knowledge and synthesized triplets, improving standard and zero-shot 3D classification across benchmarks.
- ULIP aligns image, text, and point-cloud features by adapting a pre-trained vision-language space to 3D representations.
- 3D recognition is constrained by small datasets and pre-defined categories, with costly collection and annotation limiting generalization.
- The framework uses a small number of automatically synthesized ShapeNet55 triplets instead of requiring large-scale manually collected triplets.
- ULIP is architecture-agnostic and can be integrated with different 3D backbones for standard and zero-shot classification.
- ULIP surpasses PointMLP by around 3% in standard 3D classification on ScanObjectNN and PointCLIP by around 28.8% top-1 accuracy on ModelNet40 zero-shot classification.
2. Related Work
Prior multimodal work largely aligns images and text, while 3D methods have explored modality-specific pre-training and image-mediated transfer. ULIP instead learns a unified space spanning language, images, and point clouds.
- Image-text methods either model region-word interactions with transformers or align single image and text representations as in CLIP.
- CLIP-style alignment supports downstream directions including text-based image manipulation, open-vocabulary detection, and language grounding.
- PointCLIP converts point clouds into depth maps and applies CLIP directly for zero-shot 3D classification.
- ULIP is orthogonal to these 3D encoders while extending multimodal alignment to language, images, and point clouds.
- PointBERT and PointMAE pre-train 3D encoders by masking 3D patches and reconstructing or predicting missing tokens.
3. Learning a Unified Representation of Language, Images, and Point Clouds
ULIP constructs image-text-point-cloud triplets from ShapeNet55 and trains a 3D encoder against frozen vision-language features. Contrastive losses align modalities, after which the encoder supports downstream classification.
- Creating Training Triplets for ULIP: ULIP pre-trains on triplets containing an image, text description, and point cloud for each ShapeNet55 CAD model.
- Creating Training Triplets for ULIP: ShapeNet55 provides around 52.5K CAD models with metadata describing each model’s semantic information.
- Creating Training Triplets for ULIP: Multi-view rendering produces 30 RGB images and 30 depth maps per object, with one candidate randomly selected during each pre-training iteration.
- Creating Training Triplets for ULIP: Text features are generated from CAD metadata using prompt templates, including a dedicated prompt for point-cloud models.
- Aligning Representations of Three Modalities: A frozen CLIP image-text space supervises the 3D encoder, whose features are aligned to image and text features through contrastive losses.
- Aligning Representations of Three Modalities: The final objective combines image-text, image-point-cloud, and point-cloud-text losses, while only the 3D encoder is updated by default.
4. Experiments
ULIP is evaluated across standard and zero-shot 3D classification using several backbones and datasets, with analyses of data efficiency, modality alignment, and cross-modal retrieval. It improves baseline recognition and supports multimodal 3D applications.
- Experimental settings: Experiments use PointNet++, PointBERT, and PointMLP on ModelNet40 and ScanObjectNN for standard and zero-shot 3D classification.ModelNet40 contains synthetic CAD models, while ScanObjectNN contains real-world scanned objects.
- Experimental settings: ULIP pre-trains 3D networks before fine-tuning while leaving the backbone structure unchanged, so it introduces no extra inference latency.The framework follows the original baseline settings apart from ULIP pre-training.
- Standard 3D Classification: ULIP improves PointBERT and PointMLP by around 3% on ScanObjectNN, while ULIP+PointMLP† surpasses RepSurf-U(2×) by 3.4% Overall Accuracy.ULIP+PointMLP† achieves new state-of-the-art performance on this benchmark.
- Standard 3D Classification: ULIP improves Overall Accuracy for all evaluated baselines on ModelNet40, with ULIP+PointMLP* achieving new state-of-the-art performance and improved class-mean accuracy.ModelNet40 performance is described as saturated around 94% for recent methods.
- Zero-Shot 3D Classification: ULIP-based methods outperform PointCLIP in zero-shot 3D classification, including by around 29% top-1 accuracy on the Hard ModelNet40 set.The gains also persist on real scanned objects in ScanObjectNN, where all backbones outperform PointCLIP by approximately 30%.
- Analyses: Aligning 3D representations with both text and image representations consistently outperforms alignment with either modality alone across baselines and datasets.The framework also shows improved low-data performance and meaningful image-to-point-cloud retrieval, including fine-grained aircraft differences.
5. Conclusions
ULIP aligns images, language, and point clouds in one feature space to improve 3D backbones. It achieves state-of-the-art results in standard and zero-shot 3D classification, while showing potential for cross-modal retrieval.
- ULIP aligns image, language, and point-cloud representations in a shared feature space.
- ULIP improves the representations and recognition performance of different 3D backbone models.
- ULIP achieves state-of-the-art performance in both standard and zero-shot 3D classification tasks.
- ULIP shows promising potential for cross-modal retrieval applications.
A.1. PointNeXt Backbone Experiments
The authors test ULIP with the lightweight PointNeXt backbone by pre-training it with ULIP and fine-tuning on ScanObjectNN. ULIP significantly improves both reported accuracy metrics.
- PointNeXt is a concurrent lightweight backbone based on PointNet++ with promising ScanObjectNN results.
- ULIP pre-trains PointNeXt, then uses the resulting weights for ScanObjectNN fine-tuning.
- ULIP significantly improves PointNeXt in both Overall Accuracy and Class-mean Accuracy.
A.2. Details of Evaluation Sets in Zero Shot Classification
The zero-shot ModelNet40 evaluation uses increasingly strict validation sets to address overlap between ShapeNet55 pre-training categories and target categories. The sets remove exact names, then exact names and synonyms.
- All Set: The All Set includes every ModelNet40 category for zero-shot evaluation.
- Medium Set: The Medium Set removes categories whose exact names occur in the ShapeNet55 pre-training dataset.
- Hard Set: The Hard Set removes categories whose exact names or synonyms occur in the ShapeNet55 pre-training dataset.
A.3. Indoor 3D Detection Experiments
ULIP is also evaluated for indoor 3D detection using the Group-Free-3D basic model on ScanNet-v2. The authors observe significant improvements in this setting.
- Indoor 3D detection is evaluated on ScanNet-v2 using the Group-Free-3D basic model.
- The Group-Free-3D basic model shows significant improvements when using ULIP in the reported setting.