Source-linked AI summary
CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP
Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, Wenping Wang
TL;DR
3D scene understanding lacks the broad semantic knowledge and annotation efficiency demonstrated by CLIP in 2D. CLIP2Scene transfers CLIP image-text knowledge to a 3D point-cloud network using semantic and spatial-temporal consistency regularization, achieving annotation-free segmentation and stronger label-efficient fine-tuning. The paper also reports cross-domain generalization, while noting false-positive predictions around ground-truth objects.
Problem
3D methods rely heavily on expensive point-cloud annotations and typically fail to recognize novel objects absent from training data.
Method
CLIP2Scene pre-trains a 3D network with semantic-driven cross-modal contrastive learning, using CLIP text semantics and image features for semantic and spatial-temporal consistency regularization.
Results
20.8% and 25.08% mIoU are achieved for annotation-free 3D semantic segmentation on nuScenes and ScanNet, while fine-tuning improves mIoU by 8% and 1% with 1% and 100% labelled data, respectively.
Takeaways & Limitations
CLIP knowledge can be transferred to a 3D network for annotation-free scene segmentation and improved label-efficient fine-tuning across indoor, outdoor, and cross-domain evaluations.
Takeaways & Limitations
The method observes false-positive predictions around ground-truth objects, an issue left for future work.
Abstract
from arXiv · showhide
Contrastive Language-Image Pre-training (CLIP) achieves promising results in 2D zero-shot and few-shot learning. Despite the impressive performance in 2D, applying CLIP to help the learning in 3D scene understanding has yet to be explored. In this paper, we make the first attempt to investigate how CLIP knowledge benefits 3D scene understanding. We propose CLIP2Scene, a simple yet effective framework that transfers CLIP knowledge from 2D image-text pre-trained models to a 3D point cloud network. We show that the pre-trained 3D network yields impressive performance on various downstream tasks, i.e., annotation-free and fine-tuning with labelled data for semantic segmentation. Specifically, built upon CLIP, we design a Semantic-driven Cross-modal Contrastive Learning framework that pre-trains a 3D network via semantic and spatial-temporal consistency regularization. For the former, we first leverage CLIP's text semantics to select the positive and negative point samples and then employ the contrastive loss to train the 3D network. In terms of the latter, we force the consistency between the temporally coherent point cloud features and their corresponding image features. We conduct experiments on SemanticKITTI, nuScenes, and ScanNet. For the first time, our pre-trained network achieves annotation-free 3D semantic segmentation with 20.8% and 25.08% mIoU on nuScenes and ScanNet, respectively. When fine-tuned with 1% or 100% labelled data, our method significantly outperforms other self-supervised methods, with improvements of 8% and 1% mIoU, respectively. Furthermore, we demonstrate the generalizability for handling cross-domain datasets. Code is publicly available https://github.com/runnanchen/CLIP2Scene.
1. Introduction
CLIP2Scene transfers CLIP’s 2D image-text knowledge to 3D scene understanding through semantic and spatial-temporal consistency regularization. It enables annotation-free segmentation and improves label-efficient fine-tuning.
- Motivation: 3D scene understanding remains annotation-intensive and typically cannot recognize novel objects absent from training data.High-quality 3D annotations are expensive, while novel-object recognition may require additional labeling.
- Method: Semantic consistency uses CLIP text semantics to select less-conflicting positive and negative point samples for contrastive learning.This addresses optimization conflict in prior cross-modal knowledge distillation methods.
- Results: 20.8% and 25.08% mIoU are achieved for annotation-free 3D semantic segmentation on nuScenes and ScanNet, respectively.These results use no labelled training data.
- Results: 8% and 1% mIoU improvements are obtained over state-of-the-art self-supervised methods with 1% and 100% labelled nuScenes data, respectively.The work also reports strong cross-domain evaluation on SemanticKITTI.
- Method: CLIP2Scene pre-trains a 3D point cloud segmentation network with semantic-driven cross-modal contrastive learning.The framework transfers CLIP knowledge through semantic and spatial-temporal consistency regularization.
- Method: Spatial-temporal consistency aligns temporally coherent point-cloud features with corresponding image features.The regularization exploits inter-sweep correspondence in multi-sweep point clouds.
2. Related Work
Prior work studies zero-shot 3D recognition and cross-modal knowledge transfer, while CLIP2Scene extends CLIP-based transfer to 3D scene understanding. Its framework combines CLIP-derived semantic samples with point-image consistency regularization.
- Zero-shot Learning in 3D: Zero-shot 3D learning targets recognition of objects unseen during training, with prior work addressing hubness, transductive settings, and segmentation.PointCLIP applies CLIP to zero-shot and few-shot 3D classification, whereas CLIP2Scene goes further toward 3D scene understanding.
- CLIP2Scene: CLIP2Scene extracts text, image-pixel, and point features before constructing CLIP-guided positive and negative samples.The resulting pairs support semantic and spatial-temporal consistency regularization.
- CLIP2Scene: Point features are pulled toward corresponding text embeddings, while temporally coherent point features mimic corresponding pixel features.These operations implement the framework’s two consistency regularizers.
- Cross-modal Knowledge Distillation: Cross-modal distillation methods transfer image knowledge to point clouds using techniques such as InfoNCE and superpixels.PPKT and SLidR are described as representative image-to-point self-supervised methods.
3. Methodology
CLIP2Scene transfers CLIP’s 2D image-text knowledge to a 3D point-cloud network through semantic and spatial-temporal consistency regularization. It addresses cross-modal optimization conflicts and noisy pixel-point correspondence while exploiting temporal coherence.
- Framework: CLIP2Scene transfers CLIP knowledge to a 3D network for annotation-free and fine-tuned 3D scene understanding.The framework combines Semantic Consistency Regularization, Semantic-guided Spatial-Temporal Consistency Regularization, and Switchable Self-Training Strategy.
- Semantic Consistency Regularization: Previous cross-modal methods can treat same-instance points as negatives, creating optimization conflicts that impair representation learning.CLIP2Scene instead uses text semantics to reduce this conflict when constructing contrastive samples.
- Semantic Consistency Regularization: Semantic Consistency Regularization uses CLIP-derived pixel-text and point-text pairs to select contrastive positive and negative samples.The contrastive objective pulls point features toward corresponding class text embeddings and pushes features from different classes away.
- Semantic-guided Spatial-Temporal Consistency Regularization: Semantic-guided spatial-temporal regularization maps temporally coherent LiDAR sweeps to images and constrains point features within local grids.The method registers later sweeps to the first frame, places temporally coherent points in shared grids, and forms fusion features from image and point features.
- Semantic-guided Spatial-Temporal Consistency Regularization: Each local grid generates a dynamic fusion center from image and point features, with CLIP text similarity determining their soft attention weights.This soft constraint is intended to alleviate noisy image semantics, imperfect calibration, and noisy prediction issues while regularizing temporally coherent features.
4. Experiments
Experiments evaluate CLIP2Scene on outdoor and indoor LiDAR datasets for annotation-free segmentation, limited-label fine-tuning, ablations, and cross-domain generalization. The framework uses CLIP features with semantic and spatial-temporal regularization, and its performance is examined qualitatively and against self-supervised baselines.
- Experimental setup: Experiments use SemanticKITTI, nuScenes, and ScanNet, with nuScenes pre-training on keyframes from 600 scenes.nuScenes has 16 LiDAR segmentation classes; SemanticKITTI has 19 classes.
- Ablation studies: Removing spatial-temporal consistency causes a dramatic performance decrease, while image-only supervision and self-prediction training both reduce performance.The ablations also report that direct KL-divergence image-feature distillation fails to transfer semantic information because of noisy pixel semantics and imperfect calibration.
- Limitations: False-positive predictions remain around ground-truth objects in annotation-free segmentation results.The authors state that they will address this issue in future work.
5. Conclusion
CLIP2Scene transfers CLIP image-text knowledge to a 3D network through semantic and spatial-temporal regularization. The resulting pre-trained network supports annotation-free 3D semantic segmentation and significantly outperforms state-of-the-art self-supervised methods after fine-tuning with labelled data.
- CLIP2Scene transfers CLIP image and text features to a 3D network through Semantic Regularization and Spatial-Temporal Regularization.
- The pre-trained network achieves annotation-free 3D semantic segmentation and significantly outperforms state-of-the-art self-supervised methods when fine-tuned with labelled data.