PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP

Junbo Zhang, Runpei Dong, Kaisheng Ma

arXiv:2303.04748v2cs.CV

TL;DR

3D scene understanding relies on laborious human annotations, and existing benchmarks may provide only close-set object semantics. CLIP-FO3D transfers CLIP features to 3D scenes without supervision, reporting annotation-free open-vocabulary segmentation and stronger zero-shot and data-efficient results than prior state-of-the-art methods.

  • Problem

    Training 3D scene understanding models requires laborious human annotations, while current benchmarks can contain only close-set object semantics.

    Source: §1 · p. 1

  • Method

    CLIP-FO3D extracts dense CLIP pixel features, projects multi-view features to 3D points, and trains a 3D model with feature distillation without supervision.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    CLIP-FO3D achieves annotation-free open-vocabulary segmentation and outperforms previous state-of-the-art methods in zero-shot and data-efficient learning.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The resulting 3D representations inherit CLIP’s open-world knowledge, allowing recognition beyond object concepts.

    Source: §6 · p. 9

Abstract

from arXiv · show

Training a 3D scene understanding model requires complicated human annotations, which are laborious to collect and result in a model only encoding close-set object semantics. In contrast, vision-language pre-training models (e.g., CLIP) have shown remarkable open-world reasoning properties. To this end, we propose directly transferring CLIP's feature space to 3D scene understanding model without any form of supervision. We first modify CLIP's input and forwarding process so that it can be adapted to extract dense pixel features for 3D scene contents. We then project multi-view image features to the point cloud and train a 3D scene understanding model with feature distillation. Without any annotations or additional training, our model achieves promising annotation-free semantic segmentation results on open-vocabulary semantics and long-tailed concepts. Besides, serving as a cross-modal pre-training framework, our method can be used to improve data efficiency during fine-tuning. Our model outperforms previous SOTA methods in various zero-shot and data-efficient learning benchmarks. Most importantly, our model successfully inherits CLIP's rich-structured knowledge, allowing 3D scene understanding models to recognize not only object concepts but also open-world semantics.

1. Introduction

The paper targets the gap between laborious, close-set 3D annotation and CLIP’s open-world representations. It transfers CLIP features into 3D scenes without supervision and reports annotation-free open-vocabulary segmentation plus stronger zero-shot and data-efficient benchmark performance.

1. Introduction

  • 3D benchmarks can contain only close-set labels, such as 20 ScanNet classes [14], while CLIP supports open-vocabulary reasoning and transfer to 3D remains understudied.Traditional 3D scene understanding also requires laborious human annotations.Source: §1 · p. 1
  • The method extracts local pixel-level CLIP features, projects multi-view image features to 3D points, and distills those targets into a 3D scene model.The transfer uses no 2D/3D annotations or vision-language grounding annotations.Source: §1 · p. 2
  • The authors present CLIP-FO3D as an unsupervised cross-modal pre-training framework that benefits data efficiency when annotations are limited.They report outperforming previous state-of-the-art methods in zero-shot and data-efficient learning.Source: §1 · p. 2
  • CLIP-FO3D inherits CLIP’s open-world knowledge, enabling 3D models to understand text queries beyond object concepts.The paper frames preserving this knowledge as a reason to avoid training with human annotations, which it says restrict the feature space to limited vocabularies.Source: §1 · p. 2
  • CLIP-FO3D achieves annotation-free open-vocabulary segmentation on large vocabularies and long-tailed categories, and outperforms prior state-of-the-art methods in zero-shot and data-efficient learning.The reported segmentation includes NYU-40 classes and categories beyond the NYU label set.Source: §1 · p. 2

2. Related Work

Prior work in 2D dense prediction and 3D open-vocabulary recognition often relies on human annotations, while 3D representation learning also uses cross-modal distillation. This section positions CLIP-FO3D as unsupervised cross-modal 3D representation learning.

2.1. Open-vocabulary Dense Prediction in 2D Vision

  • 2D open-vocabulary dense prediction methods use annotations or distill foundation-model knowledge, while MaskCLIP applies CLIP without training but struggles with complex 3D scene views.CLIP-FO3D instead modifies CLIP’s inputs and forwarding process to extract free pixel-level features from 3D scene RGB views.Source: §2.1 · p. 2, Figure 2 · p. 3

2.2. Zero-shot 3D Visual Recognition

  • Zero-shot 3D scene methods have used supervised seen-category labels or human annotations, whereas CLIP-FO3D transfers CLIP knowledge without annotations.The cited approaches include segmentation with generative models or word-embedding prototypes, detection with pseudo-labels, and caption-based alignment.Source: §2.2 · p. 3

2.3. 3D Representation Learning

  • CLIP-FO3D is framed as unsupervised cross-modal 3D representation learning that distills CLIP’s vision knowledge and benefits scene understanding when annotations are limited.The section contrasts this with 3D pre-training based on contrastive learning, masked auto-encoding, or their combination.Source: §2.3 · p. 3

3. Method

CLIP-FO3D adapts CLIP to produce local pixel features, projects them onto 3D points, and trains a 3D model by feature distillation. The resulting representations remain aligned with CLIP’s feature space and support open-world 3D understanding without supervision.

3.1. Extract Free Pixel-level CLIP Features

  • Because MaskCLIP features are low-resolution and globally mixed, CLIP-FO3D uses multi-scale crops and super-pixels to target objects and parts at varied scales.Super-pixels are generated with SLIC and provide locally visually similar regions.Source: §3.1 · p. 4
  • Added local classification tokens attend only to patches assigned to their super-pixel, leaving original ViT tokens unchanged; each local feature is applied to every pixel in that region.The local token feature map is produced at the cropped image’s input size.Source: §3.1 · p. 4
  • Features from overlapping crops are resized, stitched, and averaged per pixel, then extracted once per RGB view as offline training targets rather than real-time outputs.The feature extraction process is described as slow and difficult to apply to real-time inference.Source: §3.1 · p. 4

3.2. Feature Distillation with 2D Teacher

  • Each 3D point gathers projected pixel features across RGB views, with depth filtering from RGB-D data, and averages them into a CLIP-aligned target feature.Projection uses camera pose, intrinsics, and the world-to-grid transformation matrix.Source: §3.2 · p. 4
  • CLIP-FO3D trained by feature distillation outperforms its Target Feature, whose pixel features capture better semantics than MaskCLIP, on annotation-free ScanNet and S3DIS segmentation.The table caption reports these qualitative comparisons but supplies no metric values.Source: Table 1 · p. 5
  • The 3D model learns these targets by feature distillation, enabling open-vocabulary segmentation and open-world reasoning after training without 2D, 3D, or grounding annotations.The distillation distance is negative cosine similarity, consistent with CLIP-driven classification’s use of cosine distance.Source: §3.2 · p. 5

4. Experiments

The experiments evaluate annotation-free segmentation, zero-shot transfer, and data-efficient fine-tuning on ScanNet and S3DIS. Results include gains over prior methods and training from scratch, alongside segmentation of open-vocabulary and long-tailed categories.

4.1. Experimental Setup

  • Training uses unlabeled ScanNet RGB-D images and meshes, with evaluation on ScanNet validation and S3DIS Area 5.Source: §4.1 · p. 5

4.3. Zero-shot Semantic Segmentation

  • 10.1%, 18.3%, 33.8%, 35.0%, and 34.7% hIoU gains over [7] on ScanNet with 2, 4, 6, 8, and 10 unseen classes, respectively.For zero-shot use, CLIP-FO3D can generate pseudo-labels for unlabeled points.Source: Table 4 · p. 6, §4.3 · p. 7

4.2. Annotation-free Semantic Segmentation

  • On ScanNet and S3DIS, CLIP-FO3D segments without training labels and outperforms its target features on ScanNet, correcting some misleading targets through distillation.Source: §4.2 · pp. 5–6
  • Target Features outperform MaskCLIP-3D on the extended NYU-40 vocabulary, which is divided into Head, Common, and Tail categories.Source: Table 2 · p. 5, §4.2 · p. 6
  • CLIP-FO3D segments ten frequent raw ScanNet categories beyond NYU-40; qualitative masks also cover long-tailed labels and differ on ambiguous object pairs.Source: §4.2 · p. 6, Figure 4 · p. 7

4.4. Data-efficient 3D Scene Understanding

  • With 20, 50, 100, and 200 randomly sampled annotated points per scene, the method improves ScanNet mIoU over training from scratch by 11.3%, 6.0%, 5.4%, and 4.1%, respectively.Source: Table 7 · p. 7, §4.4 · p. 8

5. Discussion

The discussion examines CLIP-FO3D's open-world scene representations and how text queries can retrieve semantically relevant 3D regions. The authors connect this capacity to CLIP's real-world knowledge and describe a possible robotics application.

Ablation study

  • Table 8 ablates multi-scale inputs and local classification tokens on annotation-free and four-unseen-class zero-shot segmentation benchmarks.Source: §5 · p. 8

Open-world 3D scene understanding

  • Text queries are encoded by CLIP, matched by similarity to point features, and thresholded into 3D masks; “write” retrieves a whiteboard and desk.Source: §5 · p. 8
  • The visualizations also indicate that the representations capture affordance, activity, color, and function words, not only object concepts.Source: §5 · p. 8
  • The authors state that annotation-trained models can recognize object concepts, whereas annotation-free CLIP-FO3D preserves CLIP's open-world properties; language models may extend robot functionality.Source: §5 · p. 8

6. Conclusion

CLIP-FO3D transfers CLIP features to 3D models without supervision, projecting pixel-level features from RGB views onto points for feature-distillation training. It achieves annotation-free segmentation results and outperforms prior methods on zero-shot and data-efficient tasks while retaining open-world knowledge.

  • CLIP-FO3D transfers CLIP’s feature space to 3D without supervision, projecting pixel-level RGB-view features onto points and training a 3D model with feature distillation.Source: §6 · p. 9
  • The model achieves annotation-free semantic segmentation on standard benchmarks and open-vocabulary concepts, and outperforms prior methods in zero-shot and data-efficient learning tasks.Source: §6 · p. 9
  • CLIP-FO3D inherits CLIP’s open-world properties, enabling 3D models to encode knowledge beyond object concepts.Source: §6 · p. 9

A. Advantages of Annotation-free Training

The section contrasts annotation-free transfer with approaches trained on human-labeled categories: annotations can restrict the feature space, while CLIP-FO3D retains open-world concepts. Its advantages include recognizing long-tailed categories and queries such as colors and affordances, although it performs less well than LSeg on some common benchmark categories.

A. Advantages of Annotation-free Training

  • Annotation-free training is presented as preserving CLIP’s rich information, whereas human annotations restrict the feature space and can cause open-world knowledge to be forgotten during fine-tuning [32].Source: A. Advantages of Annotation-free… · p. 12
  • CLIP-FO3D performs less well than LSeg on some common benchmark categories, but the authors report advantages in recognizing long-tailed categories and open-world knowledge such as color and affordance.LSeg is trained on seven segmentation datasets with human annotations covering hundreds of categories; the authors characterize its feature space as restricted to those concepts.Source: A. Advantages of Annotation-free… · p. 12
  • For long-tailed categories absent from LSeg’s pretraining datasets, CLIP-FO3D highlights target regions while LSeg produces non-discriminative results.The examples include yogurt, milk, and crisps, as well as proper names such as Gryffindor and Minion.Source: A. Advantages of Annotation-free… · pp. 12–13
  • CLIP-FO3D’s pixel features also encode color and affordance queries, which the authors say are absent from LSeg’s feature space because those queries are irrelevant to its pretrained object concepts.Source: A. Advantages of Annotation-free… · p. 13
  • The authors distinguish their fully annotation-free approach from a concurrent zero-shot 3D method that uses LSeg trained with extensive human annotations [49].Source: A. Advantages of Annotation-free… · p. 13

B. Implementation Details

Implementation uses multi-scale image crops and local super-pixel features to obtain CLIP representations, with ablations removing either multi-scale processing or local features. Class predictions use averaged CLIP embeddings from hand-crafted prompts.

B. Implementation Details

  • Multi-scale region extraction uses sliding crop windows, and images are partitioned into 50 SLIC super-pixels to obtain locally similar regions representing objects or object parts.The authors use fewer super-pixels than SLidR because multi-scale cropping has already improved resolution.Source: B. Implementation Details · p. 13
  • The ablation without multi-scale processing removes region extraction and feature fusion, while the ablation without local features replaces local CLIP extraction with MaskCLIP.Source: B. Implementation Details · p. 13
  • For point classification, the method averages CLIP embeddings from 80 hand-crafted prompts per class and uses each resulting text embedding as a classification weight.Source: B. Implementation Details · p. 13

C. Dataset Partitioning

The experiments partition ScanNet’s extended vocabulary into head, common, and tail groups after removing labels without specific semantics. Zero-shot tests use designated unseen classes in ScanNet and S3DIS, including a separate six-class S3DIS benchmark.

C. Dataset Partitioning

  • The ScanNet vocabulary is extended with NYU-40 labels, excluding labels without specific semantics, and the remaining categories are divided evenly into Head, Common, and Tail groups.Source: C. Dataset Partitioning · p. 13
  • ScanNet zero-shot tests designate 2, 4, 6, 8, or 10 unseen classes, with the specific categories listed for each setting.Source: C. Dataset Partitioning · p. 13
  • S3DIS tests include an established setting with beam, column, window, and sofa unseen, plus a six-class setting previously used by PLA.The six classes are board, door, floor, sofa, table, and window.Source: C. Dataset Partitioning · p. 13

D. More Qualitative Results

The section presents additional qualitative results for annotation-free semantic segmentation and open-world 3D scene understanding on ScanNet.

Loading 2303.04748v2…