PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenScene: 3D Scene Understanding with Open Vocabularies

Songyou Peng, Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser

arXiv:2211.15654v2cs.CV

TL;DR

Traditional 3D scene understanding is usually trained with labeled data for individual tasks, limiting open-ended queries. OpenScene learns dense 3D features aligned with CLIP text and image features, enabling zero-shot open-vocabulary understanding; it achieves strong segmentation results and supports diverse queries without labeled 3D data.

  • Problem

    Traditional 3D systems use task-specific supervision and closed label sets, providing limited support for rare objects and queries without 3D supervision.

  • Method

    OpenScene aligns dense 3D point features with CLIP text and image features using multi-view fusion, 3D feature distillation, and 2D-3D ensembling.

  • Results

    OpenScene achieves state-of-the-art zero-shot 3D semantic segmentation, outperforms supervised methods for many-class segmentation, and enables open-vocabulary scene queries without labeled 3D data.

  • Takeaways & Limitations

    A single task-agnostic model supports queries about objects, materials, affordances, activities, and room types across 3D scenes.

  • Takeaways & Limitations

    OpenScene provides mainly qualitative results for non-segmentation tasks because ground-truth 3D benchmarks are scarce.

Abstract

from arXiv · show

Traditional 3D scene understanding approaches rely on labeled 3D datasets to train a model for a single task with supervision. We propose OpenScene, an alternative approach where a model predicts dense features for 3D scene points that are co-embedded with text and image pixels in CLIP feature space. This zero-shot approach enables task-agnostic training and open-vocabulary queries. For example, to perform SOTA zero-shot 3D semantic segmentation it first infers CLIP features for every 3D point and later classifies them based on similarities to embeddings of arbitrary class labels. More interestingly, it enables a suite of open-vocabulary scene understanding applications that have never been done before. For example, it allows a user to enter an arbitrary text query and then see a heat map indicating which parts of a scene match. Our approach is effective at identifying objects, materials, affordances, activities, and room types in complex 3D scenes, all using a single model trained without any labeled 3D data.

1. Introduction

OpenScene addresses the limited query diversity of supervised, task-specific 3D systems by aligning dense 3D point features with text and image features in CLIP space. This enables zero-shot open-vocabulary queries across objects, materials, affordances, functions, and physical properties.

  • Motivation: Traditional 3D systems use task-specific supervision and closed label sets, limiting performance on rare objects and queries without 3D supervision.These systems commonly target predefined categories such as 20-class semantic segmentation.
  • Method: OpenScene co-embeds dense 3D point features with text strings and image pixels in the CLIP feature space.It associates points with pixels from posed images and trains a 3D network using CLIP pixel features as supervision.
  • Method: The method fuses multi-view pixel features, distills them into a sparse 3D convolutional network, and ensembles the resulting 2D and 3D features per point.The 3D network extracts features from point-cloud geometry while minimizing differences from aggregated pixel features.
  • Applications: Open-vocabulary similarity queries can identify semantics, materials, affordances, functions, and physical properties across 3D scenes.For example, the query “soft” highlights couches, beds, and comfy chairs.
  • Results: The zero-shot approach outperforms fully supervised methods for rare classes and for indoor segmentation with 40, 80, or 160 classes.It also performs better than other zero-shot baselines and transfers to datasets with different label sets without retraining.

2. Related Work

Prior 3D scene understanding largely relies on labeled data and predefined categories, while related 2D and zero-shot approaches retain supervision or limited scene scope. OpenScene instead uses pretrained open-vocabulary 2D features to support broader 3D scene-level queries without labeled 2D or 3D training data.

  • Closed-set 3D Scene Understanding: Prior closed-set 3D methods use ground-truth labels for tasks including classification, detection, localization, and segmentation.A closely related CLIP-pretrained method still trains its point decoder with 3D annotations and targets supervised segmentation.
  • 2D-supervised 3D Understanding: Methods using 2D supervision commonly backproject predicted labels or contrast point-pixel features, but their 2D networks may require ground-truth 2D labels.OpenScene also uses image features as pseudo-supervision, without requiring 2D labels.
  • Open-vocabulary 3D Understanding: Existing approaches mainly use small predefined label sets and weaken as category counts increase because tail classes have few training examples.OpenScene segments arbitrary label sets without retraining and covers rare objects, materials, and physical properties.
  • Open-Vocabulary 2D Scene Understanding: Open-vocabulary 2D embeddings are often image-level and therefore unsuitable for dense pixel-level prediction without additional dense-feature methods.Recent 2D work attempts to correlate dense image features with language embeddings.
  • Zero-shot Learning for 3D Point Clouds: Prior open-vocabulary 3D scene methods include small partial scenes requiring ground-truth training data or zero-shot point-cloud segmentation trained on predefined seen classes.OpenScene relies solely on pretrained open-vocabulary 2D models for scene-level tasks without ground-truth 2D or 3D training data.

3. Method

OpenScene builds per-point features by fusing multi-view 2D visual-language embeddings and distilling them into a 3D network, aligning both with CLIP text features. It then selects or ensembles these representations for arbitrary text-driven 3D scene understanding.

  • 3.1. Image Feature Fusion: OpenScene extracts per-pixel embeddings from pretrained open-vocabulary 2D segmentation models and back-projects them onto 3D scene points.The frozen image encoder produces dense embeddings for each RGB image before 2D–3D pairing.
  • 3.1. Image Feature Fusion: It associates each 3D point with visible pixels across views and average-pools their features into one fused 2D representation.The method uses camera calibration for projection and depth-based occlusion tests where depth images are available.
  • 3.2. 3D Distillation: A 3D encoder maps the input point cloud to per-point embeddings and is trained with cosine similarity loss to reproduce the fused 2D features.The 3D backbone is MinkowskiNet18A, with output dimensionality changed to the feature dimension C.
  • 3.2. 3D Distillation: The distilled 3D features remain in CLIP’s embedding space, enabling arbitrary text prompts for 3D scene understanding without 2D observations.This transfers open-vocabulary image-language knowledge into a point network that uses only 3D point positions as input.
  • 3.3. 2D-3D Feature Ensemble: OpenScene ensembles fused 2D and distilled 3D features by comparing each with text embeddings and selecting the feature with the highest ensemble score.The 2D representation is useful for small or geometrically ambiguous objects, while the 3D representation helps with objects having distinctive shapes.
  • 3.3. 2D-3D Feature Ensemble: For zero-shot semantic segmentation, each 3D point receives the text-prompt label with the maximum cosine similarity to the ensemble feature.The same similarity mechanism is used across the paper’s scene understanding tasks.

4. Experiments

OpenScene is evaluated on closed-set and zero-shot 3D semantic segmentation across indoor and outdoor benchmarks, including increasingly large label sets. It outperforms prior zero-shot methods, handles rare classes well, and benefits from combining 2D and 3D features.

  • Experimental Setup: OpenScene is evaluated on ScanNet, Matterport3D, and nuScenes, spanning indoor RGB-D meshes and outdoor LiDAR scenes.Matterport3D is also used to stress open-vocabulary queries.
  • Zero-Shot Segmentation: 62.8% mIoU versus 7.7% mIoU: OpenScene significantly outperforms 3DGenZ on four unseen ScanNet classes without labeled 2D or 3D data.It also outperforms MSeg Voting, with feature regression preserving class similarities that classify-then-vote methods discard.
  • Benchmark Comparisons: OpenScene outperforms the zero-shot baseline across all three datasets and comes closest to supervised state of the art on Matterport3D.The reported Matterport3D gaps are -11.6 mIoU and -8.0 mAcc.
  • Scaling Class Vocabulary: As the label set grows from 21 to 160 classes, the zero-shot method overtakes fully supervised training, especially for less-common long-tail classes.OpenScene uses one class-agnostic model for all K values, whereas the supervised baseline trains separately for each label set.
  • Ablation Studies: The full 2D-3D ensemble performs best across datasets, metrics, and 2D feature choices, indicating that both domains contribute robust descriptive information.OpenSeg generally provides better accuracy and generalizability than LSeg.
  • Ablation Studies: Approximately 70% of predictions select 3D features, while 2D feature usage increases with the number of classes and supports long-tailed categories.The analysis associates 2D features with smaller objects and classes having fewer training examples.

5. Applications

OpenScene extends 3D scene understanding beyond fixed labels through zero-shot text and image queries. It supports object search, image-based detection, and exploration of materials, properties, affordances, activities, and room-related concepts without labeled 3D data.

  • Applications: OpenScene’s co-embedded 3D features enable arbitrary text and image queries for open-vocabulary scene understanding applications.The applications described are explicitly zero-shot and use no labeled data from 3D scene-understanding datasets.
  • Open-Vocabulary 3D Object Search: Text-query retrieval ranks Matterport3D points by cosine similarity to a CLIP text embedding, returning the best matching scene regions.The evaluation samples ten raw categories and visually verifies top matches for precision.
  • Open-Vocabulary 3D Object Search: Open-vocabulary retrieval identifies relatively rare object classes at least as well as the manual ground-truth labeling process in the reported comparison.Some top matches absent from the ground truth included 13 telephones.
  • Image-Based 3D Object Detection: Image-based 3D detection embeds internet images with CLIP, compares them with 2D-3D point features, and thresholds similarities to produce detections and masks.Figure 7 illustrates matching queried images to 3D points in a scene.
  • Open-Vocabulary Scene Exploration: Scene exploration uses CLIP’s broader concept space to query physical properties, materials, affordances, functions, and related scene information beyond category labels.Examples include “Comfy” matching beds and couches, “Hot” matching ovens and fireplaces, and “Play” matching a piano keyboard.
  • Applications: The paper presents these open-vocabulary applications as a key result because they operate without massive 3D labeling efforts.The applications include text-based search, image-based retrieval, and scene exploration.

6. Limitations and Future Work

OpenScene demonstrates strong zero-shot and open-vocabulary performance, while the authors identify limitations in inference and evaluation coverage. Future work should broaden quantitative assessment beyond closed-set semantic segmentation.

  • OpenScene achieves state-of-the-art zero-shot 3D semantic segmentation and outperforms supervised approaches with many class labels without labeled 3D data.
  • The inference algorithm could better exploit pixel features available at test time through earlier fusion, which had limited success.
  • Evaluation is extensive for closed-set 3D semantic segmentation but only qualitative for other open-vocabulary tasks because ground-truth 3D benchmarks are scarce.
  • Future work should quantify open-vocabulary query success on tasks without existing ground truth.

—Supplementary Material—

The supplementary document provides implementation details, additional method investigations, and expanded experimental results for open-vocabulary retrieval and scene exploration.

  • The listed authors are affiliated with Google Research, ETH Zurich, MPI for Intelligent Systems, Waymo LLC, and Simon Fraser University.
  • The supplement first describes implementation details and then presents additional investigations of the methods.
  • It includes full results for open-vocabulary object retrieval experiments.
  • It also provides additional open-vocabulary scene exploration results.

A. Implementation Details

The supplementary implementation uses multi-view feature fusion, depth-based visibility checks where available, dataset-specific projections, and standard training choices for 3D distillation.

  • 3D distillation uses Adam with an initial learning rate of 1e−4 for 100 epochs, with dataset-specific voxel sizes.
  • Matterport3D and nuScenes use all scene images for fusion, whereas ScanNet samples one of every 20 video frames.
  • ScanNet and Matterport3D apply depth-based occlusion tests, using σ = 0.2D and σ = 0.02D respectively.
  • nuScenes performs no occlusion test because depth images are unavailable and uses synchronized images with corresponding LiDAR points from the last timestamp of each 0.5 second segment.
  • For nuScenes, most LiDAR points receive one projected pixel feature, while points with two views use average pooling across 16 benchmark classes.
  • The implementation maps MSeg predictions to dataset label sets and uses majority voting for multi-view surface-point predictions.
  • Task-agnostic models trained on ScanNet transfer to Matterport3D label sets, and vice versa, without retraining.
  • Prompting class names as “a XX in a scene” yields a +2.3 mIoU boost with the LSeg ensemble model on ScanNet.

B. Additional Analysis

Additional analyses examine cross-dataset transfer, multi-view fusion, feature selection, and the scope of zero-shot learning in 3D.

  • Domain Transfer with Open Vocabularies: Models trained on one dataset transfer directly to another dataset with different label sets because they predict CLIP features rather than fixed classes.
  • Ablation on multi-view fusion strategy: Average pooling performs best among the evaluated multi-view fusion strategies and is used throughout the experiments.
  • Visualization of our 2D-3D ensemble model: The ensemble uses 3D features for large areas and favors 2D features for smaller objects and finer-grained concepts.
  • Visualization of our 2D-3D ensemble model: As the label count increases from 21 to 160 classes, the ensemble selects more 2D features; a pool table illustrates this finer-grained advantage.
  • Definition of Zero-Shot Learning (ZSL): In practical zero-shot learning, general-purpose feature extractors pretrained on large datasets are commonly used despite stricter theoretical definitions excluding seen-class training data.

C. Full Results of Open-vocabulary Object Retrieval

OpenScene uses CLIP-aligned 2D-3D features to rank 3D scene vertices for text queries, including highly specific object names. The retrieval examples show precise matches, while the authors note that further work is needed to characterize limitations.

  • Experimental setup: OpenScene produces ranked retrieval lists by sorting Matterport3D vertices according to cosine similarity with each CLIP text query.The experiment uses 2D-3D ensemble features and returns one match per region.
  • Experimental setup: The experiment restricts queries to dataset ground-truth category names, selecting specific categories to test difficult retrieval cases.The selection strategy also uses category frequency to estimate expected matches.
  • Evaluation display: Retrieval figures rank the best match first, marking correct matches in green, incorrect matches in red, and near misses in gray.A red wireframe sphere identifies the top retrieved 3D point.
  • Results: The method retrieves specific objects effectively: “yellow egg-shaped vase” returns a matching vase first, while “teddy bear” returns two teddy bears among the top results.The cited matches were not labeled in the ground truth.
  • Results: These retrieval results suggest that the 2D-3D open-vocabulary features are effective for specific object types, although their limitations require further study.The authors frame this as a result supported by the examples rather than a complete characterization.

D. More Results of Open-vocabulary 3D Scene Exploration

OpenScene supports open-vocabulary exploration of 3D scenes through text queries, producing similarity heat maps across scene vertices. Results span object, room, activity, color, material, and abstract-concept queries, with quantitative evaluation also comparing model variants and supervised methods.

  • Open-vocabulary exploration: Text queries are encoded with CLIP, compared by cosine similarity against every 3D vertex feature, and visualized as color-coded similarity maps.Yellow indicates high similarity, green middle similarity, and blue low similarity.
  • Open-vocabulary exploration: The exploration results cover object categories, room types, activities, colors, materials, and abstract concepts.Figures F-K demonstrate this broad query range.
  • Open-vocabulary exploration: Open-vocabulary queries can identify scene attributes and abstract concepts that are difficult to label in a supervised setting, including “store” and “cluttered.”The “store” query highlights closets and cabinets, while “cluttered” highlights a busy closet.
  • Quantitative evaluation: The 3DSSG material-estimation experiment compares OpenScene variants trained on ScanNet with fully supervised MinkowskiNet, finding that 2D-3D ensembling is the best OpenScene variant but underperforms supervised methods for abundant classes.The supplied passage reports these findings for the 3DSSG test set, though its final clause is truncated.
  • Object retrieval: Object retrieval ranks matching 3D points from left to right, with correct matches marked green and the single incorrect match marked red.The Matterport test-set query and ground-truth-instance count appear in the left column.
Loading 2211.15654v2…