PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory

Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, Arthur Szlam

arXiv:2210.05663v3cs.ROcs.CV

TL;DR

Robots need flexible 3D semantic memories that avoid fixed labels and scene-specific human annotation, while web-trained models remain primarily 2D. CLIP-Fields maps spatial locations to semantic and visual embeddings using weak supervision from pretrained models, enabling open-vocabulary scene queries and real-world semantic navigation. It outperforms RGB-based baselines in few-shot instance identification and semantic segmentation and supports natural-language robot navigation.

  • Problem

    Robotic spatial memories are often coarse or label-restricted, while web-trained vision-language representations have mainly supported 2D inputs rather than 3D reasoning.

  • Method

    CLIP-Fields learns implicit mappings from 3D scene locations to semantic and visual embeddings using RGB-D data, pretrained-model outputs, and contrastive losses.

  • Results

    CLIP-Fields outperforms RGB-based models in low-shot instance identification and semantic segmentation and enables real-robot navigation from natural-language queries.

  • Takeaways & Limitations

    The representation supports open-vocabulary 3D scene understanding, spatial queries, and semantic navigation using little or no labeled data.

Abstract

from arXiv · show

We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization. CLIP-Fields learns a mapping from spatial locations to semantic embedding vectors. Importantly, we show that this mapping can be trained with supervision coming only from web-image and web-text trained models such as CLIP, Detic, and Sentence-BERT; and thus uses no direct human supervision. When compared to baselines like Mask-RCNN, our method outperforms on few-shot instance identification or semantic segmentation on the HM3D dataset with only a fraction of the examples. Finally, we show that using CLIP-Fields as a scene memory, robots can perform semantic navigation in real-world environments. Our code and demonstration videos are available here: https://mahis.life/clip-fields

I. INTRODUCTION

CLIP-Fields addresses the need for flexible, weakly supervised 3D semantic memory by combining neural scene representations with web-scale vision-language models. It supports semantic scene queries and robotic applications from relatively few examples.

  • Existing robotic spatial memories are often coarse, class-restricted, and dependent on scene-specific human annotation.
  • Web-scale vision-language models provide strong semantic abstractions from 2D images, but their use with 3D reasoning remains an open question.
  • CLIP-Fields builds a queryable 3D scene representation intended to function as spatial-semantic memory for mobile robots.
  • The method maps spatial locations to semantic features using RGB-D data and weak supervision from off-the-shelf web-trained vision-language models.
  • The full pipeline can be self-supervised without explicit human supervision, while capturing scene-specific information.
  • Experiments cover instance segmentation and identification, view localization, text localization, and real-robot navigation from natural-language commands.

III. BACKGROUND

CLIP-Fields builds on web-trained multimodal representations, open-label detection, sentence embeddings, and coordinate-based neural fields. The approach emphasizes using pretrained models as-is rather than requiring task-specific fine-tuning.

  • CLIP aligns image and text embeddings in a shared latent space using contrastive training on paired web images and captions.
  • Open-label models allow runtime-defined labels, and CLIP-Fields uses Detic pretrained on ImageNet-20k for open-label object detection.
  • Sentence-BERT improves language-query performance by providing representations pretrained for semantic similarity.
  • Neural fields use coordinate-based networks to map spatial or temporal coordinates to scene or object properties.
  • The system primarily uses large-dataset pretrained models as-is, while leaving open the possibility of fine-tuning when necessary.

IV. APPROACH

CLIP-Fields represents a 3D scene with implicit functions that associate coordinates with semantic and visual embeddings. It constructs training data from RGB-D videos and odometry, then learns the fields using labels and embeddings generated by pretrained models.

  • A. Problem Statement: CLIP-Fields defines scene-dependent functions f, h : R3 → Rn that map 3D points to semantic and visual feature vectors.
  • A. Problem Statement: The implicit functions serve as a spatial memory or geometric database by connecting scene coordinates with scene information.
  • A. Problem Statement: Segmentation assigns each pixel the label whose embedding best aligns with the semantic feature of its corresponding 3D point.
  • A. Problem Statement: Object navigation searches for the 3D point maximizing alignment with a semantic or visual query embedding.
  • A. Problem Statement: View localization retrieves scene points whose visual embeddings align most strongly with a query image embedding.
  • B. Dataset Creation: Training begins with RGB-D frames and odometry, converts depth images into world-coordinate point clouds, and labels their points.
  • B. Dataset Creation: The dataset combines detector-provided semantic labels and confidence scores with CLIP visual embeddings and point-to-camera distances.
  • B. Dataset Creation: Figure 2 shows that dataset creation uses collected RGB-D video and off-the-shelf pretrained models without direct human supervision.

C. Model Architecture

CLIP-Fields represents a scene with a spatial trunk and task-specific heads that map 3D locations to semantic, visual, or instance embeddings. These representations are trained with confidence- or distance-weighted contrastive objectives, optionally augmented by instance-identification cross-entropy.

  • CLIP-Fields uses a trunk g that maps 3D locations to representation vectors and separate heads for objectives such as language or visual retrieval.
  • Multi-resolution hash encoding represents coordinates with a feature-pyramid structure intended to retain local and global information.
  • The semantic objective trains f(P) toward pretrained semantic label embeddings and away from embeddings of other labels, weighted by label confidence.
  • The visual objective trains h(P) toward CLIP embeddings of image frames or patches containing the scene point, weighted by camera distance.
  • An optional instance-identification head maps point representations to one-hot instance vectors using cross-entropy when human instance labels are available.
  • The final loss combines language and visual contrastive losses with an α-scaled instance-identification loss.

E. Training

CLIP-Fields is trained on scene RGB-D data and associated spatial information, with contrastive learning aligning spatial embeddings to positive representations and separating them from negatives. The evaluation uses HM3D Semantic data for visual segmentation and a real-robot dataset for spatial-memory experiments.

  • Training uses scene datasets and simultaneously optimizes the implicit maps with the described contrastive losses.
  • Contrastive learning pulls each embedding toward positive labels or image patches and pushes it away from negative labels or images.
  • The visual segmentation experiments use a subset of the Habitat-Matterport 3D Semantic dataset, while robot experiments use a Hello Robot platform.
  • The segmentation setup assumes RGB-D scene views with limited annotations, including either human annotations or supervision from large web-image-trained models.

Baselines:

The evaluation compares CLIP-Fields with pretrained 2D RGB segmentation baselines under few-shot instance and semantic segmentation settings. CLIP-Fields outperforms those baselines across instance-normalization choices and achieves higher semantic-segmentation AP than every RGB baseline with three labelled views.

  • Baselines: The few-shot baselines are Mask-RCNN and DeepLabV3 models with ResNet50 backbones, pretrained on ImageNet-1K and COCO before limited-data fine-tuning.
  • Evaluation: Evaluation projects depth pixels into world coordinates using camera matrices and odometry before querying CLIP-Fields representations.
  • Low-shot instance identification: Few-shot instance identification uses 1-5 densely instance-annotated images with associated instance IDs.
  • Low-shot instance identification: CLIP-Fields largely outperforms RGB models in instance segmentation whether average precision is normalized over seen instances or all scene instances.
  • Low-shot semantic segmentation: With three labelled views, CLIP-Fields achieves higher semantic-segmentation AP than any baseline RGB model.

3) Zero-shot semantic segmentation:

CLIP-Fields uses labels from web-trained models and 3D multi-view consistency for zero-shot semantic segmentation. The resulting predictions improve over the labeling model and remain reasonably accurate under label noise.

  • Zero-shot setup: CLIP-Fields trains on Detic labels and CLIP visual representations without extra scene annotations.The method uses labels from unlabeled training images and augments them with CLIP visual representations.
  • Zero-shot results: Enforcing 3D structure and multi-view consistency considerably improves test-time segmentation predictions over the Detic labeling model.Both systems receive the scene’s semantic label list without additional annotations.
  • Zero-shot results: 3D consistency helps identify scene properties and recover objects that are occluded or only partially visible.Back-projecting rays supports recognition when objects are not visible from every angle, which is useful in busy indoor environments.
  • Robustness to label errors: CLIP-Fields label accuracy increases almost linearly as the base model’s semantic label accuracy increases.The experiments report no dramatic accuracy decrease when base-model accuracy falls below 1, while reasonable accuracy is maintained when base models are reasonably accurate.

5) View Localization:

The paper evaluates CLIP-Fields as a learned spatial representation for locating images and natural-language targets in 3D scenes. Queries span literal object names, visual descriptions, and semantic descriptions.

  • View localization: CLIP-Fields localizes a query image by comparing its CLIP embedding with stored point embeddings using dot products.Locations with the highest dot products are treated as the best matches, as visualized in Figure 8.
  • Task setup: The downstream evaluation uses a learned mapping from real-world coordinates to vectors containing semantic and visual properties.The representation is evaluated through robot semantic-navigation tasks.
  • Task setup: The robot task is defined as “Go and look at X,” where X is a natural-language query supplied by the user.The queries are organized into literal, visual, and semantic categories.
  • Query categories: Literal queries name objects directly, visual queries describe appearance, and semantic queries refer to object functions or uses.Examples include “the refrigerator,” “the red fruit bowl,” and “something to read.”
  • Experimental scenes: The real-world scenes comprise a lab kitchen and a library/lounge, with RGB-D and odometry data collected using an iPhone 13 Pro with LiDAR.A subset of RGB images was labeled with Detic using ScanNet200 labels.

3) Robot execution:

CLIP-Fields supports robot execution of natural-language look-at commands by matching query representations to 3D scene locations. In real experiments, the robot navigated to different points, with success depending on the quality and specificity of the learned representations.

  • Robot execution: Figure 10 shows semantic navigation in a lab kitchen and a library/lounge for objects specified by different queries.Color-coded borders indicate approximate correctness from the robot’s point of view.
  • Robot execution: Figure 11 depicts language-query retrieval by comparing encoded queries with stored CLIP-Fields representations to extract the best matches.The figure illustrates the query-to-location matching used during execution.
  • Robot execution: The robot combines Sentence-BERT and CLIP encodings to represent a text query before retrieving its best-matching 3D coordinate.It then uses Hector SLAM to navigate to that region and points its camera toward the selected coordinate.
  • Results: The robot navigated to different environmental points from semantic natural-language queries.The experiments evaluate CLIP-Fields as a scene memory for real-world semantic navigation.
  • Results: Literal queries can tolerate intentional misspellings when the web-image models correctly identified the queried object during data preparation.When preparation misidentifies an object, CLIP-Fields also fails to identify it correctly.
  • Results: Semantic queries sometimes confuse related concepts, while visual queries can favor semantic similarity over exact visual attributes.A more specific visual query, such as “red plastic bowl,” retrieved the correct object in the reported example.

VII. APPENDIX

The appendix provides implementation and reproducibility materials for CLIP-Fields. It releases training code and lists optimization, architecture, and Instant-NGP hyperparameters.

  • Reproducibility: The authors release open-source code with instructions for training CLIP-Fields on new environments.The code is also included in the supplementary information and is intended to support reproducing the work and experiments.
  • Hyperparameters: Table I lists the optimization hyperparameters used in the method.
  • Hyperparameters: Table II lists the architecture and Instant-NGP hyperparameters.

B. Real world experiment logs

Real-world Kitchen experiments tested literal, visual, and semantic natural-language queries, with many successful object localizations alongside failures and ambiguities. The logs show that CLIP-Fields often found reasonable substitutes, but some distinctions remained difficult.

  • Overall observations: The experiments demonstrate navigation to different environmental points from semantic natural-language queries, with success judged by reaching and pointing the camera at the target region.The qualitative logs provide concrete successes, failures, and ambiguities from the Kitchen scenario.
  • Literal queries: Literal queries successfully located plates, the microwave, fridge, coffee machine, sink, and coffee-related objects, while the toaster-oven query found the microwave instead.The microwave and fridge examples were successful; the toaster oven was misidentified as a microwave oven.
  • Visual queries: Visual queries found the white ceramic bowl, red plastic bowl, espresso machine, blue recycling bin, and potted plants, while the red fruit bowl query found a white bowl.The potted-plant result was successful but ambiguous when isolating the black flower pot.
  • Semantic queries: Semantic queries successfully mapped “Wash my dishes,” “Throw my trash,” “Fill out my water bottle,” “Make some coffee,” and “Warm up my lunch” to reasonable kitchen targets.These targets included the dishwasher, recycling bins, glass bottles, coffee maker and grinders, and microwave.
  • Semantic queries: “Wash my hand” selected the dishwasher instead of the sink, and “Put away my leftovers” pointed toward the trash cabinet rather than the fridge or cabinets.The text attributes the leftovers error potentially to trash cabinets being identified as cabinets by the detectors.
Loading 2210.05663v3…