PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple but Effective: CLIP Embeddings for Embodied AI

Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, Aniruddha Kembhavi

arXiv:2111.09888v2cs.CVcs.LG

TL;DR

The paper asks whether CLIP visual representations improve navigation-heavy Embodied AI without specialized task architectures or extra modalities. It builds simple CLIP-based agents, evaluates their representations and task performance, and finds strong cross-task results plus an initial zero-shot navigation capability. The study also identifies frozen backbones and vision-language-only pretraining as limitations and directions for improvement.

  • Problem

    Navigation-heavy Embodied AI requires visual representations that encode semantic and geometric primitives such as object presence, reachability, and free space.

  • Method

    The paper builds simple agents using frozen CLIP representations, task goals, recurrent memory, and action heads, then evaluates task performance, representation probes, and zero-shot Object Goal Navigation.

  • Results

    CLIP-based agents perform strongly across navigation-heavy tasks, outperform ImageNet counterparts on primitive probes, and achieve 0.08 Success Rate on unseen versus 0.17 on seen objects.

  • Takeaways & Limitations

    CLIP representations can support effective navigation-heavy Embodied AI and an initial form of zero-shot navigation to objects excluded from training targets.

  • Takeaways & Limitations

    The experiments use frozen backbones, and the encoders are trained with visual and textual information rather than interactive actions.

Abstract

from arXiv · show

Contrastive language image pretraining (CLIP) encoders have been shown to be beneficial for a range of visual tasks from classification and detection to captioning and image manipulation. We investigate the effectiveness of CLIP visual backbones for Embodied AI tasks. We build incredibly simple baselines, named EmbCLIP, with no task specific architectures, inductive biases (such as the use of semantic maps), auxiliary tasks during training, or depth maps -- yet we find that our improved baselines perform very well across a range of tasks and simulators. EmbCLIP tops the RoboTHOR ObjectNav leaderboard by a huge margin of 20 pts (Success Rate). It tops the iTHOR 1-Phase Rearrangement leaderboard, beating the next best submission, which employs Active Neural Mapping, and more than doubling the % Fixed Strict metric (0.08 to 0.17). It also beats the winners of the 2021 Habitat ObjectNav Challenge, which employ auxiliary tasks, depth maps, and human demonstrations, and those of the 2019 Habitat PointNav Challenge. We evaluate the ability of CLIP's visual representations at capturing semantic information about input observations -- primitives that are useful for navigation-heavy embodied tasks -- and find that CLIP's representations encode these primitives more effectively than ImageNet-pretrained backbones. Finally, we extend one of our baselines, producing an agent capable of zero-shot object navigation that can navigate to objects that were not used as targets during training. Our code and models are available at https://github.com/allenai/embodied-clip

1. Introduction

The paper investigates whether CLIP visual representations improve navigation-heavy Embodied AI tasks. Simple RGB-only CLIP baselines perform strongly, encode useful semantic and geometric primitives, and support initial zero-shot object navigation.

  • CLIP visual representations are investigated for Embodied AI tasks requiring low-level actions and encoding navigational primitives such as free space and geometric structure.
  • Simple frozen CLIP ResNet-50 baselines use RGB observations, goal specifications, and an RNN without depth maps, spatial or semantic maps, or auxiliary training tasks.
  • The baselines top two AI2-THOR leaderboards and outperform Habitat challenge winners for Object Goal Navigation and Point Goal Navigation.
  • +3 points absolute improvement appears on three probing studies, while object localization improves by +6.5 points (+16 % relative) over ImageNet-pretrained representations.
  • ImageNet Top-1 Accuracy alone is not a good indicator of Object Goal Navigation suitability, whereas probing semantic and geometric information may be more useful.
  • A CLIP vision-and-language agent achieves roughly half the seen-object Success Rate when navigating to unseen objects, providing an initial zero-shot Object Goal Navigation result.

2. Related Work

Embodied AI spans simulated navigation and interaction tasks that require combinations of perception, reasoning, and action. Building on CLIP’s success in instruction following, this work studies atomic-action, navigation-heavy tasks.

  • Embodied AI simulators support photorealistic scenes, physics, interaction, and large accompanying scene datasets.
  • Existing Embodied AI tasks include navigation, rearrangement, furniture moving, packing, and cloth folding, requiring combinations of skills.
  • CLIP has helped instruction following along navigation graphs, where agents identify objects and locations from names and descriptions.
  • This paper extends that direction to navigation-heavy tasks using atomic actions such as Move Forward, Turn Left, Pick Up, and Place.

3. Using CLIP in Embodied AI

The paper applies CLIP visual backbones to four navigation-heavy tasks across RoboTHOR, Habitat, and iTHOR using closely related baseline architectures. The core model combines visual features and task goals with recurrent memory, while the experiments use CLIP ResNet-50 features.

  • CLIP is pretrained by contrasting corresponding and non-corresponding image–caption pairs, producing powerful visual encoders.
  • The study evaluates CLIP visual encoders on Object Goal Navigation, Point Goal Navigation, and Room Rearrangement across RoboTHOR, Habitat, and iTHOR.
  • The baselines share a core architecture but differ in how they use goal descriptions and process visual features.
  • For RoboTHOR Object Goal Navigation, a frozen CLIP ResNet-50 encodes RGB input, conditions visual features on a trainable goal embedding, and sends them to a GRU for action prediction.
  • The experiments consider only CLIP ResNet-50, although the baselines can be extended to other CLIP variants.

4. When is CLIP effective?

CLIP-based agents perform competitively across navigation-heavy embodied tasks, often surpassing leaderboard methods despite using simple RGB-only baselines without depth or task-specific components.

  • Across benchmarks: CLIP baselines are competitive with state-of-the-art methods across ObjectNav, PointNav, and Room Rearrangement benchmarks and simulators.The evaluated tasks include RoboTHOR and Habitat ObjectNav, Habitat PointNav, and iTHOR Room Rearrangement.
  • Baseline design: CLIP agents use RGB-only inputs and avoid the task-specific designs, depth maps, and extensive engineering used by many leaderboard methods.The baselines replace visual encoders while retaining simple recurrent agent designs.
  • RoboTHOR OBJECTNAV: 1.7x improvement in RoboTHOR OBJECTNAV SPL and Success Rate over the Action Boost model is reported for the CLIP agent.The comparison is against the prior best Action Boost model on the RoboTHOR leaderboard.
  • iTHOR ROOMR: 1.94x improvement in iTHOR ROOMR FixedStrict over ANM raises performance from 9% to 17%.The ImageNet ResNet-50 agent reaches 7% FixedStrict, indicating that deeper ImageNet features alone do not explain the gain.
  • Habitat OBJECTNAV: 1.45x SPL improvement over ImageNet pretraining places the Habitat CLIP agent fourth on the ObjectNav leaderboard, roughly alongside the leading entry.It surpasses both winners of the 2021 challenge without depth, human annotations, or task-specific components.
  • Habitat POINTNAV: +0.05 SPL and +0.03 Success Rate over ImageNet, with a 1.8x reduction in distance to goal, are reported for Habitat PointNav.The CLIP agent also beats the 2019 challenge winner by +0.14 SPL, although performance was still increasing at the end of training.

5. Why is CLIP effective?

The paper investigates why CLIP representations help navigation-heavy Embodied AI by testing whether they encode semantic and geometric primitives. Linear probes show that CLIP features outperform ImageNet features across all four primitives, especially object localization.

  • Probe design: The study probes object presence, object localization, reachability, and free space using simple linear classifiers on ImageNet- or CLIP-pretrained ResNet-50 features.These primitives are selected because they are relevant to the embodied tasks studied.
  • Results: CLIP representations outperform ImageNet representations on every primitive, with improvements ranging from 3 to 6.5 points.The evaluation uses test-set scores for F1 on object presence and localization, and accuracy on free space and reachability.
  • Results: +6.5 absolute and +16% relative improvement occurs on object localization with CLIP representations.Object localization is useful for the two OBJECTNAV tasks and the ROOMR task.
  • Dataset: The probe dataset contains frames sampled from 60 training, 15 validation, and 15 test iTHOR scenes across kitchens, living rooms, and bedrooms.Bathrooms are excluded, and frames receive task-specific ground-truth annotations.
  • Implementation: For each frame, convolutional features are pooled into embeddings and classified with a single supervised linear layer.The experiments use ResNet-50 encoders with either CLIP or ImageNet pretraining.

6. Does ImageNet performance correlate with Embodied AI success?

The paper asks whether ImageNet accuracy predicts embodied navigation success. Across four encoders, ImageNet Top-1 accuracy alone is not a reliable indicator, while CLIP encoders produce larger OBJECTNAV gains between comparable models.

  • Experimental setup: Four pretrained encoders are evaluated with the same OBJECTNAV architecture: ImageNet ResNet-18 and ResNet-50, and CLIP ResNet-50 and ResNet-50x16.The comparison isolates the visual encoder while keeping the OBJECTNAV model architecture fixed.
  • Findings: Within both ImageNet and CLIP model families, larger models improve ImageNet Top-1 Accuracy and Success Rate, but the Success Rate gain is small.The larger gap occurs between ImageNet and CLIP models rather than within either family.
  • Findings: For the same ResNet-50 architecture, higher ImageNet Top-1 Accuracy does not lead to higher OBJECTNAV Success Rate.The paper therefore recommends probing representations for semantic and geometric information instead of relying on ImageNet accuracy alone.

7. Discussion

The discussion presents CLIP as an effective but still preliminary basis for embodied navigation, including an initial zero-shot OBJECTNAV agent. It highlights frozen visual backbones and the absence of interactive action data as limitations and future directions.

  • Limitations: The experiments use frozen backbones, leaving robust techniques for updating visual encoders during task completion as future work.This is identified as a limitation of the study.
  • Limitations: The encoders are trained with visual and textual information, while incorporating interactive actions during representation learning is proposed as a way to obtain richer representations.The paper presents this as another limitation and possible research direction.
  • Zero-shot OBJECTNAV: The zero-shot OBJECTNAV agent trained on 8 of 12 objects achieves 0.17 Success Rate on seen objects and 0.08 on unseen objects.The agent is trained in RoboTHOR for 60M steps using DD-PPO and a simple model operating on CLIP representations.
  • Conclusion: The conclusion reports that CLIP representations are effective for navigation-heavy Embodied AI, outperform ImageNet counterparts on relevant primitives, and support an initial zero-shot OBJECTNAV baseline.The paper anticipates further work on visual representations and zero-shot navigation.

A. Baseline Architecture Details

The baseline architectures replicate task-authors’ designs while replacing their visual encoders with frozen CLIP ResNet-50 features. ImageNet baselines instead use a frozen ImageNet-pretrained ResNet-50.

  • Architecture: Each baseline replicates the task authors’ architecture with a frozen CLIP ResNet-50 visual encoder substituted in.The substitution is the primary implementation change described for these baselines.
  • Architecture: ImageNet baseline agents use a frozen ResNet-50 pretrained on ImageNet.

A.1. Room Rearrangement in iTHOR

The ROOMR model processes two RGB observations with a frozen CLIP ResNet-50 encoder and combines their feature maps for downstream prediction.

  • Two 3×224×224 RGB images are encoded into 2048×7×7 feature tensors by a frozen CLIP ResNet-50.The encoder’s final attention-pooling and classification layers are removed.
  • The two feature maps are stacked with their element-wise product to form a 6144×7×7 tensor.The resulting representation is s = [I1, I2, I1 ∗I2].
  • A 1×1 convolution produces a 512-channel attention mask from the combined feature tensor.

A.2. Habitat OBJECTNAV and POINTNAV

The Habitat POINTNAV and OBJECTNAV baselines share one architecture, differing only in the input goal and its encoding. Each model uses a frozen CLIP ResNet-50 to produce a visual embedding from the RGB observation, alongside goal and previous-action encodings.

  • Habitat POINTNAV and OBJECTNAV use the same architecture, differing only in the input goal and its encoding.OBJECTNAV represents the goal as an embedded object-category index, whereas POINTNAV uses a relative 2-dim polar coordinate.
  • OBJECTNAV encodes an integer object category g ∈{0, ..., 20} by indexing an embedding matrix to form a 32-dim goal encoding G.
  • POINTNAV encodes the target as a 2-dim polar coordinate expressed relative to the agent’s current position.
  • A frozen CLIP ResNet-50 encodes each 3×224×224 RGB image into a 2048×7×7 tensor, which is average pooled and flattened into a 2048-dim visual embedding V.The final attention pooling and classification layers are removed before spatial average pooling.
  • The previous action is represented by indexing an embedding matrix to form a 32-dim action encoding A.
Loading 2111.09888v2…