PaperlayerView exact revision on arXiv

Source-linked AI summary

CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation

Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, Shuran Song

arXiv:2203.10421v2cs.CVcs.LGcs.RO

TL;DR

Robots need to find arbitrary objects described in language without expensive in-domain navigation training, motivating L-ZSON. The paper introduces CoW baselines and PASTURE to study this setting, finding strong uncommon-object performance but limited use of language descriptions. A simple untrained CoW matches Habitat MP3D navigation efficiency with a 500M-step ZSON method and improves RoboTHOR success by 15.6 percentage points over a leading ZSON model.

  • Problem

    Robots should find arbitrary objects described by people, but existing object-navigation benchmarks and methods do not fully cover language-specified, zero-shot goals.

  • Method

    The paper adapts open-vocabulary models into 21 CLIP on Wheels baselines and introduces the PASTURE benchmark for L-ZSON.

  • Results

    CoW baselines often struggle with language descriptions but perform well on uncommon objects; the best untrained CoW matches Habitat MP3D navigation efficiency and improves RoboTHOR success by 15.6 percentage points.

  • Takeaways & Limitations

    Open-vocabulary models can support competitive zero-shot object navigation without navigation training, while language grounding remains an empirical challenge.

  • Takeaways & Limitations

    The evaluation lacks large-scale real-world benchmarking and does not consider different agent embodiments or continuous action spaces.

Abstract

from arXiv · show

For robots to be generally useful, they must be able to find arbitrary objects described by people (i.e., be language-driven) even without expensive navigation training on in-domain data (i.e., perform zero-shot inference). We explore these capabilities in a unified setting: language-driven zero-shot object navigation (L-ZSON). Inspired by the recent success of open-vocabulary models for image classification, we investigate a straightforward framework, CLIP on Wheels (CoW), to adapt open-vocabulary models to this task without fine-tuning. To better evaluate L-ZSON, we introduce the Pasture benchmark, which considers finding uncommon objects, objects described by spatial and appearance attributes, and hidden objects described relative to visible objects. We conduct an in-depth empirical study by directly deploying 21 CoW baselines across Habitat, RoboTHOR, and Pasture. In total, we evaluate over 90k navigation episodes and find that (1) CoW baselines often struggle to leverage language descriptions, but are proficient at finding uncommon objects. (2) A simple CoW, with CLIP-based object localization and classical exploration -- and no additional training -- matches the navigation efficiency of a state-of-the-art ZSON method trained for 500M steps on Habitat MP3D data. This same CoW provides a 15.6 percentage point improvement in success over a state-of-the-art RoboTHOR ZSON model.

1. Introduction

The paper frames L-ZSON as language-driven, zero-shot object navigation and introduces CoW baselines alongside the PASTURE benchmark to evaluate it. Experiments show strong performance on uncommon objects but difficulty exploiting language descriptions, while the best CoW matches or exceeds trained ZSON baselines on established benchmarks.

  • Existing navigation benchmarks contain many everyday categories but omit objects that may matter in specific human contexts.A lost “toy airplane” is given as an example of an object not annotated in those datasets.
  • L-ZSON requires agents to find objects specified by natural language without navigation training on the target objects.
  • CoW adapts open-vocabulary models to L-ZSON without navigation training while retaining text-based goal specification.The study evaluates 21 variants across models, exploration policies, backbones, prompting, and post-processing strategies.
  • PASTURE evaluates uncommon objects, appearance and spatial descriptions, and hidden objects that are not visually observed.Its tasks target capabilities absent from traditional fixed-category object-navigation benchmarks.
  • CoW baselines often struggle with language descriptions but perform well on uncommon objects.

2. Related Work

Related work spans classical and learned exploration, goal-conditioned navigation, instruction following, and zero-shot object navigation. L-ZSON combines arbitrary language-specified object goals with zero-shot evaluation and extends prior category-based ZSON settings.

  • Mapping and exploration: Classical exploration decomposes navigation into map reconstruction, localization, and planning, whereas learned methods require training steps and reward engineering.The paper evaluates both exploration families within CoW.
  • Goal-conditioned navigation: Goal-conditioned navigation includes point, image, and object goals, with this paper focusing on object-goal navigation.
  • Instruction following in Navigation: Instruction-following navigation uses step-by-step language, while L-ZSON specifies the object goal rather than a detailed route.
  • Zero-shot object navigation: Prior ZSON evaluates unseen object categories, whereas L-ZSON additionally considers object attributes and descriptions of hidden objects.

3. The L-ZSON Task

L-ZSON defines zero-shot navigation to language-specified objects across scenes and initial poses, using embodied observations and actions until the target is successfully found. The task extends category-only object navigation with granular descriptions and attributes.

  • L-ZSON specifies navigation goals as natural-language object descriptions that may include attributes such as type, appearance, or spatial relations.Examples include “plant,” “snake plant,” and “plant under the bed.”
  • Each episode combines a scene, a language-described object, and an initial agent pose, with no such tuple appearing during training.
  • The agent receives observations and sensor readings, executes navigation actions, and may terminate with STOP.
  • Success requires the agent to be within c units of the target while satisfying the task’s visibility criterion.

4. CLIP on Wheels (CoW) Baselines

CoW baselines combine RGB-D mapping, exploration, zero-shot object localization, and target-driven planning without fine-tuning. They explore until the language target is localized, then use projected relevance to plan toward it.

  • CoW overview: A CoW updates a top-down map from RGB-D observations and pose estimates while combining an exploration policy with zero-shot object localization.
  • Depth-based Mapping: CoW maps use egocentric depth observations to represent free and occupied space in a top-down grid.Known camera intrinsics, agent height, and action-based pose deltas support map construction; the map resolution is 0.125m.
  • Exploration: Frontier-based exploration moves toward boundaries between known free space and unknown space to discover new regions.
  • Target-driven planning: Object localization projects image-based relevance into the depth map, where high-relevance regions become navigation targets.
  • Object Localization: The baselines test CLIP referring expressions, image patches, and gradient relevance alongside MDETR and OWL-ViT localization.
  • Post-processing: Localization predictions are thresholded into masks, with downstream planning evaluated using either full masks or only their center pixels.Center-pixel processing reflects that successful navigation requires detecting only part of an object.

5. The PASTURE Benchmark

PASTURE is an evaluation benchmark for L-ZSON that probes seven capabilities using ROBOTHOR validation scenes. It includes uncommon objects and descriptions based on appearance and spatial relationships, with controlled variations and detailed dataset statistics.

  • PASTURE builds on ROBOTHOR validation scenes to evaluate seven core L-ZSON capabilities.The benchmark targets scenes with parallel real-world environments to support future real-world benchmarking.
  • Uncommon objects: Each room adds 12 uncommon objects, including objects identified by image text, appearance attributes, or culturally specific names.Examples include “whiteboard saying CVPR,” “wooden toy airplane,” and “maté.”
  • Appearance descriptions: Appearance descriptions combine size, color, material, and object identity, such as “small, black, metallic alarm clock.”Small objects are defined by a 3D bounding-box diagonal threshold, while colors and materials are determined by inspection.
  • Spatial descriptions: Spatial descriptions specify an object’s relation to other objects, including being on top of one object and near others.Relations are determined using THOR metadata and distance thresholds, then inspected for correctness.
  • Dataset Creation and Statistics: PASTURE comprises 2,520 tasks across 15 rooms, 12 object instances, seven settings, and two starting locations.Its scale is similar to ROBOTHOR’s 1,800 and HABITAT MP3D’s 2,195 validation sets.
  • Dataset Creation and Statistics: The benchmark varies each original room by adding uncommon objects, adding object instances, or removing target objects.For appearance attributes, 47% of objects are small; objects receive averages of 1.2 color and 0.6 material descriptors.

6. Experiments

The experiments evaluate zero-shot CoW baselines across PASTURE, ROBOTHOR, and HABITAT using standard navigation metrics and varied exploration and localization components. CoWs perform well on uncommon objects and can match or exceed trained baselines, but language descriptions, distractors, and localization remain challenging.

  • Experimental setup: SUCCESS measures stopping within 1.0m of the target, while SPL measures success weighted by inverse path length and therefore navigation efficiency.In ROBOTHOR and PASTURE, the target must also be visible at episode completion.
  • Experimental setup: 21 CoW baselines are evaluated across PASTURE, ROBOTHOR, and HABITAT, with no navigation fine-tuning for the open-vocabulary models.The study varies exploration policies, object localization modules, and backbone architectures under zero-shot inference.
  • CoWs on PASTURE: Post-processing generally helps, whereas higher-compute ViT-B/16 architectures often perform comparably or worse than ViT-B/32; OWL-ViT models perform best.The PASTURE benchmark also shows that distractors hurt performance and natural-language specification does not remove this difficulty.
  • CoWs on PASTURE: CoWs often find uncommon PASTURE objects at higher SUCCESS rates than common ROBOTHOR objects, despite lower average PASTURE success.For example, OWL ViT-B/32 with post-processing exceeds its ROBOTHOR SUCCESS by approximately 6 percentage points on uncommon objects.
  • CoWs on PASTURE: Appearance descriptions are more useful than spatial descriptions, but neither generally improves performance over ROBOTHOR baselines, and distractors cause further degradation.Higher-success models do not preserve the same relative performance when finding hidden objects, while distractors produce similar trends with worse results.
  • Ablations and failure analysis: FBE exploration outperforms learnable alternatives on PASTURE and ROBOTHOR, while exploration and object-localization errors account for large fractions of failures.Distractors increase object-localization failures, supporting the observed difficulty of using attribute prompts.
  • Comparison to prior art: The best CoW matches SemanticNav-ZSON at 4.9 versus 4.8 SPL on HABITAT and improves over EmbCLIP-ZSON ROBOTHOR SUCCESS by 15.6 percentage points.This CoW uses zero navigation-training steps, compared with 500M and 60M training steps for the respective trained methods.

7. Limitations and Conclusion

The paper identifies real-world benchmarking, localization-threshold tuning, and embodiment as important limitations, while formalizing CoW mapping from depth observations and estimated poses.

  • Limitations and Conclusion: The largest limitation is the lack of large-scale, real-world benchmarking, despite evaluation across Habitat, RoboTHOR, and Pasture.The authors also note that this gap exists in much of the related literature.
  • Limitations and Conclusion: CoW inherits limitations from its localization and exploration components, including the need to tune confidence thresholds for precision-recall trade-offs.This tuning balances localization precision and recall.
  • Limitations and Conclusion: The study does not evaluate different agent embodiments or continuous action spaces, although agent morphology may affect downstream performance.The authors identify this as a pertinent investigation for future work.
  • Depth-based Mapping Details: CoW maps observations by estimating pose through concatenated action transforms, back-projecting depth, and registering observations in a common world frame.Pose estimates remain sensitive to sensor noise and action failures, producing potential pose drift.
  • Depth-based Mapping Details: The map is projected onto the ground plane and discretized, retaining free and occupied space rather than the full 3D environment.Known agent height and the down-gravity direction support this projection.

B. Exploration Details

The appendix details learned exploration training, CLIP-based localization and interpretability components, and evaluation procedures for localization quality and qualitative trajectories.

  • Exploration Details: Learned exploration agents use AllenAct and DD-PPO with state-visitation rewards, trained separately on RoboTHOR and Habitat MP3D training sets.Agents receive rewards for visiting previously unvisited voxels and a step penalty.
  • Localization Details: The gradient-based relevance visualization keeps relevance low when the target is absent and high when it is present, avoiding spurious absent-target relevance caused by normalization.When the plant is visible, the proposed relevance map resembles the original method.
  • Localization Details: The localization evaluation reports macro F1 by averaging per-category F1 scores, with false positives and false negatives defined from predicted and ground-truth masks.The measure is designed to be more lenient than the traditional Jaccard index because partial object identification can suffice for navigation.
  • Exploration Details: Qualitative evaluation visualizes backprojecting 2D object relevance into 3D and provides successful and failed navigation trajectories.These visualizations illustrate key parts of the CoW pipeline and its outcomes.

E. Dataset Details

The evaluation uses official HABITAT MP3D and ROBOTHOR validation scenes alongside PASTURE, which remaps object goals with appearance, spatial, and hidden-object descriptions. Qualitative retrieval suggests CLIP can semantically distinguish the selected uncommon objects.

  • HABITAT MP3D uses 11 test scenes and 2,195 evaluation episodes, while ROBOTHOR uses 15 scenes and 1,800 episodes.
  • Object relevance is projected from egocentric CLIP-Grad predictions into a 3D map, making high-relevance areas navigation targets.
  • PASTURE includes uncommon objects such as a tie-dye surfboard, whiteboard saying CVPR, llama wicker basket, rice cooker, and mate gourd.
  • Trajectory visualizations show egocentric frames with progress encoded from blue to white and target and distractor objects boxed green and red, respectively.
  • PASTURE remaps object goals into appearance, spatial, and hidden descriptions, including “small, green, plastic spray bottle,” “spray bottle on a coffee table near a house plant,” and “spray bottle under the bed.”
  • CLIP retrieval returns reasonable results for uncommon ROBOTHOR and PASTURE objects, suggesting semantic distinction despite noisy internet image-text training data.

F. Prompt Ensemble Ablation

The prompt ablation compares OpenAI’s 80-prompt ensemble with a single photo prompt for fixed CLIP-Grad localization. The ensemble usually performs better, but prompt choice has limited downstream influence.

  • PASTURE’s appearance and spatial captions and hidden-object references provide distinct language-conditioned evaluation cases for the benchmark.
  • 0.2 SPL is the reported boost from the prompt ensemble for CLIP-Grad on ROBOTHOR.
  • The 80-prompt ensemble outperforms “a photo of a {}.” in most cases for fixed CLIP-Grad B/32 localization.
  • Prompting deltas are small, indicating that prompt choice is a less critical design decision in the CoW framework.
  • Main-paper CLIP experiments use the 80-prompt ensemble with the class label specific to each task, such as “spray bottle under the bed.”

G. Category-level Results

Category-level results are reported for OWL and CLIP-Grad across HABITAT MP3D, ROBOTHOR, and PASTURE, including attribute, hidden-object, and uncommon-object settings.

  • Category-level comparisons cover appearance and spatial PASTURE splits, distractor variants, hidden-object settings, uncommon objects, ROBOTHOR, and HABITAT.
  • The reported category-level models are OWL and CLIP-Grad B/32 with post-processing.
  • The uncommon-object category-level results are presented separately from ROBOTHOR and HABITAT category-level results.

H. Additional Failure Analysis

Additional analysis compares localization failures among CLIP-Ref., CLIP-Patch, CLIP-Grad., and OWL, while attribute results characterize the effects of caption type and distractors.

  • Failure analysis: CLIP-Patch and CLIP-Grad. have higher fractions of object-localization failures than OWL and lower downstream success rates on average.
  • Failure analysis: CLIP-Ref. operates in a very low-success regime, with performance examples below 3%, and shows less consistent failure patterns.
  • Attribute navigation: Appearance-based captions consistently outperform spatial captions in attribute navigation, while OWL consistently outperforms CLIP-Grad.
  • Attribute navigation: Distractors consistently reduce performance relative to no-distractor results, suggesting limited use of attribute-remapped classes.
  • Category-level coverage: The category-level analysis includes hidden-object, uncommon-object, ROBOTHOR, and HABITAT results in Tables 9–12.
Loading 2203.10421v2…