PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models

Yuxuan Kuang, Hai Lin, Meng Jiang

arXiv:2402.10670v2cs.CLcs.RO

TL;DR

ObjectNav methods trained on limited household datasets face challenges understanding free-form instructions for open-set objects and generalizing zero-shot to new environments. OpenFMNav combines language-model proposals, vision-language object discovery and detection, a semantic score map, and common-sense frontier reasoning. It outperforms strong HM3D ObjectNav baselines, with real-robot demonstrations supporting its reported open-set and generalization claims.

  • Problem

    ObjectNav methods trained on limited household datasets with close-set objects face challenges understanding free-form instructions that demand open-set objects and generalizing zero-shot to diverse environments.

    Source: §1 · p. 1

  • Method

    OpenFMNav uses language models to propose objects, vision-language models to discover and detect scene objects, and common-sense reasoning over a Versatile Semantic Score Map to guide frontier exploration.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    On HM3D ObjectNav, OpenFMNav surpasses Zhou et al. (2023) by over 15% on success rate and surpasses all strong baselines on all metrics, with real-robot demonstrations validating open-set use and real-world generalization.

    Source: §1 · p. 2, §4.4 · p. 7

  • Takeaways & Limitations

    The authors conclude that foundation-model reasoning and generalization support effective open-set zero-shot ObjectNav, with real-robot demonstrations validating the approach in real-world environments.

    Source: §6 · p. 9

    The method requires relatively accurate depth sensors, stable Internet connections for foundation-model APIs, and may have latency from LLM use.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Object navigation (ObjectNav) requires an agent to navigate through unseen environments to find queried objects. Many previous methods attempted to solve this task by relying on supervised or reinforcement learning, where they are trained on limited household datasets with close-set objects. However, two key challenges are unsolved: understanding free-form natural language instructions that demand open-set objects, and generalizing to new environments in a zero-shot manner. Aiming to solve the two challenges, in this paper, we propose OpenFMNav, an Open-set Foundation Model based framework for zero-shot object Navigation. We first unleash the reasoning abilities of large language models (LLMs) to extract proposed objects from natural language instructions that meet the user's demand. We then leverage the generalizability of large vision language models (VLMs) to actively discover and detect candidate objects from the scene, building a Versatile Semantic Score Map (VSSM). Then, by conducting common sense reasoning on VSSM, our method can perform effective language-guided exploration and exploitation of the scene and finally reach the goal. By leveraging the reasoning and generalizing abilities of foundation models, our method can understand free-form human instructions and perform effective open-set zero-shot navigation in diverse environments. Extensive experiments on the HM3D ObjectNav benchmark show that our method surpasses all the strong baselines on all metrics, proving our method's effectiveness. Furthermore, we perform real robot demonstrations to validate our method's open-set-ness and generalizability to real-world environments.

1 Introduction

ObjectNav methods trained on limited household datasets struggle with free-form instructions for open-set objects and zero-shot generalization. OpenFMNav uses foundation models to interpret instructions and explore, outperforming strong baselines on HM3D ObjectNav and demonstrating real-robot generalization.

1 Introduction

  • Prior supervised and reinforcement-learning methods train on limited household datasets with close-set objects, while human instructions may describe attributes or imply an object without naming it.These methods struggle with such instructions and with zero-shot generalization to diverse unseen environments.Source: §1 · pp. 1–2
  • Earlier approaches to free-form open-set navigation either remain limited to household settings or require occupancy maps and pre-exploration unavailable in unseen environments.These constraints are described for demand-driven navigation and an attribute-based approach, respectively.Source: §1 · p. 2
  • OpenFMNav uses large language models to propose objects from instructions and vision-language models to discover and detect scene objects, building a Versatile Semantic Score Map for common-sense frontier reasoning.The map updates during navigation and supports exploration and exploitation without prior household-dataset training.Source: §1 · p. 2
  • OpenFMNav surpasses the prior open-set zero-shot method Zhou et al. (2023) by over 15% on success rate and all strong baselines on all metrics.The authors also report real-robot demonstrations validating open-set instruction understanding and generalization to real-world environments.Source: §1 · p. 2

2 Related Work

Related work frames ObjectNav as object-centric navigation without step-by-step routes, then reviews zero-shot methods and foundation models. OpenFMNav’s framing emphasizes free-form open-set instructions and navigation in unseen environments.

2.1 Embodied Navigation

  • ObjectNav requires semantic recognition and efficient exploration without step-by-step instructions, and its continuous, object-centric setting is described as readily deployable and extensible to downstream tasks.Unlike VLN, it also accommodates ambiguous human demands rather than only detailed route instructions.Source: §2.1 · pp. 2–3
  • OpenFMNav reframes ObjectNav from close-set category names to free-form instructions with open-set objects, without requiring prior occupancy maps or initial pre-exploration.The paper presents this as supporting navigation in unseen environments and zero-shot generalization to the real world.Source: §2.1 · p. 3

2.2 Zero-Shot Object Navigation

  • Prior zero-shot methods use image-feature similarities and implicit maps or object detectors, metric maps, and language-model reasoning; some methods also train locomotion modules.The paper notes that such training means those methods are not truly zero-shot.Source: §2.2 · pp. 3–4

2.3 Foundation Models

  • Foundation models are presented as generalizable zero-shot models whose pretraining can provide physical-world common sense, while in-context learning grounds outputs in predefined instructions.The paper uses vision-language models and language models because its setting involves vision and language.Source: §2.3 · p. 4

3 Method

OpenFMNav turns an instruction and egocentric RGBD observations into object proposals, a semantic map, and frontier goals for an underlying controller. Its discovery, perception, mapping, and reasoning stages repeat as the agent navigates.

3.1 Problem Statement and Method Overview

  • Given an instruction and RGBD observations, ProposeLLM proposes objects, DiscoverVLM discovers scene objects, PerceptVLM segments prompted objects, and ReasonLLM selects a frontier goal from the VSSM for the controller.The process loops until the object is found or navigation fails; success is finding the queried object within the maximum timestep.Source: §3.1 · p. 4

3.2 Discovery and Perception

  • ProposeLLM extracts candidate objects and attributes from free-form instructions, while DiscoverVLM can actively find novel scene objects and update the proposals.The proposal, discovery, and prior object sets are combined into a prompt for PerceptVLM to detect and segment appearing objects.Source: §3.2 · pp. 4–5

3.3 Mapping and Reasoning

  • VSSM projects mask confidence scores into a top-down map with object-semantic, occupied-area, and explored-area channels, and its scores support updates as the scene changes.ReasonLLM rates semantically described frontiers using common-sense reasoning and selects the highest-scoring goal; FMM turns that goal and occupancy map into low-level actions.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate OpenFMNav on HM3D against navigation baselines and ablate framework components; reported findings include benchmark gains and real-world robustness to varied instructions.

4.4 Results and Analysis

4.1 Experimental Setup

  • HM3D evaluation uses 20 validation scenes and 2,000 episodes across six goal classes, reporting Success Rate and Success weighted by Path Length.Source: §4.1 · p. 6

4.2 Implementation Details

  • The implementation uses GPT-4 for language reasoning, GPT-4V for discovery, and Grounded-SAM for object masks, with Chain-of-Thought prompting.Simulation settings include a 20-timestep update interval and discovery frequency of 0.01.Source: §4.2 · p. 6

4.3 Baseline Methods

  • Baselines span classical frontier exploration, close-set semantic mapping, CLIP transfer, GLIP-based mapping, and foundation-model navigation.Source: §4.3 · pp. 6–7

4.5 Ablation Studies

  • Ablations report GPT-4 +1.8%, CoT +3.6%, scoring prompting +5.4%, and DiscoverVLM +5.4%; component changes worsen performance.Ablations use a uniformly sampled subset, so results may differ slightly from the full-method result; Figure 3 groups failures as Collision, Exploration, and Detection.Source: Table 2 · p. 7, §4.5 · pp. 7–8, Figure 3 · p. 7

4.4 Results and Analysis

  • Compared with semantic-map methods, VSSM adds out-of-vocabulary labels, while its maintained explicit map supports LLM exploration–exploitation reasoning and further instructions.Unlike goal-specific implicit maps, the map is maintained and updated.Source: §4.4 · p. 7

4.5 Ablation Studies

  • Real-world qualitative studies report robustness to distractors, open-set objects, and free-form demands.Source: Figure 4 · p. 8

5 Navigation in the Real World

Real-robot demonstrations show OpenFMNav handling free-form instructions, distractors, and open-set objects. Its setup uses a TurtleBot4 with RGBD perception in rooms containing common and less common objects.

5 Navigation in the Real World

  • The demonstrations are presented as validation of the method’s ability to understand free-form instructions and perform open-set zero-shot navigation in the real world.Source: §5 · p. 8

5.1 Real Robot Setup

  • The TurtleBot4 uses a Kinect RGBD camera and an action space limited to stopping, moving forward, and turning left or right.Forward motion is 0.25 m and turns are 30 degrees.Source: §5.1 · p. 8

5.2 Qualitative Studies

  • Real-world studies report effective zero-shot navigation with free-form instructions, including robustness to distractors, open-set objects, and free-form demands.The demonstrations use multi-room environments such as offices, labs, and meeting rooms.Source: §5.2 · p. 8, §5.1 · p. 8
  • ProposeLLM extracts the attribute “red” from “red chair,” and PerceptVLM masks only the red chair among three chairs.Source: §5.2 · p. 8
  • The demonstrations include navigation to an uncommon, out-of-vocabulary object: a robot arm.Source: §5.2 · p. 8
  • For an ambiguous request to wash hands, ProposeLLM proposes sink, soap, and towel, while exploration can reveal an additional goal.A tap is identified as a goal, allowing direct navigation without further exploration.Source: §5.2 · pp. 8–9

6 Conclusions

The conclusion presents OpenFMNav as a foundation-model framework for open-set zero-shot object navigation. It reports effective navigation and experiments showing the framework’s superiority, alongside real-robot demonstrations.

6 Conclusions

  • OpenFMNav is presented as a framework for open-set zero-shot object navigation that leverages foundation models to understand free-form instructions, reason, and navigate.Source: §6 · p. 9
  • The conclusion reports that extensive experiments showed the framework’s superiority.Source: §6 · p. 9
  • Real-robot demonstrations are reported as validation of the method’s open-set-ness and generalizability to real-world environments.Source: §6 · p. 9

Ethics Statement

The paper describes open-set zero-shot object navigation as applicable to diverse scenarios and as addressing ambiguous or free-form instructions. It also flags safety and privacy risks from foundation models.

Ethics Statement

  • The method is described as usable for zero-shot robotic navigation in diverse scenarios, including home and warehouse robots.Source: Ethics Statement · p. 9
  • The paper says addressing ambiguous or free-form instructions benefits human-robot interaction.Source: Ethics Statement · p. 9
  • Foundation-model safety issues, privacy leaks, and jailbreaking are identified as risks that need further attention.Source: Ethics Statement · p. 9

Limitations

The method depends on relatively accurate depth sensing to build its 2D map, and noisy depth observations or camera poses can degrade performance. Stable internet access for foundation-model APIs and LLM latency also constrain deployment.

Limitations

  • Noisy depth observations or camera poses can degrade performance because building the 2D map requires relatively accurate depth sensors.Source: Limitations · p. 9
  • Stable internet connections are required for foundation-model APIs, limiting potential large-scale deployment in harsh environments.Source: Limitations · p. 9
  • LLM responses may not be real-time, creating latency issues; the authors point to quantization and edge computing as possible mitigations.Source: Limitations · p. 9

A API Usage

Table 3 is titled “API usage.”

A API Usage

B Hyperparameters

The supplied materials identify Table 4 as covering hyperparameters and Figure 5 as showing initial prior objects.

B Hyperparameters

C Prompts and Examples

The appendix points readers to prompt figures for several model components and specifies a dot-separated object prompt for PerceptVLM. It also gives a concrete example of that formatting.

C Prompts and Examples

C.2 Prompts for DiscoverVLM

  • The cited passage states that ReasonLLM prompts are shown in Figure 7, while the figure caption identifies Figure 7 as showing DiscoverVLM prompts, leaving the model attribution inconsistent.Source: §C.2 · p. 13, Figure 7 · p. 13

C.3 Prompts for PerceptVLM

  • For PerceptVLM, object names are joined with dots to form the object prompt pt.Source: §C.3 · p. 13
  • The object list [chair, bed, plant, toilet, tv, couch] becomes the prompt “chair.bed.plant.toilet.tv.couch”.Source: §C.3 · p. 13

C.4 Prompts for ReasonLLM

Loading 2402.10670v2…