PaperlayerView exact revision on arXiv

Source-linked AI summary

L3MVN: Leveraging Large Language Models for Visual Target Navigation

Bangguo Yu, Hamidreza Kasaei, Ming Cao

arXiv:2304.05501v2cs.RO

TL;DR

Visual target navigation requires robots to locate specified household objects in unknown environments without relying solely on costly learned scene priors. L3MVN uses large language models to rank semantic frontiers through zero-shot and feed-forward approaches, reporting improved success rate and efficiency on Gibson and HM3D plus real-world validation.

  • Problem

    Visual target navigation requires robots to locate specified household objects in unknown environments, while prior methods often require costly learning of scene priors.

  • Method

    L3MVN builds maps and uses large language models to infer frontier relevance through zero-shot and feed-forward approaches for selecting long-term exploration goals.

  • Results

    L3MVN significantly improves success rate and efficiency on Gibson and HM3D while avoiding large-scale learning processes, with additional real-world validation.

  • Takeaways & Limitations

    Language models and cost-utility exploration support more efficient semantic navigation and enable practical robot deployment.

Abstract

from arXiv · show

Visual target navigation in unknown environments is a crucial problem in robotics. Despite extensive investigation of classical and learning-based approaches in the past, robots lack common-sense knowledge about household objects and layouts. Prior state-of-the-art approaches to this task rely on learning the priors during the training and typically require significant expensive resources and time for learning. To address this, we propose a new framework for visual target navigation that leverages Large Language Models (LLM) to impart common sense for object searching. Specifically, we introduce two paradigms: (i) zero-shot and (ii) feed-forward approaches that use language to find the relevant frontier from the semantic map as a long-term goal and explore the environment efficiently. Our analysis demonstrates the notable zero-shot generalization and transfer capabilities from the use of language. Experiments on Gibson and Habitat-Matterport 3D (HM3D) demonstrate that the proposed framework significantly outperforms existing map-based methods in terms of success rate and generalization. Ablation analysis also indicates that the common-sense knowledge from the language model leads to more efficient semantic exploration. Finally, we provide a real robot experiment to verify the applicability of our framework in real-world scenarios. The supplementary video and code can be accessed via the following link: https://sites.google.com/view/l3mvn.

I. INTRODUCTION

Visual target navigation requires robots to explore unknown 3D environments while using semantic reasoning to locate an object category. L3MVN addresses costly learned scene priors by using language models to rank relevant frontiers through zero-shot and feed-forward paradigms.

  • Visual target navigation asks a robot to efficiently explore an unknown 3D environment and locate an instance of a specified object category.
  • Traditional geometric-map methods struggle to generalize to new instructions because they lack scene priors, while learning-based methods require substantial training resources.
  • L3MVN uses language models to infer which frontier is most relevant to the target object from descriptions of observed frontier contents.
  • The approach is evaluated in Gibson and HM3D simulation environments and in a real-world robot setting.
  • The framework analyzes zero-shot and feed-forward paradigms for semantic reasoning about objects using language models.

II. RELATED WORK

Prior visual navigation methods use end-to-end learning, graphs, demonstrations, or map-based policies, but commonly require scene-prior learning. L3MVN instead uses large language models to support frontier-based navigation and generalization.

  • A. Visual Navigation: ObjectNav requires an agent to find an object category in an unknown environment, while classical map-based approaches can struggle with inaccurate, unexplored, or changing scenes.
  • A. Visual Navigation: End-to-end methods use visual encoders, recurrent memory, graphs, large-scale training, demonstrations, augmentation, or auxiliary tasks to improve navigation.
  • B. Navigational Policy Learning: Deep reinforcement-learning and module-based methods still learn scene priors from datasets, whereas L3MVN obtains priors from language models.
  • C. Learning from Pretrained Models: Existing pretrained-model approaches apply models to visual encoding, object localization, or transfer, but several still require additional learning for navigation.
  • C. Learning from Pretrained Models: L3MVN specifically learns a frontier-based navigation policy from large language models, addressing a use of natural language processing not previously applied to this task.

III. THE PROPOSED METHOD

The proposed method defines visual target navigation as finding an object category in an unknown environment under a bounded action space. Episodes begin from a random position with a target category and end when the agent stops near the target.

  • Proposed Framework: The framework uses two large-language-model paradigms to infer semantic relevance from observed frontiers for target navigation.
  • Task Definition: Each episode is defined by a scene, target category, and random initial position, with the agent navigating to an object in the appointed category.
  • Task Definition: The action space contains movement, rotation, and stop actions for navigating and terminating an episode.
  • Task Definition: A stop action is successful when the agent is within 0.1m of the target object, and each episode permits at most 500 time steps.

B. Overview

L3MVN builds a semantic map and frontier map from observations, uses language-model relevance to select a long-term frontier goal, and applies a local policy to navigate there and search.

  • B. Overview: The agent uses observations to build a semantic map, extracts frontiers, and selects a long-term goal according to target-object relevance inferred by language models.
  • B. Overview: The local policy plans a path and executes actions after the long-term goal is selected.
  • 2) Frontier Map:: Frontiers are extracted from explored and obstacle maps by identifying edges, dilating obstacle boundaries, clustering frontier cells, and removing small clusters.
  • 2) Frontier Map:: SCU(a) = U(a) −λCUC(a) combines a utility function and cost function, with λCU controlling their relative importance.

D. Global Policy

The global policy summarizes objects around each frontier and uses language models to estimate which frontier is relevant to the target. It combines zero-shot scoring or feed-forward prediction with entropy-based filtering of informative objects.

  • Global frontier representation: Each frontier is represented by a query sentence summarizing the semantic objects detected within a search window.The resulting frontier information is used by both language-based selection paradigms.
  • Zero-shot paradigm: Zero-shot selection scores which target category is best described by each frontier query.The approach uses a pre-trained language model without task-specific frontier-selection training.
  • Feed-forward paradigm: Feed-forward selection embeds each frontier query and predicts a distribution over target categories with a fine-tuned neural network.The network consumes the language-model embedding and produces category-level predictions.
  • Language-based relevance: Masked language models compare frontier descriptions using semantic and grammatical plausibility, favoring commonsense-compatible object combinations.For example, a description containing sink, bathtubs, and toilet receives a higher score than one replacing toilet with tv.
  • Entropy-based filtering: Low-entropy objects are treated as more informative because their conditional target-category distributions are less uniform.The method computes conditional probabilities from ground-truth co-occurrences and retains the 15 lowest-entropy objects for query construction.

2) Zero-shot Approach:

The zero-shot approach creates one target-specific query for each frontier and selects the frontier whose language-model score is highest for the target object.

  • Query construction: For each frontier, the method constructs one query string per target category using the objects detected in that frontier area.Each query combines the frontier’s detected objects with a candidate target object.
  • Frontier selection: The selected frontier is the one with the highest target-specific query sentence probability under the language model.This score provides the frontier relevance estimate used for long-term goal selection.

3) Feed-forward Approach:

The feed-forward approach embeds a single object-summary query for each frontier and uses a fine-tuned prediction head to score target categories. Frontier selection also incorporates the map’s cost-utility score when language evidence is weak or extreme.

  • Query construction: Each frontier is described by the query W_fi = “This frontier contains o1, · · · , and ok.”The query lists the objects detected around that frontier.
  • Prediction head: The language model converts each query into an embedding, and a fine-tuned neural-network head outputs logits for the target categories.The inferred frontier corresponds to the maximum value of the target object’s output.
  • Evaluation: Table I reports the comparative navigation results across the evaluated map-based methods.The supplied passage identifies the table as the comparison-study results, without providing its individual values.
  • Score integration: The map score SCU is combined with the language-model score SLLM to handle frontiers lacking informative objects or containing unrelated objects.A bound B normalizes SCU before the selection strategy is applied.

E. Local Policy

The local policy uses the Fast Marching Method to navigate toward the selected long-term goal, repeatedly updating the local map and goal from new observations. The study evaluates the method in simulation and on a real-world robot platform.

  • Local navigation: The Fast Marching Method plans navigation from the current position to the selected long-term goal.The agent then selects a local goal within a restricted range and executes an action to reach it.
  • Local updates: The local map and local goal are updated at each step using new observations.This module-based design avoids learning obstacle avoidance from scratch and improves training efficiency.
  • Evaluation: The method is evaluated against map-based baselines in simulation and applied on a real-world robot platform.The real-world experiment is intended to validate practicality for navigational tasks.

1) Dataset:

Experiments use Gibson and HM3D simulated indoor environments, six object categories, and RGB-D observations with map-based language-model variants. Evaluation reports Success Rate, SPL, and DTG.

  • Experiments use Gibson and HM3D, with 25 training and 5 validation scenes for Gibson and 75 training and 20 validation scenes for HM3D.
  • The six goal categories are chair, couch, potted plant, bed, toilet, and TV.
  • The Habitat observation space contains 480 × 640 RGB-D images, base odometry, and an integer goal-object representation.
  • The zero-shot variant uses RoBERTa-large to evaluate queries generated from semantic observations around frontiers and the target object.
  • The feed-forward variant fine-tunes RoBERTa-large and trains head networks on HM3D room-object data to evaluate frontier queries.
  • Evaluation uses Success Rate, SPL, and DTG; DTG is the distance from the agent to the target when the episode ends.

4) Baselines:

The comparison includes random, classical frontier-based, map-sampling, semantic-map, and potential-function baselines. The proposed framework outperforms all baselines across Gibson and HM3D.

  • The baselines include Randomly Walking, Frontier-based Policy, Randomly Sampling on Map, SemExp, and PONI.
  • 30?
  • Random walking fails in almost all episodes, while random long-term-goal sampling on a map exceeds the classical frontier-based method.
  • SemExp improves exploration through semantic information, and PONI further improves performance while reducing computational costs.
  • The proposed framework consistently outperforms all baselines across both datasets, with feed-forward relevance estimates more accurate than zero-shot estimates in large indoor scenes.

6) Ablation study:

Ablations identify the language module and cost-utility exploration as important components, while real-world tests expose sensor discrepancies between simulation and deployment. The module-based design supports transfer despite noisy measurements.

  • Ablation study: The complete HM3D model achieves the best ablation performance, while removing the LLM or replacing cost-utility exploration reduces success rate and SPL.
  • Ablation study: Ground-truth semantic segmentation improves performance in all tested cases, identifying semantic-mapping errors as a main failure source.
  • Real-world experiment: Real-world evaluation uses ROS, a Jackal robot, a Realsense D455 camera, and Ouster lidar while matching sensor settings to simulation where possible.
  • Real-world experiment: Real-world depth, lidar, and odometry measurements are less accurate than simulation, producing noisy points around objects and walls.
  • Real-world experiment: The module-based framework transfers with minimal fine-tuning because it takes a scene map as input rather than direct noisy images.
  • Conclusion: The study reports improved success rate and efficiency, with language models and cost-utility exploration supporting more efficient navigation.
Loading 2304.05501v2…