PaperlayerView exact revision on arXiv

Source-linked AI summary

LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action

Dhruv Shah, Blazej Osinski, Brian Ichter, Sergey Levine

arXiv:2207.04429v2cs.ROcs.AIcs.CLcs.LG

TL;DR

Robotic navigation systems often require either image goals or costly language-annotated trajectories. LM-Nav combines pretrained language, vision-language, and navigation models without fine-tuning, and follows natural-language instructions in complex outdoor environments, succeeding in 85% of 20 experiments.

  • Problem

    Language-based robotic navigation lacks an inexpensive alternative to trajectories annotated with textual instructions, while image goals provide an unnatural interface for users.

  • Method

    LM-Nav combines pretrained language, vision-language, and goal-conditioned navigation models to parse landmarks, ground them in images, and plan robot actions without fine-tuning.

  • Results

    85% of 20 experiments followed instructions consistently without collisions or disengagements, with failures attributed to visually localizing hard landmarks.

  • Takeaways & Limitations

    LM-Nav demonstrates that pretrained models can support long-horizon natural-language navigation by a real-world mobile robot in complex outdoor environments.

  • Takeaways & Limitations

    LM-Nav focuses on landmarks and disregards verbs or nuanced commands, and its navigation model is specific to outdoor navigation with the Clearpath Jackal robot.

Abstract

from arXiv · show

Goal-conditioned policies for robotic navigation can be trained on large, unannotated datasets, providing for good generalization to real-world settings. However, particularly in vision-based settings where specifying goals requires an image, this makes for an unnatural interface. Language provides a more convenient modality for communication with robots, but contemporary methods typically require expensive supervision, in the form of trajectories annotated with language descriptions. We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories, while still providing a high-level interface to the user. Instead of utilizing a labeled instruction following dataset, we show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data. We instantiate LM-Nav on a real-world mobile robot and demonstrate long-horizon navigation through complex, outdoor environments from natural language instructions. For videos of our experiments, code release, and an interactive Colab notebook that runs in your browser, please check out our project page https://sites.google.com/view/lmnav

1 Introduction

LM-Nav addresses the challenge of having robots follow high-level human instructions by combining independently pre-trained language, vision-language, and visual-navigation models. The system converts free-form instructions into grounded landmarks and executes a planned route using visual observations.

  • Motivation: Robotic instruction following requires understanding human commands and executing a diverse repertoire of real-world behaviors.Prior navigation methods largely learn from trajectories annotated with textual instructions.
  • System: GPT-3 decodes instructions into textual landmarks, while CLIP grounds those landmarks in ViNG’s topological map.CLIP infers a joint likelihood over landmarks and map nodes.
  • System: A novel search algorithm maximizes a probabilistic objective to find a robot plan, which ViNG then executes.The plan is derived from grounded landmarks and nodes in the topological map.
  • System: LM-Nav combines three independently pre-trained models to connect language instructions with embodied robotic control.The components are GPT-3 for language parsing, CLIP for visual-language grounding, and ViNG for navigation execution.

2 Related Work

Prior language-guided navigation methods commonly rely on structured state spaces, language-annotated trajectories, or low-level text-conditioned skills. LM-Nav instead builds on sequence prediction, pretrained vision-language models, and visual navigation policies trained without language annotations.

  • Language-guided navigation: Earlier navigation methods map free-form commands to grammar-defined formal languages, whereas related sequence-prediction approaches more closely resemble vision-and-language navigation.These methods often operate on structured state spaces, while VLN leverages fine-grained instructions.
  • Vision-and-language navigation: Most recent VLN methods use over 1M simulated, language-annotated demonstrations and sim-to-real transfer, requiring photorealistic simulators that are difficult to build for outdoor environments.The cited demonstrations cover indoor and driving scenarios, with deployment primarily in simple indoor environments.
  • Pretrained models for embodied tasks: Embodied applications of large pretrained language and image models typically fine-tune embeddings on language-labeled robot data or assume agents can execute textual instructions or text-conditioned skills.These approaches span textual, visual, and embodied domains, with some methods not addressing low-level control.
  • Pretrained models for embodied tasks: Such embodied approaches require low-level skills for rudimentary textual commands, implying language annotations for robotic experience and strong assumptions about robot capabilities.LM-Nav instead combines pretrained vision and language models with pretrained visual policies that do not use language annotations.
  • Vision-based mobile navigation: Vision-based mobile navigation learns goal-reaching policies from photorealistic simulators or supervised collection, while self-supervised methods use unlabeled trajectories with automatically generated labels and hindsight relabeling.These self-supervised policies can be trained on large, diverse datasets and generalize to previously unseen environments.

3 Preliminaries

LM-Nav combines three large pre-trained models for language processing, image-language association, and visual navigation. GPT-3 extracts textual landmarks, CLIP associates those landmarks with images, and ViNG supplies image-conditioned navigation through the environment.

  • 3 Preliminaries: LM-Nav consists of three large pre-trained models for language processing, image-language association, and visual navigation.These components provide complementary interfaces for interpreting instructions, matching observations, and controlling the robot.
  • 3 Preliminaries: GPT-3 parses textual instructions into a sequence of landmarks.The language model is a Transformer-based generative model trained on large internet-text corpora.
  • 3 Preliminaries: The system uses visual navigation connectivity, landmark-image associations, and graph search to derive a walk through the environment.The VNM constructs a topological graph from raw observations, while the VLM scores landmarks against graph nodes for route selection.
  • 3 Preliminaries: CLIP jointly encodes landmark descriptions and images into an embedding space to estimate their association likelihood.The system obtains embeddings for textual landmarks and image observations, enabling cross-modal matching.
  • 3 Preliminaries: ViNG predicts temporal distances between image pairs and corresponding actions, providing an interface between images and robot embodiment.As a goal-conditioned visual navigation model, ViNG learns navigation behavior and navigational affordances from visual observations.

4 LM-Nav: Instruction Following with Pre-Trained Models

LM-Nav converts natural-language navigation commands into landmark sequences, grounds those landmarks in graph nodes using CLIP, and selects a traversable waypoint sequence with pretrained navigation estimates. Its planning formulation combines landmark-association probabilities with graph traversal likelihoods and solves for a walk maximizing successful instruction execution.

  • 4 LM-Nav: Instruction Following with Pre-Trained Models: LM-Nav combines an LLM, VLM, and VNM to parse instructions, associate landmarks with graph nodes, and estimate navigation between node pairs.The LLM produces landmarks, the VLM estimates P(v|ℓ), and the VNM supplies traversal-based probabilities.
  • 4.1 Problem Formulation: The planner infers waypoints v1, ..., vk for landmarks ℓ1, ..., ℓn, typically requiring k ≥ n because traversal may need intermediate nodes.This is formulated probabilistically using distributions p(vi|ℓj).
  • 4.1 Problem Formulation: The discounted Markovian traversability model assigns termination probability 1 − γ at each step and reaching vi+1 probability γ^D(vi,vi+1).D(vi, vi+1) is the VNM-predicted number of time steps between nodes.
  • 4.2 Parsing Free-Form Textual Instructions: GPT-3 extracts an ordered sequence of landmarks from free-form user instructions using a prompt containing 3 examples of correct extractions.The method was reported to work for the tested instructions.
  • 4.3 Visually Grounding Landmark Descriptions: CLIP estimates P(vi|ℓj) by comparing each node image with the caption prompt “This is a photo of a [ℓj]”.The resulting probabilities are combined with inferred edge distances to select the optimal walk.
  • 4.3 Visually Grounding Landmark Descriptions: Graph search initializes Q[0,S] = 0, applies Dijkstra, then updates Q[i,v] with Q[i −1,v] + CLIP(v,ℓi) before returning a backtracked destination.The destination is selected by arg max(Q[n, ∗]).
  • 4.4 Graph Search for the Optimal Walk: Dynamic programming computes Q(i,v) as the maximal objective for a walk ending at v after visiting landmarks through index i.The resulting walk maximizes successful instruction execution and is executed sequentially by the VNM using its action estimates.

5 System Evaluation

LM-Nav was evaluated on a real outdoor robot using frozen pre-trained components and natural-language instructions, achieving reliable instruction following while exposing failures from missing or unrecognized landmarks. Component ablations show the importance of GPT-3, CLIP-ViT, the VNM, and reachability-aware planning.

  • Evaluation setup: LM-Nav was deployed on a Clearpath Jackal UGV with IMU, GPS, wheel encoders, and front- and rear-facing RGB cameras, while the VNM ran onboard.LLM and VLM queries were pre-computed remotely, and the resulting path was commanded wirelessly.
  • Instruction disambiguation: Different instruction detail produced different traversals: shorter prompts favored direct paths, whereas fine-grained route descriptions selected alternate landmark paths.The system could disambiguate multiple paths sharing the same start and goal locations.
  • Failure modes: LM-Nav failed when an instructed landmark was missing or unidentifiable, such as bypassing a fire hydrant and routing around the wrong side of a building.The failure reflects the assumption that landmarks exist in the environment and can be identified by the VLM.
  • System performance: Across 20 instructions, LM-Nav successfully followed instructions in 85% of experiments without collisions or disengagements.This corresponded to an average of 1 intervention per 6.4km of traversals.
  • Component evaluation: GPT-3 outperformed fairseq, GPT-J-6B, GPT-NeoX-20B, and spaCy for parsing free-form instructions into ordered landmarks, while CLIP-ViT produced the most reliable landmark detections.The VLM detection experiment used 100 robot-camera images and 30 commonly occurring landmarks; RCNN struggled with common objects in these out-of-distribution images.
  • Component ablations: Ablating the VNM caused frequent failures from collisions with small obstacles, while removing reachability information produced plans up to 6× longer than LM-Nav.GPS-Nav used GPS-based distances and a naive straight-line controller, limiting obstacle and traversability reasoning.

6 Discussion · Appendix · A Prompt Engineering

LM-Nav combines pretrained language, vision-language, and navigation models to navigate from textual instructions without user annotations, but its landmark-based interface limits command understanding. The appendix describes simple GPT-3 and CLIP prompting, with in-context examples for complex landmark ordering and over 95% landmark detection.

  • 6 Discussion: LM-Nav combines an LLM, VLM, and VNM to control a mobile robot from textual instructions without user annotations for navigational data.The LLM extracts landmarks, the VLM matches observations to landmarks, and the VNM supports navigation.
  • 6 Discussion: LM-Nav focuses only on landmarks and disregards verbs or other commands, making verb and nuanced-command grounding an important future direction.Examples include commands such as going straight for three blocks or driving past a dog slowly.
  • A Prompt Engineering: Prompt engineering encodes a task in model input, and the appendix applies this process to the LLM and VLM.The section frames prompting as representation optimization for task-specific use of large language models.
  • A.1 LLM Prompt Engineering: GPT-3 reliably extracts landmarks from free-form instructions and can parse simple queries effectively with a single zero-shot prompt.The outputs were robust to small changes in input prompts.
  • A.1 LLM Prompt Engineering: For complex instructions involving reordered landmark occurrences, the prompt adds three examples to leverage GPT-3 in-context learning.A cited example asks for a glass building after passing a white car.
  • A.1 LLM Prompt Engineering: The prompt used in all experiments successfully enabled GPT-3 to extract all landmarks.The paper directs comparisons with other extraction methods to Section 5.4 and Appendix C.2.
  • A.2 VLM Prompt Engineering: CLIP uses the prompt family “This is a photo of” followed by the landmark description, which detected over 95% of encountered landmarks.The experiments did not require more careful prompt engineering, although prompt ensembles have been reported to improve robustness.

B Building the Topological Graph with VNM

LM-Nav constructs its topological graph by combining VNM learned distances with GPS-based spatial proximity and timestamps, then removes redundant edges through transitive reduction.

  • Graph construction: Graph connectivity combines VNM distance estimates, GPS spatial proximity, and temporal proximity during data collection.Edges represent either physically traversed connections inferred from close timestamps or nearby nodes satisfying learned-distance and spatial thresholds.
  • Graph construction: Nodes captured within 2s are connected with edge weight equal to their VNM learned distance.The timestamp condition identifies observations captured in quick succession and adds physically traversed edges.
  • Graph construction: Otherwise, two nodes connect when their learned distance is below τ and their GPS spatial distance is below η.The graph-building algorithm returns an edge weighted by Dij when both threshold conditions hold.
  • Graph simplification: Transitive reduction removes redundant edges because the resulting graph may be quite dense.This operation simplifies the graph after connectivity has been inferred.

C Miscellaneous Ablation Experiments · C.1 Ablating the Search Objective

The search objective combines landmark likelihood with traveled-distance minimization. Ablating the distance term produces less efficient plans and lower planning success, particularly in complex environments, while the full objective filters erroneous landmark selections.

  • C.1 Ablating the Search Objective: The graph search objective factors into visiting required landmarks via Pl(¯v|¯l) and minimizing traveled distance via Pt(¯v).These components separately represent landmark selection and path-length minimization.
  • C.1 Ablating the Search Objective: The ablation selects nodes only by Pl, producing a Max Likelihood planner that chooses the most likely node for each landmark.Unlike LM-Nav, it does not reason about landmarks’ relative topological positions.
  • C.1 Ablating the Search Objective: The Max Likelihood planner suffers greatly in efficiency because it does not incentivize shorter paths.Table 4 reports the comparative planner metrics, while Figure 7 illustrates the resulting path-length difference.
  • C.1 Ablating the Search Objective: The ablation also reduces planning success, especially in complex environments.The passage attributes these failures partly to erroneous landmark detections from the vision-language model.
  • C.1 Ablating the Search Objective: 5× more efficient plan results from LM-Nav’s significantly shorter path than maximum likelihood planning in Figure 7.The figure contrasts paths between the same types of detected nodes and shows LM-Nav’s shorter route.
  • C.1 Ablating the Search Objective: Maximum likelihood planning can select an image matching a blue-dumpster prompt that actually contains a blue semi-truck and orange trailer.The example is presented as a possible variable-binding failure.
  • C.1 Ablating the Search Objective: The distance term Pt(¯v) imposes a soft constraint on landmark search, eliminating most erroneous-selection cases and significantly increasing planning success.LM-Nav still experiences some failures, but the second objective factor suppresses many of them.

C.2 Ablating the LLM

This section details experiments ablating the language model’s landmark-extraction component. Performance is measured by average extraction success, which evaluates both landmark correctness and preservation of ground-truth order.

  • Experimental setup: The experiments compare different methods for extracting landmarks, with reproducible source code provided in the repository.The implementation is available in ablation text to landmark.ipynb, referenced in Appendix D.
  • Evaluation metric: Average extraction success compares an extracted landmark list with the ground-truth list using longest common subsequence.The metric is defined for a ground-truth list Lgt and extracted list Lm.
  • Evaluation metric: The metric rewards methods that extract correct landmarks in the same order as the ground-truth sequence.Articles are ignored when comparing landmarks because they are not expected to affect downstream tasks.
  • Implementation: Experiments used model-serving APIs with default parameters, except temperature was set to 0 for deterministic landmark extraction.GPT-3 was accessed through OpenAI’s API, while the other open-source models used GooseAI; all experiments used the Appendix A prompt.

E Experiment Videos

Experiment videos show LM-Nav deployed on a Clearpath Jackal, following free-form textual instructions while navigating complex environments and disambiguating fine-grained commands.

  • Experiment Videos: LM-Nav experiment videos feature deployment on a Clearpath Jackal mobile robotic platform.The videos are available at sites.google.com/view/lmnav.
  • Experiment Videos: The videos highlight LM-Nav following free-form textual instructions in complex environments.
  • Experiment Videos: The videos also demonstrate disambiguation between fine-grained commands.
Loading 2207.04429v2…