Source-linked AI summary
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
Gengze Zhou, Yicong Hong, Qi Wu
TL;DR
NavGPT addresses the under-explored question of whether LLMs can reason about interactive worlds, actions, and consequences for navigation. It uses explicit text-based reasoning over visual observations, instructions, history, and navigation options for zero-shot VLN, and experiments show high-level planning and spatial-awareness capabilities, although zero-shot VLN performance remains below trained methods.
Problem
LLM reasoning for understanding interactive worlds, actions, and consequences in navigation remains under-explored despite the scale and emergent reasoning abilities of modern LLMs.
Method
NavGPT is a fully automatic LLM-based navigation system that combines visual descriptions from visual foundation models, language guidance, navigation history, and explicit Thoughts and Actions for zero-shot VLN.
Results
Experiments show that GPT-4 can decompose instructions, use navigation-relevant commonsense, identify landmarks, track progress, adjust plans, generate trajectory instructions, and draw top-down trajectories.
Takeaways & Limitations
Explicit LLM reasoning reveals potential for embodied navigation planning and may benefit from interaction with specialized downstream models or multimodal navigation LLMs.
Takeaways & Limitations
NavGPT’s zero-shot VLN performance remains below trained methods because it is limited by visual-scene caption quality, information loss, and navigation-history tracking constraints.
Abstract
from arXiv · showhide
Trained with an unprecedented scale of data, large language models (LLMs) like ChatGPT and GPT-4 exhibit the emergence of significant reasoning abilities from model scaling. Such a trend underscored the potential of training LLMs with unlimited language data, advancing the development of a universal embodied agent. In this work, we introduce the NavGPT, a purely LLM-based instruction-following navigation agent, to reveal the reasoning capability of GPT models in complex embodied scenes by performing zero-shot sequential action prediction for vision-and-language navigation (VLN). At each step, NavGPT takes the textual descriptions of visual observations, navigation history, and future explorable directions as inputs to reason the agent's current status, and makes the decision to approach the target. Through comprehensive experiments, we demonstrate NavGPT can explicitly perform high-level planning for navigation, including decomposing instruction into sub-goal, integrating commonsense knowledge relevant to navigation task resolution, identifying landmarks from observed scenes, tracking navigation progress, and adapting to exceptions with plan adjustment. Furthermore, we show that LLMs is capable of generating high-quality navigational instructions from observations and actions along a path, as well as drawing accurate top-down metric trajectory given the agent's navigation history. Despite the performance of using NavGPT to zero-shot R2R tasks still falling short of trained models, we suggest adapting multi-modality inputs for LLMs to use as visual navigation agents and applying the explicit reasoning of LLMs to benefit learning-based models.
1 Introduction
NavGPT investigates whether LLMs can reason about interactive navigation from textualized observations, actions, and history. It presents a zero-shot VLN system whose experiments expose high-level planning and reasoning capabilities while identifying information loss as a performance bottleneck.
- LLM reasoning in navigation remains under-explored, particularly whether models can understand interactive worlds, actions, and consequences in textual form.
- NavGPT is a fully automatic LLM-based navigation system that processes multimodal inputs, unrestricted language guidance, open-world interaction, and navigation history.
- Experiments show high-level planning through sub-goal decomposition, commonsense integration, landmark identification, progress tracking, and plan adjustment for anomalies.
- LLMs can generate navigation instructions from trajectories and draw top-down metric maps, revealing historical and spatial awareness for navigation.
- The authors attribute NavGPT’s zero-shot performance gap to trained models primarily to information loss in visual captioning and history summarization.
- NavGPT performs zero-shot VLN using off-the-shelf LLMs without learnable modules or prior interactive-navigation experience.
2 Related Work
Related work frames VLN as a generalization challenge in unseen environments and reviews LLM-based efforts to bring language reasoning and knowledge into navigation.
- VLN seeks broadly applicable embodied navigation agents, with generalization to unseen environments identified as a central challenge.
- Data augmentation, memory mechanisms, and pre-training address data scarcity but remain tied to sampled data from fixed scene sets.
- Large-scale language-model training has produced increasingly capable models with potential applications across domains.
- Prior LLM-based navigation work includes extracting landmarks, generating code, and using commonsense relations between targets and observed objects.
3 Method
NavGPT converts visual observations, instructions, candidate directions, and navigation history into prompts for an LLM that explicitly reasons before selecting actions. Visual foundation models translate scenes into language and auxiliary detections, while buffers and summarization manage history.
- VLN Problem Formulation: VLN represents each state with an instruction, egocentric observations, candidate viewpoints, and a navigation graph constraining available actions.
- VLN Problem Formulation: NavGPT performs zero-shot VLN because its policy parameters come from the LLM’s language corpus rather than VLN datasets.
- NavGPT: At each step, NavGPT combines visual perception, language instruction, navigation history, and navigation principles through a prompt manager to predict the next action.
- Visual Perception: Visual foundation models translate observations into natural-language descriptions and lower-level object detections that the prompt manager aggregates.
- NavGPT: The navigation history stores observation, reasoning, and action triplets; GPT-3.5 summarizes observations to keep the prompt within the LLM’s context capacity.
- Visual Perception: Each viewpoint uses 8 heading directions and 3 elevation levels, producing 24 egocentric views for visual processing.
- Reasoning and Action: NavGPT expands the action space with non-environment-triggering reasoning traces, prompting the LLM to think before choosing navigation actions.
- Reasoning and Action: Explicit reasoning supports complex planning, including decomposing instructions and planning toward sub-goals under new observations.
4 Experiment
Experiments evaluate NavGPT on R2R navigation, qualitative reasoning, spatial-history understanding, supervised-method comparison, and visual-component effects. Results show explicit planning and spatial reasoning, while performance remains constrained by scene-description precision, object tracking, and accumulated history information loss.
- Experimental Setup: NavGPT is evaluated with GPT-4 and GPT-3.5 on R2R, using BLIP-2 for image translation and Faster-RCNN for object detection.The experiments use the 783 trajectories in the 11 validation-unseen environments for evaluation and comparison.
- Qualitative Results: GPT-4 performs high-level navigation reasoning by tracking progress, decomposing long instructions into sub-goals, identifying landmarks, and handling unexpected observations.When observations are unexpected, NavGPT can explore and use commonsense knowledge to support decisions.
- History and Spatial Relations: GPT-4 extracts landmarks and action-linked navigation histories from redundant observations, tracks progress, and draws top-down trajectories using relative viewpoint relations.The trajectory reconstruction uses actions, observations, and the complete navigation history.
- Comparison with Supervised Methods: Compared with previous models trained on R2R, NavGPT’s zero-shot sequential navigation shows a significant performance discrepancy.The authors attribute the limitation primarily to imprecise language-based scene descriptions and insufficient object tracking.
- Limitations: NavGPT depends on visual-caption quality and summarizes growing navigation histories, causing information loss that can weaken tracking over longer trajectories.Missing target objects in captions force exploration, while summarization limits the continuity of environmental perception.
- Effect of Visual Components: Using finer-grained visual descriptions surpasses variants #1 and #2 by 6.48% and 2.78%, respectively.The study varies field of view, viewing angles, and the number of views used to describe each observation.
- Effect of Visual Components: Adding object information increases SR by 4.86%, while adding depth information further raises SR by 0.7% and SPL by 1.29%.Object information emphasizes salient scene objects, whereas depth helps the agent estimate its position relative to the destination.
5 Conclusion
NavGPT is an autonomous LLM system for language-guided navigation that processes multimodal inputs, unrestricted guidance, open-world interaction, and navigation history. Its zero-shot VLN performance remains below trained methods, while GPT-4 reasoning traces indicate potential for embodied navigation planning.
- NavGPT processes multimodal inputs, unrestricted language guidance, open-world environments, and navigation history for language-guided navigation.
- Zero-shot VLN performance is limited by the quality of visual-scene language descriptions and object-tracking abilities.
- NavGPT’s zero-shot VLN performance is still not compatible with trained methods.
- GPT-4 reasoning traces illuminate the latent potential of LLMs for embodied navigation planning.
Supplementary Material for NavGPT
The supplementary material expands NavGPT’s prompts and observation-description examples, documents GPT-4 prompting for instruction generation and trajectory drawing, and presents failure cases and broader impacts.
- Section A provides additional details about NavGPT components, prompts, and observation-description examples.
- Section B describes prompting GPT-4 to generate instructions and draw top-down trajectories.
- Section C illustrates NavGPT limitations through failure cases, while Section D discusses broader impacts.
A.1 Convert Visual Perception to Language Description (§3.22)
NavGPT converts visual observations into language by capturing three elevation views per direction, captioning them with BLIP-2, and summarizing their descriptions with GPT-3.5.
- Each viewpoint direction is observed through three egocentric images captured at elevation angles −30°, 0°, and 30°.The three images have a 45° field of view with 15° overlap.
- BLIP-2 captions the three images, after which GPT-3.5 summarizes the descriptions into one sentence.
- The summarizer reduces redundancy caused when the same object appears across multiple elevation images.
- The prompt manager combines summarized directional descriptions with detected objects, depth, and navigable-viewpoint information.
Observation for a viewpoint
NavGPT represents viewpoint observations and navigation history as language prompts, then uses GPT-4 to reason over navigation steps and produce actions, instructions, or top-down trajectories.
- The navigation principle defines the VLN task and NavGPT’s reasoning format and rules at each navigation step.
- A summarized viewpoint description replaces earlier full observations in navigation history, while the current viewpoint retains its full description.
- NavGPT’s prompt frames navigation as moving through static viewpoints in a predefined graph toward an instructed target with the least steps.
- A successful example records the initial apartment observation, reasoning about the only navigable direction, and the resulting movement decision.
- GPT-4 receives the navigation system prompt and navigation history to draw a top-down map of viewpoint positions.
- GPT-4 can also write an instruction for another agent to follow the trajectory represented by navigation history.
C NavGPT Failure cases (§4.2)
NavGPT’s failures arise from information loss when visual scenes are compressed into language and from weak object tracking across viewpoints. The authors suggest dynamically generating descriptions to preserve essential environmental information during navigation.
- NavGPT can miss required target objects when they are absent from the observation description, forcing further exploration.The failure reflects information loss caused by the lower granularity of linguistic descriptions relative to visual perception.
- Summarized historical observations omit viewpoint details, preventing NavGPT from comparing scenes to maintain a continuous environmental understanding.Without a reference between viewpoints, the agent may not determine whether a previous action was completed.
- In one example, NavGPT incorrectly believed it had turned right into the next room even though it had not entered it.
- Dynamic language-description generation is proposed to let the agent extract essential environmental information during navigation instead of relying on static descriptions.The proposal is compared with the interaction-based approach of ChatCaptioner.
D Broader Impact
NavGPT is presented as an early step toward versatile VLN agents, but its current evaluation remains confined to a simulated open-source dataset. The authors emphasize that deployment requires more reliable planning and sequential action prediction because generative-model robustness is not yet guaranteed.
- The experiments are conducted on an open-source VLN dataset in a simulated environment to keep agent behavior controllable.The authors connect this setting to safety and ethical concerns at the current stage.
- The robustness of generative-model performance cannot yet be guaranteed for practical implementation.
- Further research should improve prompting for accurate planning and precise sequential action prediction before real-world deployment.The authors identify these improvements as important for safety.