PaperlayerView exact revision on arXiv

Source-linked AI summary

NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models

Gengze Zhou, Yicong Hong, Zun Wang, Xin Eric Wang, Qi Wu

arXiv:2407.12366v2cs.CVcs.AIcs.CLcs.RO

TL;DR

LLM-based VLN agents have a performance gap against specialist models, while integrations can underuse LLMs’ language-based communication. NavGPT-2 combines a VLM with navigation policy modules and step-wise reasoning data; its 5B model outperforms NaviLLM (7B) by 4% SR on the test split. The authors conclude that its VLM latent supports policy learning across vision, language, and action.

  • Problem

    LLM-based VLN agents show a performance gap against VLN-specialized models, and direct tuning can discard language abilities for explanation and human communication.

    Source: §1 · p. 2

  • Method

    NavGPT-2 uses a multi-image VLM, GPT-4V-generated step-wise reasoning data, and the VLM latent for language decoding and graph-based action prediction.

    Source: §1 · p. 3, Data Acquisition and Curation · p. 8

  • Results

    NavGPT-2’s 5B model outperforms NaviLLM (7B) by 4% SR on the test split.

    Source: §4.2 · p. 11

  • Takeaways & Limitations

    The VLM latent serves as a visual-linguistic representation for policy networks to learn alignment among vision, language, and action.

    Source: §5 · p. 14

    Reasoning is single-step and local, omits navigation history in the VLM, and is not strictly synchronized with policy actions.

    Source: D Limitations and Future Work · p. 25

Abstract

from arXiv · show

Capitalizing on the remarkable advancements in Large Language Models (LLMs), there is a burgeoning initiative to harness LLMs for instruction following robotic navigation. Such a trend underscores the potential of LLMs to generalize navigational reasoning and diverse language understanding. However, a significant discrepancy in agent performance is observed when integrating LLMs in the Vision-and-Language navigation (VLN) tasks compared to previous downstream specialist models. Furthermore, the inherent capacity of language to interpret and facilitate communication in agent interactions is often underutilized in these integrations. In this work, we strive to bridge the divide between VLN-specialized models and LLM-based navigation paradigms, while maintaining the interpretative prowess of LLMs in generating linguistic navigational reasoning. By aligning visual content in a frozen LLM, we encompass visual observation comprehension for LLMs and exploit a way to incorporate LLMs and navigation policy networks for effective action predictions and navigational reasoning. We demonstrate the data efficiency of the proposed methods and eliminate the gap between LM-based agents and state-of-the-art VLN specialists.

1 Introduction

LLMs offer general language understanding and human-interpretable navigational reasoning, but existing VLN integrations face performance and capability trade-offs. NavGPT-2 combines VLMs with navigation modules to pursue effective action prediction while retaining language-based reasoning and interaction.

1 Introduction

  • LLMs are attractive for robotic navigation because they can process free-form language and provide human-interpretable navigational reasoning.The introduction also highlights their potential for commonsense reasoning and communication with people.Source: §1 · p. 1
  • Zero-shot VLN relies on prompts, captions, and language summaries, yet incurs fragile prompting and information loss and trails specialist models by about 40% success rate on R2R.The authors also question whether LLMs correctly understand spatial structure and the consequences of physical motions.Source: §1 · pp. 2–3
  • Fine-tuning LLMs for VLN still underperforms specialists and can discard general language abilities needed to explain navigation and communicate with people.The introduction attributes the performance gap in part to limited training data and a mismatch between LLM pretraining objectives and VLN targets.Source: §1 · p. 3
  • NavGPT-2 combines a multi-image VLM with visual instruction tuning on step-wise reasoning data and uses its latent representation for language and graph-based action decoding.The design aims to retain LLM language abilities while supporting navigation history and backtracking.Source: §1 · p. 3
  • NavGPT-2 aims to explain navigation decisions and support interactive use, including responding to user intervention and visual questions.The authors present combining VLN specialists with VLMs without LLM training as a contribution.Source: Figure 1 · p. 2, §1 · p. 3

2 Related Works

Related work covers VLN alignment and planning, LLM-based navigation strategies, and methods for aligning visual content with pretrained language models. NavGPT-2 adopts a query-based resampler for multi-view visual input.

Vision-and-Language Navigation

  • VLN methods address vision-language alignment and action planning, using approaches such as historical-state memorization, self-correction, navigation maps, and external knowledge prompts.Alignment approaches include generic visual-linguistic representations, data augmentation, and training strategies.Source: §2 · p. 4

Large Language Models in VLN

  • LLMs have been integrated into VLN through model ensembles, zero-shot agents, and fine-tuning, while zero-shot systems still show a gap from supervised methods even with GPT-4.The passage also describes fine-tuning language models on VLN data and multitask learning for VLN generalists.Source: §2 · p. 4

Modality Alignment in Large Vision-Language Models

  • VLM alignment methods use query-based visual resamplers or fully connected projections; NavGPT-2 adopts a Q-former to control visual-content length for multi-view inputs.The passage identifies query-based resamplers in Flamingo and BLIP-2 and later implementations, and projections in LLaVA and MiniGPT-v2.Source: §2 · p. 4

3 Method

NavGPT-2 combines a vision-language model with a topological-graph navigation policy, using visual and instruction representations for reasoning and action prediction. Its training separates reasoning-data tuning from policy learning.

3.1 VLMs Latent as Visual-Linguistic Representation

3.2 Graph Based Navigation Policy

  • The topological graph stores visited and adjacent unexplored nodes, supporting planning across the graph and backtracking to unvisited nodes.Visited-node views are average-pooled; unexplored nodes use partial pooling of views from adjacent visited nodes.Source: §3.2 · pp. 6–7, Node Embedding · p. 7
  • Node embeddings are cross-attended with LLM-encoded instructions, then graph-aware self-attention models node relationships using distances and visual similarities.A two-layer feed-forward network scores nodes, and the agent follows the shortest path to the highest-scoring target while masking visited nodes.Source: Cross-Modal Encoding · p. 7, Global Action Prediction · p. 8

3.3 Multi-stage Learning for Action and Reasoning

  • Training has two stages: tune the Q-former with the LLM and vision encoder frozen, then tune the navigation policy with the VLM frozen.The first stage starts from InstructBLIP after visual instruction tuning on academic-task VQA datasets.Source: §3.3 · p. 8
  • GPT-4V generates single-step reasoning from intermediate ground-truth trajectory observations, defining the immediate surroundings and intended next direction or action.The authors randomly select 10k intermediate steps from the R2R training trajectories for GPT-4V input.Source: Data Acquisition and Curation · p. 8, Figure 4 · p. 8
  • Policy learning combines behavior-cloning and DAgger losses, with DAgger pseudo-labels determined by the shortest path to the destination from the partial graph.The partial graphs are generated through on-policy action sampling, and the losses are combined using a balancing factor for behavior cloning.Source: Policy learning with DAgger · p. 9

4 Experiments

NavGPT-2 is evaluated against navigation specialists, language-based methods, and DUET variants, with results covering R2R performance, reasoning quality, data efficiency, and cross-dataset generalization.

4 Experiments

  • Evaluation covers path length and error, SR and OSR, SPL, NDTW, and SDTW, capturing success, efficiency, and path fidelity.SR counts paths ending within 3 meters of the target; SPL combines success with efficiency.Source: Evaluation Metrics · p. 9

4.1 Implementation Details

  • NavGPT-2 uses a frozen InstructBLIP-based VLM and trains the Q-former before fine-tuning the downstream policy network.The best model uses R2R data combined with synthetic PREVALENT data.Source: §4.1 · p. 9

4.2 Comparison with State of the Art

  • 4% SR over NaviLLM on the R2R test split; NavGPT-2 also exceeds the baseline by 5% SR and 2% SPL, and original DUET by 3% SR at the same MP3D training scale.The comparison reports performance on the test split and test unseen split, respectively; the current SOTA uses additional training environments.Source: Table 1 · p. 10, §4.2 · p. 11

4.3 Navigational Reasoning Generation

  • 1.66 Accuracy, 1.93 Informativeness, and 1.78 Rationality (out of 3) were NavGPT-2’s human-study scores; GPT-4V scored 2.31, 2.95, and 2.34, respectively.The example reasoning describes surroundings, tracks instruction progress, and infers the next step and expected observation.Source: §4.3 · pp. 11–12, Table 2 · p. 11

4.4 The Effect of Data Amount

  • NavGPT-2 matches full-data DUET performance on validation-unseen SR using 50% less R2R training data, and outperforms all tested DUET data-scale variants.The comparison tests DUET trained with 10%, 50%, and all R2R training data.Source: §4.4 · p. 12, Table 3 · p. 12

4.5 Cross Dataset Generalization Ability

  • 3.67% higher SR than DUET on RxR English unseen; NavGPT-2 also significantly outperforms DUET zero-shot on visually and structurally distinct HM3D.The RxR evaluation tests finer-grained instructions with richer landmarks and longer trajectories.Source: §4.5 · pp. 12–13, Table 4 · p. 13

4.6 Ablation Study

  • Removing the navigation-specific policy causes a drastic performance drop, while pretraining the Q-former on reasoning brings slight SR improvement.The ablation attributes the latter improvement to spatial descriptions and landmarks that may improve textual representations of multi-view observations.Source: §4.6 · p. 13, Table 5 · p. 14
  • 3.79% SR gain on Val Unseen came from FlanT5 XL (1.5B) to XXL (5B), while larger Vicuna models performed worse.The authors hypothesize that FlanT5’s full encoder attention better supports visual-instruction alignment than Vicuna’s causal attention, leaving detailed investigation for future work.Source: Table 6 · p. 14, §4.6 · p. 14

5 Conclusion

The paper presents VLM latents as an efficient visual-linguistic representation for policy learning, while retaining navigational reasoning and communication.

5 Conclusion

  • VLM latents serve as a superior and more efficient visual-linguistic representation for navigation policy networks to learn vision-language-action alignment.The conclusion frames the approach as a scalable framework for leveraging LLM language comprehension in navigation agents.Source: §5 · p. 14
  • The approach aims to close the gap between LLM-based and VLN-specialized agents while retaining generated navigational reasoning.The conclusion connects this reasoning capacity with potential interaction and understanding of free-form human intentions.Source: §5 · p. 14
  • The proposed framework points toward versatile navigation agents that can interact with humans and understand free-form intentions.Source: §5 · p. 14

Supplementary Material for NavGPT-2

The supplement outlines additional DUET details, GPT-4V prompting, experiments, limitations, and broader impacts. It describes DUET’s policy branches and how its visual embeddings represent panoramic views and navigable directions.

Supplementary Material for NavGPT-2

A DUET Revisit

  • DUET’s downstream policy uses a text encoder and global and local branches for coarse- and fine-scale cross-modal reasoning.Source: A DUET Revisit · p. 21

A.1 Text Embedding and Visual Embedding

  • DUET encodes 36 views per node with directional and navigable embeddings, then uses a two-layer transformer to represent panoramic spatial relations.Source: §A.1 · p. 21
  • NavGPT-2 instead inputs only navigable views and adds step and location embeddings before its two-layer transformer.Source: §A.1 · p. 21

A.2 DUET Local Branch

The DUET local branch predicts actions from the current instruction and egocentric observation, while dynamic fusion reconciles its local action space with the global branch. The supplement also identifies where the GPT-4V navigation-reasoning prompt is shown.

A.2 DUET Local Branch

  • The local branch predicts from the current node’s instruction and egocentric observation, without graph information beyond the local observation.Source: §A.2 · p. 21
  • Two location embeddings encode the current node’s direction from the start and adjacent views’ egocentric directions; a four-layer cross-modal transformer masks unnavigable views.Source: Local Visual Embedding · p. 22, Local Cross-model Encoding · p. 22

A.3 Dynamic Fusion

  • Local action scores are mapped into the global action space, with scores for visited nodes outside the adjacent set forming a backtrack score.Source: §A.3 · p. 22
  • The final navigation score combines global logits with adjusted local scores using a learnable fusion scalar.Source: §A.3 · p. 22

B GPT-4V Prompt

C Additional Rerults

The additional experiments examine policy-network choices and visual features. In DUET’s global branch, replacing the visual representation with NavGPT-2’s stronger vision features yields little performance gain, which the authors hypothesize reflects the branch’s coarse-scale alignment.

C Additional Rerults

  • The experiments align DUET’s training schema by initializing it from LXMERT without VLN-specialized pretraining.Source: C Additional Rerults · p. 22

C.1 Effect of Vision Encoder

  • Switching DUET’s global branch to NavGPT-2’s stronger visual features yields little performance gain, which the authors attribute to coarse-scale alignment; fine-grained alignment occurs in the local branch.Source: Table 7 · p. 23, §C.1 · p. 23

C Additional Rerults

C.2 Effect of VLN Pretrain

Under the same training schema and data scale, NavGPT-2 performs significantly better than original DUET without VLN-tailored pretraining. The authors attribute the gain to LLM features projected into the language space, while noting qualitative hallucinations and direction errors.

C.2 Effect of VLN Pretrain

  • NavGPT-2 significantly outperforms original DUET under the same training schema and data scale, without VLN-specialized pretraining.Source: §C.2 · pp. 23, 25
  • The authors attribute the gain to LLM features projected into the language space by the Q-former and leave downstream-policy pretraining for future work.Source: §C.2 · pp. 23, 25
  • Qualitative results show correct object and scene recognition and instruction-grounded planning, but also hallucinated objects and misjudged directions.Source: Figure 6 · p. 24

C.3 Additional Qualitive Results

NavGPT-2 produces detailed accounts of surroundings and orientation, relates navigation progress to instruction segments, and predicts observations from current visual input.

C.3 Additional Qualitive Results

  • NavGPT-2 describes surrounding objects, scenes, and their orientations during navigation.Its reasoning also reflects the agent’s understanding of the environment.Source: §C.3 · p. 25
  • It relates navigation progress to specific portions of the instruction.Source: §C.3 · p. 25
  • It predicts expected observations from current visual inputs, such as a view that appears to lead to a bedroom.Source: §C.3 · p. 25

D Limitations and Future Work

The paper identifies limits in evaluating and synchronizing NavGPT-2’s reasoning with policy actions, and notes that its experiments use an open-source VLN dataset in simulation. It also reports hallucinated scenes or objects and missed object directions, leaving practical safety and communication questions for future work.

D Limitations and Future Work

E Broader Effect

  • Experiments use an open-source VLN dataset in a simulated environment, so practical application remains to be explored.Source: E Broader Effect · p. 25
  • VLMs can hallucinate scenes or objects and miss object directions; future work includes harmonizing actions with reasoning for safer real-world use.Source: E Broader Effect · p. 26
Loading 2407.12366v2…