Source-linked AI summary
Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions
Yuxing Long, Xiaoqi Li, Wenzhe Cai, Hao Dong
TL;DR
VLN requires multiple capabilities, yet prior methods rely on single-round reasoning that struggles with these subtasks. DiscussNav addresses this with zero-shot multi-expert discussions before each move, outperforming the leading zero-shot model on R2R and showing advantages on real robots.
Problem
VLN requires coordinated understanding, perception, and planning, while prior methods rely on single-round reasoning for multiple navigation subtasks.
Method
DiscussNav uses large models as specialized experts and actively consults them about instruction understanding, environment perception, and completion estimation before movement.
Results
DiscussNav outperforms the leading zero-shot VLN model on all R2R metrics and shows better real-robot navigation performance than single-round self-thinking.
Takeaways & Limitations
Discussion-based reasoning facilitates zero-shot navigation by identifying relevant information, correcting errors, and filtering inconsistent movement decisions.
Abstract
from arXiv · showhide
Visual language navigation (VLN) is an embodied task demanding a wide range of skills encompassing understanding, perception, and planning. For such a multifaceted challenge, previous VLN methods totally rely on one model's own thinking to make predictions within one round. However, existing models, even the most advanced large language model GPT4, still struggle with dealing with multiple tasks by single-round self-thinking. In this work, drawing inspiration from the expert consultation meeting, we introduce a novel zero-shot VLN framework. Within this framework, large models possessing distinct abilities are served as domain experts. Our proposed navigation agent, namely DiscussNav, can actively discuss with these experts to collect essential information before moving at every step. These discussions cover critical navigation subtasks like instruction understanding, environment perception, and completion estimation. Through comprehensive experiments, we demonstrate that discussions with domain experts can effectively facilitate navigation by perceiving instruction-relevant information, correcting inadvertent errors, and sifting through in-consistent movement decisions. The performances on the representative VLN task R2R show that our method surpasses the leading zero-shot VLN model by a large margin on all metrics. Additionally, real-robot experiments display the obvious advantages of our method over single-round self-thinking.
I. INTRODUCTION
VLN requires coordinated instruction understanding, visual perception, and completion estimation, but prior agents rely on single-round self-thinking. DiscussNav instead uses multi-expert discussions to gather task-specific information before movement.
- VLN movement decisions require understanding instructions, identifying relevant landmarks, observing scenes, and estimating completion at every step.
- Previous VLN and zero-shot language-model methods passively process navigation information and make each movement prediction through one model’s single-round reasoning.
- GPT4 still struggles to handle multiple navigation subtasks simultaneously, while trained models face a substantial simulation-to-real transfer gap.
- DiscussNav draws on expert consultation by actively discussing with domain experts before making movement decisions.
- The framework assigns large models specialized VLN roles and evaluates discussion-based navigation on R2R and real robots.
A. Vision-and-Language Navigation
VLN develops embodied agents that follow natural-language instructions through real 3D environments. Prior work emphasizes simulator-data training strategies, while dialogue-based approaches involve human participation and recent large-model research explores broader capabilities.
- VLN targets embodied agents that follow natural-language instructions while navigating real 3D environments.
- Supervised navigation methods improve simulated performance through data augmentation, memory mechanisms, and pretraining but largely overlook real-world deployment.
- Prior dialogue-enabled navigation methods require human annotation or participation, limiting their fully automated zero-shot character.
- Large language models provide broad abilities such as information summarization, code generation, and task planning for embodied-agent research.
III. METHOD
DiscussNav is a GPT4-driven zero-shot navigation agent that fills discussion templates and consults specialized experts before deciding movement. Its experts cover instruction analysis, visual perception, completion estimation, and decision testing.
- DiscussNav uses GPT4 and question templates to collect multi-expert information before prompting movement decisions.
- The framework establishes domain experts by assigning large models specific roles and task requirements through prompts.
- Instruction Analysis Experts: Instruction-analysis experts decompose action sequences and extract navigational landmarks from human instructions.
- Vision Perception Experts: Vision-perception experts use scene and object models to identify instruction-relevant visual information, including scene-level and object-level landmarks.
- Completion-estimation and decision-testing experts summarize navigation history, assess executed actions, and reconcile diverse movement predictions.
C. Navigation Discussions with Multiple Experts
DiscussNav organizes navigation as sequential consultations: it parses instructions, queries landmark-relevant views, tracks action completion, and tests multiple movement predictions before execution.
- Instruction Analysis: DiscussNav first consults action-decomposition and landmark-extraction experts to convert instructions into action sequences and navigational landmarks.
- Vision Perception: At each step, vision experts inspect 12 surrounding directions using landmark-type-specific questions to update environmental understanding.
- Qualitative Results: Real-robot discussions support open-vocabulary and fine-grained landmark perception while correcting errors and filtering inconsistent movement decisions.
- Completion Estimation: The agent summarizes navigation history and consults completion-estimation experts to determine which action remains to be executed.
- Movement Decision: DiscussNav generates N movement predictions and executes directly when they agree, otherwise sending thought-prediction pairs to decision-testing experts.
A. Experiment Setup
The experiments use multiple large-model domain experts with specified roles, alongside DiscussNav and its supporting perception models. Evaluation follows standard R2R navigation metrics.
- GPT4 constructs instruction-analysis, completion-estimation, and decision-testing experts, while ChatGPT constructs trajectory-summary and thought-fusion experts.
- InstructBLIP FlanT5XL supports scene observation, and RAM-14M supports object perception within the expert system.
- R2R evaluation reports TL, NE, SR, OSR, and SPL, covering path length, endpoint error, success, oracle stopping, and length-penalized success.
B. Simulator Quantitative Experiments
Simulator experiments show that DiscussNav substantially outperforms prior zero-shot methods on R2R, while ablations show performance declines when expert discussions are removed.
- Comparison with Previous Methods: 26.47% Success Rate and 37.93% SPL improvements over NavGPT accompany DiscussNav’s lead over all zero-shot methods and two train-only methods on five R2R metrics.
- The effect of Discussions with Domain Experts: Ablating discussions with different experts produces varying performance declines, supporting the effectiveness of multi-expert discussions for zero-shot VLN.
- The effect of Discussions with Domain Experts: The ablation validation split samples one trajectory from each of 72 environments spanning 61 training and 11 validation-unseen scenes.
C. Real Robot Quantitative Experiments
Real-robot experiments evaluate DiscussNav against pretrained and zero-shot baselines on a Turtlebot navigating semantically rich indoor instructions. DiscussNav performs better on these real-world tasks.
- The Turtlebot 4 Lite uses an OAK-D Lite camera and RPLIDAR-A1 lidar to execute 20 instructions containing open-vocabulary landmarks, fine-grained landmarks, and multiple room changes.
- DiscussNav shows better real-robot performance than the compared baselines, with advantages in instruction understanding, fine-grained landmark observation, and movement decisions.
- DuET fails to transfer simulator-learned navigation abilities to the real world, while NavGPT can navigate but is limited on semantic-rich and long instructions.
D. Qualitative Results
Qualitative analysis shows that expert discussions improve landmark perception, correct decomposed action errors, and support more reliable decisions when instructions or directions are difficult.
- Discussions help identify open-vocabulary landmarks, correct errors in decomposed action sequences, and resolve confusion among similar movement directions.
- For multiple fine-grained landmarks, DiscussNav first obtains relevant visual information and then consults completion-estimation experts before deciding.
- The real-robot analysis is presented in Table III for indoor-scene experiments.
V. CONCLUSION AND FUTURE WORK
DiscussNav is a zero-shot VLN framework that improves navigation by enabling active discussions with multiple domain experts before movement. The authors report benefits over self-thinking-based reasoning and propose expanding experts and embodied-task applications.
- DiscussNav enables a navigation agent to gather professional insights through active discussions with multiple domain experts before moving.
- Experiments indicate that discussion-based reasoning enhances large language model performance on embodied tasks compared with self-thinking-based reasoning.
- Future work will expand the domain-expert roster and extend discussion-based navigation to broader embodied tasks through diverse discussion topics.