Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions

Yuxing Long; Xiaoqi Li; Wenzhe Cai; Hao Dong

Source-linked AI summary

Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions

Yuxing Long, Xiaoqi Li, Wenzhe Cai, Hao Dong

arXiv:2309.11382v1Sep 20, 2023cs.ROcs.AIcs.CLcs.CV

TL;DR

VLN requires multiple capabilities, yet prior methods rely on single-round reasoning that struggles with these subtasks. DiscussNav addresses this with zero-shot multi-expert discussions before each move, outperforming the leading zero-shot model on R2R and showing advantages on real robots.

Problem
VLN requires coordinated understanding, perception, and planning, while prior methods rely on single-round reasoning for multiple navigation subtasks.
Source: §I · p. 1
Method
DiscussNav uses large models as specialized experts and actively consults them about instruction understanding, environment perception, and completion estimation before movement.
Source: §I · p. 2, C. Navigation Discussions with… · p. 4
Results
DiscussNav outperforms the leading zero-shot VLN model on all R2R metrics and shows better real-robot navigation performance than single-round self-thinking.
Source: B. Simulator Quantitative Experiments · p. 5, C. Real Robot Quantitative Experiments · p. 6
Takeaways & Limitations
Discussion-based reasoning facilitates zero-shot navigation by identifying relevant information, correcting errors, and filtering inconsistent movement decisions.
Source: D. Qualitative Results · p. 6, §V · p. 6

Abstract

from arXiv · show

Visual language navigation (VLN) is an embodied task demanding a wide range of skills encompassing understanding, perception, and planning. For such a multifaceted challenge, previous VLN methods totally rely on one model's own thinking to make predictions within one round. However, existing models, even the most advanced large language model GPT4, still struggle with dealing with multiple tasks by single-round self-thinking. In this work, drawing inspiration from the expert consultation meeting, we introduce a novel zero-shot VLN framework. Within this framework, large models possessing distinct abilities are served as domain experts. Our proposed navigation agent, namely DiscussNav, can actively discuss with these experts to collect essential information before moving at every step. These discussions cover critical navigation subtasks like instruction understanding, environment perception, and completion estimation. Through comprehensive experiments, we demonstrate that discussions with domain experts can effectively facilitate navigation by perceiving instruction-relevant information, correcting inadvertent errors, and sifting through in-consistent movement decisions. The performances on the representative VLN task R2R show that our method surpasses the leading zero-shot VLN model by a large margin on all metrics. Additionally, real-robot experiments display the obvious advantages of our method over single-round self-thinking.

I. INTRODUCTION

VLN requires coordinated instruction understanding, visual perception, and completion estimation, but prior agents rely on single-round self-thinking. DiscussNav instead uses multi-expert discussions to gather task-specific information before movement.

VLN movement decisions require understanding instructions, identifying relevant landmarks, observing scenes, and estimating completion at every step.Source: §I · p. 1
Previous VLN and zero-shot language-model methods passively process navigation information and make each movement prediction through one model’s single-round reasoning.Source: §I · p. 1
GPT4 still struggles to handle multiple navigation subtasks simultaneously, while trained models face a substantial simulation-to-real transfer gap.Source: §I · pp. 1–2
DiscussNav draws on expert consultation by actively discussing with domain experts before making movement decisions.Source: Figure 1 · p. 1, §I · p. 2
The framework assigns large models specialized VLN roles and evaluates discussion-based navigation on R2R and real robots.Source: §I · p. 2

A. Vision-and-Language Navigation

VLN develops embodied agents that follow natural-language instructions through real 3D environments. Prior work emphasizes simulator-data training strategies, while dialogue-based approaches involve human participation and recent large-model research explores broader capabilities.

VLN targets embodied agents that follow natural-language instructions while navigating real 3D environments.Source: A. Vision-and-Language Navigation · p. 2
Supervised navigation methods improve simulated performance through data augmentation, memory mechanisms, and pretraining but largely overlook real-world deployment.Source: A. Vision-and-Language Navigation · p. 2
Prior dialogue-enabled navigation methods require human annotation or participation, limiting their fully automated zero-shot character.Source: A. Vision-and-Language Navigation · p. 2
Large language models provide broad abilities such as information summarization, code generation, and task planning for embodied-agent research.Source: B. Large Models · p. 2

III. METHOD

DiscussNav is a GPT4-driven zero-shot navigation agent that fills discussion templates and consults specialized experts before deciding movement. Its experts cover instruction analysis, visual perception, completion estimation, and decision testing.

DiscussNav uses GPT4 and question templates to collect multi-expert information before prompting movement decisions.Source: A. DiscussNav Agent for Visual… · p. 3, Figure 2 · p. 3
The framework establishes domain experts by assigning large models specific roles and task requirements through prompts.Source: B. Domain Experts Driven by Large Models · p. 3

Instruction Analysis Experts

Instruction-analysis experts decompose action sequences and extract navigational landmarks from human instructions.Source: B. Domain Experts Driven by Large Models · p. 3

Vision Perception Experts

Vision-perception experts use scene and object models to identify instruction-relevant visual information, including scene-level and object-level landmarks.Source: B. Domain Experts Driven by Large Models · p. 3

Completion-estimation and decision-testing experts summarize navigation history, assess executed actions, and reconcile diverse movement predictions.Source: B. Domain Experts Driven by Large Models · pp. 3–4, Figure 3 · p. 4

C. Navigation Discussions with Multiple Experts

DiscussNav organizes navigation as sequential consultations: it parses instructions, queries landmark-relevant views, tracks action completion, and tests multiple movement predictions before execution.

Instruction Analysis

DiscussNav first consults action-decomposition and landmark-extraction experts to convert instructions into action sequences and navigational landmarks.Source: C. Navigation Discussions with… · p. 4

Vision Perception

At each step, vision experts inspect 12 surrounding directions using landmark-type-specific questions to update environmental understanding.Source: C. Navigation Discussions with… · p. 4

Qualitative Results

Real-robot discussions support open-vocabulary and fine-grained landmark perception while correcting errors and filtering inconsistent movement decisions.Source: Figure 4 · p. 5, C. Navigation Discussions with… · p. 5

Completion Estimation

The agent summarizes navigation history and consults completion-estimation experts to determine which action remains to be executed.Source: C. Navigation Discussions with… · p. 5

Movement Decision

DiscussNav generates N movement predictions and executes directly when they agree, otherwise sending thought-prediction pairs to decision-testing experts.Source: C. Navigation Discussions with… · p. 5

A. Experiment Setup

The experiments use multiple large-model domain experts with specified roles, alongside DiscussNav and its supporting perception models. Evaluation follows standard R2R navigation metrics.

GPT4 constructs instruction-analysis, completion-estimation, and decision-testing experts, while ChatGPT constructs trajectory-summary and thought-fusion experts.Source: A. Experiment Setup · p. 5
InstructBLIP FlanT5XL supports scene observation, and RAM-14M supports object perception within the expert system.Source: A. Experiment Setup · p. 5
R2R evaluation reports TL, NE, SR, OSR, and SPL, covering path length, endpoint error, success, oracle stopping, and length-penalized success.Source: A. Experiment Setup · p. 5

B. Simulator Quantitative Experiments

Simulator experiments show that DiscussNav substantially outperforms prior zero-shot methods on R2R, while ablations show performance declines when expert discussions are removed.

Comparison with Previous Methods

26.47% Success Rate and 37.93% SPL improvements over NavGPT accompany DiscussNav’s lead over all zero-shot methods and two train-only methods on five R2R metrics.Source: B. Simulator Quantitative Experiments · p. 5, Table I · p. 6

The effect of Discussions with Domain Experts

Ablating discussions with different experts produces varying performance declines, supporting the effectiveness of multi-expert discussions for zero-shot VLN.Source: B. Simulator Quantitative Experiments · p. 6, Table II · p. 6
The ablation validation split samples one trajectory from each of 72 environments spanning 61 training and 11 validation-unseen scenes.Source: B. Simulator Quantitative Experiments · p. 6

C. Real Robot Quantitative Experiments

Real-robot experiments evaluate DiscussNav against pretrained and zero-shot baselines on a Turtlebot navigating semantically rich indoor instructions. DiscussNav performs better on these real-world tasks.

The Turtlebot 4 Lite uses an OAK-D Lite camera and RPLIDAR-A1 lidar to execute 20 instructions containing open-vocabulary landmarks, fine-grained landmarks, and multiple room changes.Source: C. Real Robot Quantitative Experiments · p. 6
DiscussNav shows better real-robot performance than the compared baselines, with advantages in instruction understanding, fine-grained landmark observation, and movement decisions.Source: C. Real Robot Quantitative Experiments · p. 6
DuET fails to transfer simulator-learned navigation abilities to the real world, while NavGPT can navigate but is limited on semantic-rich and long instructions.Source: C. Real Robot Quantitative Experiments · p. 6

D. Qualitative Results

Qualitative analysis shows that expert discussions improve landmark perception, correct decomposed action errors, and support more reliable decisions when instructions or directions are difficult.

Discussions help identify open-vocabulary landmarks, correct errors in decomposed action sequences, and resolve confusion among similar movement directions.Source: D. Qualitative Results · p. 6
For multiple fine-grained landmarks, DiscussNav first obtains relevant visual information and then consults completion-estimation experts before deciding.Source: D. Qualitative Results · p. 6
The real-robot analysis is presented in Table III for indoor-scene experiments.Source: Table III · p. 6

V. CONCLUSION AND FUTURE WORK

DiscussNav is a zero-shot VLN framework that improves navigation by enabling active discussions with multiple domain experts before movement. The authors report benefits over self-thinking-based reasoning and propose expanding experts and embodied-task applications.

DiscussNav enables a navigation agent to gather professional insights through active discussions with multiple domain experts before moving.Source: §V · p. 6
Experiments indicate that discussion-based reasoning enhances large language model performance on embodied tasks compared with self-thinking-based reasoning.Source: §V · p. 6
Future work will expand the domain-expert roster and extend discussion-based navigation to broader embodied tasks through diverse discussion topics.Source: §V · p. 6