PaperlayerView exact revision on arXiv

Source-linked AI summary

Speaker-Follower Models for Vision-and-Language Navigation

Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, Trevor Darrell

arXiv:1806.02724v2cs.CVcs.CL

TL;DR

Vision-and-language navigation requires reasoning over ambiguous instructions, visual context, and limited route annotations. The paper embeds a speaker model into a follower, using it for synthetic-data augmentation and pragmatic route selection alongside a panoramic action space; the full system substantially improves navigation performance, reaching 53.5% success on the unseen test environment.

  • Problem

    Navigation instructions underspecify low-level behavior, while agents must resolve ambiguity and plan in changing visual environments from limited annotated trajectories.

  • Method

    A speaker–follower system uses a speaker to generate synthetic route instructions during training and score candidate routes during inference, with a panoramic high-level action space.

  • Results

    53.5% final success rate on the unseen test environment, an absolute improvement of 30% over existing approaches.

  • Takeaways & Limitations

    Speaker-driven augmentation, pragmatic reasoning, and panoramic actions each improve the instruction follower, with the full model more than doubling baseline success in unseen environments.

  • Takeaways & Limitations

    The search assumes instructions typically describe acyclic routes, so it avoids routes that pass through the same state twice.

Abstract

from arXiv · show

Navigation guided by natural language instructions presents a challenging reasoning problem for instruction followers. Natural language instructions typically identify only a few high-level decisions and landmarks rather than complete low-level motor behaviors; much of the missing information must be inferred based on perceptual context. In machine learning settings, this is doubly challenging: it is difficult to collect enough annotated data to enable learning of this reasoning process from scratch, and also difficult to implement the reasoning process using generic sequence models. Here we describe an approach to vision-and-language navigation that addresses both these issues with an embedded speaker model. We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction. Both steps are supported by a panoramic action space that reflects the granularity of human-generated instructions. Experiments show that all three components of this approach---speaker-driven data augmentation, pragmatic reasoning and panoramic action space---dramatically improve the performance of a baseline instruction follower, more than doubling the success rate over the best existing approach on a standard benchmark.

1 Introduction

Vision-and-language navigation requires agents to connect natural-language instructions with visual environments while resolving ambiguity and planning routes. The paper addresses limited annotated data and weak direct instruction-to-action modeling with a speaker–follower approach and panoramic actions.

  • Motivation: Navigation agents must relate natural-language instructions to visual environments while acting from a start location toward a goal.Instructions identify landmarks and decisions, leaving some behavior to be inferred from context.
  • Motivation: Existing approaches generally map instructions directly to actions using a single model trained on limited annotated trajectories.This setup leaves ambiguous references, alternative routes, and incompletely specified destinations as nontrivial reasoning challenges.
  • Approach: The proposed system treats navigation as trajectory search with an instruction follower mapping descriptions to routes and a speaker mapping routes to instructions.Both modules use sequence-to-sequence architectures, enabling the speaker to model how routes are described.
  • Approach: The panoramic action space represents high-level movement between adjacent locations rather than low-level visuomotor controls.This representation is intended to match the granularity of human-generated navigation instructions.
  • Approach: Speaker-driven augmentation expands training data with synthetic route–instruction pairs, while pragmatic inference scores candidate routes by how well the speaker would explain the instruction.The speaker is used during both training and test-time route selection.
  • Results: 53.5% final success rate on the unseen test environment represents a 30% absolute improvement over existing approaches.The paper evaluates the speaker–follower model on the Room-to-Room dataset.

2 Related Work

The paper connects vision-and-language navigation to instruction following, pragmatic language understanding, semi-supervision, and visual grounding. Its distinction is integrating a speaker model into a visual navigation pipeline for both training and inference.

  • Natural language instruction following: Prior situated instruction-following systems either use structured language representations or map language and world observations directly to actions.The embodied navigation setting adds rich visual interaction and path-finding requirements.
  • Pragmatic language understanding: The approach draws on Rational Speech Acts models, which represent speakers and listeners as probabilistically reasoning about each other in context.Earlier applications largely addressed simpler communicative decision-making settings.
  • Pragmatic language understanding: Unlike prior work described here, the paper integrates pragmatic reasoning with a complex visual pipeline and uses it during both inference and training.The training-time use improves the quality of a base listener model.
  • Semi- and self-supervision: The semi-supervised strategy relates to bootstrapping and co-training, while differing from SEQ4 by avoiding a differentiable surrogate for the decoding objective.SEQ4 reconstructs routes in synthetic, nonvisual domains with language as a latent variable.
  • Semi- and self-supervision: Semi-supervised annotation expansion also connects to computer-vision data distillation and self-play or critic-based supervision in reinforcement learning.These neighboring approaches use model predictions or other agents to provide additional training signals.
  • Grounding language in vision: Visual grounding usually maps expressions to regions or masks in static images, whereas this task requires active interaction to find a path through an environment.The navigation setting couples language grounding with sequential action.

3 Instruction Following with Speaker-Follower Models

The speaker–follower system combines route following, route description generation, synthetic-data training, pragmatic candidate selection, and panoramic high-level actions. These components address data scarcity, ambiguous instructions, and the difficulty of low-level visual control.

  • 3 Instruction Following with Speaker-Follower Models: The system uses an instruction follower and a learned speaker that generates route descriptions from visual observations and actions.The follower models PF(r | d), while the speaker models PS(d | r).
  • 3.1 Speaker-Driven Data Augmentation: The speaker is trained on ground-truth routes and instructions, then supplies synthetic instructions for sampled routes as additional follower supervision.The augmented data are combined with original data before fine-tuning on the original training set.
  • 3.2 Speaker-Driven Route Selection: Pragmatic inference selects routes that provide a high-probability explanation for the observed instruction under the speaker model.This reframes route choice as counterfactual reasoning about which route likely caused the instruction.
  • 3.2 Speaker-Driven Route Selection: Because exact maximization is infeasible, the method generates follower-based candidate routes and rescored them using combined follower and speaker probabilities.The interpolation weight λ is tuned on validation data.
  • 3.2 Speaker-Driven Route Selection: State-factored search generates up to K = 40 candidates and can improve follower performance even before pragmatic rescoring.The search stores and expands high-scoring routes by discrete location, heading, and completion status.
  • 3.2 Speaker-Driven Route Selection: Best performance uses λ close to 1, and speaker-only scoring remains substantially better than follower-only scoring.This indicates route descriptions provide most of the selection signal in the reported setting.
  • 3.2 Speaker-Driven Route Selection: The search avoids revisiting states because route scores are non-increasing and instructions typically describe acyclic routes.This is an author-supported assumption underlying the search behavior.
  • 3.3 Panoramic Action Space: The panoramic action space replaces low-level visuomotor control with 360-degree perception and direct movement toward navigable high-level directions.The implementation discretizes each panorama into 36 view angles and uses attention before choosing a direction.

4 Experiments

Experiments on R2R evaluate the speaker–follower model through dataset-based comparisons, component ablations, and qualitative analysis. Speaker-driven augmentation, pragmatic inference, and panoramic actions each improve navigation performance, culminating in 53.5% success on the unseen test environment.

  • Experimental Setup: The R2R dataset contains 7,189 paths with three human instructions per path, while success rate measures endpoints within 3m of the goal.Validation distinguishes environments seen during training from unseen environments.
  • Component Contributions: The experiments compare a baseline follower against systems adding speaker-driven augmentation, pragmatic inference, and panoramic action space.Table 1 adds components independently and removes them from the full system to assess their contributions.
  • Component Contributions: 40.3% to 46.8% validation-seen success and 19.9% to 24.6% validation-unseen success result from speaker-generated route-instruction augmentation.The augmented data improves generalization to new scenes, although noisy synthetic data is followed by fine-tuning on original human data.
  • Component Contributions: 49.0% to 57.2% validation-seen success and 30.5% to 39.3% validation-unseen success result when the speaker rescores follower-generated route candidates.The comparison isolates pragmatic inference from follower-only candidate scoring.
  • Component Contributions: 70.1% validation-seen and 54.6% validation-unseen success are achieved after replacing visuomotor control with the panoramic action representation.The final model exceeds twice the baseline follower’s success rate in unseen environments.

5 Conclusions

Vision-and-language navigation requires reasoning over changing visual contexts and long-term decisions, and the paper improves instruction following by combining structured action spaces with an explicit speaker model.

  • The baseline sequence-to-sequence model performs comparatively poorly because navigation couples language interpretation with local perception and long-term decision-making.
  • The proposed approach integrates a follower with an explicit speaker that predicts how navigation routes are described, while structuring the action space.
  • Pragmatic inference helps resolve ambiguous instructions, such as selecting the door leading to the room containing the referenced bed rather than stopping at a similar-looking table.

A Overview

The supplementary document describes candidate route generation, analyzes pragmatic-inference parameters and implementation details, and adds qualitative examples and challenge-submission information.

  • The document presents an algorithm for generating candidate routes.
  • It analyzes parameters for speaker-driven route selection, or pragmatic inference, along with other implementation details.
  • It provides additional qualitative examples of pragmatic inference and describes the submission to the Vision and Language Navigation Challenge.

B State-Factored Search for Candidate Route Generation

State-factored search generates candidate routes for pragmatic inference by retaining the best-scoring route for each state and expanding routes until enough completed candidates are found.

  • The algorithm returns completed candidate routes for use in pragmatic inference.
  • State-factored search takes a start state and candidate count K, then iteratively selects the highest-scoring unexpanded route.
  • The search maps each state to the best route ending there, whether expanded or unexpanded, and updates that mapping when a higher-scoring route is found.
  • Completed routes and partial routes are stored separately, and the procedure stops after K completed routes are found or no candidates remain.
  • Successor routes are generated by taking all possible actions, extending the current route by one state and assigning the resulting total model score.

C Analysis of Inference Parameters and Implementation Details

The analysis examines speaker weighting, candidate-route count, search strategy, and embeddings. Speaker scoring is crucial, candidate increases help unseen-environment performance before saturation, and state-factored search outperforms beam search on val-unseen.

  • Inference parameters: λ = 0.95 gives the lowest navigation error on val unseen, while larger speaker weights produce more actions on average.
  • Inference parameters: Speaker scoring is crucial: the full model with speaker weight λ = 0.95 substantially outperforms follower-only route scoring with λ = 0.
  • Inference parameters: Increasing the number K of candidate routes improves val unseen success through 40 candidates, after which the gain tends to saturate.
  • Inference parameters: With only five route candidates, the model achieves 50.3% success rate on val unseen, versus 35.5% for greedy decoding and 43.5% for search without pragmatic inference.
  • Implementation details: State-factored search performs better than standard beam search on val unseen, while GloVe initialization slightly increases success rate.

D Qualitative Examples

Qualitative examples show that pragmatic inference helps the follower resolve ambiguous navigation commands in both seen and unseen validation environments. Attention visualizations additionally expose which instruction words and panoramic views guide the agent at each step.

  • Qualitative trajectory comparisons: Pragmatic inference improves step-wise navigation trajectories on both seen and unseen validation splits.The examples compare base-follower trajectories with trajectories using the speaker model for pragmatic inference.
  • Qualitative trajectory comparisons: Across the qualitative examples, the speaker model is used to address ambiguity in commands and improve route selection.The examples include ambiguous movement and landmark-based navigation cases on validation environments.
  • Attention visualizations: The visualizations show textual attention over instructions alongside the three most attended panoramic view angles and the agent’s chosen direction.Attention weights are displayed for the ranked view angles, while a red arrow marks the next action.

E Submission to Vision and Language Navigation Challenge

The full Speaker-Follower method achieved the top challenge performance, while pragmatic inference resolved ambiguous instructions in qualitative examples. Challenge-compliant sequential route logging preserved success-rate evaluation but increased oracle success through recorded backtracking.

  • Challenge submission: 53.49% success rate kept Speaker-Follower atop the Vision and Language Navigation Challenge leaderboard using single speaker and listener models.The submission used predictions from the full method without additional ensembling.
  • Challenge submission: Challenge-compliant inference logged all search states in traversal order, backtracking to common ancestors when switching between routes.This procedure maintained physical plausibility and prohibited teleportation in the final trajectory.
  • Challenge submission: The sequential procedure matched pragmatic inference in success rate because both returned routes with the same end states.Oracle success nevertheless rose from 63.9% to 96.0% because the trajectory recorded locations visited across routes.
  • Challenge submission: 35.08% success rate without pragmatic inference on the challenge test set was close to the corresponding validation-unseen performance.The same evaluation reported 44.45% oracle success rate, 6.62m navigation error, and 14.82m average trajectory length.
  • Limitations: The method primarily optimizes success rate and navigation error, leaving trajectory-length reduction through inference and modeling for future work.Suggested directions include ordering routes by location and ranking partial incomplete routes.
  • Qualitative examples: Pragmatic inference disambiguated “walk a bit,” enabling the follower to reach the correct location and stop by the zebra striped rug.Without pragmatic inference, the follower stopped at the wrong location and failed to enter the door.
  • Qualitative examples: Pragmatic inference also selected the correct bedroom and stopping location by evaluating how well candidate routes matched the instruction.In the unseen examples, it resolved the two-bedroom ambiguity and turned at “the end of the rug” before stopping near the mirror.
Loading 1806.02724v2…