PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout

Hao Tan, Licheng Yu, Mohit Bansal

arXiv:1904.04195v1cs.CLcs.CVcs.LG

TL;DR

Instruction-guided navigation systems often generalize poorly from seen to unseen environments. The paper combines mixed imitation and reinforcement learning with environmental-dropout back-translation, and reports rank-1 performance on the VLN R2R leaderboard under all evaluation setups.

  • Problem

    Existing navigation models perform substantially worse in unseen environments than in seen ones, limiting generalization in the R2R task.

  • Method

    The agent is trained with mixed imitation and reinforcement learning, then fine-tuned on back-translated environment, route, and instruction triplets created using environmental dropout.

  • Results

    The model achieves rank-1 on the VLN R2R challenge leaderboard under all experimental setups.

  • Takeaways & Limitations

    Environmental dropout enables fine-tuning with generated triplets from mimicked unseen environments to improve navigation generalization.

  • Takeaways & Limitations

    Traditional back-translation uses routes in seen training environments, whose limited variety forms a bottleneck for performance.

Abstract

from arXiv · show

A grand goal in AI is to build a robot that can accurately navigate based on natural language instructions, which requires the agent to perceive the scene, understand and ground language, and act in the real-world environment. One key challenge here is to learn to navigate in new environments that are unseen during training. Most of the existing approaches perform dramatically worse in unseen environments as compared to seen ones. In this paper, we present a generalizable navigational agent. Our agent is trained in two stages. The first stage is training via mixed imitation and reinforcement learning, combining the benefits from both off-policy and on-policy optimization. The second stage is fine-tuning via newly-introduced 'unseen' triplets (environment, path, instruction). To generate these unseen triplets, we propose a simple but effective 'environmental dropout' method to mimic unseen environments, which overcomes the problem of limited seen environment variability. Next, we apply semi-supervised learning (via back-translation) on these dropped-out environments to generate new paths and instructions. Empirically, we show that our agent is substantially better at generalizability when fine-tuned with these triplets, outperforming the state-of-art approaches by a large margin on the private unseen test set of the Room-to-Room task, and achieving the top rank on the leaderboard.

1 Introduction

The paper targets instruction-guided navigation in unseen environments, where existing R2R agents perform substantially worse than in seen rooms. It combines mixed imitation and reinforcement learning with environmental-dropout back-translation to improve generalization.

  • Task: R2R requires an agent to follow natural-language instructions through real Matterport3D indoor environments while selecting actions from panoramic views.The agent must understand instruction elements and localize corresponding scene views at each timestep.
  • Challenge: Performance commonly drops substantially in unseen rooms, motivating methods focused on generalization beyond the training environments.The paper frames this capability as important for practical robots operating in the wild.
  • Method: The first training stage mixes imitation learning and reinforcement learning to combine off-policy and on-policy optimization.The authors report that this mixture significantly outperforms using either IL or RL alone.
  • Method: The second stage fine-tunes the model with back-translated triplets generated from environmental-dropout environments, routes, and speaker-produced instructions.Environmental dropout addresses the limited variability of seen training environments by mimicking new environments.
  • Results: The fine-tuned model substantially outperforms prior state-of-the-art systems and achieves rank-1 on the private VLN R2R leaderboard under all reported evaluation setups.The setups are single run, beam search, and pre-exploration.

2 Related Work

Prior work established instruction-based navigation and semi-supervised back-translation, but R2R systems still show a performance gap between seen and unseen environments. This paper focuses on improving generalization in unseen environments.

  • Embodied Vision-and-Language: Embodied vision-and-language research includes agents that perceive scenes, navigate with maps and planning, and act from visual input.These works span simulation and real-world-oriented robotic settings.
  • Instruction-based Navigation: Instruction-based navigation asks an agent to move from a start viewpoint to an end viewpoint according to language instructions in an environment.R2R distinguishes this line of work through photo-realistic Matterport3D imagery.
  • Instruction-based Navigation: Existing R2R systems show a performance gap between seen and unseen environments, motivating the paper’s focus on generalizability.Earlier approaches include speaker-follower, reinforcement-learning, imitation-learning, cross-modal, and look-ahead models.
  • Back-translation: Back-translation augments paired instruction-route data by generating additional source sentences with a backward translator for fine-tuning.The approach originated in neural machine translation and was later applied to instruction-based navigation.

3 Method

The method builds an instruction-following navigation agent with a base encoder-decoder, mixed imitation and reinforcement learning, and environmental-dropout back translation for unseen-environment generalization.

  • Base Agent Model: The agent observes panoramic views and navigable viewpoints, then selects actions according to the instruction and navigation history.Each view combines RGB image features with orientation, while navigable viewpoints are represented by their orientations.
  • Base Agent Model: A bidirectional LSTM encoder represents the instruction, and an attentive LSTM decoder combines visual, action, and instruction features to score navigable viewpoints.The decoder computes an attentive visual feature, incorporates the previous action, and aligns instruction-aware hidden states with candidate viewpoint features.
  • Base Agent Model: The decoder uses the instruction-aware hidden vector as its recurrent input, making attended instruction information accessible during navigation.This differs from the referenced prior model, which used the ordinary hidden state as decoder input.
  • IL+RL Mixture: IL+RL mixes shared-weight imitation and reinforcement-learning agents navigating independent routes, combining teacher supervision with sampled-action rewards.The RL component uses Advantage Actor-Critic, terminal rewards of +3 or −3, and distance-based reward shaping; IL regularizes RL training.
  • Back Translation: Back translation generates instructions for unlabeled routes, producing new environment–route–instruction triplets used to fine-tune the agent with IL+RL.The navigation agent serves as the forward model and an enhanced stacked bidirectional LSTM speaker serves as the backward model.
  • Environmental Dropout: Environmental dropout removes a consistent object class across views to mimic novel environments, avoiding the inconsistent removals caused by traditional feature dropout.Independent feature masks can leave an object visible in another view or create conflicting observations across viewpoints.

4 Experimental Setup

The experiments use the Matterport3D Room-to-Room benchmark, with Success Rate as the primary metric and early stopping based on unseen validation performance.

  • Dataset and Simulator: The dataset contains training, seen-validation, unseen-validation, and unseen-test splits across Matterport3D environments.The supplied passage gives partial counts for these splits, including 61 training environments, 11 unseen-validation environments, and 18 unseen-test environments.
  • Evaluation Metrics: Success Rate is the primary evaluation metric, counting navigation as successful when the error is below 3 meters.Navigation Length, Navigation Error, and SPL are additional metrics.
  • Evaluation Metrics: Navigation Error measures the distance between the target viewpoint and the agent’s stopping position.
  • Leaderboard Evaluation: The leaderboard compares Navigation Length, Success Rate, and SPL across single-run, beam-search, and pre-exploration setups.The table caption identifies the primary metric for each setup and distinguishes the best and second-best results.
  • Implementation Details: Environmental dropout masks are applied at each training iteration, with early stopping selected using unseen-validation Success Rate.Sharing the mask across environments within a batch is reported to stabilize training.

5 Results

On the private unseen test set, the model outperforms prior entries across single-run, beam-search, and pre-exploration evaluations, with sizable margins on reported metrics.

  • Overall Leaderboard Results: The model outperforms all other leaderboard entries across single-run, beam-search, and pre-exploration setups on 18 new environments.
  • Single Run: 3.5% higher Success Rate and 9% higher SPL than the second-best model are achieved in the primary single-run setup.Single run forbids multiple trials and pre-navigation exploration or mapping.
  • Beam Search: 5.9% higher Success Rate than the second-best model is achieved with beam search.The reported SPL range for beam-search models is 0.01–0.02 because of their long Navigation Length.
  • Pre-Exploration: 3.4% higher Success Rate and 2.0% higher SPL than Wang et al. (2019) are achieved in the pre-exploration setup.This setup updates the agent model using information gathered by pre-exploring the test environment.

6 Ablation Studies

Ablations attribute gains to both mixed imitation–reinforcement learning and environmental-dropout back translation, while data diversity across environments improves supervised learning more than data volume alone.

  • Full Model: The full model improves validation-unseen Success Rate by 8.6% over the baseline.The ablation summary attributes substantial contributions to both IL+RL and back translation with environmental dropout.
  • Supervised Learning: IL+RL outperforms imitation-only and reinforcement-learning-only baselines by 2.9% and 22.5%, respectively.The authors associate the mixture with overcoming misleading teacher actions and stabilizing reinforcement-learning training.
  • Semi-Supervised Learning: Back translation with environmental dropout improves the best supervised model by 5.7%, three times the gain from back translation without new environments.
  • Semi-Supervised Learning: Feature dropout performs 3.8% below environmental dropout in the reported comparison.
  • Semi-Supervised Learning: Tying environmental-dropout masks between speaker and follower improves performance relative to using independent masks.The authors report that independent masks cause a substantial performance drop.
  • Environment Diversity: Adding environments produces a higher Success Rate growth rate than adding the same amount of data sampled across all 60 training environments.The predicted Success Rate is also higher when training uses more environments.

7 Analysis

The analysis identifies environment variety as a bottleneck for navigation performance and back translation, while environmental dropout improves results on unseen validation environments.

  • 7.1 More Environments vs. More Data: Using more environments produces faster success-rate growth than using more data with the same amount of training data.The comparison uses data selected either by increasing the number of environments or from all 60 training environments.
  • 7.2 Back Translation on Unseen Environments: 61.9% success rate from back translation on unseen validation environments exceeds 46.5% from back translation on training environments.The comparison indicates that new environments are important for effective back translation.
  • 7.2 Back Translation on Unseen Environments: Environmental dropout on testing environments further raises back-translation performance to 64.5%.This result suggests that the available environment variety for back translation remains insufficient without dropout.
  • 7.3 Environmental Dropout with Semantic Views: With semantic view features, environmental dropout reaches a 52.0% success rate on the unseen validation set.The semantic views encode object types by color, and environmental dropout removes objects from the view while retaining key structural labels.

8 Conclusion

The paper presents a navigation agent designed to generalize better to unseen environments. It combines imitation and reinforcement learning with back-translation fine-tuning using environmental dropout, achieving rank-1 across the reported VLN challenge setups.

  • 8 Conclusion: The agent combines supervised imitation learning and reinforcement learning before semi-supervised fine-tuning with speaker-generated instructions.Fine-tuning uses environmental dropout to generate new unseen environments.
  • 8 Conclusion: Environmental dropout addresses limited environment variety in back translation by generating new unseen environments.The method is evaluated on the Room-to-Room dataset.
  • 8 Conclusion: The model achieves rank-1 on the VLN challenge leaderboard under all experimental setups.The reported setups include single run, beam search, and pre-exploration.

A.1 Implementation Details

The implementation uses pretrained visual features, recurrent language and action representations, mixed imitation-reinforcement training, and environmental-dropout fine-tuning with supervised and semi-supervised losses.

  • Model and Training: The agent uses 2048-dimensional ResNet-152 image features pretrained on ImageNet.The model is first trained with mixed imitation and reinforcement learning, then fine-tuned through back translation with environmental dropout.
  • Optimization: Back-translation optimization uses supervised loss for half of each batch and semi-supervised loss for the other half.The global gradient norm is clipped by 40, and hyperparameters are tuned on unseen-validation success rate.
  • Model and Training: Word embeddings have size 256, action embeddings size 64, and LSTM units size 512, with 256 units in the bidirectional LSTM.Reinforcement learning uses discounted factor γ = 0.9 and distance-change reward shaping.
  • Environmental Dropout: Semantic-view dropout preserves wall, floor, and ceiling labels because they represent the basic environment structure.Dropping these key labels empirically provides no improvement.
Loading 1904.04195v1…