PaperlayerView exact revision on arXiv

Source-linked AI summary

A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning

Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, Zarana Parekh

arXiv:2210.03112v3cs.LGcs.CLcs.CVcs.RO

TL;DR

VLN datasets have limited human instruction data, and web pretraining has yielded limited improvements for this spatially grounded task. The paper builds a large synthetic instruction dataset and trains a transformer with imitation learning. On RxR, its agent improves NDTW over the state of the art in seen and unseen environments.

  • Problem

    VLN has scarce human instruction data, and large text and image-text pretraining has yielded limited improvements for interactive 3D navigation.

    Source: §1 · p. 1

  • Method

    The paper generates synthetic instructions for trajectories across 500+ environments, synthesizes novel-view observations, and trains a transformer agent with imitation learning on 4.2M instructions.

    Source: §1 · p. 2

  • Results

    On RxR, the agent reaches 79.1 NDTW in seen environments and 66.8 NDTW in unseen test environments, improving on the state of the art by 8% and 2%, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results point to large-scale imitation learning with near-human-quality synthetic instructions as a path to improving instruction-following agents.

    Source: §6 · p. 8

    The approach achieves strong but not state-of-the-art results on R2R, which the authors attribute to domain differences because Marky was trained on RxR.

    Source: §A.1 · p. 13

Abstract

from arXiv · show

Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments, as a step towards robots that can follow human instructions. However, given the scarcity of human instruction data and limited diversity in the training environments, these agents still struggle with complex language grounding and spatial language understanding. Pretraining on large text and image-text datasets from the web has been extensively explored but the improvements are limited. We investigate large-scale augmentation with synthetic instructions. We take 500+ indoor environments captured in densely-sampled 360 degree panoramas, construct navigation trajectories through these panoramas, and generate a visually-grounded instruction for each trajectory using Marky, a high-quality multilingual navigation instruction generator. We also synthesize image observations from novel viewpoints using an image-to-image GAN. The resulting dataset of 4.2M instruction-trajectory pairs is two orders of magnitude larger than existing human-annotated datasets, and contains a wider variety of environments and viewpoints. To efficiently leverage data at this scale, we train a simple transformer agent with imitation learning. On the challenging RxR dataset, our approach outperforms all existing RL agents, improving the state-of-the-art NDTW from 71.1 to 79.1 in seen environments, and from 64.6 to 66.8 in unseen test environments. Our work points to a new path to improving instruction-following agents, emphasizing large-scale imitation learning and the development of synthetic instruction generation capabilities.

1. Introduction

Human-annotated VLN data is scarce, while web pretraining offers limited gains for this spatially grounded task. The paper scales synthetic in-domain instructions and trains an imitation-learning agent, reporting improved RxR results.

1. Introduction

  • Human VLN annotations are small and expensive, while large web image-text and text-only pretraining has produced more limited improvements for interactive 3D navigation.The authors hypothesize that web datasets lack the spatially grounded, action-oriented language needed for VLN.Source: §1 · p. 1
  • The Marky-Gibson dataset adds 491 Gibson environments and 3.2M synthetic instruction-trajectory pairs, expanding the existing 1M Marky pairs across 61 Matterport3D environments.Novel-view image observations are also synthesized with an image-to-image GAN; the resulting dataset is two orders of magnitude larger than existing human-annotated datasets.Source: §1 · p. 2
  • Imitation learning scales to 4.2M instructions and over 700M steps of experience using behavioral cloning and DAGGER.The authors contrast this with low-data VLN work where pure imitation learning underperformed combined IL and online RL.Source: §1 · p. 2
  • 79.1 NDTW in seen RxR environments and 66.8 NDTW in unseen test environments improve on the state of the art by 8% and 2%, respectively.The best agent uses only imitation learning; synthetic-instruction self-training in new environments raises unseen-test NDTW by a further 2% to 68.6.Source: §1 · p. 2

2. Related Work

Related work moves from procedurally generated navigation tasks to photorealistic VLN and surveys pretraining and synthetic-data augmentation. The paper identifies synthetic-instruction quality and scale as remaining constraints.

Related work

  • Early instruction-following agents used simple settings and often procedurally generated instructions [4] [42] [43], while later VLN work used photorealistic 3D environments and natural language [3] [9].These developments encouraged datasets for dialog [17] [62], object search [46], and multilingual instructions [30].Source: §2 · p. 2

Pretraining and Transfer Learning

  • VLN pretraining has used pretrained language models, image-text pairs, and instruction-trajectory objectives, while recent visual encoders include CLIP [47].Despite this work, a significant gap to human performance remains.Source: §2 · pp. 2–3

Data Augmentation

  • Prior synthetic-instruction methods explored environment edits, rental-listing paths, and synthesized viewpoints [34] [60] [16] [27] [28], but quality and scale remained limiting factors.Human wayfinding evaluations found earlier instructions poorly grounded and mostly unfollowable [72]; Marky [63] addresses quality, while this work scales to 500+ environments and 3.2M instructions.Source: §2 · p. 3
  • The paper scales synthetic instructions and reports state-of-the-art VLN results with a purely imitation-learning agent, unlike most recent work focused on RL.DUET [11] is noted as an exception, combining imitation learning with a global action space based on a topological map.Source: §2 · p. 3

3. Approach

MARVAL predicts navigation actions by combining instruction, observation, action-candidate, and trajectory-history information in a multimodal transformer. It is pretrained on instruction-trajectory data and then adapted with imitation-learning strategies.

Problem setup

  • The navigation setup supplies panoramic observations and adjacent-pano action candidates from a predefined graph, with STOP available as an action.Panoramas contain 36 views (12 headings by 3 elevations); selecting an action moves the agent to an adjacent panorama.Source: §3 · p. 3

Agent architecture

  • At each step, a multimodal transformer combines the instruction, state and action history, current observation, and action candidates to predict the next action.Each previous panorama's 36 image-feature vectors are compressed into one vector; inputs are concatenated without attention masking, with learned biases between input types.Source: §3 · p. 3, Figure 2 · p. 4

Pretraining

  • Pretraining uses instruction-trajectory pairs, including synthetic instructions and novel-view observations, before finetuning on human-annotated data.The transformer is initialized from scratch rather than image-caption datasets or text corpora.Source: §3 · p. 4
  • Four pretraining tasks provide supervision: masked language modeling, progress prediction, constrained action prediction, and unconstrained action prediction.The two action-prediction logits are equally weighted after early experiments found this improved accuracy by 1–2%.Source: §3 · pp. 4–5

Finetuning and pre-exploration

  • Finetuning uses behavioral cloning on ground-truth trajectories and DAGGER, which adds current-policy states paired with expert actions to help the agent recover from errors.The authors report that most DAGGER gains occur in a single iteration; in pre-exploration, paths from unseen environments are annotated with Marky instructions and added to training data.Source: §3 · p. 5

4. Datasets and Augmentation

The section expands VLN training data with Marky-generated instructions in Gibson environments and describes how trajectories and image observations are augmented. It also reports the best pretraining configuration using Marky data across Matterport3D and Gibson, without Speaker instructions.

4. Datasets and Augmentation

  • 3.2M multilingual Marky-Gibson instructions cover 491 Gibson environments, compared with existing datasets situated in 61 Matterport3D training environments.The dataset is model-generated and supplements the previously available Matterport3D training data.Source: Table 1 · p. 6
  • Gibson navigation graphs combine RedNet edge predictions with mesh obstacle information, then use a minimum spanning tree to ensure full connectivity.The edge rule uses geodesic and straight-line distances, predicted edge probability, and vertical separation.Source: §4 · p. 6
  • Sampling from 491 Gibson train and validation environments produced 1.06M paths averaging 7.1 steps and 19.3m, each annotated in English, Hindi, and Telugu.Paths longer than 40m or 16 steps were discarded, with no more than 3K paths sampled per environment.Source: §4 · p. 6
  • The generated Gibson graphs achieve an F1 score of 0.70, with 0.695 precision and 0.713 recall against Matterport3D validation graphs.The edge-prediction thresholds were selected by grid search on Matterport3D validation environments.Source: §4 · p. 6
  • The best pretraining setting combines R2R and RxR with Marky instructions from Matterport3D and Gibson, SE3DS viewpoints, and no Speaker instructions.The selected configuration is row 9 of Table 2.Source: Table 2 · p. 7, §4 · p. 7

5. Experiments

MARVAL, trained with imitation learning, outperforms prior RL agents on RxR, with larger gains on seen than unseen environments. Experiments find benefits from Marky data, Gibson environments, viewpoint synthesis, and larger models, while R2R performance is strong but not state-of-the-art.

5. Experiments

  • Marky augmentation improves success rate by 27% on RxR and 17% on R2R over the baseline, while adding Speaker instructions to Marky data reduces performance.Speaker-only augmentation improves success rate by 2% on RxR and 8% on R2R; combining it with Marky performs worse than Marky alone.Source: §5 · p. 7
  • 79.1 NDTW on RxR Val-Seen improves the state of the art by 8%, while 66.8 NDTW on Test improves it by 2%.Self-training with Marky instructions in Test environments raises NDTW by a further 2% to 68.6; the table compares MARVAL trained with BC or DAGGER against prior RL agents.Source: §5 · p. 7, Table 3 · p. 8
  • Adding Gibson Marky data raises success rate by 11% on RxR and 12% on R2R, while SE3DS adds 6% and 12% respectively without Gibson.With Gibson included, SE3DS gains fall to 0% on RxR and 2% on R2R; changing from mT5-base to mT5-large adds 2% and 1%.Source: §5 · p. 7
  • On R2R, MARVAL achieves strong performance but not state-of-the-art; the authors attribute this to domain differences from RxR, on which Marky was trained.R2R instructions average 26 words versus 87 for RxR, and vocabulary and language-structure differences may also contribute to distribution shift.Source: Table 4 · p. 8, §5 · p. 8

6. Conclusion

The paper reports state-of-the-art RxR results with a purely imitation-learning agent and identifies large-scale imitation learning and synthetic instruction generation as a path for improving instruction-following agents.

6. Conclusion

  • A purely imitation-learning agent achieves state-of-the-art results on RxR, with a larger improvement in seen than unseen environments.The conclusion reports gains of 8% in seen environments and 2% in unseen test environments, and suggests that scaling to more indoor environments might further improve generalization.Source: §6 · p. 8

A.1. Limitations and Future Work

The approach is strong but not state of the art on R2R, which the authors attribute to domain differences; human followers also outperform MARVAL overall despite making fewer first errors. The authors identify error recovery as a key direction for future improvement.

A.1. Limitations and Future Work

  • R2R results are strong but not state of the art, which the authors attribute to domain differences and Marky’s training on RxR.Retraining Marky on R2R could address this, but R2R lacks the annotator pose traces used to train Marky on RxR.Source: §A.1 · p. 13
  • MARVAL makes fewer first errors than prior approaches, especially early in trajectories, and even fewer than human followers.The analysis compares MARVAL with EnvEdit, HAMT, and human demonstrations on RxR Val-Unseen.Source: §A.1 · p. 13
  • Human followers still outperform MARVAL overall, suggesting that future agent improvement should focus on recovering from errors.Humans have 94.5% success versus MARVAL’s 64.8%, while their NDTW is 81.8 versus 70.8.Source: §A.1 · p. 13

A.2. Implementation Details

Pretraining uses held-out unseen-environment action prediction to select a converged model snapshot. During finetuning on the smaller human-annotated datasets, only WordPiece embeddings are updated, and the best snapshot is selected by unseen validation success rate.

A.2. Implementation Details

  • Pretraining uses batches of 128, AdaFactor, 0.1 dropout, and a learning rate exponentially decayed from 0.1.Training continues until convergence, with one-step action prediction accuracy monitored on held-out instructions from RxR and R2R Val-Unseen.Source: §A.2 · p. 13
  • The best pretraining snapshot is selected by one-step action prediction accuracy, while the best finetuning snapshot is selected by Val-Unseen success rate.Finetuning runs for at most 150K iterations and monitors standard path-fidelity metrics.Source: §A.2 · p. 13
  • Finetuning updates only WordPiece embeddings while freezing the other transformer weights to improve stability and reduce overfitting on small datasets.The datasets are RxR and R2R; finetuning uses dropout of 0.2 and a constant learning rate of 0.001.Source: §A.2 · p. 13

A.3. Additional Pretraining Results

The additional pretraining results report Val-Seen performance and show trends consistent with Val-Unseen results, with a larger relative gain from increasing model size on seen environments.

A.3. Additional Pretraining Results

  • The additional pretraining results report Val-Seen performance, complementing the main paper’s Val-Unseen results.The reported splits cover RxR and R2R.Source: §A.3 · p. 13
  • Val-Seen results show the same trends as results in new, unseen environments.The passage reports this pattern across the seen and unseen settings.Source: §A.3 · p. 13
  • The relative improvement from using a larger model is greater on Val-Seen than on Val-Unseen.The comparison is between row 8 and row 7 in Table 5.Source: §A.3 · p. 13

A.4. Performance by Language

The section reports improvements across RxR languages relative to EnvEdit and illustrates both Marky-generated instructions and MARVAL’s instruction following in unseen environments. Its examples include successful trajectories and a failure in which MARVAL does not recover after an error.

A.4. Performance by Language

  • RxR results improve across languages over EnvEdit, with similar results for each language.Table 6 breaks down results for the best model and the previous best-performing model.Source: §A.4 · p. 13, Table 6 · p. 14

A.5. Marky Synthetic Instruction Examples

  • Figures 5 and 6 show Marky-generated synthetic instructions alongside their associated trajectories.The examples pair instructions with sampled trajectories.Source: §A.5 · p. 13

A.6. MARVAL Instruction-Following Examples

  • Figures 7 and 8 illustrate successful following of RxR Val-Unseen instructions, while Figure 9 shows a failure to recover after an error.In the failure example, MARVAL follows about 75% of the instruction before missing the single chair and failing to find the table with a book and pen.Source: §A.6 · p. 13, Figure 9 · p. 19
  • MARVAL makes fewer first errors than prior agents and human followers, although humans still significantly outperform it overall.The analysis suggests focusing future agent improvement on error recovery.Source: Figure 4 · p. 14

Related papers

Loading 2210.03112v3…