Source-linked AI summary
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, Jianfeng Gao
TL;DR
VLN agents must handle variable, ambiguous multimodal inputs despite limited training data on new tasks. This paper introduces a self-supervised pre-training and fine-tuning paradigm using image-text-action triplets to learn transferable representations. PREVALENT improves performance across R2R, CVDN, and HANNA, achieving new state-of-the-art results.
Problem
VLN involves highly variable, ambiguous multimodal inputs, while training data on a new task is often limited.
Method
PREVALENT pre-trains an encoder on image-text-action triplets with self-supervised objectives, producing generic representations that can be used in existing VLN frameworks.
Results
PREVALENT significantly improves existing methods across R2R, CVDN, and HANNA, achieving new state-of-the-art performance on all three benchmarks.
Takeaways & Limitations
The learned representation transfers across VLN tasks and supports better generalization to previously unseen environments and new tasks.
Takeaways & Limitations
Masked image modeling did not improve results and was excluded from the experiments.
Abstract
from arXiv · showhide
Learning to navigate in a visual environment following natural-language instructions is a challenging task, because the multimodal inputs to the agent are highly variable, and the training data on a new task is often limited. In this paper, we present the first pre-training and fine-tuning paradigm for vision-and-language navigation (VLN) tasks. By training on a large amount of image-text-action triplets in a self-supervised learning manner, the pre-trained model provides generic representations of visual environments and language instructions. It can be easily used as a drop-in for existing VLN frameworks, leading to the proposed agent called Prevalent. It learns more effectively in new tasks and generalizes better in a previously unseen environment. The performance is validated on three VLN tasks. On the Room-to-Room benchmark, our model improves the state-of-the-art from 47% to 51% on success rate weighted by path length. Further, the learned representation is transferable to other VLN tasks. On two recent tasks, vision-and-dialog navigation and "Help, Anna!" the proposed Prevalent leads to significant improvement over existing methods, achieving a new state of the art.
1. Introduction
VLN requires agents to reason over variable, ambiguous, and under-specified multimodal inputs, while prior methods learn each instruction largely in isolation. PREVALENT pre-trains joint vision-language representations with image-text-action triplets, then transfers them across three VLN tasks.
- VLN is challenging because visual and linguistic inputs are highly variable, ambiguous, and often under-specified.
- Most prior agents learn to understand each instruction from scratch without collectively leveraging prior vision-grounded domain knowledge.
- PREVALENT pre-trains an encoder to align language instructions and visual states into joint representations.At each time step, image-text-action triplets are used with masked-word and next-action prediction objectives.
- The pre-trained model is fine-tuned on R2R, CVDN, and HANNA, covering in-domain and out-of-domain VLN scenarios.R2R uses instructions given at the beginning, CVDN uses dialog history, and HANNA provides intermediate instructions during interactive navigation.
- PREVALENT achieves new state-of-the-art performance on all three tasks and adapts faster while generalizing better to unseen environments and new tasks.
2. Related Work
Prior work develops vision-language navigation with sequence models, attention, search, co-grounding, reinforcement learning, and environment exploration. Vision-language pre-training has learned cross-modal representations, but this paper extends pre-training to representations grounded in navigation actions and VLN-compatible visual inputs.
- Vision-language pre-training commonly uses BERT-like objectives for cross-modal tasks such as visual question answering, image-text retrieval, and image captioning.
- This paper presents pre-trained models that ground vision-language understanding with actions in a reinforcement learning setting.
- Existing vision-language pre-training features are not readily applicable to VLN because region extraction is computationally infeasible for panoramic views.State-of-the-art VLN systems use panoramic observations such as 36 images per view for R2R.
- Existing VLN methods use panoramic action spaces, neural decoding with search, co-grounding attention, progress monitoring, reinforcement learning, and environment exploration.
3. Background
VLN can be formulated as a partially observable sequential decision problem in which an agent maps language and visual states to actions. Standard agents use attention-based sequence-to-sequence policies, motivating pre-training of a generic vision-language encoder.
- VLN is modeled as a POMDP with visual states, discrete actions, unknown transition dynamics, and a reward function.
- At each step, the agent observes an RGB image, selects an action, receives a new image observation, and stops at the target or a trajectory limit.
- Training data pairs natural-language instructions with corresponding expert trajectories, and policies are learned by maximum likelihood estimation.
- Typical policies use attention-based sequence-to-sequence architectures trained with teacher forcing, decomposed into an encoder and an action decoder.
- The vision-language encoder maps visual states and instructions to joint representations, while the decoder uses them to produce actions.
4. Pre-training Models
The model builds joint visual-language representations with Transformer encoders and pre-trains them using image-attended masked language modeling and action prediction on image-text-action triplets.
- Input Embeddings: The input embedding layers convert panoramic views and language instructions into image-level visual embeddings and word-level sentence embeddings.Visual inputs use panoramic image features, while text uses token and position embeddings with layer normalization.
- Encoder Architecture: The backbone combines two single-modal Transformer encoders with a cross-modal encoder to produce downstream features.The cross-modal encoder uses cross-attention followed by self-attention and a feed-forward layer.
- Encoder Architecture: Cross-attention lets positions in one modality attend to all positions in the other modality through mask-controlled attention.Each Transformer block concatenates multiple attention heads, whose queries, keys, and values are linearly projected.
- Pre-training Objectives: Image-attended masked language modeling predicts randomly masked instruction words using surrounding words and image observations.Words are masked with probability 15%, grounding learned word embeddings in visual-state context.
- Pre-training Objectives: Action prediction uses the fused [CLS] representation to predict the correct action from a sampled state-action pair.The prediction is conditioned on the current visual image and instruction without referring to trajectory history.
- Pre-training Objectives: Masked image modeling was evaluated as an alternative loss design but did not show better results and was excluded from the experiments.The discussion notes that other pre-training loss designs remain possible.
5. Adapting to new tasks
The pre-trained representation is evaluated by adapting it to one in-domain VLN task and two out-of-domain tasks based on the Matterport3D simulator.
- Task Scope: The three tasks cover in-domain generalization to unseen environments and out-of-domain generalization to indirect instructions and interactive reinforcement learning.R2R is treated as in-domain, while CVDN and HANNA represent new tasks with different challenges.
- R2R: R2R tests navigation to an explicitly instructed target and uses the pre-trained contextualized word embeddings in an LSTM encoder-decoder.The fine-tuned framework is compared with random initialization and BERT-based word embeddings.
- CVDN: CVDN evaluates navigation from multi-turn question-answer dialog history toward a goal location, with longer and often indirect input.Its partner has privileged access to shortest-path next steps, and dialog histories are limited to 300 words.
- HANNA: HANNA evaluates interactive navigation in which an assistant provides location-specific language instructions and images to help find an indoor object.The pre-trained model encodes the assistant’s language instructions during the help mode.
6. Experimental Results
Experiments evaluate PREVALENT across R2R, CVDN, and HANNA using standard navigation metrics and task-specific settings. PREVALENT outperforms strong baselines, transfers across tasks, and shows evidence of improved generalization and useful pre-training objectives.
- Room-to-Room: PREVALENT outperforms existing models on SR and SPL by a large margin on both validation seen and unseen R2R environments.It leads on nearly all reported metrics.
- Room-to-Room: PREVALENT improves testing SPL and has a smaller seen–unseen gap than PRESS, supporting better generalization to unseen environments.The paper attributes this to image-attended language understanding.
- Cooperative Vision-and-Dialogue Navigation: PREVALENT significantly outperforms the Seq2Seq baseline across CVDN validation and testing unseen environments in all three supervision settings.The settings are Oracle, Navigator, and Mixed supervision paths.
- HANNA: PREVALENT outperforms ANNA on HANNA test-unseen SR, SPL, and NE while requesting slightly more help.Its SPL seen–unseen gap is 30.66 versus 38.42 for ANNA, which the paper interprets as reduced over-fitting.
- Ablation Studies: Including the action-prediction loss LPA yields consistent improvement on nearly all CVDN metrics and settings.The full LPA + LMLM objective also achieves the best performance among HANNA pre-training variants.
- Ablation Studies: Image-attended MLM variants outperform both BERT pre-training and direct BERT fine-tuning on CVDN, indicating greater effectiveness for navigation tasks.The comparison tests whether image-attended learning is necessary.
7. Conclusions
PREVALENT introduces a pre-training and fine-tuning paradigm for vision-and-language navigation. Across three benchmarks, it improves existing methods and achieves new state-of-the-art performance.
- PREVALENT is a new pre-training and fine-tuning paradigm for vision-and-language navigation problems.
- The paradigm uses limited training data more effectively to improve generalization to previously unseen environments and new tasks.
- The pre-trained encoder can be plugged into existing models to boost their performance.
- Results on R2R, CVDN, and HANNA show significant improvement over existing methods and new state-of-the-art performance.
Supplementary Material: Towards Learning a Generic Agent for
The supplementary material is part of the paper titled “Vision-and-Language Navigation via Pre-training” and records the contributors’ implementation, idea development, experimentation, and manuscript roles.
- The supplementary material accompanies the paper titled “Vision-and-Language Navigation via Pre-training.”
- Weituo implemented the algorithm, made the model work, and ran all experiments.
- Chunyuan initiated the pre-training idea, led the work, and completed the manuscript writing.
- Xiujun provided the codebase and helped with implementation, while Lawrence and Jianfeng edited the manuscript.
A. Pre-training Dataset Preparation
The pre-training dataset combines human-collected R2R triplets with Speaker-generated instructions on shortest-path trajectories. This synthesis expands the dataset to 6,582K image-text-action triplets.
- R2R contains 104K image-text-action triplets and is the largest VLN training dataset described here.
- Speaker synthesizes instructions for shortest-path trajectories because harvesting human annotations is expensive.
- The constructed dataset contains 6,582K image-text-action triplets, including 6,482K from synthesized instructions.
B. Experiments
The experiments describe CVDN input types and evaluate pre-training objectives on HANNA. The synthesized dataset accounts for 98.4% of the pre-training data, and LPA + LMLM performs best among the tested variants.
- 98.4% of the pre-training dataset is synthesized data.
- CVDN uses three named types of text inputs, documented with target-object, visual-feature, question, and answer notation.
- LPA + LMLM yields the best performance among the tested HANNA pre-training-objective variants.
C. Comparison with Related Work
The comparison distinguishes this work from PRESS and general vision-language pre-training by emphasizing visual-action information and transfer across navigation tasks. Table 8 frames the comparison around visual features, pre-training objectives, masking and pair prediction, and downstream navigation.
- Pre-training with visual and action information improves navigation performance and generalizes across different new navigation tasks.
- Existing vision-language pre-training models are not readily applicable because this work addresses different research scopes and problem setups.
- The related-work comparison covers view-level ResNet and object-level Fast R-CNN visual features.
- It contrasts attentive masked language modeling and action prediction with masking on vision-language inputs and same-pair prediction.
- The downstream setting is navigation in sequential decision-making environments.