Source-linked AI summary
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, Dhruv Batra
TL;DR
VLN agents need to ground instruction references in visual observations, but embodied path-instruction data is relatively scarce and costly to collect. The paper introduces VLN-BERT, a compatibility model pretrained through language, web image-text, and embodied path-instruction stages, and reports improved VLN path-selection performance. Its curriculum ablations indicate that the stages contribute cumulatively and synergistically.
Problem
The paper asks whether visual grounding learned from abundant “disembodied” web data can transfer to improve embodied VLN despite limited task-specific data.
Method
VLN-BERT scores alignment between instructions and trajectory observations, using sequential language-only, web image-text, and VLN path-instruction pretraining.
Results
4.6 absolute percentage points: VLN-BERT increases success rate over strong single-model baselines on the VLN path selection task.
Takeaways & Limitations
Web image-text pretraining can provide transferable visual grounding for VLN, with the full curriculum producing cumulative benefits greater than individual stage effects.
Takeaways & Limitations
The evaluation considers previously explored environments with stored navigation graphs and panoramas, rather than only newly unseen environments.
Abstract
from arXiv · showhide
Following a navigation instruction such as 'Walk down the stairs and stop at the brown sofa' requires embodied AI agents to ground scene elements referenced via language (e.g. 'stairs') to visual content in the environment (pixels corresponding to 'stairs'). We ask the following question -- can we leverage abundant 'disembodied' web-scraped vision-and-language corpora (e.g. Conceptual Captions) to learn visual groundings (what do 'stairs' look like?) that improve performance on a relatively data-starved embodied perception task (Vision-and-Language Navigation)? Specifically, we develop VLN-BERT, a visiolinguistic transformer-based model for scoring the compatibility between an instruction ('...stop at the brown sofa') and a sequence of panoramic RGB images captured by the agent. We demonstrate that pretraining VLN-BERT on image-text pairs from the web before fine-tuning on embodied path-instruction data significantly improves performance on VLN -- outperforming the prior state-of-the-art in the fully-observed setting by 4 absolute percentage points on success rate. Ablations of our pretraining curriculum show each stage to be impactful -- with their combination resulting in further positive synergistic effects.
1 Introduction
VLN requires grounding language-referenced objects in visual observations, but task-specific path-instruction data is expensive and limited. The paper introduces VLN-BERT and a web-to-embodied training curriculum, reporting substantial gains in path selection and evidence of object grounding.
- VLN agents must match visual evidence along a path to instruction references such as “banister rail” and “butterfly sculpture.”
- Task-specific VLN data is expensive to collect and relatively small, with around 14k training path-instruction pairs.
- VLN-BERT scores alignment between an instruction and an agent’s observations along a trajectory using a visiolinguistic transformer-based compatibility model.
- The curriculum sequentially trains language representations, visual grounding from Conceptual Captions image-text pairs, and action grounding from VLN path-instruction pairs.
- 4.6 absolute percentage points: VLN-BERT improves success rate over strong single-model baselines on path selection.
- 9.2 absolute percentage points: web image-text pretraining improves path-selection success rate, while curriculum stages show synergistic cumulative benefits.
- Gradient-based visualizations show that removing a referenced fridge phrase reduces the importance of fridge-containing image regions.
2 Related Work
Prior VLN work ranks candidate paths with follower or speaker models and addresses limited data through augmentation and auxiliary tasks. This paper extends transferable vision-and-language pretraining to panoramic image sequences for VLN.
- Follower and speaker models have been used to rank paths in fully observable VLN settings.
- Prior methods compensate for small VLN datasets using speaker-generated instructions, environmental dropout, language-only pretraining, and auxiliary tasks.
- Vision-and-language models pretrained on web image-text pairs have achieved state-of-the-art results across tasks including VQA, referring expressions, and image retrieval.
- The paper extends this model architecture to process panoramic image sequences for vision-and-language navigation.
3 Preliminaries: Self-Supervised Learning from the Web
Self-supervised language and vision-language pretraining learn transferable representations from web corpora. The paper builds on BERT and ViLBERT objectives to support VLN-BERT’s web-based visual grounding.
- Language Modeling with BERT: BERT represents tokenized word sequences with transformer outputs and trains using masked language modeling and next sentence prediction.
- Visiolinguistic Representations Learning with ViLBERT: ViLBERT learns joint image-text representations with separate visual and language streams connected by co-attention transformer layers.
- VLN-BERT processes image regions from panorama sequences together with instruction words under a curriculum transferring web-learned visual grounding to VLN.
- Visiolinguistic Representations Learning with ViLBERT: ViLBERT uses masked multimodal modeling to predict masked text tokens and visual-region object classes from surrounding context.
- Visiolinguistic Representations Learning with ViLBERT: A multimodal alignment objective trains ViLBERT to determine whether an image-text pair matches.
4 Approach
The approach formulates fully observed VLN as selecting the trajectory most compatible with an instruction, using a visiolinguistic model that jointly represents panoramic observations and instruction words.
- Path Selection: In fully observed VLN, navigation selects the valid trajectory best aligned with an instruction from paths sharing the same start position.The environment is represented as a navigation graph whose nodes contain 360-degree panoramas, while success requires ending within 3m of the goal.
- Path Selection: The model scores each candidate trajectory against the instruction and selects the trajectory with the highest compatibility score.This path-selection formulation focuses on learning the compatibility function while using a small beam-search-generated candidate set.
- Model Architecture: VLN-BERT models trajectory-instruction compatibility with a visiolinguistic transformer adapted from ViLBERT for transfer learning.The model jointly reasons over a sequence of panoramic observations and a sequence of instruction words.
- Input Representation: A trajectory is represented as panoramic images paired with poses, while each panorama is decomposed into image regions and combined with instruction tokens.The resulting path-instruction input interleaves image-region representations with the language sequence and special modality tokens.
K , <CLS> w1, . . . , wL <SEP> (4)
VLN-BERT encodes visual regions with trajectory-aware spatial information, trains path selection as a four-way classification problem, and uses staged pretraining to transfer language, visual, and action grounding.
- Spatial Encoding: Each image-region representation combines visual features with the source panorama index and panoramic spatial information.Spatial information includes normalized region geometry, elevation, and heading relative to the agent’s current and next viewing directions.
- Image Regions: The region extractor generates 600 × 600 perspective projections from 36 directions, then removes redundant regions while retaining at most 100 per panorama.Regions beyond 20 degrees from a perspective image’s center are discarded, and similar regions are filtered using attention detection scores.
- Path Selection Training: VLN-BERT is trained as a four-way multiple-choice task with one successful trajectory and three negative trajectories for each instruction.The model computes compatibility scores for each instruction-trajectory pair and trains the normalized scores with cross-entropy loss.
- Negative Examples: Beam search supplies varied path-instruction pairs, including negatives designed to differ semantically from the instruction.Training samples one positive and three negative pairs uniformly at random.
- Transfer Learning Curriculum: The curriculum sequentially trains language understanding, web-based visual grounding, and action grounding before path-selection fine-tuning.The stages use BERT on Wikipedia and BooksCorpus, ViLBERT on Conceptual Captions, and VLN path-instruction pairs, respectively.
5 Experiments
The experiments evaluate pretraining, baseline comparisons, ensemble performance, and whether VLN-BERT’s alignment scores focus on relevant image regions. Across these evaluations, the full curriculum improves path selection and leaderboard success.
- The experiments address whether web image-text pretraining improves VLN, how VLN-BERT compares with strong baselines, and whether it considers relevant image regions.
- Beam search generates up to 30 candidate paths per R2R instruction, and over 99% of candidate sets contain a path reaching the goal.
- Pretraining Curriculum: 14.1 absolute percentage points is the success-rate improvement from combining visual-grounding and action-grounding pretraining, exceeding the individual gains.The individual gains are 4.5 and 4.9 absolute percentage points, while the combined gain is 9.2 points over the next-best setting.
- Baseline Comparison: 59.3% SR is achieved by fully pretrained VLN-BERT in the single-model path-selection setting, 4.6 absolute percentage points above either speaker or follower.
- Leaderboard Results: 73% success rate is achieved by the speaker, follower, and VLN-BERT ensemble on Test Unseen, 4 absolute percentage points above previously published work.
- Visual Grounding Analysis: Gradient-based analysis assigns region importance by summing each panorama region’s feature-gradient components and tests changes after removing instruction phrases.
6 Conclusion
The paper demonstrates internet-to-embodied transfer of visual concept grounding for VLN path-instruction alignment. Its ablations find that every stage of the transfer curriculum contributes significantly.
- Internet-to-embodied transfer uses large-scale web image-text data to improve a discriminative path-instruction alignment model for VLN.
- Each stage of the transfer curriculum contributes significantly in the paper’s path re-ranking setting.
A.1 Implementation Details
The implementation uses BERTBASE-based vision and language streams connected by cross-modal attention. VLN-BERT is initialized from BERT and ViLBERT weights for language-only and visual-grounding pretraining.
- The experiments use a 12-layer BERTBASE architecture for both vision and language streams, with 6 cross-modal attention layers connecting them.
- Language-only pretraining initializes VLN-BERT with BERT weights pretrained on English Wikipedia and BooksCorpus.
- Visual-grounding pretraining initializes VLN-BERT with ViLBERT weights.
A.2 Qualitative Examples of Success and Failure
Qualitative examples show VLN-BERT selecting successful paths while assigning importance to instruction-relevant visual regions, alongside failures caused by incorrect stopping or mismatched landmarks. The analysis links selected paths and influential regions to grounding quality.
- Successful Examples: VLN-BERT correctly grounds “antelope head,” a phrase absent from the VLN training dataset, in one successful example.The term “antelope” appears three times in the training dataset.
- Successful Examples: Successful examples compare VLN-BERT’s selected paths with ground-truth paths and navigation errors, while highlighting its top five influential image regions.
- Failure Examples: The unsuccessful examples include passing the goal, failing to reach the goal bedroom, and broad mismatches between selected paths and instructions.
- Failure Examples: In one failure, influential “curtains” from another location may have contributed to selecting a path that missed the goal bedroom.
A.3 Qualitative Analysis of the Pretraining Curriculum
The full pretraining curriculum enables VLN-BERT to ground rare instruction phrases and select successful paths, unlike a model without visual grounding pretraining.
- Qualitative Comparison: Stage 2 visual grounding pretraining helps VLN-BERT ground rare phrases such as ‘mini fridge’ and ‘massage table’.These phrases occur only 1 and 29 times, respectively, in the VLN training dataset.
- Qualitative Comparison: Without Stage 2, VLN-BERT fails to ground these phrases and selects unsuccessful paths in both examples.
- Qualitative Comparison: With the full curriculum, VLN-BERT correctly grounds the key phrases and selects successful paths for both examples.