PaperlayerView exact revision on arXiv

Source-linked AI summary

Airbert: In-domain Pretraining for Vision-and-Language Navigation

Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, Cordelia Schmid

arXiv:2108.09105v1cs.CVcs.AIcs.CLcs.HCcs.LG

TL;DR

VLN agents face limited generalization to unseen environments because existing pretraining is either small-scale or mismatched with navigation data. The paper introduces BnB and Airbert, combining in-domain image-caption and generated path-instruction pretraining with shuffling loss, and reports state-of-the-art R2R and REVERIE performance plus gains in few-shot VLN.

  • Problem

    VLN generalization to unseen environments remains limited, while prior pretraining uses small VLN environments or generic image-caption data that mismatches navigation paths and instructions.

  • Method

    The paper collects BnB in-domain image-caption pairs, automatically creates VLN-like path-instruction pairs, and pretrains Airbert with masking and shuffling losses.

  • Results

    Airbert achieves state-of-the-art performance on R2R and REVERIE and significantly improves performance in challenging one/few-shot VLN evaluation.

  • Takeaways & Limitations

    In-domain pretraining provides a basis for VLN agents that generalize across standard benchmarks and from a few training houses to unseen environments.

  • Takeaways & Limitations

    The approach is evaluated with a ViLBERT-like architecture chosen for fair comparison, although it is not limited to that architecture.

Abstract

from arXiv · show

Vision-and-language navigation (VLN) aims to enable embodied agents to navigate in realistic environments using natural language instructions. Given the scarcity of domain-specific training data and the high diversity of image and language inputs, the generalization of VLN agents to unseen environments remains challenging. Recent methods explore pretraining to improve generalization, however, the use of generic image-caption datasets or existing small-scale VLN environments is suboptimal and results in limited improvements. In this work, we introduce BnB, a large-scale and diverse in-domain VLN dataset. We first collect image-caption (IC) pairs from hundreds of thousands of listings from online rental marketplaces. Using IC pairs we next propose automatic strategies to generate millions of VLN path-instruction (PI) pairs. We further propose a shuffling loss that improves the learning of temporal order inside PI pairs. We use BnB pretrain our Airbert model that can be adapted to discriminative and generative settings and show that it outperforms state of the art for Room-to-Room (R2R) navigation and Remote Referring Expression (REVERIE) benchmarks. Moreover, our in-domain pretraining significantly increases performance on a challenging few-shot VLN evaluation, where we train the model only on VLN instructions from a few houses.

1. Introduction

VLN agents struggle to generalize to unseen environments because domain-specific data is scarce and existing pretraining is limited by visual or task distribution gaps. Airbert addresses these limitations with the in-domain BnB dataset, VLN-like path-instruction construction, and temporal shuffling pretraining, achieving strong benchmark and few-shot results.

  • VLN performance remains limited on previously unseen environments, despite the task’s relevance to applications such as domestic robotics and personal assistants.
  • Existing pretraining uses either small sets of training houses with limited visual diversity or generic image-caption data that differs from VLN’s dynamic paths and navigable instructions.
  • BnB contains 1.4M indoor images and 0.7M captions collected from online rental marketplaces, providing large-scale in-domain image-caption data for VLN pretraining.
  • The method transforms static image-caption pairs into visual paths and navigation-like instructions, while a shuffling loss learns temporal alignment between paths and instructions.
  • Airbert supports discriminative and generative VLN architectures, achieves state-of-the-art results on R2R and REVERIE, and improves performance in one/few-shot evaluation.

2. Related work

VLN research has progressed from sequence-to-sequence action prediction toward multimodal transformer pretraining, while large-scale vision-language pretraining supplies broad image-text knowledge. The paper argues that VLN requires in-domain, sequentially oriented training strategies rather than generic visual-language pretraining alone.

  • VLN methods include sequence-to-sequence action prediction, attention-based cross-modal alignment, reinforcement learning, speaker-based augmentation, and environment dropout.
  • Transformer-based pretraining learns generic multimodal representations, but prior VLN pretraining remains constrained by limited environments and small visual diversity.
  • Large-scale vision-language pretraining uses web-collected image-caption pairs to learn vision-text correlations, but is not designed for sequential decision making in embodied VLN.
  • This work focuses on in-domain training strategies for VLN rather than improving general-purpose vision-language pretraining architectures.

3. BnB Dataset

BnB uses rental-marketplace listings to collect large-scale indoor image-caption data and automatically convert it into VLN-like path-instruction pairs. The construction strategies address domain gaps in sequential views, panoramic context, linguistic fluency, and actionable navigation language.

  • BnB collection: BnB is collected from U.S. online rental listings to obtain high-quality English captions and visual similarity with Matterport environments.The process downloads listings and metadata, removes outdoor images, and filters invalid captions, emails, URLs, and duplicates.
  • BnB collection: The dataset contains image-caption pairs from rental homes, whose structured listing information may support additional future applications.
  • Path construction: BnB image-caption pairs differ from VLN path-instruction pairs because navigation uses sequential panoramic observations and often multi-sentence instructions.
  • Path construction: Concatenation randomly selects and orders typically 4–7 listing image-caption pairs, inserting conjunctions or punctuation between captions to form paths and instructions.
  • Path construction: Visual-context methods address limited viewpoints by merging images and extracting multiple regions as an approximation of panoramic representations.
  • Instruction construction: Instruction rephrasing inserts BnB noun phrases into navigation templates, while instruction generation produces path instructions from concatenated image sequences using a trained video-captioning-like model.

4. Airbert: A Pretrained VLN Model

Airbert is a ViLBERT-based multimodal transformer pretrained on path-instruction pairs with masking and shuffling objectives, then adapted to discriminative or generative VLN models. Its design targets temporal alignment between visual paths and instructions while supporting downstream navigation tasks.

  • Architecture and downstream adaptation: Airbert is a multimodal transformer pretrained on path-instruction pairs and adaptable to downstream VLN tasks.The model is introduced as a BnB-pretrained architecture that can support both discriminative and generative settings.
  • Architecture and downstream adaptation: The model encodes multiple images and captions from each path-instruction pair using a ViLBERT-like architecture.An [IMG] token separates image-region features from different locations, while visual and textual inputs are jointly processed.
  • Pretraining data: BnB and Conceptual Captions are used in successive pretraining steps to reduce the domain gap for downstream VLN tasks.The approach replaces ordinary VL pretraining with in-domain path-instruction data while retaining image-caption pretraining in the pipeline.
  • Pretraining objectives: Airbert combines masking and pairing objectives with a shuffling loss that trains temporal alignment between paths and instructions.The shuffling objective contrasts aligned path-instruction pairs with pairs formed by shuffling images or captions.
  • Downstream adaptation: Airbert is fine-tuned for both path-selection and recurrent action-generation formulations of goal-oriented and object-oriented navigation.For path selection, the method uses target-task masking and shuffling losses before contrasting positive and negative candidate paths.

5. Experimental Results

Experiments evaluate BnB pretraining, path-instruction construction, shuffling loss, and Airbert against established baselines on R2R, REVERIE, and few-shot VLN settings. Airbert improves performance across standard and restricted-data evaluations, with shuffling and in-domain data contributing measurable gains.

  • Experimental setup: Experiments cover ablations, comparisons with state-of-the-art models, R2R and REVERIE benchmarks, and few-shot VLN training on one or few houses.R2R success rate is emphasized for unseen validation, while REVERIE evaluates navigation and remote object localization.
  • Pretraining with BnB: BnB path-instruction construction improves performance over VLN-BERT, with combined rephrasing, image merging, and captionless insertion yielding 7.2% improvement.Directly using BnB image-caption pairs improves performance by 3.2%, while the combined strategy improves 3.8% over concatenation.
  • Shuffling loss: Shuffling improves temporal-order learning during pretraining and fine-tuning, producing 2.3% and 0.4% improvements respectively, and 3.1% with Speaker data.Combining BnB-PI data, Speaker data, and shuffling reaches 68.67% SR on R2R with a single model.
  • Fine-tuning strategies: Visual-path shuffling produces the highest fine-tuning improvement, +2.7% on val unseen, without external parsers or additional VLP models.Hard negatives and keyword highlighting improve performance by 2.1-2.3% but require extra components.
  • Comparison against state-of-the-art: Airbert alone outperforms VLN-BERT by 9.4% on unseen R2R environments, while an Airbert ensemble reaches 77% test-set success rate and ranks first at submission.Ensembling Airbert gives a 1.4% gain over the VLN-BERT ensemble; Airbert also supports generative R2R models and REVERIE evaluation.
  • Few-shot VLN: Airbert outperforms VLN-BERT by 27.6% with 1 house and 22% with 6 houses on unseen few-shot validation, with six-house fine-tuning nearly matching full-data VLN-BERT.The evaluation restricts training access to examples from one or a few environments.

6. Conclusion

The paper presents BnB and Airbert as an in-domain pretraining approach for VLN, combining automatically created path-instruction pairs with temporal-order reasoning. Airbert achieves state-of-the-art results on R2R and REVERIE and substantially improves one/few-shot VLN performance.

  • Contributions: BnB provides large-scale, in-domain image-text data from houses listed on online rental marketplaces, with domain gaps mitigated through automatically created path-instruction pairs.The dataset and path-instruction construction address the mismatch between marketplace image-caption pairs and VLN inputs.
  • Results: Airbert achieves state-of-the-art performance on R2R through discriminative path selection and on REVERIE through a generative setting.The model is designed as a generic transformer backbone adaptable to both settings.
  • Results: In-domain pretraining produces large performance improvements in a challenging one/few-shot VLN setup.The setup trains on examples from only one or a few environments while evaluating generalization to unseen environments.

A. BnB dataset

The BnB dataset is built from image-caption pairs collected from online rental marketplaces and is extended with automatically generated path-instruction pairs. The dataset design targets in-domain visual and linguistic data for VLN pretraining.

  • Dataset overview: BnB combines image-caption pairs from online rental marketplaces with automatically generated path-instruction pairs for VLN pretraining.The supplementary section covers the source image-caption data, on-the-fly path-instruction generation, and strategies such as rephrasing and captionless insertion.
  • Filtering image-caption pairs: Outdoor images are discarded because they are rarely present in downstream VLN environments and their captions can be irrelevant or noisy for indoor navigation.Examples include captions referring to bedrooms, neighborhood activities, or emotions rather than the image content relevant to navigation.

A.2. Dataset details and Statistics

BnB contains hundreds of thousands of marketplace-derived image-caption resources, from which path-instruction pairs are sampled and augmented at large combinatorial scale. The generated instructions are designed to resemble downstream VLN instruction lengths, while shuffled negatives support temporal-order learning.

  • BnB image-caption pairs: 150K Airbnb listings yield 713K image-caption pairs and 676K images without captions.The collection is described as a sufficiently large and diverse in-domain distribution despite most listings containing fewer than 20 images.
  • Creating BnB path-instruction samples: Path-instruction samples are created on-the-fly by sampling a listing, selecting K image-caption pairs with K ∼U[4, 7], and constructing a visual trajectory with an instruction.The procedure is intended to mimic an agent’s visual trajectory and corresponding instruction through an environment.
  • Creating BnB path-instruction samples: Each positive path-instruction pair is accompanied by 9 additional negatives formed by shuffling image or caption sequences so their order no longer aligns.These negatives are used to train the shuffling loss.
  • Statistics for BnB PI pairs: 200 billion path-instruction pairs can theoretically be created by simple concatenation, increasing to over 300 quadrillion with visual-context augmentations and fluent instructions.The figures reflect the large number of possible combinations rather than a fully materialized dataset.
  • Instruction rephrasing: 11,626 fill-in-the-blank templates are created from the R2R training set for instruction rephrasing.Most templates contain 2-7 blanks filled with noun phrases from BnB captions.
  • Statistics for BnB PI pairs: Automatically created BnB path-instruction instructions have lengths similar to R2R and REVERIE instructions, whereas BnB image-caption captions are much shorter.The comparison uses a representative sample of 50K path-instruction pairs.

A.3. Examples of BnB PI Pairs

BnB path-instruction pairs are generated through concatenation, rephrasing, instruction generation, image merging, and captionless image insertion. These strategies trade off fluency, vocabulary coverage, visual context, and path diversity.

  • Generation strategies: BnB path-instruction pairs use naive concatenation, instruction rephrasing, instruction generation, image merging, and captionless image insertion.The examples illustrate multiple strategies for constructing paired visual paths and instructions.
  • Instruction construction: Instruction rephrasing inserts BnB image-description noun phrases into real instruction templates, producing more natural transitions than simple concatenation.Simple concatenation lacks action verbs between sentences, creating a domain shift from real instructions.
  • Instruction generation: Instruction generation produces fluent sentences but cannot fully exploit BnB’s diverse captions because its vocabulary is limited by the downstream VLN dataset.The paper gives an example where generated instructions omit noun phrases related to images in the path and leaves better caption generation for future work.
  • Path augmentations: Image merging expands relevant visual context from single images to semi-panoramic views.The examples include merged bedroom and kitchen views.
  • Path augmentations: Captionless image insertion increases path diversity by mimicking viewpoints that are not mentioned in the instruction.These inserted images are indicated by dotted borders in the examples.

B. Implementation details

Airbert reuses the VLN-BERT architecture and supports both discriminative path-selection and generative sequential-action settings. It is pretrained on BnB and then fine-tuned for R2R and REVERIE tasks.

  • Architecture: Airbert uses the same architecture as VLN-BERT, with six layers in each stream.
  • Architecture: Each image-region feature combines visual attention features, region-location encoding, and an image-order embedding.
  • Training: Airbert is pretrained on BnB for 15 epochs using batches of 64 and a learning rate of 4×10^-5, with 95% of listings sampled for training.
  • Discriminative setting: In the discriminative setting, Airbert directly predicts path-instruction compatibility scores for R2R navigation.
  • Fine-tuning: Fine-tuning follows task-specific setups, including R2R path selection, generated path-instruction augmentation, imitation learning, and A2C reinforcement learning.
  • Generative setting: In the generative setting, Airbert initializes a recurrent VLN-BERT model that predicts navigable actions step by step for R2R and REVERIE.

C.1. Results on R2R with Generative Models

The generative evaluation compares Airbert with alternative Recurrent VLN-BERT backbones on R2R. Airbert substantially improves over OSCAR and VLN-BERT on unseen environments while matching PREVALENT comparably.

  • Results: Airbert achieves substantial performance improvement over OSCAR and VLN-BERT backbones on unseen R2R environments.
  • Results: Airbert achieves comparable performance with PREVALENT initialization on unseen R2R environments.

C.2. Qualitative results

Qualitative examples show Airbert generalizing to new houses, rare or unseen objects, and similar navigation instructions. However, both models still fail on some spatial relations and directional instructions.

  • New houses: Airbert recognizes varied room types and performs better than VLN-BERT in substantially different environments such as churches and castles.
  • New objects: Airbert follows instructions involving rare or unseen household objects that VLN-BERT fails to handle.
  • Similar environments and instructions: Airbert reaches an acceptable range of 3m from the goal in an example where VLN-BERT stops at the wrong stairs.
  • Failure cases: Both VLN-BERT and Airbert still struggle with relational instructions such as “between” and directional instructions such as “on the left.”
  • Few-shot evaluation: Airbert achieves 49.5% and 58% unseen-validation performance with 1 and 6 training environments, respectively, compared with VLN-BERT’s 27% and 37%.
  • Similar environments and instructions: The shuffling loss helps Airbert use intermediate landmarks, improving navigation when environments and instructions resemble the training set.
Loading 2108.09105v1…