PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Vision-and-Language Navigation from YouTube Videos

Kunyang Lin, Peihao Chen, Diwei Huang, Thomas H. Li, Mingkui Tan, Chuang Gan

arXiv:2307.11984v1cs.CVcs.CL

TL;DR

VLN agents must connect natural-language instructions with visual observations, but existing pre-training approaches do not directly teach environment-layout reasoning. Lily builds path-instruction data from YouTube house tours, combines it with trajectory judgment and other pretext tasks, and achieves state-of-the-art performance on R2R and REVERIE.

  • Problem

    VLN agents need to model relations between language and visual observations, while existing pretext tasks do not directly address environment-layout reasoning.

    Source: §2.1 · p. 2

  • Method

    Lily pre-trains on YouTube-VLN path-instruction pairs, using action-aware instructions and trajectory judgment alongside masked language, masked vision, and path-ranking tasks.

    Source: §3.2 · p. 4, §4.2 · p. 5

  • Results

    Lily achieves state-of-the-art performance on the R2R and REVERIE benchmarks.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The approach improves layout reasoning by training the agent to judge whether trajectories are reasonable.

    Source: §6 · p. 8

Abstract

from arXiv · show

Vision-and-language navigation (VLN) requires an embodied agent to navigate in realistic 3D environments using natural language instructions. Existing VLN methods suffer from training on small-scale environments or unreasonable path-instruction datasets, limiting the generalization to unseen environments. There are massive house tour videos on YouTube, providing abundant real navigation experiences and layout information. However, these videos have not been explored for VLN before. In this paper, we propose to learn an agent from these videos by creating a large-scale dataset which comprises reasonable path-instruction pairs from house tour videos and pre-training the agent on it. To achieve this, we have to tackle the challenges of automatically constructing path-instruction pairs and exploiting real layout knowledge from raw and unlabeled videos. To address these, we first leverage an entropy-based method to construct the nodes of a path trajectory. Then, we propose an action-aware generator for generating instructions from unlabeled trajectories. Last, we devise a trajectory judgment pretext task to encourage the agent to mine the layout knowledge. Experimental results show that our method achieves state-of-the-art performance on two popular benchmarks (R2R and REVERIE). Code is available at https://github.com/JeremyLinky/YouTube-VLN

2. Related Work

VLN research has moved from sequence-to-sequence models to transformer-based vision-and-language pre-training, while dataset scale and trajectory quality remain central concerns. Prior work draws on simulator data and web images, whereas this paper’s related-work discussion identifies in-domain data and layout reasoning as key directions.

2.1. Vision-and-Language Navigation

  • Transformer-based vision-and-language pre-training followed early sequence-to-sequence VLN models to better model cross-modal relations.The related-work discussion lists multiple BERT-based pre-training approaches.Source: §2.1 · p. 2

2.2. Datasets for VLN

  • Simulator datasets are limited in scene count, while paths assembled from web images may be confusing or ambiguous.Prior work also uses self-exploration and augmentation to expand simulator data.Source: §2.2 · p. 3
  • The paper positions large-scale in-domain pre-training data with reasonable layouts as an alternative to simulated or spliced-image trajectories.The related-work discussion describes YouTube-VLN as providing diverse visual environments and reasonable layouts.Source: §2.2 · p. 3

3. Building VLN Dataset from YouTube Videos

YouTube-VLN constructs path-instruction pairs from house-tour videos using room-based trajectories and action-aware instruction generation. The dataset pipeline selects informative visual frames and attaches pseudo-labeled actions to instructions.

3. Building VLN Dataset from YouTube Videos

  • YouTube-VLN builds reasonable path-instruction pairs from house-tour videos using entropy-based trajectory generation and an action-aware instruction generator.The dataset is presented as featuring diverse environments, real layouts, and native actions.Source: §3 · p. 3
  • The dataset draws on 4,078 videos totaling 433 hours and retains 587k indoor frames after filtering redundant or noisy frames.Videos came from various uploaders; filtering removed frames featuring people or outdoor scenes.Source: §3.1 · p. 3
  • Adjacent frames with the same room type form trajectory nodes, with transition nodes sometimes inserted between room nodes.Room types are recognized with CLIP, and transition nodes use frames captured while moving between rooms.Source: §3.1 · p. 4
  • The lowest-classification-entropy image represents a node, and adjacent frames are merged to approximate its panoramic visual context.The method selects an informative frame from potentially large groups of images.Source: §3.1 · p. 4

3.2. Action-Aware Instruction Generation

  • Instruction templates use CLIP-derived object and room descriptions for noun blanks, while predicted actions fill verb blanks.The templates are generated from R2R instructions following Airbert.Source: §3.2 · p. 4
  • An action inverse model pseudo-labels native actions between nodes, which are converted into verbs such as “go forward,” “turn left,” and “turn right.”The model is reported to have 96% prediction accuracy; the action label fills the verb blank closest to the node description.Source: §3.2 · p. 4

4. Learning VLN from YouTube Videos

Lily uses a vision-and-language backbone to model trajectories and instructions, then pre-trains it on YouTube-VLN with trajectory judgment and established pretext tasks. The resulting backbone is adapted to goal-oriented and object-oriented navigation benchmarks.

4. Learning VLN from YouTube Videos

  • Lily combines a vision-and-language backbone that models trajectories and instructions with a decision module for action prediction or path-instruction matching.The paper uses ViLBERT for comparison with Airbert.Source: §4 · p. 4

4.1. Model Architecture

  • The architecture encodes sequential visual-region features and text tokens separately before their interaction in a cross-modal transformer.Special tokens mark image, classification, and sequence boundaries.Source: §4.1 · p. 4

4.2. Learning Layout from Trajectory Judgment

  • Trajectory judgment trains the agent to distinguish reasonable trajectories from shuffled, unreasonable ones by predicting trajectory reasonableness.The task requires reasoning over visual information and room types to assess consistency with real environment layouts.Source: §4.2 · p. 5
  • Negative trajectory examples are generated by shuffling transition nodes, shuffling all nodes, or inserting nodes from other videos.The paper describes these as rich and hard negatives intended to make the task more difficult.Source: §4.2 · p. 5
  • Pre-training combines trajectory judgment with masked language modeling, masked vision modeling, and path ranking on YouTube-VLN.Path ranking selects the best-matching path-instruction pair from several pairs.Source: §4.2 · p. 5

4.3. Adapting Pre-trained Backbone for VLN

  • The pre-trained model is adapted to R2R goal-oriented navigation and REVERIE object-oriented navigation in the Matterport3D simulator.R2R is evaluated in discriminative and generative settings; REVERIE is evaluated in the generative setting.Source: §4.3 · p. 5

5. Experiments

Lily improves on prior methods across R2R’s discriminative and generative settings and REVERIE, and also outperforms existing pre-training methods when fine-tuned on one environment. Ablations report gains from YouTube data, reasonable trajectories, pseudo-labeled actions, the trajectory judgment task, and entropy-based frame selection.

5.1. Experimental Setup

  • Evaluation uses R2R for goal-oriented navigation and REVERIE for navigation plus object grounding, with success, path-efficiency, and related metrics.R2R includes 21,567 path-instruction pairs from 90 Matterport3D scenes.Source: §5.1 · p. 6

5.2. Ablation Studies on Pre-Training

  • YouTube data exceeds Airbnb data by 0.89% success rate on val unseen; reasonable trajectories exceed shuffled trajectories by 1.27% on the same split.The comparison keeps pre-training instruction counts equal; YouTube-VLN has 587k images from 433 hours of house-tour videos, versus Airbnb’s 713k image-caption pairs.Source: §5.2 · p. 6
  • Pseudo-labeled actions improve success rate by 1.10% on val unseen and 0.30% on seen; trajectory judgment adds 2.05% on seen and 0.60% on unseen.The trajectory judgment variant also improves relative-orientation accuracy by 30%.Source: §5.2 · pp. 6–7
  • Entropy-based frame selection reaches 66.70% success rate on val unseen, versus 64.41% for random selection and 65.30% for temporal difference.The paper describes entropy selection as finding a reliable node frame while keeping adjacent nodes in different room types.Source: §5.2 · p. 7, Table 2 · p. 7

5.3. Comparison with State-of-the-Arts

  • On R2R discriminative evaluation, Lily raises success rate by 10.74% over VLN-BERT on val unseen and reaches 79% success rate on the test unseen leaderboard.Against Airbert, success rate rises from 73.85% to 79.31% on val seen and from 68.67% to 70.00% on val unseen.Source: §5.3 · p. 7, Table 3 · p. 7, Table 4 · p. 7
  • On R2R generative val unseen and test unseen, Lily improves success rate over DUET by 2% and 3%, respectively.Lily also increases SPL by 2% on val unseen.Source: §5.3 · p. 7, Table 6 · p. 8
  • On REVERIE val unseen, Lily exceeds DUET by 1.13% in success rate and raises SPL from 46.98% to 48.11%.On test unseen, it improves success rate by 1.81% and SPL by 1.28%; the paper reports state-of-the-art performance on all splits.Source: §5.3 · pp. 7–8, Table 5 · p. 8

5.4. Learning Navigation from One Environment

  • With one-environment fine-tuning, Lily improves val seen success rate by 3.60% over VLN-BERT and 1.43% over Airbert, and val unseen success rate by 28.43% over VLN-BERT.The study averages results across five randomly selected environment sets.Source: §5.4 · p. 8, Table 7 · p. 8

6. Conclusion

Lily creates a VLN-like dataset from real house-tour videos and trains an embodied agent on it. The paper reports state-of-the-art performance on R2R and REVERIE.

6. Conclusion

  • Lily combines real-video path-instruction data, entropy-based trajectory construction, action-aware instruction generation, and trajectory judgment pre-training; it reports state-of-the-art results on R2R and REVERIE.The trajectory-judgment task is intended to improve layout reasoning.Source: §6 · p. 8

A. More Details on YouTube Video Collection

  • The video collection restricts sources to categorized real-estate tour playlists and treats each video ID as a house ID.The authors report spending less than an hour on manual search for suitable YouTubers.Source: A. More Details on YouTube Video… · p. 12

B. More Details on Frame Filtering

  • Videos are sampled at 0.5 frames per second, then frames with people, outdoor scenes, or no extractable region features are filtered out.The filtering uses a Places365-pretrained ResNet for outdoor scenes and a COCO-pretrained Mask R-CNN for people.Source: B. More Details on Frame Filtering · p. 12

C. More Details on Trajectory Generation

Trajectory construction combines sampled room nodes with transition nodes, while instruction generation pairs room descriptions with predicted actions. Supplementary figures show example paths and dataset distributions.

C. More Details on Trajectory Generation

  • Trajectories typically contain 4–7 nodes, with 2–7 room nodes sampled in temporal order and remaining positions filled by transition nodes.Similar room-type region features from consecutive frames are merged to approximate panoramic visual context.Source: C. More Details on Trajectory Generation · p. 12
  • Figure 5 presents YouTube-VLN Dataset statistics, including distributions of frames per video, actions, and predicted scene categories on YouTube frames; the displayed action labels include forward, left, and right.Source: Figure 5 · p. 13, C. More Details on Trajectory Generation · p. 13
  • Examples show trajectories with 2 room and 3 transition nodes, 4 room and 1 transition node, or 3 room and 3 transition nodes.Translucent images represent merged images.Source: Figure 6 · p. 14, Figure 7 · p. 14, Figure 8 · p. 15

D. More Details on Instruction Generation

  • Instruction generation uses 14,031 R2R-derived templates, selecting R noun-phrase blanks and R−1 verb-phrase blanks for a trajectory with R room nodes.Room-node captions fill noun blanks, while an inverse-action model supplies transition actions for nearby verb blanks.Source: D. More Details on Instruction… · pp. 14–15

E. Statistics and Visualizations of YouTube-VLN Dataset

The dataset statistics describe the collected videos, filtered images, room categories, and pseudo-labeled actions. A separate architecture figure depicts the adapted model in discriminative and generative settings.

E. Statistics and Visualizations of YouTube-VLN Dataset

  • The dataset was built from 4,078 videos and contains 568K images after noisy-frame filtering.Most videos contain more than 25 effective frames.Source: E. Statistics and Visualizations of… · p. 15
  • CLIP assigns frames to 12 Matterport room types; the reported distribution emphasizes core rooms, while pseudo-labeled actions are evenly distributed across three types.The authors state that core-room coverage supports learning layout priors and action labels represent directions between room nodes.Source: E. Statistics and Visualizations of… · p. 15
  • Figure 9 depicts the adapted model for downstream VLN tasks in both discriminative and generative settings.Source: Figure 9 · p. 16

F. More Implementation Details

The implementation details enumerate language, vision, and navigation-node embeddings, then describe self-attention, cross-modal attention, and feed-forward layers.

F. More Implementation Details

F.1. Pre-training Details

The pre-training and fine-tuning details specify training configurations for the discriminative and generative settings. Pre-training uses a video-level train/test split, while the fine-tuning stages use separate task-specific setups.

F.1. Pre-training Details

  • Pre-training runs for 500 epochs on four GPUs, with 95% of videos used for training and 5% for testing; Figure 10 plots training cross-entropy and test accuracy over epochs.The batch size is 8 and the learning rate is 2 × 10−5.Source: §F.1 · p. 16, Figure 10 · p. 17

F.2. Fine-tuning Details

  • In the discriminative setting, VLN is treated as path selection, with MLM and MVM fine-tuning followed by path-ranking fine-tuning for 30 epochs per stage.The checkpoint with the highest success rate on the validation unseen split is used for test evaluation and leaderboard submission.Source: §F.2 · p. 17
  • For generative fine-tuning, the agent predicts actions sequentially using DUET, initialized with the pre-trained language and cross-modal modules.The setting covers reaching the goal in R2R or finding an object in REVERIE.Source: §F.2 · p. 17

G. Effectiveness of Trajectory Judgment Task on Layout Reasoning Ability

The trajectory judgment task tests whether pre-training supports reasoning about the relative direction of an unexplored room. The task-trained agent reaches substantially higher accuracy than the agent trained without it.

G. Effectiveness of Trajectory Judgment Task on Layout Reasoning Ability

  • The task asks the agent to predict which of 12 intervals contains the relative orientation of a candidate node identified by room type.The room type and current-node panorama are supplied as text and visual input, respectively.Source: G. Effectiveness of Trajectory… · p. 17
  • The evaluation compares two agents trained for 500 epochs, with and without the trajectory judgment task.Accuracy on the orientation question is used to assess layout reasoning ability.Source: G. Effectiveness of Trajectory… · p. 17
  • Around 40% accuracy was reached with trajectory-judgment pre-training, versus 10% without it, nearly equal to 1/12.The task-trained model's training loss drops rapidly and test accuracy rises stably, while the other model's training loss almost does not decrease.Source: G. Effectiveness of Trajectory… · p. 18

H. Transferability Results to Other VLN Benchmarks

Lily is evaluated by transferring models trained on R2R or REVERIE to three other VLN benchmarks without fine-tuning. The reported results show consistent gains over state of the art on their unseen validation splits.

H. Transferability Results to Other VLN Benchmarks

I. Qualitative Results

The qualitative examples illustrate Lily using layout knowledge to choose a route and correctly interpret an action instruction. The authors note that the method remains limited to graph-based environments and identify continuous trajectories and broader data as future directions.

I. Qualitative Results

  • In Figure 11, Lily reaches an office more directly than Airbert, and in Figure 12 it follows “turn left” while Airbert continues forward.The authors attribute the first contrast to layout knowledge and the second to action understanding.Source: §I · p. 18, Figure 11 · p. 19, Figure 12 · p. 20

J. Potential Future Research and Social Impact

  • The method is restricted to graph-based environments; continuous real-world action would require more continuous navigation trajectories for pre-training.The authors also propose improving dataset-construction models and adding richer video and instruction templates.Source: J. Potential Future Research and… · p. 18
  • The authors envisage video-based skill learning helping agents deliver or clean, while noting that access to videos people want kept private requires safeguards.They identify data security as an important issue.Source: J. Potential Future Research and… · p. 18

Related papers

Loading 2307.11984v1…