Source-linked AI summary
Scaling Data Generation in Vision-and-Language Navigation
Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, Yu Qiao
TL;DR
VLN training needs more diverse traversable environments and supervision, while existing scaling requires careful choices across data generation and training. ScaleVLN combines additional HM3D and Gibson environments with large-scale trajectory and instruction synthesis, yielding strong results across navigation tasks, including 80% R2R test success with simple imitation learning.
Problem
VLN training faces limited diversity of traversable environments and insufficient supervision for learning generalizable agents.
Method
ScaleVLN generates augmented VLN data from HM3D and Gibson environments, analyzes its components, and studies its use in pre-training and fine-tuning.
Results
80% success rate on the R2R test split is achieved with simple imitation learning, alongside new state-of-the-art results on CVDN, REVERIE, and R2R-CE.
Takeaways & Limitations
Fully traversable graphs, recovered photo-realistic images, diverse scenes, and appropriate data combination improve downstream navigation and generalization.
Takeaways & Limitations
The generated instructions are low quality, although they substantially influence navigation learning.
Abstract
from arXiv · showhide
Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents. To tackle the common data scarcity issue in existing vision-and-language navigation datasets, we propose an effective paradigm for generating large-scale data for learning, which applies 1200+ photo-realistic environments from HM3D and Gibson datasets and synthesizes 4.9 million instruction trajectory pairs using fully-accessible resources on the web. Importantly, we investigate the influence of each component in this paradigm on the agent's performance and study how to adequately apply the augmented data to pre-train and fine-tune an agent. Thanks to our large-scale dataset, the performance of an existing agent can be pushed up (+11% absolute with regard to previous SoTA) to a significantly new best of 80% single-run success rate on the R2R test split by simple imitation learning. The long-lasting generalization gap between navigating in seen and unseen environments is also reduced to less than 1% (versus 8% in the previous best method). Moreover, our paradigm also facilitates different models to achieve new state-of-the-art navigation results on CVDN, REVERIE, and R2R in continuous environments.
1. Introduction
VLN requires diverse visual-language supervision and traversable environments, but scaling training introduces graph, image-quality, instruction-generation, and data-utilization challenges. The paper proposes a large-scale paradigm and reports strong gains across navigation settings.
- VLN agents need diverse visual-language data to interpret instructions, perceive environments, and learn interaction in photo-realistic scenes.
- Large-scale training must address navigation-graph construction, corrupted rendered images, instruction generation, and effective use of augmented data.
- The paper introduces a reproducible paradigm that uses HM3D and Gibson environments, builds navigation graphs, samples trajectories, generates instructions, and trains downstream agents.
- 80% success rate on the R2R test split is achieved with simple imitation learning, exceeding the previous best method’s 73%.The model uses neither pre-exploration, beam search, nor model ensembling.
2. Related Works
Related work addresses VLN data scarcity by collecting annotations, creating environments, generating data automatically, and scaling pre-training. This paper positions its paradigm as a route to systematic data-acquisition and training guidance.
- VLN research covers instruction following, dialog-based navigation, object grounding, and navigation in continuous environments approximating the real world.
- Expensive navigational data collection creates a data-scarcity problem for visual navigation.
- Prior work scales training through additional human annotations, new environments, automatically generated data, and large-scale pre-training.
- The paper seeks practical guidance for scaling VLN training through comprehensive analysis of data acquisition and agent training.
3. Scaling Data for Learning VLN
ScaleVLN generates R2R-style instruction-path data from additional traversable environments through graph construction, image recovery, trajectory sampling, and instruction generation. The resulting dataset contains nearly 4.94 million pairs.
- ScaleVLN creates R2R-style instruction-path pairs for downstream R2R, CVDN, and R2R-CE tasks.
- Constructing Navigation Graphs: High-quality navigation graphs require broad spatial coverage, fully traversable edges, suitable edge lengths, and centrally positioned nodes.
- Recovering Faulty Rendered Images: Co-Mod GAN recovers broken, distorted, or missing regions in rendered HM3D and Gibson images using rendered-and-camera-image training pairs from Matterport3D.
- Generating Data: The pipeline samples shortest routes with three to five intermediate nodes and uses the EnvDrop Speaker, an LSTM-based model, to generate instructions.
- 4,941,710 instruction-trajectory pairs are generated, approximately 352 times larger than R2R and 4.62 times larger than PREVALENT.
4. Experiments
Experiments analyze how navigation-graph quality, environment diversity, image quality, instruction quality, and training-stage data choices affect VLN performance. The augmented data also transfers across multiple navigation tasks, achieving state-of-the-art results on R2R, REVERIE, CVDN, and R2R-CE.
- Scale VLN Data, What Really Matters?: Fully traversable, dense navigation graphs produce effective augmented data, whereas obstacle-crossing edges can create noisy trajectories that fail to improve unseen-environment performance.The comparison attributes the difference to graph quality and trajectory realism.
- Scale VLN Data, What Really Matters?: More diverse environments improve performance more than generating more samples from the same environments.With equal augmented-scene counts, higher sampling density helps, but matching sample counts across more environments performs better; the number of scenes has stronger impact than sample count.
- Scale VLN Data, What Really Matters?: Recovering rendered images improves results for both HM3D and Gibson data, with larger benefits for low-quality Gibson scans.Using faulty Gibson images can be especially damaging because many reconstructed meshes produce broken or distorted views.
- Scale VLN Data, What Really Matters?: Generated instruction quality strongly influences navigation performance, while the evaluated instruction models produce low-quality language.The authors identify pairing augmented trajectories with better instructions as promising future work.
- How to Utilize Large-Scale Data?: Combining original and augmented data in pre-training and fine-tuning improves generalization, reducing the seen–unseen gap to less than 1% SR.Keeping HM-E data during fine-tuning adds +2.51% SR on Val-Unseen; the reported seen and unseen scores are 80.02% and 79.10%.
- How to Utilize Large-Scale Data?: MLM and SAP are effective and complementary pre-training tasks, whereas adding MRM slightly reduces performance.Combining MLM and SAP yields +16.49% SPL over no pre-training.
5. Conclusion
The paper introduces ScaleVLN, a large-scale VLN data-generation paradigm, and uses it to establish state-of-the-art results across multiple benchmarks.
- ScaleVLN applies thousands of photo-realistic HM3D and Gibson environments to create millions of instruction-trajectory training pairs.
- The study evaluates navigation-graph quality, camera-quality image recovery, augmented-data use in pre-training and fine-tuning, and pre-training-task choices.
- ScaleVLN achieves new state-of-the-art results on R2R, REVERIE, CVDN, and R2R-CE across distinct instruction styles and action spaces.
6. Acknowlegement
The authors acknowledge reviewer feedback and funding support for the work.
- The authors thank ICCV reviewers for their helpful suggestions.
- The work is partially supported by China’s National Key R&D Program.
- Additional support comes from the Shanghai Committee of Science and Technology, and Taesung Park provided suggestions on image recovery.
Appendices
The appendices provide implementation details, additional experiments, leaderboard results, and further analyses of ScaleVLN.
- Appendix A: Appendix A describes pre-training objectives and details of the REVERIE experiments.
- Appendix B: Appendix B examines visual encoders, model initialization, depth features, different VLN agents, and long-horizon navigation on R4R.
- Additional results: The appendices also include R2R leaderboard results and object-grounding results for REVERIE.
A.1. Pre-Training Objectives (§4.1)
The agent is pre-trained with proxy tasks that use instructions and demonstration paths to improve language, visual, and action-related representations.
- Masked Language Modeling (MLM): MLM predicts randomly masked instruction words using textual context and the full trajectory.Fifteen percent of instruction tokens are randomly masked.
- Masked Region Modeling (MRM): MRM predicts semantic labels for masked history-observation regions from instructions and neighboring regions.View images are randomly removed with 15% probability, and training minimizes KL-divergence to target label distributions.
- Single Action Prediction (SAP): SAP predicts the next action from the instruction and the agent’s given path.The model assigns probabilities to candidate actions and minimizes the negative log probability of the target view action.
A.2. Implementation Details of REVERIE (§4.1)
The REVERIE implementation augments training with object-grounded trajectories and evaluates how visual encoders, initialization, depth, and dataset scaling affect navigation. These experiments identify strong effects from encoder choice and additional ScaleVLN data across models and longer-horizon tasks.
- Data generation: REVERIE trajectories target visible objects through sampled paths ending at their viewpoints, producing 518,233 HM3D and 311,976 Gibson paths.Instructions are generated in the REVERIE style from these object-directed trajectories.
- Ablations: The visual encoder has a major influence on SPL, indicating that encoder choice affects efficient navigation.The comparison uses CLIP ViT backbones with Base/16, Large/14, and Huge/14 configurations.
- Ablations: BERT initialization does not improve downstream performance and harms validation-unseen performance, whereas LXMERT initialization can improve results.The authors attribute BERT’s weakness to a domain gap between unimodal language representations and CLIP visual representations.
- Dataset scaling: Adding ScaleVLN data benefits multiple VLN agents, including Seq2Seq, EnvDrop, and HAMT.The agents are trained with CLIP ViT-B/16 features under configurations adapted from their original methods.
- Dataset scaling: +2.7% SR, +1.5% nDTW, and +2.7% SDTW result on R4R when ScaleVLN is added during pre-training.R4R uses concatenated adjacent R2R trajectories to form longer navigation paths.
B.6. Leaderboard Results of R2R (§4.4)
The R2R leaderboard comparison shows that ScaleVLN’s single-run result remains competitive with methods using beam search or test-environment pre-exploration. It is reported as the leading non-human submission.
- Leaderboard comparison: ScaleVLN’s single-run result outperforms the other top-seven submissions by a large margin despite their use of beam search or pre-exploration.Most competing methods have extremely low SPL when ranked by success rate because they use beam search; pre-explored methods also remain worse when ranked by SPL.
- Leaderboard comparison: The authors report ScaleVLN as ranked first on the R2R test-unseen leaderboard apart from human followers.The comparison is based on the leaderboard results reported on 28 July 2023.
B.7. REVERIE Object Grounding Result (§4.4)
ScaleVLN achieves state-of-the-art REVERIE object grounding performance, while its navigation graphs are denser and more traversable than AutoVLN’s. The image-recovery pipeline produces clearer views and restores missing scene regions for augmented training data.
- Object grounding: ScaleVLN achieves state-of-the-art REVERIE object-grounding performance, comparable to the previous best AutoVLN method.The reported metrics are remote object-grounding success rate and its path-length-weighted variant.
- Navigation graphs: ScaleVLN’s navigation graphs are denser, cover more regions, avoid obstacles, and remain fully traversable in open space.The comparison is visualized against AutoVLN across several HM3D environments.
- Image recovery: The recovery method restores missing outdoor and indoor regions and generally produces sharper object boundaries with fewer blurry or distorted areas.Examples include sky, trees, floors, walls, ceiling lights, chairs, and door frames.