Source-linked AI summary
EnvEdit: Environment Editing for Vision-and-Language Navigation
Jialu Li, Hao Tan, Mohit Bansal
TL;DR
VLN agents face limited training data and environmental diversity, making generalization to unseen environments challenging. EnvEdit edits existing environments for training, and reports improved metrics and new state-of-the-art results on R2R and RxR leaderboards.
Problem
VLN datasets are limited in scale and environmental diversity, making it challenging for agents to generalize to unseen environments.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Ensembling agents trained on different edited environments shows that the editing methods are complementary.
Source: §7 · p. 8The work explores two classic style-transfer and image-synthesis approaches; more advanced image-to-image translation models may generate higher-quality environments and further improve results.
Source: L. Limitations · p. 17
Abstract
from arXiv · showhide
In Vision-and-Language Navigation (VLN), an agent needs to navigate through the environment based on natural language instructions. Due to limited available data for agent training and finite diversity in navigation environments, it is challenging for the agent to generalize to new, unseen environments. To address this problem, we propose EnvEdit, a data augmentation method that creates new environments by editing existing environments, which are used to train a more generalizable agent. Our augmented environments can differ from the seen environments in three diverse aspects: style, object appearance, and object classes. Training on these edit-augmented environments prevents the agent from overfitting to existing environments and helps generalize better to new, unseen environments. Empirically, on both the Room-to-Room and the multi-lingual Room-Across-Room datasets, we show that our proposed EnvEdit method gets significant improvements in all metrics on both pre-trained and non-pre-trained VLN agents, and achieves the new state-of-the-art on the test leaderboard. We further ensemble the VLN agents augmented on different edited environments and show that these edit methods are complementary. Code and data are available at https://github.com/jialuli-luka/EnvEdit
1. Introduction
EnvEdit edits training environments to address VLN agents’ difficulty generalizing across diverse, unseen settings. Across R2R and RxR, it improves results for pretrained and non-pretrained agents and reaches new leaderboard state of the art.
1. Introduction
- VLN datasets have limited scale and environmental diversity, while navigation settings can differ substantially, making generalization to unseen environments difficult.Prior work reports substantially worse agent performance in unseen environments.Source: §1 · p. 1
- Feature dropout and environment mixing address environment bias, but mixing existing environments does not introduce unseen modifications to them.The paper contrasts these methods with edits that change style, appearance, and objects.Source: §1 · p. 1
- Image-text pretraining can mismatch VLN inputs because its panoramas and instructions differ from panoramic observations and human-written navigation instructions.The cited discussion notes that even substantial pretraining data yields limited VLN performance gains.Source: §1 · p. 2
- EnvEdit creates training environments that vary in style, object appearance, and object classes while largely retaining original semantic information.This lets the method reuse human-annotated instructions; a style-aware speaker also generates instructions for unannotated paths.Source: §1 · p. 2
- ENVEDIT reaches new state-of-the-art results on both R2R and RxR test leaderboards, improving all reported metrics for pretrained and non-pretrained agents.For non-pretrained agents, gains include 1.6% in SR and 1.4% in SPL on R2R, and 5.3% in nDTW and 8.0% in sDTW on RxR.Source: §1 · p. 2
2. Related Work
Prior VLN research addresses environment bias through feature engineering and image-text pretraining, while environment-level augmentation remains less explored. EnvEdit uses style transfer and image synthesis to create novel training environments.
2. Related Work
- VLN agents commonly use cross-modal alignment, sequence models, and imitation-learning and reinforcement-learning training; this paper builds on EnvDrop [61] and tests compatibility with pretrained models [9] [26].Source: §2 · p. 2
- Environment-bias methods include feature-level engineering [61] [66] and image-text or pretrained-model approaches [9] [20] [26] [46], but their data can differ from VLN panoramas and human-written instructions.Source: §2 · pp. 2–3
- Instruction augmentation generates language for unannotated paths [16] [61], whereas environment mixing [42] recombines existing paths without creating new styles unseen by the agent.EnvEdit instead creates environments through style transfer and image synthesis.Source: §2 · p. 3
- Computer-vision augmentation includes geometric and noise-based transformations [29] as well as GAN-based methods [5] [17] [30] [55] [58] [69]; EnvEdit applies style transfer [18] and image synthesis [54].Source: §2 · p. 3
3. Method Overview
The method defines navigation over panoramic observations and uses a three-stage training procedure: create edited environments, train on original and edited settings, then generate instructions for unannotated paths.
3.1. Problem Setup
- At each step, the VLN agent chooses among navigable locations using a panoramic observation discretized into 36 views, and predicts STOP when navigation ends.View features combine visual representations with orientation features; each navigable location is represented by its closest view and orientation.Source: §3.1 · p. 3
3.2. Training Procedures
- Environment creation uses style transfer and image synthesis to produce environments differing from originals in style, appearance, and object classes.The style-transfer models are off-the-shelf, while the image generator and style encoder are trained on seen R2R environments.Source: §3.2 · p. 4
- Navigation training uses both original and edited environments, with half of each instruction-path batch observing each environment type.The procedure combines imitation learning and reinforcement learning, following [61].Source: §3.2 · p. 4
- Back translation generates instructions for unannotated paths, and the style-aware speaker conditions generation on the route’s start-view style embedding.The decoder attends to route context when predicting the next instruction word.Source: §3.2 · p. 4
4. Environment Editing
EnvEdit edits environments along style, object appearance, and object-class dimensions while preserving enough semantics to reuse the original instructions. Style transfer changes style alone, while conditional image synthesis can vary appearance and object classes.
4. Environment Editing
- EnvEdit edits environment style (S), object appearance (A), and object class (O), aiming to preserve semantic correspondence with the original instructions.The method uses style transfer and image synthesis to create edited environments.Source: §4 · p. 4, §4.1 · p. 4, §4.2 · p. 5
4.1. Style Transfer
- Style transfer preserves the original environment’s semantics while changing style, allowing its human-annotated instructions to be reused.A style embedding is sampled from the Painter By Numbers style-embedding distribution and incorporated during decoding through conditional instance normalization.Source: §4.1 · p. 4
- The same sampled style embedding is applied across all 36 views in a panorama to keep the agent’s visual observation stylistically consistent.The paper reports that this setup is crucial for creating effective edited environments.Source: §4.1 · p. 5
4.2. Image Synthesis
- Conditional image synthesis uses the original semantic segmentation to preserve scene semantics while varying object shape and appearance; changing segmentation classes can also change objects.The synthesis model is a GAN conditioned on semantic masks, with SPADE normalization; style embeddings control synthesis style.Source: §4.2 · p. 5
- The method creates appearance-only environments by retaining original style, and other variants that change style and appearance or randomly mask objects and alter all three environment components.For object editing, a randomly selected semantic class is assigned to a mask class during training, enabling generation of random masks for that class.Source: §4.2 · p. 5
5. Experimental Setup
The experiments use R2R and multilingual RxR, with training, seen-validation, unseen-validation, and test splits. Evaluation uses six navigation metrics, with different primary metrics for the two datasets.
5.1. Datasets
- R2R contains English instructions, while RxR contains English, Hindi, and Telugu instructions; both datasets have training, seen- and unseen-validation, and test splits.Unseen-validation and test environments do not appear in training.Source: §5.1 · p. 5
5.2. Evaluation Metrics
- Evaluation uses SR, SPL, TL, NE, nDTW, and sDTW, with SR and SPL primary for R2R and nDTW and sDTW primary for RxR.Source: §5.2 · pp. 5–6
- Table 1 reports R2R val-unseen performance for agents trained with one edited-environment type; its indicators distinguish changed from unchanged environment components.The table identifies ViT-B/32 and ViT-B/16 image features and marks reproduced results.Source: Table 1 · p. 6
6. Results and Analysis
Edited environments improve unseen-set performance, with gains varying across edit types and visual backbones. The analyses also examine pretrained agents, method ablations, ensembling, test-set comparisons, and qualitative edits.
6.1. Results for Environment Editing Methods
- Est improves R2R validation-unseen SR by 5.2% and SPL by 4.9% with ViT-B-32, and by 3.1% SR and 2.7% SPL with ViT-B-16, versus their baselines.Any newly created environment outperforms the baseline by a large margin on the validation-unseen set.Source: §6.1 · p. 6
- Keeping style unchanged while modifying object appearance improves SR by 2.0% with ViT-B/32 and 0.2% with ViT-B/16; edit-method rankings also vary by backbone and base agent.The authors attribute the style-preserving result to retaining correspondence with original instructions while providing diversity, and recommend Est as a starting point across models.Source: §6.1 · p. 6
6.3. Method Ablations
- Changing style within a panorama lowers SR by 2.3% and SPL by 1.7%; fixing one style for the whole environment reduces improvements by 2.4% and 2.0%, respectively.The authors attribute the advantage of a fixed style per viewpoint to balancing consistent observations with style variation.Source: Table 3 · p. 7, §6.3 · p. 7
- Table 4 reports around 1% higher SR and SPL with the style-aware speaker for both ViT-B/32 and ViT-B/16 features.The authors say incorporating environment style helps generate instructions that better match the environments.Source: Table 4 · p. 7, §6.3 · p. 7
6.4. Combining Multiple Environments
- Ensembling agents trained on Est, Eis1, and Emis1 improves SR by 0.6% and SPL by 1.2% over using Est alone with EnvDrop.Similar improvement is reported with HAMT as the base agent.Source: Table 5 · p. 7, §6.4 · p. 7
6.5. Test Set Results
- The test-set analysis uses single-run evaluation; Tables 6 and 7 compare agent performance on R2R and RxR, respectively.The supplied passages identify the comparison scope but do not report table values.Source: §6.5 · p. 7, Table 6 · p. 8, Table 7 · p. 8
6.6. Qualitative Analysis for Edited Environments
- Qualitative examples show style transfer retaining original scene semantics with an artistic look, while image synthesis changes object appearance and looks closer to real environments.Masking a cabinet in an Emis1 example adds environmental diversity.Source: §6.6 · p. 8, Table 8 · p. 8
7. Conclusion and Discussion
The paper concludes that editing existing environments improves VLN performance across R2R and RxR and that agents trained on different edits can complement one another.
7. Conclusion and Discussion
- ENVEDIT edits existing environments in style, object appearance, and object classes; the authors report improved metrics on R2R and RxR and new state-of-the-art results on both test leaderboards.Ensembling agents trained on different edited environments is also reported to improve overall performance.Source: §7 · p. 8
A. Appendix Overview
- The appendix covers editing and training implementation, evaluation metrics, back-translation results, pretrained-agent adaptation, preprocessing, curriculum learning, class masking, unseen-environment performance, and qualitative examples.It also identifies a separate section for limitations.Source: A. Appendix Overview · p. 12
B. Implementation Details
- For R2R image synthesis, the paper trains the generation model for 10 epochs on training environments using semantic segmentations as input; VLN training uses default hyperparameters from prior work.For RxR, the instruction encoder is changed to cased multilingual BERTBASE; back translation uses the speaker and agent hyperparameters from prior work.Source: B. Implementation Details · p. 12
C. Evaluation Metrics
- The evaluation uses six metrics: SR, SPL, trajectory length, navigation error, nDTW, and sDTW; SR and SPL are primary for R2R, while nDTW and sDTW are primary for RxR.The passage defines success as stopping within 3 meters of the target and describes SPL as penalizing long paths.Source: C. Evaluation Metrics · p. 12
D. Back Translation Results
After back translation, the best SR environment differs by visual backbone. Image-synthesis edits also show larger reported improvement with back translation than style-transfer edits.
D. Back Translation Results
- After back translation, Eis1 gives the best SR for ViT-B/32 and Emis1 for ViT-B/16, outperforming their baselines by 4.9% and 3.1%, respectively.Training with any created environment improves the baseline by a large margin in these experiments.Source: D. Back Translation Results · p. 12
- With back translation, image-synthesis environments improve performance by 3.5%, versus 2.1% for style-transfer environments.The authors note that models trained with different environments benefit differently from back translation.Source: D. Back Translation Results · p. 12
E. Performance on Rec-BERT
ENVEDIT improves R2R performance when applied to Rec-BERT, while preprocessing changes the baseline and reduces the performance gap between edited variants. Adapting ViT-B/16 features is constrained by unavailable pretraining code.
E. Performance on Rec-BERT
- Eis1 could improve the baseline by 0.5% in SR and 1.1% in SPL; augmenting with the other two environments could improve the baseline by 0.5% in both metrics.Table 10 reports performance on the R2R validation-unseen set.Source: E. Performance on Rec-BERT · p. 13, Table 10 · p. 13
- The Rec-BERT experiments also explore ViT-B/16 features, but unavailable source code for PREVA-LENT makes in-domain pretraining followed by transfer difficult.The ViT-B/16 features are mapped from 512 to 2048 dimensions with a linear layer and dropout.Source: E. Performance on Rec-BERT · p. 13
- Augmenting the original environments with Eis1 achieves the best performance, improving the baseline by 3.0% in SR and 1.0% in SPL; Est and Em yield 1.6% and 1.3% SR improvements, respectively.Source: E. Performance on Rec-BERT · p. 13
F. Image Preprocessing Variants
- Removing resize-and-crop preprocessing significantly improves the baseline, while editing narrows the performance gap between the two preprocessing variants.The authors attribute the baseline change to its Place365 features being extracted without resize and crop.Source: F. Image Preprocessing Variants · p. 13
G. Curriculum Learning
Curriculum training underperforms joint training on original and edited environments in the reported experiments. Masking one object class performs best in the tested setup, while masking more classes reduces performance.
G. Curriculum Learning
- Training first on the original environment and then on edited environments performs much worse than the reverse order, which the authors attribute to higher-quality original data.The two training steps each use 50,000 iterations; back translation is not applied.Source: G. Curriculum Learning · p. 14, Table 12 · p. 14
- Curriculum learning yields lower performance than training on original and edited environments in the same batch.The authors suggest that better-designed difficulty ranking may be needed for higher performance.Source: G. Curriculum Learning · pp. 13–14, Table 12 · p. 14
- Sequential training on multiple edited environments followed by the original environment performs worse than training on Est alone; adaptive curriculum learning reaches 57.6%, still below Est alone.The authors identify combining editing environments effectively as an open problem requiring further investigation.Source: G. Curriculum Learning · p. 14, Table 12 · p. 14
H. Editing Different Number of Objects
- Masking more object classes consistently reduces performance, and masking one class gives the best result in this experimental setup.The authors frame the choice as balancing correspondence with original instructions against diversity in new environments.Source: H. Editing Different Number of Objects · pp. 14–15, Table 13 · p. 14
I. Detailed Results on Different Environments
Different unseen environments benefit unequally from the editing methods: some show substantial gains, and the authors report complementary information across edited environments.
I. Detailed Results on Different Environments
- 15.0% in SR and 13.8% in SPL on Environment 5, and 7.2% in SR and 4.6% in SPL on Environment 11, are gains from training on Est.For Eis1, Environment 1 receives the largest improvement; the authors suggest its difference in object appearance may explain this pattern.Source: §I · p. 16
- Unseen environments benefit differently from the edited environments, which the authors describe as containing complementary information for generalization.They identify learning from multiple new environments as a future direction.Source: §I · p. 16
J. Analysis of Consistency Across Different Environments
The editing method that helps most varies across environments and VLN models, though each tested editing method improves performance. Qualitative examples distinguish the edits, and the authors identify their limited editing approaches as a limitation.
J. Analysis of Consistency Across Different Environments
- Different VLN models generalize differently across environments, so their best editing method varies; the authors recommend Est as a simple starting point.EnvDrop+BT and Rec-BERT perform better than EnvDrop on Env5, while Emis1 can benefit EnvDrop+BT on Env11.Source: J. Analysis of Consistency Across… · pp. 16–17
K. Qualitative Examples for Generated Environments
- Est preserves original scene semantics better but has an artistic overall style, whereas Eis1 and Emis1 add object-appearance diversity and look closer to real environments.These are qualitative observations from the generated-environment examples.Source: K. Qualitative Examples for Generated… · p. 17
L. Limitations
- The study tests two classic style-transfer and image-synthesis approaches; more advanced image-to-image translation may improve environment quality, and use beyond VLN would require adaptation.The authors leave exploration of other tasks for future work.Source: L. Limitations · p. 17