Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments

1 Introduction

• Standard VLN assumes navigation among neighboring nav-graph nodes with perfect travel between them, while VLN-CE requires low-level actions in a 3D simulator.These settings share the high-level instruction-guided task, but VLN-CE drops the nav-graph assumptions.Source: §1 · p. 1
• Published VLN-CE episode success rates were less than half those in standard VLN, and sim-2-sim transfer had been under-explored and uncertain.The paper identifies transfer and more capable VLN-CE models as avenues for improvement.Source: §1 · pp. 1–2
• +12% success rate over prior published state of the art (32% vs 44%) on the VLN-CE test set demonstrates the value of transferring a VLN agent.The final model is transferred from the abstract VLN setting to VLN-CE.Source: §1 · p. 2
• The authors build a modular harness to run VLN agents in VLN-CE, adding local navigation and subgoal generation to mimic nav-graph candidates.They use systematic experiments to quantify which differences between the settings most affect performance.Source: §1 · p. 2
• The experiments quantify transfer losses from task-data differences, visual domain shift, navigation policies, and subgoal candidate generation.The analysis identifies episodes with significant elevation change, such as stairs, as challenging for the model.Source: §1 · p. 2
• The authors present transfer as an alternative direction for improving instruction-guided navigation in continuous environments and plan to release their model-agnostic harness.They describe the remaining challenges as directions for further research.Source: §1 · p. 2

2 Related Work

• VLN ⟳ BERT combines a transformer-based agent with recurrent history context, and the authors adopt it as their core VLN agent.The related work also notes pre-training and data augmentation for limited data and challenges in jointly processing vision, language, and action.Source: §2 · p. 3
• VLN-CE replaces navigation graphs with low-level actions, and prior approaches include end-to-end agents, language-aligned path supervision, and waypoint actions trained with reinforcement learning.The cited prior work also used continuous environments to study topological map generation and semantic mapping.Source: §2 · p. 3
• Anderson et al. transferred VLN agents to reality using subgoal prediction and ROS navigation, reporting a more than 2x performance drop.The authors characterize effective VLN sim-to-real transfer as an open research question.Source: §2 · p. 3
• This work examines sim-2-sim transfer from high-level simulation to lower-level simulation and diagnoses challenges also identified in sim-to-real transfer.The authors note that some mitigation methods are infeasible in VLN and impractical in reality.Source: §2 · p. 3

3 Task Descriptions

• VLN selects among neighboring nav-graph nodes, whereas VLN-CE navigates a continuous 3D mesh with low-level actions.VLN-CE uses FORWARD 0.25m, TURN-LEFT 15°, TURN-RIGHT 15°, and STOP actions.Source: §3 · pp. 3–4
• VLN agents receive an instruction, a panoramic observation represented by 36 views, and subgoal candidates from which they select their next location.The panorama has 12 evenly spaced headings at each of three elevation angles; the process continues until STOP.Source: §3 · p. 4
• R2R contains 7,189 trajectories, and the VLN-CE subset contains 5,611 trajectories traversible in continuous environments.Val-Seen has novel paths and instructions in training scenes; Val-Unseen also uses novel scenes.Source: §3 · p. 4
• The two tasks use different distance calculations, with VLN using Euclidean node distances and VLN-CE using geodesic distances on a 3D navigation mesh.This tends to produce longer path lengths and lower SPL in VLN-CE.Source: §3 · pp. 4–5

4 Porting a VLN agent to Continuous Environments

• A modular harness ports VLN agents to VLN-CE by combining subgoal generation with a lower-level navigation policy.The harness is designed for drop-in replacement of its subgoal-generation, VLN-agent, and navigation-policy components.Source: §4 · p. 5

4.1 Core VLN Agent

• The experiments use VLN ⟳ BERT, a recurrence-modified transformer agent that encodes RGB vision with a ResNet-152 trained on Places365.The harness itself is agnostic to the selected VLN agent.Source: §4.1 · pp. 5–6

4.2 Navigation Policies

• The navigation-policy comparison includes teleportation and an oracle policy as upper bounds, plus a local policy admissible in VLN-CE.The local policy builds an occupancy map from egocentric depth, plans with the Fast Marching Method, and stops within 0.15m of the target or after 40 actions.Source: §4.2 · p. 6

4.3 Subgoal Candidate Generation

• The subgoal generation module predicts candidates from panoramic RGB and a 360° laser obstacle map, then returns the five highest-probability candidates.A U-Net predicts a radial map; the system trains it against nav-graph subgoals using Sinkhorn divergence and Habitat-rendered panoramas.Source: §4.3 · pp. 6–7

5.1 How different is the VLN-CE subset of R2R?

• VLN ⟳ BERT performs worse on the VLN-CE subset of R2R, with success drops of 3 points in Val-Seen and 1 point in Val-Unseen.The subset has 22% fewer trajectories than R2R because of conversion errors.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 What is the visual domain gap from VLN to VLN-CE?

• VLN and VLN-CE observations differ because VLN uses high-quality Matterport3D panoramas while VLN-CE uses renders from 3D scene reconstructions.Examples include differences in lighting, furniture resolution, and object presence.Source: Figure 2 · p. 8, §5.2 · p. 8
• Training VLN ⟳ BERT on reconstructed vision recovers seen-environment performance and leaves Val-Unseen success rate only 1 point below the original model.The reconstruction-trained model achieves Val-Unseen success rate 60 with either MP3D panoramas or Habitat renders.Source: Table 2 · p. 9, §5.2 · p. 9
• Habitat renders reduce success rate by 13 points in Train∗, 10 points in Val-Seen, and 3 points in Val-Unseen compared with MP3D panoramas.The larger drops in seen environments suggest possible overfitting to training-environment visual representations.Source: §5.2 · pp. 8–9

5.3 What is the navigation gap between VLN and VLN-CE?

• The navigation comparison first tests VLN-CE actions with an Oracle Policy, then tests a realistic Local Policy while retaining nav-graph subgoals.The table compares navigation policies under the assumption of a known nav-graph.Source: §5.3 · p. 9, Table 3 · p. 10
• The Local Policy trails the Oracle Policy by 3 success points in Val-Seen and 5 in Val-Unseen, while its >0.5m navigation-failure rate is 15x higher.The Local Policy’s SPL drop matches its success drop; the experiment assumes candidates from the nearest nav-graph node after navigation terminates.Source: §5.3 · pp. 9–10

5.4 What is the subgoal candidate gap?

• The SGM experiments compare graph-provided candidates with predicted candidates; the predicted-candidate rows form a complete VLN-CE harness with lower-level navigation.Online prediction is required because VLN-CE does not provide nav-graph candidates.Source: Table 4 · p. 11, §5.4 · p. 11
• Even optimal SGM predictions lose 3 points of Val-Seen success and 6 points of Val-Unseen success relative to nav-graph candidates.The prediction space discretizes candidate locations into polar bins with 0.20m range and 7.5° resolution.Source: §5.4 · p. 11
• Replacing nav-graph candidates with SGM predictions decreases success by 12 SR in Val-Seen and 8 SR in Val-Unseen.The authors interpret these drops as a large domain gap between the nav-graph and SGM.Source: §5.4 · p. 11

5.5 Comparison With Previous VLN-CE Models

• 44 SR on Test is 12 points above the next-best model, a 38% relative improvement, after 12 GPU-hours of fine-tuning.The final agent combines SGM-generated candidates, the reconstruction-trained VLN ⟳ BERT fine-tuned on those candidates, and Local Policy navigation.Source: Table 5 · p. 12, §5.5 · p. 12

Fine-Tuning VLN

• Fine-tuning the reconstruction-trained VLN ⟳ BERT on SGM candidates raises Val-Unseen success by 2 points but leaves gaps to optimal and nav-graph candidates.The remaining gaps are 6 points to optimal subgoal candidates and 12 points to candidates directly from the nav-graph.Source: Fine-Tuning VLN · pp. 11–12

5.5 Comparison With Previous VLN-CE Models

• The final agent surpasses previously published models on all VLN-CE Challenge Leaderboard splits, including Test.Prior methods were trained exclusively in VLN-CE, and the best among them required more than 7,000 GPU-hours.Source: §5.5 · p. 12

• 12 SR of the diagnosed drop comes from subgoal candidate generation, with 6 SR attributable to episodes requiring at least 1.0m of elevation change.The authors identify insufficient recall of elevation-changing subgoals as the likely issue.Source: §5.6 · pp. 12–13, Figure 3 · p. 13
• Even with greedy-oracle selection and Oracle Policy navigation, SGM candidates yield only 38% success on Train∗ episodes requiring at least 1.0m elevation change.Across all episodes, oracle success is 89% in Train∗, 93% in Val-Seen, and 90% in Val-Unseen.Source: §5.6 · p. 13

Elevation-Based Performance of VLN-CE

• Val-Unseen success for VLN-CE ⟳ BERT is 19 SR on high-elevation episodes versus 48 SR on low-elevation episodes.The authors link the gap to poor recall of elevation subgoals, which are less represented in training data and poorly predicted by 2D free-space.Source: Elevation-Based Performance of VLN-CE · p. 13, Figure 4 · p. 14
• In VLN, reconstruction-trained VLN ⟳ BERT reaches 68 SR on high-elevation Val-Unseen episodes versus 57 SR on low-elevation episodes.The authors estimate that solving the VLN-CE elevation problem could mitigate 6 points of the 12 SR drop attributed to candidate generation.Source: Elevation-Based Performance of VLN-CE · p. 13

5.7 Qualitative Example.

• In the qualitative example, the Local Policy routes around a countertop when a selected subgoal cannot be reached in a straight line.The agent makes appropriate candidate selections over five actions despite observing from poses off the nav-graph.Source: §5.7 · p. 14

• Transfer from topological VLN to VLN-CE improves the prior state of the art by 12% and identifies subgoal candidate generation as a primary hindrance.The authors outline candidate generation in multi-floor environments as a problem for future work.Source: §6 · p. 14

7.1 VLN

• Retrained VLN ⟳ BERT is 2 SR and 1 SPL lower than the published result on Val-Unseen, but matches its Val-Seen SR and SPL.Performance was consistent across three random seeds.Source: BERT: Validation Performance · p. 17

7.2 Oracle Policy Detail

• The projection procedure maps predicted distance and relative heading to global 3D coordinates, assumes the target is at the agent’s elevation, then snaps to a reachable navigation-mesh position.The snapped position must have finite geodesic distance from the agent’s current position.Source: §7.2 · p. 17

7.3 Subgoal Module Ablations

• 8 SR in Val-Unseen: reconstructed vision and 360° scanning together improve the SGM over the version used by Anderson et al..Reconstructed vision improves performance across all splits; reducing the scan from 360° to 270° causes a further drop.Source: §7.3 · p. 17, Table 6 · p. 18

7.4 Distribution of Navigation Errors

• In Val-Unseen, the Local Policy has an 8.04% rate of errors at 0.20m and fails to reach within 1.0m of the subgoal 5.73% of the time.The Oracle Policy produces errors of at least 0.20m in 0.53% of navigations.Source: §7.4 · pp. 17–18, Table 7 · p. 18