Language-Aligned Waypoint (LAW) Supervision for Vision-and-Language Navigation in Continuous Environments

1 Introduction

• Shortest-path supervision is goal-oriented and may not correspond to following the natural-language instruction, especially when an agent leaves the reference path.The language-aligned path can differ from the shortest route to the goal.Source: §1 · p. 1
• LAW supervision directs the agent toward the nearest waypoint on the language-aligned path at every step.The paper proposes a denser signal that leads the agent back to the reference path.Source: §1 · pp. 1–2
• Dense waypoint supervision is especially important in VLN-CE, where episodes average 55.88 steps versus 4–6 nodes in discrete VLN.The authors investigate how waypoint-supervision density affects task performance.Source: §1 · p. 2
• The paper adds waypoint accuracy and combines language-aligned waypoints with sub-instruction information to assess instruction following.It complements normalized Dynamic Time Warping (nDTW) with these measures.Source: §1 · p. 2
• Language-aligned supervision trains agents to follow instructions more closely than goal-oriented supervision.The introduction also presents waypoint accuracy and sub-instruction information as ways to assess instruction following.Source: §1 · p. 2

2 Related Work

• Discrete-nav-graph research has explored language–vision grounding, progress monitoring, trajectory-based instruction generation, and environmental dropout.These lines of work include Wang et al. (2019), Ma et al. (2019), Fried et al. (2018), and Tan et al. (2019).Source: §2 · p. 2
• The cited methods fail to achieve similar performance on the more challenging VLN-CE task, where agents navigate a continuous 3D simulation; Chen et al. (2021) propose a modular topological-map approach that achieves better results.Source: §2 · p. 2
• Continuous-environment approaches by Chen et al. (2021) and Krantz et al. (2021) focus on reaching the goal, whereas this work targets accurate adherence to the instructed path.Krantz et al. (2021) predict panoramic waypoints and use low-level control for navigation.Source: §2 · p. 2
• Discrete-VLN instruction-adherence methods include Bayesian state tracking, object-and-action attention, curriculum learning, and sequential sub-instruction traversal.Hong et al. (2020) also introduce FG-R2R with sub-instruction-to-sub-path mappings.Source: §2 · p. 2
• Unlike Jain et al. (2019), which uses CLS-based rewards, LAW dynamically selects the closest reference waypoint and supplies cross-entropy supervision at every step.LAW is online imitation learning rather than a policy-gradient formulation; Jain et al. (2019) use a sparse end-of-episode reward during stage two.Source: §2 · p. 2

3 Approach

• VLN-CE adapts R2R to Habitat and provides episodes with language instructions and reference paths.Reference paths are built from panoramic-camera nav-graph nodes and dense step waypoints spaced for a 0.25 m agent step.Source: §3 · pp. 2–3
• The model adapts Cross-Modal Attention (CMA), using recurrent networks to encode agent history and predict actions from attended visual and instruction features.Training starts with teacher-forced behavior cloning on augmented data, then fine-tunes with DAgger.Source: Figure 2 · p. 3, §3 · p. 3
• The method uses path waypoints as language-aligned waypoint supervision, contrasting with prior goal-oriented supervision; its language-aligned sensor helps return an agent to the next waypoint when it wanders off path.Source: §3 · p. 3
• About 6% of VLN-CE R2R episodes have nDTW(shortest, LAW step) < 0.8.This indicates a subset of episodes where the shortest path differs from the dense language-aligned path by this nDTW criterion.Source: §3 · p. 3
• At each step, the ground-truth action follows the shortest path from the agent’s position to its nearest language-aligned waypoint.The objective uses cross-entropy loss to maximize the log likelihood of that action; φ selects the nearest waypoint and g chooses the action.Source: §3 · p. 3

Dataset and evaluation

• Waypoint Accuracy measures the fraction of waypoints visited within 0.5 m, complementing nDTW and other standard navigation metrics.FG-R2R supplies sub-instruction-to-sub-path mappings for qualitative analysis.Source: §4 · p. 3

Qualitative analysis

• On an unseen R2R episode, the language-aligned agent’s trajectory follows the instruction more closely than the goal-oriented agent’s, with higher nDTW and Waypoint Accuracy.Figure 4 also shows how FG-R2R maps sub-instructions to waypoints.Source: Figure 4 · p. 4

Quantitative Results

• LAW pano outperforms goal-oriented supervision across all metrics in seen and unseen validation environments, including nDTW and Waypoint Accuracy.The table caption identifies the comparison and instruction-following metrics.Source: §4 · p. 4, Table 1 · p. 5

Waypoint-density ablation

• Language-aligned waypoint densities perform similarly overall, while LAW#4 is better than LAW pano for paths where one sub-instruction maps to several panoramic waypoints.The authors suggest short R2R trajectories make LAW pano denser than needed; Table 2 spans sparse #2 to dense step supervision.Source: §4 · p. 4, Table 2 · p. 5

Path-alignment analysis

• LAW performs better than goal on episodes with lower path similarity across both nDTW and Waypoint Accuracy.Figure 3 bins episodes by similarity between the language-aligned and goal-oriented paths.Source: §4 · p. 4, Figure 3 · p. 4

Additional experiments

• On unseen VLN-CE RxR, LAW step improves Waypoint Accuracy by 6% and nDTW by 2% over goal; the best mixed-loss model reaches 53% nDTW versus 54% for LAW pano.The mixed goal-oriented and language-oriented losses do not outperform LAW pano.Source: §4 · p. 5

Conclusion

• Language-aligned supervision improves instruction following in VLN compared with the goal-oriented supervision commonly used in prior work.The authors describe LAW as a simple, useful strategy for VLN-CE.Source: §5 · p. 5
• Waypoint Accuracy makes it easier to interpret how agent navigation corresponds to following sub-instructions in the input language.The conclusion says quantitative and qualitative results demonstrate the benefit of LAW supervision.Source: §5 · p. 5
• The authors characterize LAW as a simple but useful strategy for improving VLN-CE.Source: §5 · p. 5

• LAW means language-aligned waypoints that make the navigation path align with the language instruction.Source: §A.1 · p. 8
• A nav-graph is the scene’s discrete navigation graph.Source: §A.1 · p. 8
• Pano paths use nav-graph nodes corresponding to panoramic-camera positions in R2R.Source: §A.1 · p. 8
• Step paths densely interpolate pano paths with shortest geodesic segments and waypoints spaced for a 0.25 m agent step.Source: §A.1 · p. 8
• “Shortest” denotes the goal-oriented path to the goal.Source: §A.1 · p. 8
• The “goal” model is supervised only with the goal.Source: §A.1 · p. 8

• The analysis uses nDTW to compare the VLN-CE R2R reference path with the shortest path.Source: §A.2 · p. 8
• About 6% of episodes across training and validation have nDTW(shortest, LAW step) < 0.8.Source: §A.2 · p. 8
• The reference-path versus shortest-path nDTW distributions differ, and the language-aligned sensor’s distribution is closer to ground-truth trajectories.Source: §A.2 · p. 8

• CMA encodes RGB observations with ImageNet-pretrained ResNet50, depth with a point-goal-trained modified ResNet50, and instruction tokens with GLoVE embeddings and a bidirectional LSTM.The instruction embeddings are passed through a bidirectional LSTM to obtain word-feature representations.Source: §A.3 · p. 8
• A first GRU encodes agent-state history to generate attended instruction and visual features, while a second GRU uses those features to predict an action.The attention mechanism is scaled dot-product attention.Source: §A.3 · p. 8
• The model uses scaled dot-product attention to generate visual attentions from attended instruction features.Source: §A.3 · p. 8

A.4.1 Metrics

• The standard metrics include trajectory length, navigation error, success and oracle success rates, SPL, nDTW, and SDTW.nDTW measures trajectory match to the ground truth; SDTW applies nDTW only to successful episodes.Source: §A.4.1 · p. 8

A.4.2 Quantitative Results

• Figure 5 compares nDTW distributions across train, validation-seen, and validation-unseen splits, showing many episodes where the shortest path differs from the language-aligned path.Source: Figure 5 · p. 9
• Language-aligned models with varying waypoint density perform similarly, while LAW pano improves over goal-oriented supervision.Source: §A.4.2 · p. 8, Table 3 · p. 9
• On the VLN-CE test split, LAW pano increases OS while matching goal on SR and SPL.Source: §A.4.2 · p. 8, Table 4 · p. 9

A.4.3 Qualitative Analysis

• LAW pano correctly predicts more sub-instructions than goal, and Waypoint Accuracy makes the fraction of correctly predicted waypoints more intuitive to interpret than nDTW.Source: §A.4.3 · p. 9, Table 5 · p. 10

A.5 Mixing goal-oriented and language-oriented losses

• Training with mixed goal-oriented and language-oriented losses does not outperform training with only language-oriented loss.Source: §A.5 · p. 10, Table 6 · p. 11

A.6 Evaluation on VLN-CE RxR dataset

• On the RxR-Habitat English split, LAW methods outperform goal on unseen environments, with LAW step increasing WA by 6% and nDTW by 2%.Source: Table 7 · p. 11, §A.6 · p. 11