Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments

Jacob Krantz; Erik Wijmans; Arjun Majumdar; Dhruv Batra; Stefan Lee

Source-linked AI summary

Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments

Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, Stefan Lee

arXiv:2004.02857v2Apr 6, 2020cs.CVcs.CLcs.RO

TL;DR

Prior vision-and-language navigation commonly assumes fixed topologies, oracle movement, and precise localization, limiting its reflection of real-world robot instruction following. The paper introduces VLN-CE, a continuous 3D task with low-level actions, develops transferred and baseline models, and finds substantially lower performance with significant gaps relative to nav-graph settings. VLN-CE therefore provides a testbed for studying learned navigation together with control and mapping, while transferred trajectories remain subject to reconstruction errors.

Problem
Prior nav-graph navigation assumes known topologies, oracle movement, and precise localization, making it a poor proxy for real-world robotic instruction following.
Method
The paper instantiates VLN in continuous Matterport3D environments using low-level actions, transfers R2R trajectories, and evaluates sequence-to-sequence, cross-modal, and single-modality models.
Results
The best agent successfully navigates to the goal in approximately a third of unseen-environment episodes while averaging 88 actions, and continuous-to-graph comparisons reveal significant performance gaps.
Takeaways & Limitations
VLN-CE provides a more realistic testbed for instruction-following robots and enables integrative studies of learned agents with low-level control and mapping.
Takeaways & Limitations
Transferred nav-graph trajectories can contain reconstruction errors because node locations may be unreachable and meshes or objects may differ from captured panoramas.

Abstract

from arXiv · show

We develop a language-guided navigation task set in a continuous 3D environment where agents must execute low-level actions to follow natural language navigation directions. By being situated in continuous environments, this setting lifts a number of assumptions implicit in prior work that represents environments as a sparse graph of panoramas with edges corresponding to navigability. Specifically, our setting drops the presumptions of known environment topologies, short-range oracle navigation, and perfect agent localization. To contextualize this new task, we develop models that mirror many of the advances made in prior settings as well as single-modality baselines. While some of these techniques transfer, we find significantly lower absolute performance in the continuous setting -- suggesting that performance in prior `navigation-graph' settings may be inflated by the strong implicit assumptions.

1 Introduction

Prior VLN settings simplify navigation through fixed panorama graphs, oracle movement, and precise localization, whereas VLN-CE uses continuous environments and low-level actions. The paper develops models and evaluates transferred techniques, finding substantially harder long-horizon navigation and significant performance gaps between settings.

Problem: Nav-graph VLN represents environments as fixed topologies of panoramic nodes and navigable edges, providing agents prior layout information even in unseen settings.Agents commonly select directions in panoramas and snap to adjacent nav-graph nodes.
Problem: Oracle navigation abstracts away visual control by deterministically traversing meters between nodes, effectively replacing one panorama with another.This omits the continuous observations and obstacle-related movement challenges encountered by real agents.
VLN-CE setting: VLN-CE replaces graph transitions with free movement through continuous 3D environments using low-level actions, without providing the agent’s location or heading.The setting also exposes agents to action-dependent views, obstacle avoidance, and longer horizons averaging approximately 55 actions per path.
Approach: The paper develops sequence-to-sequence and cross-modal attention models, plus input-modality ablations, to test how VLN mechanisms transfer to VLN-CE.The analysis identifies depth as an integral signal for embodied navigation and evaluates single-input baselines.
Results: Approximately one-third of episodes are successfully navigated by the best agent in unseen environments, with an average trajectory of 88 actions.The result reflects performance on the long-horizon VLN-CE task.
Results: Transferring continuous-environment paths back to nav-graphs reveals significant performance gaps, indicating that nav-graphs provide a strong prior and may make prior VLN results overly optimistic.The paper positions VLN-CE as a testbed for studying integrated control and mapping with learned navigation agents.

2 Related Work

Language-guided navigation tasks differ in instruction source, environment realism, and navigation constraints. VLN-CE extends the Matterport3D-based R2R task to continuous navigation, providing crowdsourced instructions in realistic environments with unconstrained agent motion.

Prior tasks: Touchdown and related work study language-guided navigation in Google Street View panoramas, while other settings use automatically generated instructions or unrealistic synthetic scenes.These tasks differ in both instruction construction and environmental realism.
VLN and R2R: VLN provides nav-graph trajectories and crowdsourced instructions in Matterport3D through the Room-to-Room dataset.VLN-CE builds directly on these annotations.
VLN-CE: VLN-CE converts R2R panorama-based trajectories into fine-grained paths in continuous Matterport3D environments with unconstrained navigation.This shift lifts several assumptions associated with nav-graph-based navigation.
Task dimensions: Existing language-guided navigation tasks vary mainly in instruction source, environment realism, and whether navigation is graph-constrained or unconstrained.Table 1 compares tasks along these axes.
Related embodied tasks: Other embodied language tasks specify goals or query knowledge about objects and rooms rather than providing intermediate navigational directions.This loose coupling between language and intermediate actions distinguishes them from language-guided navigation.

3 VLN in Continuous Environments (VLN-CE)

VLN-CE transfers R2R instructions and trajectories into continuous Matterport3D environments, where agents navigate freely using low-level actions. The transfer process addresses mismatched waypoint locations and reconstruction errors, but only 77% of trajectories remain navigable and tasks require much longer action sequences than R2R.

Task and Environment: VLN-CE transfers R2R trajectories and instructions into reconstructed Matterport3D environments using the Habitat simulator.The dataset contains 4475 converted trajectories with natural-language instructions and pre-computed low-level-action paths.
Task and Environment: Agents observe egocentric 256×256 RGB-D images and choose among four low-level actions: move forward 0.25m, turn 15 degrees, or stop.The observation and action spaces emulate a ground-based robot with a forward-mounted camera and zero turning radius.
Trajectory Transfer: Direct projection of nav-graph nodes fails for 73% of nodes, but downward ray casting, navigability checks, and manual corrections transfer 98.3% successfully.Transferred waypoint locations must be occupiable by a 1.5m-tall, 0.2m-diameter ground-based agent, with less than 0.5m displacement.
Trajectory Transfer: A* checks pairwise waypoint reachability, and a trajectory is navigable only when every waypoint can be reached within 0.5m.This procedure identifies whether converted trajectories have valid paths through the reconstructed mesh.
Transferability and Failures: 77% of R2R trajectories transfer successfully; failures arise from invalid nodes or disconnected reconstruction regions caused by mesh and scene discrepancies.Examples include holes in reconstructed meshes, furniture or chairs at waypoint locations, and manually connected nav-graph nodes whose paths are blocked in the reconstruction.
Transferability and Failures: 55.88 low-level actions are required per VLN-CE trajectory on average, compared with 4-6 steps in R2R.Continuous control therefore produces substantially longer-horizon trajectories than panorama-to-panorama navigation.

4 Instruction-guided Navigation Models in VLN-CE

The paper develops sequence-to-sequence and cross-modal attention models for VLN-CE, using egocentric RGB-depth perception and natural-language instructions. It also examines transferred training techniques and finds depth to be an important signal for embodied navigation.

4 Instruction-guided Navigation Models in VLN-CE: The VLN-CE models include a simple sequence-to-sequence baseline and a more powerful cross-modal attentional model, both conceptually related to prior nav-graph VLN systems.These models support comparisons with earlier architectures and tests of whether prior improvements transfer to continuous environments.
4 Instruction-guided Navigation Models in VLN-CE: Both agents represent RGB frames with an ImageNet-pretrained ResNet50 and depth frames with a point-goal-navigation-pretrained ResNet50.RGB features are spatial activations V={v_i}, while depth features are D={d_i}.
4.1 Sequence-to-Sequence Baseline: The baseline recurrent policy combines mean-pooled RGB features, depth features, and an LSTM encoding of the instruction to predict each action.Its straightforward structure enables input-modality ablations and establishes a baseline for VLN-CE.
4.2 Cross-Modal Attention Model: Cross-modal and spatial attention address instructions whose relative references are difficult to ground from mean-pooled features and can focus on instruction content relevant to the next decision.Completed parts of an instruction may be irrelevant to subsequent actions.
4.2 Cross-Modal Attention Model: The cross-modal model uses one recurrent network to track visual observations and another to make decisions from attended instruction and visual features.The decision network also receives an action encoding and the first recurrent network’s hidden state.
4.2 Cross-Modal Attention Model: The attentional architecture encodes instructions with a bidirectional LSTM, computes attended instruction features, and then uses them to attend over visual and depth features.The attention operation is specified as scaled dot-product attention.
4.3 Auxiliary Losses and Training Regimes: Training experiments transfer auxiliary losses, exposure-bias mitigation, and synthetic data augmentation strategies from prior VLN work, without claiming an exhaustive survey of effective techniques.The paper specifically applies DAgger and converts approximately 150k synthetic trajectories to continuous environments.
4.3 Auxiliary Losses and Training Regimes: Imitation learning uses teacher forcing with inflection weighting, while a progress monitor predicts the fraction of the trajectory completed using a mean squared error loss.Inflection weighting emphasizes time steps where the action changes, and progress estimation supervises stopping-related behavior.

5 Experiments

Experiments evaluate VLN-CE baselines, modality ablations, model architectures, training augmentations, and qualitative behavior. Performance remains challenging in unseen environments, with depth and cross-modal attention providing notable benefits while nav-graph comparisons expose large gaps.

Setting and Metrics: The evaluation reports TL, NE, OS, SR, SPL, and nDTW, emphasizing SR and SPL for performance and nDTW for trajectory-shape differences.Models are trained and early-stopped using val-unseen performance.
No-Learning Baselines: Random and hand-crafted agents achieve approximately 3% success in val-unseen, reflecting the difficulty of long continuous trajectories.The hand-crafted agent samples a heading, moves forward for 37 actions, and then stops.
Seq2Seq and Single-Modality Ablations: 20% success: the baseline Seq2Seq model significantly outperforms random and hand-crafted baselines in val-unseen.All models in this comparison use imitation learning without data augmentation or auxiliary losses.
Seq2Seq and Single-Modality Ablations: Depth is essential for learning: No Depth and No Vision models fail to outperform chance, with success rates of ≤1%.No Instruction and No Image each achieve 17% success, indicating different limitations for instruction and visual grounding.
Cross-Modal Attention vs. Seq2Seq: The Cross-Modal model improves SPL over Seq2Seq under all tested settings, including from 0.22 to 0.30 SPL with all three augmentations.This is an improvement of 0.08 SPL, or 36% relative.
Training Augmentation: DAgger improves both model types by 0.03–0.05 SPL, while progress monitoring or data augmentation alone can reduce SPL by 0.01–0.03.Combining all three techniques yields 0.35 SPL on val-seen, 0.30 SPL on val-unseen, and success in 32% of new-environment episodes.
Nav-Graph Comparison: On the VLN test set, the model achieves 0.21 SPL versus 0.38 for RCM and near 0.47 for the state of the art.The comparison highlights a substantial gap between continuous-environment performance and nav-graph-based results.

6 Discussion

The discussion frames VLN-CE as a continuous, low-level-action testbed that removes unrealistic nav-graph assumptions. It also identifies modular control architectures as an important direction for future work.

6 Discussion: VLN-CE studies instruction following with low-level actions in continuous environments, lifting unrealistic assumptions from prior nav-graph settings.The task enables experiments at the interface between high-level language-guided decisions and low-level control.
6 Discussion: The presented agents map observations directly to low-level control end to end, while modular approaches such as learned directives to a motion controller remain future work.The authors describe modular control as an exciting direction rather than an evaluated component.

Supplementary

The supplementary material describes converting continuous VLN-CE paths into nav-graph trajectories by iteratively selecting nearby graph nodes. The resulting representations can poorly match paths through sparsely covered 3D space.

Path Conversion: The conversion starts from the VLN start node and seeks a nav-graph path that follows the sequence of continuous positions.The initial continuous position aligns with the start node.
Path Conversion: At each timestep, the algorithm selects the nearest node among the current node and its adjacent navigable nodes.The current node and one-step neighborhood are updated repeatedly for p2 through pT.
Path Representation: Nav-graph trajectories can have high error or oscillate because continuous agents often traverse spaces absent from the sparse graph.Figure 6 contrasts continuous paths with their graph conversions and shows the graph’s limited coverage of continuous motion.