Source-linked AI summary
Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation
Yicong Hong, Zun Wang, Qi Wu, Stephen Gould
TL;DR
VLN agents trained for discrete or continuous environments do not readily generalize across the two settings because discrete navigation relies on connectivity graphs and high-level actions. The paper predicts accessible waypoints from refined graphs, augments them during training, and reports state-of-the-art results on R2R-CE and RxR-CE while reducing the discrete-to-continuous gap.
Problem
Discrete and continuous VLN use different action and accessibility assumptions, limiting transfer between agents designed for the two environments.
Method
A candidate waypoints predictor uses Habitat-Matterport3D graphs refined from Matterport3D to generate accessible waypoints, with waypoint augmentation during training.
Results
The method achieves new state-of-the-art results on R2R-CE and RxR-CE, including 39% and 19.61% SPL on their test sets.
Takeaways & Limitations
Predicted waypoints enable discrete-style agents to navigate continuous environments with high-level actions and generalize across agents and unseen environments.
Takeaways & Limitations
The predictor can miss directions at rare structures such as stairs because training samples for those structures are insufficient.
Abstract
from arXiv · showhide
Most existing works in vision-and-language navigation (VLN) focus on either discrete or continuous environments, training agents that cannot generalize across the two. The fundamental difference between the two setups is that discrete navigation assumes prior knowledge of the connectivity graph of the environment, so that the agent can effectively transfer the problem of navigation with low-level controls to jumping from node to node with high-level actions by grounding to an image of a navigable direction. To bridge the discrete-to-continuous gap, we propose a predictor to generate a set of candidate waypoints during navigation, so that agents designed with high-level actions can be transferred to and trained in continuous environments. We refine the connectivity graph of Matterport3D to fit the continuous Habitat-Matterport3D, and train the waypoints predictor with the refined graphs to produce accessible waypoints at each time step. Moreover, we demonstrate that the predicted waypoints can be augmented during training to diversify the views and paths, and therefore enhance agent's generalization ability. Through extensive experiments we show that agents navigating in continuous environments with predicted waypoints perform significantly better than agents using low-level actions, which reduces the absolute discrete-to-continuous gap by 11.76% Success Weighted by Path Length (SPL) for the Cross-Modal Matching Agent and 18.24% SPL for the Recurrent VLN-BERT. Our agents, trained with a simple imitation learning objective, outperform previous methods by a large margin, achieving new state-of-the-art results on the testing environments of the R2R-CE and the RxR-CE datasets.
1. Introduction
VLN research has largely separated discrete and continuous navigation because discrete agents rely on connectivity graphs and high-level actions, whereas continuous agents must infer accessibility. This paper bridges the domains with predicted waypoints that let discrete-style agents navigate continuous environments.
- Motivation: Discrete and continuous VLN differ because discrete agents rely on connectivity graphs and high-level actions, while continuous agents infer accessibility through low-level controls.This separation has limited direct transfer of advances such as visiolinguistic transformers and scene memory to continuous navigation.
- Approach: The paper adapts the Matterport3D graph to Habitat-Matterport3D and trains a candidate waypoints predictor to estimate accessible locations during navigation.The predictor constructs a local navigability graph from visual observations and supports high-level action selection in continuous spaces.
- Approach: Waypoint positions are augmented during training to expose agents to diverse observations and step lengths, improving their generalization ability.The augmentation moves waypoint positions while preserving the same target.
- Results: 39% and 19.61% SPL are achieved on the R2R-CE and RxR-CE test sets, respectively, with predicted waypoints significantly improving continuous-environment navigation.The experiments use CMA and Recurrent VLN-BERT, demonstrating transfer across distinct navigator architectures.
2. Related Work
Related work established graph-based high-level navigation and explored continuous VLN and hierarchical visual navigation separately. The paper positions its approach as a decoupled waypoint-prediction module that supplies accessible directions without redesigning the navigator.
- Vision-and-Language Navigation: Graph-based VLN uses panoramic actions to teleport agents among adjacent Matterport3D waypoints by selecting images aligned with navigable directions.Subsequent work extended this setup through cross-modal learning, data augmentation, waypoint tracking, and Transformer pre-training.
- Continuous VLN: Continuous VLN requires agents to infer spatial accessibility, unlike graph-based methods that receive navigable directions from a predefined connectivity graph.This accessibility requirement makes graph-trained agents inapplicable without adaptation.
- Hierarchical Visual Navigation: Hierarchical visual navigation combines mapping, planning, and control, including methods that construct topological maps or predict sub-goals.The paper distinguishes its method by estimating key positions around the agent and decoupling waypoint prediction from decision making.
- Hierarchical Visual Navigation: Unlike Waypoint Models, the proposed method provides accessible directions without modifying network architectures or training methods for the navigator.This design uses learned navigability as an interface between continuous environments and discrete-style agents.
3. Background
The background contrasts low-level control in continuous environments with graph-based high-level actions, emphasizing view selection and waypoint teleportation as the key advantages of connectivity graphs. Experiments then isolate how navigable directions and movement distance affect performance.
- Navigation Setups: Continuous VLN agents choose low-level controls from egocentric observations, while graph-based agents select among navigable directions and teleport between connected waypoints.Low-level episodes are about 10 times longer than high-level episodes, making training more expensive.
- High-Level Actions: View selection transfers control inference into selecting a navigable direction, while waypoint teleportation moves the agent directly to a distant graph waypoint.Together, these mechanisms expand the decision space and reduce the number of navigation decisions.
- Connectivity Graphs: Connectivity graphs encode sparse keypoints and directional information that support future navigation decisions, rather than representing every explorable position.Figure 3 distinguishes structurally important keypoints from general navigability.
- Contrastive Experiments: Knowing navigable directions brings success rate close to graph navigation, and teleportation reduces execution time while slightly increasing success rate.The experiments identify navigability as the main bottleneck when the connectivity graph is unavailable.
- Contrastive Experiments: View selection significantly improves performance across fixed step lengths, while larger step lengths improve efficiency by reducing the number of decisions.An appropriate distance such as 1 meter can perform well, but the best value is unknown beforehand.
4. Candidate Waypoints Predictor
The candidate waypoints predictor infers accessible positions in continuous environments, creating a local sub-graph that lets agents use high-level actions. It processes panoramic RGBD observations with visual encoders, a Transformer, and a heatmap-based waypoint classifier.
- The predictor generates virtual waypoints and a local sub-graph of edges from the agent toward accessible positions.This enables high-level navigation actions in continuous spaces.
- Network Architecture and Processing: Panoramic RGB and depth observations are encoded and merged before Transformer processing models spatial relationships between views.The visual encoders use separate ResNet-50 models for RGB and depth, while the Transformer contains two layers with 12 self-attention heads.
- Network Architecture and Processing: The classifier predicts waypoint probabilities over a 120-by-12 heatmap spanning 120 angles and distances from 0.25 to 3.00 meters.Non-maximum suppression converts the heatmap into K neighboring waypoints.
- Training Data: Training adapts the Matterport3D connectivity graph to continuous Habitat-MP3D environments before constructing waypoint supervision.The adapted graph represents partial accessibility, with nodes concentrated at sparse keypoints and edges indicating directions worth exploring.
DDPPO Res-50
This section presents the waypoint-predictor setup and graph statistics used to compare predicted waypoints with ground-truth connectivity. Training targets are represented as localized heatmaps, while predictor quality is assessed by count, accessibility, and point-distance metrics.
- The waypoint predictor maps RGBD panoramic inputs to waypoint positions in the agent’s neighborhood.The associated figure identifies the predictor as the module used to generate candidate waypoints.
- The connectivity-graph statistics compare MP3D with the refined Habitat-MP3D environment graphs.The supplied passage reports average node connectivity of 3.31 (4.07) edges and average edge lengths of 1.87 (2.26) meters for the compared environments.
- Ground-truth waypoint targets are encoded as a 120-by-12 heatmap over 3-meter neighborhoods divided into angular sectors and distance rings.Each waypoint is represented by a Gaussian distribution with variance of 1.75m and 15°.
- Predictor evaluation uses |∆|, %Open, Chamfer distance, and Hausdorff distance to measure count difference, open-space accessibility, and point-set distances.The passage reports that U-Net predictions are closest to ground truth but are often blocked by obstacles.
5. Bridging the Discrete to Continuous Gap
The experiments test whether predicted and augmented waypoints can transfer graph-trained agents to continuous navigation. Across CMA and VLN-BERT, waypoint-based training narrows the graph-to-continuous gap and improves benchmark performance over prior methods.
- Bridge the Gap: Ground-truth graph navigation scores 8.92% and 18.69% higher SPL than low-level actions for CMA and VLN-BERT, respectively.The comparison quantifies the discrete-to-continuous challenge before introducing predicted waypoints.
- Bridge the Gap: Training on predicted waypoints brings agents to performance similar to navigation on ground-truth graphs, with VLN-BERT reducing SR and SPL gaps by 77% and 90%.The predictor also reduces obstacle collisions from about 15% to 7% in the reported fixed-step experiment.
- Waypoint Augmentation: Waypoint augmentation samples new positions from the selected-view heatmap, diversifying views, step lengths, and obstacle interactions during training.Augmented-waypoint training further improves performance, and CMA’s unseen-environment SPL exceeds graph-based CMA.
- Comparison to SoTA: The method outperforms previous approaches across R2R-CE and RxR-CE dataset splits using predicted waypoints.On R2R-CE test, CMA improves over Waypoint Models by 6% SR and 3% SPL while using single-GPU imitation learning; on RxR-CE, both agents exceed the previous best by more than 10% SR and 6% nDTW.
6. Conclusion
The paper presents candidate waypoint prediction as a bridge from discrete VLN to continuous navigation. It reports generalization across agents and unseen environments, with applications beyond VLN proposed as future directions.
- The candidate waypoints predictor produces accessible waypoints, enabling discrete-environment agents to navigate continuous environments with high-level actions.
- Experiments show generalization across different agents and unseen environments, substantially reducing the discrete-to-continuous gap.
- The method achieves state-of-the-art performance on the R2R-CE and RxR-CE datasets.
- Future work includes weakly supervised and language-conditioned waypoint prediction, as well as applications to other embodied navigation and task-completion problems.
Appendices
The appendices describe how Matterport3D connectivity graphs are projected, corrected, merged, and fitted to Habitat-Matterport3D and continuous trajectories. They also document graph-based trajectory filtering and comparable performance on unseen environments.
- Graph construction: Matterport3D graphs are projected into Habitat using trajectory-derived node positions, then corrected to remove obstacle violations and preserve navigability.The construction criteria require obstacle-free nodes, few added nodes, straight edges, and merging of edges shorter than 0.25 meters.
- Graph construction: 264 candidate positions within a 0.35m-radius circle are sampled for each node requiring adjustment and scored using graph-quality measurements.
- Graph construction: Invalid edges are repaired with Habitat shortest-path detours, while nearby nodes are merged repeatedly until the graph is navigable with edges at least 0.5 meters long.
- Trajectory fitting: Trajectory endpoints are fitted to graph nodes within 0.8 meters, adding connecting nodes when necessary.
- Trajectory processing: The filtered dataset contains 10,755 train, 1,755 validation-seen, and 745 validation-unseen episodes, with 6.63 average nodes per trajectory.
- Evaluation: Agents achieve similar performance on Habitat-Matterport3D graphs despite lower-quality images and more complicated graphs, including in unseen environments.
B.1. Architecture (§5.1)
The architecture processes RGB, depth, language, and directional features for candidate waypoints, using either CMA or VLN-BERT to select a navigable view. The predictor operates on a 2D-planar graph.
- Visual and waypoint features: RGB and depth encoders process candidate images at waypoint directions, and their outputs feed both the waypoint predictor and navigation policies.The encoders are frozen while the navigators are trained.
- Visual and waypoint features: Candidate RGB and depth representations are merged after learnable projections before entering the policy networks.The representations correspond to the directions containing predicted waypoints.
- Directional encoding: Relative candidate-view directions are explicitly encoded using the view heading angle relative to the agent’s orientation.
- Directional encoding: The waypoint predictor creates a 2D-planar graph, so directional encodings omit elevation; predicting 3D waypoints is identified as future work.
- Language encoding: CMA uses a bidirectional LSTM for instructions and a zero initial state, whereas VLN-BERT uses BERT representations and the [CLS] output as its initial state.
- Navigation policies: CMA and VLN-BERT use attention-based visual-language policies, and inference selects the candidate view with the greatest probability.
B.2. Training (§5.1)
Training uses imitation learning with waypoint-based oracle actions, with a specialized oracle for RxR-CE. The appendices also report rare predictor failures, multilingual training costs, evaluation metrics, and trajectory visualizations.
- Training objective: All agents are trained with imitation learning using cross-entropy loss on predicted action probabilities and oracle actions.
- Oracle actions: For the standard setting, the oracle selects the predicted waypoint with shortest geodesic distance to the target and stops within 1.5 meters.
- Training limitations: In rare cases, the predictor produces no waypoint closer to the target; an unseen-environment test found that only 2% of agents failed to reach within 3 meters.
- Oracle actions: For RxR-CE, the oracle uses a sub-goal defined by the intersection of a 3-meter ring and the ground-truth path.
- Training setup: RxR-CE training combines results from three language-specific models initialized with multilingual BERT features.
- Visualization: Predicted waypoints usually occupy accessible spaces and reduce the number of decisions needed, but predictor failures can block exploration and inhibit navigation.
D. Simulator Configurations (§5.1)
The experiments use distinct Habitat configurations for R2R-CE and RxR-CE, including different agent dimensions and sliding behavior. RxR-CE’s non-sliding setting increases difficulty, while enabling sliding improves both evaluated agents’ results.
- Environment and agent settings: R2R-CE and RxR-CE use different standard agent dimensions for fair comparison with prior work.The RxR-CE agent is shorter and can access locations that the R2R-CE agent cannot.
- Turning angles: 30° turning angles in official RxR-CE configurations can prevent access to some navigable directions and create zigzag ground-truth paths.The reported inflection coefficient is 1.9 for RxR-CE trajectories, compared with about 3.2 in R2R-CE using 15° turning angles.
- Sliding comparison: Enabling sliding significantly improves results for both the CMA and VLN-BERT agents.The comparison is reported in Table 8, which evaluates performance with and without sliding.
E. Limitations (§4.1 & §5.2)
The proposed candidate waypoints predictor has several documented limitations, including a fixed candidate budget, failures at rare structures, reliance on unadjusted predictions, uncertain out-of-domain transfer, and separation from navigation state.
- Number of candidates: The predictor limits each position to at most five waypoints, although some spatial structures have more navigable directions.The authors suggest dynamically varying the number of waypoints by position.
- Rare structures: At rare structures such as stairs, insufficient training samples can prevent the predictor from producing a waypoint on the stairs and inhibit navigation.The proposed direction for improvement is targeted sampling and a loss function that balances learning.
- Online adjustment: Because the agent fully trusts predicted waypoints, prediction errors can cause deadlocks, especially in RxR-CE.The authors propose adjusting waypoints using local structure or control outcomes.
- Out-of-domain transfer: Although accurate on unseen MP3D environments, the predictor’s transferability to distinct out-of-domain scenes remains unclear.The authors identify adaptation to new environments without predefined connectivity graphs as a future direction.
- State conditioning: Decoupling waypoint prediction from decision making reduces the agent’s state space but excludes navigation progress and instruction landmarks from prediction.The authors note that incorporating this state information could produce more effective waypoints for reaching the target.