Source-linked AI summary
Waypoint Models for Instruction-guided Navigation in Continuous Environments
Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, Oleksandr Maksymets
TL;DR
The paper examines how action-space expressivity affects navigation success and execution efficiency in continuous language-guided navigation. It develops language-conditioned waypoint models across action spaces, evaluates task performance and estimated LoCoBot execution time, and finds a trade-off between faster trajectories and higher navigation metrics.
Problem
Prior work has limited evidence about how action-space design affects navigation success and the efficiency of executing language-guided trajectories.
Method
The authors develop language-conditioned waypoint prediction networks with varied expressivity and evaluate them using navigation metrics and estimated execution time from a profiled LoCoBot.
Results
More expressive models produce simpler, faster trajectories, while constrained action spaces achieve higher navigation metrics; the best submission improves success by 4% and SPL by 5 points over prior work.
Takeaways & Limitations
Waypoint models offer favorable real-world execution properties, whereas less expressive action spaces can better approximate shortest paths and improve traditional VLN metrics.
Takeaways & Limitations
Performance differences of 1 SPL may not be significant because of variance associated with reinforcement-learning training methods.
Abstract
from arXiv · showhide
Little inquiry has explicitly addressed the role of action spaces in language-guided visual navigation -- either in terms of its effect on navigation success or the efficiency with which a robotic agent could execute the resulting trajectory. Building on the recently released VLN-CE setting for instruction following in continuous environments, we develop a class of language-conditioned waypoint prediction networks to examine this question. We vary the expressivity of these models to explore a spectrum between low-level actions and continuous waypoint prediction. We measure task performance and estimated execution time on a profiled LoCoBot robot. We find more expressive models result in simpler, faster to execute trajectories, but lower-level actions can achieve better navigation metrics by approximating shortest paths better. Further, our models outperform prior work in VLN-CE and set a new state-of-the-art on the public leaderboard -- increasing success rate by 4% with our best model on this challenging task.
1. Introduction
The paper studies how waypoint action-space expressivity affects instruction-guided navigation in continuous environments, balancing navigation quality with robotic execution efficiency. It introduces language-conditioned waypoint models and evaluates their trajectories and leaderboard performance in VLN-CE.
- Motivation: VLN-CE exposes the limitations of end-to-end low-level control, which requires agents to learn language grounding and navigation over roughly 55 actions per episode.Prior models mirroring successful VLN agents perform substantially worse in VLN-CE.
- Motivation: Frequent low-level stops, starts, and turns can slow real-robot execution, increase state-estimation error, strain hardware, and demand additional power for repeated policy inference.These concerns motivate examining action spaces beyond individual low-level actions.
- Approach: The proposed models explore waypoint action spaces from continuous relative points to fixed directional steps, including mixed discrete-continuous parameterizations.This spectrum varies the expressivity of predicted waypoints while retaining low-level navigation toward them.
- Approach: Given language and panoramic RGBD observations, the waypoint network predicts polar-coordinate waypoints, followed by a low-level continuous navigator that moves toward each waypoint.The waypoint predictor handles obstacle-avoidance concerns, while the navigator executes the resulting straight-line movement.
- Findings: More expressive models produce simpler, faster trajectories, whereas constrained action spaces can improve navigation by approximating shortest paths more closely.This establishes a trade-off between execution efficiency and standard navigation performance.
- Findings: Our models set a new VLN-CE leaderboard state of the art, increasing success rate by an absolute 4%.The contribution set includes waypoint prediction networks, empirical expressivity analysis, and low-level navigation comparisons.
2. Related Work
Related work spans instruction-guided navigation across languages, environments, observability, and action spaces. This paper focuses on VLN-CE and connects waypoint prediction with hierarchical navigation and sim-to-real transfer.
- Instruction-Guided Navigation: Instruction-guided navigation research varies by instruction source and language, environment setting and realism, navigation affordance, and agent embodiment.These dimensions range from templated or multilingual instructions to continuous real-world navigation.
- Instruction-Guided Navigation: VLN uses natural-language instructions and sparse Matterport3D navigation graphs, while VLN-CE lifts the task into continuous 3D environments.The paper studies waypoint models with more abstract action spaces in VLN-CE.
- Hierarchical Visual Navigation: Waypoint-based agents form a hierarchical design in which high-level waypoint selection is paired with low-level navigation.The paper combines waypoint prediction and selection while conditioning both on the task goal.
- Hierarchical Visual Navigation: Prior hierarchical methods make assumptions about observability or vehicle affordances that limit direct transfer to indoor, partially observable ground-based navigation.Examples include outdoor methods relying on near-full observability or aerial free-space.
- Sim-to-Real Transfer: Sim-to-real VLN work has used instruction-independent waypoint proposals to emulate online navigation graphs, but performance drops relative to known graphs, identifying waypoint prediction as a transfer bottleneck.The paper instead conditions waypoint prediction on the navigation instruction.
3. Task Description
VLN-CE asks an agent to follow natural-language instructions to a goal in unseen continuous 3D environments using egocentric RGBD perception. The paper extends the visual input panoramically and separates waypoint selection from low-level action execution.
- VLN-CE Task: The task requires reaching an instruction-specified goal in a previously unseen environment while navigating continuous 3D space with obstacles and occlusion.The agent receives egocentric RGBD observations and must stop at the goal location.
- Perception: The visual setup extends VLN-CE to 360° panoramic observations composed of twelve 30°-offset frames, each with a 90° field of view.Each frame has 256 x 256 resolution.
- Action Space: Waypoint-based agents can remain independent of the low-level action space used to reach predicted waypoints.This separation enables comparison across different execution action spaces.
4. Method
The waypoint prediction network maps natural-language instructions and panoramic RGBD observations to relative waypoints or STOP actions, using cross-modal attention, visual history, and configurable action distributions. Training uses a PPO-based objective with exploration, value, progress, success, slack, and unreachable-waypoint considerations.
- Waypoint Prediction Network: The waypoint prediction network predicts relative waypoints or STOP directly from pixels and natural language, then a lower-level navigator turns and moves toward each waypoint.Separating language processing from waypoint execution lets the navigator operate using relative polar coordinates.
- Waypoint Prediction Network: At each step, panoramic RGBD observations provide 12 views, from which the model selects a coarse heading and predicts an angular offset and distance.The waypoint is represented in relative polar coordinates using the selected heading, refined offset, and predicted distance.
- Cross-Modal Attention: Two levels of cross-modal attention combine visual history, instruction features, spatial frames, pose features, and panorama views before recurrent action prediction.The architecture includes instruction-to-history attention, spatial attention over RGBD frames, and attention across panorama frames.
- Visual History: The model incorporates previous waypoint actions and visual context from the prior timestep to handle larger observation changes caused by waypoint-based movement.Previous visual features are selected near the prior waypoint heading and mean-pooled into a context vector.
- Action Prediction: Waypoint expressivity varies by making offset and distance predictions continuous, discrete, or constant within specified angular and distance domains.Continuous predictions use truncated Gaussians; constant predictions reproduce the 0.25m forward step of the standard VLN-CE action space.
5. Experiments
Experiments examine how waypoint-action expressivity affects conventional navigation performance and practical execution efficiency in VLN-CE. Less expressive models generally improve standard success metrics, while waypoint models produce faster trajectories and achieve strong leaderboard performance.
- Impact of Waypoint Expressivity: 3-8% success separates continuous-offset models from discrete or fixed-offset counterparts, showing that offset expressivity materially affects navigation performance.Continuous offsets provide more positional control at longer distances.
- Impact of Waypoint Expressivity: 2-3% higher success favors fixed-distance HPNs over corresponding WPNs, while approximately 4x more actions improve path approximation and SPL.WPNs instead use straight-line segments averaging 1.6m, reducing starts, stops, and turns.
- Path Efficiency under a LoCoBot Motion Model: 144 seconds separates the best WPN from the best HPN in estimated execution time, nearly a 2x reduction for the WPN.Models predicting travel distance have lower EET than models stepping in fixed 0.25m increments.
- Path Efficiency under a LoCoBot Motion Model: 6.9 cm/s versus 2.6 cm/s gives the best WPN a 2.7x higher estimated average speed than the best HPN.The WPN also achieves SCT of 23 versus 11 despite lower SPL.
- Comparison with Discrete Action Models: 4 SR and 5 SPL above prior work establish HPN+DN as the new VLN-CE state of the art, despite evaluation with an unseen navigator.The reported gains are 14% relative for SR and 20% relative for SPL.
- Comparison with Discrete Action Models: 300M training steps failed to bring the matched prior observation-and-action-space agent to convergence, identifying its longer time horizon as a challenge.This ablation was intended to isolate differences between HPN+DN and prior work.
6. Discussion
The paper finds a trade-off between execution efficiency and traditional navigation metrics: expressive waypoint models produce faster trajectories, while constrained actions can score better on VLN metrics. Its best submission improves leaderboard success and SPL over prior work, but a gap remains between topological VLN and continuous VLN-CE.
- Discussion: More expressive waypoint models have favorable real-world execution properties, including an approximately 2x reduction in expected execution time.The paper also emphasizes their modular architecture for interfacing with robot-specific navigation stacks.
- Discussion: Less expressive action spaces achieve higher traditional VLN metrics by better approximating shortest paths.
- Discussion: The best submission improves success by 4% and SPL by 5 points over prior work, corresponding to 14% and 20% relative improvements.
- Discussion: A significant gap remains between topological VLN and continuous VLN-CE, alongside a related sim2real gap.The paper identifies an effective interface between language understanding and robotic control as an outstanding need.
7. Supplementary
The supplementary material details reinforcement-learning settings, LoCoBot motion-model construction, and the completion-time metric used to evaluate trajectories. It also describes adapting the metric’s oracle time to the robot’s dynamics, action space, and multi-floor environments.
- Hyperparameter Details: A slack reward scalar of -0.05 reduces reward variance, accelerates convergence, and produces more efficient paths compared with the default -0.01.
- Hyperparameter Details: The discount factor γ = 0.99 balances immediate gains from larger steps with long-term success.Setting γ to 1 produces HPN-style actions, whereas γ = 0 or 0.5 encourages maximum-distance steps toward the goal.
- Constructing a LoCoBot Motion Model: LoCoBot execution time is estimated from rotation and translation functions fitted using repeated empirical timings for profiled base controllers.The resulting motion model supports both continuous and discrete navigators, and analogous models can be computed for other robots.
- Adapting Success Weighted by Completion Time: Success weighted by Completion Time scales binary success by the trajectory’s relative completion time, using estimated completion time C and oracle time T.The oracle-time computation adapts RRT* to LoCoBot dynamics, the model’s action space, and multi-floor navigation.