Source-linked AI summary
Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, Anton van den Hengel
TL;DR
Robots still face the challenge of linking natural language to vision and action in previously unseen real environments. The paper introduces the Matterport3D Simulator and R2R benchmark, applying vision-and-language methods to embodied navigation. The results show that sequence-to-sequence agents improve over random behavior, while generalization to unseen environments remains difficult.
Problem
VLN addresses the need to link natural language, vision, and action when robots navigate in unstructured, previously unseen environments.
Method
The paper presents a real-image Matterport3D reinforcement-learning simulator and the R2R benchmark for building-scale natural-language navigation.
Results
Sequence-to-sequence agents improve over RANDOM, achieving a 20.4% test-set success rate with student-forcing, while unseen validation success is 21.8% versus 38.6% in seen environments.
Takeaways & Limitations
VLN methods can be successfully applied, but generalizing to previously unseen environments is a significant challenge.
Takeaways & Limitations
Matterport3D scenes are biased toward clean, luxurious spaces with few people or animals, and viewpoints often provide commanding views unlike typical robot positions.
Abstract
from arXiv · showhide
A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visually-grounded navigation instructions, we present the Matterport3D Simulator -- a large-scale reinforcement learning environment based on real imagery. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings -- the Room-to-Room (R2R) dataset.
1. Introduction
Vision-and-Language Navigation links natural language, visual perception, and action in previously unseen real environments. The paper introduces a realistic simulator and the R2R benchmark to enable reproducible study of this open-set problem.
- VLN requires linking natural language to vision and action in unstructured, previously unseen environments.
- VLN and VQA are visually grounded sequence-to-sequence tasks, but VLN produces longer action sequences that manipulate the camera viewpoint.
- Unlike approaches using rendered images or labels, VLN interprets open-vocabulary commands against imagery from previously unseen real environments.This preserves the open-set nature of relating real language to real imagery.
- The Matterport3D Simulator is a large-scale interactive reinforcement-learning environment built from 10,800 panoramic RGB-D images across 90 real-world buildings.Using real images preserves visual and linguistic richness compared with synthetic environments.
- R2R contains 21,567 open-vocabulary, crowd-sourced instructions averaging 29 words, each describing trajectories through typically multiple rooms.The task asks agents to navigate to goal locations in previously unseen buildings.
- The paper introduces the simulator, presents R2R as the first real-building VLN benchmark, and applies sequence-to-sequence neural networks to establish baselines.
2. Related Work
Prior navigation work often simplifies visual perception or language, while vision-and-language benchmarks generally do not let agents move or control cameras. The paper addresses this gap with navigation in visually and linguistically complex real environments.
- Navigation and language: Many navigation approaches assume enumerated goals or objects identified by labels, or operate in visually restricted environments requiring limited perception.
- Vision and language: Existing vision-and-language benchmarks support end-to-end learning from natural images but do not allow agents to move or control the camera.
- Navigation based simulators: Compared with synthetic 3D reinforcement-learning environments, the simulator uses natural images of real scenes, preserving unique visual diversity and richness.
- RL in navigation: Earlier language-based navigation reinforcement-learning studies used visually or linguistically less complex settings, such as template instructions in Doom or a fully observable blocks world.
3. Matterport3D Simulator
The Matterport3D Simulator provides a large-scale visual RL environment built from real building imagery, with graph-constrained navigation and configurable observations. Its design supports embodied-agent research while retaining substantial scene diversity, although the underlying scans exhibit selection biases.
- Environment: The simulator is a large-scale visual reinforcement-learning environment based on Matterport3D panoramic views of diverse real building-scale scenes.The source dataset includes 10,800 panoramic views from 90 environments spanning houses, apartments, hotels, offices, and churches.
- Environment: Agents move among panoramic viewpoints represented by 3D position, heading, and camera elevation, receiving first-person RGB observations.The simulator generates perspective views from precomputed cube-mapped images and can be extended with depth and segmentation observations.
- Action space: At each step, agents select a visible reachable viewpoint and camera adjustments, or remain at the current viewpoint while changing the camera.Reachability is constrained by the current camera frustum, while actions remain deterministic.
- Action space: A weighted, undirected navigation graph restricts movement to robot-navigable viewpoint transitions detected from scene geometry and manually verified for missed obstacles.Edges encode straight-line distance, edges longer than 5m are removed, and manual checks correct obstacles such as windows and mirrors.
- Design scope: The simulator leaves goals, rewards, and contextual inputs task-dependent, enabling the same environment to support different datasets and embodied tasks.Python bindings and configurable image parameters facilitate integration with learning and RL frameworks.
- Limitations: The Matterport3D scans overrepresent clean, luxurious spaces, contain few people or animals, and often place viewpoints where robots may not naturally stand.Additional building scans can alleviate these biases to some extent.
4. Room-to-Room (R2R) Navigation
Room-to-Room defines navigation as following natural-language instructions from a start pose to a goal in a previously unseen building. It pairs crowd-sourced directions with graph-based trajectories and evaluates both destination accuracy and the requirement to stop.
- 4.1. Task: The R2R task gives an embodied agent a natural-language instruction and an initial RGB observation, requiring navigation from a start pose to a goal location.The episode ends when the agent selects a special stop action and succeeds when it reaches the goal closely enough.
- 4.2. Data Collection: The dataset contains 7,189 sampled paths, predominantly connecting different rooms, with average path length 10m and four-to-six graph edges.Paths shorter than 5m or outside the edge-count range were discarded.
- 4.2. Data Collection: Each path receives three instructions collected through an interactive 3D WebGL interface that lets workers inspect the trajectory from multiple viewpoints.Workers write directions intended to guide a robot from the shared starting location to the goal.
- 4.3. R2R Dataset Analysis: The resulting dataset contains 21,567 open-vocabulary instructions averaging 29 words, with a constrained vocabulary of around 3.1k words and widely varying abstraction levels.The variation in instruction abstraction is identified as an important task-handling issue.
- 4.4. Evaluation Protocol: Navigation error is the shortest graph distance between the final position and goal, with success defined as error below 3m rather than matching the entire trajectory.This reflects that instructions may not specify a unique path.
- 4.4. Evaluation Protocol: The evaluation requires agents to recognize the goal and explicitly stop, making stopping part of successful task completion.The paper notes that this differs from vision-only navigation settings that do not train agents to stop.
5. Vision-and-Language Navigation Agents
The paper evaluates sequence-to-sequence navigation agents using LSTM encoders and decoders with instruction attention, alongside learning-free and human baselines. It compares teacher-forcing and student-forcing training for navigation in the R2R environment.
- Sequence-to-Sequence Model: The main learned agent is an LSTM sequence-to-sequence policy that encodes instructions, observes images and previous actions, and uses attention to predict actions.The encoder processes word embeddings, while the decoder combines visual and action features with attended language representations.
- Model action space: The simplified action space contains six actions: left, right, up, down, forward, and stop.Forward moves to the reachable viewpoint closest to the center of the visual field.
- Training Regimes: The models use cross-entropy loss against the next action on the ground-truth shortest-path trajectory.The target action is defined from the agent’s current pose toward the target location.
- Training Regimes: Teacher-forcing conditions each prediction on the ground-truth previous action, whereas student-forcing samples the next action from the model’s output distribution.Student-forcing addresses the changing input distribution between training and testing caused by teacher-forcing.
- Training Regimes: Scheduled sampling performed worse than student-forcing in initial experiments.The task permits determining the shortest trajectory to the goal from arbitrary preceding states.
- Baselines: The evaluation includes RANDOM and SHORTEST learning-free baselines and human trajectories collected for 1,390 test instructions.Human workers freely navigated the scenes in first person while following the provided instructions.
6. Results
Student-forcing outperformed teacher-forcing and achieved a 20.4% test success rate, but performance was substantially better in seen than unseen environments. The results indicate strong overfitting and difficulty generalizing navigation behavior to new buildings.
- Human Performance: People sometimes reduce navigation ambiguity with gestures and dialog, mechanisms unavailable in this evaluation.Right-left confusion can be recoverable when instructions contain sufficient visually grounded references.
- Results: 20.4% success was achieved on the test set by the student-forcing sequence-to-sequence model.Student-forcing was more effective than teacher-forcing, although it required longer training because it explored more of the environment.
- Generalization: 38.6% versus 21.8% success was obtained in seen versus unseen validation environments for student-forcing.Performance in unseen environments plateaued quickly while continued training improved performance in training environments, even with dropout and weight decay.
- Generalization: The results expose significant challenges in training agents that generalize well to previously unseen environments.The authors state that practices optimized for existing vision-and-language datasets are unlikely to suffice for models operating in new environments.
7. Conclusion and Future Work
The paper introduces a reproducible, interactive, visually realistic framework and the R2R benchmark for evaluating navigation-language understanding in previously unseen real buildings. It concludes that VLN is tractable with existing methods, but generalization remains difficult, while real-location reconstructions offer a scalable resource for related tasks.
- The Matterport3D Simulator balances reproducibility, interactivity, and visual realism for investigating VLN.
- R2R is the first dataset evaluating natural-language navigation in previously unseen real images at building scale.
- Existing vision-and-language methods can be successfully applied to VLN, but generalizing to previously unseen environments remains a significant challenge.
- Crowd-sourced reconstructions of real locations are a highly scalable and underutilized resource for VLN and related embodied vision-and-language problems.
- The released Matterport3D data represent only 90 of more than 700,000 building scans already collected.