Source-linked AI summary
Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
Jing Gu, Eliana Stefani, Qi Wu, Jesse Thomason, Xin Eric Wang
TL;DR
VLN research targets embodied agents that understand human language, perceive environments, and navigate to complete tasks, but the field faces multimodal, reasoning, data, and generalization challenges. This survey systematically reviews benchmarks, methods, evaluation, progress, limitations, and future directions, concluding that current VLN still lacks external knowledge, broader environments, and robust object interaction.
Problem
VLN seeks agents that communicate naturally, perceive their surroundings, and navigate to complete real-world tasks, while current systems face multimodal, reasoning, data-scarcity, and unseen-environment generalization challenges.
Method
The paper systematically categorizes VLN benchmarks, hierarchically classifies methods, and reviews evaluation, progress, challenges, and future opportunities.
Results
The survey identifies representation learning, action strategy learning, data-centric learning, and prior exploration as broad approaches addressing VLN’s central challenges.
Takeaways & Limitations
VLN research spans communication, navigation, interaction, and exploration, providing a structured reference for understanding current tasks, methods, and evaluation.
Takeaways & Limitations
Current VLN often lacks external knowledge, diverse environments beyond American houses, and solutions to the last-mile problem of object interaction.
Abstract
from arXiv · showhide
A long-term goal of AI research is to build intelligent agents that can communicate with humans in natural language, perceive the environment, and perform real-world tasks. Vision-and-Language Navigation (VLN) is a fundamental and interdisciplinary research topic towards this goal, and receives increasing attention from natural language processing, computer vision, robotics, and machine learning communities. In this paper, we review contemporary studies in the emerging field of VLN, covering tasks, evaluation metrics, methods, etc. Through structured analysis of current progress and challenges, we highlight the limitations of current VLN and opportunities for future work. This paper serves as a thorough reference for the VLN research community.
1 Introduction
Vision-and-Language Navigation (VLN) studies embodied agents that communicate in natural language, perceive environments, and navigate or interact to complete tasks. This survey organizes VLN tasks, methods, challenges, and future directions.
- VLN and its motivation: Such agents could autonomously perform household, delivery, and hazardous-condition tasks following human commands.The scientific goal also involves interpreting language, perceiving visual surroundings, and using both to navigate successfully.
- VLN and its motivation: VLN aims to build embodied agents that communicate with humans in natural language and navigate real 3D environments.The agent observes and interacts with the environment while following instructions or requesting guidance from an oracle.
- Task diversity: VLN benchmarks vary from passively following one instruction to conversing with an oracle and interacting with objects.Task objectives range from strictly following described routes to exploring environments and manipulating objects.
- Challenges and organization: Key VLN challenges include multimodal alignment, navigation reasoning, data scarcity, and generalization from seen to unseen environments.The survey groups solutions around representation learning, action strategy learning, data-centric learning, and prior exploration.
- Survey contributions: The survey contributes systematic benchmark categorization, hierarchical classification of solutions, and discussion of opportunities and future directions.
2 Tasks and Datasets
VLN tasks are organized by communication complexity and task objective, spanning initial instructions, oracle guidance, dialogue, navigation, and object interaction. The surveyed datasets also differ in environment type, action space, and task complexity.
- Communication Complexity: Communication complexity ranges from understanding an initial instruction to requesting help and conducting natural-language dialogue during navigation.The three levels are Initial Instruction, Guidance, and Dialogue.
- Task Objective: Task objectives range from fine-grained route following to coarse-grained goal finding that requires path reasoning and possibly additional oracle help.Some object-interaction tasks also require dialogue-based supervision.
- Benchmark examples: Representative benchmarks include R2R for fine-grained navigation, REVERIE and SOON for object-oriented or coarse-to-fine instructions, and ALFRED for interactive household tasks.ALFRED combines coarse- and fine-grained instructions with object interaction in an interactive visual environment.
- Environment and action space: Most current datasets use predefined viewpoints, while VLNCE and Robo-VLN move toward continuous action spaces and freely navigable environments.These changes are intended to facilitate transfer learning to real agents.
- Environment and action space: Benchmarks span indoor, outdoor, and drone settings, including street-view environments and synthetic 3D navigation worlds.Outdoor datasets commonly use Google Street View, while drone-navigation datasets often use Unity3D.
- Oracle Guidance and Human Dialogue: Oracle guidance lets agents request help when uncertain, while dialogue tasks require natural-language questions and responses during navigation.Examples include Just Ask, VNLA, HANNA, CVDN, Talk the Walk, TEACh, and DialFRED.
3 Evaluation
VLN evaluation measures goal achievement, progress, path efficiency, and fidelity to reference routes. The metrics distinguish reaching the goal from following the desired path.
- Goal-oriented metrics: Success Rate measures how frequently an agent completes a task within a specified distance of the goal.Goal Progress measures the reduction in remaining distance to the target.
- Goal-oriented metrics: Path Length measures total navigation distance, while Shortest-Path Distance measures mean distance from the final location to the goal.Longer paths are undesirable because they increase duration and robot wear.
- Goal-oriented metrics: Oracle Success Rate counts whether any visited node comes within a threshold of the target rather than only the final node.
- Path-fidelity metrics: Coverage weighted by Length Score and normalized Dynamic Time Warping evaluate how closely an agent’s trajectory follows a reference path.These are path-fidelity metrics rather than measures of goal proximity alone.
- Path-fidelity metrics: Success weighted by normalized Dynamic Time Warping restricts nDTW to successful episodes, combining success with path fidelity.
4 VLN Methods
The survey organizes VLN methods into four categories addressing multimodal representation, action selection, data scarcity, and generalization to unseen environments. Representation learning further includes modality-specific features, structured relations, memory, and auxiliary tasks.
- 4 VLN Methods: VLN methods are categorized as Representation Learning, Action Strategy Learning, Data-centric Learning, and Prior Exploration.The categories address multimodal understanding, decision-making, limited training data, and adaptation to unseen environments.
- Representation Learning: Representation learning helps agents relate instruction words to perceived environmental features.
- Representation Learning: Pretrained vision or text encoders provide single-modality knowledge for VLN agents.Examples include ResNet, Vision Transformers, and BERT-based representations; Airbert reports good few-shot performance after in-domain pretraining.
- Representation Learning: Semantic representations can improve performance in unseen environments, while undifferentiated visual features may sometimes hurt performance.
- Representation Learning: Graph structures, memory modules, and auxiliary losses encode relations, navigation history, and task-relevant information.Graphs connect instruction and observations; memory retains navigation history; auxiliary tasks can explain actions or predict future decisions.
4.2 Action Strategy Learning
Action strategy learning addresses the difficult choice among many actions in complex environments. Surveyed approaches use reinforcement learning, exploration, planning, stopping decisions, and interaction with an oracle.
- 4.2 Action Strategy Learning: Action strategy learning helps agents select effective actions in environments with many choices and complicated dynamics.
- Reinforcement Learning: Reinforcement learning models VLN as a sequential decision-making problem but must address success signals received only at episode end.The survey describes rewards based on goals, instruction fidelity, landmark alignment, and navigation metrics.
- Reinforcement Learning: Recursively alternating imitation and reinforcement learning improves performance.
- Exploration and Exploitation: Exploration helps agents gather environmental information, but greater exploration can increase path length and duration.Student-forcing combines sampled actions with supervision from shortest-path actions.
- Planning: Planning future observations, states, rewards, waypoints, neighboring views, and stopping locations supports action selection.
- Interaction: Agents can request guidance using action probabilities or separately trained models, with both rule-based and neural dialogue methods available.
4.3 Data-centric Learning
Data-centric learning uses existing data more effectively or creates synthetic data to address VLN’s limited and costly training resources. Methods include augmentation, curriculum learning, cross-task transfer, instruction reinterpretation, and exploration-based adaptation.
- 4.3 Data-centric Learning: Data-centric methods focus on better use of existing data and creation of synthetic data.VLN training data are relatively small compared with task complexity, while collection is expensive and time-consuming.
- Data Augmentation: Trajectory-instruction augmentation generates additional path-instruction pairs with speaker modules and alignment or adversarial filtering.
- Data Augmentation: Masking visual features, remixing house scenes, changing styles or objects, and counterfactual replacement create new environments or training examples.
- Curriculum Learning: Curriculum learning gradually increases difficulty using instruction length or trajectory attributes such as the number of rooms traversed.The survey reports that curriculum learning helps smooth training.
- Cross-task Transfer: Cross-task knowledge transfer trains navigation jointly with dialog-history navigation or question answering.
- Instruction Processing: Multiple interpretations and shorter, more concise instructions can support textual understanding and clearer guidance.
- Prior Exploration: Prior exploration helps agents adapt to unseen environments and bridge the seen–unseen performance gap.Approaches include self-supervised imitation, test-environment path augmentation, and environment-based exploration.
5 Related Visual-and-Language Tasks
Related visual-and-language navigation tasks extend beyond the paper’s emphasis on photo-realistic VLN environments. Visual navigation omits the textual modality, while alternative settings include 2D maps and synthetic environments.
- 5 Related Visual-and-Language Tasks: The paper emphasizes VLN tasks in photo-realistic environments while recognizing 2D maps and synthetic environments as alternatives.
- 5 Related Visual-and-Language Tasks: VLN requires following language instructions and sometimes requesting assistance, whereas visual navigation generally navigates toward a target without textual understanding.
6 Conclusion and Future Directions
The paper broadly reviews VLN tasks, evaluation, and methodologies, then identifies limitations and directions for expanding the field. Key concerns include limited knowledge use, object interaction, collaboration, simulation realism, privacy, and environmental diversity.
- Conclusion: The survey broadly reviews VLN agents, task variation by communication level and objective, evaluation, and methodologies.It presents the discussion as introductory and identifies progress and directions for expansion.
- Future Directions: Current VLN methods rarely use external knowledge, leaving object interaction as a last-mile challenge for embodied agents.The paper contrasts learning to bring an object with learning to pick it up.
- Collaborative VLN: Most benchmarks focus on one navigating agent, whereas complex real-world scenarios may require collaboration among robots or between humans and robots.The paper highlights needs in swarm intelligence, information communication, and performance evaluation.
- Simulation to Reality: Transferring VLN from simulation to real robots causes performance loss because simulators often assume graph-based navigation, perfect localization, teleportation, known topology, and static environments.The paper suggests realistic imagery and probabilistic transitions to narrow the simulation-to-reality gap.
- Ethics & Privacy; Multicultural VLN: Future VLN systems must address privacy risks and limited environmental diversity across homes, cities, and countries.The paper specifically notes sensitive information leakage and reliance on American indoor and outdoor data.
A Dataset Details
The survey reports that VLN datasets are concentrated in a small set of simulators and indoor environments, while outdoor environments are more complex and object-rich.
- Dataset Details: Most indoor datasets use Matterport3D, most outdoor datasets use Google Street View, and indoor datasets outnumber outdoor datasets.The survey also notes that available simulators are limited relative to the number of datasets.
- Dataset Details: Outdoor environments are usually more complex and contain more objects than indoor environments.This distinction is presented as a characteristic of the dataset settings.
B Simulator
VLN datasets rely on simulators offering different combinations of visual, semantic, physical, and navigational capabilities. The survey describes environments ranging from graph-based indoor platforms to landmark-based outdoor scenes.
- Simulators: House3D provides first-person RGB images, semantic and instance masks, and depth information in a SUNCG-based virtual 3D environment.It supplies multiple visual and semantic observations to the agent.
- Simulators: Matterport3D is a large-scale indoor embodied-AI simulator where agents navigate among viewpoints along a predefined graph.It includes houses, apartments, hotels, offices, and churches and underlies many indoor VLN datasets.
- Simulators: Habitat is a physics-enabled 3D platform emphasizing system response speed, with Matterport3D, Gibson, and Replica built in.The survey contrasts Habitat 2.0 with other simulation environments on response speed.
- Simulators: Gibson provides interactive real-world perception scenes with RGB panoramas, global camera poses, and reconstructed 3D meshes.Matterport3D is also integrated into Gibson.
- Simulators: House3D enforces physical constraints, while LANI uses a fenced grass field with randomly placed landmarks for navigation tasks.Drone navigation tasks have also been built on LANI, and outdoor photorealistic simulation remains difficult to construct.
C Room-to-Room Leaderboard
The survey compiles reported results for the frequently used Room-to-Room benchmark and cautions that some evaluation settings are not directly comparable.
- Room-to-Room Leaderboard: Room-to-Room is the benchmark used most frequently to evaluate different VLN methods.The survey collects reported metrics from corresponding papers and the official leaderboard.
- Room-to-Room Leaderboard: Beam search and prior exploration make their reported performance not directly comparable with other Room-to-Room methods.Beam search explores more routes, while prior exploration uses additional observations in the test environment.
- Room-to-Room Leaderboard: Table 4 presents the Room-to-Room leaderboard as of March 2022.The table is identified as a leaderboard rather than a general dataset or simulator summary.