Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models

1 Introduction

• VLN is a multimodal, cooperative task in which agents follow human instructions, explore 3D environments, and communicate amid ambiguity.Research spans photorealistic simulators and real environments, with benchmarks using different problem formulations.Source: §1 · pp. 1–2
• The survey addresses a gap in earlier VLN surveys, which predate foundation models and lack a comprehensive overview of foundation-model methods and opportunities, especially LLM applications.It aims both to map progress and opportunities for foundation models and to organize VLN challenges and solutions for foundation-model researchers.Source: §1 · p. 2
• The survey adopts a top-down approach, using the LAW framework to connect foundation models’ roles in world and agent models with VLN’s planning and reasoning challenges.It reviews world models, human models, and VLN agents, then discusses challenges and future opportunities.Source: §1 · pp. 2–3
• The review frames world models around visual representation and generalization, human models around interpreting intentions, and VLN agents around grounding, communication, reasoning, and planning.Table 1 summarizes benchmarks across world, human interaction, agent, and dataset-collection dimensions.Source: §1 · p. 2, Table 1 · p. 3

2.1 Cognitive Underpinnings of VLN

• The survey situates VLN in cognitive accounts of navigation that include landmark-based piloting, path integration, and unified spatial representations.The cognitive map hypothesis proposes a unified spatial representation supporting memory and navigation; spatial language is also linked to spatial tasks and landmark recognition.Source: §2.1 · pp. 3–4
• The survey notes that language and spatial cognition are intertwined, and presents VLN as relevant to both embodied AI and understanding cognitive navigation skills.The discussion connects language use with visual perception and action.Source: §2.1 · p. 4

2.2 Relevant Tasks and Scope of the Survey

• The survey focuses on visual-environment models that address multimodal understanding and grounding, rather than symbolic navigation or surveys centered on perception and physical embodiment.Its top-down review organizes frontier foundation-model methods around world models, human models, and VLN agents.Source: §2.2 · p. 4

2.3 VLN Task Formulations and Benchmarks

• VLN agents follow human instructions from egocentric views to reach a destination through discrete views or lower-level actions, sometimes exchanging information or performing added tasks.Success is defined by arriving within a specified distance of the destination.Source: §2.3 · p. 4
• Benchmarks vary across navigation world, human interaction, agent type and action space, additional tasks, and dataset-collection methods.Table 1 organizes these dimensions, including domain, communication format, language granularity, and route-demonstration source.Source: §2.3 · p. 4
• R2R uses fine-grained instructions in Matterport3D, while RxR adds multilingual, time-aligned instructions and VLN-CE transfers discrete paths to continuous spaces.Robo-VLN further uses continuous action spaces in a robotics-oriented setting.Source: §2.3 · p. 5
• VLN evaluation includes Navigation Error, Success Rate, and Success Rate Weighted Path Length, alongside metrics that assess path fidelity and instruction following.The latter include Coverage Weighted by Length Score, normalized Dynamic Time Warping, and its success-rate-weighted variant.Source: §2.3 · p. 5

2.4 Foundation Models

• Foundation models are trained on large-scale datasets and show generalization across downstream applications; vision-language models integrate visual and textual data.The survey names BERT and GPT-3 as text-only examples, and LXMERT, CLIP, and GPT-4 as vision-language examples.Source: §2.4 · p. 6

3 World Model: Learning and Representing the Visual Environments

• World models help VLN agents represent surroundings, predict how actions change the world, and align perception and action with instructions.The survey identifies history encoding and generalization to unseen environments as two central challenges.Source: §3 · p. 6

3.1 History and Memory

• Navigation-history methods include recurrent state tokens and Transformer encodings of sequential views; a panorama-plus-history design separates spatial relations within views from temporal dynamics across them.That hierarchical design avoids recurrent state-token updates and supports efficient, large-scale pretraining on instruction-path pairs.Source: §3.1 · p. 6
• Graph-based history methods encode geometric cues and may add top-down or neighborhood maps; recent LLM agents also use maps to store viewpoints, objects, and spatial relations.Map representations include grid, semantic, and local metric maps, as well as linguistically formed and topological maps.Source: §3.1 · p. 7

3.2 Generalization across Environments

• Limited environment coverage makes generalization to unseen environments a central challenge, and prior methods use semantic features, environment dropout, and cross-environment image similarity to improve it.The survey points to large-scale environment data as a way to avoid overfitting to training environments.Source: §3.2 · p. 7
• Generalization methods include CLIP-based visual representations, synthetic environment augmentation, and pretraining on collected environments.The survey describes video-derived temporal representations as useful for navigation and says large-scale in-domain pretraining has become crucial for bridging agent-human performance gaps.Source: §3.2 · p. 7

4 Human Model: Interpreting and Communicating with Humans

• Human models must interpret situation-specific instructions, with ambiguity and generalization across visual environments identified as two main challenges.Single-turn instructions can be difficult when landmarks are not currently visible or are indistinguishable from multiple viewpoints.Source: §4 · p. 8, §4.1 · p. 8

4.1 Ambiguous Instructions

• Foundation models help resolve ambiguous instructions by grounding language in visual context and commonsense, including through landmark cues, retrieved knowledge, and actionable instruction breakdowns.Examples include CLIP-based sub-instructions and landmark retrieval, knowledge-enhanced reasoning, and LLM commonsense reasoning about landmark co-occurrences.Source: §4.1 · p. 8
• Information-seeking approaches address when and what to ask and who provides answers; LLMs and VLMs can serve as questioners or helpers, including in VLN-Copilot.The surveyed work includes conformal prediction and in-context learning for determining when and what to ask.Source: §4.1 · pp. 8–9

4.2 Generalization of Grounded Instructions

• Limited and linguistically diverse navigation data can constrain grounding in unseen environments, and foundation models address this through pretrained representations and instruction generation.The passage distinguishes generalization of language style from the challenge of grounding instructions in unseen environments.Source: §4.2 · p. 9
• Pretrained language and vision-language representations support instruction generalization, while methods range from BERT fine-tuning to web-paired multimodal pretraining and cross-lingual representations.The survey also describes prompt-based language-embedding adaptation and VLM representations for navigation policy learning and reasoning.Source: §4.2 · p. 9
• Instruction synthesis ranges from offline speakers trained on instruction-trajectory pairs to generators that produce descriptions during navigation.Marky addresses low-quality generated instructions with text-aligned visual landmarks, while other approaches incorporate spatial cues, progress, or iterative self-training.Source: §4.2 · p. 9

5.1 Grounding and Reasoning

• Foundation-model VLN agents must ground instructions in visual environments while reasoning about prior actions and which sub-instruction to execute.Unlike static vision-language tasks, VLN requires reasoning over spatial and temporal dynamics in both instructions and the environment.Source: §5.1 · p. 10
• Pre-training approaches add scene, object, orientation, history, order, future-view, path-reconstruction, and entity-grounding tasks to improve grounding.The survey contrasts these approaches with explicit semantic modeling and notes that foundation-model pre-training already provides general spatial and temporal semantics.Source: §5.1 · p. 10

5.2 Planning

• Graph-based planners extend local action choices with global graph information, while LLM-based planners generate and dynamically adjust textual plans or sub-goals.Graph approaches also use hierarchical zone/node selection, waypoint prediction, maps, or long-term waypoint effects; LLM examples include object-aware plans, task decomposition, and scene-graph-informed high-level plans.Source: §5.2 · pp. 10–11

5.3 Foundation Models as VLN Agents

• VLN agent backbones have shifted from LSTM-based Seq2Seq models to Transformers and large-scale pre-trained systems.Single-stream VLM agents jointly process language, vision, and history tokens; waypoint predictors help transfer discrete-environment methods to continuous navigation.Source: §5.3 · p. 11
• LLMs can act as agents through textualized visual observations, zero-shot navigation, fine-tuning, or chain-of-thought reasoning.Examples include multi-expert systems, memory topology maps, task decomposition, and simulating future environments for navigational reasoning.Source: §5.3 · pp. 11–12

Benchmarks

• Current VLN datasets have limitations in quality, diversity, bias, and scalability; R2R instruction-trajectory pairs favor shortest paths, which may not reflect real-world navigation.The survey calls for unified, realistic tasks and platforms, including standardized sim-to-real evaluation and benchmarks reflecting dynamic environments and everyday activities.Source: §6 · p. 12

World Model: From 2D to 3D

• Current generic 2D world representations fall short of spatial language understanding in VLN’s fundamentally 3D environments.Existing explicit 3D representations can restrict objects to a closed set, limiting open-vocabulary use; adapting large-scale 3D representations for VLN remains under exploration.Source: §6 · pp. 12–13

Human Model: From Instruction to Dialogue

• Open-ended dialogue benchmarks permit free-form exchanges, but current approaches still rely on rule-based dialogue templates.The survey points to situated, task-oriented dialogue management and foundation-model exploration as future directions.Source: §6 · p. 13

Agent Model: Adapting Foundation Models for VLN

• LLMs lack direct visual perception and can hallucinate objects, while foundation models still face challenges in adapting to navigation tasks.Captioning observations for LLM prompts can lose essential visual semantics; VLMs also lack embodied experience from their primarily internet-based development and need fine-tuning for robust agentic decisions.Source: §6 · p. 13
• In VLN’s constrained indoor settings, LLMs can provide coarse step-by-step directions, but perception and motion-control components remain central to agent decisions.The survey characterizes LLM planning as a supplementary guide rather than the sole decision-making factor.Source: §6 · p. 14

Deployment: From Simulation to Real Robots

• Simulated settings often lack real-world complexity and variability, while lower-quality rendered images contribute to a perception gap and reduced performance and accuracy.Embodiment gaps and data scarcity are also identified as bottlenecks; robot teleportation is described as an alternative for scaling VLN data.Source: §6 · p. 14

• Web-scale pretraining can carry biases, including fairness concerns for multilingual users, while continual training and real-world deployment raise privacy risks.Source: §7 · p. 14