MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation

1 Introduction

• Earlier zero-shot VLN agents textualize visual observations and use multiple expert summaries, losing information and limiting multimodal use; local-only decisions can lead to aimless exploration after errors.These methods select positions or directions using GPT-4 based on textual prompts.Source: §1 · pp. 1–2
• MapGPT builds an online linguistic map with node information and topological relationships in prompts, creating a global view while omitting precise GPS coordinates that GPT may struggle to understand.The map is designed to support global exploration.Source: §1 · p. 2
• MapGPT iteratively updates a human-work-plan-like multi-step plan, enabling systematic exploration of candidate nodes or sub-goals and backtracking for re-exploration when needed.Planning is adapted to the map rather than merely documenting each step's reasoning.Source: §1 · p. 2
• MapGPT achieves state-of-the-art zero-shot performance on R2R and REVERIE, with 31.6% SR on REVERIE, surpassing some REVERIE-trained methods.The benchmarks contain step-by-step and high-level instructions, respectively.Source: §1 · p. 2
• MapGPT uses one navigation expert that works flexibly with GPT-4 and GPT-4V, adapts to varying instruction styles, and achieves state-of-the-art zero-shot performance on both R2R and REVERIE.Its map-guided prompting and adaptive path-planning mechanism are described as encouraging systematic exploration to improve navigation.Source: §1 · p. 2

Vision-and-Language Navigation (VLN)

• Existing GPT-based zero-shot VLN agents reason locally, and their designs limit instruction styles or validation beyond R2R (Zhou et al., 2023; Long et al., 2023).NavGPT has limited performance and uses a two-stage language-only system, while DiscussNav uses sequential experts and fuses five predictions; these designs address step-by-step R2R instructions but are not validated on REVERIE-style instructions.Source: §2 · pp. 2–3

Large Language Models (LLMs)

• LLM-based zero-shot navigation remains limited, and prompting LLMs, including GPT-4V, for global thinking and planning remains unexplored (Yang et al., 2023).LangNav uses GPT-4 to synthesize data, then fine-tunes Llama2 rather than using an LLM directly as a zero-shot agent (Pan et al., 2023).Source: §2 · p. 3

Maps for Navigation

• Metric maps trade map size against computational efficiency, while topological maps enable global exploration but prior methods are designed for model learning.Using prompts to build maps and leverage LLMs for zero-shot reasoning and planning based on them remains unexplored.Source: §2 · p. 3

3.1 Single Expert Prompt System

• On R2R, its two-stage system uses 672 input and 115 output tokens per step, versus NavGPT’s 2,465 input and 317 output; one expert handles prompts and navigation.The system can use GPT-4V directly with visual observations or use text descriptions with GPT-4.Source: §3.1 · p. 3

3.1.1 Task Description

• The task description gives GPT the task background, defines step inputs, and specifies basic requirements for completing the task.Source: §3.1.1 · p. 4

3.1.2 Fundamental Inputs

• The agent receives raw instructions, navigable-view observations, directional action choices plus stop, and prior actions; GPT returns one action label, with surroundings used only for REVERIE stopping.Observations can be images for GPT-4V or captions and detection results for the two-stage system.Source: §3.1.2 · p. 4

3.2 Map-Guided Prompting

• MapGPT builds an online graph of observed nodes and edges, then renders topological relationships as text because GPT-4V struggles with precise coordinates.Newly navigable neighboring nodes and their edges update the graph at each step.Source: §3.2 · p. 5

3.2.1 Constructing Maps with Prompts

• Prompts represent explored-node trajectories and connectivity, annotate accessible actions with node IDs, and supply inaccessible unexplored nodes’ images or descriptions to support backtracking.Explored nodes are identified in observation order, while inaccessible nodes provide supplementary information for choosing where to revisit.Source: §3.2.1 · pp. 5–6

3.3 Adaptive Path Planning

• At each step, the agent combines its thought, map, and prior plan to update a multi-step plan, considering multiple sub-goals and adapting to exploration or backtracking.The current plan becomes input to the next step, so the agent can continue toward sub-goals or return to a previously visited node for re-exploration.Source: §3.3 · p. 6

4.1 Experimental Settings

• R2R provides detailed step-by-step instructions, while REVERIE gives high-level object-finding instructions; evaluation focuses on navigation rather than REVERIE object grounding.The navigation metrics are NE, SR, OSR, and SPL.Source: §4.1 · p. 6

4.2.1 Results on the Room-to-Room Dataset

• 4.8% higher SR and 15.6% higher OSR than NavGPT were achieved by MapGPT (GPT-4) on the R2R validation unseen set; MapGPT (GPT-4V) achieved 34.8% SPL.The comparison used 11 scenes and 783 trajectories; MapGPT (GPT-4) also showed a noticeable reduction in NE relative to NavGPT.Source: §4.2.1 · p. 7, Table 2 · p. 7

4.2.2 Results on the REVERIE Dataset

• 31.6% SR was achieved by MapGPT on the sampled REVERIE validation unseen set, outperforming zero-shot NavGPT and some training-only models across all metrics.MapGPT was also highly competitive with pretrained and fine-tuned HAMT.Source: §4.2.2 · p. 7, Table 3 · p. 7
• MapGPT backtracked in 49% of REVERIE cases, correcting an erroneous navigation path at least once in 80% of those cases, compared with NavGPT’s 18% and 53%, respectively.Backtracking was defined as repeated visits to previously visited locations.Source: §4.2.2 · p. 7

4.3 Ablation Study

• Topological maps expressed in natural language improved performance, whereas coordinate maps and global action planning did not help GPT-based zero-shot navigation.Adaptive path planning instead has GPT output a multi-step path segment, flexibly attending to candidate nodes or sub-goals and correcting earlier errors.Source: §4.3 · pp. 7–8

4.4 Case Study

• MapGPT (GPT-4V) used the map’s topological relationships to explore four likely candidates among six unexplored places, adaptively revising its plan before finding the hidden bathroom.The plan included possible backtracking to place 2 and was updated after discovering a direct connection between places 5 and 4.Source: §4.4 · p. 8, Figure 3 · p. 8

5 Conclusion

• MapGPT achieves state-of-the-art zero-shot performance on the VLN task, with global thinking and path-planning capabilities.The conclusion attributes these results to extensive experiments.Source: §5 · p. 9
• Map-guided prompting builds online maps and provides GPT with node information and topological relationships to activate global exploration.Source: §5 · p. 9
• Adaptive planning enables multi-step path planning based on the map, helping the agent systematically explore potential objectives.Source: §5 · p. 9

• MapGPT is evaluated only in a simulator that incorporates certain ideal assumptions.Source: Limitations · p. 9
• GPT’s real-world image data in pre-training gives zero-shot VLN potential for generalization and sim-to-real transfer.Source: Limitations · p. 9
• Developing LLM-based agents in the real world and addressing its challenges is identified as a meaningful future direction.Source: Limitations · p. 9

A.1 Prompts

• Unified GPT-4V task prompts specify task background, input definitions, and output requirements.These prompts are fed directly into GPT-4V’s system content.Source: §A.1 · p. 11
• Prompt modifications adapt R2R and REVERIE [Qi et al., 2020b], with REVERIE requiring target-object checking before stopping.The unified agent focuses on navigation, ignoring REVERIE’s extensive interactive object actions; R2R requires stopping at the destination.Source: §A.1 · p. 11
• Each step’s observations, history, maps, and other environmental information are incorporated into GPT’s user messages.Source: §A.1 · p. 11
• In the two-stage system, observations are represented as N navigable-viewpoint entries, each pairing a scene description with its objects.Each entry describes the observation toward a specific navigable point.Source: §A.1 · p. 11
• Actions use six directional concepts—forward, left, right, around, up, and down—to specify movement toward a described place.The action template can pair a direction and place with a scene description and listed objects.Source: §A.1 · p. 11
• In the one-stage GPT-4V system, interleaved image IDs and observed images let the agent reference images in actions, history, and maps.An action can identify a place by its corresponding image ID.Source: §A.1 · p. 12

A.2 Implementation Details

• In Matterport3D, the agent receives visual observations and connected navigable candidates, selects a candidate label through GPT, then executes its viewpoint ID in the simulator.The simulator provides discrete navigation among predefined viewpoints and teleports the agent to the selected viewpoint.Source: §A.2 · p. 12
• MapGPT was implemented as a one-stage GPT-4V agent and a two-stage GPT-4 system using BLIP-2 captions and Faster R-CNN object detection.The two-stage setup follows NavGPT for comparison with methods designed for GPT-4; both systems use map-guided prompting and adaptive path planning, and REVERIE requires only simple modifications.Source: §A.2 · p. 12

B More Qualitative Examples

• The agent can misread instruction details, turning into an incorrect bedroom instead of walking straight into the bedroom in the eleven o’clock direction, then stopping there.Figure 6 presents this as stopping erroneously in a similar location.Source: B More Qualitative Examples · p. 12, Figure 6 · p. 15
• In complex scenes with insufficient instruction clues, the agent may fail to explore the correct direction in time.Figure 6 illustrates delayed backtracking when the correct route is 0→3.Source: B More Qualitative Examples · p. 12, Figure 6 · p. 15
• After exploring places 3 and 4 from place 1, MapGPT backtracks to place 1, explores inaccessible places 6 and 7, and reaches place 7 at step 10.The example demonstrates map-guided global exploration and planning.Source: B More Qualitative Examples · p. 12, Figure 5 · p. 14