Source-linked AI summary
MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation
Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, Kwan-Yee K. Wong
TL;DR
Existing zero-shot VLN agents mainly reason over local environments, limiting their global view during navigation. MapGPT builds an online linguistic topological map and updates multi-step plans, achieving state-of-the-art zero-shot performance on R2R and REVERIE while supporting GPT-4 and GPT-4V.
Problem
Existing zero-shot VLN agents make decisions from local environments without an effective global view, and their text-based processing can hinder multimodal-model application.
Method
MapGPT incorporates node information and topological relationships from an online linguistic map into prompts and adaptively updates multi-step path plans.
Results
MapGPT achieves state-of-the-art zero-shot performance on both R2R and REVERIE and is applicable to GPT-4 and GPT-4V.
Takeaways & Limitations
The method supports global exploration and systematic step-by-step exploration of potential objectives through map-based path planning.
Takeaways & Limitations
MapGPT is evaluated only in a simulator incorporating certain ideal assumptions, leaving direct real-world deployment as future work.
Abstract
from arXiv · showhide
Embodied agents equipped with GPT as their brains have exhibited extraordinary decision-making and generalization abilities across various tasks. However, existing zero-shot agents for vision-and-language navigation (VLN) only prompt GPT-4 to select potential locations within localized environments, without constructing an effective "global-view" for the agent to understand the overall environment. In this work, we present a novel map-guided GPT-based agent, dubbed MapGPT, which introduces an online linguistic-formed map to encourage global exploration. Specifically, we build an online map and incorporate it into the prompts that include node information and topological relationships, to help GPT understand the spatial environment. Benefiting from this design, we further propose an adaptive planning mechanism to assist the agent in performing multi-step path planning based on a map, systematically exploring multiple candidate nodes or sub-goals step by step. Extensive experiments demonstrate that our MapGPT is applicable to both GPT-4 and GPT-4V, achieving state-of-the-art zero-shot performance on R2R and REVERIE simultaneously (~10% and ~12% improvements in SR), and showcasing the newly emergent global thinking and path planning abilities of the GPT.
1 Introduction
Existing zero-shot VLN agents rely on local observations and can explore aimlessly after navigation errors. MapGPT adds a linguistic topological map and adaptive planning to support global exploration, achieving strong zero-shot results across R2R and REVERIE.
- 1 Introduction: Existing zero-shot VLN agents prompt GPT-4 to select positions from local observations, but may continue exploring aimlessly after erroneous exploration.These systems also rely on textual conversion and multiple summarization steps.
- 1 Introduction: MapGPT introduces an online linguistic map with node information and topological relationships, encouraging GPT’s global exploration.The method preserves topology rather than difficult-to-interpret precise GPS coordinates.
- 1 Introduction: MapGPT uses adaptive multi-step path planning to explore candidate nodes or sub-goals systematically and backtrack when necessary.The mechanism updates plans iteratively based on the map.
- 1 Introduction: MapGPT applies to GPT-4 and GPT-4V and achieves state-of-the-art zero-shot performance on both R2R and REVERIE.The reported REVERIE success rate is 31.6%, surpassing some learning-based methods trained on REVERIE.
2 Related Work
Prior VLN work includes learning-based methods trained on domain-specific datasets and LLM-based agents that make decisions within adjacent navigable points. MapGPT extends this line with map-guided prompting and adaptive path planning for global exploration.
- 2 Related Work: Learning-based VLN approaches train model architectures on domain-specific datasets for navigation.These methods include pretrained and task-specific approaches.
- 2 Related Work: LLM-based agents have attracted attention for embodied tasks, while prior VLN systems make decisions within adjacent navigable points.MapGPT is presented as a map-guided alternative for global exploration.
3 Method
MapGPT uses a single-expert prompt system that combines task and navigation inputs with an online linguistic topological map. An adaptive planner updates multi-step plans so the agent can explore candidate nodes and backtrack across the environment.
- 3.1 Single Expert Prompt System: The single navigation expert accepts GPT-4V visual inputs directly or textual descriptions in a two-stage GPT-4 setup.This design removes separate history-summary and instruction-decomposition experts while supporting both visual and textual inputs.
- 3.1 Single Expert Prompt System: The prompt system organizes task description, instruction, action history, observations, and action space into the language-model input.The model generates a thought and selects one action from the available options.
- 3.1.2 Fundamental Inputs: Visual observations focus on navigable viewpoints, using original images for GPT-4V or captions and detections for the two-stage system.Object lists are additionally used for REVERIE stopping decisions but not for R2R.
- 3.2.1 Constructing Maps with Prompts: The map is a dynamically updated graph of observed nodes and connectivity, with node IDs added to prompts so the agent can reference explored and accessible locations.Supplementary information preserves unexplored or inaccessible nodes for backtracking and re-exploration.
- 3.3 Adaptive Path Planning: At each step, MapGPT combines the current thought, map, and previous plan to generate an updated multi-step path plan.The previous plan is fed into the next iteration, allowing the agent to continue exploring sub-goals or backtrack to earlier nodes.
- 3.2 Map-Guided Prompting and 3.3 Adaptive Path Planning: MapGPT combines an online topological map with adaptive multi-step planning to support global exploration and systematic navigation.The map records node relationships, while planning can focus on multiple candidate nodes or sub-goals and update as exploration proceeds.
4 Experiments
Experiments evaluate MapGPT on R2R and REVERIE, comparing zero-shot settings, map and planning designs, and a successful exploration case. MapGPT improves navigation performance while adaptive planning supports systematic exploration and error correction.
- 4.2.1 Results on the Room-to-Room Dataset: 47.7% success rate is achieved by the GPT-4V one-stage MapGPT agent on the R2R various-scenes evaluation.On the validation unseen set, GPT-4 and GPT-4V achieve 38.8% and 43.7% success rates, respectively.
- 4.2.1 Results on the Room-to-Room Dataset: 15.6% improvement in OSR over NavGPT on the R2R validation unseen set indicates higher attainable navigation performance with MapGPT.MapGPT also leads NavGPT by 4.8% in SR, while GPT-4V reaches 34.8% SPL.
- 4.2.2 Results on the REVERIE Dataset: 31.6% SR on REVERIE demonstrates MapGPT’s competitiveness against zero-shot NavGPT and some training-only models across all metrics.The comparison uses a randomly sampled subset of the validation unseen set.
- 4.2.2 Results on the REVERIE Dataset: MapGPT backtracks in 49% of REVERIE cases, successfully correcting an erroneous navigation path at least once in 80% of those cases.NavGPT backtracks in 18% of cases and corrects errors in 53% of those cases.
- 4.3 Ablation Study: Topological map prompting improves performance over coordinate maps, while adaptive planning lets GPT attend to multiple nodes or sub-goals and correct previous errors.The ablation compares coordinate maps with topological maps and global action planning with adaptive path planning.
- 4.3 Ablation Study: In a REVERIE example, MapGPT systematically explores four candidate places and discovers the bathroom at place 8 after local observations remain insufficient.The case illustrates global exploration, map understanding, and adaptive multi-step path planning.
5 Conclusion
MapGPT is a zero-shot VLN agent that combines online map-guided prompting with adaptive multi-step planning. Experiments report state-of-the-art zero-shot performance alongside global thinking and path-planning capabilities.
- 5 Conclusion: MapGPT builds online linguistic maps with node information and topological relationships to activate global exploration and adaptive multi-step path planning.The approach systematically explores potential objectives through map-based planning.
Limitations
The study evaluates MapGPT only in a simulator with certain ideal assumptions, leaving direct real-world deployment and associated challenges for future work.
- Limitations: MapGPT is evaluated only in a simulator with ideal assumptions, so direct real-world development and deployment remain future directions.The paper identifies addressing real-world challenges as a meaningful next step.
A.1 Prompts
MapGPT uses unified task prompts and stepwise environmental templates to support navigation across R2R and REVERIE, with both text-based and multimodal GPT inputs.
- The unified prompts define task background, input definitions, and output requirements for GPT-4V.
- R2R and REVERIE require only a few prompt modifications, including ignoring REVERIE object interactions for unified navigation.
- At each step, environmental observations, history, and maps are incorporated into GPT user messages.
- Navigable viewpoints are represented as directional actions paired with scene and object descriptions.
- GPT-4V prompts can directly reference interleaved images by image ID in action, history, and map templates.
A.2 Implementation Details
MapGPT is implemented in the Matterport3D simulator with both direct multimodal GPT-4V and two-stage GPT-4 variants, while preserving the same core map-guided mechanisms.
- The Matterport3D simulator provides visual observations and connected navigable candidates that GPT selects for execution.
- MapGPT includes a one-stage GPT-4V agent and a two-stage GPT-4 system using BLIP-2 captions and Faster R-CNN detections.
- Map-guided prompting and adaptive path planning are applied to both system variants, with simple modifications for REVERIE.
B More Qualitative Examples
Qualitative examples show MapGPT using map-based exploration, backtracking, and stopping decisions, while failures arise from instruction-detail errors and delayed exploration of difficult directions.
- Successful example: A successful R2R trajectory explores branches from place 1, backtracks, reaches previously inaccessible places 6 and 7, and stops at place 7 in step 10.
- Failure cases: Typical failures involve entering a similar incorrect location or failing to explore the correct direction promptly in complex scenes.
- Prompt example: The task prompt combines global instructions, explored-place history, trajectory IDs, map connectivity, and supplementary unvisited places.
- Dataset adaptation: REVERIE task prompts omit object interactions because the unified system focuses on navigation.
- Planning example: The planning example turns around from place 4 to place 3 and then plans to proceed to place 7.
- Stopping example: A stopping example concludes that entering the bedroom satisfies the instruction and selects the stop action.