Source-linked AI summary
WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigation
Dujun Nie, Xianda Guo, Yiqun Duan, Ruijun Zhang, Long Chen
TL;DR
Zero-shot object navigation requires finding target objects in unseen environments while limiting costly interaction and reasoning about future states. WMNav integrates VLMs into a world-model navigation framework with predicted-state memory, feedback-based task decomposition, and staged action proposals. It surpasses existing zero-shot benchmarks on HM3D and MP3D in success rate and exploration efficiency.
Problem
Zero-shot object navigation requires locating specific objects in unseen environments, but existing VLM agents lack a fully modular world-model design for predicting future states and reducing costly interaction.
Method
WMNav uses VLMs to predict possible outcomes, stores target-presence likelihoods in an online Curiosity Value Map, decomposes subtasks with feedback, and uses two-stage action proposals.
Results
WMNav surpasses existing zero-shot benchmarks in success rate and exploration efficiency on HM3D and MP3D.
Takeaways & Limitations
WMNav establishes a VLM-based world-model direction for zero-shot object navigation in complex, unknown environments.
Abstract
from arXiv · showhide
Object Goal Navigation-requiring an agent to locate a specific object in an unseen environment-remains a core challenge in embodied AI. Although recent progress in Vision-Language Model (VLM)-based agents has demonstrated promising perception and decision-making abilities through prompting, none has yet established a fully modular world model design that reduces risky and costly interactions with the environment by predicting the future state of the world. We introduce WMNav, a novel World Model-based Navigation framework powered by Vision-Language Models (VLMs). It predicts possible outcomes of decisions and builds memories to provide feedback to the policy module. To retain the predicted state of the environment, WMNav proposes the online maintained Curiosity Value Map as part of the world model memory to provide dynamic configuration for navigation policy. By decomposing according to a human-like thinking process, WMNav effectively alleviates the impact of model hallucination by making decisions based on the feedback difference between the world model plan and observation. To further boost efficiency, we implement a two-stage action proposer strategy: broad exploration followed by precise localization. Extensive evaluation on HM3D and MP3D validates WMNav surpasses existing zero-shot benchmarks in both success rate and exploration efficiency (absolute improvement: +3.2% SR and +3.2% SPL on HM3D, +13.5% SR and +1.1% SPL on MP3D). Project page: https://b0b8k1ng.github.io/WMNav/.
I. INTRODUCTION
WMNav frames zero-shot object navigation as a challenge of reasoning about unseen indoor environments while reducing costly interaction through VLM-based world modeling. It combines predicted-state memory, feedback-driven task decomposition, and staged action proposals, achieving state-of-the-art results on HM3D and MP3D.
- Zero-shot object navigation requires locating unseen target objects in unknown environments using semantic knowledge and efficient movement.
- Existing methods often rely on environmental interaction and cannot exploit predicted future states and action outcomes for anticipatory planning.
- WMNav uses VLMs as a world model to predict future outcomes from panoramic scenes without task-specific training, prebuilt maps, or prior surroundings knowledge.
- The online Curiosity Value Map stores predicted target-presence likelihoods across scenarios as memory for navigation.
- Subtask decomposition with feedback and a two-stage action proposer improve reasoning reliability and exploration efficiency.
- WMNav achieves state-of-the-art zero-shot object-navigation results and outperforms benchmark methods on HM3D and MP3D.
II. RELATED WORK
Related work includes supervised navigation approaches that depend on training and recent zero-shot methods based on open-vocabulary scene understanding.
- Supervised object-navigation methods train visual encoders with reinforcement or imitation learning, or construct semantic maps from training data.
- These supervised approaches can be effective in known environments but struggle with unseen objects and rooms because of their training dependency.
- Recent zero-shot methods address this setting through open-vocabulary scene understanding.
B. Foundation Model Guided Navigation
Foundation-model navigation combines VLM visual grounding with LLM common-sense reasoning, while WMNav applies VLMs in a world-model framework for visually grounded navigation. The task requires navigating from RGB-D observations and poses to a target object in an unknown indoor environment.
- VLMs provide visual grounding, whereas LLMs contribute common-sense reasoning but lack rich visual processing and spatial reasoning.
- World models predict environment state transitions or abstract future outcomes to support planning and improve interaction efficiency.
- WMNav uses a VLM world model that predicts scene outcomes, maintains curiosity and cost memory, and guides navigation decisions.
- Object goal navigation requires exploring an unknown indoor environment and reaching an arbitrary object instance within a category from RGB-D observations and pose information.
B. Overview
WMNav uses panoramic observations and a VLM-based world model to predict goal likelihoods, project them onto a top-down map, and retain them in curiosity-map memory for navigation.
- The framework captures six RGB-D views through agent rotations and converts them into a panoramic image for comprehensive perception.
- Its world model combines PredictVLM with memory built from the Curiosity Value Map and cost.
- PredictVLM estimates the target’s likelihood in each panoramic direction and outputs a curiosity score from 0 to 10.
- The predicted directional scores are projected from the panoramic view onto a top-down map and used to construct the Curiosity Value Map.
2) Curiosity Value Map Construction:
WMNav constructs an online Curiosity Value Map by projecting viewpoint-specific predictions into top-down coordinates and combining them with prior map state.
- Each map pixel stores a curiosity value from 0 to 10 representing the likelihood that the corresponding scene position contains the goal.
- Visited regions found not to contain the goal receive curiosity value 0, while directly discoverable regions receive value 10.
- Initially, all Curiosity Value Map pixels are set to 10 because the agent has no information about the scene.
- Depth information and agent pose project ego-centric navigable areas and their scores into the top-down map.
- The current map is updated by combining the projected navigable-area map with the previous Curiosity Value Map.
3) Cost:
WMNav’s cost module supplies environmental rewards through the current subtask and a binary goal flag, which condition subsequent VLM prompts.
- The cost module uses the subtask and goal flag as environmental rewards for navigation.
- The goal flag records whether PlanVLM finds the goal in the selected image and is either True or False.
- The cost is included in PlanVLM and ReasonVLM prompts to implicitly optimize their navigation-policy outputs.
D. Subtask Decomposition
WMNav decomposes navigation into feedback-rich subtasks, selects high-curiosity views for detailed planning, and uses a two-stage action strategy for exploration and goal approach.
- D. Subtask Decomposition: Subtask decomposition obtains denser environmental feedback by replacing the final goal with an intermediate objective at each step.
- D. Subtask Decomposition: The system selects the navigable direction with the highest curiosity score before performing more specific planning on its image.
- D. Subtask Decomposition: PlanVLM uses the selected image and previous subtask to produce a new subtask and a goal flag.
- E. Two-stage Action Proposer: The action proposer samples vectors at regular angular intervals, filters actions in explored regions, and refines movement distance and angular spacing.
- E. Two-stage Action Proposer: The first stage explores regions likely to contain the goal, while the second approaches the goal and stops at its location.
1) Exploration Stage:
During exploration, WMNav filters candidate actions and uses ActionVLM to select the most appropriate action until the target is found.
- The action proposer filters sampled actions during exploration.
- ActionVLM selects an executable action from the image and labeled candidate action sequence.
2) Goal-approaching Stage:
During goal approaching, WMNav densely samples navigable actions and selects one representing the goal location, with stopping based on distance to the goal.
- The goal proposer densely samples actions from navigable image regions for precise localization.
- GoalVLM selects the sampled action that best represents the goal location.
- The goal flag is True when DistanceToGoal < dthres and False otherwise.DistanceToGoal is the Euclidean distance between the current position and goal position.
- Evaluation uses Success Rate and Success Rate Weighted by Inverse Path Length to measure completion and efficiency.SR measures completed episodes; SPL weights inverse path length by success rate.
B. Implementation Details
WMNav is evaluated on HM3D and MP3D using SR and SPL, and its design combines VLM reasoning with action decomposition to avoid reliance on trained policy modules.
- WMNav is compared with representative object-navigation methods on the MP3D and HM3D benchmarks.
- +3.2% SR and +3.2% SPL on HM3D, and +13.5% SR and +1.1% SPL on MP3D, versus state-of-the-art zero-shot methods.
- The two-stage action proposer uses VLM spatial reasoning without local or learning-based policy modules.
- Table III evaluates open-source and proprietary VLMs, with larger Qwen2.5-VL models performing better and proprietary models ahead of current open-source models.
D. Ablation Study
Ablation studies assess WMNav’s modules and memory strategies, while the conclusion links the Curiosity Value Map, task decomposition, and two-stage proposing to more efficient navigation.
- Subtask Decomposition, Curiosity Value Map, and Two-stage Action Proposer each improve navigation performance in ablations.
- Text-Image Memory performs worse than No Memory, while the Curiosity Value Map improves both SR and SPL.The paper attributes the text-image result to hallucinated erroneous memory information and the map result to quantitative memory construction.
- WMNav uses an online Curiosity Value Map to predict target-presence likelihood and address redundant back-and-forth movement.
- The framework combines a VLM world model, concise memory mapping, and task breakdown for zero-shot object navigation in unknown environments.