Source-linked AI summary
HGWM: Hierarchical Graph-Guided World Model for Zero-Shot Object Navigation via Graph Matching
I-Tak Ieong, Hao Tang
TL;DR
Zero-shot object navigation requires agents to anticipate target locations while retaining evidence from earlier observations. HGWM combines a category-conditioned room–anchor prior with persistent graph memory and staged heading evaluation. It improves SR and SPL over WMNav on HM3D v0.1 and MP3D.
Problem
Zero-shot navigation must anticipate where a target may be found while retaining evidence from previous observations, despite ambiguous observations and incomplete spatial memory.
Source: §1 · p. 1Method
HGWM combines a hierarchical goal prior with persistent spatial-semantic memory, then uses coarse visual-semantic scoring and fine direction-specific prior matching to guide actions.
Source: §1 · p. 1Results
HGWM gains 4.5 pp SR and 0.3 pp SPL over WMNav on HM3D v0.1, and 1.4 pp SR and 0.7 pp SPL on MP3D.
Source: §1 · p. 1Takeaways & Limitations
The simulation findings support combining relational priors and persistent graph memory for target-directed search.
Source: §5 · p. 9Experiments use static Habitat environments and do not quantify physical-deployment effects such as localization or depth noise, moving objects, or atypical layouts.
Source: §4.7 · p. 8
Abstract
from arXiv · showhide
Zero-shot object navigation requires an agent to reason about where a target may be found while retaining evidence from previous observations. We present HGWM, a hierarchical graph-guided world model that combines a category-conditioned room--anchor prior with persistent spatial-semantic memory. A coarse stage uses a vision-language model to score candidate headings, while a fine stage explicitly compares observed objects and relations with the goal prior. Phase-dependent fusion integrates these signals with a language-model recommendation to guide local navigation. HGWM achieves 62.6\% success rate (SR) and 31.5\% success weighted by path length (SPL) on HM3D v0.1, improving over WMNav by 4.5 and 0.3 percentage points, respectively. On MP3D, it achieves 46.8\% SR and 17.9\% SPL, with gains of 1.4 and 0.7 points. On HM3D v0.2, the full model reaches 74.4\% SR and 31.9\% SPL; removing individual components reduces SR by 0.9--2.3 points and SPL by 0.6--1.5 points. These results support the use of relational priors and persistent graph memory for target-directed search.
1 Introduction
HGWM combines a hierarchical room–anchor prior with persistent spatial-semantic memory to guide target search. It reports improved SR and SPL over WMNav on HM3D v0.1 and MP3D.
1 Introduction
- Navigation must retain evidence across viewpoints and use room–object relations without treating common associations or VLM detections as reliable in every scene.Structural checks may reject misleading interpretations but can also suppress valid atypical observations.Source: §1 · p. 1
- HGWM first generates a hierarchical goal graph, then populates persistent spatial-semantic memory from RGB-D observations and uses coarse scoring plus fine prior-conditioned matching to select actions.Grounding relates hypotheses to sensor evidence without proving either graph correct.Source: §1 · p. 1
- The evaluation covers HM3D and MP3D, with four additional HM3D v0.2 component ablations, analyses of success and path efficiency, and qualitative navigation examples.Source: §1 · p. 1
- 4.5 pp SR gain on HM3D v0.1 and 1.4 pp on MP3D over WMNav, with SPL gains of 0.3 and 0.7 pp, respectively.Source: §1 · p. 1
2 Related Work
Prior navigation work uses learned or pretrained representations, spatial memory, and graph structures. HGWM’s focus is how a category-conditioned relational hypothesis influences candidate-heading scores and navigation decisions.
Semantic search and spatial maps
- HGWM addresses how category-conditioned relational hypotheses influence candidate-heading scores, complementing prior work on spatial memory and target-directed search.The surveyed approaches include learned task capabilities, pretrained representations, frontier maps, waypoint reasoning, and target-location beliefs.Source: §2 · p. 1
Scene graphs for embodied reasoning
- Scene-graph methods already support embodied context and perception review; HGWM couples observed memory to a separate room–anchor hypothesis and explicit compatibility signal.SG-Nav builds an online hierarchy of objects, groups, and rooms and uses graph-based re-perception.Source: §2 · p. 2
Goal graphs and the ObjectNav setting
VLM decision memory
- HGWM retains WMNav’s map-and-policy structure [17] while adding a separate goal prior, persistent relational evidence, and a fine compatibility channel.Its decision rule combines visual promise, graph compatibility, and an LLM direction recommendation using coefficients selected from the current matching state.Source: §2 · p. 2
Predictive models and HGWM’s contribution
- Unlike learned predictive environment models, HGWM compares a pretrained-model-generated symbolic hypothesis with observations without training dynamics or synthesizing future observations.Its contribution combines category-to-graph expansion, direction-specific compatibility, and phase-dependent fusion within persistent decision memory.Source: §2 · p. 2
3 The Proposed Method
HGWM maintains a symbolic predictive navigation state that couples a category-conditioned goal prior with persistent observation memory and candidate-heading scores. Its coarse visual-semantic scores, fine graph compatibility, and phase-dependent fusion guide search and local action selection.
3 The Proposed Method
- HGWM’s “world model” is a symbolic state comprising a goal prior, observation memory, and scores for candidate headings.It is not a learned dynamics simulator.Source: §3 · p. 3
3.1 Task and Decision State
- The agent uses RGB-D observations and pose to select a search direction and short-horizon subtask before a local navigator chooses a feasible displacement.The 40-step horizon counts decision iterations, not uniform metric distance.Source: §3.1 · p. 3
3.2 Goal Prior and Observation Memory
- For a target category, an LLM supplies likely and alternative rooms, anchor objects, and room connections as a cached episode-level hypothesis, not a map of the scene.Hypothetical nodes remain separate from observations; detected anchors can support a direction before the target appears, while an absent anchor does not prove target absence.Source: §3.2 · p. 3
- Spatial-semantic memory stores object names, inferred rooms, observation counts, and estimated positions, then associates directional descriptions with nearby depth-based view rays.Positions are coarse view estimates, not precise object projections.Source: §3.2 · p. 3
3.3 Coarse Visual-Semantic Scoring
- The coarse VLM stage scores each heading from the panorama, target, and prior context, using 10 for a model-reported visible target and 0 for a dead end without one.Intermediate opportunities score 1–9; the scale is a model judgment, not validated detector confidence, and missing scores default to 5.Source: §3.3 · p. 3
3.4 Fine Symbolic Compatibility
- Fine compatibility compares room, object, spatial, semantic, and depth features, with heuristic relation proposals refining observed graph evidence.The relation updates can mishandle atypical arrangements and do not verify truth; matching uses symbolic features rather than a learned graph-embedding network.Source: §3.4 · pp. 3–4, Figure 3 · p. 4
3.5 Phase-Dependent Fusion
- Phase selection uses fine-score thresholds to combine coarse scores, fine compatibility, and an LLM direction recommendation through heuristic coefficients and clipping.The illustrated fine scores are heuristic compatibility values, not calibrated probabilities; the algebraic comparison of fine and coarse contributions is a diagnostic, not a measured navigation ablation.Source: Figure 2 · p. 4, §3.5 · p. 5, Table 1 · p. 5
3.6 Memory Update and Local Action Selection
- Fused directional values update a spatial curiosity map, whose stored regional means guide heading selection; a planning VLM proposes the next subtask and goal flag, while depth-derived navigability constrains local motion.The conservative map update can preserve an erroneous low score, and model-driven stopping or navigation can fail despite a high score.Source: §3.6 · p. 5
4 Experiments
Experiments evaluate HGWM in simulation on HM3D and MP3D, using benchmark comparisons, component analyses, and qualitative trajectory examples. Results are accompanied by limits on run-to-run interpretation, latency accounting, and robustness evidence.
4 Experiments
- The reported experiments are simulations on HM3D and MP3D, without physical-robot trials or a systematic sensor-noise robustness protocol.The authors identify these as boundaries of the evaluation scope.Source: §4 · p. 5
4.4 Qualitative Navigation Behavior
- Selected WMNav and HGWM trajectories show HGWM visiting different regions with less repeated exploration, but do not establish an aggregate reduction in looping.The examples expose observations, model responses, and decisions along search trajectories and complement the quantitative evaluation.Source: Figure 4 · p. 6, §4.4 · p. 7
4.1 Datasets and Evaluation Metrics
- Experiments use HM3D v0.1, HM3D v0.2, and MP3D; SR measures completion and SPL navigation efficiency, with split results kept separate.Main comparisons use HM3D v0.1, while HM3D v0.2 is used for component comparisons; differences are absolute percentage-point changes within a dataset and split.Source: §4.1 · p. 6
4.2 Implementation and Reproducibility
- Gemini-1.5-Pro provides scene understanding and coarse scores, while Qwen2.5-7B-Instruct generates priors, proposes relation updates, and recommends directions.Goal priors are cached within each episode; the setup records model calls and elapsed time, and the main HM3D v0.1 setting is run three times.Source: §4.2 · p. 6
4.3 Main Results
- 62.6% SR and 31.5% SPL on HM3D v0.1 beat WMNav by 4.5/0.3 pp; on MP3D, 46.8% SR and 17.9% SPL exceed it by 1.4/0.7 pp.Across three HM3D v0.1 runs, SR ranges from 62.3 to 62.8 and SPL from 31.2 to 31.7; these are not confidence intervals, and the SPL lower bound reaches WMNav’s point estimate.Source: §4.3 · pp. 6–7, Table 2 · p. 7
4.5 Backbone Sensitivity
- On HM3D v0.1, replacing Gemini-1.5-Pro with Gemini 2.0 Flash changes SR from 62.6% to 59.2% and SPL from 31.5% to 30.8%.The reported decreases are 3.4 pp in SR and 0.7 pp in SPL; the comparison covers two backbone settings.Source: Table 4 · p. 7, §4.5 · p. 7
4.6 Component Analysis on HM3D v0.2
- On HM3D v0.2, full HGWM achieves 74.4% SR and 31.9% SPL. Basic VLM navigation obtains 67.8% SR and 29.3% SPL, while voxel memory with direct VLM scoring obtains 70.9% SR and 30.1% SPL.Scene–goal graph matching improves SR by 1.5 points and SPL by 1.4 points over prior prompting.Source: §4.6 · pp. 7–8
4.7 Efficiency and Limitations
- Gemini-1.5-Pro inference takes approximately 2.3 seconds per call, compared with 12–25 ms for graph updates at tested episode lengths.Per-call latency is not full-step latency; the static Habitat experiments do not quantify physical-deployment noise, moving objects, atypical layouts, or paired significance tests.Source: §4.7 · p. 8
5 Conclusion
HGWM combines relational priors, graph memory, and direction scoring; simulation results support the combined design, while several evaluation questions remain open.
5 Conclusion
- Removing individual components reduces SR by 0.9–2.3 points and SPL by 0.6–1.5 points relative to full HGWM on HM3D v0.2.The paper concludes that the findings support the combined design.Source: §5 · p. 9
- Component interactions, paired uncertainty, query cost, and deployment beyond simulation remain open evaluation questions.These findings support the combined design, but do not resolve those questions.Source: §5 · p. 9
- Code, prompt builders, and the case gallery are available online.The paper provides repository and project-page links.Source: §5 · p. 9
A Goal Prior Construction
The goal-prior prompt is instantiated for each target category and directs an LLM to construct hierarchical spatial knowledge, later parsed as a goal subgraph.
A Goal Prior Construction
- The prompt is dynamically instantiated for each target category, with the goal placeholder replaced by the requested object.The described strategy template is distinct from the executable builder, whose source index identifies its schema and examples.Source: §A.1 · p. 10
- The LLM organizes likely primary and secondary rooms, adjacent zones, pathways, visual access points, and navigation landmarks.The resulting knowledge is parsed into a goal subgraph whose nodes represent rooms or objects and whose edges encode spatial relationships.Source: §A.1 · p. 10
- The prompt requests target-oriented anchors, visual pathways, distance indicators, and positioning cues to guide search.It also asks for exploration priorities and recovery strategies when initial paths fail.Source: §A.1 · p. 10
B Implementation Details
Implementation combines a configured RGB-D agent and pretrained language models with coarse directional scores, symbolic graph matching, phase-dependent weights, and persistent memory.
B.1 Agent Configuration
- The agent uses a 40-step episode limit, a 640×480 RGB-D camera, and a 1.0 m success threshold, with Gemini-1.5-Pro as the primary VLM.The agent body radius and height are 0.18 m and 0.88 m, respectively.Source: §B.1 · p. 10
B.2 Model Choices
- Gemini-1.5-Pro handles scene understanding and coarse semantic scoring; Qwen2.5-7B-Instruct generates priors, proposes relation updates, and recommends directions.The reported experiments do not include a matched cost or spatial-reasoning comparison against GPT-4V.Source: §B.2 · p. 10
C.1 Score Domains and Phase Selection
- Coarse scores cover six panorama headings on a 0–10 scale, while fine-overlap thresholds select verification, focused search, or frontier exploration.The phase schedule uses f_max ≥ 0.7 for verification, then mean overlap ≥ 0.4 or maximum overlap ≥ 0.5 for focused search; otherwise it explores.Source: §C.1 · p. 10
C.2 Symbolic Feature Definitions
- Fine matching compares symbolic room, object, and spatial relations, alongside semantic and depth features; relation matching uses canonical endpoint names and labels.Room scoring includes room-name and spatial-region overlap plus a heuristic connectivity term; spatial scoring weights room, object, and containment relations 0.4, 0.4, and 0.2.Source: §C.2 · p. 11
C.3 Fine Weights, Bonuses, and Direction Factors
- Fine-score weights depend on feature conditions and scene quality; bonuses reward specified partial matches and reliable relations, while direction factors use nearby stored observations.The default feature-weight vector is (0.35,0.30,0.20,0.10,0.05) in room, object, spatial, semantic, and depth order.Source: §C.3 · p. 11
C.4 Memory, Prompts, and Execution
- Goal priors are generated at runtime, cached within an episode, and cleared on reset; independent episodes may use different priors and model outputs.The documented Table 6 weights are implementation settings, not fitted performance results.Source: §C.4 · p. 12, Table 6 · p. 12
C.5 Graph Representation and Memory Updates
- Before scoring, the graph interface normalizes fields and merges retrieved memory with current observations; object positions are coarse estimates tied to view-center rays.Canonical endpoint names support matching across instance identifiers, and missing observation, relation, or depth evidence contributes zero to its match term.Source: §C.5 · p. 12
C.6 Ablation Settings and Evaluation Tools
The evaluation distinguishes ablation settings and paired episode analysis, but does not establish the separate roles of coarse and fine stages or end-to-end runtime cost.
C.6 Ablation Settings and Evaluation Tools
- The no-arbitration control removes the LLM contribution without rescaling other terms, while other variants change graph context or score scaling.The prior-prompt-only setting bypasses graph scoring and planning context; the graph-memory/direct-scoring setting retains graph context but uses raw visual direction scores.Source: §C.6 · p. 12
- The analysis matches episodes and resamples paired scene clusters, but the reported experiments provide point differences rather than bootstrap intervals.Runs save configuration, seed, episode identities, client-call counts and times, graph diagnostics, and episode outcomes.Source: §C.6 · p. 12
C.7 Runtime
- Runtime measurements do not establish end-to-end latency, asymptotic scaling, or total evaluation cost.Source: §C.7 · p. 12
D.1 Roles of Coarse and Fine Stages
- The stage comparison does not separately measure coarse recall, fine precision, pruning errors, or loop frequency, so it cannot establish a proven division of labor.Prior consistency can reject correct but unusual observations while allowing plausible errors to remain.Source: §D.1 · p. 12
D.2 Full Component Configuration
- The full HGWM row changes both memory and scoring, so its comparison does not isolate either component’s effect.The table caption also notes that this configuration does not measure ambiguity robustness.Source: Table 8 · p. 13
D.3 Additional Component Results
The additional analyses compare component removals and graph matching, report run-to-run variation, and outline further evaluations. The authors note that existing controls do not isolate matching’s direct scoring effect.
D.3 Additional Component Results
- Removing the prior or relation refinement lowers SR/SPL by 2.3/1.1 and 2.3/1.5 points; arbitration and non-adaptive weights lower them by 1.7/0.8 and 0.9/0.6 points.These ablations are compared with full HGWM at 74.4% SR and 31.9% SPL; individual removal effects need not be additive.Source: §D.3 · p. 13
D.4 Prior Information and Graph Matching
- Matching scene and goal graphs improves SR by 1.5 points and SPL by 1.4 points over prior prompting, but existing controls do not isolate matching from other changed components.The authors describe a stricter matched comparison as a proposed extension.Source: §D.4 · p. 13
D.5 Three-Run Range on HM3D v0.1
- Across three HM3D v0.1 runs, SR ranges from 62.3–62.8 and SPL from 31.2–31.7; these marginal ranges are not confidence intervals or standard deviations.The range applies only to the main Gemini-1.5-Pro configuration; its SPL lower bound equals WMNav’s point estimate.Source: §D.5 · p. 13
D.6 Further Evaluation
- A fixed-state replay is proposed to characterize the direct numerical matching channel, but it cannot estimate closed-loop SR or SPL.The replay would hold the prior, graph, coarse scores, phase, and LLM output constant.Source: §D.6 · pp. 13–14
- Further evaluation could compare memory and scoring configurations under matched episodes and budgets, and test robustness to depth, localization, unusual layouts, dynamic scenes, and physical-robot settings.The authors also identify annotations of relation updates, revisitation, and failure categories as useful for explaining successes and failures.Source: §D.6 · p. 14
E Limitations and Future Work
The authors identify limits in the evaluation’s simulation scope and component attribution, alongside assumptions and deployment challenges. They propose broader robustness and physical-robot studies.
E Limitations and Future Work
- No physical-robot results or controlled noise-robustness tests are available, and backbone sensitivity covers only two VLM settings on one split.Source: E Limitations and Future Work · p. 14
- Memory baselines change representation and scoring together, while component interactions and statistical uncertainty remain unestablished.Source: E Limitations and Future Work · p. 14
- The framework assumes static scenes; dynamic objects or movable furniture could violate the structural priors encoded in Ggoal.Source: E Limitations and Future Work · p. 14
- Performance depends on foundation-model capabilities, detection errors can propagate into graph construction, and heuristic pruning can remove correct atypical observations.Source: E Limitations and Future Work · p. 14
- Realtime deployment remains challenging because VLM inference takes 2.3 seconds per call despite caching; temporal graph updates, multiple targets, active perception, and robot trials remain future directions.Source: E Limitations and Future Work · p. 14
F Implementation Availability
The implementation materials document code, experiment records, and archived navigation cases. The case collection preserves qualitative traces but is not a benchmark sample for estimating performance.
F Implementation Availability
- The code directory includes configuration files, prompt builders, ablation settings, analysis scripts, and implementation notes; experiment records include configuration and episode identifiers and model-call statistics.The authors state that benchmark datasets and model services remain subject to their providers’ terms.Source: F Implementation Availability · p. 14
G Archived Navigation Cases
- The archive covers 18 cases, indexes 338 step directories, and retains 337 available text logs; selected images preserve original bytes under a deterministic first/middle/last-step rule.Additional frames and GIFs are omitted, and one text record is missing.Source: G Archived Navigation Cases · pp. 14–15
- Archived traces expose prompts and model responses for qualitative inspection, but goal flags are not verified simulator successes and the cases cannot estimate SR, SPL, or failure frequencies.The index identifies missing text records and retains error-marked steps.Source: G Archived Navigation Cases · p. 15