Source-linked AI summary
CGFM-Nav: Cognitive Graph-Field Memory for Semantic-Guided Lifelong Multimodal Embodied Navigation
Yuxiang Xiao, Xibei Chen, Xin Zhou, Jie Chen, Yifeng Zhang, Guillaume Sartoretti
TL;DR
VLN environment representations often separate explicit semantic memory from continuous exploration guidance, limiting their joint support for lifelong navigation. CGFM couples a persistent multimodal scene graph with a goal-conditioned semantic-frontier field and builds CGFM-Nav around closed-loop VLM reasoning and verification. On GOAT-Bench, CGFM-Nav improves overall success rate and SPL under the same Qwen3-VL-8B backbone, while image-goal performance remains more dependent on fine-grained visual correspondence.
Problem
Existing VLN representations typically provide either exploration-oriented maps without explicit object relations or discrete graphs without continuous exploration guidance.
Method
CGFM couples a persistent multimodal scene graph with a goal-conditioned semantic-frontier field, while CGFM-Nav adds task-relevant subgraph selection, VLM reasoning, and verification feedback.
Results
Under the same Qwen3-VL-8B backbone, CGFM-Nav improves overall SR from 53.2% to 63.0% and SPL from 30.0% to 39.6%.
Takeaways & Limitations
Combining explicit semantic memory with semantic-guided exploration provides an effective representation for lifelong multimodal embodied navigation.
Takeaways & Limitations
Image-goal navigation remains below MSGNav with GPT-4o, indicating continued dependence on stronger visual representations or navigation-oriented VLM backbones.
Abstract
from arXiv · showhide
Vision-and-Language Navigation (VLN) requires agents to reason over accumulated observations while continuously exploring unseen regions. However, existing environment representations often struggle to jointly support explicit semantic memory and continuous exploration guidance. To address this challenge, we propose Cognitive Graph-Field Memory (CGFM), a persistent multimodal scene representation that couples explicit relational memory with continuous spatial intuition. CGFM organizes objects, spatial relations, and visual observations into a multimodal scene graph, enabling target retrieval and long-horizon reasoning across navigation tasks. When no reliable target match is identified, graph-based evidence is projected into a goal-conditioned semantic-frontier field to guide exploration toward semantically promising frontiers and regions. Building upon CGFM, we introduce CGFM-Nav, a foundation-model-based framework for lifelong multimodal navigation that integrates task-relevant subgraph selection, VLM reasoning, and verification feedback into a closed decision loop. Preliminary experiments on GOAT-Bench show that, under the same Qwen3-VL-8B backbone, CGFM-Nav improves the overall success rate from 53.2% to 63.0% and SPL from 30.0% to 39.6%, demonstrating the effectiveness of combining explicit semantic memory with semantic-guided exploration.
2 Implicit Semantic Intuition Field
CGFM couples explicit scene-graph memory with a continuous semantic-frontier field. The graph supports retrieval of known targets, while projected evidence guides exploration when no reliable match exists.
- 2 Implicit Semantic Intuition Field: When no reliable target match is identified, graph evidence is projected into a semantic-frontier field that guides exploration toward promising regions.The field indicates areas that are both semantically relevant to the goal and worth exploring.
- 2 Implicit Semantic Intuition Field: Existing 2D maps support exploration but lack explicit relations, whereas graph representations support retrieval but provide limited continuous exploration guidance.The paper identifies this joint capability as an unresolved challenge in training-free VLN environment representations.
- 2 Implicit Semantic Intuition Field: CGFM combines explicit relational memory with continuous spatial intuition in a unified graph-field representation.The scene graph and semantic field are complementary components of the same cognitive representation.
- 2 Implicit Semantic Intuition Field: The multimodal scene graph stores objects, spatial relations, and visual observations for target retrieval and long-horizon reasoning.When a task-relevant target node exists, the agent navigates to it directly.
- 2 Implicit Semantic Intuition Field: CGFM-Nav combines task-relevant subgraph selection, VLM reasoning, and verification feedback in a continually updated navigation loop.The framework executes either goal navigation or frontier/semantic exploration guided by CGFM.
A. Problem Formulation and Overview
CGFM-Nav addresses lifelong open-vocabulary navigation by retaining a persistent scene graph, reconstructing a goal-conditioned field, and maintaining short-term decision memory across subtasks.
- A. Problem Formulation and Overview: Each episode contains K sequential subtasks with object-category, language, or reference-image goals in an initially unknown environment.The agent operates without task-specific policy training while retaining accumulated environment memory across subtasks.
- A. Problem Formulation and Overview: At each decision cycle, RGB-D observations support selection of a high-level navigation destination, followed by low-level execution until verification or budget exhaustion.The high-level destination is denoted h_t, while primitive actions are executed by a low-level planner.
- A. Problem Formulation and Overview: The maintained state comprises a persistent multimodal scene graph G_t, a goal-conditioned semantic-frontier field F_t^(k), and short-term decision memory D_t^(k).These components jointly represent persistent environment knowledge, current-goal exploration guidance, and recent decisions.
- A. Problem Formulation and Overview: At each new subtask, the graph and verification history persist, the field is reconstructed for the new goal, and decision memory is reset.Rejected candidates are recorded in graph nodes and suppressed in the field rather than tracked separately.
- A. Problem Formulation and Overview: CGFM-Nav’s architecture forms a closed-loop process in which navigation and verification outcomes update the maintained state.The overall architecture and closed-loop navigation process are illustrated in Fig. 2.
1) Perception and Multimodal Scene Graph Construction:
The framework incrementally builds a multimodal scene graph from RGB-D observations and projects its goal-relevant evidence into a synchronized semantic field for frontier selection.
- 1) Perception and Multimodal Scene Graph Construction:: RGB-D frames are processed for open-vocabulary detection, segmentation, and visual-semantic features before association into a multimodal scene graph.Graph nodes store object category, 3D location, visual-semantic features, and accumulated observation images; edges encode spatial or semantic relations.
- 2) Semantic-Frontier Field Construction:: The semantic field converts graph evidence into exploration values for previously unobserved regions.Each graph node receives goal relevance from CLIP similarity, with background subtraction and verification gating.
- 2) Semantic-Frontier Field Construction:: CGFM-Nav stores RGB-D observations in a structural scene graph whose goal-conditioned evidence constructs the semantic-frontier field.The field, compact key subgraph, and decision memory support object, image, frontier, and semantic exploration decisions.
- 2) Semantic-Frontier Field Construction:: Field updates remain synchronized with the evolving graph by recomputing only cells near changed object projections.The rest of the field is retained unchanged during local updates.
- 2) Semantic-Frontier Field Construction:: Frontier candidates are scored by nearby semantic-field values and geodesic travel distance, favoring closer and more relevant frontiers.If the best score is below a confidence threshold, the agent falls back to the nearest frontier.
C. CGFM-Guided Navigation
CGFM-Nav uses CGFM-guided subgraph selection to provide the VLM with compact, task-relevant environment evidence for navigation reasoning.
- CGFM-Nav combines memory retrieval, VLM reasoning, action selection, and closed-loop refinement around the CGFM representation.These four modules support zero-shot VLN through the unified graph-field memory.
- Goal-conditioned relevance supplies semantic seed nodes, while VLM selection adds candidates supported by relational or commonsense cues.The residual graph is used for complementary selection beyond direct CLIP similarity.
- The selected node set is processed with edge pruning and image allocation to form the key subgraph used for navigation.
2) VLM Reasoning with Decision Memory:
The VLM reasons over a selected subgraph together with a compressed decision history, choosing among object targets, image targets, and exploration.
- The decision log records the cycle, destination type, candidate identity and category, and a compact reasoning excerpt.It preserves short-term temporal information without accumulating raw VLM responses.
- Before prompting, the decision log is compressed into previously visited or rejected candidates and supplied to the VLM.
- The VLM selects an object node, an observation image, or an EXPLORE request.This separates navigation toward graph-represented targets from semantic-frontier-guided exploration.
3) Action Selection:
Action selection converts VLM decisions into object, image, or exploration destinations, while verification feedback updates memory and suppresses rejected-object guidance.
- Action selection: Object candidates use stored node positions, whereas image candidates are re-detected and localized by YOLOv8-World and the VLM.
- Action selection: Exploration selects the highest-scoring frontier, falls back to the nearest frontier when scores are weak, and uses semantic peaks when no frontier remains.
- Verification: Verification returns YES, NO-UNCERTAIN, or NO-CONFIRMED, completing, deferring, or rejecting the subtask.Candidates are rejected after all viewpoints are exhausted without a positive response.
- Closed-loop update: Navigation outcomes and verification feedback enter decision memory, while rejected object nodes stop contributing to the semantic field for the remaining goal.
III. EXPERIMENTS AND RESULTS
The evaluation uses GOAT-Bench’s Val Unseen split and compares MSGNav and CGFM-Nav under a shared Qwen3-VL-8B backbone using Success Rate and SPL.
- Experiment settings: The evaluation covers 36 episodes with 278 subtasks from the first episode of each scene in GOAT-Bench’s Val Unseen split.GOAT-Bench contains 360 episodes across 36 scenes and 2669 subtasks overall.
- Metrics: Success Rate measures the fraction of evaluated tasks that succeed, while SPL weights success by the ratio of shortest-path distance to traveled path length.
- Comparison: MSGNav and CGFM-Nav use Qwen3-VL-8B-Instruct as their VLM backbone, while original MSGNav results use GPT-4o.
B. Results
CGFM-Nav improves multimodal navigation performance on GOAT-Bench and combines structured scene-graph memory with semantic-guided exploration. The remaining image-goal gap indicates that fine-grained visual correspondence remains important.
- Results: 63.0% overall SR and 39.6% SPL improve over 53.2% SR and 30.0% SPL under the same Qwen3-VL-8B backbone.The gains are reported on the Val Unseen subset of GOAT-Bench.
- Results: CGFM-Nav achieves consistent gains across category, language, and image goals by combining scene-graph memory with semantic-frontier guidance.The representation reuses previous observations while directing exploration toward semantically promising regions.
- Results: CGFM-Nav with Qwen3-VL-8B outperforms MSGNav with GPT-4o in overall, category, and language performance.This comparison suggests that environment representation can compensate for part of the performance gap between VLM backbones.
- Results: Image-goal navigation remains constrained by fine-grained visual correspondence despite semantic-guidance improvements.The authors identify stronger visual representations or navigation-oriented VLM backbones as likely directions for further improvement.
- Results: CGFM-Nav is presented as a training-free framework integrating structured scene-graph memory with semantic-guided exploration.The conclusion reports consistent navigation improvements on GOAT-Bench and frames explicit semantic memory plus spatial intuition as an effective cognitive representation.