PaperlayerView exact revision on arXiv

Source-linked AI summary

STEGNav: Spatio-Temporal Event Graph Reasoning for Multimodal Lifelong Object Navigation

Yang Chen, Zhenyu Huang, Wenbo Fu, Danyang Peng, Shi-Yu Tian, Kun-Yang Yu, Lan-Zhe Guo

arXiv:2608.28279v1cs.RO

TL;DR

Multimodal lifelong navigation exposes limits in state-centric scene graphs, particularly for instance distinction, frontier integration, and experience reuse. STEGNav introduces a training-free spatio-temporal event graph with query-conditioned spatial grounding and dual-window temporal memory, achieving strong results across GOAT-Bench and HM3D. The authors conclude that this representation supports instance-aware grounding, exploration, and cross-subtask experience reuse, while tighter integration with low-level control remains future work.

  • Problem

    State-centric scene graphs struggle to distinguish similar instances, jointly represent semantic targets and exploration frontiers, and exploit navigation memory and trajectory experience.

  • Method

    STEGNav is a training-free framework that combines query-conditioned instance and frontier grounding with trajectory-aware dual-window memory in a spatio-temporal event graph.

  • Results

    STEGNav improves navigation reliability across multimodal lifelong navigation and category-level ObjectNav, achieving 66.3% SR on GOAT-Bench and highest SR on both HM3D benchmarks.

  • Takeaways & Limitations

    The combined spatial and temporal representation supports instance-aware target grounding, exploration, and reuse of verified cross-subtask navigation experience.

  • Takeaways & Limitations

    Future work must better integrate high-level event-graph reasoning with low-level navigation control to improve navigation efficiency.

Abstract

from arXiv · show

Multimodal lifelong navigation requires an agent to autonomously explore unseen environments while sequentially completing navigation tasks specified by object categories, language descriptions, or reference images. Existing methods primarily accomplish these tasks by constructing state-centric semantic scene graphs. By treating scene graphs as persistent repositories of semantic observations, these methods struggle to distinguish similar instances, jointly represent semantic targets and exploration frontiers, and effectively exploit navigation memory and trajectory experience. To address these limitations, we propose Spatio-Temporal Event Graph Navigation (STEGNav), a training-free framework that extends conventional scene graphs into spatio-temporal event graphs along complementary spatial and temporal axes. The spatial axis performs query-conditioned instance grounding and jointly represents semantic targets and occupancy-aware exploration frontiers characterized by reachability, path cost, and exploration utility. The temporal axis employs trajectory-aware dual-window memory to retain recent decision--trajectory events and verified cross-subtask navigation outcomes. A VLM-based navigation agent reasons over the resulting spatio-temporal event graph and selects either a target instance or an exploration frontier as its next navigation goal. STEGNav achieves 66.3% SR and 39.7 SPL on GOAT-Bench, as well as SR scores of 64.0% and 69.4% on HM3Dv1 and HM3Dv2, respectively. Ablation studies and error analyses validate the complementary effects of the two axes, demonstrating that event-driven spatio-temporal representations improve navigation reliability and cross-subtask experience reuse.

Introduction

Multimodal lifelong navigation challenges state-centric scene graphs with instance ambiguity, separated exploration information, and weak navigation-history reuse. STEGNav addresses these gaps through complementary spatial and temporal event-graph mechanisms and reports improved benchmark performance.

  • State-centric scene graphs struggle to distinguish individual instances, represent target-state spatial information, and reuse prior decision experience.
  • STEGNav transforms scene graphs into spatio-temporal event graphs through query-conditioned instance grounding and occupancy-aware frontier integration.The spatial graph lets the navigation VLM select either a target instance or an exploration frontier.
  • Dual-window memory organizes recent decisions and trajectories alongside verified outcomes from previous subtasks.The short-term window captures recent navigation events, while the long-term window retains verified cross-subtask results.
  • 66.3% SR and 39.7 SPL on GOAT-Bench Val-Unseen, with SR improving 3.9 percentage points over the strongest competing method.
  • The framework is training-free and is evaluated for multimodal lifelong navigation on GOAT-Bench and category-level ObjectNav generalization on HM3D.

Related Work

Object navigation has progressed from supervised, distribution-limited policies and isolated target search toward long-horizon multimodal tasks. STEGNav builds on scene-graph representations by adding spatially integrated exploration information and temporally reusable experience.

  • Early object-navigation methods often relied on reinforcement or imitation learning, requiring substantial supervision and showing limited unseen-environment generalization.
  • Zero-shot object navigation has expanded from isolated single-target search to long-horizon navigation with multimodal target sequences.
  • Scene graphs encode semantic and spatial relations among objects and regions, supporting high-level reasoning and long-horizon planning.
  • STEGNav extends state-centric scene graphs by associating queries with candidate instances, integrating navigation status and exploration utility, and retrieving relevant outcomes.
  • The spatial and temporal extensions form a reusable, event-driven representation for multimodal lifelong navigation.

Preliminaries

The navigation setting combines multimodal subtasks, online scene-graph and occupancy-map construction, and VLM-based target selection. Existing category filtering and separated frontier representations leave instance identity and joint navigation reasoning unresolved.

  • Each subtask specifies a target category and modality, while success requires stopping within 1.0 m of an instance satisfying the query.Modalities include object, language, and image queries.
  • The agent maintains an online semantic scene graph and accumulates RGB-D observations into an occupancy grid distinguishing free, occupied, and unexplored regions.
  • Unexplored boundaries are clustered into frontier regions whose exploration gains are estimated from surrounding unexplored space.
  • A VLM category prefilter retains scene nodes relevant to the current target, but it cannot distinguish different physical instances within one category.
  • Separating semantic nodes from occupancy-derived frontiers prevents joint reasoning over target identity, traversability, and exploration utility.

Method

STEGNav builds a query-conditioned spatial graph that unifies target instances with occupancy-aware exploration frontiers, then adds temporal memory for recent trajectories and verified cross-subtask outcomes. The resulting event graph exposes these elements jointly to a VLM, which selects navigation goals from a unified representation.

  • Spatial axis: The spatial axis constructs a query-conditioned semantic–navigational graph containing semantic instances, navigation relations, and exploration gains.
  • Spatial axis: Query-conditioned instance grounding associates multimodal queries with individual scene instances, while category queries remain unpruned at the instance level.Image and language queries use visual, linguistic, and spatial context; category-level queries do not perform instance-level pruning.
  • Spatial axis: Occupancy-derived frontier nodes encode spatial centers, reachable path distances, estimated exploration gains, and recent visitation states.Frontier–semantic edges describe the local semantic context reached through each frontier, while navigation edges record reachability and geodesic path cost.
  • Graph serialization: Candidate objects and exploration frontiers are represented within the same graph instead of being supplied to the VLM as separate inputs.The serialized graph exposes query-conditioned instance descriptions, relevance scores, reachability, frontier utility, recent trajectories, and verified historical outcomes.
  • Temporal axis: The temporal axis maintains recent navigation decisions and trajectories in a short-term window while retaining verified outcomes from previous subtasks in a long-term window.Short-term events store selected nodes, trajectory segments, traversed regions, path lengths, and outcomes; the short-term window is cleared when the subtask ends.
  • Temporal axis: Long-term outcomes are VLM-verified without simulator ground truth and stored as recall anchors that are re-associated with the current scene graph.Verification uses the query, terminal visual evidence, and distance to the selected node; accepted anchors retain instance, category, modality, query, appearance, and room information.

Experiments

Experiments evaluate STEGNav on multimodal lifelong navigation, category-level ObjectNav, component ablations, and failure causes. Results show strong success-rate gains, complementary component effects, and reduced high-level navigation failures, while low-level control remains a limitation.

  • Benchmarks: GOAT-Bench Val-Unseen contains 36 scenes with 5–10 sequential subtasks per episode, testing multimodal grounding, long-horizon exploration, and cross-subtask experience reuse.The benchmark uses object, language, and image queries.
  • Main Results: 66.3% SR and 39.7 SPL were achieved on GOAT-Bench Val-Unseen, with SR 3.9 percentage points above MSGNav and SR/SPL gains of 6.7%/0.8 over EvoMemNav.MSGNav is compared only on SR because it does not report SPL under the adopted protocol.
  • Main Results: 64.0%/29.8 SR/SPL on HM3Dv1 and 69.4%/28.2 on HM3Dv2 gave STEGNav the highest SR on both benchmarks, although not the best SPL.The results support generalization to category-level ObjectNav, while path efficiency remains weaker on HM3D.
  • Ablation Study: QIG raised overall SR/SPL to 59.4%/36.2, while OFG further raised them to 66.9%/41.1 and increased image-query SR/SPL by 10.2%/5.8.QIG targets same-category instance distinction; OFG links targets with reachable frontiers and exploration gains.
  • Ablation Study: DWM produced the best ablation result at 70.9%/42.7 overall SR/SPL and increased image-query SR from 68.2% to 73.9%.The complete model outperformed the standard 3D scene-graph variant by 15.1% SR and 9.3 SPL.
  • Error Analysis: STEGNav reduced failures from 123 to 81, including 40.0% fewer inefficient-exploration failures and 53.8% fewer instance-confusion failures.Execution and stopping failures decreased only from 18 to 14, leaving low-level control and termination as residual error sources.

Conclusion

STEGNav extends scene graphs with complementary spatial and temporal axes to support instance-aware grounding, efficient exploration, and cross-subtask experience reuse. Experiments show improved success, path efficiency, and category-level ObjectNav generalization, while tighter integration with low-level control remains future work.

  • STEGNav transforms static scene graphs into event-driven representations spanning target instances, occupancy-aware frontiers, reachability, and navigation experience.Its spatial axis models semantic–navigational structure, while its temporal axis preserves decision–trajectory events and verified outcomes.
  • The dual-axis design supports instance-aware target grounding, efficient exploration, and reuse of verified navigation outcomes across subtasks.
  • GOAT-Bench experiments improve both navigation success and path efficiency, while HM3Dv1 and HM3Dv2 results validate category-level ObjectNav generalization.
  • Future work will integrate high-level event-graph reasoning more tightly with low-level navigation control to improve efficiency.

Implementation Details

STEGNav’s implementation builds an online graph from RGB-D observations, occupancy mapping, query-conditioned grounding, frontier construction, and dual-window trajectory memory. A VLM selects goals using this serialized representation, with verification and persistent historical associations supporting navigation reuse.

  • Overall Navigation Procedure: At each high-level step, RGB-D observations update the semantic scene graph and occupancy map before spatial and temporal graph augmentation.The VLM then selects either an object or frontier goal for navigation.
  • Perception and Online Mapping: The system uses YOLOv8x-World, SAM-L, CLIP ViT-H-14-quickgelu, persistent 3D object nodes, 3.5 m graph connections, and a 0.1 m TSDF grid.
  • Query-Conditioned Instance Grounding: Instance grounding retains up to 20 category-relevant nodes, pre-ranks at most 10 candidates, and keeps the five highest-scoring instances, with CLIP fallback when VLM grounding is unavailable.
  • Query-Conditioned Instance Grounding: Category-level queries bypass instance pruning and re-ranking, whereas image and description queries change selections only when another candidate improves similarity by at least 0.05.
  • Occupancy-Aware Frontier Grounding: Frontier nodes encode reachability, geodesic path cost, exploration gain, visitation state, and visual context, enabling common comparison with semantic targets.Frontiers connect to nearby semantic anchors on the current traversable island.
  • Trajectory-Aware Dual-Window Memory: The short-term window stores the latest 20 decisions and is cleared per subtask, while historical anchors are re-associated using persistent identifiers, CLIP similarity, and VLM verification.
  • Trajectory-Aware Dual-Window Memory: Verification uses observations and the current scene graph rather than simulator ground truth, and successful goals are checked within the navigation procedure’s stated distance criteria.
  • Core Prompt Templates: The implementation uses one gpt-5.4-mini completion per request without majority voting or self-consistency sampling.

Case Study

The case studies show STEGNav combining frontier exploration with semantic and spatial reasoning for language-guided instance grounding and category-level object navigation. Both examples successfully reach their targets with reported path-efficiency scores.

  • Case Study: Together, the examples illustrate unified exploration and target-oriented navigation across language-guided instance and category-level object tasks.
  • Language-guided target grounding: The language-guided case progressively explores informative regions before grounding “refrigerator next to the kitchen cabinet” using semantic and spatial cues.It reaches the target within 0.40 m with an SPL of 0.587 in 4 navigation steps.
  • Object navigation with exploration: The object-navigation case selects informative frontiers, discovers the piano’s region, and then approaches a suitable target viewpoint in the unseen environment.The trajectory reaches the target within 0.97 m with an SPL of 0.913 in 6 navigation steps.
Loading 2608.28279v1…