Source-linked AI summary
SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigation
Jiwen Zhang, Zejun Li, Siyuan Wang, Xiangyu Shi, Zhongyu Wei, Qi Wu
TL;DR
Zero-shot VLN agents lack the learned spatial priors of supervised systems and therefore rely on limited local observations. The paper enables pre-exploration, builds a Spatial Scene Graph, and introduces SpatialNav with structured spatial and visual tools; experiments show strong gains over prior zero-shot agents across discrete and continuous environments. The approach narrows the gap with learning-based methods, while its integration with supervised agents and reconstruction pipeline remain open limitations.
Problem
Zero-shot VLN agents rely primarily on local observations without the spatial priors learned from large-scale training, leaving a significant performance gap versus learning-based methods.
Method
The paper allows pre-exploration, constructs a Spatial Scene Graph of global layout and semantics, and uses it through an agent-centric map, compass-like visual representation, and remote object localization.
Results
57.7%, 49.6%, 64.0%, and 32.4% success rates are achieved on val-unseen R2R, REVERIE, R2R-CE, and RxR-CE, surpassing previous zero-shot agents and several supervised-learning methods.
Takeaways & Limitations
Spatial scene graphs enable zero-shot navigation performance comparable to learning-based methods across environments and agents.
Takeaways & Limitations
Integration of spatial scene graphs into supervised agents remains unexplored, and the reconstruction pipeline is not fully evaluated for robustness or operational cost.
Abstract
from arXiv · showhide
Although learning-based vision-and-language navigation (VLN) agents can learn spatial knowledge implicitly from large-scale training data, zero-shot VLN agents lack this process, relying primarily on local observations for navigation, which leads to inefficient exploration and a significant performance gap. To deal with the problem, we consider a zero-shot VLN setting that agents are allowed to fully explore the environment before task execution. Then, we construct the Spatial Scene Graph (SSG) to explicitly capture global spatial structure and semantics in the explored environment. Based on the SSG, we introduce SpatialNav, a zero-shot VLN agent that integrates an agent-centric spatial map, a compass-aligned visual representation, and a remote object localization strategy for efficient navigation. Comprehensive experiments in both discrete and continuous environments demonstrate that SpatialNav significantly outperforms existing zero-shot agents and clearly narrows the gap with state-of-the-art learning-based methods. Such results highlight the importance of global spatial representations for generalizable navigation.
1. Introduction
Zero-shot VLN agents rely on local observations and lack the spatial priors learned by supervised agents, making disambiguation and exploration difficult. SpatialNav addresses this through pre-exploration, a Spatial Scene Graph, and three navigation components, achieving strong results across environments.
- Zero-shot VLN agents generalize without task-specific training but still show a significant performance gap versus learning-based methods.
- Local observations leave zero-shot agents without spatial priors, causing short-sighted decisions and difficulty distinguishing multiple instruction-consistent actions.
- The proposed setting allows agents to fully explore environments before task execution and encodes reconstructed layouts and semantics in a Spatial Scene Graph.
- SpatialNav combines an agent-centric spatial map, compass-like visual representation, and remote object localization for navigation.
2. Related Works
VLN research spans supervised navigation in discrete and continuous environments, zero-shot agents built with (M)LLMs, and exploration-based methods. SpatialNav builds on these directions by organizing pre-explored environments into a spatial scene graph.
- Supervised VLN remains the mainstream paradigm, but large-scale domain-specific training limits generalization to unseen environments.
- Zero-shot VLN agents use prompting strategies such as action-aware reasoning, deliberative planning, progress estimation, and mistake reflection, yet still lag behind learning-based agents.
- SpatialNav extends these directions by constructing a spatial scene graph from pre-exploration to encode structured spatial and semantic information.
- Prior exploration methods use successful trajectories, topological-map planners, or pre-explored videos to support navigation and goal identification.
3. Methods
The method converts pre-explored point clouds into a hierarchical spatial scene graph and equips SpatialNav with structured, orientation-consistent, and future-aware representations. These components support long-horizon decisions beyond local perception.
- Spatial Scene Graph: The problem setup assumes pre-exploration and enriches reusable point clouds with floor–room–object structure and semantic labels.
- Spatial Scene Graph: The spatial scene graph is built through floor segmentation, room segmentation, room classification, and object detection before organizing entities hierarchically.
- SpatialNav Agent: SpatialNav queries the graph to capture surrounding-room layout, align egocentric views with spatial orientation, and retrieve future object semantics.
- Agent-centric Spatial Map: The agent-centric map uses the current position to select nearby rooms and projects them into a top-down representation with heading aligned upward.
- Compass-like Visual Observation: Compass-like observations discretize panoramas into eight 90° field-of-view directional views and organize them into one representation.
- Remote Object Localization: Remote object localization retrieves object categories and distances near candidate navigable places and adds them as concise textual context.
- SpatialNav Agent: Together, these components enable long-horizon, goal-aware decisions beyond local perception.
4. Experiments
Experiments evaluate SpatialNav across discrete and continuous VLN environments, comparing it with supervised and zero-shot baselines and analyzing spatial-map, annotation, backbone, and representation choices. SpatialNav consistently improves over zero-shot methods, while performance depends on spatial context, annotation quality, and visual grounding.
- Experimental Setup: Experiments use R2R and REVERIE in discrete environments and R2R-CE and RxR-CE in continuous environments, reporting standard VLN metrics.The metrics include TL, NE, SR, OSR, SPL, and nDTW; success requires stopping within 3 meters of the goal.
- Experimental Setup: SpatialNav is compared with supervised and zero-shot agents, including state-of-the-art baselines within each group and a ground-truth-annotation upper-bound variant.SpatialNav† replaces predicted spatial annotations with ground-truth room and object annotations.
- Main Results: +9.3% SR and +11.7% SPL are achieved over SpatialGPT on R2R val-unseen, while SpatialNav consistently outperforms zero-shot baselines across discrete datasets.Its zero-shot performance is comparable with several supervised agents, including PREVALENT, VLN-BERT, and DUET.
- Main Results: +21.6% SR and +24.8% SPL are achieved over VLN-Zero on R2R-CE, with further gains over prior zero-shot methods on RxR-CE.SpatialNav uses a spatial scene graph that jointly models global layout and detailed semantics, unlike VLN-Zero’s symbolic scene graph.
- Main Results: Ground-truth spatial annotations further improve performance across four datasets, showing that annotation quality strongly affects navigation performance.The results motivate more accurate and scalable automatic spatial annotation methods.
- Further Analysis: Spatial-map augmentation improves GPT-5.1 and Gemini-2.5-Pro results but slightly degrades Qwen3-VL-Plus-based NavGPT performance.This comparison tests whether spatial-map effectiveness depends on the MLLM backbone.
- Ablation Studies: The 1024×1024 compass-style representation uses about 640 visual tokens, versus over 1700 for eight sequential views, while achieving navigation accuracy suitable for the default setting.Sequential views achieve the best overall performance but incur higher latency and cost.
- Ablation Studies: A 7.68m spatial-map radius provides the best balance, whereas 3.84m adds little information and 11.52m introduces distracting, instruction-irrelevant structures.The 7.68m setting achieves the strongest SR, SPL, and nDTW; the larger radius slightly decreases SR despite higher OSR.
5. Conclusions
SpatialNav pre-explores environments to construct spatial scene graphs and uses them for zero-shot navigation. Experiments show generalization across environments and agents, with performance comparable to learning-based methods.
- 5. Conclusions: SpatialNav pre-explores environments to construct spatial scene graphs containing global spatial information, then uses those graphs for zero-shot navigation.The method is reported to generalize across environments and agents.
Limitations
SpatialNav’s limitations concern integration beyond zero-shot agents, the computational and operational cost of constructing spatial scene graphs, and challenges in room-level segmentation.
- Integrating spatial scene graphs into supervised learning-based VLN agents remains unexplored because existing methods may not support top-down map-structured inputs.The authors propose collecting spatial-map annotations and redesigning training to expose agents to this information.
- Constructing a full spatial scene graph introduces additional computation overhead, and the reconstruction pipeline’s robustness and failure cases are not evaluated.The setting assumes point clouds are already available despite potentially expensive 3D reconstruction during pre-exploration.
- Room-level segmentation remains challenging in open or ambiguously bounded spaces, where automated methods may require manual correction.The authors characterize this as consistent with real-world robotic mapping, where users can refine room layouts after map construction.