Source-linked AI summary
Structured Scene Memory for Vision-Language Navigation
Hanqing Wang, Wenguan Wang, Wei Liang, Caiming Xiong, Jianbing Shen
TL;DR
VLN agents commonly rely on latent recurrent memory, which limits precise access to past observations, environment layouts, and long-term planning. The paper introduces Structured Scene Memory (SSM), a structured external memory with disentangled visual-geometric representation, iterative reasoning, and frontier-based global decision making. SSM achieves state-of-the-art performance on R2R and R4R across several metrics.
Problem
Current VLN agents use latent recurrent memory, limiting precise access to past observations, environment-layout exploration, and long-term planning.
Method
SSM externally stores scene layouts and past percepts, separates visual and geometric cues, supports iterative reasoning, and uses frontier exploration over a global action space.
Results
The SSM-based agent achieves state-of-the-art performance on the R2R and R4R datasets across several metrics.
Takeaways & Limitations
Structured, persistent scene memory supports fine-grained instruction grounding, long-term reasoning, and global decision making in VLN.
Abstract
from arXiv · showhide
Recently, numerous algorithms have been developed to tackle the problem of vision-language navigation (VLN), i.e., entailing an agent to navigate 3D environments through following linguistic instructions. However, current VLN agents simply store their past experiences/observations as latent states in recurrent networks, failing to capture environment layouts and make long-term planning. To address these limitations, we propose a crucial architecture, called Structured Scene Memory (SSM). It is compartmentalized enough to accurately memorize the percepts during navigation. It also serves as a structured scene representation, which captures and disentangles visual and geometric cues in the environment. SSM has a collect-read controller that adaptively collects information for supporting current decision making and mimics iterative algorithms for long-range reasoning. As SSM provides a complete action space, i.e., all the navigable places on the map, a frontier-exploration based navigation decision making strategy is introduced to enable efficient and global planning. Experiment results on two VLN datasets (i.e., R2R and R4R) show that our method achieves state-of-the-art performance on several metrics.
1. Introduction
Current VLN agents encode past perceptions in latent recurrent states, limiting access to prior observations, environment layouts, and local decision making. The paper proposes SSM, an external structured memory that supports disentangled scene representation, long-term reasoning, and global planning.
- Current Seq2Seq VLN models mix perceived information in recurrent units, restricting access to past observations and environment layouts.
- Latent recurrent memory and local action spaces make long-range planning difficult in partially observed environments.
- SSM is an external graph-structured memory that stores percepts, represents environment layouts, and provides a global action space over explored locations.The agent can move from its current direction to a previously visited distant location when needed.
- SSM separates visual information in location nodes from geometric relations in edges, supporting instruction grounding to perception and action.
- Experiments on R2R and R4R demonstrate the effectiveness of the full approach and its core model designs.
2. Related Work
Prior VLN work has advanced learning paradigms, supervision, representation learning, and instruction grounding, but recurrent agents still struggle with precise long-term memory, layout exploration, and global action selection. SSM addresses these limitations with structured scene storage and frontier-based planning.
- VLN research has explored imitation learning, reinforcement learning, auxiliary supervision, representation learning, instruction grounding, and self-pretraining.
- Existing Seq2Seq agents encode past observations as recurrent hidden states, making precise long-term memorization and environment-layout exploration difficult.
- SSM stores scene representations in structured memory, provides a global action space, and facilitates long-term reasoning.
- Frontier-exploration decision making addresses the expanding action space that otherwise makes policy learning difficult and convergence slow.
3. Our Approach
The approach replaces reactive latent-state memory with Structured Scene Memory (SSM), an external graph that stores explored places and supports global navigation decisions. Its collect-read controller grounds instructions in disentangled visual and geometric representations, then performs iterative reasoning and frontier-based exploration.
- Existing VLN agents use recurrent latent states and local current-view actions, limiting precise long-term recall, layout exploration, and correction of wrong decisions.
- SSM maintains an online directed graph whose nodes represent visited places and whose edges encode connections between them.
- SSM disentangles visual information in node embeddings from geometric orientation cues in edge embeddings for instruction grounding.
- SSM expands the action space to navigable sub-nodes across the explored map, allowing the agent to jump to previously observed locations rather than relying on localized backtracking.
- A collect-read controller selects perception- and action-relevant content from SSM and uses iterative message passing to capture S-hop neighborhood context.
- Frontier exploration first selects a frontier node as a sub-goal and then chooses one of its navigable sub-nodes, reducing the difficulty of learning over a growing global action space.
4. Experiment
Experiments on R2R and R4R show that SSM improves navigation performance and supports robust long-range decisions through structured memory and global action spaces. Ablations attribute gains to disentangled scene representation, iterative reasoning, and frontier-exploration-based decision making.
- Performance on R2R Dataset: SSM outperforms competing VLN models across most R2R metrics, including SR, SPL, NE, TL, and OR on multiple validation and test splits.It achieves SR 0.61 on R2R test unseen versus 0.53 for EGP, while improving SPL across validation seen, validation unseen, and test unseen.
- Visual Results: SSM enables recovery from ambiguous or initially misleading instructions by accessing past observations and planning over explored locations.In representative R2R and R4R cases, the agent redirects toward critical landmarks or returns to the correct direction and reaches the target.
- Performance on R4R Dataset: SSM achieves CLS 0.65 on R4R validation seen and 0.53 on validation unseen, compared with 0.60 and 0.44 for the second-best method.The model also gains on SR, nDTW, and SDTW, with a CLS seen-to-unseen gap of 0.12.
- Diagnostic Experiments: Disentangled visual and geometric scene representations with fine-grained instruction grounding improve VLN performance over corresponding ablations.SSM separates visual information in nodes from geometric relations in edges for instruction grounding.
- Diagnostic Experiments: Removing iterative reasoning or restricting decisions to local visible directions significantly reduces performance relative to the full SSM model.These ablations support the roles of long-range message passing and a global navigation action space.
5. Conclusion
The paper concludes that recurrent latent memory is insufficient for long-term structural dependencies in partially observable VLN. SSM provides explicit structured memory for accessing past percepts, exploring layouts, and supporting fine-grained grounding, long-term reasoning, and global decisions.
- Conclusion: SSM replaces recurrent latent memory with structured explicit memory that represents past percepts and environment layouts for VLN planning.The agent achieves state-of-the-art performance on the R2R and R4R datasets.