Structured Scene Memory for Vision-Language Navigation

1. Introduction

• Prior VLN work advanced learning paradigms [50, 48], supervision from synthesized data [13, 45, 14] and auxiliary tasks [25] [33], multimodal embeddings [24, 40, 49], and path planning [28] [34] [47].Source: §1 · p. 1
• Seq2Seq VLN agents mix observations into recurrent hidden states, limiting access to past perceptions and explored layouts and restricting decisions to locally navigable directions.This limits planning over the explored environment and can lead to suboptimal decisions.Source: §1 · p. 1
• Robotics has long used map-building to facilitate path planning, and deep learning has been applied to constructing space representations [17] [12] [7] [8].Recurrent memory states are unstable over long timescales [44], whereas compartmentalized external memories preserve and recall long-term information [44] and have been used in language modeling [44] and 3D reinforcement-learning agents [38].Source: §1 · pp. 1–2
• SSM stores navigation percepts in a graph-structured topological representation, capturing layouts and providing a global action space of explored navigable locations for flexible planning.The agent can redirect to a previously visited location rather than remain limited to its current direction.Source: §1 · p. 2
• Nodes encode visual information and edges geometric relations; an adaptive collect-read controller gathers context for iterative long-range reasoning, while frontier exploration supports policy learning as the action space grows.Visual and geometric cues are disentangled so instructions can ground perception-related descriptions to nodes and action-related descriptions to edges.Source: §1 · p. 2
• Experiments on the Room-to-Room (R2R) [3] and Room-for-Room (R4R) [26] datasets demonstrate the effectiveness of the full approach and core model designs.Source: §1 · p. 2

Related work

• VLN methods span imitation learning [3], hybrid or ensemble RL, synthetic-sample supervision [13] [45] [14], and auxiliary tasks [25] [33].Source: §2 · p. 2
• Other VLN directions address self-correction [28] [34], active exploration [47], environment-agnostic representations, instruction grounding [23] [40] [22], web-paired pretraining [35] [18], and continuous environments [29].Source: §2 · p. 2
• SSM separates visual and geometric cues for instruction grounding and provides global actions, unlike [11]’s mixed scene map and subset planner trained only with IL.Existing Seq2Seq agents encode past observations as recurrent hidden states, limiting precise long-lag memorization and exploration of environment layouts; SSM uses structured memory and navigation-utility-driven policy learning.Source: §2 · p. 2
• External-memory research enables long-term memorization and explicit read/write operations; SSM adapts this approach to store exploration percepts, capture layouts, and gather context with a collect-read controller.Prior external-memory applications include language modeling, robot control, and meta-learning.Source: §2 · p. 2
• Navigation research uses SLAM-derived maps for planning and learns policies over spatial or topological representations, but VLN methods often retain local action spaces and latent memory.Prior VLN planning approaches include rule-based beam search, self-correction, and active perception; frontier exploration limits action-space growth to avoid expensive learning and slow convergence.Source: §2 · pp. 2–3

3.1. Structured Scene Memory (SSM) Construction

• SSM is an online-updated topological graph whose nodes encode visual cues and directed edges encode geometric cues, while its sub-nodes expose navigable viewpoints across the map as actions.This global action space lets the agent jump to previously observed viewpoints rather than rely only on currently visible choices.Source: §3.1 · pp. 3–4

3.2. Fine-Grained Instruction Grounding based Iterative Reasoning

• The collect-read controller separately grounds perception- and action-related instruction features to visual nodes and orientation edges, then uses iterative message passing for long-range reasoning.GRU-based updates aggregate neighbor information, allowing node features to capture context within an S-hop neighborhood.Source: §3.2 · p. 4

3.3. Frontier-Exploration based Decision Making

• SSM selects a frontier node as a sub-goal, then chooses one of its navigable viewpoints; its frontier policy is trained to favor navigation utility and supports long-term backtracking.The agent travels to a selected frontier along the shortest path when it is not already there, then selects among that frontier’s sub-nodes.Source: §3.3 · p. 5

3.4. Implementation Details

• The implementation trains the navigation model with imitation learning and reinforcement learning, using teacher and student forcing for imitation learning.Reinforcement-learning rewards reflect changes in distance to the destination at frontier or action selection and successful episode termination; R2R training also uses back-translation augmentation.Source: §3.4 · p. 5

4.2. Performance on R4R Dataset

• In an ambiguous doorway instruction, SSM returned from mistaken P2 to the correct direction at P1 and reached the target, unlike the basic agent.The example followed a successful first instruction before the basic agent was misled by the second.Source: Figure 4 · p. 7, §4.2 · p. 8

4.1. Performance on R2R Dataset

• On R2R, SSM outperformed Active Perception [47] on SR and SPL, surpassed AuxRN on SR, NE, TL, and OR without auxiliary supervision, and reached test-unseen SR 0.61 versus EGP [11]’s 0.53.The SPL advantage over Active Perception [47] held across validation seen, validation unseen, and test unseen; without data augmentation, SSM already exceeded existing methods on SR, NE, and OR except [47].Source: §4.1 · pp. 6–7
• When the kitchen landmark was initially unobservable, SSM planned across explored locations, redirected from P2 to P3, found the kitchen, and completed the instruction.The basic agent instead chose the wrong direction and failed.Source: §4.1 · p. 7

4.2. Performance on R4R Dataset

• On R4R, CLS was 0.65 vs 0.60 on validation seen and 0.53 vs 0.44 on validation unseen; SSM also improved SR, nDTW, and SDTW.Its seen–unseen CLS gap was 0.12, the smallest among compared methods, which the authors cite as evidence of a more generalized navigation policy.Source: Table 2 · p. 7, §4.2 · p. 8

4.3. Diagnostic Experiments

• Ablations found that the full SSM outperformed the basic agent across R2R validation metrics, while removing fine-grained grounding, iterative reasoning, or global action choices reduced performance.Frontier exploration consistently beat one-step decisions over the global action space; restricting choices to visible directions also performed significantly worse than the full model.Source: §4.3 · p. 8, Table 3 · p. 8

• The SSM-based VLN agent achieves state-of-the-art results on challenging R2R and R4R datasets.Source: §5 · p. 8
• SSM provides structured, explicit memory that lets VLN agents access past percepts and explore environment layouts.Source: §5 · p. 8
• The persistent space representation supports fine-grained instruction grounding, long-term reasoning, and global decision-making.Source: §5 · p. 8