PaperlayerView exact revision on arXiv

Source-linked AI summary

JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation

Shuang Zeng, Dekang Qi, Xinyuan Chang, Feng Xiong, Shichao Xie, Xiaolong Wu, Shiyi Liang, Mu Xu, Xing Wei, Ning Guo

arXiv:2509.22548v2cs.CVcs.RO

TL;DR

VLN methods often rely on explicit semantic memories that lose spatial information, repeat computation, and grow with navigation history. JanusVLN addresses this with separate fixed-size implicit memories for visual semantics and spatial geometry, using RGB-only 3D priors and incremental KV caching. It achieves state-of-the-art performance on VLN-CE and reduces inference overhead while preserving or improving performance.

  • Problem

    Existing VLN memories based on textual maps or historical frames suffer spatial information loss, redundant computation, and memory bloat, while common visual encoders lack 3D geometric understanding.

  • Method

    JanusVLN combines an MLLM with a feed-forward 3D geometry encoder and stores separate visual-semantic and spatial-geometric historical KV caches in initial and sliding windows.

  • Results

    JanusVLN achieves state-of-the-art results on VLN-CE without auxiliary 3D data and reduces inference overhead by 69%-90% in memory-size ablations.

  • Takeaways & Limitations

    The results support dual implicit neural memory as an efficient RGB-only paradigm for combining semantic and spatial information in VLN.

  • Takeaways & Limitations

    The authors note that autonomous navigation technology has dual-use risks, including unauthorized surveillance or military applications.

Abstract

from arXiv · show

Vision-and-Language Navigation requires an embodied agent to navigate through unseen environments, guided by natural language instructions and a continuous video stream. Recent advances in VLN have been driven by the powerful semantic understanding of Multimodal Large Language Models. However, these methods typically rely on explicit semantic memory, such as building textual cognitive maps or storing historical visual frames. This type of method suffers from spatial information loss, computational redundancy, and memory bloat, which impede efficient navigation. Inspired by the implicit scene representation in human navigation, analogous to the left brain's semantic understanding and the right brain's spatial cognition, we propose JanusVLN, a novel VLN framework featuring a dual implicit neural memory that models spatial-geometric and visual-semantic memory as separate, compact, and fixed-size neural representations. This framework first extends the MLLM to incorporate 3D prior knowledge from the spatial-geometric encoder, thereby enhancing the spatial reasoning capabilities of models based solely on RGB input. Then, the historical key-value caches from the spatial-geometric and visual-semantic encoders are constructed into a dual implicit memory. By retaining only the KVs of tokens in the initial and sliding window, redundant computation is avoided, enabling efficient incremental updates. Extensive experiments demonstrate that JanusVLN outperforms over 20 recent methods to achieve SOTA performance. For example, the success rate improves by 10.5-35.5 compared to methods using multiple data types as input and by 3.6-10.8 compared to methods using more RGB training data. This indicates that the proposed dual implicit neural memory, as a novel paradigm, explores promising new directions for future VLN research. Ours project page: https://miv-xjtu.github.io/JanusVLN.github.io/.

1 INTRODUCTION

JanusVLN addresses the spatial, computational, and memory limitations of explicit semantic navigation memories by introducing compact dual implicit memories for visual semantics and spatial geometry. It combines RGB-only 3D priors with incremental historical-information updates and reports state-of-the-art VLN-CE results without auxiliary 3D data.

  • Motivation: Existing VLN methods use textual cognitive maps or historical frames, which lose spatial information or require redundant historical reprocessing.Textual descriptions also introduce redundancy and noise, making critical information difficult to extract from cluttered memory.
  • Motivation: 2D-image-trained visual encoders capture semantics but remain deficient in 3D geometry and spatial reasoning despite RGB images containing depth and perspective cues.The paper identifies this mismatch as a fundamental limitation for complex navigation.
  • JanusVLN: JanusVLN introduces separate spatial-geometric and visual-semantic memories as compact, fixed-size neural representations whose size does not grow with trajectory length.The design is inspired by implicit spatial representations associated with human navigation.
  • JanusVLN: JanusVLN incorporates a feed-forward 3D visual geometry foundation model into an MLLM to obtain spatial structure solely from RGB video, without explicit 3D data.Historical key-values from the geometry and semantic encoders form the dual memories and are updated through initial and sliding windows.
  • Results: JanusVLN achieves state-of-the-art results on VLN-CE benchmarks without requiring auxiliary 3D data.The experiments evaluate the method on R2R-CE and RxR-CE.

2 RELATED WORK

Related VLN research has increasingly adopted MLLMs and RGB-only video, while spatial-reasoning methods often rely on explicit 3D inputs. These directions leave open the challenge of efficient navigation with spatial understanding from ordinary RGB streams.

  • MLLM-based VLN: VLN research has shifted toward adapting Multimodal Large Language Models using large-scale training data.These models provide strong visual perception and semantic understanding for navigation.
  • RGB-only VLN: RGB-only video models aim to improve generalization and practical value for monocular VLN systems.However, many such agents construct only explicit semantic memory.
  • Spatial reasoning: Spatial-reasoning methods commonly add point clouds or depth maps, but their reliance on auxiliary hardware can limit practical viability.This motivates methods that obtain spatial information without costly additional 3D sensing.

3 METHOD

JanusVLN uses dual semantic and spatial-geometric encoders with fixed-window implicit KV memories to process RGB video and instructions efficiently. The memories retrieve historical information incrementally, fuse complementary features, and support action prediction with nearly constant historical-processing cost.

  • 3.1 PRELIMINARY: VLN-CE provides RGB video and language instructions, and the agent repeatedly predicts Move Forward, Turn Left, Turn Right, or Stop actions.Each action corresponds to a 30° rotation, a 25 cm forward step, or stopping.
  • 3.1 PRELIMINARY: VGGT extracts geometric tokens from frames using an encoder and fusion decoder, while its prediction head can produce point maps and confidence maps.JanusVLN uses the encoder and fusion decoder as its 3D visual geometry encoder rather than directly predicting 3D attributes.
  • 3.2 DUAL IMPLICIT MEMORY: JanusVLN caches deeply processed historical key-values as compact implicit knowledge representations instead of storing raw historical frames.These representations support retrieval and reasoning with minimal computational cost.
  • 3.2 DUAL IMPLICIT MEMORY: The hybrid cache retains recent KV caches in a FIFO sliding window while permanently preserving initial-frame caches as global attention anchors.The initial and recent windows together support incremental updates while limiting memory growth.
  • 3.2 DUAL IMPLICIT MEMORY: For each new frame, cross-attention retrieves historical information from implicit memory, avoiding redundant feature extraction from past frames.VGGT inference time therefore increases only marginally instead of reprocessing the entire sequence.
  • 3.3 JANUSVLN ARCHITECTURE: The semantic encoder and VGGT spatial encoder separately produce semantic and spatial-geometric features, which are aligned, fused, and passed with the instruction embedding to the MLLM for the next action.A lightweight projection layer fuses the complementary feature types.

4 EXPERIMENTS

Experiments evaluate JanusVLN on simulated VLN-CE benchmarks and real-world spatial-navigation tasks, showing strong performance and benefits from its dual implicit memory and 3D geometric priors.

  • Experimental Setup: JanusVLN is evaluated on R2R-CE and RxR-CE unseen splits, using Matterport3D scenes in the Habitat simulator.The real-world setup uses a Unitree Go2 with an Insta360 X5 camera and remote A10-GPU inference.
  • Main Results: 10.5-35.5 SR improvement is reported over methods using multiple input types, while single RGB input also outperforms depth-based methods by 12.6-16.7.JanusVLN also improves SR over explicit textual maps or historical-frame methods by 20.8, 10.8, and 3.6, respectively.
  • Main Results: Real-world tasks test depth perception, 3D orientation and relative positioning, and spatial association.Examples include selecting the farthest yellow stool, standing beside rather than in front of a plant, and identifying a stool beside an orange cabinet.
  • Ablation Study: Removing spatial memory reduces SPL from 49.2 to 40.9, while removing semantic memory decreases SR by 13.8%.Removing both memory modules causes a near-collapse in performance, supporting their complementary roles.
  • Ablation Study: Replacing VGGT with DINOv2, SigLIP 2, or randomly initialized VGGT provides no notable gains, unlike the pretrained 3D encoder.The reported advantage is attributed to complementary 3D spatial-geometric priors rather than simply adding parameters.
  • Ablation Study: 69%-90% lower inference overhead is achieved by dynamically caching historical KV instead of recomputing the full sequence.Performance improves as memory grows and saturates at 48 frames; preserving the initial window also matters.

5 CONCLUSION

JanusVLN introduces a dual implicit neural memory framework that combines compact spatial-geometric and visual-semantic representations for efficient RGB-only VLN. The framework retains initial and sliding-window key-value caches to avoid recomputation.

  • JanusVLN is presented as the first VLN framework featuring a dual implicit neural memory.
  • The framework uses two complementary, fixed-size, compact neural memories for spatial-geometric and visual-semantic information.
  • JanusVLN combines an MLLM with a feed-forward 3D spatial geometry foundation model to perceive spatial structures solely from RGB video.
  • Retaining only initial and sliding-window key-value caches enables efficient incremental memory updates without recomputation.
  • The experiments demonstrate JanusVLN’s superiority and motivate a shift toward 3D spatial-semantic synergy in VLN.

ETHICAL STATEMENT

The ethical statement highlights beneficial applications of JanusVLN while acknowledging that autonomous navigation technology can be misused. It calls for responsible development and application of AI.

  • JanusVLN is anticipated to support navigational assistance for visually impaired people, domestic service robots, and disaster search and rescue.
  • The statement acknowledges potential misuse of advanced autonomous navigation technology in unauthorized surveillance or military operations.
  • The authors call on the academic community to establish and follow guidelines for responsible AI development and application.

REPEATABILITY

JanusVLN’s implementation details are provided for reproducibility, and the authors plan to release code, configurations, and fine-tuned weights under relevant licenses.

  • Implementation details for JanusVLN are provided in Section 4.1 to support reproducibility.
  • The authors will publicly release source code, model configurations, and fine-tuned model weights in accordance with relevant licenses.
  • The planned releases are intended to help researchers replicate the findings and build upon the work.

A THE USE OF LARGE LANGUAGE MODELS (LLMS)

The paper limits LLM use to manuscript writing assistance, while describing how JanusVLN’s spatial encoder and semantic encoder use historical key-value caches for attention and memory.

  • A THE USE OF LARGE LANGUAGE MODELS (LLMS): LLMs were used only to improve writing quality and identify grammatical or stylistic errors while preserving the manuscript’s original meaning.
  • JanusVLN’s spatial encoder incorporates historical key-value caches during global attention by concatenating cached and current-frame keys and values.
  • Qwen2.5-VL computes attention between new-frame visual queries and historical plus current-frame keys and values in its standard KV-cache mechanism.

C MORE ABLATION STUDIES

JanusVLN maintains strong real-world and benchmark performance, with gains attributed to spatial memory and robust intrinsic navigation capabilities. Ablations identify lightweight feature addition and supplementary-data choices as effective design settings, while larger external datasets remain future work.

  • Real-world experiments: 23.6% improvement on navigation tasks requiring spatial understanding over the variant without spatial memory.Real-world evaluation used 25 instructions repeated three times, with success defined as stopping within 1 meter of the target.
  • Benchmark results: 44.9% Success Rate, up from 40.8%, on the HM3D-OVON val unseen benchmark.The benchmark is described as more diverse and updated, supporting the model’s reported generalization capability.
  • Fusion strategies: Performance peaked when spatial features received weight 0.2, while simple addition slightly outperformed Cross-Attention.The authors leave more sophisticated fusion strategies for future work.
  • Training data ablation: JanusVLN outperformed prior methods using partial supplementary datasets even without additional data.Adding ScaleVLN and DAgger data individually improved performance, and using both sources produced further gains.
  • Training data ablation: Larger-scale external datasets are reserved for future work to construct more powerful navigation agents.This marks the current training-data scope boundary despite the model’s performance without supplementary data.

D STATISTICAL ANALYSIS

Statistical analyses show stronger spatial awareness and improved handling of the longest instructions, while failures concentrate on trajectory recovery and premature stopping. The authors associate these failures with limited corrective data and missing real-world scale in spatial estimates.

  • Spatial understanding: JanusVLN’s advantage is more pronounced on instructions requiring spatial understanding than its average gain across all tasks.These instructions include relations such as farthest, nearest, rightmost, leftmost, front, and back.
  • Instruction length: Both StreamVLN and JanusVLN perform well on 1-150-word instructions but decline on 150-400-word instructions.The decline indicates a need to improve decomposition and comprehension of moderately complex directives.
  • Instruction length: For 400-550-word instructions, StreamVLN eventually reaches zero performance, whereas JanusVLN improves.The passage attributes JanusVLN’s improvement to its dual implicit memory paradigm and the detailed guidance in lengthy instructions.
  • Failure cases: JanusVLN often fails to recover after deviating from the optimal trajectory, causing compounding errors and eventual failure.The authors report that the limited non-optimal trajectory data collected through DAgger is insufficient for robust error correction.
  • Failure cases: JanusVLN sometimes stops prematurely after seeing the destination because VGGT spatial information lacks real-world scale for accurate distance estimation.This overly aggressive stopping policy can leave the robot outside the success radius.

E MORE QUALITATIVE RESULTS

Qualitative analyses visualize how JanusVLN’s spatial geometric tokens support depth and occlusion reasoning, and present examples across real-world and simulated unseen environments. The examples cover varied instruction complexity, sites, spatial tasks, and long trajectories.

  • Spatial geometric tokens: Spatial geometry tokens produce depth maps and point clouds that reveal useful scene structure for navigation.The visualizations support localization of the farthest chair and reveal a chair hidden behind a sink counter.
  • Real-world results: Real-world examples include simple and complex instructions, diverse sites, and spatial-understanding tasks.The authors use these examples to demonstrate JanusVLN’s reported generalization in real-world settings.
  • Simulated results: Simulated examples use complex trajectories and long instructions from unseen validation sets of R2R-CE and RxR-CE.Figures 12 and 13 provide qualitative results for the two simulated environments.
Loading 2509.22548v2…