Source-linked AI summary
StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling
Meng Wei, Chenyang Wan, Xiqian Yu, Tai Wang, Yuqiang Yang, Xiaohan Mao, Chenming Zhu, Wenzhe Cai, Hanqing Wang, Yilun Chen, Xihui Liu, Jiangmiao Pang
TL;DR
Real-time VLN must process continuous visual streams while preserving fine-grained understanding, long-term context, and computational efficiency. StreamVLN addresses this with fast sliding-window dialogue context and slow 3D-aware memory pruning, achieving superior VLN-CE performance with low latency.
Problem
Continuous video-based VLN requires long-term context and responsive action generation, but visual tokens grow linearly and existing compression or sampling methods can lose temporal and visual detail.
Method
StreamVLN extends Video-LLMs into an interleaved vision-language-action model using a fast sliding-window KV cache and slow-updating memory with training-free voxel-based 3D spatial pruning.
Results
StreamVLN achieves superior performance with lower latency on standard VLN-CE benchmarks while supporting long video streams with bounded context growth and inference latency.
Takeaways & Limitations
The slow-fast context design provides an efficient and scalable approach for continuous, real-time, long-horizon vision-and-language navigation.
Abstract
from arXiv · showhide
Vision-and-Language Navigation (VLN) in real-world settings requires agents to process continuous visual streams and generate actions with low latency grounded in language instructions. While Video-based Large Language Models (Video-LLMs) have driven recent progress, current VLN methods based on Video-LLM often face trade-offs among fine-grained visual understanding, long-term context modeling and computational efficiency. We introduce StreamVLN, a streaming VLN framework that employs a hybrid slow-fast context modeling strategy to support multi-modal reasoning over interleaved vision, language and action inputs. The fast-streaming dialogue context facilitates responsive action generation through a sliding-window of multi-turn dialogues, while the slow-updating memory context compresses historical visual states using a 3D-aware token pruning strategy. With this slow-fast design, StreamVLN achieves real-time dialogues through KV cache reuse, supporting long video streams with bounded context size and inference cost. Experiments on VLN-CE benchmarks show state-of-the-art performance with low latency, ensuring robustness and efficiency in real-world deployment. The project page is: https://streamvln.github.io/.
I. INTRODUCTION
StreamVLN targets continuous VLN challenges in fine-grained multimodal alignment, long-term reasoning, and efficient real-time interaction. It combines fast dialogue streaming, slow memory updates, and 3D spatial pruning to support long video streams with low latency.
- Continuous real-world VLN requires grounding language in visual observations while planning actionable trajectories.
- Video-LLM navigation methods trade temporal resolution, visual detail, and computational efficiency when processing continuous video streams.Fixed-frame sampling can miss fine-grained temporal changes, while token compression can sacrifice temporal and visual details.
- StreamVLN combines a fast-streaming dialogue context with a slow-updating memory context for responsive action decoding and long-term reasoning.A fixed dialogue window reuses KV states, while completed windows are consolidated into memory.
- 3D spatial pruning removes redundant visual patches by using voxel-based spatial cues, supporting streaming video processing and KV cache compression.The strategy addresses high-resolution visual-token redundancy remaining after temporal sampling.
- StreamVLN works on long video streams with bounded context growth and inference latency, achieving superior performance with low latency on VLN-CE benchmarks.The model can be trained on short clips such as 16 frames while operating on long streams.
II. RELATED WORK
Related VLN research spans discrete navigation, continuous-environment control, and multimodal language-model planning. These directions seek more flexible and scalable navigation while task-specific models remain stronger than some training-free modular approaches.
- Early VLN focused on discrete scene graphs, where agents navigated between predefined nodes and emphasized high-level decisions.
- Continuous-environment VLN addresses realistic navigation requiring low-level actions rather than teleportation between predefined locations.
- MLLM-based VLN includes training-free modular planners and fine-tuned Video-LLM-based models for multimodal navigation.
- Training-free LLM planners remain separated from task-specific models by a performance gap, motivating flexible and scalable navigation frameworks.
III. METHOD
StreamVLN extends a Video-LLM into an online interleaved vision-language-action model for continuous multi-turn navigation. Its sliding-window KV cache and memory-token updates balance responsive decoding with long-horizon context.
- StreamVLN generates actions from continuous video input through online multi-turn dialogues interleaving observations and actions.Each new observation is appended to the token stream, and the next action is generated autoregressively.
- KV cache reuse can eliminate over 99% of prefilling time, but unrestricted cache growth creates substantial memory overhead during long sessions.A 2K-token cache can consume around 5GB of memory, and overly long contexts can degrade reasoning performance.
- A fixed-size sliding-window KV cache retains recent dialogues while limiting active context.When the window fills, non-observation prompt and action states are discarded and past visual states are processed into memory tokens.
- Past sliding-window states are converted into memory tokens that provide context for subsequent windows.
- The latest action is decoded from the current observation together with cached states and the current window’s KV cache.
C. Slow-Updating Memory Context
The slow-updating memory context addresses redundant visual observations without altering pretrained features. StreamVLN uses depth-guided voxel pruning to retain recent spatially representative tokens for memory construction.
- Fine-grained VLN trajectories contain redundant observations, while high-resolution Video-LLMs retain substantial patch-level spatial redundancy.
- Feature-level compression can substantially degrade performance by altering the pretrained input distribution and undermining pretraining knowledge.
- Training-free voxel-based 3D spatial pruning back-projects image patches into shared 3D space using depth information.
- When tokens from different frames map to the same voxel within a duration, the method retains only the token from the most recent observation.
- The pruning mask selects preserved token states, with a threshold that can suppress frames containing too few retained patches.
D. Co-Training with Multi-Source Data.
StreamVLN co-trains on navigation-specific and general vision-language data to combine corrective navigation supervision with broader multimodal reasoning.
- 450K navigation clips from 60 Matterport3D environments are sourced from R2R, R2R-EnvDrop, and RxR.
- An additional 300K samples from ScaleVLN increase scene diversity for navigation generalization.
- 240K DAgger corrective demonstrations from model rollouts are added after initial training for co-training.
- 248K video-based VQA samples preserve general reasoning capabilities alongside navigation supervision.
A. Experimental Setup
StreamVLN is evaluated on continuous-control R2R-CE and RxR-CE navigation benchmarks and deployed on a robot-server system for real-time testing.
- R2R-CE contains 5.6K English trajectories averaging 10 meters, while RxR-CE contains 126K multilingual instructions with paths averaging 15 meters.
- Both VLN-CE benchmarks use realistic indoor navigation under continuous control, with evaluation focused on validation-unseen generalization.
- StreamVLN runs on a remote RTX 4090 workstation receiving continuous visual data from a Go2 robot and returning executable actions.
- Inference averages 0.27 seconds for four actions, while communication takes 0.2 seconds indoors and 1.0 second outdoors.
B. Implementation Details
StreamVLN is built from LLaVA-Video and trained in two stages, using oracle and DAgger trajectories alongside multimodal data for streaming navigation.
- StreamVLN is based on the LLaVA-Video 7B model with Qwen2-7B as its language model.
- Training first fine-tunes one epoch on oracle VLN trajectories, then trains one additional epoch on DAgger trajectories mixed with general multimodal data.
- Each training step processes 128 video clips, and training requires approximately 1500 A100 GPU hours.
C. Comparisons with State-of-the-Arts
StreamVLN delivers strong benchmark and real-world navigation results, while also outperforming generalist navigation models on ScanQA and handling diverse environments.
- VLN-CE benchmark: 56.9% SR and 51.9% SPL on R2R, and 52.9% SR and 46.0% SPL on RxR, establish state-of-the-art RGB-only performance among compared methods.
- VLN-CE benchmark: StreamVLN surpasses HMAT trained on 3 million trajectories while using only 150K ScaleVLN trajectories.
- Voxel pruning: Voxel pruning reduces memory tokens by 28% on R2R and 22% on RxR without noticeably affecting performance.
- Real-world experiments: Across hallway, bedroom, and office scenarios, StreamVLN completes both simple and challenging long-horizon tasks, unlike baselines with narrower capabilities.
- ScanQA: StreamVLN outperforms NaviLLM and NaVILA on the ScanQA validation set using 16 multi-view images per scan.
- Real-world experiments: Qualitative results span Home, Workspace, Mall, and Outdoor environments, including complex instructions with landmarks and real-world disturbances.
D. Ablation Studies
The ablations show that training-data composition, fine-grained memory, and an eight-turn sliding window materially affect navigation performance and training cost. KV-cache reuse is evaluated under a fixed eight-turn window.
- Data Ablation: RxR data yields notable gains of +7.8 SR / +7.3 SPL, while removing DAgger data reduces the benefit by +5.5 SR / +3.8 SPL.
- Data Ablation: ScaleVLN data adds +2.9 SR / +3.7 SPL when incorporated into the co-training composition.
- Effectiveness of KV-Cache Reuse: The KV-cache reuse experiment compares multiple-turn decoding settings with the sliding window fixed at eight conversation turns.
- Sliding Window Size: An eight-turn sliding window retains the best balance of navigation performance and training cost, whereas smaller windows increase samples to approximately 815K or 1.5M.
V. CONCLUSION
StreamVLN uses a hybrid memory design to reuse past key/value states during long-video navigation. Its fast sliding window and slow long-term memory support efficient, coherent action generation with lower latency on VLN-CE benchmarks.
- StreamVLN reuses past key/value states through a fast-updating sliding window and a slow-updating long-term memory.
- The framework achieves superior performance with lower latency on standard VLN-CE benchmarks for long-horizon navigation.