Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation

Hai Zhang; Siqi Liang; Li Chen; Yuxian Li; Yukuan Xu; Yichao Zhong; Fu Zhang; Hongyang Li

Source-linked AI summary

Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation

Hai Zhang, Siqi Liang, Li Chen, Yuxian Li, Yukuan Xu, Yichao Zhong, Fu Zhang, Hongyang Li

arXiv:2602.05827v1Feb 5, 2026cs.CVcs.RO

TL;DR

Beyond-the-View Navigation challenges instruction-following agents to reach distant, unseen targets from simple intents, while short-horizon LLM supervision limits their foresight and longer supervision destabilizes training. SparseVideoNav addresses this gap with sparse video generation, achieving efficient long-horizon trajectory inference and strong zero-shot real-world performance.

Problem
Beyond-the-View Navigation requires agents to locate distant, unseen targets from simple intents, but existing LLM methods rely on short-horizon supervision and dense instructions.
Method
SparseVideoNav introduces sparse video generation with a structured four-stage training pipeline and a 20-second sparse future for action prediction.
Results
SparseVideoNav achieves state-of-the-art zero-shot performance across six real-world scenes and 2.5× the success rate of SOTA LLM baselines on beyond-the-view scenarios.
Takeaways & Limitations
SparseVideoNav demonstrates that sparse video generation can support beyond-the-view navigation, including challenging night scenes.
Takeaways & Limitations
The curated 140-hour dataset is not web-scale, and inference remains slightly slower than existing LLM-based navigation paradigms.

Abstract

from arXiv · show

Why must vision-language navigation be bound to detailed and verbose language instructions? While such details ease decision-making, they fundamentally contradict the goal for navigation in the real-world. Ideally, agents should possess the autonomy to navigate in unknown environments guided solely by simple and high-level intents. Realizing this ambition introduces a formidable challenge: Beyond-the-View Navigation (BVN), where agents must locate distant, unseen targets without dense and step-by-step guidance. Existing large language model (LLM)-based methods, though adept at following dense instructions, often suffer from short-sighted behaviors due to their reliance on short-horimzon supervision. Simply extending the supervision horizon, however, destabilizes LLM training. In this work, we identify that video generation models inherently benefit from long-horizon supervision to align with language instructions, rendering them uniquely suitable for BVN tasks. Capitalizing on this insight, we propose introducing the video generation model into this field for the first time. Yet, the prohibitive latency for generating videos spanning tens of seconds makes real-world deployment impractical. To bridge this gap, we propose SparseVideoNav, achieving sub-second trajectory inference guided by a generated sparse future spanning a 20-second horizon. This yields a remarkable 27x speed-up compared to the unoptimized counterpart. Extensive real-world zero-shot experiments demonstrate that SparseVideoNav achieves 2.5x the success rate of state-of-the-art LLM baselines on BVN tasks and marks the first realization of such capability in challenging night scenes.

I. INTRODUCTION

Beyond-the-View Navigation requires agents to locate distant, unseen targets from simple intents rather than dense instructions, exposing the short-horizon limitations of current LLM methods. SparseVideoNav introduces sparse video generation to provide long-horizon foresight while reducing the computational burden of video-based navigation.

Beyond-the-View Navigation (BVN) asks agents to autonomously locate distant, unseen targets without dense, step-by-step guidance.
Short-horizon supervision leaves existing LLM-based methods vulnerable to unexpected turning, spinning, and dead-end trapping in BVN.Extending the supervision horizon often destabilizes LLM training.
Video generation models are proposed as a suitable interface for BVN because they are pretrained to capture long-horizon futures aligned with language instructions.
SparseVideoNav introduces sparse video generation to extend prediction horizons while reducing the dense temporal information and computational overhead required by continuous video generation.
140 hours of curated navigation data support SparseVideoNav, which achieves state-of-the-art zero-shot performance across six real-world scenes.
2.5× the success rate of SOTA LLM baselines is achieved on beyond-the-view scenarios, with 17.5% success in challenging night scenes.

B. Video Generation Models for Embodied Agents

Video generation models encode dynamic variation useful for embodied tasks, but navigation requires a sparse-future paradigm to extend prediction horizons and remain efficient. SparseVideoNav combines sparse supervision, staged training, curated real-world data, and stabilized video capture for navigation.

Video generation models encode dynamic variation between frames and have been applied across robotic manipulation, autonomous driving, and navigation.
SparseVideoNav uses a four-stage pipeline in which sparse future video supervises the backbone before conditioning action learning.The stages adapt text-to-video to image-to-video, inject history, distill denoising steps, and learn actions.
A. Sparsification: Sparse video supervision extends prediction toward distant targets while balancing horizon and visual fidelity through a fixed interval of 3.The sparse schedule covers 20 seconds at 4 FPS, while the first two observation chunks remain continuous for 8 timesteps.
A. Sparsification: The sparse generation timesteps are [T +1, T +2, T +5, T +8, T +11, T + 14, T + 17, T + 20], covering a 20-second horizon at 4 FPS.
Simulation-only training can cause mode collapse, while existing real-world navigation datasets often suffer from fisheye distortion and limited scale.
140 hours of real-world navigation videos are processed into approximately 13,000 trajectories averaging 140 frames at 4 FPS.

C. Stage 1: T2V →I2V

Stage 1 adapts a text-to-video backbone into an image-to-video model that generates sparse future observations conditioned on the current visual context and language. Training uses flow matching on interpolated noisy and future chunk latents, with mean-square velocity prediction.

T2V → I2V: Stage 1 adapts Wan’s text-to-video backbone to image-to-video generation so future observations remain consistent with the initial observation.The adaptation is necessary because the original T2V model primarily conditions future generation on language rather than visual inputs.
Sparse future training: The model trains on sparse future chunk latents spanning offsets from cT+1 through cT+20, alongside random noise and a sampled timestep.An intermediate latent is formed from the noise and sparse future latents by linear interpolation.
Flow matching: Flow matching defines the intermediate latent as a linear interpolation between random noise x0 and the sparse future latent x1.The passage identifies this interpolation as the training input construction.
Training objective: The ground-truth velocity vt supervises training through mean square error between the predicted velocity and vt.The language embedding l and model weights θ condition and parameterize the predicted velocity.

D. Stage 2: History Injection

Stage 2 injects the agent’s observation history into the video-generation backbone while controlling the computational cost of processing long, high-dimensional visual sequences. Later stages address dynamic-scene denoising and action alignment using distilled generation and relabeled actions.

History Injection: VGMs require an explicit mechanism to incorporate the entire observation history, unlike LLMs that can directly process long sequences of image tokens.SparseVideoNav adds a cross-attention block within each Wan transformer block to inject history information.
History Injection: Q-Former and Video-Former modules extract spatio-temporal features from the long, high-dimensional history before injection.Q-Former processes temporal features, followed by Video-Former processing along the spatial dimension.
History Injection: Stage 2 optimizes predicted velocity against ground-truth velocity while conditioning on language, the current chunk, and the processed history embedding hT.The objective extends the Stage 1 flow-matching formulation with hT.
Efficiency bottleneck: Navigation’s highly dynamic scene transitions make high-fidelity future-frame generation difficult with few-step denoising, limiting real-world deployment.This challenge differs from manipulation tasks, whose limited visual changes permit high-fidelity reconstruction with few denoising steps.
Efficiency bottleneck: PCM is adapted to flow matching to distill a history-injected image-to-video teacher into an identical student using four noise-schedule phases.The student learns solution points along the teacher’s probability-flow ODE trajectory.
Action Learning: The frozen distilled image-to-video model and language instruction condition a DiT-based action head through cross-attention to predict continuous actions.DDIM reconstructs relabeled actions to address the mismatch between generated futures and original action labels.

IV. EVALUATIONS

The evaluation tests instruction following, beyond-the-view navigation, and efficiency across six unseen real-world scenes, including indoor, outdoor, and night environments. Models are evaluated zero-shot on standard and challenging navigation tasks with repeated trials and controlled comparisons.

Evaluation goals: The experiments assess instruction following, beyond-the-view navigation, and the efficiency–performance trade-off of SparseVideoNav.These questions are evaluated in the referenced experimental sections.
Evaluation Protocols: Six diverse unseen scenes span indoor, outdoor, and night categories for zero-shot generalization testing.The scenes are Room, Lab Building, Yard, Park, Square, and Mountain.
Evaluation Protocols: Each scene contains four navigation tasks: two standard instruction-following tasks and two beyond-the-view tasks.The task specifications are deferred to Section D.
Evaluation Protocols: Each model is tested 10 times per task to support statistical reliability.The passage introduces this repeated-trial protocol after describing the task distribution across scenes.
Experimental Setup: Comparisons define success as stopping within 1.5 meters of the target and control time and weather by testing models within the same time window.This criterion avoids penalizing models for stopping with a lateral rather than direct target orientation.
Hardware Deployment: The real-world experiments use a Unitree Go2 robotic dog with stabilized RGB observations, while InternVLA-N1 additionally uses an Intel RealSense D455 for RGB-D input.The cameras are mounted at approximately 1 m above the ground, with the D455 pitched 15° downward.

B. Main Results

SparseVideoNav achieves state-of-the-art zero-shot performance across real-world IFN and BVN scenes, including challenging environments where established baselines fail. Qualitative results show successful navigation through dead ends, narrow ramps, and steep hillsides.

SparseVideoNav achieves unanimously state-of-the-art zero-shot performance across all real-world scenes on both IFN and BVN tasks.
15.0% average success-rate improvement over StreamVLN is reported on both IFN and challenging BVN tasks.
SparseVideoNav is the sole method reported as capable of navigating to distant goals in extreme environments where diminished visibility causes baseline BVN failures.
SparseVideoNav successfully navigates challenging scenarios including dead ends, narrow accessible ramps, and hillsides with high inclination angles.
Closed-loop feedback combined with generated guidance mitigates unexpected turning and premature dead-end trapping associated with short-horizon LLM supervision.

C. Ablation Study

The ablation study evaluates data scale, sparse generation, distillation, and history compression as components affecting SparseVideoNav’s efficiency and effectiveness. Sparse design and distillation provide favorable efficiency gains while retaining competitive performance.

Data Scalability: 140 hours of training data produce a downward FVD trend across 8h, 50h, and 140h settings, indicating scalability to larger real-world navigation datasets.
Diffusion Distillation: Four denoising steps achieve visual fidelity comparable to the original model using 50 steps.
Sparse Video Generation: Short-horizon continuous generation yields suboptimal performance, while extending the horizon partially mitigates but does not remove the disparity with SparseVideoNav.
Sparse Video Generation: 1.7× faster inference and 1.4× lower cumulative convergence time are achieved by SparseVideoNav versus the undistilled 50-step, 20-chunk variant, despite a slight performance compromise.
Diffusion Distillation: Approximately 10× inference acceleration from diffusion distillation comes with a slight performance compromise.

D. Further Discussion

SparseVideoNav shows adaptability and robustness in challenging real-world conditions, including dynamic pedestrians and camera-height changes, while its broader evaluations outperform existing LLM baselines. The authors also identify data scale and latency as remaining limitations.

Dynamic Pedestrians Avoidance: SparseVideoNav successfully avoids oncoming pedestrians and reaches the door during deployment.
Camera Height Insensitivity: SparseVideoNav remains robust when the camera height is fixed at 50cm, despite training data collected at approximately 1m.
Limitations: The curated 140-hour dataset is not exhaustive compared with web-scale data, and inference remains slightly slower than existing LLM-based navigation paradigms.

APPENDIX

The appendix details SparseVideoNav’s motivation, data curation, architecture, training configuration, and action-prediction design. It also explains how sparse video guidance can support backtracking from dead ends.

Motivating Questions: Sparse video guidance can imagine backtracking trajectories to escape dead ends, unlike language-model point goals confined to the current view.
Data Curation: The data curation pipeline temporally samples raw video, estimates camera poses with Depth Anything 3, and extracts actions from relative pose transformations.
Training Configurations: The four-stage training pipeline uses AdamW on 32 NVIDIA H200 GPUs and requires approximately 64 hours to complete.
Network Architecture: SparseVideoNav integrates a video-generation backbone, a history-compression module, and an inverse-dynamics-based action-prediction head.
Action Prediction: The action head encodes 640 sparse future latent tokens and predicts continuous 8-step action trajectories with a Diffusion Transformer.

D. Task Specifications and Details

The evaluation covers 24 zero-shot navigation tasks distributed across six real-world scenes, with both instruction-following and beyond-the-view settings represented in every scene.

Task Composition: 24 zero-shot evaluation tasks span six real-world scenes, with each scene containing two IFN tasks and two BVN tasks.

E. Data Ethics and License

The appendix describes privacy protection, ethical data collection, licensing, and representative navigation-task specifications. Sensitive information is removed before dataset use or storage.

Privacy Protection: Human faces and vehicle identifiers are permanently blurred before captured video frames are used for training or stored.
Data Collection Ethics: Human operators record in diverse public environments while following ethical guidelines and avoiding restricted or private areas without authorization.
License: The system builds on Wan2.1 under Apache License 2.0, while SparseVideoNav code and dataset are released under CC BY-NC-SA 4.0.
Task Specifications: The task specifications include high-level instructions such as finding a metal board, locating a bench, reaching a cone, and stopping near landmarks.