Source-linked AI summary
Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation
Meng Wei, Chenyang Wan, Jiaqi Peng, Xiqian Yu, Yuqiang Yang, Delin Feng, Wenzhe Cai, Chenming Zhu, Tai Wang, Jiangmiao Pang, Xihui Liu
TL;DR
Existing VLN methods often couple reasoning, planning, and control into short-horizon action prediction, causing fragmented motion, latency, and difficulty with dynamic obstacles. DualVLN decouples these functions into a VLM-based waypoint planner and a lightweight diffusion policy, and it surpasses prior methods across VLN benchmarks while showing robust real-world navigation.
Problem
Existing VLN models largely map vision-language inputs directly to short-horizon discrete actions, producing fragmented motion, high execution latency, and limited coordination across decision levels.
Method
DualVLN uses System 2 for robust image-grounded pixel-goal reasoning and System 1 for continuous trajectory generation, connecting both through explicit goals and latent representations.
Results
DualVLN consistently surpasses prior state-of-the-art methods on VLN-CE and VLN-PE benchmarks and demonstrates robust long-horizon planning, real-time execution, and dynamic obstacle avoidance in real-world evaluations.
Takeaways & Limitations
The dual-system design provides a foundation for more robust, efficient, and generalizable navigation by decoupling semantic grounding from low-level action execution.
Abstract
from arXiv · showhide
While recent large vision-language models (VLMs) have improved generalization in vision-language navigation (VLN), existing methods typically rely on end-to-end pipelines that map vision-language inputs directly to short-horizon discrete actions. Such designs often produce fragmented motions, incur high latency, and struggle with real-world challenges like dynamic obstacle avoidance. We propose DualVLN, the first dual-system VLN foundation model that synergistically integrates high-level reasoning with low-level action execution. System 2, a VLM-based global planner, "grounds slowly" by predicting mid-term waypoint goals via image-grounded reasoning. System 1, a lightweight, multi-modal conditioning Diffusion Transformer policy, "moves fast" by leveraging both explicit pixel goals and latent features from System 2 to generate smooth and accurate trajectories. The dual-system design enables robust real-time control and adaptive local decision-making in complex, dynamic environments. By decoupling training, the VLM retains its generalization, while System 1 achieves interpretable and effective local navigation. DualVLN outperforms prior methods across all VLN benchmarks and real-world experiments demonstrate robust long-horizon planning and real-time adaptability in dynamic environments.
1 INTRODUCTION
VLN has progressed toward continuous and physically realistic settings, while VLMs offer broader generalization. DualVLN addresses limitations of tightly coupled short-horizon control by separating robust reasoning from agile execution.
- VLN has evolved from discrete goal planning through continuous action spaces toward physically realistic simulations with locomotion controllers.
- Existing VLA models directly map vision-language inputs to short-horizon discrete actions, producing fragmented motions and high execution latency.These models also entangle reasoning, global planning, and local control.
- DualVLN decouples navigation into System 2 for robust VLM reasoning and explicit pixel goals, and System 1 for lightweight continuous trajectory generation.Latent representations connect the two systems, while System 2 is frozen after pixel-goal training and prompt-tuned queries extract compact implicit goals.
- Decoupled sequential training lets System 2 scale with multi-source reasoning data while System 1 uses fewer low-level goal-reaching examples and asynchronous high-frequency RGB inputs.The separation is described as preserving VLM generalization during downstream low-level planning.
- Explicit pixel goals improve System 2 interpretability and generalization, while implicit latent features provide richer, more adaptive guidance for System 1.
- DualVLN surpasses prior state-of-the-art methods on VLN-CE and VLN-PE, with real-world evaluations showing long-horizon planning, real-time execution, and dynamic obstacle avoidance.The paper also introduces Social-VLN for social awareness and task recovery in dynamic environments.
2 RELATED WORK
Prior VLN work uses language-token prediction, pixel grounding, or end-to-end trajectory generation, but existing approaches leave gaps in asynchronous high-frequency control and long-horizon unseen-environment navigation.
- Many VLM-based navigation methods formulate actions as text and treat navigation as next-token prediction.
- Pixel-grounding methods such as RoboPoint and NaviMaster still require additional modules for execution.
- End-to-end methods map VLM latent features directly to continuous trajectories, but synchronized frameworks limit high-frequency decision-making in dynamic environments.
- Other dual-system architectures focus on tabletop tasks and do not address long-horizon planning or cross-building navigation.
- Visual navigation policies address explicit goals, obstacle avoidance, generalization, training efficiency, sim-to-real transfer, and image-goal navigation through distinct model families.DualVLN’s System 1 is an RGB-only policy conditioned on VLM-derived latent goals.
3 METHOD
DualVLN combines a VLM planner that grounds mid-term waypoints with an asynchronous diffusion policy that converts latent and visual conditions into continuous trajectories.
- Framework Overview: System 2 predicts mid-term waypoint goals in image pixel space, while System 1 generates continuous trajectories from current observations and asynchronous latent features.
- Framework Overview: System 2 receives egocentric image sequences and instructions, outputting view-adjustment actions or a 2D pixel coordinate for the next waypoint.
- Framework Overview: System 1 conditions diffusion-based trajectory generation on latent goal embeddings and high-frequency RGB inputs.
- System 2: System 2 iteratively chooses view adjustment or pixel-goal prediction so targets are grounded from informative perspectives despite occlusions and difficult viewpoints.
- Latent Goal Representation: Learnable latent queries appended to System 2’s context extract task-relevant latent goal representations through prompt tuning.
- System 1: System 1’s diffusion transformer generates 32 dense waypoints using low-frequency System 2 latents and high-frequency RGB inputs during asynchronous inference.The RGB conditioning fuses System 2’s last-frame features with the current observation and compresses them into 32 tokens for fast inference.
- Flow Matching: Training samples a diffusion timestep and Gaussian noise to form a noisy trajectory, then trains the transformer to predict its velocity by minimizing mean squared error.The noise schedule uses decreasing α_u and increasing σ_u.
- Implementation: System 2 fine-tunes QwenVL-2.5, while System 1 uses four learnable latent queries, a DepthAnythingV2-Small ViT RGB encoder, and a compact DiT for low-latency inference.
4 SOCIAL VISION-AND-LANGUAGE NAVIGATION BENCHMARK.
Social-VLN extends VLN evaluation from static layouts to dynamic, human-centered scenarios requiring obstacle avoidance and trajectory recovery. It measures both task completion and safety around strategically placed humanoid agents.
- Motivation: Existing VLN-CE benchmarks focus on static layouts and do not evaluate dynamic obstacle handling, social awareness, or recovery to the original task trajectory.
- Benchmark Construction: Social-VLN extends R2R-CE by placing multiple humanoid agents strategically along ground-truth VLN trajectories in simulation.
- Metrics: Human Collision Rate (HCR) quantifies failures caused by unsafe interactions with dynamic pedestrians.The benchmark evaluates safety awareness alongside standard task-completion metrics.
- Training Data: A human-detection and modified-A* pipeline generated 763K social-navigation episodes across 60 MP3D scenes for training.
5 EXPERIMENTS.
Experiments evaluate DualVLN across simulated VLN benchmarks, dynamic Social-VLN, real-world robots, and ablations of its dual-system design. Results show broad gains over prior methods, while analyses identify the importance of decoupled training, explicit pixel goals, latent goals, and data-efficient System 1 learning.
- Simulation Experiments: DualVLN achieves substantial gains over prior RGB-based approaches on VLN-CE using only first-person RGB inputs.The comparison includes multi-sensor, VLM-free, and video-LLM-based baselines on R2R and RxR Val-Unseen splits.
- Simulation Experiments: DualVLN surpasses all baselines on VLN-PE despite not being fine-tuned on VLN-PE trajectories.VLN-PE uses a physical locomotion controller and evaluates effectiveness and robustness under robot dynamics and control errors.
- Simulation Experiments: DualVLN achieves better task completion with obstacle avoidance than StreamVLN on Social-VLN, although both methods suffer substantial performance drops.The success rate decreases by about 27% for DualVLN and 26% for StreamVLN relative to standard VLN tasks.
- Real-World Cross-Embodiment Experiments: Real-world evaluations show safe, smooth, long-horizon navigation across office, canteen, street, and convenience-store scenarios and multiple robot platforms.Experiments use Turtlebot4, Unitree Go2, and Unitree G1 robots with asynchronous remote-server inference.
- Ablation Study: Joint one-stage training without explicit intermediate pixel goals converges more slowly and deteriorates System 2 generalization.The ablation supports decoupled training with intermediate pixel goals as important for efficient learning and preserving VLM reasoning strength.
- Ablation Study: Removing explicit pixel-goal text or latent goal queries weakens performance, while System 1 reaches competitive performance with 1% of System 2’s trajectories and nearsaturation at 10%.Explicit goals guide the diffusion policy and improve interpretability; latent queries provide adaptive information flow, while additional System 1 data beyond 10% adds little.
6 CONCLUSION
DualVLN decouples semantic grounding from action execution and combines explicit pixel waypoints with latent goal representations. The resulting system is presented as more robust, efficient, generalizable, and smoother across diverse environments and tasks.
- DualVLN decouples high-level semantic grounding from low-level action execution.
- The model combines explicit pixel-grounded waypoints with implicit latent goal representations.
- DualVLN produces smoother trajectories and demonstrates strong performance across diverse environments and tasks.
7 CONTRIBUTIONS AND ACKNOWLEDGMENTS
The section records contributors by responsibility and acknowledges institutional and codebase support. Roles span modeling, data curation, benchmarking, real-world deployment, and advising.
- Meng Wei, Xiqian Yu, Jiaqi Peng, Wenzhe Cai, Delin Feng, and Chenming Zhu are credited for model work.
- Chenyang Wan and Meng Wei are credited with VLN data curation.
- Simulation and benchmarking are credited to Meng Wei, Chenyang Wan, Delin Feng, Yuqiang Yang, and Wenzhe Cai.
- Yuqiang Yang, Meng Wei, and Jiaqi Peng are credited with real-world deployment.
- Tai Wang, Jiangmiao Pang, and Xihui Liu are credited as advisors.
- The research is supported by Shanghai Artificial Intelligence Laboratory and acknowledges collaborators on InternVLA-N1 and the InternNav codebase.
A.1 SYSTEM 2: QWENVL2.5
System 2 is trained to produce view adjustments, pixel goals, and STOP decisions from observations and instructions, while subsequent processing converts discrete guidance into smooth trajectories. Progressive training extracts latent goal representations from QwenVL and conditions a diffusion policy on them for obstacle-aware execution.
- System 2 outputs: System 2 predicts self-directed view adjustments, pixel-goal grounding, and STOP actions from observations and instructions.
- Self-Directed View Adjustment: When future trajectories are not projectable, System 2 predicts up to four consecutive turn actions for view adjustment.
- Pixel-Goal Grounding: When a future waypoint is visible, the model predicts the farthest successfully projected waypoint as the pixel goal.
- Trajectory processing: Discrete action waypoints are interpolated into 32 smooth fixed-interval trajectory waypoints.
- Latent representation and policy: Learnable latent queries extract goal representations from frozen QwenVL, and a DiT-based diffusion policy generates smooth world-coordinate trajectories conditioned on them.
- Stage 2 training: Stage 2 trains latent representations and a diffusion policy to predict obstacle-aware trajectories, while preserving generalized pixel-goal grounding through progressive two-stage training.
B ATTENTION MAP ANALYSIS FOR PIXEL-GOAL GROUNDING
The attention analysis examines how System 2 uses multimodal context when predicting pixel goals and STOP actions. Attention shifts from broad contextual cues toward precise goal regions in deeper layers, with deepest layers also emphasizing STOP.
- Attention-map setup: Attention maps visualize System 2’s focus across language instructions, historical frames, and the current observation during pixel-goal prediction.
- Pixel-goal grounding: Shallow layers attend to objects, scene layouts, and directional cues, while deeper layers increasingly focus on the target pixel-goal region.
- Task completion: The deepest transformer layers assign significant attention to STOP when predicting task completion.