PaperlayerView exact revision on arXiv

Source-linked AI summary

VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memory

Shaoan Wang, Yuanfei Luo, Xingyu Chen, Aocheng Luo, Dongyue Li, Chang Liu, Sheng Chen, Yangang Zhang, Junzhi Yu

arXiv:2601.08665v1cs.ROcs.CV

TL;DR

Existing VLA navigation models are often reactive and lack explicit reasoning and persistent memory for complex, long-horizon tasks. VLingNav combines adaptive chain-of-thought, visual-assisted linguistic memory, reasoning-annotated data, and expert-guided online RL; it reports state-of-the-art benchmark performance and zero-shot real-world transfer, while remaining limited by monocular input and a single-system architecture.

  • Problem

    Current VLA models lack explicit reasoning and persistent semantic memory, while imitation-learning training limits performance beyond expert demonstrations.

  • Method

    VLingNav integrates adaptive chain-of-thought, visual-assisted linguistic memory, a reasoning-annotated dataset, and online expert-guided reinforcement-learning post-training.

  • Results

    VLingNav achieves state-of-the-art or competitive performance across embodied-navigation tasks with one unified model and transfers zero-shot to real-world robots.

  • Takeaways & Limitations

    Deliberate reasoning at an average 2.1% of key steps is reported as sufficient to substantially boost overall task success while supporting efficient deployment.

  • Takeaways & Limitations

    The model primarily uses monocular egocentric observations and a single-system architecture, constraining perception and prediction frequency in dynamic environments.

Abstract

from arXiv · show

VLA models have shown promising potential in embodied navigation by unifying perception and planning while inheriting the strong generalization abilities of large VLMs. However, most existing VLA models rely on reactive mappings directly from observations to actions, lacking the explicit reasoning capabilities and persistent memory required for complex, long-horizon navigation tasks. To address these challenges, we propose VLingNav, a VLA model for embodied navigation grounded in linguistic-driven cognition. First, inspired by the dual-process theory of human cognition, we introduce an adaptive chain-of-thought mechanism, which dynamically triggers explicit reasoning only when necessary, enabling the agent to fluidly switch between fast, intuitive execution and slow, deliberate planning. Second, to handle long-horizon spatial dependencies, we develop a visual-assisted linguistic memory module that constructs a persistent, cross-modal semantic memory, enabling the agent to recall past observations to prevent repetitive exploration and infer movement trends for dynamic environments. For the training recipe, we construct Nav-AdaCoT-2.9M, the largest embodied navigation dataset with reasoning annotations to date, enriched with adaptive CoT annotations that induce a reasoning paradigm capable of adjusting both when to think and what to think about. Moreover, we incorporate an online expert-guided reinforcement learning stage, enabling the model to surpass pure imitation learning and to acquire more robust, self-explored navigation behaviors. Extensive experiments demonstrate that VLingNav achieves state-of-the-art performance across a wide range of embodied navigation benchmarks. Notably, VLingNav transfers to real-world robotic platforms in a zero-shot manner, executing various navigation tasks and demonstrating strong cross-domain and cross-task generalization.

1 Introduction

Existing VLA navigation agents improve adaptability but remain largely reactive, with limited explicit reasoning, persistent memory, and reinforcement-learning-based policy refinement. VLingNav addresses these gaps through adaptive reasoning, visual-assisted linguistic memory, expanded reasoning data, and expert-guided online RL, achieving strong benchmark and real-world transfer results.

  • Motivation: Traditional navigation pipelines rely on manually defined interfaces, while VLA models unify multimodal understanding and action generation but remain reactive.The introduction contrasts modular systems' interface and error-accumulation issues with VLA adaptability.
  • Motivation: Current VLA models lack adaptive deliberation and persistent semantic memory, limiting their ability to handle ambiguity and retain historical context.They commonly use fixed inference budgets and limited context windows.
  • VLingNav: AdaCoT dynamically switches between fast execution and deliberate planning, while VLingMem recalls past observations to prevent repetitive exploration and infer movement trends.The two components target adaptive reasoning and long-term spatial dependencies.
  • Training: Nav-AdaCoT-2.9M provides adaptive reasoning annotations, and online expert-guided RL extends training beyond supervised imitation.The dataset teaches when and what to think about, while post-training supports self-improving behaviors.
  • Results: VLingNav achieves state-of-the-art performance across diverse benchmarks and zero-shot transfer to real-world robots, including unseen navigation tasks.Reported gains cover success rate and efficiency metrics, with no additional fine-tuning for real-world transfer.

2 Related Works

Related work spans modular and end-to-end navigation, video-based VLA models, reasoning-enhanced agents, memory mechanisms, and RL post-training. VLingNav is positioned as an adaptive-thinking, language-centered memory system using continuous actions and expert knowledge for online learning.

  • Embodied Navigation: Embodied navigation covers VLN, ObjectNav, ImageNav, and EVT, requiring robots to follow language instructions while perceiving, reasoning, and planning in unseen environments.These subtasks broaden navigation beyond classical point-to-point movement.
  • Navigation Paradigms: Modular methods offer interpretability through components such as SLAM and planning, whereas end-to-end methods map sensor inputs directly to actions but can generalize poorly out of distribution.The trade-off is between manually aligned modules and data-driven action generation.
  • VLA Navigation: NaVid and Uni-NaVid advance video-based VLA navigation, but long video streams create inference costs and existing models often lack explicit reasoning and memory.Prior models may retain history implicitly through visual features without an explicit memory structure.
  • Reasoning and Memory: VLingNav proposes adaptive thinking to balance reasoning efficiency with navigation capability, alongside language memory augmented by visual features.The memory design is motivated by language pretraining and differs from latent-, vision-, or map-based alternatives.
  • Reinforcement Learning: Prior navigation RL work largely uses autoregressive actions, whereas VLingNav adopts continuous MLP-based actions and incorporates expert knowledge to improve online learning.The design targets limited policy spaces and slow inference associated with alternative action formulations.

3 Methodology

VLingNav combines video-based visual encoding, adaptive reasoning, linguistic memory, and continuous trajectory prediction for embodied navigation. Its online pipeline selectively reasons, updates memory, and converts multimodal representations into robot motion.

  • VLingNav Overview: VLingNav extends LLaVA-Video-7B with an action model that jointly supports text generation and trajectory planning.The action model predicts waypoint trajectories conditioned on the VLM backbone’s outputs.
  • Observation Encoding: Dynamic FPS sampling preserves recent frames densely while sampling older frames sparsely to reduce visual-token computation.The sampling rate decreases with historical time interval, while selectively preserving important images.
  • Observation Encoding: Visual observations are encoded into 729-patch, 1152-dimensional features and compressed with grid pooling for efficient historical summarization.A pretrained SigLIP-400M encoder produces the visual features before pooling.
  • Observation Encoding: Timestamp-aware tokens encode temporal intervals so the model can distinguish historical frames sampled at different rates.Rotary Position Embedding represents the interval between each historical observation and the current frame.
  • Adaptive Reasoning and Memory: The VLM first predicts <think_on> or <think_off>, then generates reasoning and environmental summaries only when explicit thought is triggered.Reasoning covers visual perception, task decomposition, visited-location assessment, and next-action selection; summaries become linguistic memory.
  • Action Model and Online Inference: The final VLM hidden state conditions an MLP action model that outputs the robot’s next motion trajectory during online inference.The online procedure repeatedly encodes observations, samples and pools cached features, updates memory when reasoning is enabled, and executes the predicted action.

4 Data Collection

VLingNav’s data collection combines navigation benchmarks, adaptive reasoning annotations, and open-world video data. The resulting corpus supplies multimodal, trajectory-level supervision for training the model’s reasoning and navigation behaviors.

  • Dataset Construction: The navigation data draws from HM3D ObjNav, MP3D ObjNav, HM3D OVON, EVT-Bench tracking, and HM3D Instance ImageNav.Several sources use shortest-path trajectories, while HM3D ObjNav uses human demonstrations and EVT-Bench supplies multi-person indoor tracking data.
  • Dataset Construction: Nav-AdaCoT-2.9M contains 2.9M step-by-step adaptive CoT trajectories aligned with observations and instructions.Its structured reasoning supervision bridges perception, language, and action rather than providing only instructions and expert action labels.
  • Autonomous Adaptive CoT Labeling: The autonomous labeling pipeline uses a vision-language model to generate step-by-step rationales and linguistic environmental summaries.The generated outputs are filtered with rule-based and subsequent quality-control procedures.
  • Open-World Video Data: The full training corpus contains 4.5M samples: 2.9M embodied-navigation samples and 1.6M open-world-video samples.The open-world video data is included to support visual understanding and adaptive reasoning across varied environments.

5 Training Recipe

VLingNav uses staged training that combines adaptive visual reasoning, navigation imitation learning, and online expert-guided reinforcement learning. Its continuous probabilistic action model and hybrid rollouts target efficient control, exploration, and robustness.

  • Training stages: Adaptive-CoT pre-training equips the VLM with visual reasoning through one epoch of supervised token-level cross-entropy training.
  • Training stages: Supervised fine-tuning co-trains embodied navigation and open-world video data to retain general visual reasoning while learning task-specific navigation skills.
  • Training stages: The training objective combines trajectory mean squared error with cross-entropy supervision for CoT reasoning and VQA outputs, balanced by α.
  • Online post-training: Online post-training collects on-policy trajectories and combines outcome-driven optimization with expert-guided supervision to address imitation-learning covariate shift.
  • Action modeling: A lightweight probabilistic projection head models continuous actions as a multivariate Gaussian, balancing high-precision control with efficient inference.
  • Online post-training: Hybrid rollouts alternate between successful naive trajectories and expert-guided recovery demonstrations, while stochastic exploration is replaced by mean actions during validation.
  • Inference: Inference predicts a CoT indicator, optionally generates reasoning, and feeds the final hidden state to an action module for future trajectory prediction.

6 Experiments

VLingNav is evaluated against state-of-the-art methods across multiple embodied navigation benchmarks using a shared checkpoint without task-specific fine-tuning. Evaluation covers navigation success, efficiency, tracking, and collisions.

  • Evaluation setup: Experiments compare VLingNav with state-of-the-art methods across modular, end-to-end, and VLA model categories.
  • Evaluation setup: A shared model checkpoint is evaluated on ObjectNav, Embodied Visual Tracking, and ImageNav benchmarks without additional task-specific fine-tuning.
  • Metrics: Performance is measured using Success Rate, Success-weighted Path Length, Tracking Rate, and Collision Rate.
  • Benchmark comparisons: Tables 2 and 3 report object-goal navigation comparisons on HM3D, MP3D, and HM3D-OVON benchmarks, marking the best and second-best results.

6.2 Simulation Experiments

Simulation experiments evaluate VLingNav across object navigation, visual tracking, and image-goal navigation benchmarks. The results report strong benchmark performance, efficiency gains, and adaptive reasoning visualizations.

  • Object Goal Navigation: 79.1 SR and 42.9 SPL on HM3Dv1 exceed Uni-NaVid’s 73.7 SR and 37.1 SPL by +5.4 SR and +3.9 SPL.
  • Object Goal Navigation: 83.0 SR on HM3Dv2 surpasses FiLM-Nav’s 77.0 SR by +6.0 SR, while VLingNav’s SPL is slightly lower under the comparison setup.
  • Object Goal Navigation: HM3D OVON evaluates generalization across seen, seen-synonym, and unseen object-category splits.
  • Embodied Visual Tracking: VLingNav improves EVT-Bench performance over TrackVLA++ by 1.1 (+1.7%) in SR and 4.7 (+6.8%) in TR.
  • Visualization: Simulation visualizations combine egocentric observations, scene maps, instructions, adaptive CoT outputs, and predicted trajectories.
  • Image Goal Navigation: VLingNav achieves 60.8 SR and 37.4 SPL on HM3D Instance ImageNav, with SPL improving over UniGoal by 13.7 (+57.8%).

6.3 Real-World Experiments

Real-world experiments deploy VLingNav on a quadruped platform across office, household, and outdoor scenarios. The same model weights support object navigation, visual tracking, and image-goal navigation, with remote inference at approximately 2.5 FPS.

  • Platform and deployment: The robot platform uses a Unitree Go2 with an Intel RealSense D457 camera, transmitting RGB observations over Wi-Fi to a remote server.
  • Platform and deployment: Remote deployment incurs approximately 100 ms communication overhead while achieving around 2.5 FPS during long-horizon experiments.
  • Platform and deployment: The robot tracks predicted trajectories using a nonlinear model predictive control module based on a kinematic unicycle model.
  • Object Goal Navigation: Object navigation is compared with Uni-NaVid across home, office, and outdoor scenes using three target objects per scene and 10 repeated trials per object.
  • Embodied Visual Tracking: Visual tracking evaluates single-target and distracted scenarios across open, cluttered, and crowded environments, with 10 repeated trials per scenario.
  • Image Goal Navigation: Image-goal navigation compares VLingNav with UniGoal across home, office, and outdoor categories using repeated trials for image-specified goals.
  • Evaluation scope: Real-world evaluation covers object navigation, embodied visual tracking, and image-goal navigation across office, household, and outdoor settings using the deployed model weights.

6.4 Emergence of Cross-Task and Cross-Domain Capabilities

Joint multi-task training produces emergent cross-task and cross-domain navigation capabilities. VLingNav transfers these capabilities to real-world tasks, including image-goal tracking and non-human target tracking.

  • Joint training on multi-task navigation datasets yields cross-task and cross-domain capabilities beyond any single task.
  • The real-world deployment results qualitatively demonstrate VLingNav’s navigation performance.
  • VLingNav directly tracks image-goal targets despite training the visual tracking task with only language-format instructions.
  • VLingNav composes search and tracking behaviors across language-described and image-goal targets.
  • VLingNav tracks dynamic non-human targets despite being trained only to track humans.

6.5 Ablation Studies

Ablations show that adaptive reasoning, persistent multimodal memory, online post-training, open-world video co-training, and multi-task learning each improve navigation performance, while excessive training eventually degrades it.

  • Reasoning strategies: Adaptive CoT achieves the highest performance across all benchmarks, outperforming absent, dense, and fixed-interval reasoning strategies.Fixed-interval reasoning improves performance moderately but remains inflexible.
  • Memory modalities: Removing VLingMem causes a substantial performance drop, especially in large or multi-room layouts where agents revisit dead ends or enter loops.Visual-only and linguistic-only memories partially recover performance but remain below the full approach.
  • Open-world video co-training: Open-world video co-training significantly improves performance over training solely on embodied-navigation data.The passage attributes gains to richer semantic priors, cross-modal grounding, generalization, and a reduced sim-to-real gap.
  • Training steps: Success rate rises with training steps, but excessive training produces diminishing returns and eventual degradation, likely from simulation overfitting.The study notes that 1 epoch is approximately 10K training steps.
  • Online post-training: Online post-training significantly outperforms the SFT checkpoint across all evaluated benchmarks, with Hybrid Rollout producing the best rollout performance.The post-training phase teaches shortcut discovery, error recovery, and handling of distribution shift beyond static demonstrations.
  • Multi-task learning: Multi-task models consistently outperform single-task models, including on specialized benchmarks, and improve out-of-distribution generalization.The ablation evaluates joint training across ObjectNav, EVT, and ImageNav.

7 Discussion

VLingNav’s discussion attributes its effectiveness to adaptive computation, persistent linguistic memory, and online expert-guided reinforcement learning. Together, these mechanisms support efficient reasoning, long-horizon robustness, and generalization across tasks and real-world platforms.

  • Effectiveness of Adaptive Thinking: AdaCoT allocates cognitive resources adaptively, using fast execution for simple scenes and deliberate reasoning at ambiguous or critical decisions.
  • Effectiveness of Adaptive Thinking: Deliberate reasoning at an average 2.1% of steps substantially boosts overall task success, supporting efficient deployment on resource-constrained robots.
  • Synergy of Visual-Assisted Linguistic Memory: VLingMem converts key visual observations into linguistic summaries that preserve semantic context and help prevent redundant exploration.
  • Synergy of Visual-Assisted Linguistic Memory: Persistent linguistic memory complements AdaCoT by supplying historical context when detailed chain-of-thought reasoning is not activated.
  • Beyond Imitation Learning: Expert-guided RL post-training enables online exploration and policy refinement beyond imitation learning constrained by expert-data quality and coverage.
  • Generality and Real-world Generalization: A single unified model trained on Nav-AdaCoT-2.9M reaches state-of-the-art or competitive performance across tasks and transfers zero-shot to real-world robots.

8 Conclusion and Limitation

VLingNav integrates adaptive reasoning, multimodal memory, and online expert-guided RL to achieve strong benchmark performance and zero-shot real-world transfer. The authors identify perceptual, decision-frequency, and locomotion limitations as directions for future work.

  • Conclusion: VLingNav combines adaptive reasoning, multimodal memory, and online expert-guided RL in a VLA model for embodied navigation.
  • Conclusion: The model achieves state-of-the-art performance across embodied-navigation benchmarks and transfers directly to real-world robot platforms without fine-tuning.
  • Limitations: Monocular egocentric observations constrain perception through limited field of view, motivating future integration of multi-view inputs.
  • Limitations: The single-system architecture restricts prediction frequency, impeding rapid decision-making and obstacle handling in highly dynamic environments.
  • Limitations: The MPC-based waypoint controller lacks flexible locomotion, limiting movement speed and reachable areas.
Loading 2601.08665v1…