Source-linked AI summary
NaVILA: Legged Robot Vision-Language-Action Model for Navigation
An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, Xiaolong Wang
TL;DR
Legged-robot vision-and-language navigation requires translating human instructions into reliable low-level motion in challenging, cluttered environments. NaVILA uses a two-level VLA and visual locomotion policy, and reports gains on classic and simulated benchmarks plus real-world robot deployment.
Problem
Legged-robot VLN must connect language-based reasoning and closed-loop planning with low-level control, while existing end-to-end systems raise questions about representing actions beyond quantized commands.
Method
NaVILA generates spatial mid-level actions in language with a VLA, then uses a visual locomotion policy to execute them through robot control.
Results
NaVILA improves success rate by over 17% on classic VLN benchmarks and by 14% over a blind policy in VLN-CE-Isaac, while achieving 88% success on 25 real-world instructions.
Takeaways & Limitations
The framework supports navigation across diverse environments and legged robots while preserving a modular separation between reasoning and locomotion skills.
Takeaways & Limitations
NaVILA fails in some real-world cases, and its image-based VLMs are computationally intensive.
Abstract
from arXiv · showhide
This paper proposes to solve the problem of Vision-and-Language Navigation with legged robots, which not only provides a flexible way for humans to command but also allows the robot to navigate through more challenging and cluttered scenes. However, it is non-trivial to translate human language instructions all the way to low-level leg joint actions. We propose NaVILA, a 2-level framework that unifies a Vision-Language-Action model (VLA) with locomotion skills. Instead of directly predicting low-level actions from VLA, NaVILA first generates mid-level actions with spatial information in the form of language, (e.g., "moving forward 75cm"), which serves as an input for a visual locomotion RL policy for execution. NaVILA substantially improves previous approaches on existing benchmarks. The same advantages are demonstrated in our newly developed benchmarks with IsaacLab, featuring more realistic scenes, low-level controls, and real-world robot experiments. We show more results at https://navila-bot.github.io/
I. INTRODUCTION
NaVILA addresses legged-robot vision-and-language navigation by separating language-based navigation reasoning from low-level locomotion control. Its two-level design supports challenging environments, diverse training data, and deployment across robots.
- Motivation: Legged-robot VLN extends mapless language-guided navigation to cluttered environments such as messy laboratories, houses, and uneven outdoor terrain.
- Framework: NaVILA uses a VLA to generate spatial mid-level language actions, while a visual locomotion policy follows them through low-level control.Example commands include “turn right 30 degrees.”
- Framework: Decoupling execution lets one VLA transfer across robots by swapping locomotion policies and avoids overfitting outputs to robot-specific low-level commands.
- Training: NaVILA trains its VLA with historical context, navigation prompts, human touring videos, and a curated blend of navigation and general vision-language data.The training strategy is intended to retain broad generalization while specializing the model for navigation.
- Results: Over 17% improvement in success rate is reported on classic VLN benchmarks, alongside 14% improvement in success rate over a blind policy in VLN-CE-Isaac.The paper also reports deployment across Go2, H1, and T1 robots and 88% success on 25 real-world instructions.
- Framework: The framework combines high-level visual-language understanding with low-level locomotion control for real-time operation and adaptation across real-world environments.
A. Taming VLMs for Vision Language Navigation
NaVILA adapts image-based VLMs for navigation by combining current observations with historical memory and training on heterogeneous navigation data. Human touring videos are converted into continuous navigation examples with generated actions and instructions.
- Model choice: Image-based VLMs are selected because image-text pretraining is more available and these models provide stronger generalization and broader knowledge than video-based alternatives.
- Temporal context: The VLA distinguishes the latest frame for immediate decisions from historical frames that form a memory bank for progress tracking and planning.
- Human-video data: Metric camera-pose estimation extracts step-by-step actions, while VLM captioning and LLM rephrasing generate natural-language instructions for each trajectory.
- Data blend: Training blends continuous-environment navigation data, augmented instructions, trajectory summarization, spatial question answering, and general VQA data to improve scene understanding and retain broad capabilities.
- Training and inference: The model is fine-tuned through supervised training, and inference parses generated language to extract action types and corresponding values.
B. Visual Locomotion Policy
NaVILA’s visual locomotion policy converts high-level navigation commands into joint control for a legged robot. It uses LiDAR-derived height maps and simulation training to support robust deployment in challenging environments.
- Robot and training: The Go2 platform uses a head-mounted LiDAR sensor, and the policy is trained in Isaac Sim with Isaac Lab before direct real-world deployment.
- Control policy: The policy outputs desired joint positions, converts them into simulator torques, and is trained with PPO while separating privileged critic observations from real-world actor observations.
- Terrain perception: The height map supports obstacle avoidance in simulation and detects glass surfaces where depth and RGB images fail.
- Terrain perception: LiDAR point clouds are converted into a 2.5D height map to perceive surrounding terrain for locomotion.
III. EXPERIMENTS
Experiments evaluate NaVILA across VLN, cross-dataset transfer, spatial scene understanding, and low-level control. The results report improvements over comparison methods across these settings.
- Evaluation setup: Experiments compare NaVILA on VLN-CE benchmarks, cross-dataset transfer, spatial scene understanding, and low-level policy control.The evaluation includes R2R-CE and RxR-CE val-unseen splits, ScanQA, and comparison with policy distillation-based approaches.
- VLN-CE benchmarks: NaVILA outperforms baselines without simulator-pretrained waypoint predictors on both R2R-CE and RxR-CE using a single model.The comparison includes methods using additional inputs such as depth, panoramic views, or odometry.
- Cross-dataset transfer: 10% improvement in SR is achieved over NaVid on zero-shot RxR-CE evaluation after training exclusively on R2R samples.The RxR training set is omitted during training.
- Spatial scene understanding: 20 points higher on the CIDEr score is achieved than NaviLLM on the ScanQA validation benchmark.With 64 frames, NaVILA also exceeds 3D multimodal models that require 3D scans or RGBD data with camera poses.
- Low-level policy: NaVILA’s low-level policy outperforms ROA in linear velocity error, angular velocity error, and collision rate.The strongest reported difference is a significantly lower collision rate.
C. Legged Robot Navigation Performance in Simulation
VLN-CE-Isaac evaluates legged-robot navigation in high-fidelity simulation with detailed joint movements and environment interactions. Results compare vision-based and blind policies across Go2 and H1 platforms.
- Benchmark: VLN-CE-Isaac addresses the absence of legged-robot VLN benchmarks with high-fidelity simulation of joint movements and environmental interactions.It uses R2R scenes and selects 1,077 traversable trajectories from 1,839 R2R Val-Unseen trajectories.
- Benchmark: The benchmark evaluates the full navigation pipeline, from high-level planning to precise robotic execution, using prior VLN metrics.Its design excludes unrealistic passages, such as 10 cm gaps that legged robots cannot practically traverse.
- Robot platforms: NaVILA supports evaluation on both Unitree Go2 and Unitree H1 robots within VLN-CE-Isaac.The H1 robot achieves lower success than Go2, consistent with its larger humanoid body size.
- Policy comparison: 14% higher Success Rate is achieved by the vision-based policy than the blind policy on Go2, and 21% higher on H1.The passage attributes these gains to superior obstacle avoidance capability.
- Realistic execution: 15% lower Success Rate on Go2 and 27% lower on H1 are observed when the Oracle low-level policy is not provided.The gaps indicate greater challenges and realism in the benchmark than perfect command execution assumes.
D. Real World Evaluation
NaVILA is evaluated on real-world navigation tasks spanning quadruped and humanoid robots, multiple environments, and simple-to-complex instructions. Results indicate generalization across robots and environments, with voice control and diverse terrain navigation demonstrated.
- Evaluation setup: Real-world experiments use 25 instructions, each repeated three times, across Workspace, Home, and Outdoor environments.Tasks include simple one- or two-command instructions and complex instructions requiring multiple rooms or landmarks.
- Cross-robot generalization: NaVILA generalizes across quadruped and humanoid robots, using the same VLA model on the Booster Dynamics T1 without retraining.The humanoid evaluation includes changes in camera height and viewing angle.
- Real-world results: Human touring videos improve outdoor generalization and success rates across all evaluated environments.The paper reports higher success rates with human-video data in the real-world experiments.
- Capabilities demonstrated: The deployment integrates speech recognition for voice commands and demonstrates long-horizon navigation, obstacle traversal, and challenging-terrain navigation.Commands begin with “Hey Robot!” and the robot can move through different house areas to a specified goal.
- Limitations: The paper identifies remaining failures in some real-world cases and points to larger-scale realistic-simulation training as a route toward improved generalizability and spatial understanding.It also notes that image-based vision-language models are computationally intensive.
2) Human Touring Video Data:
The paper studies whether human touring videos improve navigation and how much historical visual context is useful. Human-video data improves several R2R-CE metrics, while eight history frames are sufficient for most instruction horizons under latency constraints.
- Human touring video data: Adding human touring videos improves OS, SR, and SPL by approximately 5% on R2R-CE.The result comes from an ablation comparing training with and without the additional real-world data.
- Real-world transfer: Human-video training produces better outdoor generalization and higher success rates across all real-world environments.The real-world experiments also report fewer navigation errors with the data pipeline.
- Memory ablation: Eight history frames are sufficient to cover most instruction horizons on R2R-CE, with limited gains from larger memory sizes.The real-world system uses eight frames because of latency constraints.
B. More Qualitative Results
Qualitative results illustrate the role of visual locomotion and the breadth of NaVILA’s deployment settings. The vision policy avoids obstacles that cause the blind policy to become stuck, while demonstrations span varied environments and terrains.
- Obstacle avoidance: The Go2 vision policy detects and navigates around an obstacle that causes the blind policy to fail and get stuck.The VLA commands are unchanged; the distinction is the additional LiDAR-based visual obstacle-handling capability.
- Instruction following: The qualitative examples include navigation instructions involving turning, continuing forward, stepping onto grass, and stopping near a landmark.The examples illustrate short natural-language commands used for navigation.
- Diverse environments: NaVILA demonstrates robust performance across urban streets, campus sidewalks, courtyards, houses, challenging terrains, dynamic objects, and varied lighting.These settings are presented as diverse and challenging real-world environments.
3) Spatial Scene Understanding:
NaVILA’s spatial scene-understanding evaluation uses ScanQA image sequences and asks the model to ground and locate objects. An auxiliary trajectory-summarization task converts video history frames into navigation instructions.
- Spatial scene understanding: On ScanQA, NaVILA qualitatively grounds and locates objects from a sequence of images sampled from a video.The result is presented as spatial scene understanding in Figure 10.
- Trajectory summarization: The auxiliary video-navigation task processes trajectory videos into history frames and prompts an LLM to infer navigation instructions.The prompt frames the model as a robot designed for navigation and asks it to describe the trajectory.
2) VLA Hyperparameters:
The appendix specifies training settings for NaVILA’s VLA and locomotion components, including learning-rate details, terrain diversity, sensor parameters, and compute requirements.
- The instruction fine-tuning stage uses a 1e−4 learning rate with cosine decay and a 0.03 warm-up ratio.
- The locomotion policy is trained on flat, rough, slope, and obstacle terrains with domain randomization and LiDAR or height-map sensing.The associated reward, domain-randomization, and sensor settings are listed in Tables X–XII.
- The first two NaVILA training stages use 16 A100 GPU nodes, while final visual instruction tuning uses 4 A100 GPU nodes.Connector initialization takes 4 hours, visual-language pretraining 30 hours, and final instruction tuning 18 hours.
- During inference, the VLA model runs on a single RTX 4090 GPU at roughly 1 FPS.
D. Parameter-efficient Quantization
NaVILA applies weight-only quantization to reduce inference cost while retaining robust navigation capabilities, supporting more practical robot deployment.
- W4A16 quantization halves memory requirements and improves processing speed by about 40% while preserving robust navigation capabilities.The computational-cost evaluation uses an RTX 4090 with 1737 context tokens and 10 generated tokens on an R2R-CE sample.
- The efficiency gains could enable direct deployment on the robot by reducing image transmission time.The paper leaves direct deployment as future work.
- The locomotion policy is reported to remain collision-free around high grass, certain transparent glass, and large objects under strong sunlight.It also shows robustness on sand and grass terrains.
- A real-world failure case occurs when the robot follows the prompt initially but fails to reach the bedroom after a deviation.The authors attribute this to ineffective error correction and suggest improving generalizability, spatial understanding, simulation diversity, and explicit reasoning data.