PaperlayerView exact revision on arXiv

Source-linked AI summary

NaVILA: Legged Robot Vision-Language-Action Model for Navigation

An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, Xiaolong Wang

arXiv:2412.04453v2cs.ROcs.CV

TL;DR

Language-guided legged navigation requires connecting reasoning and planning with low-level control in challenging environments. NaVILA connects a VLA’s spatial language commands to a visual locomotion policy for execution. It improves classic VLN success rates by over 17% over prior state-of-the-art methods and demonstrates strong real-world performance across legged robots.

  • Problem

    Translating language instructions into legged-robot actions requires integrating reasoning, closed-loop planning, and low-level control for navigation in challenging environments.

    Source: §I · p. 2

  • Method

    NaVILA couples a VLA that outputs spatial language commands with a low-level visual locomotion policy that executes them.

    Source: §I · p. 2

  • Results

    Over 17% improvement in classic VLN success rate over prior state-of-the-art methods accompanies gains over distillation-based locomotion policies and strong real-world performance across legged robots.

    Source: §I · p. 2, §V · p. 9

  • Takeaways & Limitations

    The same VLA can be applied across different robots by replacing the low-level locomotion policy.

    Source: §I · p. 2

    NaVILA still fails in some real-world cases.

    Source: §V · p. 9

Abstract

from arXiv · show

This paper proposes to solve the problem of Vision-and-Language Navigation with legged robots, which not only provides a flexible way for humans to command but also allows the robot to navigate through more challenging and cluttered scenes. However, it is non-trivial to translate human language instructions all the way to low-level leg joint actions. We propose NaVILA, a 2-level framework that unifies a Vision-Language-Action model (VLA) with locomotion skills. Instead of directly predicting low-level actions from VLA, NaVILA first generates mid-level actions with spatial information in the form of language, (e.g., "moving forward 75cm"), which serves as an input for a visual locomotion RL policy for execution. NaVILA substantially improves previous approaches on existing benchmarks. The same advantages are demonstrated in our newly developed benchmarks with IsaacLab, featuring more realistic scenes, low-level controls, and real-world robot experiments. We show more results at https://navila-bot.github.io/

I. INTRODUCTION · II. METHOD

NaVILA couples a VLA that produces natural-language navigation actions with a real-time locomotion policy that executes joint movements. It reports improved VLN success rates and deployment on multiple legged robots.

I. INTRODUCTION

  • VLN follows language instructions through unseen environments without maps [1] [2] [3] [4] [5], while legs extend navigation to cluttered spaces and difficult terrain.Navigation requires language reasoning, closed-loop planning, and low-level control. Existing end-to-end VLAs [7] [8] fine-tune VLMs on manipulation demonstrations to produce low-level actions, motivating alternatives to translating language reasoning directly into precise, non-verbal commands.Source: §I · p. 2
  • Language-based mid-level actions enable robot-independent VLA reuse, diverse training sources, and separate navigation and locomotion timescales.Training can incorporate human videos and reasoning QA; slower VLA inference leaves real-time locomotion to handle obstacle avoidance and improve robustness. Swapping locomotion policies enables deployment on Unitree Go2, Unitree H1, and Booster T1.Source: §I · p. 2
  • Historical context, current observations, navigation-specific prompts, YouTube touring videos, and curated dataset blending adapt an image-based VLM for navigation while retaining general vision-language training.The authors identify this as the first demonstration that direct training on human videos improves navigation in continuous environments.Source: §I · p. 2
  • A single-stage visual locomotion policy uses LiDAR-derived height maps and randomization, converting VLA outputs into command velocities tracked through joint-position control.Randomization addresses the sim-to-real gap, supporting robust, safe deployment under conditions including strong sunlight and proximity to certain transparent surfaces.Source: §I · p. 2
  • Over 17% improvement in success rate over state-of-the-art methods is reported on classic VLN benchmarks.Real-world deployment achieves an 88% success rate on 25 instructions, including a 75% success rate on complex instructions across diverse scenes.Source: §I · p. 2
  • 14% improvement in success rate over a blind policy is reported for the vision-based policy on VLN-CE-Isaac.The new Isaac Sim benchmark models detailed joint movements and robot–environment interactions absent from prior VLN work. The single-stage locomotion policy also outperforms previous policy distillation-based methods.Source: §I · p. 2

II. METHOD

  • NaVILA processes single-view images to generate natural-language waypoint instructions, which a locomotion policy translates into precise joint movements for real-time control.The two-level framework separates high-level visual-language understanding from low-level execution.Source: §II · p. 2

A. Taming VLMs for Vision Language Navigation

NaVILA adapts image-based VILA to navigation by distinguishing current observations from historical memory and learning from human videos, simulated trajectories, and auxiliary datasets. Full-model fine-tuning produces language outputs that are parsed into action types and spatial arguments.

VILA Preliminary

  • NaVILA builds on image-based VILA [13] [14] [15] [17] [18] [19], leveraging its multi-image reasoning rather than video encoders constrained by limited high-quality video-text pre-training data.VILA converts images into visual tokens, maps them into the language domain through an MLP projector, and combines them with text for autoregressive LLM generation.Source: A. Taming VLMs for Vision Language… · p. 3

Navigation Prompts

  • Navigation prompts distinguish the latest observation from historical memory, preserving the first frame and uniformly sampling earlier frames.Textual cues identify current and historical observations alongside the navigation instruction, avoiding additional special tokens and keeping the LLM’s inputs and outputs in language.Source: A. Taming VLMs for Vision Language… · p. 3

Learning from Human Videos

  • 20K trajectories from 2K egocentric YouTube touring videos provide continuous-navigation supervision through estimated camera poses and automatically generated instructions.Entropy-based sampling [26] selects trajectories; MASt3R [27] estimates poses for step-wise actions, while VLM [13] captioning and LLM rephrasing generate instructions.Source: A. Taming VLMs for Vision Language… · pp. 3–4, Figure 4 · p. 5

Supervised Fine-tuning Data Blend

  • The supervised fine-tuning blend combines real-video navigation, simulated navigation, auxiliary navigation tasks, and general VQA to balance embodied specialization with broad knowledge.Habitat shortest-path rollouts from R2R-CE [29] and RxR-CE supply step-wise observations and oracle actions. Merging up to three consecutive actions diversifies distance and angle labels, while label rebalancing addresses underrepresented stop actions.Source: A. Taming VLMs for Vision Language… · p. 4
  • Auxiliary training adds EnvDrop instructions, trajectory summarization, ScanQA spatial questions, and general VQA datasets [23].Trajectory summarization predicts annotated instructions from sampled trajectory frames; ScanQA uses multi-view RGB images from real-world scans. General VQA data is included to retain the model’s general capabilities.Source: A. Taming VLMs for Vision Language… · p. 4

Training and Inference Paradigm

  • One epoch of full-model fine-tuning applies the four-source data blend to VILA’s stage-two checkpoint, followed by parsing language outputs into executable action specifications.The vision encoder, connector, and LLM are all unfrozen. A regular-expression parser extracts action types and distance or angle arguments, successfully matching and mapping every action across the reported experiments.Source: A. Taming VLMs for Vision Language… · p. 4

B. Visual Locomotion Policy · III. EXPERIMENTS

NaVILA’s visual locomotion policy converts language navigation commands into leg-joint actions, using LiDAR terrain perception and simulation training before direct real-world deployment. The experiments assess navigation and spatial understanding, locomotion training, simulated locomotion navigation, and real-robot deployment.

B. Visual Locomotion Policy

  • The Go2 locomotion policy converts high-level navigation commands into desired positions for 12 leg-joint motors and transfers directly from simulation to the real robot.Training uses Isaac Sim with Isaac Lab. The robot’s six base degrees of freedom remain unconstrained, and desired joint positions are converted into torques using stiffness and damping.Source: B. Visual Locomotion Policy · p. 5
  • PPO [37] trains an asymmetric actor–critic: the critic receives privileged terrain information, while the actor uses only observations available on the real robot.The critic receives current proprioception, velocity commands, and a terrain-height scan. The actor excludes unavailable linear velocity and instead uses proprioceptive history to infer it implicitly.Source: B. Visual Locomotion Policy · p. 5
  • Language actions map to fixed forward, turning, or stopping velocities, with execution duration adjusted to match the VLM’s specified movement.Forward motion uses 0.5 m s^-1; left and right turns use π/6 rad s^-1 and -π/6 rad s^-1, respectively; stopping uses zero velocity.Source: B. Visual Locomotion Policy · p. 5
  • Over 60K FPS training throughput on an RTX 4090 GPU is achieved with Isaac Lab ray-casting and single-stage visual locomotion reinforcement learning.Unlike two-stage teacher–student approaches, this approach eliminates policy distillation. Direct environment interaction also allows exploration and potentially novel strategies.Source: B. Visual Locomotion Policy · p. 5
  • LiDAR supplies a 2.5D terrain height map, chosen for transparent-object detection and robust performance under strong sunlight.The Unitree L1 covers 360° × 90°; preprocessing selects each voxel’s lowest in-range value and applies a maximum filter over the last five point clouds.Source: B. Visual Locomotion Policy · p. 5

III. EXPERIMENTS

  • The experiments compare VLA performance against state-of-the-art methods on VLN-CE and general spatial understanding, and compare single-stage locomotion with policy-distillation approaches.These comparisons are posed as experimental questions; this passage supplies no outcomes.Source: §III · p. 5
  • Further experiments investigate simulator evaluation of locomotion navigation, NaVILA’s effectiveness and flexibility in those scenarios, and deployment in real-robot vision-and-language navigation.The simulator and real-robot investigations are assigned to Sections III-C and III-D, respectively.Source: §III · p. 5

A. High-level VLA Performance · B. Low-level RL Policy Performance

NaVILA outperforms navigation baselines without simulator-pretrained waypoint predictors and improves spatial scene understanding. Its low-level RL policy outperforms ROA on velocity tracking errors and collision rate.

A. High-level VLA Performance

  • NaVILA surpasses all baselines without simulator-pretrained waypoint predictors on R2R-CE [29] and RxR-CE [30] Val-Unseen using one single-view RGB model.Evaluation uses continuous photorealistic indoor environments and metrics NE, OS, SR, SPL, and nDTW. The authors report the first single-view RGB-trained VLN agent achieving comparable or superior results to models using panoramic views, odometry, or simulator-pretrained waypoint predictors.Source: A. High-level VLA Performance · pp. 5–6, Table I · p. 6
  • 10% improvement in SR over NaVid [12] is reported on RxR-CE [30] Val-Unseen without training on RxR.Following the cross-dataset protocol of [12], NaVILA trains exclusively on R2R samples and evaluates zero-shot on RxR.Source: Table II · p. 6, A. High-level VLA Performance · p. 6

B. Low-level RL Policy Performance

  • The ROA comparison uses the same 2.5D heightmap input but trains an adaptation encoder through supervision from a privileged encoder.ROA’s privileged encoder processes height scan points and other privileged observations, providing the policy-distillation baseline for NaVILA’s RL policy.Source: B. Low-level RL Policy Performance · p. 6

A. High-level VLA Performance

  • 20 points higher on CIDEr than NaviLLM, NaVILA answers ScanQA [32] Validation questions using multi-view images from real-world scans.This evaluation tests spatial scene understanding, including object localization, referring, and spatial reasoning.Source: A. High-level VLA Performance · p. 6, Table III · p. 7
  • 64 input frames yield NaVILA performance superior to state-of-the-art 3D-based large multimodal models on ScanQA Validation.Those models require 3D scans or RGBD data with camera poses, whereas NaVILA uses multi-view images.Source: A. High-level VLA Performance · p. 6, Table III · p. 7

B. Low-level RL Policy Performance

  • NaVILA’s low-level RL policy outperforms ROA on linear velocity error, angular velocity error, and collision rate, with particularly lower collision rates.Velocity errors measure command-following accuracy, while collision rate measures obstacle avoidance.Source: B. Low-level RL Policy Performance · pp. 6–7

C. Legged Robot Navigation Performance in Simulation · D. Real World Evaluation

NaVILA’s vision-based locomotion improves success over a blind policy in VLN-CE-Isaac, a benchmark incorporating realistic robotic execution. Real-world evaluations show better SR and NE than GPT-4o, improved generalization with human videos, and humanoid transfer without VLA retraining.

C. Legged Robot Navigation Performance in Simulation

  • 1,077 of 1,839 R2R Val-Unseen trajectories form VLN-CE-Isaac, an Isaac Sim benchmark evaluating navigation from high-level planning through precise robotic execution.Trajectories are selected for traversability and high-quality meshes, while simulation captures joint movements and environmental interactions. Evaluation uses the same metrics as prior work.Source: C. Legged Robot Navigation Performance… · p. 7, Figure 6 · p. 7
  • 14% higher Success Rate on Go2 and 21% higher on H1 distinguish the vision-based policy from the blind policy in VLN-CE-Isaac.The comparison tests Unitree Go2 and Unitree H1, with the gains attributed to superior obstacle avoidance.Source: Table IV · p. 7, C. Legged Robot Navigation Performance… · p. 7
  • 15% lower success rate on Go2 and 27% lower success rate on H1 occur without Oracle’s idealized low-level policy.Oracle assumes perfect command execution without realistic physics, so these gaps highlight the benchmark’s execution challenges. H1 also achieves substantially lower success than Go2, attributed to its larger size.Source: Table IV · p. 7, C. Legged Robot Navigation Performance… · p. 7

D. Real World Evaluation

  • NaVILA significantly outperforms GPT-4o in real-world SR and NE across Workspace, Home, and Outdoor environments.Evaluation uses 25 instructions, each repeated three times. Simple tasks contain one or two commands without inter-room navigation; complex tasks contain at least three commands spanning multiple rooms or landmarks.Source: D. Real World Evaluation · p. 7, Table VI · p. 8
  • Human videos improve outdoor generalization and increase success rates across all evaluated real-world environments.These findings come from an ablation of training with human videos.Source: D. Real World Evaluation · pp. 7–8
  • NaVILA transfers the same VLA model to a Booster Dynamics T1 humanoid without retraining and consistently outperforms baselines.The transfer accommodates changes in camera height and viewing angle.Source: D. Real World Evaluation · p. 8
  • Integrating speech recognition enables voice-controlled navigation through NaVILA.The integration is demonstrated qualitatively in Fig. 7.Source: D. Real World Evaluation · p. 8

IV. RELATED WORK · V. CONCLUSION AND LIMITATIONS

NaVILA combines language-level navigation with real-time locomotion, reporting a 17% gain on classic VLN benchmarks and strong real-world performance. Its limitations include some real-world failures and the computational demands of image-based vision-language models.

IV. RELATED WORK

  • VLN progressed from discrete node-based navigation [1] to continuous environments [12], exposing motion-control challenges overlooked by earlier settings.Visual navigation developed from map-based approaches toward imitation learning and reinforcement learning. Large pretrained models [25] and pretraining techniques [24] brought discrete VLN toward human-level performance, whereas continuous navigation requires movement and rotation rather than teleportation.Source: §IV · p. 8
  • Simulator-trained waypoint approaches improve continuous navigation but face generalization limitations and omit low-level motion planning and obstacle avoidance.Their candidates cover only nearby positions; NaVILA instead addresses high-level decisions and full robot motion control. The paper also introduces a VLN benchmark built on Isaac Sim for more realistic simulation.Source: §IV · p. 8
  • Existing robot foundation models emphasize manipulation [7] [8] or restricted navigation goals, motivating NaVILA’s focus on complex instruction-following.Ding et al. generate executable low-level actions for legged robots, while skill-bank approaches use VLM or LLM controllers to select specialized policies. The passage identifies difficulty comprehending complex navigation instructions as a remaining gap.Source: §IV · p. 8
  • NaVILA introduces single-stage RL with LiDAR inputs during training to improve locomotion-learning efficiency and robustness.Proprioception-only methods struggle with obstacle avoidance, while vision-based methods are vulnerable to extreme conditions such as intense sunlight. Lee et al. require two-stage training, whereas Miki et al. use different height-map construction procedures during training and deployment, including an external tool[110].Source: §IV · p. 8
  • Real-world demonstrations show NaVILA following long-horizon instructions, avoiding obstacles, traversing challenging terrain, and climbing stairs.Figure 7 also shows speech recognition [70] enabling voice commands beginning with “Hey Robot!” and navigation through multiple household areas to a specified goal.Source: Figure 7 · p. 9

V. CONCLUSION AND LIMITATIONS

  • 17% gain on classic VLN benchmarks accompanies NaVILA’s reported superiority over distillation-based low-level policies and vision-blind policies on VLN-CE-Isaac1K.The conclusion also reports strong real-world performance across diverse environments and legged robots.Source: §V · p. 9
  • NaVILA separates high-level language commands from real-time locomotion and obstacle avoidance, preserving reasoning and enabling direct learning from human videos.The conclusion attributes reduced overfitting, better generalization, and robustness across robots to this two-level design.Source: §V · p. 9
  • Real-world failures and computationally intensive image-based vision-language models remain limitations of NaVILA.The authors suggest larger-scale training in realistic simulations to improve generalizability and spatial understanding, and advances in long-context LLMs for more efficient sequence processing.Source: §V · p. 9

2) Human Touring Video Data: · 3) Different Memory Sizes:

Human touring video data improves simulated and real-world navigation, while eight history frames cover most R2R-CE instruction horizons with limited gains from larger memories. Real-world experiments use eight frames because of latency constraints.

2) Human Touring Video Data:

  • Adding human touring video data also yields higher success rates and fewer navigation errors in real-world experiments.These real-world findings are reported in Table VI.Source: 2) Human Touring Video Data · p. 14
  • The touring-video results support the data pipeline’s effectiveness and NaVILA’s ability to integrate data from diverse sources.The authors identify this easy integration as a source of framework scalability.Source: 2) Human Touring Video Data · p. 14
  • Approximately 5% improvements in OS, SR, and SPL follow adding YouTube human touring video data in R2R-CE simulations.The † marker identifies models trained without human touring videos.Source: Table VIII · p. 14, 2) Human Touring Video Data · p. 14

3) Different Memory Sizes:

  • 8 history frames are sufficient to cover most instruction horizons on R2R-CE.Memory size is defined as the number of history frames.Source: 3) Different Memory Sizes · p. 14
  • Increasing memory beyond eight frames provides limited additional navigation performance gains on R2R-CE.The memory-size ablation therefore finds little benefit from retaining longer histories for this benchmark.Source: 3) Different Memory Sizes · p. 14
  • 8-frame memory is used in real-world experiments because of latency constraints.The deployment choice accounts for latency rather than simply maximizing history length.Source: 3) Different Memory Sizes · p. 14

B. More Qualitative Results · 1) VLN-CE-Isaac: · 2) Real-World:

Qualitative results show the Go2 vision policy avoiding an obstacle that traps the blind policy, while NaVILA demonstrates robust real-world performance and generalization across diverse environments.

1) VLN-CE-Isaac:

  • The vision-based locomotion policy uses LiDAR input to autonomously avoid dangers that the high-level VLA misses.The policy was trained to handle obstacles.Source: 1) VLN-CE-Isaac · p. 14
  • The high-level VLA, not specifically trained for obstacle avoidance, failed to navigate effectively around the obstacle.Source: 1) VLN-CE-Isaac · p. 14
  • The Go2 vision policy avoided an obstacle that left the blind policy stuck.The blind policy followed the VLA’s commands without additional sensory input.Source: Figure 8 · p. 14, 1) VLN-CE-Isaac · p. 14

2) Real-World:

  • NaVILA demonstrates robust performance and generalization across diverse real-world settings.These are qualitative demonstrations rather than quantified benchmark comparisons.Source: 2) Real-World · p. 14, Figure 9 · p. 15
  • Real-world demonstrations span urban streets, campus sidewalks, courtyards, and different houses.Source: Figure 9 · p. 15
  • The demonstrated real-world settings introduce challenging terrains, dynamic objects, and different lighting conditions.Source: Figure 9 · p. 15

3) Spatial Scene Understanding: … 2) VLA Hyperparameters:

NaVILA’s qualitative ScanQA results show object grounding and localization from video frames, while auxiliary trajectory summarization targets scene understanding and instruction-format familiarity. Instruction fine-tuning uses a learning rate of 1e−4 with cosine decay and a 0.03 warm-up ratio.

3) Spatial Scene Understanding:

1) Video Navigation Trajectory Summarization:

  • The auxiliary trajectory-summarization task infers navigation instructions from video history frames to improve scene understanding and instruction-format familiarity.Following [12], prompt templates cast the LLM as a navigation robot, insert frame tokens, and ask it to describe the navigation trajectory.Source: 1) Video Navigation Trajectory… · p. 16

2) VLA Hyperparameters:

3) Locomotion Motion Policy: · 4) VLA Compute Resources:

NaVILA’s Go2 locomotion policy is trained on flat, rough, slope, and obstacle terrains. Its 8B VLA model can run inference on a single RTX 4090 GPU at roughly 1 FPS.

3) Locomotion Motion Policy:

4) VLA Compute Resources:

D. Parameter-efficient Quantization · E. Limitations

W4A16 quantization reduced NaVILA’s memory requirements and improved processing speed while preserving robust navigation. A real-world failure exposed ineffective error correction; proposed improvements include larger-scale realistic simulation training and explicit reasoning data.

D. Parameter-efficient Quantization

  • Quantization could enable direct onboard NaVILA deployment and reduce image transmission time, but onboard deployment remains future work.Source: D. Parameter-efficient Quantization · p. 16
  • The locomotion policy demonstrates collision-free avoidance of high grass, certain transparent glass, and large objects under strong sunlight, with robustness on sand and grass.Source: Figure 12 · p. 17
  • About 40% faster processing and halved memory requirements followed AWQ W4A16 quantization of FP16 NaVILA-8B, while navigation remained robust.Computational cost was tested on an RTX 4090 using 1737 context tokens and 10 generated tokens for an R2R-CE sample.Source: D. Parameter-efficient Quantization · p. 16, Table XIII · p. 17

E. Limitations

  • A real-world robot initially followed its prompt but failed to reach the bedroom because it could not effectively correct deviations.The authors identify generalizability and spatial understanding as key areas for improvement.Source: E. Limitations · p. 17
  • Larger-scale training in more realistic simulations could improve NaVILA by providing more diverse navigation scenarios and error recovery cases.Source: E. Limitations · p. 17
  • Incorporating explicit reasoning data during training could help NaVILA anticipate and correct mistakes.Source: E. Limitations · p. 17

Related papers

Loading 2412.04453v2…