PaperlayerView exact revision on arXiv

Source-linked AI summary

NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation

Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, He Wang

arXiv:2402.15852v7cs.CVcs.RO

TL;DR

VLN must generalize navigation from linguistic instructions to unseen environments, including the transition from simulation to reality. NaVid uses a video-based VLM that plans executable actions from monocular RGB video and instructions without maps, odometers, or depth, and reports state-of-the-art-level performance in simulation and real-world evaluations.

  • Problem

    VLN generalization across unseen scenes and from simulation to real-world environments remains a longstanding challenge, especially with input domain gaps and data scarcity.

  • Method

    NaVid combines video observation encoding with a language model to infer executable next-step navigation actions from monocular RGB video and human instructions.

  • Results

    NaVid achieves state-of-the-art-level navigation performance in simulation, improves cross-dataset evaluation, and reaches about 66% success on 200 real-world instructions across four indoor scenes.

  • Takeaways & Limitations

    The results demonstrate generalization to real-world VLN using only monocular video inputs and no odometers, depth sensors, or maps.

  • Takeaways & Limitations

    NaVid has long latency from computation costs and may lose performance on very long-horizon instructions because of long-context-token limitations and limited high-quality long-video annotations.

Abstract

from arXiv · show

Vision-and-language navigation (VLN) stands as a key research problem of Embodied AI, aiming at enabling agents to navigate in unseen environments following linguistic instructions. In this field, generalization is a long-standing challenge, either to out-of-distribution scenes or from Sim to Real. In this paper, we propose NaVid, a video-based large vision language model (VLM), to mitigate such a generalization gap. NaVid makes the first endeavor to showcase the capability of VLMs to achieve state-of-the-art level navigation performance without any maps, odometers, or depth inputs. Following human instruction, NaVid only requires an on-the-fly video stream from a monocular RGB camera equipped on the robot to output the next-step action. Our formulation mimics how humans navigate and naturally gets rid of the problems introduced by odometer noises, and the Sim2Real gaps from map or depth inputs. Moreover, our video-based approach can effectively encode the historical observations of robots as spatio-temporal contexts for decision making and instruction following. We train NaVid with 510k navigation samples collected from continuous environments, including action-planning and instruction-reasoning samples, along with 763k large-scale web data. Extensive experiments show that NaVid achieves state-of-the-art performance in simulation environments and the real world, demonstrating superior cross-dataset and Sim2Real transfer. We thus believe our proposed VLM approach plans the next step for not only the navigation agents but also this research field.

I. INTRODUCTION

VLN requires agents to follow linguistic instructions in unseen, continuous environments, but generalization across scenes and from simulation to reality remains difficult. NaVid addresses this with a video-based VLM that uses monocular RGB video and instructions to plan executable next-step actions without maps, odometers, or depth.

  • VLN challenges agents to interpret complex visual observations and fine-grained linguistic instructions while navigating diverse, especially unseen environments.
  • Generalization remains difficult because existing continuous-environment methods face data scarcity and domain gaps from RGBD, odometer, and map inputs.
  • NaVid uses a robot’s monocular video and human-issued instructions to infer executable next-step actions end to end in continuous environments.Unlike coarse navigation generalists and LLM-based discrete-space systems, it directly predicts low-level actions with quantitative arguments.
  • NaVid transfers pretrained vision and language knowledge to navigation by encoding instruction-relevant and instruction-agnostic visual features from each frame.
  • Video-based modeling encodes visual observations without depth, odometer data, or maps, addressing generalization challenges associated with odometer noise and domain discrepancies.
  • 66% success rate was achieved on 200 instructions across four diverse indoor scenes in Sim-to-Real deployment using only RGB videos.The evaluation also reports SOTA-level performance on VLN-CE R2R and significant cross-dataset improvement on R2R-RxR.

III. PROBLEM FORMULATION

VLN-CE is formulated as next-step low-level action planning from a monocular RGB video history and a natural-language instruction. NaVid encodes current and historical observations with instruction-aware and instruction-agnostic visual tokens before an LLM outputs executable actions.

  • III. PROBLEM FORMULATION: At time t, NaVid receives instruction I and video frames {x0, · · ·, xt} to plan the next low-level action at+1.The observation space contains monocular RGB video only, while actions combine qualitative types with quantitative arguments.
  • IV. THE PROPOSED NAVID AGENT: NaVid is a video VLM-based navigation agent that transfers general VLM knowledge to realistic VLN-CE action planning.The agent is introduced as an end-to-end system for the formulation defined in Section III.
  • A. Overall Architecture: NaVid uses a vision encoder, query generator, LLM, and two cross-modality projectors to transform video observations and instructions into language-aligned tokens.The video is encoded into observation tokens, while instructions are separately tokenized for processing by the language model.
  • B. The VLN-CE Modeling of NaVid: Each frame produces instruction-queried tokens for instruction-relevant features and instruction-agnostic tokens for globally encoding fine-grained visual information.The frame embedding uses Nx = 256 visual patches before these tokenization operations.
  • B. The VLN-CE Modeling of NaVid: A QFormer-based query generator uses cross-modality interaction between visual and textual embeddings to produce instruction-aware queries.The resulting queried tokens are obtained through cross-attention between visual embeddings and generated queries.
  • B. The VLN-CE Modeling of NaVid: Instruction-agnostic tokens are generated by grid pooling and cross-modality projection, preserving fine-grained information needed for executable robot actions.Grid pooling squeezes the visual tokens from Nx to Nv; two-token frame representations designed for high-level question answering are insufficient here.
  • B. The VLN-CE Modeling of NaVid: Historical frames receive 4 instruction-agnostic tokens each, whereas the current frame receives 64 to reflect their different geometric-information requirements.The current frame supports primary action reasoning, while historical frames provide context for tracing navigation progress.
  • B. The VLN-CE Modeling of NaVid: Special tokens distinguish historical and current observations, and <NAV> prompts the LLM to process instructions and output robotic actions in linguistic form.The input format places historical frames, the current frame, and instruction content into explicitly delimited token sequences.

C. The Training of NaVid

NaVid uses hybrid training to address limited simulation-data diversity, authenticity, and scale. The strategy combines non-oracle trajectories with action planning and instruction-reasoning auxiliary tasks.

  • C. The Training of NaVid: NaVid’s hybrid training strategy targets limited diversity, authenticity, and scale in available navigational simulation data.It combines non-oracle trajectory collection with auxiliary-task co-training.
  • C. The Training of NaVid: Non-oracle trajectories are added because training only on oracle paths diverges from practical conditions and diminishes navigation-strategy robustness.The collection procedure is inspired by Dagger and exposes NaVid to its own executed trajectories.
  • C. The Training of NaVid: 320k oracle step-wise samples from 61 MP3D indoor scenes are combined with 180k non-oracle samples collected by deploying a trained agent.The combined trajectories form the final VLN-CE R2R navigation training data.
  • C. The Training of NaVid: NaVid co-trains VLN-CE action planning with auxiliary tasks for environmental understanding and instruction following.The auxiliary tasks complement action planning by targeting capabilities required for navigation.
  • C. The Training of NaVid: The instruction-reasoning task asks NaVid to infer corresponding instructions from video-based navigation trajectories.It contains 10k trajectories and reuses the shared input-output organization for trajectory descriptions and human-labeled instructions.

D. Implementation details

The experiments evaluate NaVid in simulated and real indoor environments using standard continuous-navigation benchmarks and robot deployments. Implementation details cover training, action parsing, hardware, baselines, and evaluation metrics.

  • Training configurations: Training uses 24 NVIDIA A100 GPUs for approximately 28 hours, totaling 672 GPU hours.Video-caption data are sampled at 1 FPS, while navigation-action data retain all frames, mostly below 300 frames.
  • Evaluation configurations: Predicted language actions are converted to valid actions with regular-expression matching, and real-world inference runs remotely while commanding a local robot.The parser achieves a 100% valid-action rate in VLN-CE val-unseen R2R evaluation.
  • Simulated environments: NaVid is evaluated on VLN-CE R2R and RxR, with 10,819 R2R training examples, 1,839 R2R val-unseen examples, and 1,517 RxR val-unseen examples.The splits support cross-split and cross-dataset evaluation.
  • Real-world environments: Real-world tests use four indoor scenes—Meeting_room, Office, Lab, and Lounge—with simple landmark and complex composite instructions.Composite instructions require completing 2–5 landmark-following tasks.
  • Real-world environments: Experiments run on a Turtlebot4 equipped with a Kinect DK camera, while odometry-based baselines additionally use an RPLIDAR A1M8 and Nav2.The camera captures RGB and depth images; the lidar supports localization and mapping for applicable baselines.
  • Metrics: Evaluation reports success rate, oracle success rate, SPL, trajectory length, and navigation error, with SPL as the primary metric.SPL reflects both navigation accuracy and efficiency.
  • Baselines: Baselines directly predict low-level action primitives, including RGBD and RGB variants of Seq2Seq and CMA.Seq2Seq uses a recurrent policy, whereas CMA uses cross-modal attention between instructions and observations.

B. Comparison on Simulated Environment

In simulated VLN-CE environments, NaVid achieves strong RGB-only navigation and generalizes across datasets. Video-based trajectory modeling substantially outperforms text- and map-based alternatives, while remaining effective on longer-horizon instructions.

  • VLN-CE R2R: 37.4% SR and 35.9% SPL improve over Seq2Seq-RGB and CMA-RGB, while NaVid surpasses WS-MGMap by 1.6% SPL using only RGB observations.The method does not use depth or odometry information.
  • Long-horizon instructions: Web-scale video co-tuning includes sequences up to 300 frames, supporting longer historical context than most VLN tasks typically require.This training strategy is presented as a possible explanation for robust long-horizon performance.
  • VLN-CE RxR: 41.7% better SR and 236.5% better SPL than A2Nav demonstrate strong zero-shot cross-dataset performance on RxR Val-Unseen.NaVid is trained on R2R trajectory-instruction samples and evaluated on unseen RxR data.
  • Trajectory modeling: Video-based trajectory modeling yields 35.9% SPL versus 20.8% with text history and 8.97% with 2D maps.The paper attributes this advantage to retaining richer visual information from navigation history.
  • Baseline comparison: NaVid significantly outperforms LM-Nav on RxR Val-Unseen despite LM-Nav using predefined oracle landmarks.The comparison links NaVid’s advantage to following verbs and directive commands alongside landmarks.

C. Comparison on Real-world Environment.

In real-world environments, NaVid improves substantially over end-to-end and map-based baselines across diverse indoor scenes. It follows both simple and complex instructions using only RGB video, including targeted stopping and rotation behaviors.

  • Overall real-world performance: NaVid shows significant improvement over Seq2Seq, CMA, and WS-MGMap in the real-world evaluation.Seq2Seq and CMA perform poorly, while WS-MGMap benefits from an ongoing semantic map.
  • Instruction following: NaVid accurately parks near a specified target despite initially facing a similarly classified chair.This rotation-following case tests both target disambiguation and orientation control.
  • Overall real-world performance: NaVid completes about 84% of instructions and about 48% of complex instructions across four diverse real-world scenes using only RGB videos.The evaluation includes Meeting Room, Office, Lab, and Lounge environments.
  • Instruction following: NaVid follows similar forward instructions and stops close to their distinct specified stop conditions.The visual examples test instruction understanding under different stopping requirements.

D. Ablation Studies.

The ablations show that training-data composition, special tokens, and visual-token count materially affect NaVid’s navigation performance and efficiency. More data helps primarily before DAgger saturation, while four visual tokens balance performance and inference cost.

  • Training strategy: Co-tuning data is critical, while instruction-reasoning and DAgger navigation-action samples improve performance and support generalizability.The authors suggest that collecting more navigation-related data could further improve NaVid.
  • Network architecture: Removing task and observation special tokens causes a noticeable performance drop, supporting their role in NaVid’s architecture.The ablation removes the [NAV], [HIS], and [OBS] tokens while retaining instruction-queried and instruction-agnostic content tokens.
  • Visual-token count: 1.60% higher success rate but 122% higher time costs result when visual tokens increase from setting (2) to setting (3).The authors therefore identify four visual tokens per frame as the best balance between visual information and computational cost.
  • Data scale: Increasing navigation data from fewer than 280k to 330k produces an obvious performance boost during pre-dagger training, whereas post-dagger gains are minimal.The analysis tracks SR, OS, SPL, and NE across pre-dagger and post-dagger phases.

VI. DISCUSSION AND CONCLUSION

NaVid is presented as a video-based VLM for VLN that uses monocular video and instructions without maps, odometers, or depth sensors. Experiments report strong simulation performance, real-world generalization, and a limitation from latency and long-horizon context.

  • Discussion and conclusion: NaVid achieves SOTA navigation performance without relying on odometers, depth sensors, or maps.It encodes historical and current navigation data with self-defined special tokens and is trained on 510k navigation samples.
  • Discussion and conclusion: NaVid uses a video-based VLM to process historical and current observations and plan navigation actions from monocular video and human instructions.The training data include action-planning and instruction-reasoning samples collected from continuous environments.
  • Limitations: NaVid’s computation cost causes long latency, while very long-horizon instructions can trigger performance drops from long-context-token limits.The authors point to action chunking, quantization, larger backbones, and long-video data as possible future remedies.
  • Future work: Future work extends NaVid toward mobile manipulation by designing actions for simultaneous mobile-base and robot-arm control.The authors also identify annotated mobile-manipulation videos as necessary for learning instructions and physical interactions.

APPENDIX

The appendix organizes supplemental implementation, instruction, robot-setup, baseline, and experiment details. It also documents real-world indoor environments, example instructions, and the robot hardware used in those experiments.

  • Supplemental organization: The supplemental material contains sections on implementation, instructions, robot setup, baselines, and additional experiment results.These topics are listed as Sections A through E.
  • Real-world experiments: Real-world experiments primarily focus on indoor scenes, instructions, and robot setup, following previous vision-and-language reports.The appendix provides further context for the experimental configuration.
  • Real-world environments: Four indoor environments—Meeting Room, Office, Lab, and Lounge—contain diverse objects and varied lighting conditions.These environments are visualized in Fig. 9.
  • Instructions: An example instruction requires moving to a wall, turning left, passing through a door, facing a chair, and stopping.The instruction illustrates the multi-step, fine-grained commands used for navigation.

C. Robot setup.

The real robot is a Turtlebot 4 equipped with an Azure Kinect DK and a PRLIDAR A1M8. NaVid itself uses only the Kinect’s RGB images, while lidar supports a separate lidar-odometry computation.

  • Hardware: The experimental robot is based on a Turtlebot 4 and uses an Azure Kinect DK to capture RGB and depth images.Although both modalities are captured, the proposed method uses only RGB images.
  • NaVid inputs: NaVid uses only RGB images from the Azure Kinect DK, excluding the captured depth images from its inputs.This matches the method’s monocular-video input design.
  • Odometry setup: A PRLIDAR A1M8 captures a 1D laser point cloud used by lidar odometry to compute the robot’s location and orientation.The lidar-odometry output is part of the robot setup rather than NaVid’s stated input.

D. Baseline details

The paper compares NaVid with image-based and graph-based navigation baselines, including trained and directly used VLM variants. A waypoint-output variant is also evaluated and performs poorly under the paper’s action formulation.

  • VLM baselines: Directly used LLaVA and LLaMA-VID models do not consistently produce valid action outputs despite prompting, chain-of-thought, and captioned historical information.Most responses concern scene descriptions and navigation skills rather than valid action outputs.
  • LLaVA-Nav: LLaVA-Nav is trained using the navigation data and describes sampled historical keyframes as text because LLaVA processes images and text.It uses the same output format as NaVid, while uniformly sampling keyframes, including the first and last frames, to address token limits.
  • Waypoint-NaVid: Waypoint-NaVid directly predicts the next waypoint’s location and orientation, sampling a valid waypoint within the current observation’s field of view.The waypoint is defined by the shortest path to follow the trajectory.
  • Waypoint-NaVid: 16% success rate is achieved by Waypoint-NaVid under thresholds of less than 30cm distance error and less than 30 degrees orientation error.The paper reports that this poor waypoint-prediction performance leads to frequent failures.
  • LM-Nav: LM-Nav is evaluated using Habitat ground-truth landmarks and ground-truth shortest paths because its ViNG visual navigation model is unavailable.These resources enable LM-Nav to navigate effectively to each landmark.

E. Additional Experiments

Additional experiments examine NaVid on sparse-instruction object-goal navigation, single-step prediction, trajectory summarization, and qualitative simulator visualizations. The results indicate robust object-goal performance, action-prediction capability, and concise trajectory descriptions.

  • Object goal navigation: NaVid outperforms Gow and ZSON on object-goal navigation despite not being specifically designed for that task.The paper presents this as evidence of robust performance under sparse instructions.
  • Object goal navigation: NaVid achieves comparable SPL to ESC while using only RGB video instead of ESC’s ground-truth location, orientation, and depth data.The task is evaluated in open-vocabulary and zero-shot settings, as indicated by the object-goal comparison table.
  • Single-step prediction: Single-step prediction evaluates next-movement quality using success rate, stop success rate, and angle-based measures on new R2R Val-Unseen samples.The samples contain image sequences paired with a prediction of the next movement.
  • Trajectory summary: 10k instruction-reasoning samples support trajectory-summary testing across real-world and simulator environments.The model briefly describes trajectories, demonstrating that it can understand its motions.
  • Visual experiments: Figures 15–16 visualize simulator results for VLN-CE R2R, while Figures 17–18 visualize simulator results for VLN-CE RxR.The paper directs readers to an attached video for additional visual results.
Loading 2402.15852v7…