PaperlayerView exact revision on arXiv

Source-linked AI summary

Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks

Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, He Wang

arXiv:2412.06224v2cs.ROcs.CV

TL;DR

Existing embodied-navigation approaches are often limited to specific tasks, configurations, or datasets, hindering unified general-purpose navigation. Uni-NaVid addresses this gap with a video-based VLA model trained across four navigation tasks using shared RGB-video and language inputs, online token merging, and joint data. It achieves strong benchmark performance and real-world non-blocking navigation, while remaining bounded by its four-task training scope, standard-size robot assumption, and short-horizon discrete trajectories.

  • Problem

    Existing embodied-navigation methods commonly address one task with specialized modules and datasets, limiting applicability across diverse navigation demands.

  • Method

    Uni-NaVid unifies four navigation tasks in a video-based VLA model using online token merging and joint training with 3.6M navigation samples plus 2.3M video data samples.

  • Results

    Uni-NaVid achieves SOTA or SOTA-comparable performance across diverse navigation benchmarks, including +25.7% Success Rate on R2R versus NaVid, and demonstrates non-blocking real-world navigation.

  • Takeaways & Limitations

    Jointly learning diverse navigation tasks in one RGB-video VLA model supports strong multi-task performance and real-world navigation efficiency.

  • Takeaways & Limitations

    Uni-NaVid is evaluated on four defined tasks, assumes a standard-size robot, and currently predicts only short-horizon discrete trajectories.

Abstract

from arXiv · show

A practical navigation agent must be capable of handling a wide range of interaction demands, such as following instructions, searching objects, answering questions, tracking people, and more. Existing models for embodied navigation fall short of serving as practical generalists in the real world, as they are often constrained by specific task configurations or pre-defined maps with discretized waypoints. In this work, we present Uni-NaVid, the first video-based vision-language-action (VLA) model designed to unify diverse embodied navigation tasks and enable seamless navigation for mixed long-horizon tasks in unseen real-world environments. Uni-NaVid achieves this by harmonizing the input and output data configurations for all commonly used embodied navigation tasks and thereby integrating all tasks in one model. For training Uni-NaVid, we collect 3.6 million navigation data samples in total from four essential navigation sub-tasks and foster synergy in learning across them. Extensive experiments on comprehensive navigation benchmarks clearly demonstrate the advantages of unification modeling in Uni-NaVid and show it achieves state-of-the-art performance. Additionally, real-world experiments confirm the model's effectiveness and efficiency, shedding light on its strong generalizability.

I. INTRODUCTION

Uni-NaVid unifies four embodied navigation tasks within one video-based VLA model, addressing the task-specific design and data limitations of prior approaches. It combines online token merging, multi-task data, and joint training to support efficient navigation from RGB video and language instructions.

  • Existing embodied-navigation methods usually target individual tasks with specialized modules and datasets, limiting multi-purpose applicability and cross-task synergy.
  • Uni-NaVid unifies vision-and-language navigation, object-goal navigation, embodied question answering, and human following in one model.The model is intended to handle distinct task settings and objectives within a shared navigation framework.
  • Online token merging compresses near and far historical frames at different ratios, preserving spatial and temporal information while reducing inference token counts.The mechanism addresses memory overload and computational latency during online navigation.
  • 3.6M navigation samples support training across the four tasks, supplemented by 2.3M real-world video question-answering and captioning samples.The auxiliary data is intended to enhance scene understanding and sim-to-real generalization.
  • Experiments across four navigation benchmarks report SOTA or SOTA-comparable performance, while ablations support benefits from joint multi-task learning and real-world tests show non-blocking robust navigation.The pipeline uses a single VLA model with RGB video streams and instructions as inputs.
  • The model encodes RGB video and language instructions, projects merged visual tokens into the language space, and predicts four subsequent low-level actions.Its action space includes forward, turning, and stop actions for continuous-environment navigation.

IV. MODEL OF UNI-NAVID

Uni-NaVid encodes ego-centric video into visual tokens and applies hierarchical online merging to retain recent detail while compressing older history. This reduces token growth and inference latency for real-world navigation.

  • Observation Encoding: Each video frame is encoded into Nx × C visual tokens, with Nx set to 256 patches and C denoting embedding dimension.The vision encoder produces framewise visual features before temporal processing.
  • Observation Encoding: Progressively growing visual-token sequences increase LLM inference time, making uncompressed video-based navigation impractical for real-world deployment.The passage reports typical LLM inference times of 1–2 seconds per step as token counts grow.
  • Online Visual Token Merging: Visual tokens are grouped into current, short-term, and long-term memories, each pooled at different resolutions according to temporal distance.Current tokens preserve immediate detail, while short- and long-term tokens summarize progressively older observations.
  • Online Visual Token Merging: The token-merging mechanism reuses prior tokens online, processes only newly relevant boundaries, and merges similar long-term tokens to avoid linear growth.Long-term tokens are merged when cosine similarity exceeds the threshold τ, empirically set to 0.95.
  • Efficiency: 0.2 seconds per inference is achieved on average with online merging, compared with substantially slower processing for existing video-based large language models.The efficiency advantage becomes more pronounced for longer video sequences.

C. Action Planning

Uni-NaVid projects merged visual tokens into the LLM’s language space, combines them with navigation instructions and a task indicator, and predicts four discrete actions at once.

  • Vision-Language Alignment: The cross-modality projector maps merged visual tokens into the LLM’s input representation space for vision-language alignment.The projector is a two-layer MLP optimized end-to-end.
  • Input and Output Format: The final input sequence concatenates long-, short-, and current-memory tokens, the ⟨NAV⟩ indicator, and language tokens.The task indicator supports task-specific learning and a consistent output format.
  • Input and Output Format: Uni-NaVid outputs four action tokens representing a short-horizon action sequence for navigation.The model predicts four successive actions, denoted by EA_T through EA_T+3.
  • Action Space: The discrete action set contains FORWARD, TURN-LEFT, TURN-RIGHT, and STOP, with forward movement of 25 cm and turns of 30°.Predicting the next four steps is reported to provide optimal performance and encourage longer-horizon forecasting.

V. DATA COLLECTION AND TRAINING

Uni-NaVid is trained by combining multi-task navigation data from synthetic environments with real-world video understanding data. The training data visualization summarizes 5.9M samples, video-frame counts, and common instruction words.

  • Training Data: Training uses multi-task navigation data from 861 synthetic scenes with a uniform input-output format, supplemented by real-world video question-answering data.The synthetic data provides navigation skills, while video QA supports interpretation of real-world images and open-vocabulary knowledge acquisition.
  • Training Data: Figure 3 visualizes the combined training data, video frame counts, and most common words in navigation instructions.The figure reports 5.9M training samples in its visualization.

A. Multi-Task Navigation Data.

The authors collect and evaluate unified navigation data across four tasks, using a common video-instruction-action format and diverse benchmarks. They also combine navigation data with video understanding data during training.

  • Multi-Task Navigation Data: 3.6 million samples are collected across vision-and-language navigation, object-goal navigation, embodied question answering, and human following.All four tasks are curated within a unified framework in the Habitat simulator environment.
  • Human Following: The human-following benchmark adds descriptions for eight avatars, distracting people, and photo-realistic large-scale scenes to test language-guided following in crowds.The benchmark addresses the absence of an existing dataset for language-described following in multi-person environments.
  • Unified Data Format: Unified samples contain an ego-centric RGB video, a natural-language instruction, and four corresponding future actions.The data is collected from synthetic HM3D and MP3D scenes using a common format.
  • Training Strategy: Training jointly uses 2.3M public video-question-answering samples and 3.6M multi-task navigation samples after projector pretraining.The two-stage procedure first trains the cross-modality projector, then fine-tunes the projector and LLM jointly.
  • Evaluation: Experiments assess individual-task performance, multi-task synergy, and the effectiveness of the model’s key design choices.Evaluation spans multiple navigation benchmarks, with scene splits checked to avoid overlap between training and validation scenes.

A. Deployment Details of Uni-Navid.

Uni-NaVid is evaluated across diverse navigation and video-question-answering settings, using online RGB video and instructions while producing action sequences for continuous environments. Results show strong performance across VLN, ObjectNav, EQA, human following, and VQA, with efficient real-world execution.

  • Deployment and evaluation: Each benchmark uses online RGB video and a textual instruction, with the model outputting the next four actions for execution in continuous environments.For EQA, navigation continues until STOP, after which the navigation token is removed and the question is answered from navigation history.
  • Deployment and evaluation: Approximately 0.2 seconds are required to generate the next four actions during real-world deployment.The robot asynchronously compresses and uploads observations while executing pending actions.
  • Vision-and-language navigation: +25.7% Success Rate on R2R is achieved over NaVid, while performance remains consistent across R2R and the more challenging RxR benchmark.The reported RxR result is +3.6 SR(%), and removing RxR samples still yields +23.9 SR(%) against NaVid.
  • Object goal navigation: +4.7% SR and +8.8% SPL are achieved over previous state-of-the-art methods on HM3D ObjectNav.The model also improves over zero-shot and fine-tuned methods on the open-vocabulary HM3D-OVON benchmark.
  • Human following: +21.0% SR and +4.4% FR are achieved over comparison methods for human following while maintaining low CR under every tested setup.The comparison methods were also evaluated with simulator ground-truth human bounding boxes.
  • Question answering: +17.9% BLEU-1, +5.7% ROUGE, +16.2% METEOR, and +13.1% CIDEr are reported on ScanQA, while open-ended VQA performance is comparable to state of the art.On embodied question answering, Uni-NaVid outperforms methods operating in discrete landmark-based environments and is comparable to GPT-4V with scene captions on OpenEQA.

C. Qualitative Results in Real-World

Uni-NaVid demonstrates zero-shot real-world navigation across individual and compositional tasks, while ablations show benefits from joint training, data scale, and visual memory.

  • Compositional tasks: Uni-NaVid sequentially completes complex instructions that combine multiple navigation tasks in unseen environments.The compositional instructions and environments are novel to the approach.
  • Real-world navigation: Uni-NaVid performs diverse navigation tasks in novel real-world environments under zero-shot conditions.The evaluated tasks include vision-and-language navigation, object-goal navigation, embodied question answering, and human following.
  • Real-world navigation: The agent handles long-horizon routes, searches for out-of-view objects, answers subsequent questions, and follows humans despite appearance variation.These capabilities are demonstrated across hallways, rooms, object-search scenarios, and human-following settings.
  • Ablation study: Multi-task training consistently improves performance across navigation tasks, with larger gains for VLN, ObjectNav, and EQA than Following.The authors attribute Following’s smaller gain to its lower reliance on historical context.
  • Ablation study: Performance improves with larger data volumes, but gains diminish from 3M to 6M samples, potentially because simulator data diversity is limited.Following also converges more slowly under heavy occlusion from obstacles or other humans.
  • Ablation study: Removing visual memory causes a −80.3% SR drop in VLN but only a −8% SR decline in Following.The contrast reflects VLN’s stronger dependence on aligning visual history with instructions, whereas Following relies more on recent frames.

VII. LIMITATIONS

The paper identifies scope boundaries in task coverage, robot embodiment, and action prediction, while proposing broader benchmarks and motion planning as future directions.

  • Current limitations: Uni-NaVid is trained and evaluated on four well-defined navigation tasks, leaving many practical navigation datasets outside its current scope.The authors suggest incorporating data from additional datasets to enhance navigation capabilities.
  • Current limitations: The method assumes a robot of standard size, limiting direct generalization to robots with different physical dimensions.The authors suggest incorporating prior knowledge of robot size for broader embodiments.
  • Current limitations: Uni-NaVid predicts short-horizon sequences of discrete low-level actions rather than continuous and smooth trajectories.The paper points to motion-planning and autonomous-driving techniques as possible extensions.
  • Future work: Future benchmarking could help position general-purpose navigation methods and drive progress in the navigation community.The authors identify general-purpose benchmarking as an emerging need.
  • Future work: Future architecture work targets very long-horizon tasks and advanced motion planning, including navigation across buildings.The paper also identifies robotic guide dogs, home-service robots, and mobile manipulation as application directions.

A. Vision-and-language Navigation (VLN)

The paper presents VLN as instruction-following through unseen environments and describes Uni-NaVid’s visual-history processing for open-vocabulary navigation.

  • VLN task: VLN requires following free-form instructions between landmarks and stopping at the described destination in unseen environments.The landmarks and motions are open-vocabulary, making instruction-history alignment central to the task.
  • VLN task: Benchmark instructions describe routes using landmarks, turns, rooms, and surrounding objects.The R2R and RxR examples require sequential movement through home interiors and completion at described destinations.
  • Human following: The human-following extension uses natural-language descriptions to identify and follow a specified person among multiple candidates.Its benchmark places multiple humans with diverse characteristics in each scene.
  • Visual token processing: Uni-NaVid processes current observations at 64 pooled visual tokens after dividing each image into an 8×8 grid.Average pooling over each grid produces the current-observation representation.
  • Visual token processing: Older observations are compressed into short-term 2 × 2 tokens and eventually into single long-term tokens after more than 65 steps.Long-term tokens are retained when the visual similarity condition falls below τ = 0.95.

B. Token Organization

Uni-NaVid organizes visual, instruction, and task-control tokens while training on heterogeneous navigation and auxiliary video data to support multi-task action prediction.

  • Token organization: Observation indicator and image-separator tokens distinguish visual content from instructions and separate information from adjacent frames.A navigation special token addresses ambiguity between answering an EQA question and outputting navigation actions.
  • Training strategy: Uni-NaVid uses a two-stage training strategy that first adapts the projector and language model with image/video QA data, then trains on navigation data.The auxiliary data is collected from LLaMA-VID and Pandm.
  • Navigation data: Each navigation sample contains a history video, an instruction, and four future actions.The history consists of accumulated frames up to the current timestep.
  • Navigation data: VLN data combines ground-truth trajectories with DAGGER-collected samples, producing nearly 1.69M additional navigation samples.The expert actions for DAGGER samples come from deterministic path planning to the next non-arrived landmark.
  • Navigation data: Shortest-path ObjectNav samples achieve only 30.1% SR because they do not teach exploration or recovery from mistakes.This motivates collecting navigation data that better represents object-search behavior.
  • Instruction augmentation: Instruction augmentation improves VLN by +2.31% in SR and ObjectNav by +3.7% in SR.The augmentation diversifies grammar and descriptions while preserving the original meaning.

XII. REAL-WORLD DEPLOYMENT

Uni-NaVid is deployed on a robotic dog with RGB video communication to a remote server, where it generates navigation commands for evaluation across multiple benchmarks.

  • System setup: The robotic dog uses a RealSense D455 camera, while LiDAR is reserved for local motion planning.The model uses 640 × 480 RGB frames with a 90° field of view; LiDAR is not used for perception.
  • System setup: Uni-NaVid does not rely on odometry algorithms or noiseless depth, supporting deployment in real-world environments.
  • System setup: The model runs on a remote NVIDIA A100 server that receives compressed images and instructions over the Internet and returns FORWARD, LEFT, RIGHT, or STOP commands.
  • Evaluation: Evaluation spans vision-and-language navigation, embodied question answering, and human following across public and newly constructed benchmarks.The reported benchmark suite includes VLN-CE R2R and RxR, MP3D-EQA and OpenEQA, and human-following datasets from HM3D, HSSD, and MP3D.
  • Evaluation: The experiments use task-specific metrics, including SR, OSR, SPL, FR, CR, ACC, EM, BLEU, ROUGE, METEOR, CIDEr, and Score.

B. Real-world Experiments

Real-world experiments compare Uni-NaVid with NaVid on simple and complex navigation instructions, while additional studies examine cross-dataset transfer, efficiency, and design choices.

  • Real-world VLN: 50 real-world VLN instructions comprise 25 simple and 25 complex cases for comparison with NaVid.Simple instructions target a single landmark, whereas complex instructions combine multiple movements and landmarks.
  • Real-world VLN: Uni-NaVid achieves high success rates on simple instructions and significant improvements over NaVid on complex instructions.
  • Evaluation measures: Human-following evaluation uses SR, FR, and CR, while OpenEQA results are reported with EM and broken down by ScanNet, HM3D, and ALL sources.
  • Cross-dataset evaluation: Without RxR training data, Uni-NaVid still outperforms existing approaches, although all metrics decrease relative to training with RxR data.The authors relate the discrepancy to differences in trajectory characteristics between R2R and RxR.
  • Ablation: Reducing current observation tokens causes a significant performance drop, with human following failing across all sequences in one setting.The authors associate the failure with insufficient information for tracking human motion; the study also varies τ.
  • Efficiency: Uni-NaVid reaches approximately 0.2 seconds of inference time and maintains consistent running times as navigation steps increase.The reported efficiency is attributed to the model architecture and online token merging, unlike competing approaches whose running times increase significantly.

F. Qualitative Experiments

Qualitative visualizations illustrate Uni-NaVid across vision-language navigation, object-goal navigation, embodied question answering, and human following.

  • Qualitative coverage: The qualitative results cover VLN-CE R2R and RxR, HM3D ObjectNav, MP3D-EQA, and HM3D human following.These examples span navigation, object search, question answering, and person-following tasks.
  • Vision-language navigation: VLN examples include multi-step instructions involving rooms, stairs, landmarks, and stopping locations.
  • Object-goal navigation: Object-goal examples request searches for toilets, beds, and chairs.
  • Human following: Human-following examples specify people using clothing and appearance attributes such as yellow shirts, gray shorts, or blue T-shirts.
Loading 2412.06224v2…