Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks

I. INTRODUCTION

• Online token merging compresses near history less and distant history more while reusing prior frames; foresight predicts future actions, enabling 5Hz inference.The mechanism aims to retain fine-grained spatial and structured temporal information while reducing token count and inference time.Source: §I · p. 2
• Uni-NaVid unifies vision-and-language navigation, object-goal navigation, embodied question answering, and human following, with a new language-guided human-following benchmark.Training data include 3.6M navigation samples and 2.3M internet video-question-answering and captioning samples intended to improve scene understanding and sim-to-real generalization.Source: §I · p. 2
• Uni-NaVid achieves state-of-the-art or comparable performance across four navigation benchmarks, while real-world experiments show robust, non-blocking navigation across diverse instructions and environments.The paper also reports that ablations validate the benefits of jointly learning multiple navigation tasks.Source: §I · p. 2

II. RELATED WORKS

• Earlier IL/RL generalist methods [85] [93] [66] struggled to generalize to novel real-world environments, while LLM-based approaches [114] [61] [35] [60] incur prompting costs.Other LLM-based approaches [120] [114] [61] [60] simplify navigation with discretized decisions and pre-defined graphs, limiting output flexibility and complicating deployment.Source: §II · p. 2, §I · p. 2
• The study uses continuous-environment VLN-CE, HM3D on Habitat for object-goal navigation, MP3D-EQA, and a self-built textual-description human-following benchmark in Habitat 3.0.VLN-CE is a continuous-environment variant of R2R and RxR; HM3D shares action settings with VLN-CE, and MP3D-EQA was selected for its maintained baselines.Source: §II · p. 3
• Zero-shot LLM navigation methods [120, 61, 60, 78] turn visual observations into text and select discrete landmarks, while end-to-end video-language models [106] face long-horizon efficiency challenges.Uni-NaVid instead uses online visual token merging to support long-horizon training efficiency and non-blocking real-world execution.Source: §II · p. 3

III. PROBLEM FORMULATION

• At time T, the agent receives an instruction and egocentric RGB video, then plans k=4 low-level actions from FORWARD, TURN-LEFT, TURN-RIGHT, and STOP for an unseen environment.The discrete actions correspond to a 30-degree rotation or a 25-centimeter forward movement and are compatible with continuous environments.Source: §III · pp. 3–4
• The pipeline encodes video frames into visual tokens, merges them spatially and temporally, aligns them with language tokens, and has an LLM infer the next four action tokens.The described implementation uses EVA-CLIP as the vision encoder and an MLP projector to align merged visual tokens with language tokens.Source: §III · p. 4

A. Observation Encoding.

• Each video frame is encoded into 256 visual tokens using a vision encoder, with each token having embedding dimension C.The implementation uses EVA-CLIP.Source: A. Observation Encoding · p. 4
• As video grows, the T × Nx visual-token sequence lengthens LLM inference, typically taking 1–2 seconds per inference and hindering real-world deployment.Source: A. Observation Encoding · p. 4

B. Online Visual Token Merging

• Token merging exploits the greater importance of recent observations and redundancy across neighboring frames and pixels by grouping tokens into current, short-term, and long-term memories.Source: B. Online Visual Token Merging · p. 4
• With B set to 64 and pooling factors of 2, 8, and 16, current, short-term, and long-term memories contain 64, 4, and 1 tokens, respectively.Current tokens preserve comprehensive immediate visual information; short- and long-term tokens retain navigation history. These hyperparameters were chosen empirically to balance token count and visual representation.Source: B. Online Visual Token Merging · p. 4
• When a new frame arrives, the online process pools only the latest current tokens and the oldest short-term tokens, reusing previous tokens to update short- and long-term memory.Source: B. Online Visual Token Merging · p. 4
• Long-term tokens with cosine similarity above threshold τ are merged using their prior frame counts; at τ = 0.95, distinct tokens are added, keeping history compact at length M ≪ T − B − 1.Only tokens at group boundaries require parallelizable grid pooling, supporting computationally efficient online deployment.Source: B. Online Visual Token Merging · pp. 4–5
• 0.2 seconds per inference is the reported average for Uni-NaVid’s online merging strategy, which is faster than existing video-based large language models.The improvement becomes increasingly notable for longer video sequences.Source: B. Online Visual Token Merging · p. 5

C. Action Planning

• A two-layer cross-modality MLP projector aligns merged visual tokens with the LLM input space, while Vicuna-7B’s tokenizer and embedding layer encode instructions [54] [51].The projector is optimized end-to-end; merged visual tokens comprise long-term, short-term, and current observations [82].Source: C. Action Planning · p. 5
• The model concatenates visual tokens, a <NAV> task indicator, and instruction tokens, then feeds them to the LLM to predict four action tokens.The input format combines long-term, short-term, and current tokens with the instruction; outputs are <Action 0> through <Action 3>.Source: C. Action Planning · p. 5
• The discrete actions are FORWARD, TURN-LEFT, TURN-RIGHT, and STOP; forward moves 25 cm and turns rotate 30° under the standard configuration [76] [106].Predicting the next four steps empirically yields optimal performance, encourages long-horizon forecasts while retaining sufficient observations, and enables non-blocking asynchronous navigation.Source: C. Action Planning · p. 5

• Extensive real-world navigation data can be prohibitively expensive to collect, motivating the use of these two training strategies.Source: §V · pp. 5–6
• Multi-task navigation data from 861 synthetic scenes is collected in a uniform input and output format to help Uni-NaVid acquire general navigation skills.Source: §V · p. 6
• Co-tuning with real-world video-based question-answering data enhances interpretation of real-world images and supports open-vocabulary knowledge acquisition.Source: §V · p. 6

A. Multi-Task Navigation Data.

• The unified Habitat dataset contains 3.6 million samples across four navigation tasks.The paper describes it as the largest multi-task navigation dataset to date within Habitat.Source: A. Multi-Task Navigation Data · p. 6
• The four task datasets comprise 2.4M VLN samples, 483k HM3D ObjectNav samples, 240k EQA video-action plus 10k video-answering samples, and 544k human-following samples.The VLN collection includes VLN-CE R2R [44] and RxR [47]; the EQA samples come from MP3D-EQA [21].Source: A. Multi-Task Navigation Data · p. 6
• The new human-following benchmark adds descriptions for eight avatars, distractor humans, and photo-realistic Habitat Matterport scenes to support crowded, language-described following.The robot and target start nearby while distracting avatars move randomly; Fig. 4 illustrates the benchmark setup.Source: A. Multi-Task Navigation Data · p. 6, Figure 4 · p. 6
• Navigation samples share an egocentric RGB video, natural-language instruction, and four future actions, with robot height and radius varied to reduce embodiment overfitting.Data were collected in synthetic HM3D and MP3D scenes.Source: A. Multi-Task Navigation Data · pp. 6–7

B. Training Strategy of Uni-NaVid

• Training uses a two-stage process: projector alignment first, then projector and LLM fine-tuning on 2.3M video-question-answering and 3.6M navigation samples.Online token merging is applied to both data types; VQA samples omit the navigation task indicator ⟨NAV⟩.Source: B. Training Strategy of Uni-NaVid · p. 7
• Training took approximately 35 hours on 40 NVIDIA H800 GPUs, sampled video at 1 FPS, and optimized trainable parameters for one epoch.The vision encoder EVA-CLIP [82] and Vicuna-7B were initialized with pretrained weights.Source: B. Training Strategy of Uni-NaVid · p. 7

VI. EXPERIMENT

• Uni-NaVid is evaluated on VLN-CE R2R and RxR, HM3D ObjectNav and zero-shot HM3D-OVON, MP3D-EQA and OpenEQA, and the proposed language-described human-following benchmark.The experiments assess individual-task performance, multi-task synergy, and the effectiveness of the method’s design.Source: §VI · p. 7
• Video-understanding evaluation uses ScanQA, MSVD, MSRVTT, and ActivityNet, while navigation metrics include SR, OS, SPL, TL, FR, CR, and NE.The evaluation verifies that training and validation scenes do not overlap and uses each benchmark’s default success criteria.Source: §VI · p. 7

A. Deployment Details of Uni-Navid.

• Across evaluated tasks, Uni-NaVid takes online RGB video and a textual instruction as input, predicts the next four actions, and stops when the first predicted action is STOP.For EQA, it navigates until a stop command, then removes the <NAV> token and answers using navigation history; real-world action generation takes approximately 0.2 seconds, with observation uploads asynchronous to action execution.Source: A. Deployment Details of Uni-Navid · pp. 7–8

B. Individual Task Results

• Uni-NaVid achieves SOTA-level performance on VLN-CE R2R and RxR using RGB video, with a +25.7% Success Rate gain over NaVid on R2R and +3.6 SR(%) on RxR.RxR has more diverse, longer trajectories with detailed landmark descriptions; without RxR training samples, Uni-NaVid still achieves a +23.9 SR(%) advantage over NaVid.Source: B. Individual Task Results · p. 8, Table II · p. 8, Table III · p. 8
• On HM3D ObjectNav, Uni-NaVid achieves the best performance, improving SR by +4.7% and SPL by +8.8% over previous state-of-the-art methods.Methods without odometry must rely on implicit memory to retain trajectory history; the authors suggest reinforcement learning could further improve Uni-NaVid’s ObjectNav performance.Source: B. Individual Task Results · p. 8, Table IV · p. 8
• On HM3D-OVON, Uni-NaVid improves over zero-shot VLFM and outperforms fine-tuned DAgRL+OD on both VAL SEEN and VAL UNSEEN splits.The paper presents this open-vocabulary object-goal evaluation as evidence of generalizability.Source: Table V · p. 8, B. Individual Task Results · p. 9
• On MP3D-EQA, Uni-NaVid outperforms approaches operating in discrete landmark-based environments; ground-truth navigation trajectories further improve its performance.On the more challenging OpenEQA benchmark, it achieves performance comparable to GPT-4V with scene captions.Source: B. Individual Task Results · p. 9, Table VI · p. 9
• On human following, Uni-NaVid exceeds comparison methods by +21.0% SR and +4.4% FR while maintaining low CR, even against methods using ground-truth bounding boxes.The paper also reports consistent SOTA performance in additional HSSD and MP3D environments.Source: B. Individual Task Results · p. 9, Table VII · p. 9
• On ScanQA, Uni-NaVid achieves the best performance on four metrics: BLUE-1 (+17.9%), ROUGE (+5.7%), METEOR (+16.2%), and CIDEr (+13.1%).On open-ended video QA benchmarks, it performs comparably to state-of-the-art methods sharing the Vicuna-7B backbone; the paper notes exact-match scoring is poorly suited to its flexible responses.Source: B. Individual Task Results · p. 9, Table VIII · p. 9, Table IX · p. 9

C. Qualitative Results in Real-World

• Uni-NaVid successfully executes complex compositional navigation tasks sequentially when both instructions and environments are novel.The supplementary-video results show the agent completing multiple navigation tasks in sequence.Source: Figure 6 · p. 10
• In novel real-world settings, Uni-NaVid handles four navigation tasks, including long-horizon instruction following, out-of-view object search, question answering, and following people unlike training avatars.Experiments were conducted zero-shot across diverse environments.Source: C. Qualitative Results in Real-World · p. 10, Figure 5 · p. 10
• Action probabilities show the agent distinguishes locations and sequences turning right or left before moving forward.The action predictions are presented for VLN tasks involving challenging open-vocabulary objects and break-in navigation.Source: C. Qualitative Results in Real-World · p. 10, Figure 7 · p. 11

D. Ablation Study

• Multi-task training improves all navigation tasks, with larger gains in VLN, ObjectNav, and EQA than in Following.The authors attribute Following’s smaller gains to its lower reliance on historical context.Source: D. Ablation Study · pp. 10–11, Figure 8 · p. 11
• Performance improves across all tasks with more training data, but gains diminish from 3M to 6M samples.The authors suggest limited simulator-data diversity may explain diminishing gains; Following also converges more slowly because obstacles and other people heavily occlude the target, motivating more high-quality Following data.Source: D. Ablation Study · p. 11, Figure 8 · p. 11
• Removing <NAV> and VQA data reduces performance across tasks; without <NAV>, EQA confuses answering with acting, while without VQA, question answering nearly fails.The authors attribute the latter decline to catastrophic forgetting of open-world knowledge during navigation-only training; related findings are reported in [14].Source: D. Ablation Study · p. 11, Table X · p. 11
• Removing visual memory causes an −80.3% SR drop in VLN and an −8% SR drop in Following.Both short- and long-term visual tokens improve performance; memory supports alignment of visual history with instructions in VLN, while Following mainly relies on recent frames to track the target.Source: D. Ablation Study · p. 11

VII. LIMITATIONS

• The model is trained and evaluated on four well-defined tasks despite broader practical navigation datasets [86], whose data could further enhance its capabilities.The paper notes that collecting data from these datasets could improve navigation capability.Source: §VII · p. 11
• Uni-NaVid assumes a standard-size robot; incorporating prior knowledge of robot size is proposed to extend it to robots of general sizes [24] [29].The standard-size assumption is specified in Section V-A.Source: §VII · p. 11
• Its predictions are short-horizon sequences of discrete actions; continuous, smooth trajectories using motion-planning or autonomous-driving techniques are proposed as a remedy [79] [81] [18] [52].The limitation concerns the model's current trajectory output.Source: §VII · p. 11

VIII. DISCUSSION AND CONCLUSION

• Uni-NaVid uses online visual token merging to run at 5 Hz and achieves SOTA-level performance from monocular video, with real-world generalizability across navigation tasks.It is trained on 3.6 million data points from four tasks: vision-and-language navigation, object goal navigation, embodied question answering, and human following.Source: §VIII · pp. 11–12

IX. TASK DEFINITION

• The task-definition section details four embodied navigation tasks, also enumerated as vision-and-language navigation, object goal navigation, embodied question answering, and human following.The supplied task-definition passage identifies the section's scope but does not provide the task descriptions themselves.Source: §VIII · p. 12, §IX · p. 18

VIII. DISCUSSION AND CONCLUSION

• The authors call for a general-purpose navigation benchmark to help researchers position their work and advance the community.They identify benchmarking as a future direction amid continued development of embodied navigation.Source: §VIII · p. 12
• Future architecture work aims to support very long-horizon tasks, such as navigating across buildings, and incorporate advanced motion planning [81] [18].The paper frames this as a way to further improve practicality.Source: §VIII · p. 12
• The authors propose applying Uni-NaVid to robotic guide dogs, home service robots, and other embodied AI tasks such as mobile manipulation [92] [55].These are applications and extensions the authors hope to pursue.Source: §VIII · p. 12

A. Vision-and-language Navigation (VLN)

• VLN requires following free-form, open-vocabulary landmark and motion instructions in continuous unseen environments, with R2R and RxR requiring ordered landmark navigation.In VLN-CE R2R and RxR, success requires stopping within 3 meters of the destination, with a maximum of 500 steps. [44] [47]Source: A. Vision-and-language Navigation (VLN) · p. 18

B. Object Goal Navigation (ObjectNav)

• ObjectNav starts an agent at a random position and orientation in an unseen environment and asks it to find an instance of a specified object category. [76]In HM3D, success requires stopping within 1 meter of the target object under 500 steps.Source: B. Object Goal Navigation (ObjectNav) · p. 18

C. Embodied Question Answering (EQA)

• Embodied Question Answering requires navigating an unseen 3D environment to identify a question-related target and answer in natural language. [21]In MP3D-EQA, an episode succeeds if the answer matches the ground truth within 500 steps.Source: C. Embodied Question Answering (EQA) · pp. 18–19

D. Human Following

• The human-following task uses natural-language descriptions to locate and follow a specific person in potentially crowded, dynamic environments.Its Habitat 3.0 benchmark places multiple people in each scene; success requires stopping within 2 meters of the correct person and facing them. [68]Source: D. Human Following · p. 19

X. IMPLEMENTATION DETAILS

• The implementation details identify online visual token merging as Uni-NaVid’s key technique for efficiently encoding long-horizon egocentric video.Source: §X · p. 19, A. Online Visual Token Merging · p. 19

A. Online Visual Token Merging

• Grid pooling creates 64 tokens for the current frame, then reduces older observations to 2 × 2 tokens for a short-term memory buffer.The current frame is divided into an 8×8 grid and average-pooled; older current-observation tokens are pooled before entering the buffer.Source: A. Online Visual Token Merging · p. 19
• After 65 steps, the oldest short-term frame is pooled to one token and added to long-term memory if the list is empty or cosine similarity is below τ = 0.95.The threshold is reported as empirically chosen to balance efficiency and effectiveness [80].Source: A. Online Visual Token Merging · p. 19

B. Token Organization

• The <NAV> token signals navigation and prompts action-token output, while visual indicators and frame separators distinguish visual content across frames.The navigation token addresses ambiguity in embodied question answering, where the model might otherwise answer directly rather than output actions.Source: B. Token Organization · p. 19

C. Traing Strategy

• Training follows two stages: projector pretraining on Image QA followed by projector-and-LLM fine-tuning on Video QA, then training both on navigation data.Image and Video QA data come from LLama-Vid [51] and Pandm [19]; default parameters follow NaVid.Source: C. Traing Strategy · p. 19

XI. DATA PREPARATION DETAILS

• Because large-scale high-quality real-world annotations are difficult to collect, the authors gather navigation data in synthetic environments.The collected samples cover several navigation tasks and combine navigation-history video, instructions, and future actions; the action space uses four actions.Source: §XI · p. 19, A. Data Collection · p. 19

A. Data Collection

• VLN-CE simulator samples from R2R [44] and RxR [46] provide 0.64 M ground-truth-trajectory video-action examples, supplemented by nearly 1.69 M DAGGER [74] samples.A 70k VLN subset also pairs short trajectories with low-level action instructions; the authors report that performance drops when such instructions expand significantly.Source: A. Data Collection · p. 20
• ObjectNav data include 483 k successful HM3D [70] trajectories, while embodied question answering contributes 240 k video-action samples and 10 k video-answering samples from MP3D-EQA [21].Object instructions use a “Search for a/an [Object]” template; using shortest-path object trajectories yielded only 30.1% SR because the model did not learn exploration or recovery.Source: A. Data Collection · p. 20
• Human-following data comprise 544 k samples from a Habitat 3.0 environment, with 2–6 people per episode and a robot guided to follow a designated target while avoiding obstacles.The environment is based on HM3D [70], and a low-level planner guides each person through randomly generated waypoints.Source: A. Data Collection · p. 20

B. Instruction Augmentation

• Instruction augmentation could increase performance by +2.31% in SR for VLN and +3.7% in SR for ObjectNav; cross-dataset instruction augmentation may be promising for further investigation.Source: B. Instruction Augmentation · p. 20

A. Robot Setup.

• The robot setup uses a Unitree GO2 with a head-mounted RealSense D455 capturing 640 × 480 RGB frames at 90◦ HFOV; Wi-Fi supports communication and LiDAR-L1 is used only for local motion planning.Source: A. Robot Setup · pp. 20–21
• Uni-NaVid does not rely on odometry algorithms [108, 109] or noiseless depth [76] [48], easing deployment in real-world environments.Source: A. Robot Setup · p. 21

B. Real-world System Architecture

• The model runs on a remote NVIDIA A100 server, receiving compressed images over the Internet and returning commands for local execution; inference takes about 0.2 seconds and communication about 0.3 seconds.Online token merging supports efficient image processing; the robot uses Unitree Dog’s off-the-shelf local motion planner.Source: B. Real-world System Architecture · p. 21
• For non-blocking navigation, the robot executes four commands sequentially, captures an image after each action, and prioritizes the most recent command when multiple commands arrive for one step.Source: B. Real-world System Architecture · p. 21

1) Benchmark.:

• Evaluation covers Val-Unseen VLN-CE R2R and RxR, MP3D-EQA validation and OpenEQA, plus human-following benchmarks based on HM3D, HSSD, and MP3D.The HM3D human-following benchmark is self-built and also supplies training data; HSSD and MP3D benchmarks test following across datasets and scenes.Source: 1) Benchmark · p. 21

2) Metrics.:

• Vision-and-language navigation and object goal navigation use SR, OSR, and SPL; SR uses 3 m and 1 m goal thresholds for VLN and ObjectNav, respectively, within up to 500 steps.OSR credits proximity to the goal even without stopping at it; SPL combines success with path-length efficiency.Source: 2) Metrics · p. 21
• Human following uses SR for reaching the endpoint with the person, FR for the share of steps successfully following them, and CR for episodes with a collision.Source: 2) Metrics · p. 21
• Embodied QA uses ACC, while ScnaQA [7] also uses EM, BLEU, ROUGE, METEOR, and CIDEr; MSVD-QA, MSRVTT-QA, and ActivityNet-QA use ACC and GPT-assisted scores from 1 to 5.ACC measures answer correctness; the other ScnaQA metrics assess exact matching, text overlap, or similarity to reference answers.Source: 2) Metrics · p. 21

B. Real-world Experiments

• The real-world comparison with NaVid tested instructions involving landmarks and motions at two difficulty levels.Simple instructions required navigating to a single robot landmark and stopping; complex instructions required following a series of simple instructions.Source: B. Real-world Experiments · p. 22
• Navid and the authors’ method achieve high SR on simple instructions, while the authors’ method shows significant improvements on complex instructions, demonstrating its superiority over existing methods.Source: B. Real-world Experiments · p. 22

C. More Benchmark Experiments

• Without RxR training data, Uni-NaVid still outperformed existing approaches, but all metrics decreased; the paper links the gap to RxR’s more varied trajectory lengths.R2R trajectories are approximately 10 meters, while RxR trajectories range from approximately 2 to 20 meters; the text says this underscores the importance of training on diverse trajectories.Source: C. More Benchmark Experiments · p. 22
• On OpenEQA, Uni-NaVid achieved competitive performance to GPT-4V without additional frame captions or Socratic-style prompts.OpenEQA requires answering questions by analyzing egocentric video.Source: C. More Benchmark Experiments · p. 22, Table XIII · p. 22
• Uni-NaVid outperformed baselines on SR and FR across HSSD and MP3D, with notable gains on HSSD; IBVS’s lower collision rate came with worse SR and FR.The text attributes the HSSD gains in part to fewer reconstruction artifacts that can leave robots stuck, and describes IBVS as maintaining greater distance from targets.Source: C. More Benchmark Experiments · pp. 22–23, Table XIV · p. 23

D. More Ablation study

• Reducing current observation tokens caused a significant performance drop, and the following task failed across all sequences, potentially from insufficient information to track human motion.The ablation results are reported in Table XV.Source: D. More Ablation study · p. 23, Table XV · p. 23
• Increasing τ improved performance, which the paper attributes to retaining more long-term observation tokens for understanding navigation history.The results are reported in Table XV.Source: D. More Ablation study · p. 23, Table XV · p. 23

E. Time Analyze

• Approximately 0.2 seconds per inference, Uni-NaVid runs consistently while existing methods’ running times increase significantly.Fig. 12 compares average running time and token count across time steps.Source: Figure 12 · p. 23, E. Time Analyze · p. 23
• Its efficiency is attributed to an efficient architecture and online token merging that gradually increases token count.The paper contrasts its architecture with the time-consuming QFormer operations used by LLaMA-VID [51] and NaVid.Source: E. Time Analyze · p. 23

F. Qualitative Experiments

• Additional visual results are available in an attached video.Source: F. Qualitative Experiments · p. 23
• Qualitative results cover VLN-CE R2R and RxR, HM3D ObjectNav, and MP3D-EQA.The results are shown in Figs. 13, 14, and 15, respectively.Source: F. Qualitative Experiments · p. 23, Figure 13 · p. 24, Figure 14 · p. 25
• Object-search examples ask the agent to find a toilet, a bed, and a chair.Source: F. Qualitative Experiments · p. 25

Human Following

• HM3D human-following results include instructions identifying people by clothing, such as a woman in gray shorts or a man in a blue shirt and gray trousers.Fig. 16 presents visual results for HM3D human following.Source: Figure 16 · p. 27, F. Qualitative Experiments · p. 27