PaperlayerView exact revision on arXiv

Source-linked AI summary

LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation

Shaoan Wang, Aocheng Luo, Fei Huang, Jingyi Xu, Xiaoyang Wang, Yueyu Wang, Qianli Ma, Fan Yang, Ran Mei, Jia Wei, Jiangpeng Hu, Xuhao Liu, Hongming Chen, Yuanbin Shao, Yiyang Lin, Ziliang Li, Liang Pan, Xinhang Liu, Yuntao Ma, Tingxiang Fan

arXiv:2608.30935v1cs.ROcs.AI

TL;DR

Embodied navigation lacks a unified, transferable way to connect VLM spatial reasoning with precise actions across tasks and embodiments. LightNav-0 addresses this with dual-channel pointing, residual-quantized trajectories, and temporally compressed visual history in a compact pretrained VLM. LightNav-ER achieves the highest complete-set average across eight embodied-reasoning benchmarks, while LightNav-0 reports the strongest monocular success rates across ten public simulation settings and transfers to four real-world robot embodiments.

  • Problem

    Existing navigation systems rely on task- or embodiment-specific components, limiting unified transfer across tasks, environments, and robot embodiments.

  • Method

    LightNav-0 uses a compact pretrained VLM with dual-channel pointing, residual vector-quantized action tokens, temporally compressed visual history, and unified navigation supervision.

  • Results

    LightNav-ER attains the highest complete-set average across 8 embodied-reasoning benchmarks, while LightNav-0 achieves the strongest monocular success rates across all 10 public navigation simulation settings.

  • Takeaways & Limitations

    Compact VLMs can provide a unified and transferable substrate for generalist embodied navigation across tasks and robot embodiments.

Abstract

from arXiv · show

Embodied navigation requires agents to translate heterogeneous goals and visual observations into actions across tasks, environments, and robot embodiments. Modern vision-language models (VLMs) already encode spatial priors for visual grounding, spatial reasoning, and pointing, but these capabilities are rarely elicited directly for robot control. Existing navigation systems instead rely on task- or embodiment-specific components, fragmenting perception, reasoning, and action while offering limited generalization. Here we present LightNav-0, a compact generalist embodied navigation model that elicits the spatial intelligence of a pretrained VLM and aligns it with navigation, without task-specific prediction heads. LightNav-0 represents diverse navigation tasks through a unified token interface: dual-channel pointing expresses task-, scene-, and embodiment-agnostic spatial intent, while a residual vector-quantized action tokenizer maps this intent to precise, embodiment-specific trajectories. Together with temporally aware visual history compression, ER mid-training, supervised fine-tuning, and reinforcement learning, this formulation supports instruction following, open-vocabulary object navigation, and visual tracking within a single model. The navigation training corpus spans 2K+ scenes and 4K+ hours of embodied navigation data. LightNav-ER, the embodied-reasoning checkpoint used to initialize LightNav-0, attains the highest complete-set average across 8 embodied-reasoning benchmarks, while LightNav-0 achieves state-of-the-art monocular success rates across all 10 public navigation simulation settings. Real-world evaluations further demonstrate zero-shot generalization across robot embodiments, diverse scenes, and static and dynamic targets. These results establish compact VLMs as a unified and transferable backbone for generalist embodied navigation.

I. INTRODUCTION

LightNav-0 addresses the fragmentation and limited transfer of embodied navigation by using a compact pretrained VLM with a unified spatial-action interface across tasks and embodiments. Its dual-channel pointing and tokenized trajectories support cross-task and cross-embodiment navigation.

  • Motivation: Existing embodied navigation systems commonly use task-specific modules or policies, limiting open-vocabulary transfer and reuse across platforms.
  • Approach: LightNav-0 retains a compact pretrained VLM as a shared reasoning backbone without task-specific prediction heads for tasks or embodiments.
  • Approach: Dual-channel pointing uses an affordance point for feasible motion and an object point for the target, providing a shared spatial interface across navigation tasks and robot embodiments.
  • Evaluation: LightNav-0 is evaluated across 10 public simulation settings and real-world embodiments, supporting a unified, transferable basis for cross-task and cross-embodiment navigation.
  • Approach: The system combines visual reasoning traces, tokenized actions, and reinforcement learning while keeping the action interface compatible with precise navigation trajectories.

III. MODEL ARCHITECTURE

LightNav-0 converts multimodal navigation inputs into a compact sequence of spatial and action tokens, while compressing visual history according to temporal recency. Its RVQ tokenizer maps trajectories to progressively refined, executable waypoint sequences.

  • Architecture Overview: LightNav-0 conditions token generation on a natural-language instruction and egocentric RGB history, producing dual-channel pointing followed by a 10-waypoint SE(2) trajectory.
  • Architecture Overview: The model preserves the pretrained VLM architecture, augmenting only its vocabulary with affordance, object, and RVQ action tokens decoded by the original language-model head.
  • Architecture Overview: Affordance pointing identifies feasible local motion or free space, while object pointing localizes a target object or goal location.
  • Temporally Aware Visual History Compression: Recent observations receive denser sampling and finer resolution, whereas older observations are sampled less frequently and pooled more aggressively.
  • Temporally Aware Visual History Compression: The history compressor bounds context length within configurable pixel budgets while retaining temporal ordering and current-frame detail.
  • Action Tokenization: The hierarchical RVQ tokenizer quantizes 10-step SE(2) trajectories with one coarse and two residual 256-entry codebooks, progressively refining geometric precision.

C. Dual-Channel Pointing as Latent Spatial Reasoning

LightNav-0 uses dual-channel image-grid pointing to express affordance and object intent, then residual vector quantization converts that spatial trace into precise trajectories.

  • Dual-channel pointing: Each projected point is encoded as one flattened token on a shared H_g × W_g image grid.The grid index is computed from the point’s row and column coordinates.
  • Dual-channel pointing: The affordance channel identifies feasible local motion or landing locations, while the object channel localizes the target object or goal location.The channels use distinct token families over the same lattice.
  • Unified output: The navigation prefix contains two pointing tokens followed immediately by three RVQ action tokens.Causal attention turns the compact prefix into a latent spatial trace conditioning trajectory generation.
  • Residual action tokenizer: Each action chunk represents 10 future SE(2) waypoints using three 256-entry residual codebooks.The first codebook captures the coarse trajectory and later levels quantize successive residuals.
  • Residual action tokenizer: Jacobian-weighted trajectory distance balances translation and heading errors during residual codebook fitting.The weighting operates in integrated trajectory space, with λ = 0.3 used throughout experiments.
  • Residual action tokenizer: Any non-empty RVQ prefix decodes to an executable trajectory, allowing fewer tokens to trade precision for lower autoregressive cost.Full three-level decoding supports up to 256^3 code combinations.

E. Unified Autoregressive Objective

The model trains heterogeneous navigation and auxiliary reasoning examples through one causal language-model objective and shared token interface. Its data mixtures preserve spatial, temporal, and visual capabilities while supporting compact inference.

  • Shared objective: Navigation and auxiliary VQA examples use the same causal language-model training objective.This places navigation actions and reasoning outputs in one autoregressive prediction framework.
  • Shared objective: Navigation outputs serialize one affordance token, one object token, and three RVQ action tokens.Pointing and spatial VQA instead emit their corresponding indexed tokens or language responses.
  • Shared objective: The shared token space removes separate task losses and lets all samples use the same packed autoregressive training loop.The formulation exposes every task through one prediction head.
  • Efficient implementation: Dynamic packing fits approximately 8.6 samples into each 8,192-token training sequence.Fused vision rotary-position operations and aligned LM-head dimensions further reduce kernel and memory overhead.
  • Efficient implementation: Single-process vLLM inference achieves approximately 4 ms per generated token on an NVIDIA GeForce RTX 4090.Each step uses the pointing prefix followed by at most three RVQ action tokens.
  • Training mixtures: The navigation corpus spans 2K+ scenes and 4K+ hours, while ER mid-training draws from 36 sources before SFT combines navigation mixtures.The ER curriculum includes pointing, VQA, video reasoning, and general visual reasoning.
  • Training mixtures: Single-image supervision covers qualitative relations, metric depth and distance, object relations, camera motion, and physical measurement.These tasks provide indexed spatial supervision for the shared model.
  • Training mixtures: Video supervision spans robot interactions and clips up to 64 frames, with temporal ordering, trajectory-aware relations, planning, and affordance prediction.This complements the model’s embodied and temporal reasoning requirements.

2) Video Embodied QA:

The training mixture broadens embodied reasoning beyond static image grounding by combining cross-view correspondence, general visual reasoning, and task-diverse navigation data. These sources support instruction following, object navigation, and tracking within one policy.

  • Pointing and grounding: Pointing supervision is the largest specialized ER component, with 13 sources contributing 35.14% of sampling probability.It covers object localization, free-space selection, affordances, and trajectory traces.
  • Multi-image and ego–exo correspondence: Multi-image and temporal cross-view samples require associating objects across viewpoints and reconciling egocentric with exocentric observations.They also support camera-motion estimation and preservation of spatial relations across frame changes.
  • General visual reasoning: General ER sources account for 20.00% of sampling mass and cover VQA, instruction understanding, mathematical reasoning, and compositional spatial relations.This component preserves linguistic and visual breadth alongside navigation specialization.
  • Unified navigation tasks: Navigation supervision is organized by objective across instruction following, object goal navigation, and embodied visual tracking rather than by robot platform.Camera randomization samples field of view, height, and pitch to broaden training viewpoints.
  • Instruction following: Instruction-following data combine expert, synthesized, self-distilled, and ScaleVLN routes to expose fine-grained language alignment and geometric variation.The mixture includes randomized-camera trajectories from R2R and RxR.
  • Object navigation: Object-goal data combine semantic and expert demonstrations with exploration trajectories and in-loop DAgger states induced by the policy’s own actions.The targets include a feasible landing region and the referred object.
  • Visual tracking: Tracking data use randomized-camera person-following trajectories requiring persistent target identity and continuous relative-position control.This adds temporal behavior to the shared policy mixture.

4) Quality Control and Sampling:

INSIGHT-Bench combines automatically constructed 3D target inventories, route-based episode generation, diagnostic axes, and semantic quality controls. Sampling filters and validation checks constrain supervision while retaining diverse scenes and instructions.

  • Quality control and sampling: Stop supervision is retained only on final frames with visible targets, while stop samples are capped at 2% of the mixture.Episodes without any target observation are removed for instruction following and object navigation.
  • Quality control and sampling: The ER checkpoint is evaluated on 8 benchmarks spanning pointing, referring, interaction-site prediction, affordance grounding, geometric perception, and embodied QA.The benchmark set includes Point-Bench, RefSpatial, RoboSpatial, Where2Place, CV-Bench, ERQA, and EmbSpatial.
  • Quality control and sampling: Navigation evaluation uses 10 simulation settings across instruction following, closed- and open-vocabulary ObjectNav, and embodied visual tracking without benchmark-specific fine-tuning.The settings include R2R, RxR, MP3D, HM3D, HM3D-OVON, and EVT-Bench.
  • Quality control and sampling: INSIGHT-Bench contains 1,683 training scenes with 53,090 episodes and 210 evaluation scenes with 1,097 episodes.Its evaluation split is organized by scene function/layout rows and spatial-mechanism columns.
  • Pre-annotation: Multi-view RGB and metric-depth observations are converted into spatially consistent 3D target instances through open-set pointing, 3D lifting, and cross-view merging.Unstable single-view detections are removed before episode generation.
  • Data generation: Episodes sample a target and start pose, plan a navigation-mesh route, anchor the stop, and render monocular egocentric clips with action labels.The target may be absent from the policy’s initial forward-facing observation.
  • Instruction labeling: Instructions are drafted from geometry or video, rewritten under semantic checks, and rejected rewrites fall back to the source draft.Checks preserve the target noun, sector-implied direction, and avoidance of the opposite direction.
  • Quality control and sampling: A visual arrival check examines the final frames and flags episodes whose instructed target is absent from every checked frame.The default check uses the 1st, 3rd, 6th, and 10th frames counted backward from the end.

3) Diagnostic Taxonomy:

The paper introduces a diagnostic taxonomy that separates scene-level and instruction-level factors, enabling aggregate navigation success to be analyzed by functional layout and language mechanism.

  • 3) Diagnostic Taxonomy:: INSIGHT-Bench assigns episodes scene-level labels distinguishing apartments, houses, commercial scenes, institutions, and outdoor environments.These labels capture differences in room topology, openness, repeated instances, corridors, workspaces, and landmark sparsity.
  • 3) Diagnostic Taxonomy:: Instruction labels derive from geometric target-construction proofs, covering base, direction, relation, extremum, and ordinal references.The taxonomy therefore reflects the spatial mechanism required to resolve each target rather than surface wording alone.
  • 3) Diagnostic Taxonomy:: The taxonomy turns aggregate success into a diagnostic result: rows expose scene sensitivity, columns isolate language mechanisms, and cells reveal interactions.Lower relation and ordinal counts reflect stricter uniqueness and ordering gates rather than silent truncation.
  • 3) Diagnostic Taxonomy:: Training and evaluation scenes are strictly disjoint, with 210 evaluation scenes separated from 1,683 training scenes across sources.This partitioning includes fixed Habitat-GS and HM3D scene assignments and separate InteriorGS training and evaluation scenes.
  • 3) Diagnostic Taxonomy:: ER mid-training combines spatial reasoning, VQA, pointing, video QA, multi-image reasoning, situated 3D QA, affordance understanding, and scene description without navigation-specific heads.The resulting LightNav-ER checkpoint initializes later navigation alignment, while supervised training rehearses ER data alongside navigation data.

B. Supervised Fine-tuning

Supervised fine-tuning aligns the embodied-reasoning checkpoint with a shared navigation token space, while online reinforcement learning optimizes complete task rollouts and task-specific terminal objectives.

  • B. Supervised Fine-tuning: Navigation examples use one affordance-point token, one object-point token, and three RVQ tokens for short-horizon trajectories.The same serialized target structure spans instruction following, object navigation, and embodied visual tracking alongside retained ER and VQA examples.
  • B. Supervised Fine-tuning: Online reinforcement learning optimizes complete rollouts against task-level objectives that refine tracking, route-consistent reaching, efficient search, and termination.This addresses the limitation that token-level imitation does not directly optimize closed-loop task success.
  • B. Supervised Fine-tuning: GRPO samples G independent trajectories per episode seed and standardizes their rewards within each group to form advantages.Within-group standardization removes scene-difficulty offsets so gradients depend on attempt rankings from the same start state.
  • B. Supervised Fine-tuning: Each task assigns one terminal scalar reward to every decision step, giving all trajectory tokens the same advantage instead of using per-step shaping.The surrogate remains token-level, with the frozen supervised policy providing the reference anchor.
  • B. Supervised Fine-tuning: The implementation decouples simulation from optimization by assigning episode rollouts to resident-scene simulator processes and serving action generation separately.Candidate episodes are filtered toward mixed solvability cases because always-solved and never-solved groups produce little within-group reward variance.
  • B. Supervised Fine-tuning: The online pipeline collects rollouts across navigation tasks and normalizes visibility, position, persistence, alignment, arrival, termination, discovery, efficiency, and progress rewards.These normalized rewards drive GRPO policy updates across embodied visual tracking, instruction following, and object goal navigation.
  • B. Supervised Fine-tuning: Tracking rewards combine visibility and position quality with motion-plan agreement and collision penalties, while persistence penalizes early termination over a 300-step horizon.ObjectNav rewards arrival with path-length efficiency and graded proximity, whereas instruction-following rewards arrival alignment using nDTW.
  • B. Supervised Fine-tuning: Instruction-following rewards combine final-goal arrival with reference-path alignment, while ObjectNav uses path efficiency because category goals prescribe no route.A graded proximity term ranks partial progress among failures in object navigation.

4) Optimization Details:

The optimization configuration uses eight H100 GPUs with fixed rollout and policy-update settings for online reinforcement learning.

  • 4) Optimization Details:: G = 8 trajectories per seed and B = 32 seeds produce 256 episodes per update.The policy clip range is ε = 0.2 and the KL coefficient is β = 0.01.

VI. EXPERIMENTS

The experiments assess embodied reasoning, simulated navigation, and real-world transfer under explicit sensing and deployment protocols. LightNav-ER achieves the strongest complete-set spatial-intelligence average, while evaluations cover diverse tasks, scenes, and robot embodiments.

  • VI. EXPERIMENTS: The evaluation spans 8 embodied-reasoning benchmarks, 10 navigation simulation settings, and deployment on four physical robot embodiments.Navigation covers instruction following, closed- and open-vocabulary ObjectNav, and embodied visual tracking.
  • VI. EXPERIMENTS: INSIGHT-Bench evaluates 1,097 episodes from 210 scenes using a shared 120° forward RGB stream, 300-action budget, and task-specific indoor and outdoor goal radii.The success criterion requires stopping within the goal radius while the target remains in the final frame’s field of view.
  • VI. EXPERIMENTS: Comparisons include general-purpose and specialized VLMs, modular navigation systems, task-specific policies, and generalist VLM/VLA policies with sensing assumptions reported explicitly.Runs identify whether methods use single-view RGB, panoramic or multi-camera RGB, depth, and odometry.
  • VI. EXPERIMENTS: 67.4 is LightNav-ER’s highest complete-set average across 8 embodied-reasoning benchmarks, exceeding Qwen3-VL-4B by +4.3 (6.8%) and Molmo2-ER by +4.6 (7.3%).It ranks first on 4 benchmarks and second on the remaining 4, indicating broad rather than uniform dominance.
  • VI. EXPERIMENTS: ER mid-training improves 7 of 8 scores and raises the macro average from 63.1 to 67.4, an absolute improvement of +4.3 (6.8%) over Qwen3-VL-4B initialization.The largest gains are Where2Place (+12.6; 19.7%) and RefSpatial (+11.9; 26.2%).

1) Vision-Language Navigation:

LightNav-0 delivers strong monocular performance across continuous VLN, closed- and open-vocabulary ObjectNav, INSIGHT-Bench, and visual tracking. Its advantages are broad but not uniform: trajectory fidelity and specialist tracking metrics remain complementary strengths.

  • Vision-Language Navigation: LightNav-0 achieves the strongest monocular R2R result on all four reported metrics.SR rises from 66.9 to 68.5, SPL from 62.3 to 62.8, and NE decreases from 4.05 to 3.91 m.
  • Vision-Language Navigation: LightNav-0 obtains the best monocular NE, SR, and SPL on RxR, while nDTW remains below DualVLN.SR improves from 73.4 to 73.6, SPL from 63.5 to 64.5, and NE decreases from 4.09 to 3.66 m; nDTW is 67.4 versus DualVLN’s 70.0.
  • Object Goal Navigation: LightNav-0 leads monocular SR and SPL across all three closed-vocabulary ObjectNav settings without depth or odometry.It also exceeds listed multi-view systems on several MP3D and HM3D comparisons.
  • Object Goal Navigation: Open-vocabulary ObjectNav gains are largest on synonym and unseen categories, with SPL improvements reaching 34.4% and 21.7%, respectively.The policy uses a single RGB input despite comparisons against monocular methods that may use depth and odometry.
  • INSIGHT-Bench: LightNav-0 achieves the best aggregate INSIGHT-Bench result, improving SR from 27.4 to 43.7 and SPL from 24.0 to 41.5.NE decreases from NaVid’s 4.25 m to 3.88 m.
  • Embodied Visual Tracking: LightNav-0 leads SR in both tracking regimes, especially distracted tracking, while ReferTrack retains higher TR.Distracted-tracking SR rises from 73.3 to 82.6, and collision rate decreases from 5.51 to 4.62.

D. Cross-Domain and Real-World Generalization

The same LightNav-0 checkpoint transfers across game domains, robot embodiments, tasks, and real-world scenes without task- or scene-specific adaptation. These evaluations are qualitative demonstrations of breadth rather than standardized quantitative comparisons.

  • Evaluation Scope: Real-world rollouts assess transfer breadth rather than quantitative benchmark performance because virtual and physical settings lack standardized action spaces and success protocols.The demonstrations use the same checkpoint without domain-specific fine-tuning.
  • Cross-Domain Generalization: The same checkpoint operates across four game domains with different visual styles, scene structures, and control dynamics.It follows spatial instructions, tracks a moving Minecraft target, and drives through Trigger Rally checkpoints.
  • Cross-Domain Generalization: Dual-channel pointing supplies an affordance point for navigable motion and an object point for the referred target or landmark.The shared representation supports first-person navigation, target following, and vehicle control.
  • Real-World Generalization: LightNav-0 transfers to physical environments for visual tracking, instruction following, and object search without task- or scene-specific adaptation.Tracking includes previously unseen dynamic target classes such as humanoid robots, wheeled robots, and carts.
  • Real-World Generalization: A shared trajectory follower connects the unchanged RGB-to-trajectory policy to four heterogeneous robot platforms.It converts predictions into odometry and velocity commands for each platform’s built-in locomotion policy.
  • Ablation Study: Embodied-reasoning initialization raises mean SR across eight settings from 60.8 to 63.1, while dual-channel pointing raises mean SR from 54.7 to 63.1.Both ablations use single-view RGB observations and the same broad navigation setting families.

F. Scaling Analysis

Scaling experiments show that performance improves with moderate model, data, and environment growth, but these factors behave differently near the largest tested regimes. Environment coverage remains especially strong, while additional parameters alone are not consistently beneficial.

  • Model Scaling: Scaling the backbone from 2B to 4B raises R2R and RxR SR/SPL by 6.6–9.6 points across four measures.Scaling to 8B produces mixed changes, including decreases in R2R SR/SPL and RxR SR.
  • Data Scaling: Increasing the training-data fraction produces monotonic but saturating gains on both R2R and RxR.From 1/16 to full data, gains are 15.0–17.4 points; the final doubling adds only 0.8/0.9 R2R and 1.4/0.4 RxR SR/SPL points.
  • Environment Scaling: Expanding training environments improves all four R2R and RxR measures at every tested increment.From 1/8 to full coverage, gains are 16.7/16.2 points on R2R and 21.1/19.1 points on RxR, exceeding matched data-scaling gains.
  • Conclusion: LightNav-0 uses a compact VLM, unified monocular RGB policy, temporal history compression, and hierarchical RVQ action tokens across navigation tasks.The training corpus spans 2K+ scenes and 4K+ hours of embodied trajectories.
  • Limitations and Future Work: The conclusion identifies a single decision pathway without explicit separation between high-frequency local control and slower semantic deliberation as an extension direction.A proposed dual system would pair reactive obstacle avoidance with a slower VLM planner.
  • Limitations and Future Work: Internet-scale video pretraining is proposed to broaden open-world concept coverage beyond curated embodied datasets.The stated target is exposure to rarer scenes, interactions, and motion patterns.
Loading 2608.30935v1…