PaperlayerView exact revision on arXiv

Source-linked AI summary

LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild

Linkai Liu, Yuntian Zhang, Zhenshan Bing, Chen Chen, Lingjuan Lyu, Shangguang Wang, Mengwei Xu, Dongqi Cai

arXiv:2610.12368v1cs.RO

TL;DR

Navigation policies can generate trajectories without evaluating their future consequences, while visual rollouts for world-model evaluation are costly. LiteNWM shares visual encoding, jointly predicts action-conditioned latent futures, and scores candidate trajectories. Across RECON, SCAND, and SACSoN it reduces macro ATE by 17.56% over NWM-XL, while also transferring to MBRA and supporting real-robot navigation.

  • Problem

    Navigation policies can generate trajectories without explicitly evaluating their future consequences, while world-model visual rollouts are costly when evaluating multiple candidates.

    Source: §I · p. 1

  • Method

    LiteNWM shares visual encoding across candidates, jointly predicts action-conditioned future latents at multiple horizons, and uses incumbent-relative scoring to select trajectories.

    Source: §I · pp. 1–2

  • Results

    LiteNWM reduces macro ATE by 17.56% over NWM-XL on matched NoMaD proposal pools across RECON, SCAND, and SACSoN.

    Source: §V · p. 8

  • Takeaways & Limitations

    The frozen evaluator transfers to other navigation policies without proposer-specific retraining and supports closed-loop navigation on a physical robot.

    Source: §V · p. 8

Abstract

from arXiv · show

Direct visual navigation policies generate trajectories efficiently but do not explicitly evaluate their future consequences. Generative navigation world models provide this foresight through visual rollouts, which are costly when evaluating multiple candidates. We present LiteNWM, a latent navigation world model that shares visual encoding across candidates and jointly predicts their action-conditioned future representations at multiple horizons, while a learned scorer uses these predictions to select trajectories. In offline evaluations on RECON, SCAND, and SACSoN, LiteNWM reduces macro-averaged trajectory error by 17.56% relative to NoMaD+NWM-XL and achieves a 128.00-fold end-to-end speedup on an RTX 5090. The same evaluator transfers from NoMaD to MBRA without proposer-specific retraining, reducing MBRA's macro-averaged trajectory error by 16.2%. In real-robot experiments in unseen indoor and outdoor environments, LiteNWM improves navigation success from 43.3% to 83.3% relative to NoMaD. These results demonstrate that LiteNWM can be deployed for future-aware planning and closed-loop navigation on a physical robot.

I. INTRODUCTION

Navigation policies can propose trajectories without evaluating their future consequences, while visual world-model rollouts make candidate evaluation costly. LiteNWM uses shared encoding, joint latent prediction, and relative scoring; it improves trajectory accuracy and evaluation speed while transferring across proposers and to a physical robot.

I. INTRODUCTION

  • Direct visual policies do not explicitly evaluate future consequences, and candidate-conditioned RGB rollout accounts for 97.6% of captured NoMaD+NWM-XL computation.NWM addresses future-aware selection through predicted observations, but this evaluation path is costly when comparing multiple candidates.Source: §I · p. 1
  • LiteNWM targets candidate-wise rollout cost with shared visual encoding, joint H1–H4 latent prediction, and an incumbent-relative scorer that uses proposal-set context while keeping policies frozen.The paper identifies candidate-wise RGB rollout as the dominant evaluation cost.Source: §I · pp. 1–2
  • LiteNWM improves trajectory accuracy while achieving 128.00× and 16.86× end-to-end speedups over NoMaD+NWM-XL and NoMaD+NWM-S, respectively.The evaluator also transfers to MBRA without proposer-specific retraining and supports closed-loop navigation on a Unitree Go2 EDU in unseen environments.Source: §I · pp. 1–2

A. Generalist Navigation Policies and Trajectory Selection

  • Generalist policies such as GNM, ViNT, and NoMaD learn from diverse robot datasets –, while NavDP and ViLiNT add learned trajectory selection [6], [7].LiteNWM keeps trajectory generation fixed and evaluates the resulting proposals using predicted future evidence.Source: A. Generalist Navigation Policies and… · p. 2

B. Generative World Models for Navigation

  • Navigation world models use predicted candidate futures for trajectory selection; NWM also establishes a modular proposer–evaluator interface.The open question is whether action-conditioned latent features can compare short-horizon candidates without reconstructing RGB futures.Source: B. Generative World Models for… · p. 2

C. Latent Predictive and World–Action Models

  • Prior latent models span learned-latent planning and action-conditioned feature forecasting –[13]; DF3 is observation-only [14], while ReL-NWM retains sequential autoregressive rollouts [15].LiteNWM instead uses shared encoding and joint multi-horizon prediction, eliminating autoregressive feedback across horizons.Source: C. Latent Predictive and World–Action… · p. 2
  • WAM-Nav and NavWAM couple visual foresight with action generation [16], [17], whereas LiteNWM asks whether latent predictions can distinguish proposals from a frozen policy.Its scope is trajectory evaluation rather than unified world–action policy learning.Source: C. Latent Predictive and World–Action… · p. 2

III. METHOD

LiteNWM evaluates trajectories proposed by a frozen navigation policy by predicting their future latent features and comparing them with a retained incumbent. Its formulation uses a shared observation history and candidate action sequences to improve trajectory choice without replacing or fine-tuning the proposer.

III. METHOD

  • LiteNWM combines frozen navigation policies, shared visual encoding, an action-conditioned latent predictor, and a scorer that evaluates predicted futures relative to the incumbent and proposal set.The workflow retains the incumbent if no eligible challenger is selected.Source: §III · p. 2, Figure 2 · p. 3

A. Problem Formulation

  • The frozen policy generates K = 16 candidate trajectories from a four-frame visual history and goal; each trajectory is represented as a sequence of actions [Δx, Δy, Δψ].The incumbent is retained separately, and K counts challengers rather than the incumbent.Source: A. Problem Formulation · p. 2
  • The latent predictor maps observation history and each candidate action sequence to future features, supporting better trajectory choice without replacing or fine-tuning the proposer.The workflow encodes the history and goal once per state and jointly predicts H1–H4 futures for the incumbent and challengers.Source: A. Problem Formulation · p. 2, Figure 2 · p. 3

B. Latent-space Navigation World Model

LiteNWM replaces candidate-wise RGB rollout with shared encoding and joint action-conditioned latent prediction across four horizons. A scorer uses predicted futures and proposal-set context to evaluate outcomes relative to the incumbent.

B. Latent-space Navigation World Model

  • Candidate-wise RGB rollout accounts for 84.8% of NWM-S and 97.6% of NWM-XL captured latency, motivating shared encoding and latent prediction before trajectory scoring.The history and goal are encoded once per state, and the latent predictor does not take the goal as input.Source: B. Latent-space Navigation World Model · p. 3
  • Pose-conditioned warping initializes prediction queries, while six factorized blocks integrate action and history conditions for joint H1–H4 prediction.The poses are computed by SE(2) integration; the blocks refine the queries using horizon embeddings broadcast across spatial tokens.Source: B. Latent-space Navigation World Model · p. 3, Figure 4 · p. 4
  • All four horizons are predicted jointly without autoregressive feedback, and the latent predictor is supervised by frozen DINOv3-adapted future features.The scorer combines predicted futures with the current state and goal, then uses action, history, and proposal-relative features to predict outcome changes from the incumbent.Source: B. Latent-space Navigation World Model · p. 4

C. Training, Trajectory Selection, and Execution

LiteNWM trains predictor and scorer ensembles sequentially while keeping NoMaD and DINOv3 frozen. Its selection procedure gates eligible challengers, retains the incumbent when none qualify, and replans after executing a short trajectory segment.

C. Training, Trajectory Selection, and Execution

  • Predictor and scorer ensembles are trained sequentially, with NoMaD and DINOv3 frozen throughout training.The predictor combines latent regression, cosine, action-sensitivity, temporal-consistency, and uncertainty losses; the scorer uses regression, classification, and ranking losses.Source: C. Training, Trajectory Selection, and… · p. 4
  • Dual improvement requires decreases in both ATE and J, whereas harm denotes an increase in either, with tolerance 10−7 m.Source: C. Training, Trajectory Selection, and… · p. 4
  • The learned acceptance gate admits challengers only after physical and proposal-support checks and an eligibility test; otherwise, the incumbent is retained.Temperatures, dataset-specific calibration terms, and thresholds are fitted on scorer-calibration data.Source: C. Training, Trajectory Selection, and… · p. 4
  • At each control cycle, LiteNWM observes, proposes, predicts, selects, executes a short segment, and replans; MBRA outputs use a deterministic action adapter with frozen LiteNWM components.Table I describes cross-domain offline trajectory evaluation at H = 4; its † baselines report 95% trajectory-cluster-robust confidence-interval half-widths, while other ± values are standard deviations across seeds.Source: C. Training, Trajectory Selection, and… · p. 4, Table I · p. 5

IV. EXPERIMENTS

The experiments evaluate trajectory selection, predictive representations, efficiency, and physical-robot deployment using fixed development cohorts from three navigation datasets. The setup compares navigation policies, generated-vision world models, and a latent-world-model baseline, with offline accuracy metrics and horizon-wise errors.

IV. EXPERIMENTS

  • The experiments assess trajectory selection, predictive representations, end-to-end efficiency, and physical-robot deployment.The supplied setup describes fixed development cohorts and offline evaluation.Source: §IV · p. 5

A. Experimental Setup

  • Three fixed development cohorts use 100 query states per domain, a four-step horizon, and four proposal seeds for 1,200 state–seed evaluations.LiteNWM uses K = 16 frozen NoMaD challengers, while deterministic NoMaD remains the incumbent.Source: A. Experimental Setup · p. 5
  • The baselines include policies without future-aware reranking, generated-observation world models, and DINO-WM adapted to rerank candidates using predicted future features.The policy baselines are GNM, ViNT, and NoMaD; the world-model baselines include NoMaD+NWM-S and NoMaD+NWM-XL.Source: A. Experimental Setup · p. 5
  • ATE is translational RMSE across future waypoints, RPE is relative-translation RMSE between successive waypoints, and lower values indicate better accuracy.Trajectories are query-centered and normalized by dataset-average waypoint spacing; SR@1 measures offline endpoint agreement, not closed-loop navigation success.Source: A. Experimental Setup · p. 5
  • Fig. 5 reports macro and per-domain step errors for NoMaD, NoMaD+NWM-XL, and LiteNWM, with LiteNWM showing slower error growth at later timesteps across all three domains.NWM comparisons are constrained by adapting methods designed for length-eight trajectories to H = 4; absolute errors are not directly comparable with the original NWM results.Source: Figure 5 · p. 5, A. Experimental Setup · p. 5
  • Table II reports a three-domain development macro at fixed K = 16, with lower values better; its first four rows use separately trained matched scorers and its final row averages two prespecified mismatches on a frozen scorer.The caption identifies the ablation and correspondence-diagnostic design but does not provide row values.Source: Table II · p. 6

B. End-to-end Evaluation

LiteNWM improves trajectory selection across three domains, and ablations examine how predicted future features support that improvement. Its frozen evaluator also transfers to MBRA without proposer-specific retraining.

B. End-to-end Evaluation

  • LiteNWM lowers ATE by 25.21% on RECON, 14.98% on SCAND, and 14.71% on SACSoN versus NoMaD+NWM-XL, while leading on ATE, RPE, and SR@1 in all domains.SR@1 rises by 6.5, 3.5, and 7.0 percentage points on RECON, SCAND, and SACSoN, respectively.Source: B. End-to-end Evaluation · p. 6

C. Ablation Study

  • Predicted future features improve selection over no-future and static-feature controls; H4 supplies most of the gain, with intermediate horizons adding trajectory-evolution cues.The variants use identical candidate pools and separately trained scorers under matched conditions.Source: C. Ablation Study · p. 6
  • Mispaired candidate–future features increase macro ATE by 0.0294, supporting the scorer’s use of correctly paired predicted-future evidence.The 95% paired cluster-bootstrap CI is [0.0175, 0.0415]; joint bundle permutation and the no-prediction control preserve selections.Source: C. Ablation Study · p. 6

D. Transfer Across Navigation Policies

  • The frozen LiteNWM evaluator reduces MBRA ATE by 6.9% on RECON, 23.0% on SCAND, and 16.8% on SACSoN without proposer-specific training or recalibration.A deterministic adapter standardizes MBRA trajectories to LiteNWM’s four-step local action representation.Source: D. Transfer Across Navigation Policies · p. 6

E. System Performance

LiteNWM reduces the cost of future-aware trajectory evaluation and is evaluated in closed-loop navigation on a physical robot. Across three real-world scenes, it improves success and SPL relative to NoMaD.

E. System Performance

  • LiteNWM is 16.86× faster than NoMaD+NWM-S and 128.00× faster than NoMaD+NWM-XL on an RTX 5090.The end-to-end measurement includes proposal generation, future evaluation, scoring, refinement, and final selection.Source: E. System Performance · p. 6
  • Shared encoding and joint H1–H4 latent prediction replace candidate-wise RGB rollout, which accounts for 84.8% of NWM-S and 97.6% of NWM-XL captured component latency.LiteNWM predicts jointly without autoregressive feedback and scores futures without RGB decoding; its reported speedup covers the complete pipeline.Source: E. System Performance · pp. 6–7

F. Real-World Deployment

  • The offline representative comparisons show selected trajectories against ground truth over the first 3 m, while robot comparisons include exploratory NoMaD+NWM-S probes.The deployment probes characterize planning delay and are excluded from aggregate navigation-outcome comparisons.Source: Figure 6 · p. 7, Figure 7 · p. 7, F. Real-World Deployment · p. 8
  • LiteNWM navigates previously unseen indoor and outdoor layouts without scene-specific training or online model updates.The evaluation used monocular RGB observations and an RGB goal image, with 10 trials per method in each of three scenes.Source: F. Real-World Deployment · p. 7

V. CONCLUSION

  • LiteNWM improves navigation success from 43.3% to 83.3% and SPL from 0.409 to 0.810 over NoMaD across three real-world scenes.Each method was evaluated over 30 trials on a Unitree Go2 EDU.Source: §V · p. 8, Table V · p. 8

APPENDIX A SUPPLEMENTARY EVALUATION PROTOCOL

Supplementary evaluations specify grouped uncertainty estimation and test whether predictor-side actions and candidate-specific future evidence contribute to selection. The reported interventions increase macro ATE when these inputs are disrupted.

APPENDIX A SUPPLEMENTARY EVALUATION PROTOCOL

  • Four-seed paired bootstrap resampling keeps all queries and proposal seeds together within each sampled group.Confidence intervals are 95% cluster-bootstrap intervals, with Holm corrections applied using the original experiment families.Source: Appendix A · p. 9

A. Predictor Action Input

  • Permuting predictor-side challenger actions increases macro ATE by 0.0300 and changes 63.75% of selections, even with scorer-side action descriptors fixed.The intervention also holds model weights, visual inputs, incumbent, and actual candidate trajectories fixed.Source: A. Predictor Action Input · p. 9

B. Selection Changes and Dependence Grouping

  • Evidence for predicted futures, mismatched future features, and permuted predictor actions remains positive under the alternative dependence grouping.The grouping preserves queries and proposal seeds, weights queries within domains, and weights domains equally for macro estimates.Source: B. Selection Changes and Dependence… · p. 9, Table VI · p. 9

A. Selection Within the Initial Pool

Fixed-pool selection retains most of the default configuration’s error reduction, while supplementary diagnostics examine replacements, a historical acceptance gate, and ensemble variants.

A. Selection Within the Initial Pool

  • Fixed-pool selection retains 94.75%, 95.03%, and 93.32% of CEM1’s error reductions over NoMaD+NWM-XL for ATE, RPE, and FDE, respectively.Each percentage divides the baseline-to-fixed-pool error difference by the baseline-to-CEM1 difference.Source: A. Selection Within the Initial Pool · p. 9, Table VII · p. 10
  • Among 1,200 state–seed observations, the selected output improves ATE in 49.75%, increases it in 8.42%, and retains the incumbent in 41.83%.A lower-ATE challenger exists in 98.42% of observations; 597 of 698 replacements contribute to improved trajectory accuracy.Source: A. Selection Within the Initial Pool · p. 9
  • At τ = 1, the historical macro endpoint-agreement advantage over NoMaD+NWM-XL is 6.33 percentage points, contracting near saturation at looser thresholds.Lower average error coincides with improvement on 80% of RECON, 68% of SCAND, and 71% of SACSoN states.Source: A. Selection Within the Initial Pool · pp. 9–10

APPENDIX D ACCEPTANCE GATE AND ENSEMBLE DIAGNOSTICS

  • In the historical CEM3 gate diagnostic, the gate reduces harmful replacements from 23.00% to 9.92% while reducing replacement coverage from 100% to 64.83%.The no-gate minus gated macro ATE contrast is −0.00409, with a 95% CI of [−0.00912, 0.00076] that includes zero.Source: Appendix D · p. 10, Table VIII · p. 10
  • None of the six ensemble contrasts in the reported point estimates survives Holm correction.The full scorer ensemble has lower ATE than each individual scorer, while the full predictor ensemble has higher ATE than each single-predictor configuration.Source: Appendix D · p. 10, Table IX · p. 10

APPENDIX F QUALITATIVE REAL-WORLD DEPLOYMENT

The deployment examples show indoor and outdoor route progression across varied surroundings. Onboard execution uses a frozen LiteNWM evaluator, with route memory switching predefined visual goals on extended routes.

A. Route Progression

  • Indoor routes progress through doorways and corridors to a chair goal, while outdoor routes cross paved areas and vegetation toward a visual target.The ordered observations follow successive local motions under the fixed prediction horizon.Source: A. Route Progression · p. 10, Figure 10 · p. 11

B. Deployment Conditions

  • Deployment views include furnished interiors, paved outdoor areas, vegetation, building-side passages, and sunlit or shaded conditions.Selected tiles illustrate visual diversity, not independent environments or trials.Source: B. Deployment Conditions · p. 10, Figure 11 · p. 11

C. Onboard Deployment

  • LiteNWM runs frozen on a Unitree Go2 EDU's Jetson Orin; extended routes use a predefined sequence of visual goals, with route memory switching goals while the evaluator selects local trajectories at H = 4.Onboard odometry supports motion feedback and route progression.Source: C. Onboard Deployment · pp. 10–11
  • Onboard deployment skips CEM when the initial pool meets acceptance conditions; otherwise, it performs up to three refinement rounds and stops when an eligible challenger is available.Across 311 logged planning calls, four used two rounds and 19 used three.Source: C. Onboard Deployment · pp. 10–11

D. Onboard Planning Latency

Onboard latency measurements distinguish internal planning time from robot motion and initialization, while NWM-S reranking is very slow on Jetson Orin. Appendix G notes limits from fixed proposals and four-step prediction, and outlines future extensions.

D. Onboard Planning Latency

  • LiteNWM latency includes encoding, proposal generation, evaluation, scoring, conditional refinement or recovery, and selection, but excludes initialization and robot motion.Statistics pool completed calls from 30 LiteNWM trials; NoMaD latency uses 302 calls across 30 runs.Source: D. Onboard Planning Latency · p. 12
  • 692.7 s at H = 4 and 1381.2 s at H = 8: NWM-S reranking on Jetson Orin takes this long with K = 16, three repeats, and DDIM-100.These are reranking-only measurements on one saved observation–goal state, separate from the RTX 5090 benchmark.Source: D. Onboard Planning Latency · p. 12

APPENDIX G

  • Fixed-pool selection retains 94.75% of CEM1's observed ATE reduction; three CEM rounds lower macro ATE by a further 0.64% while raising RECON benchmark wall time by approximately 71%.The experiments do not distinguish limited search coverage from errors in evaluating refined trajectories.Source: Appendix G · p. 12
  • The frozen proposer receives no evaluator training feedback, and the four-step predictor does not establish accuracy beyond its horizon; aggressive score optimization may expose errors outside the proposal distribution.Longer routes therefore rely on repeated replanning.Source: Appendix G · p. 12
  • Future directions include training a generator with scorer and predictor feedback, distilling refined trajectories, and adding hierarchical subgoal selection for detours and partially observed routes.Exploration and search-and-rescue would additionally require destination choice and target discovery.Source: Appendix G · p. 12
Loading 2610.12368v1…