PaperlayerView exact revision on arXiv

Source-linked AI summary

NavWAM: A Navigation World Action Model for Goal-Conditioned Visual Navigation

Daichi Azuma, Taiki Miyanishi, Koya Sakamoto, Shuhei Kurita, Yaonan Zhu, Petr Khrapchenkov, Motoaki Kawanabe, Yusuke Iwasawa, Yutaka Matsuo

arXiv:2606.13494v1cs.ROcs.CV

TL;DR

Goal-conditioned visual navigation requires anticipating future views and goal progress under partial observability, while navigation world models still rely on external planners for action selection. NavWAM uses a diffusion-transformer policy with a shared latent sequence for future observations, values, and action chunks, making visual foresight directly executable. In the reported offline and real-robot evaluations, it improves over planning-based world-model baselines without CEM-style action search and remains competitive with a larger direct policy.

  • Problem

    Goal-conditioned navigation needs explicit visual foresight under partial observability, but navigation world models remain prediction modules that require external planning for closed-loop control.

    Source: §1 · p. 1

  • Method

    NavWAM jointly represents future egocentric observations, goal-progress values, and executable action chunks in a shared latent sequence within a diffusion-transformer policy.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    NavWAM improves navigation performance over planning-based world-model baselines without CEM-style test-time trajectory optimization and remains competitive with a larger direct navigation policy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Learning future prediction together with action and value targets makes visual foresight directly usable for closed-loop robot navigation.

    Source: §1 · p. 2, §6 · p. 8

    Evaluation focuses on indoor image-goal navigation, with 24 real-world episodes across four environments; language- or object-specified goals remain future work.

    Source: §6 · p. 8

Abstract

from arXiv · show

Goal-conditioned visual navigation requires a robot to act under partial observability by anticipating how its motion will change the future egocentric view and whether that change brings it closer to the goal. Navigation world models provide such visual foresight, but they remain prediction modules that require an external planner to convert predicted futures into closed-loop control. We propose Navigation World Action Model (NavWAM), a diffusion-transformer policy that turns navigation world-model prediction into executable action by representing future observations, goal-progress values, and action chunks in a shared latent sequence. By learning future prediction jointly with the action and value targets that determine closed-loop behavior, NavWAM makes visual foresight directly usable for robot control. We build NavWAM through simulation pretraining and real-robot adaptation, and evaluate it on image-goal navigation against planning-based world models and a representative direct navigation policy. Across offline benchmarks and closed-loop real-robot deployment, NavWAM improves over planning-based world-model baselines in our evaluations while using the default policy mode without CEM-style action search. Project page: https://dachii-azm.github.io/navwam/

1 Introduction

Goal-conditioned navigation needs explicit visual foresight, but navigation world models still depend on external planning to turn predictions into actions. NavWAM unifies future-view prediction, goal-progress estimation, and executable action generation in one policy, improving over planning-based baselines in the reported evaluations.

Motivation

  • Navigation world models predict action-conditioned future views but require an external planner to select actions, making deployment depend on planning choices.This separation is the central limitation motivating NavWAM.Source: §1 · p. 1

Contribution

  • NavWAM jointly represents future egocentric observations, goal-progress values, and executable action chunks, converting visual foresight into a closed-loop policy.The shared latent formulation replaces separate prediction and action-selection stages.Source: §1 · p. 2, Figure 1 · p. 2

Novelty

  • NavWAM learns future prediction, value estimation, and action generation in one policy representation for goal-conditioned navigation under partial observability.This targets the need to anticipate viewpoint changes and their effect on goal progress.Source: §1 · p. 2

Results

  • NavWAM improves over planning-based world-model baselines without CEM-style test-time trajectory optimization while remaining competitive with a larger direct navigation policy.The evaluation covers offline benchmarks and closed-loop real-robot deployment, using a 2B-parameter video backbone against a larger 7B-parameter VLA baseline.Source: §1 · p. 2

Evaluation

  • The paper evaluates NavWAM through simulation pretraining, real-robot adaptation, offline benchmarks, and closed-loop image-goal navigation.The primary comparison is against NWM and the representative direct policy OmniVLA.Source: §1 · p. 2

2 Related Work

Prior navigation methods either predict actions directly or generate future visual states that require separate planning. NavWAM combines future prediction, goal-progress estimation, and action generation for closed-loop goal-conditioned navigation.

Goal-Conditioned Visual Navigation

  • Goal-conditioned navigation methods span explicit geometric, topological, or semantic planning and learned direct action prediction, but usually do not share one representation for future views, progress, and actions.The cited learned approaches include diffusion policies, large-scale reinforcement learning, and vision-language-action models.Source: §2 · p. 3

Visual Foresight for Navigation

  • Pixel-space navigation methods make future egocentric prediction explicit, yet still require a planner, value map, or scoring function to convert predicted futures into actions.Examples include PathDreamer, Schrödinger’s Navigator, NWM, and NavDreamer.Source: §2 · p. 3

Video-Based Robot Policies

  • Video-based robot policies use generative modeling as a policy representation, including action diffusion, video intermediates, and joint prediction of future images and actions.Diffusion Policy [37], NoMaD [3], UniPi, Dreamitate, and GR-1 illustrate this direction.Source: §2 · p. 3
  • NavWAM adapts video-world-model policy ideas to goal-conditioned navigation under partial observability by coupling viewpoint-changing egocentric prediction with goal progress and local-frame action generation.This distinguishes the navigation setting from manipulation-oriented video policies.Source: §2 · p. 3

3 Problem Setup and World-Action Formulation

Under partial observability, navigation must couple prediction of future views with action selection toward the goal. NavWAM addresses the separation in direct policies and navigation world models by learning both within one executable world-action policy.

3 Problem Setup and World-Action Formulation

  • Image-goal navigation asks a robot to reach a target image from partial egocentric observations while outputting an executable action chunk over horizon H.The goal is a target image captured at the goal location.Source: §3 · p. 3

3.1 Visual Foresight and Action Selection

  • Successful navigation requires predicting how motion changes the view and choosing actions that bring the robot closer to the goal.Action prediction alone does not expose the future visual consequences of motion.Source: §3.1 · pp. 3–4
  • Direct policies efficiently predict actions but are not explicitly trained to predict what the robot will see after executing them.Their foresight may be implicit rather than represented as future visual prediction.Source: §3.1 · p. 4
  • Navigation world models explicitly predict future observations conditioned on candidate actions, but require a separate action-selection procedure.This leaves a gap between explicit future prediction and direct executable control.Source: §3.1 · p. 4

3.2 From Navigation World Models to NavWAM

  • NWM uses CEM-based planning to sample action sequences, predict their future observations, and select a best-scoring trajectory, increasing test-time computation and dependence on external optimization.The model predicts possible futures while the external procedure decides which future indicates goal progress.Source: §3.2 · p. 4
  • NavWAM learns joint world-action prediction to close the gap between future prediction and action selection.The formulation makes visual foresight directly usable for executable control.Source: §3.2 · p. 4
  • At inference, NavWAM predicts an action chunk, auxiliary future state, future egocentric observations, and a goal-conditioned value, then executes the action chunk receding-horizon.These targets couple future perception, goal progress, and executable motion in one prediction problem.Source: §3.2 · p. 4

4 Method

NavWAM represents goal-conditioned navigation as joint denoising over future perception, goal progress, and executable motion in a shared latent canvas. It trains multiple conditioning modes and executes predicted action chunks in a receding-horizon policy loop without requiring test-time action search.

4 Method

  • The shared canvas jointly represents visual and non-visual navigation variables, preserving a common latent representation for actions, states, values, and future images.The architecture uses latent frames rather than separate action or value heads, with the value frame representing goal progress and the action frame representing local-frame motion.Source: §4 · p. 4, §4.1 · p. 5

4.1 World-Action Latent Canvas

  • NavWAM uses a fixed nine-frame latent canvas whose observed frames condition generation of action, future state, future observations, and goal-progress value.The observed frames include a blank VAE frame, current robot state, goal image, and current egocentric observation; five prediction frames are generated.Source: §4.1 · p. 5

4.2 Training Objective

  • The diffusion objective denoises generated canvas frames from Gaussian-corrupted latents and upweights the action frame so low-dimensional actions are not dominated by image reconstruction.Conditioning includes the observed-frame mask, embeddings, and observed latent frames.Source: §4.2 · p. 5
  • A single model is trained in policy, world-model, and value modes, with a 50/25/25 sample split across action generation, action-conditioned prediction, and value estimation.The reported main results use only policy mode at inference; other modes support optional best-of-N sampling and auxiliary value estimation.Source: §4.2 · pp. 5–6

4.3 Inference

  • At test time, policy mode predicts an action chunk from the current observation and goal, executes it receding-horizon, and re-queries after the next observation.Future observations and goal-progress values provide interpretable foresight but are not required for execution.Source: §4.3 · p. 6
  • Optional value-guided best-of-N sampling selects among candidate action chunks, but the reported main results use default policy mode without CEM-style action search.Candidate futures and values are evaluated with the auxiliary world-model and value modes only in the optional setting.Source: §4.3 · p. 6

5 Experiments

The experiments evaluate NavWAM on offline image-goal benchmarks and closed-loop Diablo-robot deployment against planning-based world models and a direct navigation policy. Across these settings, NavWAM combines direct action prediction with future-view consistency and improves navigation results without CEM-style search.

5.1 Experimental Setup

  • The evaluation uses GO STANFORD for NWM-style comparison, SIT as a cleaner held-out benchmark for OmniVLA, and ATE, RPE, SR@1.0m, and subject consistency as applicable.Real-world methods receive 224 × 224 RGB observations and output local-frame actions under the same closed-loop setting.Source: §5.1 · p. 6

5.2 Results

  • With GO STANFORD fine-tuning, NavWAM achieves ATE 0.192 and RPE 0.070, outperforming Cosmos Predict2 and NWM without CEM-style action search.NavWAM directly predicts action chunks while also producing future observations and goal-progress values.Source: §5.2 · p. 7
  • NavWAM preserves visual foresight, increasing subject consistency from NWM’s 0.524 to 0.668 without task-specific fine-tuning and retaining 0.635 after fine-tuning.Qualitative predictions also remain closer to the goal scene, whereas NWM can drift to visually inconsistent futures.Source: §5.2 · p. 7
  • Adding action and state supervision reduces ATE from 0.326/0.569 to 0.107/0.287 at horizons h = 4/8, while value supervision further improves both ATE and RPE.Image-only future prediction is insufficient even with CEM using N = 120 candidate actions.Source: §5.2 · p. 7
  • On SIT, NavWAM remains competitive with OmniVLA, achieving lower ATE and slightly higher observed success rates at both horizons with a 2B-parameter backbone versus OmniVLA’s 7B.NavWAM additionally produces future observations and goal-progress values.Source: §5.2 · p. 7
  • NavWAM reaches the goal in 19/24 real-world episodes (79.2%), compared with 14/24 for OmniVLA (58.3%) and 4/24 for NWM (16.7%).The evaluation covers 24 episodes across four indoor environments, with termination at 1 meter or timeout.Source: §5.1 · p. 6, §5.2 · pp. 7–8

6 Conclusion

NavWAM turns visual foresight into a closed-loop navigation policy by jointly predicting future observations, goal progress, and executable action chunks. Its shared latent canvas encodes the goal, state, action sequence, and value targets used across simulation pretraining and real-robot adaptation.

6 Conclusion

  • NavWAM couples future perception, progress estimation, and action generation in one shared latent sequence for closed-loop navigation.The model uses a fixed nine-frame canvas containing observed and generated variables.Source: §6 · p. 8, A Method Details · p. 13, §A.1 · p. 13

A.1 Latent Canvas Frame Layout

  • The nine-frame canvas represents image-goal or world-model inputs in Frame 2, while non-image variables are broadcast over the latent spatial grid and recovered by spatial averaging.The IG variant stores the goal image; the WM variant stores the previous egocentric observation and uses text conditioning for the goal.Source: §A.1 · p. 13

A.2 Goal-Progress Value vt+H

  • Goal progress is encoded as a bounded [0, 1] value derived from trajectory distance, using Euclidean labels for real-robot data and geodesic distance for HM3D simulation.The cap dmax bounds the scalar, while HM3D uses Habitat shortest-path distances.Source: §A.2 · pp. 13–14

A.3 Robot State st / A.4 Action Chunk at:t+H−1

  • The state frame encodes normalized 2D position and yaw, while each action chunk contains H local-frame waypoint increments decoded from a 3H-dimensional action frame.The main paper uses H=4; HM3D pretraining uses H=16 before resetting to H=4 for joint fine-tuning.Source: §A.3 · p. 14, §A.4 · p. 14

A.5 Training Objective and Conditioning Modes

NavWAM trains a diffusion transformer with multiple conditioning patterns so one model learns policy generation, action-conditioned world prediction, and value estimation. A staged curriculum moves from long-horizon HM3D simulation pretraining to joint real-robot fine-tuning and optional GO STANFORD adaptation.

A.5 Training Objective and Conditioning Modes

  • The training objective applies diffusion loss to generated canvas frames and upweights the action frame to preserve the low-dimensional control signal.The action-frame multiplier is λ=5.Source: §A.5 · p. 14, §B.1 · p. 15
  • Three modes train one set of weights for control and prediction: policy mode predicts actions and futures, world-model mode predicts action-conditioned futures, and value mode predicts progress.The reported main results use policy mode at inference; world-model and value modes are auxiliary training modes.Source: §A.5 · p. 14

B.1 Training Hyperparameters

  • Denoising uses rectified-flow scaling with σdata=1.0, σcond=0, and a hybrid training noise distribution spanning σmax=200 to σmin=10−2.Inference narrows the range to σmax=80 and σmin=4.Source: §B.1 · p. 15

B.2 Training Curriculum

  • The curriculum pretrains on success-only HM3D simulator trajectories with H=16, then jointly fine-tunes on RECON, SACSON, and SCAND with H=4.The WM and IG variants branch from the Phase 2 joint fine-tuning stage and share Phase 1.Source: §B.2 · p. 15
  • Optional Phase 3 fine-tuning adapts the Phase 2 IG checkpoint on GO STANFORD training trajectories for results labeled NavWAM w/ FT.This phase uses the same schedule as the preceding fine-tuning stage.Source: §B.2 · p. 15

B.3 Datasets

The evaluation combines indoor image-goal datasets with held-out testing and real-robot deployment on a sensorized mobile platform. Training and evaluation phases use distinct dataset roles, while the reported hyperparameters remain largely shared across phases.

B.3 Datasets

  • GO STANFORD supplies indoor image-goal evaluation, with 30 held-out test episodes, while SIT provides 1,400 held-out episodes across 14 official test segments.The GO STANFORD test size follows the compute limit for reproducing NWM-style CEM planning with N=120 candidates.Source: §B.3 · p. 15, Table S2 · p. 16
  • Training uses HM3D simulation data in Phase 1, RECON/SACSON/SCAND in Phase 2, and GO STANFORD training trajectories only in optional Phase 3.Evaluation trajectories are not seen during fine-tuning.Source: §B.3 · p. 15

B.4 Hyperparameter Summary

  • The optimizer, learning rate, batch size, mode mixture, action upweight, and noise schedule stay constant across phases, while dataset, chunk size, and step count vary.Table S3 consolidates the phase-specific settings.Source: §B.4 · p. 16, Table S3 · p. 16

B.5 Robot Platform

  • Real-robot deployment uses a Direct Drive Tech Diablo with RGB-D, LiDAR, Jetson AGX Orin, and ROS 2 control for executing local-frame action chunks.The platform includes an Intel RealSense D455 and Livox Mid-360 sensor.Source: §B.5 · p. 16, Figure S1 · p. 17

C.1 Inference Efficiency

NavWAM replaces external CEM trajectory search with a single denoising chain while preserving or improving trajectory accuracy against planning-based world-model baselines. Its compute and memory costs avoid the candidate-count scaling of CEM and support roughly 5 Hz operation on the evaluated hardware.

C.1 Inference Efficiency

  • The evaluation tabulates trajectory accuracy and inference cost on the same 30-episode GO STANFORD subset, with N=120 matching the main-paper NWM protocol.CEM baselines batch candidate trajectories into one forward pass.Source: §C.1 · p. 17, Table S4 · p. 18
  • Single-pass NavWAM matches or outperforms NWM across CEM budgets, while in-domain fine-tuning adds approximately 1.7× improvement in ATE.The comparison uses GO STANFORD trajectory metrics and includes a same-backbone Cosmos Predict2 + CEM baseline.Source: §C.1 · p. 17, Table S4 · p. 18
  • CEM compute scales linearly with candidate count N, whereas NavWAM uses one denoising chain independent of candidate budget.The same-backbone Cosmos Predict2 + CEM baseline requires orders of magnitude more compute per executed action.Source: §C.1 · pp. 17–18
  • Under the paper-faithful N=120 setting, CEM baselines run at sub-Hz rates, while NavWAM replaces the CEM search loop with a single denoising chain.Measurements use a single NVIDIA Blackwell-class GPU with 224×224 RGB inputs, and cost is normalized per executed action.Source: §C.1 · pp. 17–18, Table S4 · p. 18

C.2 Component Ablation

The ablation shows that future-view supervision improves action-policy trajectory accuracy, while real-robot analyses show distinct failure patterns, coherent predictions, and value changes tied to goal visibility.

C.2 Component Ablation

  • At h=8, future-view supervision lowers ATE from 0.262 to 0.192 (−27%) and RPE from 0.103 to 0.070 (−32%).At h=4, ATE also improves from 0.090 to 0.076 and RPE from 0.045 to 0.037, showing gains at both horizons.Source: §C.2 · p. 18

C.3 Real-Robot Failure Modes

  • NavWAM has the fewest real-robot failures overall, with 3 drift failures and 2 collisions, compared with NWM’s drift-heavy and OmniVLA’s collision-heavy failures.The aggregate failure pattern is consistent with NWM’s open-loop CEM rollout, OmniVLA’s short direct trajectories, and NavWAM’s closed-loop joint supervision.Source: §C.3 · pp. 19–20, Table S6 · p. 19

C.4.1 Real-Robot Future-View Predictions

  • NavWAM produces coherent real-robot future views aligned with the goal, whereas NWM’s zero-shot predictions fail to remain consistent across office, hallway, and meeting-room scenes.The predictions correspond to the required motion, including forward movement, right turns, and left turns.Source: §C.4.1 · p. 20, Figure S2 · p. 20

C.4.2 Real-Robot Value Trajectory

  • The predicted value drops when the goal object leaves view and recovers after reorientation, indicating goal-conditioned changes during single-pass closed-loop execution.The recovery is generated by repeated model evaluation on post-action observations rather than teacher forcing or external annotation.Source: §C.4.2 · pp. 20–21, Figure S3 · p. 21

D Limitations and Broader Impact

  • The value trajectory should be interpreted qualitatively because Eq. 5 calibrates it to full-trajectory distances and compresses its absolute range on short real-robot rollouts.NavWAM’s broader deployment evidence is limited to the evaluated real-robot settings and should not be read as universal validation.Source: §C.4.2 · p. 21, D Limitations and Broader Impact · p. 21

D.1 Failure Mechanisms

NavWAM can fail when physical pose drift moves the robot off its predicted trajectory or when future-view prediction degrades near obstacles and ambiguous scenes.

D.1 Failure Mechanisms

  • Pose drift can push the robot off its predicted trajectory, causing inconsistent future views, drifting action chunks, and eventual collision.The representative failure attributes the deviation to over-rotation likely caused by wheel friction.Source: §D.1 · p. 21, Figure S4 · p. 22
  • Near obstacles or in visually ambiguous regions, degraded future-view prediction can cause the subsequent action chunk to drift with it.The failure mechanism compounds because the model then acts from an unreliable predicted future.Source: §D.1 · pp. 21–22
  • Dynamic obstacles and long-horizon navigation remain unverified because deployment covered static scenes and short- to medium-range goals within one room or corridor.Multi-room or multi-floor navigation requiring many replanning chunks was not evaluated.Source: §D.1 · p. 22

D.2 Detailed Limitations

The detailed limitations constrain NavWAM’s evaluation breadth, real-world scale, inference efficiency, value comparability, head-design exploration, and deployment safety.

D.2 Detailed Limitations

  • Evaluation focuses on image-goal navigation, leaving language-goal, object-goal, instruction-following, and embodied-question-answering settings unevaluated.Source: §D.2 · p. 22
  • Real-world validation covers 24 episodes across four indoor environments on one robot platform, leaving broader platform and environment diversity for future work.Different cameras and control rates are not covered.Source: §D.2 · p. 22
  • NavWAM’s single-pass inference cost is dominated by diffusion-chain length, while aggressive step reduction and its accuracy-latency trade-off remain unstudied.Source: §D.2 · p. 22
  • The value frame is calibrated to indoor fine-tuning trajectories, so absolute values are not directly comparable across datasets with different scales.Source: §D.2 · p. 22
  • The study uses direct readout from denoised action and value frames and does not further explore separate MLP heads because they increased complexity without improving results.Source: §D.2 · p. 22
  • Released checkpoints and code are for research use and are not safety-certified for autonomous operation around people or in safety-critical environments.Real-world deployment used operator supervision, a time budget, and a hardware safety stop.Source: §D.2 · p. 22

Related papers

Loading 2606.13494v1…