PaperlayerView exact revision on arXiv

Source-linked AI summary

MWM: Mobile World Models for Action-Conditioned Consistent Prediction

Han Yan, Zishang Xiang, Zeyu Zhang, Hao Tang

arXiv:2603.07799v1cs.CVcs.RO

TL;DR

MWM targets action-conditioned drift and training–inference mismatch in navigation world models. It combines structure pretraining, ACC post-training, and ICSD for consistency-preserving few-step diffusion. Across benchmark and real-world tasks, it reports gains in visual fidelity, trajectory accuracy, navigation success, and inference efficiency.

  • Problem

    Existing navigation world models can produce visually plausible rollouts that drift under multi-step action conditioning, while few-step distillation does not explicitly preserve rollout consistency.

    Source: §I · pp. 1–2

  • Method

    MWM combines structure pretraining with Action-Conditioned Consistency post-training and Inference-Consistent State Distillation for few-step diffusion inference.

    Source: §I · p. 2

  • Results

    MWM reports consistent gains across benchmark and real-world tasks, including 20.4% lower DreamSim, 17.5% lower FID, and at least 4× faster inference.

    Source: §I · p. 2, B. Main Results · pp. 5–6

  • Takeaways & Limitations

    MWM can serve as an interface between visual prediction and navigation control by improving the practical utility of imagined rollouts for downstream planning.

    Source: §V · p. 7

Abstract

from arXiv · show

World models enable planning in imagined future predicted space, offering a promising framework for embodied navigation. However, existing navigation world models often lack action-conditioned consistency, so visually plausible predictions can still drift under multi-step rollout and degrade planning. Moreover, efficient deployment requires few-step diffusion inference, but existing distillation methods do not explicitly preserve rollout consistency, creating a training-inference mismatch. To address these challenges, we propose MWM, a mobile world model for planning-based image-goal navigation. Specifically, we introduce a two-stage training framework that combines structure pretraining with Action-Conditioned Consistency (ACC) post-training to improve action-conditioned rollout consistency. We further introduce Inference-Consistent State Distillation (ICSD) for few-step diffusion distillation with improved rollout consistency. Our experiments on benchmark and real-world tasks demonstrate consistent gains in visual fidelity, trajectory accuracy, planning success, and inference efficiency. Code: https://github.com/AIGeeksGroup/MWM. Website: https://aigeeksgroup.github.io/MWM.

I. INTRODUCTION

MWM addresses action-conditioned rollout drift and training–inference mismatch in navigation world models through two-stage consistency training and inference-consistent distillation.

  • World models support navigation planning by generating future visual rollouts that can be scored against goal observations or task requirements.Source: §I · p. 1
  • Visually realistic predictions may still diverge from real trajectories under the same action sequence, undermining model predictive control.Source: §I · pp. 1–2
  • MWM combines structure pretraining with Action-Conditioned Consistency post-training to reduce compounding rollout error while preserving visual fidelity.Source: §I · p. 2
  • Inference-Consistent State Distillation extends consistency training to few-step diffusion inference by aligning truncated training estimates with inference-time states.Source: §I · p. 2
  • 20.4% lower DreamSim, 17.5% lower FID, and at least 4× faster inference are reported alongside trajectory and real-world navigation gains.Benchmark results include 10.9% ATE and 8.5% RPE improvements; real-world deployment reports 50% relative success-rate improvement and 32.1% lower navigation error.Source: §I · p. 2

II. RELATED WORK

Prior work develops visual navigation and diffusion-based world models, while MWM focuses on making imagined rollouts both consistent with actions and efficient enough for online planning.

  • Visual navigation research progressed from handcrafted features and metric maps toward end-to-end, goal-conditioned, and transferable learned policies.Source: §II · p. 2
  • Diffusion world models provide high-fidelity visual generation but standard sampling requires hundreds of denoising steps, limiting real-time model predictive control.Source: §II · p. 3
  • MWM uses structure-first training and consistency refinement to preserve detailed scene dynamics while improving autoregressive rollout behavior.Source: A. Overview · p. 3
  • Planning uses CEM-based search in the world-model rollout space.Source: A. Overview · p. 3

B. Two-Stage Training Pipeline For MWM

Stage I pretrains MWM as an action-conditioned diffusion model that denoises future representations from current state–action context, using a structure-preserving architecture.

2) Stage II: Action-Conditioned Consistency (ACC) Post-training:

Stage II trains MWM on self-generated rollout contexts with observation-level consistency supervision, while selectively adapting modulation layers to improve multi-step alignment.

Stage II: Action-Conditioned Consistency (ACC) Post-training:

C. Inference-Consistent State Distillation (ICSD)

ICSD aligns truncated training-time diffusion states with inference-time denoising states, enabling few-step diffusion while improving multi-step rollout consistency.

ICSD design

  • ICSD introduces an inference-consistent state within ACC post-training to support low-step diffusion self-distillation without a separate training stage or objective.It distills at the level of action-conditioned consistency rather than only matching outputs.Source: C. Inference-Consistent State… · p. 4
  • The model injects diffusion timestep and action conditions through the same AdaLN modulation network, while Stage II optimizes only the modulation layers.The updated timestep conditioning facilitates few-step inference.Source: C. Inference-Consistent State… · p. 4
  • Truncated and skip-step denoising states can become overly smooth or blurred and mismatch the inference-time endpoint.ICSD explicitly bridges this discrepancy.Source: C. Inference-Consistent State… · p. 4
  • ICSD uses a deterministic DDIM update to align the truncated training state with the inference-time denoising endpoint.This reduces truncation-induced train–test mismatch.Source: C. Inference-Consistent State… · pp. 4–5

Effect

D. Planning with MWM

MWM formulates image-goal navigation as model-predictive control over world-model rollouts, using CEM to optimize action sequences against a terminal-frame perceptual objective.

Planning formulation

  • Navigation is formulated as MPC, with CEM optimizing action sequences in the world-model rollout space.The trajectory evaluator can score language alignment or obstacles, but this work uses a terminal-frame perceptual objective.Source: D. Planning with MWM · p. 5

Planning objective

  • The terminal score compares the decoded rolled-out terminal latent with the goal image using negative LPIPS.CEM samples action sequences, ranks them by score, and updates the action distribution using top-k elites.Source: D. Planning with MWM · p. 5

A. Experimental Settings

Experiments evaluate MWM on SCAND using multimodal robot demonstrations, specified training settings, and benchmarks covering rollout consistency, visual fidelity, efficiency, and navigation.

Datasets

  • SCAND contains 8.7 hours from 138 trajectories spanning 25 miles, with multimodal sensing and joystick commands from two mobile robot platforms.The data cover diverse indoor and outdoor university environments.Source: A. Experimental Settings · p. 5

Implementation

  • ACC post-training runs for 1.5K optimization steps on one RTX PRO 6000 GPU using eight-step rollout segments.Increasing the post-training budget beyond this yields negligible additional gains.Source: A. Experimental Settings · p. 5

Main results

  • MWM achieves lower LPIPS and DreamSim than NWM across rollout horizons on SCAND, including under accelerated DDIM inference.MWM with DDIM 5 outperforms NWM with DDIM 25 while reducing denoising steps by at least 80%.Source: B. Main Results · p. 5, Table I · p. 6
  • MWM provides at least a 4× speedup while achieving lower FID than NWM with DDIM 25.NWM with DDIM 5 suffers a substantial generation-quality drop.Source: B. Main Results · p. 5, Table II · p. 6, Table III · p. 6
  • MWM achieves the lowest ATE and RPE on SCAND under CEM-based goal-image planning and reaches state-of-the-art navigation performance.Qualitative predictions remain temporally coherent and aligned with executed actions across rollout steps.Source: B. Main Results · pp. 5–6, Table IV · p. 6

C. Ablation Studies

Ablations support the design of ACC, the two-stage training paradigm, and ICSD for improving rollout consistency and inference alignment.

ACC loss design

  • LPIPS-based ACC consistently outperforms pixel-wise L1 and L2 objectives across all three SCAND metrics.L2 performs worst among the compared ACC loss choices.Source: C. Ablation Studies · p. 6

Training paradigm

  • The two-stage structure-training-plus-ACC pipeline achieves the best overall MMK2-RealNav performance, while ACC-only performs worst.Structure-only training falls between the two-stage and ACC-only strategies.Source: C. Ablation Studies · p. 6, Table VI · p. 6

Inference-consistent state

  • Without ICSD, ACC consistency supervision is undermined by mismatched intermediate states and the training-inference step-skipping gap.The ablation varies the context used during ICSD training on SCAND.Source: C. Ablation Studies · p. 6, Table VII · p. 6

D. RealWorld Evaluation

The real-world evaluation uses the MMK2 platform for RGB-only indoor goal-image navigation across varied target locations and viewpoints.

Robot setup

  • Real-world evaluation uses the AIRBOT Mobile Manipulation Kit 2 with RGB-only observations for navigation.The platform includes onboard vision sensors and LiDAR, but the navigation policies do not use additional sensing modalities.Source: 1) Robot Setup · pp. 6–7

Deployment process

  • The real-world setup records RGB observations with a head-mounted camera and executes returned actions on the robot base in open loop.Inference runs on a remote RTX PRO 6000 server.Source: 1) Robot Setup · p. 7

Scene setup and task types

  • Experiments cover cabinet, window, pillar, and corridor goals from different starting viewpoints in an indoor university building.The setup assesses robustness to viewpoint changes and error accumulation under real-world sensing and actuation noise.Source: 2) Scene setup and task types · p. 7

Quantitative evaluation

  • The evaluation aggregates quantitative real-world goal-image navigation results over four targets.The supplied table caption identifies the aggregation scope but does not provide the numerical results.Source: Table VIII · p. 7

3) Quantitative real-world evaluation:

MWM’s real-world planned rollouts align better with observations than NWM, supporting improved downstream planning utility, while deployment remains open-loop and one-shot.

Qualitative real-world evaluation

  • MWM produces planned rollouts more consistent with real observations than NWM in qualitative real-world evaluation.The improved alignment indicates reduced rollout error accumulation and supports stronger goal-directed planning and execution.Source: 4) Qualitative real-world evaluation · p. 7

Conclusions

  • Combining two-stage training with consistency-oriented few-step distillation improves prediction quality and the practical utility of imagined rollouts for navigation planning.The paper reports this conclusion across benchmark and real-world tasks.Source: §V · p. 7
  • MWM currently performs open-loop, one-shot planning without online replanning from newly observed feedback.Future work targets real-time closed-loop navigation in dynamic and uncertain environments.Source: §V · p. 8

Related papers

Loading 2603.07799v1…