PaperlayerView exact revision on arXiv

Source-linked AI summary

MWM: Mobile World Models for Action-Conditioned Consistent Prediction

Han Yan, Zishang Xiang, Zeyu Zhang, Hao Tang

arXiv:2603.07799v1cs.CVcs.RO

TL;DR

MWM targets action-conditioned drift and training–inference mismatch in navigation world models. It combines structure pretraining, ACC post-training, and ICSD for consistency-preserving few-step diffusion. Across benchmark and real-world tasks, it reports gains in visual fidelity, trajectory accuracy, navigation success, and inference efficiency.

  • Problem

    Existing navigation world models can produce visually plausible rollouts that drift under multi-step action conditioning, while few-step distillation does not explicitly preserve rollout consistency.

  • Method

    MWM combines structure pretraining with Action-Conditioned Consistency post-training and Inference-Consistent State Distillation for few-step diffusion inference.

  • Results

    MWM reports consistent gains across benchmark and real-world tasks, including 20.4% lower DreamSim, 17.5% lower FID, and at least 4× faster inference.

  • Takeaways & Limitations

    MWM can serve as an interface between visual prediction and navigation control by improving the practical utility of imagined rollouts for downstream planning.

Abstract

from arXiv · show

World models enable planning in imagined future predicted space, offering a promising framework for embodied navigation. However, existing navigation world models often lack action-conditioned consistency, so visually plausible predictions can still drift under multi-step rollout and degrade planning. Moreover, efficient deployment requires few-step diffusion inference, but existing distillation methods do not explicitly preserve rollout consistency, creating a training-inference mismatch. To address these challenges, we propose MWM, a mobile world model for planning-based image-goal navigation. Specifically, we introduce a two-stage training framework that combines structure pretraining with Action-Conditioned Consistency (ACC) post-training to improve action-conditioned rollout consistency. We further introduce Inference-Consistent State Distillation (ICSD) for few-step diffusion distillation with improved rollout consistency. Our experiments on benchmark and real-world tasks demonstrate consistent gains in visual fidelity, trajectory accuracy, planning success, and inference efficiency. Code: https://github.com/AIGeeksGroup/MWM. Website: https://aigeeksgroup.github.io/MWM.

I. INTRODUCTION

MWM addresses action-conditioned rollout drift and training–inference mismatch in navigation world models through two-stage consistency training and inference-consistent distillation.

  • World models support navigation planning by generating future visual rollouts that can be scored against goal observations or task requirements.
  • Visually realistic predictions may still diverge from real trajectories under the same action sequence, undermining model predictive control.
  • MWM combines structure pretraining with Action-Conditioned Consistency post-training to reduce compounding rollout error while preserving visual fidelity.
  • Inference-Consistent State Distillation extends consistency training to few-step diffusion inference by aligning truncated training estimates with inference-time states.
  • 20.4% lower DreamSim, 17.5% lower FID, and at least 4× faster inference are reported alongside trajectory and real-world navigation gains.Benchmark results include 10.9% ATE and 8.5% RPE improvements; real-world deployment reports 50% relative success-rate improvement and 32.1% lower navigation error.

II. RELATED WORK

Prior work develops visual navigation and diffusion-based world models, while MWM focuses on making imagined rollouts both consistent with actions and efficient enough for online planning.

  • Visual navigation research progressed from handcrafted features and metric maps toward end-to-end, goal-conditioned, and transferable learned policies.
  • Diffusion world models provide high-fidelity visual generation but standard sampling requires hundreds of denoising steps, limiting real-time model predictive control.
  • MWM uses structure-first training and consistency refinement to preserve detailed scene dynamics while improving autoregressive rollout behavior.
  • Planning uses CEM-based search in the world-model rollout space.

B. Two-Stage Training Pipeline For MWM

Stage I pretrains MWM as an action-conditioned diffusion model that denoises future representations from current state–action context, using a structure-preserving architecture.

  • Structure pretraining uses teacher forcing, providing the ground-truth next representation while conditioning denoising on the current representation and action.
  • A noised target is formed by sampling a diffusion timestep from the schedule.
  • The denoiser receives the noised next representation, current representation, action, and timestep, then predicts a clean target.
  • Training optimizes reconstruction loss over randomly sampled diffusion timesteps.
  • Actions and timesteps enter each CDiT block through AdaLN, while the current context attends to a memory of past states.

2) Stage II: Action-Conditioned Consistency (ACC) Post-training:

Stage II trains MWM on self-generated rollout contexts with observation-level consistency supervision, while selectively adapting modulation layers to improve multi-step alignment.

  • Stage II: Action-Conditioned Consistency (ACC) Post-training:: ACC exposes the model to its own autoregressive predictions, matching the self-conditioned context distribution encountered during deployment.
  • Stage II: Action-Conditioned Consistency (ACC) Post-training:: Rollouts are generated autoregressively with randomly selected truncation denoising steps, and gradients are enabled only at the final truncation prediction.
  • Stage II: Action-Conditioned Consistency (ACC) Post-training:: The pipeline freezes the CDiT backbone and updates only AdaLN modulation layers to improve rollout alignment without degrading pretrained structure and appearance.
  • Stage II: Action-Conditioned Consistency (ACC) Post-training:: The consistency loss supervises multi-frame rollouts against ground-truth observations using an averaged perceptual loss based on LPIPS.
  • Stage II: Action-Conditioned Consistency (ACC) Post-training:: Reducing the self-conditioning gap yields more reliable terminal predictions for trajectory evaluation and navigation performance.

C. Inference-Consistent State Distillation (ICSD)

ICSD aligns truncated training-time diffusion states with inference-time denoising states, enabling few-step diffusion while improving multi-step rollout consistency.

  • ICSD design: ICSD introduces an inference-consistent state within ACC post-training to support low-step diffusion self-distillation without a separate training stage or objective.It distills at the level of action-conditioned consistency rather than only matching outputs.
  • ICSD design: The model injects diffusion timestep and action conditions through the same AdaLN modulation network, while Stage II optimizes only the modulation layers.The updated timestep conditioning facilitates few-step inference.
  • ICSD design: Truncated and skip-step denoising states can become overly smooth or blurred and mismatch the inference-time endpoint.ICSD explicitly bridges this discrepancy.
  • ICSD design: ICSD uses a deterministic DDIM update to align the truncated training state with the inference-time denoising endpoint.This reduces truncation-induced train–test mismatch.
  • Effect: The aligned state improves multi-step rollout consistency under few-step sampling.The qualitative results show predicted frames that remain consistent with ground-truth frames under action-conditioned rollout.

D. Planning with MWM

MWM formulates image-goal navigation as model-predictive control over world-model rollouts, using CEM to optimize action sequences against a terminal-frame perceptual objective.

  • Planning formulation: Navigation is formulated as MPC, with CEM optimizing action sequences in the world-model rollout space.The trajectory evaluator can score language alignment or obstacles, but this work uses a terminal-frame perceptual objective.
  • Planning objective: The terminal score compares the decoded rolled-out terminal latent with the goal image using negative LPIPS.CEM samples action sequences, ranks them by score, and updates the action distribution using top-k elites.

A. Experimental Settings

Experiments evaluate MWM on SCAND using multimodal robot demonstrations, specified training settings, and benchmarks covering rollout consistency, visual fidelity, efficiency, and navigation.

  • Datasets: SCAND contains 8.7 hours from 138 trajectories spanning 25 miles, with multimodal sensing and joystick commands from two mobile robot platforms.The data cover diverse indoor and outdoor university environments.
  • Implementation: ACC post-training runs for 1.5K optimization steps on one RTX PRO 6000 GPU using eight-step rollout segments.Increasing the post-training budget beyond this yields negligible additional gains.
  • Main results: MWM achieves lower LPIPS and DreamSim than NWM across rollout horizons on SCAND, including under accelerated DDIM inference.MWM with DDIM 5 outperforms NWM with DDIM 25 while reducing denoising steps by at least 80%.
  • Main results: MWM provides at least a 4× speedup while achieving lower FID than NWM with DDIM 25.NWM with DDIM 5 suffers a substantial generation-quality drop.
  • Main results: MWM achieves the lowest ATE and RPE on SCAND under CEM-based goal-image planning and reaches state-of-the-art navigation performance.Qualitative predictions remain temporally coherent and aligned with executed actions across rollout steps.

C. Ablation Studies

Ablations support the design of ACC, the two-stage training paradigm, and ICSD for improving rollout consistency and inference alignment.

  • ACC loss design: LPIPS-based ACC consistently outperforms pixel-wise L1 and L2 objectives across all three SCAND metrics.L2 performs worst among the compared ACC loss choices.
  • Training paradigm: The two-stage structure-training-plus-ACC pipeline achieves the best overall MMK2-RealNav performance, while ACC-only performs worst.Structure-only training falls between the two-stage and ACC-only strategies.
  • Inference-consistent state: Without ICSD, ACC consistency supervision is undermined by mismatched intermediate states and the training-inference step-skipping gap.The ablation varies the context used during ICSD training on SCAND.

D. RealWorld Evaluation

The real-world evaluation uses the MMK2 platform for RGB-only indoor goal-image navigation across varied target locations and viewpoints.

  • Robot setup: Real-world evaluation uses the AIRBOT Mobile Manipulation Kit 2 with RGB-only observations for navigation.The platform includes onboard vision sensors and LiDAR, but the navigation policies do not use additional sensing modalities.
  • Deployment process: The real-world setup records RGB observations with a head-mounted camera and executes returned actions on the robot base in open loop.Inference runs on a remote RTX PRO 6000 server.
  • Scene setup and task types: Experiments cover cabinet, window, pillar, and corridor goals from different starting viewpoints in an indoor university building.The setup assesses robustness to viewpoint changes and error accumulation under real-world sensing and actuation noise.
  • Quantitative evaluation: The evaluation aggregates quantitative real-world goal-image navigation results over four targets.The supplied table caption identifies the aggregation scope but does not provide the numerical results.

3) Quantitative real-world evaluation:

MWM’s real-world planned rollouts align better with observations than NWM, supporting improved downstream planning utility, while deployment remains open-loop and one-shot.

  • Qualitative real-world evaluation: MWM produces planned rollouts more consistent with real observations than NWM in qualitative real-world evaluation.The improved alignment indicates reduced rollout error accumulation and supports stronger goal-directed planning and execution.
  • Conclusions: Combining two-stage training with consistency-oriented few-step distillation improves prediction quality and the practical utility of imagined rollouts for navigation planning.The paper reports this conclusion across benchmark and real-world tasks.
  • Conclusions: MWM currently performs open-loop, one-shot planning without online replanning from newly observed feedback.Future work targets real-time closed-loop navigation in dynamic and uncertain environments.
Loading 2603.07799v1…