PaperlayerView exact revision on arXiv

Source-linked AI summary

ABot-M0.5: Unified Mobility-and-Manipulation World Action Model

Ronghan Chen, Yandan Yang, Zuojin Tang, Dongjie Huo, Tong Lin, Haoning Wu, Haoyun Liu, Yuzhi Chen, Lulu Zheng, Botai Yuan, Tianlun Li, Mingxin Wang, Dekang Qi, Bin Hu, Wei Mei, Yuze Xuan, Haolong Yang, Yanqing Zhu, Mu Xu, Zhiheng Ma, Xinyuan Chang

arXiv:2607.00678v2cs.CVcs.RO

TL;DR

Current embodied learning methods lack world modeling or are not structured to match mobile manipulation. ABot-M0.5 combines intermediate latent actions, a dual-level Mixture-of-Transformers architecture, and Dream Forcing, achieving state-of-the-art performance across long-horizon mobile and fine-grained manipulation tasks.

  • Problem

    Existing embodied learning methods lack explicit world modeling or do not align with mobile manipulation’s demands, including bridging coarse video latents and fine-grained executable actions.

    Source: §1 · p. 3, §2.1 · p. 5

  • Method

    ABot-M0.5 uses intermediate latent actions, a dual-level Mixture-of-Transformers architecture, and Dream Forcing to align temporal granularity, action structure, and inverse-dynamics training with autoregressive inference.

    Source: §1 · p. 4

  • Results

    Across RoboCasa365, RoboTwin 2.0, LIBERO/LIBERO-Plus, and real-world tasks, ABot-M0.5 achieves state-of-the-art performance in long-horizon mobile and fine-grained manipulation, surpassing leading VLA and WAM baselines.

    Source: §6 · p. 27

  • Takeaways & Limitations

    The results indicate that extending WAMs from stationary to mobile manipulation has a practical path in aligning world modeling, action abstraction, and deployment-time behavior.

    Source: §1 · p. 4

    Validation in complex, unstructured real-world mobile manipulation beyond controlled lab settings remains future work.

    Source: §6 · p. 27

Abstract

from arXiv · show

Mobile manipulation is a key capability for general-purpose robots, yet remains challenging for current embodied learning methods. VLA policies are typically reactive and lack explicit world modeling, while existing World Action Models (WAMs) are still poorly aligned with the structure of mobile manipulation: they operate on coarse video chunks, model entangled navigation-manipulation actions, and train inverse dynamics under supervision that does not match autoregressive inference. As a result, they often miss fine-grained contact dynamics, suffer from action-distribution conflicts, and accumulate errors over long-horizon rollouts. We propose ABot-M0.5, a new WAM built on the insight that mobile manipulation requires alignment at three levels: temporal granularity, action space, and train-test consistency. To align temporal granularity, we introduce intermediate latent actions that capture local visual state transitions and serve as an bridging action space between video latents and embodiment-specific controls. To align action space, we design a dual-level Mixture-of-Transformers architecture that disentangles both modality representations and heterogeneous action subspaces such as base movement and arm manipulation. To align inference conditions, we propose the dream-forcing training strategy that progressively trains inverse dynamics on model-predicted videos, improving train-test alignment and robustness during autoregressive prediction. Experiments on challenging mobile and fine-grained manipulation benchmarks demonstrate that ABot-M0.5 achieves state-of-the-art performance in both long-horizon task success and finegrained control accuracy. These results highlight the critical importance of granularity-aligned, action-disentangled, and inference-consistent world-action modeling.

1 Introduction

ABot-M0.5 addresses mobile manipulation by aligning temporal granularity, action structure, and training with inference through a unified world-action model. Experiments show strong performance on long-horizon and fine-grained manipulation tasks.

1 Introduction

  • Reactive VLA policies lack explicit world modeling and long-horizon memory, while existing world-model methods remain structurally mismatched to mobile manipulation [4] [5] [29] [53] [54] [3] [20] [21] [73].The paper identifies temporal-granularity, action-structure, and training-context mismatches as the core bottlenecks.Source: §1 · pp. 3–4
  • Coarse video can blur contact dynamics, navigation and manipulation have different dynamics yet are often entangled, and deployment on predicted observations creates train-test mismatch and rollout errors [21] [5] [6] [20] [21] [33] [73].The cited examples include contact onset, grasp closure, alignment correction, low-frequency base motion, and high-frequency arm control.Source: §1 · pp. 3–4
  • Frame-level latent actions capture local visual transitions between video latents and robot actions, while a dual-level Mixture-of-Transformers separates modalities and base-motion and arm-manipulation action subspaces.The intermediate actions factorize video-to-action mapping and are less tied to a specific embodiment.Source: §1 · p. 4
  • Dream Forcing trains inverse dynamics on self-dreamed videos, aligning training context with inference and improving robustness to prediction errors.This directly exposes inverse-dynamics learning to model-predicted observations.Source: §1 · p. 4
  • Experiments show strong performance on challenging mobile-manipulation and manipulation benchmarks, with gains in long-horizon success and fine-grained accuracy; ablations validate each component.The experiments point to alignment of world modeling, action abstraction, and deployment-time behavior as a path for extending WAMs from stationary to mobile manipulation.Source: §1 · p. 4

2 Alignment-Aware World-Action Learning

Mobile manipulation requires world modeling, action generation, and deployment-time rollout to be aligned, rather than treating it as a longer version of stationary manipulation. The section frames ABot-M0.5 around three structural mismatches: temporal granularity, heterogeneous action structure, and rollout conditions.

2.1 Problem Setting

  • Mobile manipulation combines language-conditioned, long-horizon navigation and object interaction, requiring executable actions consistent with future world evolution.Reactive policies can become brittle because future decisions depend on how the environment evolves under the robot’s own actions; WAMs explicitly model future observations and actions.Source: §2.1 · p. 5
  • Coarse video latents and fine-grained executable actions have a granularity and semantic gap, motivating intermediate motion intents that capture local visual transitions.The proposed hierarchy anticipates global visual evolution, distills it into frame-level intents, then grounds those intents in embodiment-specific controls.Source: §2.1 · p. 5

2.2 Core Bottlenecks in Mobile Manipulation

  • Coarse chunk-level world prediction can omit brief interaction transitions, including grasp closure, contact onset, release, fine alignment, and local collision avoidance.Robot controls may be needed at every frame or control step even when future video latents summarize multiple frames.Source: §2.2 · p. 6
  • Entangling base mobility with arm manipulation combines low-frequency global motion and high-frequency contact-sensitive control, increasing optimization difficulty and weakening compositionality.The two action types must coordinate while remaining structurally distinct.Source: §2.2 · p. 6
  • Inverse dynamics trained on ground-truth futures faces imperfect self-generated visual rollouts at inference, creating exposure bias that can compound and derail long-horizon execution [2] [56].Predicted futures may contain noise, uncertainty, blurring, object drift, or hallucinated content.Source: §2.2 · p. 6
  • ABot-M0.5 addresses the three mismatches with latent actions, a dual-level Mixture-of-Transformers, and Dream-Forcing for rollout-aligned inverse dynamics learning.These components target temporal alignment, structured action modeling, and inference-consistent training, respectively.Source: §2.2 · p. 6

3 The ABot-M0.5 Model

ABot-M0.5 models mobile manipulation as a cascade from future visual dynamics to frame-level motion intents and embodiment-specific actions. Its design aligns temporal granularity, action structure, and training conditions with autoregressive inference.

3.1 Overall Architecture and Notation

  • ABot-M0.5 factorizes prediction into world modeling, motion abstraction, and control generation, using Conditional Flow Matching across stages with conditioning that follows inference order.Video prediction uses historical states and language; later stages condition on previously generated variables, aligning training-time information flow with autoregressive generation.Source: §3.1 · p. 7

3.2 Intermediate Latent Action Modeling

  • Frame-level, embodiment-agnostic latent actions bridge coarse video latents and executable controls, refining predicted world dynamics into fine-grained motion intents before hardware-specific action generation.The hierarchy separates embodiment-agnostic physical priors from hardware-specific kinematics, supporting generalization across heterogeneous robot platforms.Source: §3.2 · p. 8

3.3 Dual-Level Mixture-of-Transformers

  • Dual-level Mixture-of-Transformers separates modality representations and mobility versus manipulation action subspaces while retaining coordinated reasoning through shared attention.Each modality has dedicated projections, timestep embeddings, and output heads; action subspaces use separate FFNs and prediction heads. Subspace-aware CFM uses teacher-forced upstream representations and a shared denoising timestep for both branches.Source: §3.3 · pp. 9–10

3.4 Dream Forcing for Train-Test Aligned Action Prediction

  • Dream Forcing trains action prediction on the model’s own dreamed video latents, exposing it to imperfect visual states like those encountered at inference.Teacher forcing uses unavailable ground-truth future latents, while diffusion forcing introduces independently noised contexts that are difficult to match to inference; Dream Forcing replaces future conditioning latents with self-dreamed counterparts.Source: §3.4 · pp. 11–12

4 Training Paradigm

ABot-M0.5 trains progressively from large-scale world modeling through self-supervised latent-action learning to supervised action alignment, with system optimizations supporting long-sequence video-action training.

4.1 Pretraining Data

  • Pretraining combines public robot corpora and synthetic data for embodiment, task, and scene diversity, while unlabeled video extends supervision for visual motion abstraction.Galaxea includes coordinated base-mobility tasks, and InternData-A1 adds synthetic scene and embodiment variation.Source: §4.1 · p. 13

4.2 World Model Pretraining

  • Initialized from Wan2.2 5B [66], the autoregressive world model is fine-tuned on robot video; four semantic view slots separate third-person scene views from wrist interaction views.The pretrained dynamics provide future visual predictions for action learning, especially important when mobile manipulation changes scenes and viewpoints.Source: §4.2 · pp. 13–14

4.3 Latent Action Model Pretraining

  • Self-supervised latent actions encode visual transitions between frames; triplet-based additive and reversal consistency structure the motion space, and the frozen encoder supplies offline trajectory labels.This fine-grained intermediate representation transfers motion knowledge from unlabeled videos beyond action-annotated robot datasets.Source: §4.3 · pp. 14–15

4.4 Progressive Supervised Fine-Tuning

  • Supervised fine-tuning first predicts actions using ground-truth future video latents, shielding inverse dynamics from early prediction errors while the world model adapts to downstream data.This stabilizes initial joint learning of future video, latent actions, and executable actions.Source: §4.4 · p. 15
  • Dream forcing then conditions action prediction on model-generated futures, training inverse dynamics to tolerate prediction noise and align with autoregressive rollout.This progressive transition aligns action-learning conditions with deployment and targets stable long-horizon performance under generated futures.Source: §4.4 · p. 16

4.5 Efficient Structured Attention and Latent Augmentation

  • Approximately 5× speedup in combined forward-backward passes comes from FlashAttention-based structured attention; offset-based latent augmentation also expands valid segmentations H-fold.The attention implementation is mathematically equivalent to the structured pattern while reducing kernel overhead, padding, and memory; offset-based indexing improves temporal diversity and robustness to timing variations.Source: §4.5 · p. 17

5 Experiments

ABot-M0.5 is evaluated on mobile, tabletop, and real-world manipulation, with ablations testing its alignment and training components. The reported results show strong long-horizon mobile performance, benefits beyond mobile settings, and successful real-world deployment.

5.1 Experimental Setup

  • Experiments evaluate long-horizon mobile household manipulation on RoboCasa365, bimanual manipulation under clean and randomized scene settings on RoboTwin 2.0, multi-task and long-horizon tabletop manipulation on LIBERO/LIBERO-Plus, and real-task transfer beyond simulation.Real-world experiments use a single-arm, 6-DoF Agilex Piper and 50 real-world demonstrations per task.Source: §5.1 · pp. 18–19, §5.5 · p. 25

5.2 Main Results on Mobile Manipulation

  • On RoboCasa365, ABot-M0.5 achieves state-of-the-art performance, with particularly clear gains on long-horizon composite tasks and coherent navigation-to-manipulation rollouts.The paper attributes these gains to temporal alignment, action decoupling, and Dream-Forcing robustness during long rollouts.Source: §5.2 · p. 19, Table 2 · p. 20

5.3 Results on Manipulation Benchmarks

  • On RoboTwin 2.0 and LIBERO / LIBERO-Plus, ABot-M0.5 performs strongly across clean and randomized scenes and achieves leading or competitive results on compositional tabletop tasks.The authors interpret these results as evidence that the approach benefits fine-grained manipulation and generalizes beyond mobility-heavy settings.Source: §5.3 · pp. 20–21, Table 4 · p. 21, Table 5 · p. 22

5.4 Ablation Studies

  • Action-decoupled MoT scores 0.48 versus 0.34 for Modality-level MoT; Dream-Forcing reaches 70.56% success after 5k steps, versus 67.55% at warm start and 66.78% with teacher forcing.Figure 10 shows faster convergence for action-decoupled MoT; Dream-Forcing’s 70.56% is an absolute 3.01% improvement over warm start, while teacher forcing reaches 68.90% only after 10k steps.Source: Figure 10 · p. 22, §5.4 · p. 23, Table 8 · p. 24
  • On RoboTwin 2.0 Clean, 3-Stage Separate reaches 94.0% success rate versus 87.60% for the naive baseline, 91.06% for Channel Concat, and 90.86% for 2-Stage Separate.The structured attention mask enforces strict causality: video cannot attend to latent actions, while actions can condition on latent actions. With pdrop = 0, success is 94.0% versus 91.06% at pdrop = 0.2.Source: §5.4 · pp. 21–22, Table 7 · p. 24
  • On RoboCasa365 Target 10%, pretrained ABot-M0.5 plus SFT reaches 49.0% success versus 17.8% for Wan2.2 plus SFT, a 31.2-point gap.Pretraining supplies transferable visual and interaction priors; SFT adapts them. Pretrained+SFT also produces focused attention on target objects and manipulation regions across views.Source: §5.4 · pp. 24–25, Figure 11 · p. 24

5.5 Real-World Experiments

  • On Peg Cylinder, ABot-M0.5 scores 70% success and 96% process, versus π0.5 at 50% and 90% and Fast-WAM at 30% and 77%.Across Organize Plate, Arrange Fruits, Cup Stacking, and Arrange Flower, success rates are 70%, 80%, 80%, and 60%, with process scores above 88%; Fast-WAM reaches only 20% to 40% success on these tasks.Source: Figure 12 · p. 25, §5.5 · pp. 26–27
  • The same model performs well on fine-grained and long-horizon real-world manipulation without task-specific changes, with reliable target localization and precise insertion under language guidance.The authors attribute robust deployment to Dual-level MoT separating modalities and action distributions, and Dream-Forcing reducing the training-inference gap.Source: §5.5 · pp. 26–27

6 Conclusion and Future Work

ABot-M0.5 combines intermediate latent actions, a dual-level Mixture-of-Transformers, and Dream Forcing to align video, action spaces, and training with inference. The paper reports state-of-the-art performance across simulation and real-world tasks, and outlines plans to improve generalization, scaling, memory, and inference efficiency.

6 Conclusion and Future Work

  • ABot-M0.5 achieves state-of-the-art performance across RoboCasa365, RoboTwin 2.0, LIBERO/LIBERO-Plus, and real-world tasks, surpassing leading VLA and WAM baselines.The reported strengths are long-horizon mobile manipulation and fine-grained manipulation.Source: §6 · p. 27
  • The framework combines intermediate latent actions, a dual-level Mixture-of-Transformers, and Dream Forcing to address temporal granularity, action-space, and train-test mismatches.Dream Forcing conditions action learning on self-dreamed latents.Source: §6 · p. 27
  • Future work will extend evaluation to broader real-world data, diverse robot embodiments, and complex, unstructured mobile manipulation beyond controlled labs.The authors also plan to investigate how WAM task performance scales with model capacity, data volume, and compute budget.Source: §6 · p. 27
  • The authors plan efficient memory mechanisms for long-horizon context and faster inference through advanced decoding and hardware-aware acceleration for real-time edge-device execution.Source: §6 · p. 27

7 Contributions

The section assigns team members to data collection, modeling, evaluation, deployment, writing, challenge submission, and project leadership and advising.

7 Contributions

  • Data collection and standardization were assigned to Yandan Yang, Ronghan Chen, Yuzhi Chen, Haoyun Liu, and Dekang Qi; model and training to Ronghan Chen, Zuojin Tang, Tong Lin, and Yandan Yang.Source: §7 · p. 28
  • Post-training and evaluation were assigned to Zuojin Tang, Tianlun Li, Haoning Wu, Ronghan Chen, Tong Lin, Mingxin Wang, and Bin Hu.Source: §7 · p. 28
  • Dongjie Huo, Lulu Zheng, and Botai Yuan handled real-robot experiments and deployment; Yandan Yang, Ronghan Chen, Zuojin Tang, Dekang Qi, and Haoyun Liu handled writing.Source: §7 · p. 28
  • Challenge submission was assigned to Yanqing Zhu‡, Wei Mei, Yuze Xuan, Haolong Yang, and Dongjie Huo.Source: §7 · p. 28
  • Xinyuan Chang was project lead, and Mu Xu† and Zhiheng Ma were advisors.Source: §7 · p. 28

Related papers

Loading 2607.00678v2…