Source-linked AI summary
ABot-M0.5: Unified Mobility-and-Manipulation World Action Model
Ronghan Chen, Yandan Yang, Zuojin Tang, Dongjie Huo, Tong Lin, Haoning Wu, Haoyun Liu, Yuzhi Chen, Lulu Zheng, Botai Yuan, Tianlun Li, Mingxin Wang, Dekang Qi, Bin Hu, Wei Mei, Yuze Xuan, Haolong Yang, Yanqing Zhu, Mu Xu, Zhiheng Ma, Xinyuan Chang
TL;DR
Mobile manipulation models struggle to bridge coarse world predictions and fine-grained executable actions while maintaining long-horizon context. ABot-M0.5 addresses these gaps with intermediate latent actions, action-disentangled Transformers, and Dream Forcing, achieving state-of-the-art performance across long-horizon mobile and fine-grained manipulation tasks.
Problem
Existing embodied policies lack explicit world modeling, while world-action models do not adequately bridge coarse video latents and fine-grained heterogeneous robot actions for mobile manipulation.
Method
ABot-M0.5 combines intermediate latent actions, a dual-level Mixture-of-Transformers architecture, and Dream Forcing to align granularity, action structure, and train-test conditions.
Results
ABot-M0.5 achieves state-of-the-art performance across RoboCasa365, RoboTwin 2.0, LIBERO/LIBERO-Plus, and real-world tasks in long-horizon mobile and fine-grained manipulation.
Takeaways & Limitations
The results support granularity-aligned, action-disentangled, and inference-consistent world-action modeling for mobile manipulation.
Takeaways & Limitations
The authors identify the need to validate generalization beyond controlled laboratory settings using broader real-world data and more diverse robot embodiments.
Abstract
from arXiv · showhide
Mobile manipulation is a key capability for general-purpose robots, yet remains challenging for current embodied learning methods. VLA policies are typically reactive and lack explicit world modeling, while existing World Action Models (WAMs) are still poorly aligned with the structure of mobile manipulation: they operate on coarse video chunks, model entangled navigation-manipulation actions, and train inverse dynamics under supervision that does not match autoregressive inference. As a result, they often miss fine-grained contact dynamics, suffer from action-distribution conflicts, and accumulate errors over long-horizon rollouts. We propose ABot-M0.5, a new WAM built on the insight that mobile manipulation requires alignment at three levels: temporal granularity, action space, and train-test consistency. To align temporal granularity, we introduce intermediate latent actions that capture local visual state transitions and serve as an bridging action space between video latents and embodiment-specific controls. To align action space, we design a dual-level Mixture-of-Transformers architecture that disentangles both modality representations and heterogeneous action subspaces such as base movement and arm manipulation. To align inference conditions, we propose the dream-forcing training strategy that progressively trains inverse dynamics on model-predicted videos, improving train-test alignment and robustness during autoregressive prediction. Experiments on challenging mobile and fine-grained manipulation benchmarks demonstrate that ABot-M0.5 achieves state-of-the-art performance in both long-horizon task success and finegrained control accuracy. These results highlight the critical importance of granularity-aligned, action-disentangled, and inference-consistent world-action modeling.
1 Introduction
Mobile manipulation demands navigation, manipulation, long-horizon context, and precise interaction, but current methods suffer from structural mismatches. ABot-M0.5 addresses these through aligned latent actions, disentangled action modeling, and Dream Forcing, achieving strong benchmark results.
- Overview: The project lists code at https://github.com/amap-cvlab/ABot-Manipulation.
- Overview: ABot-M0.5 combines temporal alignment, action disentanglement, and train-test consistency for mobile manipulation.The overview identifies these three properties as the model’s organizing principles.
- Training pipeline: The three-stage pipeline consists of Pretrain, SFT1, and SFT2, with Dream-Forcing applied in the final stage.SFT2 trains inverse dynamics on model-predicted videos to improve robustness.
- Motivation: Mobile manipulation requires navigating clutter, maintaining long-horizon context, and executing precise interactions under changing viewpoints and scene dynamics.
- Core bottlenecks: Existing methods face mismatches in temporal granularity, heterogeneous action structure, and training context during autoregressive inference.These bottlenecks connect coarse video prediction, entangled mobility-manipulation controls, and ground-truth-conditioned training to deployment failures.
- Approach: ABot-M0.5 introduces intermediate latent actions, a dual-level Mixture-of-Transformers architecture, and Dream Forcing to address the three mismatches.
- Results: The model demonstrates strong results on challenging mobile manipulation and manipulation benchmarks, including long-horizon success and fine-grained manipulation accuracy.The paper also validates individual components through extensive ablations.
2 Alignment-Aware World-Action Learning
Alignment-aware world-action learning treats mobile manipulation as a joint problem of future modeling, heterogeneous control, and robust deployment. The paper motivates this view by showing why direct or entangled mappings fail.
- Problem setting: Mobile manipulation differs from stationary manipulation through larger viewpoint changes, heterogeneous mobility-and-manipulation actions, and greater rollout-robustness demands.
- Problem setting: The central gap is between coarse future video latents z_t+1:t+H and fine-grained executable actions a_t:t+H−1.Direct mapping is difficult because the spaces differ in temporal granularity and semantics.
- Bridging spaces: A hierarchical policy should predict visual evolution, distill it into frame-level motion intents, and ground those intents in embodiment-specific controls.
- Temporal mismatch: Coarse temporal chunks can blur grasp closure, contact onset, object release, fine alignment, and local collision avoidance.
- Action mismatch: Base movement is low-frequency and globally oriented, whereas arm manipulation is higher-frequency, local, and contact-sensitive.Entangling them can cause gradient interference and weaken compositional coordination.
- Train-test mismatch: Training inverse dynamics on ideal future observations creates exposure bias when deployment instead uses imperfect self-generated futures.The discrepancy can compound across long-horizon rollouts and derail execution.
3 The ABot-M0.5 Model
ABot-M0.5 organizes world-action learning as a hierarchical cascade that predicts future visual dynamics, frame-level motion intents, and executable robot actions. Its latent-action bridge, action-specialized transformers, and Dream Forcing strategy jointly address temporal, structural, and train-test mismatches.
- Hierarchical cascade: ABot-M0.5 factorizes prediction into world modeling, motion abstraction, and embodiment-specific control.The cascade predicts future video latents, refines them into frame-level latent actions, and translates those intents into executable actions.
- Intermediate latent actions: Frame-level latent actions bridge coarse video latents and fine-grained control by capturing local visual state transitions.They provide an intermediate, embodiment-agnostic representation between future visual dynamics and robot actions.
- Intermediate latent actions: The cascade separates embodiment-agnostic physical priors from hardware-specific kinematics, enabling generalization across heterogeneous robot platforms.Future video latents represent macroscopic environmental evolution, latent actions encode fine-grained motion intents, and executable actions remain embodiment-specific.
- Action decoupling: The Dual-level Mixture-of-Transformers disentangles modality representations and heterogeneous action subspaces such as base mobility and arm manipulation.Dedicated sub-towers, feed-forward networks, and prediction heads specialize learning while preserving coordinated reasoning through shared attention.
- Dream Forcing: Dream Forcing trains action prediction on self-generated visual predictions, aligning inverse-dynamics training with autoregressive inference.Unlike teacher forcing with clean ground-truth latents or diffusion forcing with difficult-to-match noise configurations, it exposes actions to model-induced visual errors.
- Design summary: The model combines intermediate latent actions, dual-level Mixture-of-Transformers, and Dream Forcing to address temporal, action-structure, and context mismatches.Together, these components target fine-grained motion abstraction, structured action decoupling, and inference-consistent learning.
4 Training Paradigm
ABot-M0.5 uses staged training to build world modeling, visual motion abstraction, and executable action generation before rollout-consistent fine-tuning. The pipeline combines heterogeneous robot data, self-supervised latent actions, camera normalization, and efficient long-sequence optimization.
- Progressive training: Three progressive phases establish world prediction, latent action abstraction, and action learning under increasingly realistic rollout conditions.The strategy moves from large-scale world modeling toward model-generated futures and executable control.
- Data preparation: Broad public and synthetic robot datasets provide coverage across embodiments, environments, task structures, and manipulation dynamics.The collection combines scale and diversity from OXE, OXE-AugE, Agibot-Beta, RoboCOIN, RoboMind, Galaxea, and InternData-A1.
- Latent action pretraining: Unlabeled or weakly labeled videos expand latent-action supervision because frame-level motion representations require visual frame pairs rather than control labels.This allows motion abstraction learning to draw on data beyond action-annotated robotic datasets.
- World model pretraining: The visual world model is pretrained as an autoregressive future-video predictor, adapting Wan2.2 5B weights to robotic environments through full-parameter fine-tuning.This stage supplies scene and object representations and reduces the burden on downstream inverse-dynamics fine-tuning.
- World model pretraining: Four canonical camera slots separate third-person scene views from wrist-mounted hand-object views across heterogeneous robot datasets.Extra views are randomly sampled, missing views are zero-padded, and padded regions are masked from attention and loss computation.
- Latent action pretraining: The latent action encoder learns structured transition embeddings from temporally ordered observations and enforces additive and reversal consistency.These constraints encourage longer transitions to decompose into shorter ones and reversed transitions to act as inverses.
- System optimization: Approximately 5× speedup in the combined forward-backward pass is achieved over a FlexAttention-style baseline for long-sequence video-action modeling.The design reduces kernel overhead, avoids unnecessary block padding, and lowers memory consumption.
5 Experiments
ABot-M0.5 is evaluated across simulated mobile-manipulation, manipulation, and real-world benchmarks, with results emphasizing long-horizon robustness and fine-grained control. Ablations connect these gains to latent actions, action decoupling, and rollout-aligned training.
- Benchmark Coverage: The evaluation spans RoboCasa365, RoboTwin 2.0, LIBERO/LIBERO-Plus, and real-world robotic tasks.These benchmarks cover household mobile manipulation, bimanual and compositional tabletop manipulation, and physical deployment.
- Mobile Manipulation: RoboCasa365 tests increasingly difficult atomic seen, composite seen, and composite unseen tasks involving long-horizon planning and precise manipulation.The category structure moves from localized interactions toward longer task chains and unseen compositions.
- Main Results: ABot-M0.5 shows particularly clear gains on long-horizon composite tasks, consistent with temporal alignment, action decoupling, and Dream-Forcing.The claimed mechanisms preserve fine-grained dynamics, coordinate mobility with manipulation, and improve robustness during long rollouts.
- Qualitative Analysis: Qualitative RoboCasa365 rollouts maintain coherent progression across navigation and manipulation while preserving fine-grained object-interaction control.This supports the quantitative finding that the framework is aligned with long-horizon embodied control rather than only low-level action prediction.
- Generalization: ABot-M0.5 performs strongly under both clean and randomized RoboTwin 2.0 settings, suggesting robustness beyond mobility-heavy tasks.The latent action abstraction benefits fine-grained control without mobility, while rollout-aligned training supports visual-variation robustness.
- Ablations: Removing latent actions degrades precise and contact-rich manipulation, while the full strategy reaches a 94.0% success rate.The reported alternatives reach 87.60%, 91.06%, and 90.86%; conditioning dropout pdrop = 0.2 reaches 91.06%, whereas pdrop = 0 reaches 94.0%.
5.4 Ablation Studies
The ablations and deployment studies examine how ABot-M0.5’s training choices affect manipulation precision, generalization, and physical execution. Real-robot results show strong performance with limited demonstrations across both fine-grained and long-horizon tasks.
- Action Decoupling: The action-decoupled Mixture-of-Transformers is evaluated on long-horizon composite tasks requiring frequent transitions between base navigation and arm manipulation.The comparison is against a modality-level Mixture-of-Transformers using a single transformer for action-modality prediction.
- Pretraining and SFT: Pretraining reaches 49.0% success in the RoboCasa365 Target 10% setting, versus 17.8% for direct fine-tuning from Wan2.2.The resulting gap is 31.2%, indicating that SFT alone is insufficient with limited downstream data.
- Pretraining and SFT: Pretraining shifts attention toward robotic-arm interaction regions, while SFT further aligns those visual priors with task semantics.The Wan baseline often attends to background clutter before these training stages improve action-related focus.
- Real-World Evaluation: On the real-world Peg Cylinder task, ABot-M0.5 achieves 70% success and a 96% process score, compared with π0.5 at 50% and 90%.FastWAM reaches 30% success and a 77% process score on the same task.
- Real-World Evaluation: Across Organize Plate, Arrange Fruits, Cup Stacking, and Arrange Flower, success rates are 70%, 80%, 80%, and 60%, respectively.FastWAM reaches only 20% to 40% in these complex scenarios, while ABot-M0.5 maintains process scores above 88%.
- Deployment Robustness: The same model handles fine-grained and long-horizon manipulation without task-specific changes.The paper attributes this flexibility to Dual-level MoT and Dream-Forcing’s reduction of the training-inference gap.
6 Conclusion and Future Work
ABot-M0.5 unifies intermediate action abstraction, action-space separation, and Dream Forcing to address structural mismatches in mobile-manipulation world-action learning. The paper concludes that deployment robustness requires more than simply scaling models or data.
- Core Innovations: Intermediate latent actions bridge video latents and robot actions, creating a fine-grained intermediate space for local visual state transitions.This factorizes direct video-to-action prediction into a video-to-latent-action-to-action pipeline.
- Core Innovations: Dual-level Mixture-of-Transformers separates heterogeneous action spaces, including mobility and manipulation behaviors.The design targets interference between fundamentally different action dynamics.
- Core Innovations: Dream Forcing trains inverse dynamics on self-dreamed latents to reduce the gap between training and autoregressive inference.The strategy is intended to improve robustness during deployment-time prediction.
- Takeaway: Experiments across simulation and real-world tasks report state-of-the-art performance in both long-horizon mobile and fine-grained manipulation.The conclusion attributes this progress to systematic engineering of future prediction, action decoupling, and deployment-time robustness.
- Future Work: Future work targets broader real-world data, diverse embodiments, long-term memory, faster inference, and validation beyond controlled laboratory settings.The stated direction is to test generalization in complex, unstructured mobile-manipulation environments.
7 Contributions
The paper assigns contributions across data, modeling, evaluation, deployment, writing, and challenge submission. The listed roles indicate a broad collaborative effort spanning the full research pipeline.
- Contributions: Data collection and standardization are credited to Yandan Yang, Ronghan Chen, Yuzhi Chen, Haoyun Liu, and Dekang Qi.
- Contributions: Model and training contributions are credited to Ronghan Chen, Zuojin Tang, Tong Lin, and Yandan Yang.
- Contributions: Post-training and evaluation are credited to Zuojin Tang, Tianlun Li, Haoning Wu, Ronghan Chen, Tong Lin, Mingxin Wang, and Bin Hu.
- Contributions: Real-robot experiments and deployment are credited to Dongjie Huo, Lulu Zheng, and Botai Yuan.