PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-ABC: Towards Generalist Mobile Manipulation via Seeing, Coordinating and Imagining

Qiwei Liang, Guangyu Chen, Shaolong Zhu, Zikuan Xiao, Jinxuan Lu, Yifan Xie, Renjing Xu, Wenbo Ding, Tianxing Chen

arXiv:2609.35652v1cs.RO

TL;DR

Mobile manipulation requires spatial perception under ego-motion and coordination of coupled arm and base actions, while existing approaches use geometric representations or separate action streams. MM-ABC combines multilevel visual features, training-only future geometry supervision, and masked coordination of separate action streams. It reports strong results across mobile and fixed-base benchmarks, including 83% mean success on five real-world tasks.

  • Problem

    Mobile manipulation requires spatially grounded perception under ego-motion and coordination between heterogeneous, coupled arm and base actions.

    Source: §1 · p. 2

  • Method

    MM-ABC combines sparse multilevel VLM features and training-only future geometry supervision with MM-APT, which coordinates separate manipulation and mobility streams through masked joint attention and clean-action prediction.

    Source: §1 · p. 2

  • Results

    MM-ABC reports strong results across mobile and fixed-base benchmarks, including 83% mean success on five real-world tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support treating mobile manipulation as control over a reconfigurable workspace, with perception, anticipation, and arm–base coordination learned together.

    Source: §7 · p. 21

Abstract

from arXiv · show

Mobile manipulation extends robot interaction beyond a fixed kinematic workspace by making the reachable region itself controllable. This flexibility introduces two central challenges: spatially grounded perception under continuous ego-motion and coordinated control of heterogeneous arm and base actions. Existing approaches strengthen geometry through explicit 3D representations or predictive world models, and often decouple mobility and manipulation into separate action streams. We argue that effective mobile manipulation requires not only decoupling, but also representations that support efficient cross-stream collaboration. We present MM-ABC, a foundation model built around Seeing, Coordinating, and Imagining Arm-Base Collaboration. MM-ABC combines sparse multi-level VLM features for spatial perception; a training-only future branch that uses world imagination and geometric intent as extra supervision, strengthening perception and manipulation-intent prediction and improving the overall learning signal; and MM-APT, which coordinates separate manipulation and mobility streams through masked joint attention and clean-action x-prediction. In controlled ablations, replacing clean-action prediction with velocity prediction lowers success on RoboCasa365 composite-seen tasks from 32.8% to 29.2%, and removing future supervision or multilevel conditioning causes larger drops. We pretrain MM-ABC on 5,000+ hours of heterogeneous robot data spanning 400K+ episodes, 12 datasets, and 17 embodiments. Experiments cover EBench, RoboCasa365, ManiSkill-HAB, LIBERO, LIBERO-Plus, and real-world mobile manipulation. MM-ABC achieves 44.71% success on EBench, 61.2% on RoboCasa365, 99.1% on LIBERO, 82.8% on LIBERO-Plus without perturbation training, and 83% mean success on five real-world tasks.

1 Introduction

Mobile manipulation makes the reachable workspace controllable but requires spatial perception under ego-motion and coordination between coupled arm and base actions. MM-ABC combines multilevel visual features, training-only future geometry supervision, and coordinated action streams, with reported results across simulation, fixed-base, and real-world tasks.

1 Introduction

  • Mobile manipulation makes workspace extent controllable, but requires spatially grounded perception during ego-motion and coordination of coupled arm and base actions.Base motion changes arm reach, while intended manipulation informs where the base should move.Source: §1 · p. 2
  • MM-APT keeps manipulation and mobility in separate streams, using masked joint attention and clean-action x-prediction to coordinate them.A controlled synthetic study and RoboCasa365 ablation favor clean-action over velocity prediction.Source: §1 · pp. 2–3
  • 44.71% success on EBench and 61.2% task-weighted average on RoboCasa365 exceed the strongest respective baselines by 3.30 and 7.0 percentage points; real-world mean success is 83%.MM-ABC also reports 99.1% mean success on LIBERO and 82.8% on LIBERO-Plus without further training.Source: §1 · p. 3
  • MM-ABC uses sparse multilevel VLM features and training-only world imagination to supervise geometric intent without deployment-time overhead.Its future branch predicts geometry-rich representations at sparse horizons, while future representations do not access action tokens and the branch is removed at deployment.Source: §1 · pp. 2–3
  • MM-30 provides 30+ hours of multi-view demonstrations across 40+ mobile manipulation tasks, with diverse objects, backgrounds, and everyday skills.The model is also pretrained on 5,000+ hours spanning 400K+ episodes, 12 datasets, and 17 embodiments.Source: §1 · p. 3

2 Related Work

Prior work expands mobile manipulation through broader policies and structured coordination, improves perception through geometry and predictive supervision, and explores clean-action prediction. MM-ABC situates its contributions within these efforts while studying collaboration between distinct arm and base streams.

2.1 Mobile Manipulation

  • Mobile manipulation work broadens tasks and embodiments, while coordination between heterogeneous control channels remains an architectural concern.Prior methods use separate action branches, structured action spaces, subsystem-specific perception, or joint attention; MM-ABC applies joint attention to specialized streams with asymmetric temporal visibility.Source: §2.1 · pp. 3–4

2.2 Representation Alignment and Predictive Supervision

  • Representation-alignment methods vary in teacher information and which policy component receives supervision, making the alignment pathway a design choice.Predictive-supervision methods can remove their prediction branches after training; MM-ABC predicts future geometric features while mutually masking future and action streams.Source: §2.2 · p. 4

2.3 Clean-Action Prediction and Cross-Stream Interaction

  • Clean-action outputs, velocity-space objectives, and sampling procedures are distinct design choices, and MM-ABC studies prediction targets in coupled arm–base streams.The paper evaluates endpoint noise, denoising across noise levels, and task allocation under few-step sampling against an analytical Bayes-optimal denoiser and in full-policy ablations.Source: §2.3 · pp. 4–5

3 Understanding Clean-Action Prediction

The section compares clean-action and velocity prediction under a controlled synthetic setup, then evaluates their learned-task implications.

3 Understanding Clean-Action Prediction

  • The matched synthetic comparison changes only the prediction target; clean chunks occupy rank 13, while velocity and noise span the ambient space.This gives the velocity head noise directions to reproduce that are absent from the clean-action subspace.Source: §3 · p. 5
  • At t=0.01, x-prediction reduces normalized endpoint-estimate variance to 0.023 versus 0.54 and yields a v/x excess-risk ratio of 1.64.At t=0.2, normalized variance is 0.35 versus 0.63; the excess-risk ratio compares each prediction head with the Bayes-optimal denoiser.Source: §3 · p. 6
  • At five Euler steps, x-prediction lowers task-allocation error to 0.0090 versus 0.0106 and off-subspace mass to 0.095 versus 0.104.Task-allocation error is lower at every tested budget from 1 to 16 steps; sliced Wasserstein distance is lower at 8 and 16 steps.Source: §3 · p. 6, Figure 1 · p. 6
  • In the learned setting, clean-action prediction beats velocity prediction by 3.6 points on RoboCasa365 composite-seen tasks.Here, allocation is inferred from observations, instruction, and robot state, with action streams exchanging information through masked joint attention.Source: §3 · p. 6

4 MM-ABC Architecture

MM-ABC combines multilevel visual context, coordinated manipulation and body-action streams, and training-only future-geometry supervision. Its training objective predicts clean action chunks and uses five-step Euler sampling at inference.

4.1 Overview

  • MM-ABC maps current multi-view images, instruction, and robot state to 64-step action chunks split into manipulation and body motion; MM-APT jointly denoises the two streams.A separate future stream predicts later geometric features during training while remaining isolated from action tokens.Source: §4.1 · pp. 6–7

4.2 Sparse Multilevel Perceptual Context

  • Sparse multilevel context retains final-layer semantics and injects intermediate VLM features through gated residual updates, which start at zero and leave context read-only to action tokens.Intermediate features come from layers 11, 19, and 27; later expert blocks retain the accumulated context.Source: §4.2 · pp. 7–8

4.3 Structured Two-Stream Action Generation

  • Separate stream transformations preserve distinct control semantics while masked joint attention enables within-segment exchange; far tokens read near tokens, but near tokens cannot read far tokens.The asymmetric access lets the longer-horizon tail build on the immediate plan without influencing it.Source: §4.3 · p. 8

4.4 Future Geometry as Auxiliary Supervision

  • A training-only World Expert predicts VGGT-Omega geometric features at steps n+32 and n+64 from current context; future queries are isolated from actions, and the future branch and teacher are removed at inference.The future targets are generated from recorded future images by a frozen geometric teacher.Source: §4.4 · pp. 8–9

4.5 Training Objective

  • Training interpolates Gaussian noise and clean chunks, weights masked action errors across active streams, and samples action chunks with five Euler steps at inference.The objective averages four independent noise/time draws that share one context computation; the robot executes a prefix before replanning.Source: §4.5 · p. 9

5 Multi-Embodiment Pretraining Data Engine

MM-ABC’s pretraining data engine combines heterogeneous robot sources in a shared masked interface, with source-specific semantic auditing and staged quality filtering. The resulting corpus spans 12 datasets and 17 embodiments, including the self-collected MM-30 mobile-manipulation set.

5.1 Corpus Composition and Training Mixture

  • The cleaned corpus contains 5,166.2 hours and 400K+ episodes from 12 datasets, 51 subsets, and 17 embodiments.Real-world and simulated demonstrations contribute 80% and 20% of duration; platforms capable of mobile manipulation contribute about 61%.Source: §5 · p. 9
  • Training probabilities use sublinear duration weighting with a 0.2 cap, so the largest profiles do not dominate the rebalanced mixture.Figure 3 aggregates the profile-level sampling probabilities by dataset, which differ from each dataset’s duration contribution.Source: §5.1 · pp. 9–10

5.2 Self-Collected Mobile Manipulation Dataset

  • MM-30 adds 30+ hours of multi-view demonstrations across 40+ mobile-manipulation tasks, varying viewpoints, initial configurations, objects, and scenes.All cleaned MM-30 demonstrations are included in pretraining.Source: §5.2 · pp. 10–12

5.3 Embodiment Diversity and Unified Control Interface

  • A fixed 80D canonical state/action interface accommodates embodiment-specific channels through a validity mask, with unavailable channels zeroed and excluded from action loss.The interface reserves left- and right-arm blocks and a body block for base, torso, lift, head, and auxiliary controls.Source: §5.3 · p. 12

5.4 Data Engine: Semantic Audit, Cleaning, and Canonicalization

  • Dataset-specific frame contracts and adapters align field semantics, units, coordinate frames, TCPs, rotations, and gripper conventions before common quality checks.Four filtering tiers check structure, physical validity, timing, and stationary behavior; retained contiguous segments are independently validated before mixture inclusion.Source: §5.4 · p. 13, Figure 6 · p. 13

6 Experiments

MM-ABC is evaluated on mobile and fixed-base simulation benchmarks, component ablations, and five real-world tasks. It reports strong results across these settings, while the ablations compare clean-action prediction, future supervision, and multilevel conditioning.

6.1 Simulation Experiments

  • 44.71% success and a 59 progress score on EBench exceed π0.5’s 41.41% and 54, respectively.The gains are 3.30 percentage points in success and 5 points in progress.Source: §6.1 · p. 14, Figure 8 · p. 15
  • 61.2% task-weighted average success on RoboCasa365 exceeds ABot-M0.5 by 7.0 percentage points overall, with gains across all three task splits.The split results are 78.4%, 52.7%, and 50.3%, respectively; composite tasks are harder than atomic tasks across the compared methods.Source: §6.1 · pp. 14–15, Table 1 · p. 15
  • MM-ABC achieves the highest mean success on SetTable (86.4%), TidyHouse (70.2%), and PrepareGroceries (65.6%).Improvements on TidyHouse and PrepareGroceries are concentrated in picking.Source: §6.1 · p. 15, Table 2 · p. 16, Table 3 · p. 16
  • MM-ABC reaches 99.1% mean success on LIBERO, 0.5 points above ABot-M0, and 82.8% total success on LIBERO-Plus without further training.The LIBERO-Plus result is the highest among compared methods, including methods trained on perturbed demonstrations.Source: §6.1 · p. 16, Table 5 · p. 17

6.2 Ablation Studies

  • On RoboCasa365 composite-seen tasks, the full model averages 32.8%; velocity prediction lowers this to 29.2%, while removing future supervision lowers it to 24.9%.Final-layer and interleaved conditioning average 25.6% and 22.0%, respectively.Source: §6.2 · pp. 17–18, Table 6 · p. 18

6.3 Real-World Experiments

  • MM-ABC averages 83% success across five real-world tasks, versus 71% for π0.5 and 54% for StarVLA-GR00T.All methods used the same 200 demonstrations per task and were evaluated over 20 trials per task.Source: §6.3 · p. 20, Figure 10 · p. 20

7 Conclusion

MM-ABC combines multilevel visual-language conditioning, a training-only future-geometry branch, and separate but coupled manipulation and body-action streams. It reports strong benchmark and real-world results, while component ablations favor the full configuration.

7 Conclusion

  • MM-ABC uses sparse DeepStack conditioning, training-only future geometry supervision, and MM-APT’s separate manipulation and body streams with masked joint attention and clean-action prediction.The future branch aligns shared perception with future geometry without adding inference cost.Source: §7 · p. 20
  • Across benchmarks, MM-ABC reaches 44.71% success on EBench, 61.2% task-weighted success on RoboCasa365, and the highest mean success on all three ManiSkill-HAB suites.It also reaches 99.1% on LIBERO and 82.8% on LIBERO-Plus; real-world mean success is 83%, compared with 71% for π0.5 and 54% for StarVLA-GR00T.Source: §7 · p. 20
  • Sparse multilevel injection outperforms final-layer and interleaved conditioning by 7.2 and 10.8 points, respectively; these results support viewing mobile manipulation as reconfigurable-workspace manipulation integrating perception, anticipation, and arm–base coordination.Source: §7 · p. 21

Related papers

Loading 2609.35652v1…