PaperlayerView exact revision on arXiv

Source-linked AI summary

MachEmbodied-U0: Unified Understanding and Generation Model for Embodied Intelligence

Haoran Wen, Wenfu Wang, Kunsong Shi, Jingke Wang, Wancheng Feng, Yiren Zhang, Yueran Zhao, Xuancheng Zhang, Nanfei Ye, Xingru Chen, Zhaohong Sun, Chengmin Yang, Zikang Yu, Penghao Bi, Jia Shi, Yu Liu, Kun Zhan, Yan Xie

arXiv:2609.25627v1cs.ROcs.CV

TL;DR

Robot control needs to connect task intent and interaction location with scene prediction and precise action, capabilities that VLA and WAM approaches do not necessarily combine. ME-U0 unifies understanding and generation for grounded visual-dynamics and action prediction. With native benchmark supervision, it reports strong results on RoboDojo, LIBERO, and LIBERO-Plus, alongside real-world manipulation validation.

  • Problem

    VLA models typically do not explicitly model scene dynamics, while WAMs do not necessarily expose the task-relevant semantic and spatial grounding needed for fine-grained manipulation.

    Source: §1 · p. 3

  • Method

    ME-U0 couples understanding and generation experts, using subtask prediction and affordance grounding to guide joint visual-dynamics and continuous-action generation.

    Source: §1 · pp. 4–5

  • Results

    Using native benchmark supervision, ME-U0 achieves 17.66 on RoboDojo, 99.0% average success on LIBERO, and 82.5% average success on LIBERO-Plus.

    Source: §1 · p. 4

  • Takeaways & Limitations

    ME-U0 combines competitive downstream benchmark performance with validation on real-world manipulation tasks.

    Source: §1 · p. 5, §8 · p. 23

    Future work will extend ME-U0 to broader task, environment, and embodiment distributions and strengthen long-term memory and in-context adaptation.

    Source: §8 · p. 23

Abstract

from arXiv · show

General-purpose robot control requires models to understand task intent, identify where to interact, capture how the scene evolves, and generate precise actions. Vision-language-action models provide strong semantic priors but typically do not explicitly model scene dynamics, while world-action models couple visual prediction with control without necessarily exposing the task-relevant semantic and spatial structure needed for fine-grained manipulation. We present MachEmbodied-U0 (ME-U0), a unified embodied foundation model connecting understanding and generation experts through a Mixture-of-Transformers architecture. Subtask prediction and affordance grounding guide joint visual-dynamics and action generation via flow matching. Visual dynamics encompass future RGB, depth, surface normals, and optical flow, providing complementary supervision for appearance, geometry, and motion. Multi-rate Rotary Position Encoding (MRPE) aligns visual dynamics with fine-grained control. We pretrain ME-U0 on approximately 4,200 hours of curated demonstrations from robotic datasets and egocentric datasets. Using only the supervision natively available in each downstream benchmark, ME-U0 achieves an average score of 17.66 on the RoboDojo simulation benchmark and average success rates of 99.0\% and 82.5\% on LIBERO and LIBERO-Plus, respectively. We additionally validate ME-U0 on real-world robotic manipulation tasks, demonstrating its effectiveness beyond simulation. Without corresponding downstream supervision, ME-U0 further demonstrates zero-shot subtask prediction, affordance grounding, and visual dynamics on simulated and real-world observations. Overall, ME-U0 combines competitive downstream control performance with transferable task-grounding and visual-dynamics capabilities across simulation and the real world.

1 Introduction

ME-U0 connects task understanding, spatial grounding, visual dynamics, and continuous control in a unified model. It targets manipulation benchmarks with native benchmark supervision and also demonstrates transfer to real-world tasks.

1 Introduction

  • VLA models provide semantic priors but typically do not explicitly model scene dynamics, while WAMs model future observations and actions without necessarily exposing task-relevant grounding or decomposition.These complementary gaps motivate connecting semantic understanding, visual dynamics, and continuous control.Source: §1 · p. 3
  • ME-U0 couples understanding and generation experts in a Mixture-of-Transformers architecture, connecting scene-grounded task interpretation with visual-dynamics generation and continuous control.Specialized parameters serve understanding and continuous generation, while shared multimodal attention coordinates them.Source: §1 · p. 4
  • 17.66 on RoboDojo, 99.0% average success on LIBERO, and 82.5% average success on LIBERO-Plus demonstrate downstream performance using only each benchmark’s native supervision.The evaluation adds no subtask, affordance, geometry, or motion annotations during post-training.Source: §1 · p. 4
  • Subtask prediction and affordance grounding guide joint flow-matching generation of future RGB, depth, surface normals, optical flow, and continuous actions, with MRPE aligning control to visual transitions.The model is pretrained on approximately 4,200 hours of curated robotic and egocentric demonstrations, across four datasets and six robot embodiments in the robotic corpus.Source: §1 · pp. 4–5
  • ME-U0 also demonstrates broad validation on RoboDojo, LIBERO, and LIBERO-Plus, with additional transfer to real-world robotic manipulation.The paper presents the model as combining downstream control performance with transferable task-grounding and visual-dynamics capabilities.Source: §1 · p. 5

2 Related Work

Prior work develops semantic robot policies, visual-dynamics-based action models, and unified multimodal architectures. These lines offer complementary capabilities while leaving design questions about future prediction and task-grounded control.

2.1 Vision-Language-Action Models

  • VLA models bring pretrained semantic knowledge to robot control, while action representation methods range from FAST’s frequency-domain tokenization to π0’s flow-matching continuous actions.π0.5 combines multi-robot and web data with high-level semantic prediction for long-horizon manipulation in unseen homes.Source: §2.1 · p. 5

2.2 World Action Models

  • WAMs couple future visual prediction with action learning, while recent methods differ on whether future generation is needed at inference or mainly as representation-learning supervision.LaWAM uses compact visual subgoals; Fast-WAM retains video co-training but omits future prediction at inference.Source: §2.2 · p. 5

2.3 Unified Models

  • Unified multimodal and embodied models combine understanding and generation through shared context or expert pathways, including systems for language, visual forecasting, and action.UAM uses a parallel Dorsal Expert for visual-dynamics prediction to address multimodal forgetting during action training.Source: §2.3 · p. 6

3 Training Data

ME-U0 is pretrained on curated robotic and egocentric demonstrations, with filtering, multimodal annotation, and robot-aligned synthesis used to build a heterogeneous training mixture.

3 Training Data

  • Approximately 5,700 hours of robotic and 3,920 hours of egocentric demonstrations form the raw pool; curation retains approximately 4,200 hours, with a subset receiving subtask, affordance, geometry, and motion annotations.Egocentric data complements robotic data with manipulation contexts and behaviors underrepresented in robot demonstrations.Source: §3 · p. 6

3.1.1 Robotic Datasets

  • The robotic corpus draws on four open-source datasets and spans six embodiments; scene–action–object tags describe task contexts, behaviors, and objects across varied combinations.Common behaviors recur across task contexts, and shared actions associate with objects of different physical properties and roles.Source: §3.1.1 · p. 6, Figure 2 · p. 7

3.1.2 Egocentric Datasets

  • Egocentric demonstrations are selected to cover underrepresented task tags and provide aligned first-person observations with low-level human motion annotations for a unified hand-centric action representation.Sources include EgoDex, EgoLive, EgoVerse, HOI4D, HOT3D, and hand-annotated Ego-Exo-4D.Source: §3.1.2 · p. 7

3.2.1 Data Curation

  • Robotic episodes undergo sequential checks for signal anomalies, state–action consistency, physical bounds, and timestamp gaps; an episode is removed if an enabled check is triggered.Filtering operates on complete episodes and preserves the original recordings and temporal continuity of retained demonstrations.Source: §3.2.1 · pp. 7–8, Figure 3 · p. 8

3.2.2 Data Annotation

  • Annotations cover temporal subtasks, spatial affordances, and visual dynamics, using staged subtask segmentation and verification, keyframe-based target grounding, and derived depth, normals, and optical flow.The visual-dynamics labels provide geometric and motion supervision; depth and normals are estimated from individual RGB frames, while optical flow is estimated between sampled pairs.Source: §3.2.2 · pp. 8–9, Figure 4 · p. 9

3.3 Egocentric Data Processing

  • Egocentric action data is standardized in a human-centric format, while Ego-to-Robot synthesis combines scene reconstruction, trajectory processing, retargeting, and rendering to produce robot-aligned demonstrations.Easy tasks use replayed trajectories; contact-sensitive tasks receive object and motion reconstruction plus trajectory refinement before robot-specific inverse kinematics and rendering.Source: §3.3 · pp. 9–10, Figure 5 · p. 10

4 Model Design

ME-U0 separates semantic understanding from multimodal generation while coordinating both through shared attention. The design grounds instructions in subtasks and affordances, aligns video with finer-rate actions, and uses unified prompts and embodiment-aware action representations.

4.1 Main Architecture

  • The MoT architecture couples an understanding expert for subtasks and affordances with a generation expert for future visual observations and robot actions.The branches use shared multimodal attention, with specialized parameters for their distinct demands.Source: §4.1 · pp. 10–11, Figure 6 · p. 11

4.1.1 Understanding Expert

  • The understanding expert predicts local manipulation subtasks and, when annotated, task-relevant targets, bounding boxes, intended interactions, and interaction points.Samples without affordance annotations retain their original training objective.Source: §4.1.1 · p. 11

4.1.2 Generation Expert

  • The generation expert jointly produces future RGB, depth, surface-normal, or optical-flow sequences and continuous robot actions.Visual modalities share a video-VAE latent backbone, while action tokens are denoised synchronously with visual tokens.Source: §4.1.2 · pp. 11–12

4.2 Multi-rate Rotary Position Encoding

  • MRPE preserves video–action temporal alignment while assigning each action a distinct position within its video-frame chunk.Actions share the corresponding video latent frame’s temporal position but receive distinct height-axis indices.Source: §4.2 · p. 12

4.3 Unified Multimodal Prompting

  • The unified prompt combines the current observation and task instruction with control metadata and explicit subtask, affordance, and video-modality queries.Control metadata specifies conventions such as relative or absolute commands and gripper encoding.Source: §4.3 · pp. 12–13

4.4 Unified Action Representation

  • Dataset-specific adapters reconcile units, coordinate conventions, and control directions before normalization while preserving meaningful embodiment-specific differences.Gripper values use a shared coordinate from fully open (0) to fully closed (1), and some sources require kinematic conversion of states.Source: §4.4 · p. 13

5 Pretraining

Pretraining balances task coverage, aligns video and action sampling across recording rates, and supervises both semantic grounding and joint visual-action generation. Its objectives combine autoregressive understanding with flow-matched generation.

5.1 Training Data Sampling

  • Task-aware sampling gives eligible tasks initial coverage from multiple episodes, then distributes remaining training blocks with sublinear, capacity-bounded weighting.With α = 0.5, the strategy balances less abundant tasks against larger buckets and draws 120 million samples from the curated mixture.Source: §5.1 · pp. 13–14

5.2 Video–Action Temporal Alignment

  • Video–action sampling keeps prediction windows within approximately 1–2 seconds while sampling actions three times as densely as video.Each video-latent step corresponds to 12 action steps after video-VAE temporal compression.Source: §5.2 · p. 14

5.3 Understanding Expert Training

  • Understanding training uses autoregressive cross-entropy on supervised response tokens for available subtask and affordance labels, conditioned on the current image, instruction, and contextual metadata.The paired tasks connect the overall goal to the current manipulation step and localize relevant objects or regions.Source: §5.3 · p. 15

5.4 Generation Expert Training

  • The generation expert jointly flow-matches future visual latents and actions, processing noisy tokens together while using modality-specific output projections.The shared objective also supports RGB, depth, surface-normal, and optical-flow targets, as well as forward- and inverse-dynamics tasks.Source: §5.4 · pp. 15–16

6 Training Infrastructure

Training infrastructure reduces dataset setup and video-loading costs through precomputed metadata, logical shards, and coordinated decoding. Shard-based loading reuses overlapping frames while retaining shuffled sample assembly.

6 Training Infrastructure

  • Dataset initialization falls from approximately one hour to less than ten minutes, while video loading reduces overall training time by more than 30% on the authors’ training platform.The infrastructure targets recurring costs in dataset initialization and video data loading.Source: §6 · p. 16

6.1 Efficient Dataset Initialization

  • Precomputed episode manifests, reusable subtask indexes, and lightweight readers avoid repeated filesystem inspection, annotation parsing, and full dataset initialization.The subtask index preserves complete annotated intervals for reuse across different prediction horizons.Source: §6.1 · pp. 16–17

6.2 Logical Shards

  • Logical shards group nearby anchors within task subtask intervals, and workers merge overlapping frame requests before decoding and assembling shuffled samples.The described example reduces twelve frame requests to six unique frames; bounded decoding jobs and asynchronous preparation manage resource use and overlap decoding with training.Source: §6.2 · p. 17, Figure 7 · p. 17, §6.3 · p. 18

6.3 Video Decoding and CPU Offloading

  • Separate CPU jobs can prepare and cache data ahead of training, letting decoding capacity scale independently of GPU resources.A bounded lead over sample consumption prevents excessive accumulation of prepared payloads.Source: §6.3 · p. 18

7 Experimental Results

ME-U0 is post-trained separately on RoboDojo and LIBERO from the same pretrained checkpoint, using only supervision available in each benchmark. It performs strongly despite the absence of auxiliary task-grounding, geometry, and motion annotations.

7.1 Post-Training Recipe

  • The RoboDojo and LIBERO post-training runs start from the same pretrained checkpoint and use 368 PPU810E accelerators per run.LIBERO uses a 24-step action horizon with end-effector pose control; RoboDojo uses a 48-step horizon with delta joint-position control.Source: §7.1 · p. 18
  • Neither benchmark supplies subtask or affordance annotations or auxiliary video-geometry and motion supervision for post-training.The authors do not add these signals, preserving the benchmarks’ native supervision setting.Source: §7.1 · p. 18
  • ME-U0 performs strongly on RoboDojo and LIBERO despite post-training without subtask, affordance, geometry, or motion supervision.The reported tables cover standard LIBERO and LIBERO-Plus robustness.Source: §7.1 · pp. 18–19, Table 4 · p. 19, Table 5 · p. 19

7.2 Simulation Benchmark Evaluation

ME-U0 posts its strongest evaluated WAM result on RoboDojo and near-ceiling success across standard LIBERO suites. On LIBERO-Plus, the same checkpoint remains strong overall, with viewpoint and robot-initial-state perturbations proving most challenging.

7.2.1 RoboDojo

  • 17.66 overall process score and 11.18% SR: ME-U0 achieves the strongest overall performance among evaluated WAMs on RoboDojo.Its process scores are 23.95 for precision and 36.98 for long-horizon execution; memory-dependent tasks score 8.42 and 7.00% SR.Source: §7.2.1 · pp. 19–20

7.2.2 LIBERO/LIBERO-Plus

  • 99.0% average success on LIBERO and 82.5% overall success on LIBERO-Plus: the same checkpoint performs strongly on both benchmarks without LIBERO-Plus-specific adaptation.LIBERO-Plus success is 70.2% under camera-viewpoint changes and 75.7% under robot-initial-state variations, the most challenging categories.Source: §7.2.2 · p. 20
  • On the four standard LIBERO suites, a single LIBERO-adapted ME-U0 checkpoint achieves success rates of 98.4% on Spatial, 99.8% on Object, 99.0% on Goal, and 98.6% on Long, averaging 99.0%.Source: §7.2.2 · p. 20
  • LIBERO-Plus is more discriminative than standard LIBERO, with lighting and language perturbation success rates of 97.6% and 90.4%, respectively.Success rates for background textures, sensor noise, and object layouts are 81.9%, 81.3%, and 84.5%, respectively.Source: §7.2.2 · p. 20

7.3 Real-World Deployment

ME-U0 is deployed on real-world manipulation tasks using its pretrained checkpoint, without additional post-training or deployment-specific adaptation. The demonstrations provide qualitative evidence of instruction-following across grasping, pick-and-place, and semantically grounded placement.

7.3 Real-World Deployment

  • The pretrained checkpoint completes three instruction-guided demonstrations: picking up a marker, placing a tissue in a box, and positioning 20 to complete 3+17 = 20.Together, the tasks span grasping, pick-and-place manipulation, and semantically grounded object placement.Source: §7.3 · p. 20
  • Deployment uses the pretrained checkpoint without additional post-training or deployment-specific adaptation, with inference performed on a remote server.The evaluated robot platforms are represented in the pretraining corpus, so this tests familiar rather than unseen embodiments.Source: §7.3 · p. 20

7.4 Zero-Shot Analysis

Using pretrained weights without additional post-training, ME-U0 demonstrates zero-shot visual dynamics, affordance prediction, and subtask prediction on held-out simulated and real-world observations. The qualitative evaluations show structured predictions across these capabilities.

7.4 Zero-Shot Analysis

  • The zero-shot evaluations use pretrained weights without additional post-training, and all samples are excluded from the pretraining corpus.The three experiments test visual-dynamics generation, affordance prediction, and subtask prediction.Source: §7.4 · pp. 20–21

7.4.1 Visual Dynamics

  • Generated depth maps, surface normals, and optical flow capture meaningful scene structure and motion on held-out simulated and real-world observations.Predictions on real-world samples are qualitatively slightly more consistent than those on simulated samples; the authors hypothesize a simulation-to-real domain gap.Source: §7.4.1 · pp. 21–22, Figure 9 · p. 21

7.4.2 Affordance

  • Affordance prediction identifies and localizes task-relevant objects or regions from an observation and instruction across simulated and real-world data.Examples include a charger, fruit, a pen, a straw, and regions inside a bowl or on a table.Source: §7.4.2 · p. 22, Figure 10 · p. 22

7.4.3 Subtask

  • Subtask predictions specify the action for the current task stage, while corresponding affordance targets identify the relevant object or region.The qualitative examples pair the predicted subtasks with their affordance targets.Source: Figure 10 · p. 22, §7.4.3 · p. 23

8 Conclusion

ME-U0 connects task-grounded understanding, visual dynamics, and continuous action generation in a unified model. The conclusion reports competitive benchmark performance, real-world validation, and zero-shot capabilities, while identifying broader task and environment coverage, long-term memory, and in-context adaptation as future work.

8 Conclusion

  • ME-U0 uses a Mixture-of-Transformers architecture to connect subtask prediction and affordance grounding with joint visual-dynamics and continuous-action generation through flow matching.Visual dynamics include future RGB, depth, surface normals, and optical flow.Source: §8 · p. 23
  • Pretrained on approximately 4,200 hours of curated robotic and egocentric demonstrations, ME-U0 achieves competitive performance on RoboDojo, LIBERO, and LIBERO-Plus and is additionally validated on real-world manipulation tasks.The robotic corpus spans six embodiments.Source: §8 · p. 23
  • ME-U0 retains zero-shot subtask prediction, affordance grounding, and visual-dynamics generation without corresponding downstream supervision.Future work will broaden task, environment, and embodiment distributions and strengthen long-term memory and in-context adaptation.Source: §8 · p. 23

9 Contributions and Acknowledgments

The acknowledgments assign named contributors to data, model, infrastructure, evaluation, project leadership, and advising, and thank additional supporters.

Contributions and Acknowledgments

  • Data contributors are Wenfu Wang, Kunsong Shi, Yiren Zhang, Jingke Wang, Yueran Zhao, Xuancheng Zhang, and Nanfei Ye; base-model contributors are Haoran Wen, Wenfu Wang, Kunsong Shi, and Wancheng Feng.The paper separately credits Wenfu Wang, Jingke Wang, Xingru Chen, Zhaohong Sun, and Chengmin Yang for training infrastructure.Source: §9 · p. 23
  • Evaluation contributors are Jingke Wang, Kunsong Shi, Wancheng Feng, Zikang Yu, Penghao Bi, Yueran Zhao, and Jia Shi; project leads are Haoran Wen and Yu Liu, with Kun Zhan and Yan Xie as advisors.Names are reported with the contribution markers shown in the acknowledgments.Source: §9 · p. 23
  • The authors thank Xuyao Huang, Runsheng Wang, Jiahao Gu, Mingcui Wang, Yue Ma, Danlu Dong, Xueyang Zhang, Mofan Zhou, and Yuying Chen for support and contributions.Source: Acknowledgments · p. 23
Loading 2609.25627v1…