PaperlayerView exact revision on arXiv

Source-linked AI summary

WholeBodyWAM: Learning Whole-Body World Action Models with Scalable Motion Priors

Bowei Zhang, Qiyao Zhang, Shuanghao Bai, Xinhua Wang, Meng Li, Yilei Wang, Leiwang Zhang, Jian Tang, Lu Zhou, Lei Sun, Zhengping Che

arXiv:2609.18197v1cs.RO

TL;DR

Humanoid world-action modeling needs scalable supervision for coordinated whole-body dynamics, but target-robot trajectories are expensive and heterogeneous motion is not directly executable. WholeBodyWAM pretrains a language-conditioned motion prior on UniMotion-4K, integrates it with video and action experts, and improves prediction, manipulation, and target-robot data efficiency.

  • Problem

    Target-robot demonstrations are expensive to collect and difficult to scale, motivating transferable supervision from abundant human and humanoid whole-body motion.

  • Method

    WholeBodyWAM canonicalizes more than 4K hours of heterogeneous motion, pretrains a language-conditioned Motion Expert, and integrates it with Video and Action Experts through asymmetric MoT attention.

  • Results

    WholeBodyWAM improves future-motion prediction, downstream control, real-world humanoid manipulation, and target-robot data efficiency as motion-pretraining scale increases.

  • Takeaways & Limitations

    Scalable motion pretraining provides a transferable whole-body predictive prior for humanoid world-action modeling and real-world manipulation.

Abstract

from arXiv · show

Humanoid whole-body manipulation requires coordinated whole-body dynamics, yet large-scale trajectories from a target robot are expensive to collect and difficult to scale. In contrast, whole-body motion from human and humanoid sources is abundantly available, although such data cannot be directly used as embodiment-specific robot actions. This work asks whether these scalable motion resources can instead provide a transferable predictive prior for humanoid world-action modeling. We introduce WholeBodyWAM, a humanoid world-action model that learns whole-body dynamics from large-scale heterogeneous motion before target-robot training. We curate UniMotion-4K, a motion corpus spanning more than 4K hours from human videos, native 3D motion datasets, and heterogeneous humanoid platforms, and canonicalize these diverse sources into a unified motion space. A language-conditioned Motion Expert is then pretrained to predict future whole-body motion without target-robot action supervision. During robot post-training, the pretrained Motion Expert is integrated with Video and Action Experts through asymmetric Mixture-of-Transformers (MoT) attention, enabling predictive scene dynamics and whole-body motion to jointly inform embodiment-specific action generation. Experiments show that WholeBodyWAM consistently benefits from increased motion-pretraining scale, improves future-motion prediction and downstream task performance, and transfers effectively to real-world humanoid manipulation. Moreover, the pretrained motion prior substantially improves data efficiency under limited target-robot demonstrations.

I. INTRODUCTION

WHOLEBODYWAM addresses the limited scalability of target-robot data by transferring predictive whole-body motion priors from heterogeneous human and humanoid motion into humanoid world-action modeling. It combines motion pretraining with video-action modeling for embodiment-specific control and reports improved real-world manipulation performance and data efficiency.

  • I. INTRODUCTION: Future whole-body motion is treated as a control-relevant predictive signal because humanoid torso, arm, and lower-body dynamics must remain coordinated during extended tasks.
  • I. INTRODUCTION: UniMotion-4K provides more than 4K hours of heterogeneous whole-body motion, canonicalized from human videos, native 3D datasets, and humanoid platforms into a shared motion space.
  • I. INTRODUCTION: WHOLEBODYWAM pretrains a task-conditioned Motion Expert on UniMotion-4K, then integrates it with Video and Action Experts to transfer predictive whole-body priors to target-robot control.The architecture uses a two-stage framework in which future whole-body motion informs embodiment-specific action generation.
  • I. INTRODUCTION: Scaling motion pretraining consistently improves future-motion prediction, downstream control, and target-robot data efficiency, with transfer demonstrated on real-world humanoid manipulation.

A. UniMotion-4K: Data Composition and Processing

UniMotion-4K combines heterogeneous human and humanoid motion sources by converting them into a unified root-free 63D representation. The pipeline filters and annotates source data, while embodiment-specific adapters translate humanoid joint states into the shared space.

  • A. UniMotion-4K: Data Composition and Processing: UniMotion-4K unifies heterogeneous human and humanoid motion in a root-free 63D space containing local rotations for 21 joints.Global root translation, root orientation, and body shape are excluded to retain articulated body configurations.
  • A. UniMotion-4K: Data Composition and Processing: Humanoid-native joint states are mapped through embodiment-specific adapters trained on retargeted paired robot states and unified 63D motions.The target robot’s adapter converts proprioceptive states into the unified space during Stage II and deployment, and remains frozen during Stage II.
  • A. UniMotion-4K: Data Composition and Processing: The corpus combines Internet videos, egocentric human data, native 3D human datasets, and humanoid motion, all converted into the shared representation.
  • A. UniMotion-4K: Data Composition and Processing: More than 30K hours of heterogeneous source data are filtered for temporal and reconstruction quality before retained clips undergo 3D motion recovery.
  • A. UniMotion-4K: Data Composition and Processing: Video-derived sequences receive motion-focused language annotations describing posture, limb coordination, locomotion, temporal transitions, and object interactions.These motion-language pairs support language-conditioned Stage I pretraining.

B. Stage I: Scalable Motion Pretraining

Stage I learns a transferable, language-conditioned prior for future whole-body motion from UniMotion-4K without target-robot action supervision. Stage II then grounds that prior in target-robot demonstrations through jointly trained video, motion, and action experts with asymmetric attention.

  • B. Stage I: Scalable Motion Pretraining: Stage I pretrains a language-conditioned Motion Expert to predict task-conditioned future whole-body motion without target-robot actions or demonstrations.The model uses 16 motion-history frames and predicts the subsequent 32 frames in the shared 63D representation.
  • B. Stage I: Scalable Motion Pretraining: Continuous flow matching trains the Motion Expert to predict a velocity field from motion history, noisy future motion, language, and the flow timestep.At inference, future motion starts from Gaussian noise and is generated by integrating the learned velocity field from σ = 1 to σ = 0.
  • B. Stage I: Scalable Motion Pretraining: Asymmetric attention blocks Motion and Action streams from future visual tokens while allowing current visual tokens and cross-stream information, preventing privileged future information leakage.Future visual latents provide predictive supervision during training, but future visual tokens are omitted at inference.
  • B. Stage I: Scalable Motion Pretraining: Stage II jointly post-trains Video, Motion, and Action Experts so future visual latents, future whole-body motion, and target-robot action chunks are predicted together.
  • B. Stage I: Scalable Motion Pretraining: Stage I and Stage II share the same 63D motion representation and prediction horizon, enabling direct transfer of the pretrained Motion Expert.

D. Closed-Loop Humanoid Control

At deployment, WholeBodyWAM uses visual observations, proprioception, motion history, and language to predict future motion and executable actions. Chunked closed-loop execution replans after each 32-step action chunk, with leg commands handled by a whole-body controller.

  • D. Closed-Loop Humanoid Control: The joint model predicts future whole-body motion and an executable, embodiment-specific action chunk from visual input, motion history, and language.Predicted motion serves as a predictive representation, while actions remain in the robot’s native action space.
  • D. Closed-Loop Humanoid Control: At each query, the policy samples motion and actions with four Euler steps, executes a 32-step chunk at 30 Hz, then replans from new observations.This is a chunked closed-loop execution strategy.
  • D. Closed-Loop Humanoid Control: Arm, hand, head, and waist commands execute directly, whereas an RL-based whole-body controller generates leg commands for locomotion and balance.
  • D. Closed-Loop Humanoid Control: The evaluation targets six real-world whole-body tasks and examines baseline control, generalization to task variations, and effects of motion modeling, scale, and data availability.The tasks include coordinated manipulation, posture transitions, and loco-manipulation.

A. Experimental Setup

WholeBodyWAM is evaluated on six real-world humanoid tasks against representative VLA and WAM baselines, including controlled distribution shifts and motion-pretraining ablations. It achieves the strongest reported average score, generalizes better under tested shifts, and benefits from motion pretraining and its action-generation interaction.

  • B. Real-World Evaluation: 72.2% average task score is the highest reported, exceeding GR00T N1.7 by 11.4 percentage points across six real-world humanoid tasks.WholeBodyWAM achieves or matches the best performance on all six tasks.
  • C. Generalization Evaluation: 55.0% vs. 46.3% and 75.0% vs. 63.3% under spatial and object shifts show WholeBodyWAM outperforming GR00T N1.7 in both tests.The shifts are a 15 cm basket displacement and unseen toy colors and shapes, respectively.
  • 1) Effect of Motion Modeling:: Removing motion pretraining lowers average task score from 72.2% to 59.1%, while blocking Motion-to-Action attention lowers it further to 46.3%.The latter retains motion prediction supervision, isolating the contribution of direct motion-to-action interaction.
  • 1) Effect of Motion Modeling:: Integrating the pretrained Motion Expert into τ0-WM raises average task score from 39.4% to 62.7%, indicating transfer beyond WholeBodyWAM’s native architecture.

2) Scaling with Motion Pretraining:

Scaling motion pretraining from 0 to 4K+ hours improves future-motion prediction, real-robot task scores, and target-robot data efficiency, while WholeBodyWAM retains efficient inference.

  • Experimental Setup: The scaling study uses nested 1K, 2K, and 4K+ hour subsets plus a 0 h variant, with identical training-step budgets and target-robot demonstrations.Stage I uses 100K steps and Stage II uses 20K steps for every variant.
  • Scaling Study: 27.5% lower average MPJRE and a 10.5-point higher average real-robot task score result when Stage I data scales from 0 to 4K+ hours.Average MPJRE decreases from 1.127° to 0.817°, while the average task score rises from 57.1% to 67.6%.
  • Target-Robot Data Efficiency: Motion pretraining improves average task scores by 4.2, 11.2, and 10.5 percentage points with 20%, 50%, and 100% of target-robot demonstrations.At 50% demonstrations, the 4K+ hour model reaches 46.9%, exceeding FastWAM trained with the full dataset at 42.9%.
  • Inference Efficiency: 363 ms end-to-end latency makes WholeBodyWAM 19.5× faster than DreamZero, 14.5× faster than LingBot-VA, and 1.45× faster than FastWAM.Latency is measured on a single NVIDIA A100 GPU using each method’s deployment configuration.
  • Overall Finding: Across six whole-body tasks, scaling motion pretraining consistently improves future-motion prediction, downstream control, and target-robot data efficiency.The conclusion also reports validation through ablations and cross-architecture transfer.

APPENDIX

UniMotion-4K combines heterogeneous human and humanoid motion sources into a filtered, unified corpus used for Motion Expert pretraining.

  • Dataset Composition: The corpus combines heterogeneous sources whose motions may overlap, so reported statistics are computed after source-specific preprocessing, recovery, unification, and quality filtering.HumanML3D and AMASS are cited as an example of underlying-motion overlap.
  • Dataset Composition: UniMotion-4K contains 1,196,017 variable-length sequences, approximately 444M motion frames, and over 4.1K hours of whole-body motion.These retained sequences are distinct from the fixed 48-frame windows sampled during Stage I pretraining.
  • Internet-Video Processing and Quality Control: Internet videos are filtered for subject tracking, body visibility, movement, truncation, unreliable pose observations, and fragmented tracks before 3D motion recovery.The pipeline rejects isolated hand detections, nearly static segments, and clips lacking reliable torso observations.
  • Internet-Video Processing and Quality Control: Recovered sequences undergo numerical and temporal-consistency checks, including minimum length, finite-value, joint-rotation, pose-change, and root-translation criteria.Only sequences passing both video-level filtering and post-recovery quality control are retained.
  • Stage I Representation and Training: Each Stage I sample uses 16 historical and 32 future frames, with clean history conditioning the Motion Expert while only future frames are flow-perturbed.Language embeddings from the frozen Wan2.2 text encoder condition the Motion Expert throughout Stage I.

2) Stage II Flow-Matching Details:

Stage II independently flow-trains Video, Motion, and Action streams, then couples modality-specific Experts through layer-wise mixed attention under an asymmetric connectivity pattern.

  • Stage II Flow-Matching Details: Stage II samples independent flow timesteps for Video, Motion, and Action streams, using timestep shifts of 5 for Video and Action and 1 for Motion.The shifted timestep is defined as σ_x = ϕ_sx(u_x), with u_x sampled uniformly from 0 to 1.
  • Stage II Flow-Matching Details: Future visual latents, future motion, and action chunks are jointly optimized with a flow-matching objective after modality-specific noise is applied.For Motion, only future frames are noised; current visual input remains clean, while the Action stream contains 32 tokens for 34-dimensional robot actions.
  • Stage II Flow-Matching Details: A 32-step action horizon corresponds to eight future video frames plus the current conditioning frame because the action-to-video frequency ratio is 4.The Action stream does not use history/future type embeddings.
  • Layer-Wise Mixture-of-Transformers Interaction: Video, Motion, and Action Experts retain separate parameters while interacting through layer-wise Mixture-of-Transformers attention in a common attention space.Each Expert has independent query, key, value, and output projections, with normalized queries and keys using rotary positional embeddings.
  • Layer-Wise Mixture-of-Transformers Interaction: For each query stream, keys and values are concatenated only from modalities permitted by its modality-aware attention mask, then projected back to that Expert stream.The common attention geometry uses 24 heads with dimension 128, while hidden dimensions remain modality-specific.

4) Attention Connectivity and Motion-to-Action Ablation:

The attention design lets Motion and Action use current visual information while exchanging information with each other, and the real-robot evaluation uses standardized TianGong 3.0 demonstrations and milestone scoring.

  • Attention Connectivity: Motion and Action queries can attend to the current visual condition but cannot access future visual tokens, while Motion and Action exchange information through mixed attention.This asymmetric connectivity makes future visual tokens unavailable to both action-related streams.
  • Motion-to-Action Ablation: The Motion-to-Action ablation removes Motion keys and values from Action queries while leaving the reverse Action-to-Motion path and all other components unchanged.Motion initialization, objectives, demonstrations, optimization, and remaining model components are held constant.
  • Real-Robot Evaluation: All real-world evaluations use the full-sized TianGong 3.0 humanoid, with coordinated demonstrations collected through an IMU, matched master arms, and a joystick.The system controls head motion, arms and end effectors, locomotion, and waist motion through these teleoperation interfaces.
  • Real-Robot Evaluation: Each task uses 100 target-robot demonstrations, and every compared method receives the same demonstration set.This standardizes the demonstration data across methods.
  • Scoring Protocol: Real-robot scores accumulate sequentially completed milestones and normalize total rollout points by each task’s maximum achievable score.All methods share task definitions, initial conditions, the robot-side control stack, and scoring procedure.
  • Evaluation Tasks: The evaluation covers manipulation tasks requiring bending, grasping, transport, placement, deformable-object handling, posture transitions, and collision-free whole-body motion.Examples include Toy Pickup, Laundry Loading, Pillow Transfer, Kneeling Toy Storage, Toy Transfer, and Box Transfer.

3) High-Level Action Interface:

WHOLEBODYWAM uses a structured whole-body action interface with robot-specific low-level controllers, while its motion expert is pretrained and integrated during post-training. Stage I acquires predictive dynamics before target-robot training, with future motion used only as a predictive representation.

  • High-Level Action Interface: The policy outputs 34 high-level commands spanning arm joints, end effectors, head, posture, waist, locomotion, and mode/status.These outputs are dispatched to TianGong 3.0 control interfaces rather than directly predicting motor torques.
  • High-Level Action Interface: Robot-specific low-level controllers execute locomotion and posture transitions, while upper-body, head, and waist commands use corresponding robot-side interfaces.Walking uses an AMP-style controller, whereas kneeling and related posture transitions use a BeyondMimic-style motion-tracking controller.
  • Training and Conditioning: The Motion Expert is pretrained in Stage I and jointly optimized with Video and Action Experts during Stage II, while the video VAE and motion adapter remain frozen in post-training.Stage I trains the embodiment-specific motion adapter separately; Stage II initializes the Motion Expert from Stage I.
  • Stage I Evaluation: Before robot post-training, Stage I achieves predictive whole-body dynamics on held-out UniMotion-4K motion, with reported reductions of 28.6% and 20.1%.The evaluation uses MPJRE on future-motion prediction; the supplied passages report the reductions but do not identify the corresponding horizons fully.
  • Inference Procedure: At each query, the model predicts a 32-frame future-motion trajectory and 32-step action chunk, executes actions at 30 Hz, then replans from new observations.Predicted future motion remains a predictive representation and is not fed back into motion history or retargeted into executable commands.

6) Implementation Details of Baselines:

Baseline comparisons use a common TianGong 3.0 interface, demonstration dataset, low-level control stack, and milestone-based evaluation protocol. Held-out motion prediction shows that pretrained Motion Experts outperform static propagation and improve as pretraining scale increases.

  • Baseline Setup: WHOLEBODYWAM is compared with FastWAM, π0.5, GR00T N1.7, and τ0-WM as representative world-action, VLA, and humanoid foundation-model baselines.All methods are adapted to the same TianGong 3.0 policy-facing action interface and trained on the same target-robot demonstration dataset.
  • Stage I Future-Motion Prediction: The evaluation predicts H16 and H32 futures from 16 historical motion frames using mean per-joint rotation error on a held-out UniMotion-4K split.The split is independent of the TianGong 3.0 demonstrations used for Stage II.
  • Stage I Future-Motion Prediction: All pretrained Motion Experts outperform the static baseline, indicating predictive dynamics beyond simple pose propagation.Performance improves consistently with increasing motion-pretraining scale.
Loading 2609.18197v1…