Source-linked AI summary
WholeBodyWAM: Generalizing Pre-trained World-Action Priors to Humanoid Loco-Manipulation via WBC-Grounded Coordination
Zhuo Li, Yiming Yao, Jim Tan, Mengjie Jing, Zhipeng Dong, Fei Chen
TL;DR
Most WAM studies focus on tabletop or arm-centric manipulation, leaving humanoid loco-manipulation less explored. WholeBodyWAM extends pre-trained world–action priors through structured WBC grounding and coordination; experiments report improved performance, including a 91.9% overall simulation task success rate over the listed WAM baselines. The paper presents this approach as a route toward scalable humanoid whole-body intelligence.
Problem
Most WAM studies focus on tabletop or arm-centric manipulation, while humanoid loco-manipulation and paired demonstrations remain limited.
Source: §I · p. 2Method
WholeBodyWAM preserves pre-trained world–action priors and uses UWBC to ground heterogeneous WBC commands while coordinating whole-body behavior.
Source: §I · p. 2, C. Unified WBC Interface · p. 4Results
91.9% overall task success rate, exceeding DreamZero, DreamZero-PT, and Cosmos3 by 26.9, 18.3, and 5.6 percentage points, respectively.
Source: B. Simulation Experiments · p. 6Takeaways & Limitations
Abstract
from arXiv · showhide
World Action Models (WAMs) offer a promising approach to general-purpose robot manipulation by jointly modeling visual dynamics and actions. However, most WAM studies focus on tabletop or arm-centric manipulation, while humanoid loco-manipulation remains less explored. To address this gap, we introduce WholeBodyWAM, which jointly predicts future visual dynamics, manipulation actions, and whole-body control intents for generalizable humanoid loco-manipulation. It preserves pre-trained world-action priors while grounding heterogeneous whole-body controller (WBC) semantics and coordinating whole-body behavior. Extensive experiments show that WholeBodyWAM achieves an overall simulation task success rate of 91.9%, with a 0.23 improvement in real-world out-of-distribution task progress and a 70% reduction in success-rate variance across WBCs relative to the respective baselines. These results suggest a path toward scalable humanoid whole-body intelligence by extending pre-trained world-action priors through structured WBC grounding and coordination, rather than relearning whole-body behavior from scratch. Project page: https://wholebodywam.github.io/.
I. INTRODUCTION
Existing WAM work largely targets tabletop or arm-centric manipulation, while humanoid loco-manipulation faces coordination, controller-semantic, and data-scarcity challenges. WholeBodyWAM extends pre-trained world–action priors with structured WBC grounding and coordination, and experiments report improved task performance, OOD generalization, and cross-WBC robustness.
Motivation
- Humanoid loco-manipulation remains underexplored, with coordination complexity, heterogeneous WBC command semantics, and scarce paired demonstrations limiting direct supervision.Existing WAM studies mostly address tabletop or arm-centric settings.Source: §I · pp. 1–2
Approach
- WholeBodyWAM asks how to generalize pre-trained world–action priors to humanoid loco-manipulation without learning whole-body behavior from scratch.The approach builds on the insight that pre-trained WAMs already capture rich manipulation priors.Source: §I · p. 2
- UWBC provides a shared semantic interface for heterogeneous WBCs, while CASA uses manipulation intent to coordinate whole-body behavior.CASA strengthens manipulation-to-UWBC information flow when greater whole-body coordination is indicated.Source: §I · p. 2
Experiments
- Experiments across six simulation and eight real-world tasks report improved task performance, OOD generalization, and cross-WBC robustness.The paper presents WholeBodyWAM as preserving pre-trained priors while grounding heterogeneous commands and coordinating whole-body behavior.Source: §I · p. 2
II. RELATED WORK
World Action Models jointly predict actions and future visual dynamics, and recent work extends them toward humanoid loco-manipulation. WholeBodyWAM differs by explicitly learning reusable grounding across heterogeneous WBC interfaces.
A. World Action Models
- WAMs jointly model robot actions and future visual dynamics to provide reusable manipulation priors, with approaches including joint video-action prediction and latent-frame action representations.DreamZero, Cosmos Policy, and Fast-WAM are examples of different WAM designs.Source: A. World Action Models · p. 2
- Humanoid WAMs such as MotionWAM and ω-0 predict whole-body motion or controller-compatible action latents, while WholeBodyWAM learns WBC grounding and coordination to adapt pre-trained WAMs.This adaptation uses limited humanoid supervision to extend a reusable world–action prior to whole-body configurations.Source: A. World Action Models · p. 2
B. Humanoid Loco-Manipulation
- Humanoid loco-manipulation policies span whole-body controllers, VLA-based policies, and WAM-based methods, which generally tailor action representations to downstream WBC interfaces.Examples include OmniH2O, HOVER, HugWBC, WholeBodyVLA, Ψ0, HEX, OpenHLM, MotionWAM, and ω-0.Source: B. Humanoid Loco-Manipulation · p. 2
- Unlike those controller-tailored representations, WholeBodyWAM uses UWBC to learn shared semantics over heterogeneous WBC commands and adapt grounding priors across controllers.Source: B. Humanoid Loco-Manipulation · p. 2
III. METHOD
WholeBodyWAM formulates humanoid loco-manipulation as joint prediction of future visual dynamics and structured whole-body actions conditioned on task context. Its factorization models action streams jointly with context and corresponding future dynamics, rather than prescribing sequential video-then-action inference.
A. Problem Formulation
- The model jointly predicts future visual dynamics and structured whole-body actions under task context.The context includes visual history, current robot state, and a language instruction.Source: A. Problem Formulation · pp. 2–3
- A shared Diffusion Transformer generates future visuals, manipulation actions, and UWBC commands from structured tokens, with CASA strengthening manipulation-to-UWBC attention when task-directional arm manipulability falls.Source: Figure 2 · p. 3
- Manipulation actions specify arm and finger targets, while UWBC commands carry whole-body actions through a distinct stream.The streams accompany future visual dynamics in the joint prediction.Source: A. Problem Formulation · p. 3
- The factorization models future scene evolution from task context and jointly models manipulation and UWBC actions conditioned on context and corresponding future dynamics.It describes dependency structure, not a sequential video-then-action inference procedure.Source: A. Problem Formulation · p. 3
B. Model Architecture
WholeBodyWAM extends a pre-trained video Diffusion Transformer with jointly processed visual, manipulation, and UWBC streams. UWBC organizes heterogeneous controller commands by shared physical semantics while allowing controller-specific fields.
B. Model Architecture
- A pre-trained video Diffusion Transformer jointly processes future visual, manipulation, and UWBC token streams, retaining the pre-trained visual–manipulation pathway while making whole-body behavior addressable.Instruction features, compressed video tokens, and encoded robot state provide conditioning.Source: B. Model Architecture · p. 3
- The model uses a modality-weighted joint flow-matching objective, with stream-specific target flow velocities and loss weights that vary with flow time.The noisy joint block contains visual, manipulation, and UWBC streams.Source: B. Model Architecture · p. 3
C. Unified WBC Interface
- UWBC addresses the differing physical semantics of task-space, discrete-mode, and joint-space WBC commands rather than flattening them into a controller-specific action vector.Source: C. Unified WBC Interface · pp. 3–4
B. Model Architecture
- Native self-attention visibility is preserved across the streams, while a gated coordination bias selectively enhances manipulation-to-UWBC attention within each prediction block.Source: Figure 3 · p. 4
C. Unified WBC Interface
- UWBC defines 46 shared continuous command dimensions with consistent physical meanings and reserves ten residual slots for controller-specific commands.Unsupported or undefined fields are masked according to each downstream controller’s capability profile.Source: C. Unified WBC Interface · p. 4
D. Coordination-Aware Self-Attention
WholeBodyWAM uses task-directional arm manipulability to gate manipulation-to-UWBC attention, strengthening whole-body coordination when an arm’s capability is reduced. Its cross-WBC evaluation reports high mean TSR and low variance across three controllers.
D. Coordination-Aware Self-Attention
- Coordination-aware self-attention selectively strengthens manipulation-to-UWBC information flow when arm manipulability along the task direction becomes insufficient.The gate is based on reduced task-directional manipulability in the current arm configurations.Source: D. Coordination-Aware Self-Attention · p. 4
- A gated bias modifies native self-attention logits, selecting UWBC queries and manipulation keys within the same prediction block.UWBC slots comprise shared physical-semantic fields and controller-specific residual slots.Source: D. Coordination-Aware Self-Attention · p. 4, Table I · p. 4
- The coordination gate measures relative manipulability reduction against sampled radial capability for each arm, with a small constant preventing division by zero.The global gate takes the larger relative reduction across both arms, so either active arm can trigger coordination.Source: D. Coordination-Aware Self-Attention · pp. 4–5
- 89.2% mean TSR and 10.5 pp2 population variance across SONIC, AMO, and GEAR WBCs.Each controller is fine-tuned separately.Source: Figure 4 · p. 5
IV. EXPERIMENTS
The experiments are organized around whether WholeBodyWAM improves task success, whether UWBC supports stability and portability across controllers, and whether manipulation priors transfer to OOD coordination settings.
IV. EXPERIMENTS
- The experiments ask whether WholeBodyWAM improves task success over WAM and VLA baselines.Source: §IV · p. 5
- They test whether UWBC improves stability and portability across heterogeneous WBCs.Source: §IV · p. 5
- They examine whether manipulation priors generalize to new whole-body coordination in OOD settings and how each mechanism contributes.Source: §IV · p. 5
A. Experimental Setup
The setup evaluates WholeBodyWAM and comparison methods on simulation and real humanoid platforms, using task success, progress, and cross-WBC variance metrics. It also compares downstream fine-tuning efficiency across demonstration budgets.
A. Experimental Setup
- Training uses 15K retargeted whole-body demonstrations and post-trains the pre-trained 14B WAM for 50,000 steps with LoRA.The demonstrations are collected in SIMPLE, converted to SMPL motion sequences, and retargeted to the G1 embodiment.Source: A. Experimental Setup · p. 5
- Real-world experiments use a Unitree G1 with two dexterous hands and head- and wrist-mounted RGB cameras; simulation uses the SIMPLE benchmark.Source: A. Experimental Setup · p. 5
- WholeBodyWAM achieves its largest downstream fine-tuning advantage with few task-specific demonstrations, consistent with effective reuse of pre-trained world–action priors through coordinated WBC grounding across methods.Source: Figure 5 · p. 5
- Baselines include adapted DreamZero and Cosmos3 world-action models, post-trained DreamZero-PT, and fine-tuned Ψ0 and GR00T N1.6 VLAs.Source: A. Experimental Setup · p. 5
- Evaluation reports TSR, normalized task progress before first failure or irreversible deviation, and population variance of mean TSR across WBCs.Each method-task combination uses 20 independent trials; lower cross-WBC variance indicates greater robustness.Source: A. Experimental Setup · p. 5
B. Simulation Experiments
WholeBodyWAM leads the simulation TSR comparison and shows stronger cross-WBC performance and lower variance than Cosmos-3. Its fine-tuning results also support reuse of pre-trained world–action priors.
Overall performance
- 91.9% overall TSR, exceeding DreamZero, DreamZero-PT, and Cosmos3 by 26.9, 18.3, and 5.6 percentage points, respectively.WholeBodyWAM also exceeds the highest VLA baseline, Ψ0 at 82.2%, by 9.7 percentage points.Source: B. Simulation Experiments · p. 6, Table II · p. 6
Cross-WBC robustness
- 89.2% average TSR and 10.5 pp2 cross-controller variance for WholeBodyWAM, versus 80.2% and 35.0 pp2 for Cosmos-3.These results are reported across AMO, SONIC, and GEAR WBCs.Source: B. Simulation Experiments · pp. 6–7
Downstream fine-tuning efficiency
- WholeBodyWAM's largest fine-tuning advantage occurs with few task-specific demonstrations, supporting reuse of pre-trained world–action priors.The comparison covers four simulation tasks and budgets of 50, 100, 200, and 300 task-specific demonstrations per task.Source: B. Simulation Experiments · p. 7
C. Real-World Experiments
Across eight real-world tasks, WholeBodyWAM outperforms DreamZero under both in-distribution and out-of-distribution conditions. It also demonstrates recovery through whole-body repositioning when a towel lies beyond its initial arm workspace.
ID performance
- 81.3% mean TSR for WholeBodyWAM versus 57.5% for DreamZero on real-world ID tasks, with gains of 35 points on BasketCarry and DoorEntry and 40 on BoxTransfer.The gap is small on manipulation-dominant TableCleanup (90% versus 85%).Source: C. Real-World Experiments · p. 7
OOD generalization
- 68.8% mean TSR for WholeBodyWAM versus 40.0% for DreamZero under OOD conditions, with a smaller ID-to-OOD drop (12.5 versus 17.5 points).Mean ID/OOD task progress is 0.92/0.82 for WholeBodyWAM versus 0.73/0.59 for DreamZero.Source: C. Real-World Experiments · p. 7
- After a failed reach for a towel displaced 10 cm beyond its initial workspace, WholeBodyWAM steps forward and adjusts its torso to grasp it on the next attempt.The recovery sequence is absent from the demonstrations.Source: Figure 8 · p. 7, C. Real-World Experiments · p. 8
D. Ablation Studies
Ablations reduce TSR when CASA, UWBC, or structured action factorization is removed, with the largest reduction for removing structured factorization.
Structured action factorization
- Removing structured action factorization reduces mean TSR from 91.9% to 80.8%, an 11.1-point drop.The ablation replaces the separate manipulation and UWBC streams with a capacity-matched monolithic stream under the same data and optimization budget.Source: D. Ablation Studies · p. 8
UWBC
- Replacing UWBC with native SONIC commands reduces TSR to 84.7%, a 7.2-point drop.The WBC pathway and supervision are retained in this ablation.Source: D. Ablation Studies · p. 8
CASA
- Removing CASA and restoring native self-attention reduces mean TSR from 91.9% to 86.9%, a 5.0-point drop.The reduction is larger on PickBetweenTables (8.3 points) and MoveBendPick (6.7 points) than on Handover (1.7 points).Source: D. Ablation Studies · p. 8
V. CONCLUSION
WholeBodyWAM generalizes pre-trained world–action priors to coordinated humanoid loco-manipulation through WBC grounding. Simulation and real-robot experiments show improved task performance, OOD generalization, and cross-WBC robustness.
Conclusion
- WholeBodyWAM preserves pre-trained world–action priors and grounds heterogeneous whole-body commands through UWBC.Coordination-aware self-attention further enhances state-dependent manipulation-to-UWBC interaction.Source: §V · p. 8
- Experiments in simulation and on real robots demonstrate improved task performance, OOD generalization, and cross-WBC robustness while maintaining strong performance on manipulation-dominant tasks.Data-scaling and ablation results further support the value of structured prior reuse.Source: §V · p. 8
- The results suggest that scalable humanoid whole-body intelligence can be built by generalizing reusable world–action priors through structured WBC grounding and coordination, rather than relearning whole-body behavior from scratch.Source: §V · p. 8