WB-WAM: Heterogeneous Body-Hand Pre-training for Humanoid Loco-Manipulation

I. INTRODUCTION

• The motivating gap is limited direct supervision of coordinated humanoid body, root, and hand motion in conventional robot pre-training data.The paper frames whole-body action coverage as a pre-training concern beyond the diversity of tasks and objects.Source: §I · p. 1
• WB-WAM jointly learns visual dynamics and whole-body action trajectories in a shared 72-D physical space, using available body, root, and hand annotations even when labels are incomplete.The action space combines body joint references, root motion, and articulated hand states from heterogeneous sources.Source: §I · p. 2
• 81.9% mean success on HumanoidArena, surpassing task-wise best reported baselines on all seven tasks; real-robot success averages 84.0% across five tasks.With aligned PICO demonstrations, adaptation using 30 robot demonstrations per task reaches 73.8% mean success across four tasks, versus 65.0% with 100 demonstrations for direct adaptation.Source: §I · p. 2
• The three-stage recipe uses heterogeneous pre-training, retargeted PICO demonstrations, and task-specific robot adaptation to link whole-body priors to downstream tasks.WB-Datasets includes 22 hours of PICO demonstrations and 1,011 robot episodes across eight tasks.Source: §I · p. 2
• The evaluations also include comparisons with six physical-robot baselines, language-conditioned manipulation, and generalization to unseen visual conditions.These are identified as additional evaluation studies and comparison scope.Source: §I · p. 2

A. Humanoid Foundation Models for Loco-manipulation

• Prior humanoid foundation models use varied action representations, while scalable video data often lack recorded actions or use proxies; controller-compatible humanoid references mainly enter through robot data or later adaptation.The paper identifies direct large-scale pre-training on temporally dense body, root, and articulated-hand references as comparatively underexplored.Source: A. Humanoid Foundation Models for… · p. 2

B. World Action Models

• WB-WAM differs from prior WAM approaches by introducing large-scale direct whole-body action supervision during generative video pre-training, unifying body and hand annotations from separate sources.The related-work passage contrasts this with evaluations focused on fixed-base or upper-body manipulation and humanoid WAMs using compact plans or controller-oriented latents.Source: B. World Action Models · p. 2

III. METHODS

• The three-stage curriculum integrates heterogeneous video-action pre-training, PICO motion transfer, and task-specific robot adaptation; the prediction space stays fixed across training.At deployment, the frozen SONIC encoder maps predicted body and root trajectories into controller latents.Source: §III · pp. 2–3, Figure 2 · p. 3

A. Preliminary: World Action Modeling with MoT

• The Mixture-of-Transformers backbone uses modality-specific video and action experts; the video expert models future-frame latents, while the action expert models normalized physical reference trajectories.The action expert attends to features from the video expert.Source: A. Preliminary: World Action Modeling… · p. 3

B. Problem Formulation

• The prediction target is future video plus a whole-body physical-reference trajectory, conditioned on egocentric RGB observation, proprioceptive state, and language instruction.The action vector has 72 dimensions comprising body joint references, root motion, and articulated hand references.Source: B. Problem Formulation · p. 3

• Stage I combines video paired with body and hand motion, egocentric video paired with hand motion, and text-conditioned body motion without video across 1,880.2 hours of data.Source: C. Stage I: Heterogeneous Body–Hand… · p. 3
• Available body, root, and hand annotations map to corresponding channels in a shared physical action space, allowing complementary supervision from separately sourced datasets.Source: C. Stage I: Heterogeneous Body–Hand… · p. 4
• A binary mask marks annotated, non-padded action entries and is applied to noisy action inputs when constructing the target flow velocity.Source: C. Stage I: Heterogeneous Body–Hand… · p. 4
• The action expert predicts action flow from noisy trajectories and flow time, conditioned on visual features, proprioception, and language; deployment denoises actions from the current frame without synthesizing future video.Source: C. Stage I: Heterogeneous Body–Hand… · p. 4
• The joint objective combines video and action flow-matching losses, and optimization on the heterogeneous dataset yields the initialization for downstream task adaptation.Source: C. Stage I: Heterogeneous Body–Hand… · p. 4

D. Stage II: Mid-training with Retargeted PICO Motion

• PICO mid-training pairs egocentric observations with retargeted body, root, and hand references, including human demonstrations of tasks later learned from robot demonstrations.Retargeting provides kinematic supervision, but alone does not establish physical-robot dynamic stability or contact feasibility.Source: D. Stage II: Mid-training with… · p. 4

E. Stage III: Task-Specific Real-Robot Post-training

• Real-robot adaptation initializes from the mid-trained model and augments joint video-and-action learning with a differentiable forward-kinematics loss.Source: E. Stage III: Task-Specific Real-Robot… · p. 4
• The forward-kinematics loss penalizes position and orientation errors for selected G1 links in the pelvis frame, measured over valid reference targets.Source: E. Stage III: Task-Specific Real-Robot… · p. 4

IV. WB-DATASETS

• WB-Datasets contains 22 hours of egocentric PICO demonstrations and 1,011 robot teleoperation episodes across eight task groups.PICO data support Stage II motion transfer, while robot demonstrations support Stage III task adaptation.Source: §IV · p. 4

A. Heterogeneous Pre-training Data

• Stage I uses 1,880.2 hours of heterogeneous data from nine external sources, including video-based and motion-only collections.Source: A. Heterogeneous Pre-training Data · p. 4
• Human motion is retargeted to the G1 morphology and Wuji hands where needed, while robot-coordinate data receive schema conversion and coordinate alignment.Unavailable modalities and action fields are recorded in metadata and validity masks.Source: A. Heterogeneous Pre-training Data · pp. 4–5
• Pre-training data checks cover invalid poses, motion discontinuities, body or hand collisions, and missing or corrupted video frames; ambiguous detector cases receive manual inspection.Source: A. Heterogeneous Pre-training Data · p. 5

B. PICO Egocentric Mid-training Data

• PICO recordings use egocentric RGB video and synchronized SMPL body motion, with GMR initialization and constrained whole-body inverse kinematics refining G1 references.Human hand keypoints are reconstructed from egocentric video and retargeted to Wuji joint configurations.Source: B. PICO Egocentric Mid-training Data · p. 5
• The PICO collection contains 13,396 episodes across 73 tasks, and its mid-training tasks share objectives with robot demonstrations to evaluate transfer under limited robot supervision.Source: B. PICO Egocentric Mid-training Data · p. 5

C. SONIC Real-Robot Post-training Data

• The SONIC teleoperation collection covers eight tasks and contains 1,011 episodes; body and root references use PICO tracking, while MANUS gloves control the Wuji hands.Source: C. SONIC Real-Robot Post-training Data · p. 5

V. EXPERIMENTS

• The experiments ask whether WB-WAM compares favorably with existing policies, whether whole-body pre-training improves downstream performance, and whether task-aligned PICO data reduce robot-demonstration needs.Source: §V · p. 5

A. Training Details

• The action expert, video expert, and proprioceptive encoder are jointly optimized across three stages, while the video VAE and UMT5 text encoder remain frozen.The video expert starts from Wan2.2; the action expert is a reduced-width DiT initialized by structural parameter transfer, and its output head is 96-D, with at most 72-D used.Source: A. Training Details · p. 5
• Pre-training runs for one epoch at batch size 16, sampling video-body-hand, video-hand, and text-body-motion data at a 3:1:1 ratio.The action branch conditions equally often on the full video or its first frame, allowing action prediction from generated future video with IDM or directly from the current observation without IDM.Source: A. Training Details · p. 5
• PICO mid-training uses a shared checkpoint trained for 10 epochs on 22 hours and 13,396 episodes across 73 tasks, before task-specific post-training for 50 epochs with 30 robot demonstrations.The PICO dataset contains 22 hours of demonstrations; the resulting shared checkpoint initializes each task policy.Source: A. Training Details · pp. 5–6

B. Deployment and System Details

• At each replanning step, the robot receives an RGB observation, language instruction, and proprioceptive state, then predicts a 32-step action chunk using 20 denoising steps.The first 20 steps execute at 20 Hz; body and root references are encoded by frozen SONIC for tracking, whereas hand references bypass the encoder and are issued as joint commands.Source: B. Deployment and System Details · p. 6

C. Evaluation Metrics

• Real-robot success rate (SR) measures final task completion, while task progress (TP) averages task milestone completion rates, including final success.Milestone rates use all trials, and mean SR and TP weight tasks equally.Source: C. Evaluation Metrics · p. 6

1) Simulation Benchmark Evaluation:

• The seven-task HumanoidArena benchmark includes three human-object interaction tasks and four human-scene interaction tasks, evaluated under the in-GMT protocol with SONIC for demonstrations and execution.Comparisons use the highest reported baseline success rate for each task under the same SONIC setting.Source: 1) Simulation Benchmark Evaluation · p. 6
• 81.9% mean success on HumanoidArena, outperforming the strongest reported baseline on all seven tasks.The gains span locomotion, posture adjustment, and object interaction.Source: 1) Simulation Benchmark Evaluation · p. 6

2) Real-World Comparison:

• The five tasks are curtain closing, table wiping, bed making, pillow moving, and cloth tidying; the first three require locomotion before or during manipulation.The latter two focus on stationary whole-body manipulation, and every method uses approximately 100 robot demonstrations per task in the main comparison.Source: 2) Real-World Comparison · p. 6

E. PICO Mid-training

• 73.8% mean success after PICO mid-training and adaptation with 30 robot demonstrations per task, versus 65.0% for direct adaptation with 100 demonstrations.The comparison uses 70% fewer robot demonstrations; at the same 30-demonstration budget, direct post-training achieves 48.8%.Source: Table III · p. 7, E. PICO Mid-training · p. 7

2) Real-World Comparison:

• 84.0% mean success across five real-robot tasks, compared with 80.0% for OpenWAM, the strongest evaluated baseline.Across the three tasks requiring locomotion, WB-WAM achieves 76.7% mean success versus OpenWAM’s 71.7%.Source: 2) Real-World Comparison · p. 7

E. PICO Mid-training

• PICO mid-training uses the full 73-task dataset, which includes approximately 150 PICO demonstrations for each of the four evaluated tasks, before task-specific post-training.The task policies start from the same shared mid-trained checkpoint and use 30 real-robot demonstrations each.Source: E. PICO Mid-training · p. 7

G. Visuomotor Generalization

• On cart pushing, WB-WAM executes on the real robot without further adaptation under an unseen visual condition that changes the cart’s appearance.The video expert can optionally roll out future observations under the modified condition.Source: G. Visuomotor Generalization · p. 7, Figure 7 · p. 7

VI. CONCLUSION

• The authors identify evaluation on one robot platform and task-specific policies as current scope limits, with unseen-task transfer and object-interaction execution accuracy left for future work.The conclusion also states that simulation benchmarks and real-world experiments demonstrate effectiveness for humanoid loco-manipulation, and that task-aligned PICO mid-training improves mean success with few robot demonstrations per task.Source: §VI · p. 8