PaperlayerView exact revision on arXiv

Source-linked AI summary

ExBody2: Advanced Expressive Humanoid Whole-Body Control

Mazeyu Ji, Xuanbin Peng, Fangchen Liu, Jialong Li, Ge Yang, Xuxin Cheng, Xiaolong Wang

arXiv:2412.13196v2cs.ROcs.AIcs.LG

TL;DR

Humanoid robots still struggle to imitate expressive whole-body human motion while remaining stable and robust across the robot–human dynamic and kinematic gap. Exbody2 addresses this with automated motion curation, generalist-to-specialist training, teacher–student distillation, and decoupled velocity and landmark tracking. It achieves broad, robust motion coverage and higher targeted fidelity, while specialist policies remain difficult to recombine smoothly.

  • Problem

    Humanoid robots lack a reliable way to achieve human-like whole-body expressiveness and versatility while maintaining stability and control across their dynamic and kinematic gap with biological bodies.

    Source: §I · p. 2

  • Method

    Exbody2 retargets human motion, filters infeasible lower-body sequences, trains a generalist policy, fine-tunes specialists, and uses decoupled motion–velocity control with teacher–student distillation.

    Source: §I · p. 2, 1) Generalist policy with automated… · p. 3, 1) Teacher Policy Training · p. 5

  • Results

    Exbody2 achieves higher fidelity than four state-of-the-art baselines in simulation and real-world tests, while fine-tuning further improves targeted motion quality and expressiveness.

    Source: §I · p. 2, A. Data-driven Generalist-specialist… · p. 3, §V · p. 9

  • Takeaways & Limitations

    The generalist–specialist pipeline balances broad adaptability with focused accuracy for expressive humanoid whole-body tracking.

    Source: A. Data-driven Generalist-specialist… · p. 3, §V · p. 9

    Specialist policies cannot yet be seamlessly recombined, limiting smooth and efficient transitions between different motion types within one tracking session.

    Source: §VI · p. 9

Abstract

from arXiv · show

This paper tackles the challenge of enabling real-world humanoid robots to perform expressive and dynamic whole-body motions while maintaining overall stability and robustness. We propose Advanced Expressive Whole-Body Control (Exbody2), a method for producing whole-body tracking controllers that are trained on both human motion capture and simulated data and then transferred to the real world. We introduce a technique for decoupling the velocity tracking of the entire body from tracking body landmarks. We use a teacher policy to produce intermediate data that better conforms to the robot's kinematics and to automatically filter away infeasible whole-body motions. This two-step approach enabled us to produce a student policy that can be deployed on the robot that can walk, crouch, and dance. We also provide insight into the trade-off between versatility and the tracking performance on specific motions. We observed significant improvement of tracking performance after fine-tuning on a small amount of data, at the expense of the others.

I. INTRODUCTION

Exbody2 addresses the challenge of producing expressive, versatile humanoid whole-body motions while maintaining stability across the robot–human dynamic and kinematic gap. Its generalist–specialist pipeline, automated data curation, and decoupled motion–velocity control improve adaptability, targeted fidelity, and robustness in simulation and real-world tests.

Motivation

  • Whole-body humanoid control remains difficult because robots must reconcile high-dimensional nonlinear dynamics with the competing demands of expressiveness, versatility, stability, and control.Traditional model-based approaches require accurate system identification, physical modeling, and intensive online computation for real-time stability.Source: §I · p. 2, §II · pp. 2–3

Core approach

  • Exbody2 combines automated data curation, generalist–specialist policies, and decoupled motion–velocity control for expressive whole-body humanoid tracking.The framework is trained through a sim-to-real pipeline using retargeted human motion and simulated data before deployment on humanoid robots.Source: §I · p. 2, §III · p. 3, A. Data-driven Generalist-specialist… · p. 3

Generalist policy

  • Automated curation removes infeasible lower-body motions while preserving upper-body diversity, balancing physical feasibility with broad expressive coverage.This addresses the mismatch between human motion datasets and robot capabilities while retaining diverse behaviors for generalist training.Source: §I · p. 2, A. Data-driven Generalist-specialist… · p. 3

Specialist policy

  • Specialist policies fine-tune the generalist policy for targeted motion groups, improving precision while retaining priors learned from broad motion exposure.Motion labels or action recognition can support dynamic selection of an appropriate specialist policy.Source: §I · p. 2, A. Data-driven Generalist-specialist… · p. 3

Motion–velocity control

  • Decoupling local keypoint tracking from velocity control lets velocity guide movement while keypoint tracking emphasizes expressive motion imitation.A teacher policy uses privileged simulation information, and DAgger-style student distillation infers that information from historical observations for real-world deployment.Source: §I · p. 2

Evaluation

  • Exbody2 achieves higher fidelity than four state-of-the-art baselines across simulation and real-world Unitree G1 tests, with further quality gains from task-specific fine-tuning.The reported evaluations emphasize a robust single generalist policy for diverse motions and improved expressiveness for targeted behaviors such as dancing.Source: §I · p. 2, A. Data-driven Generalist-specialist… · p. 3

1) Generalist policy with automated data curation:

Exbody2 builds a generalist policy by filtering motion data according to tracking errors, preserving diversity while excluding infeasible motions. The resulting policy can then be fine-tuned into specialists for higher-precision motion categories.

Generalist policy with automated data curation

  • The initial policy is trained on a diverse dataset containing many infeasible motions, then evaluated per sequence using lower-body tracking errors.Lower-body errors are emphasized because the lower body is central to dynamic feasibility and balance.Source: 1) Generalist policy with automated… · p. 3
  • The filtering error combines lower-body keybody-position error and joint-angle tracking error to rank motion sequences.The keybody term helps prevent extreme deviations such as flipping or rolling, while the joint-angle term measures precision.Source: 1) Generalist policy with automated… · p. 3
  • Moderate filtering thresholds produce the best-performing policy by balancing motion diversity against feasibility.Very small thresholds limit generalization, whereas very large thresholds retain infeasible motions that degrade training effectiveness.Source: 1) Generalist policy with automated… · p. 3
  • The selected optimal threshold generalizes to other motion datasets, supporting robust training and improved performance.The threshold is selected by evaluating policies trained on subsets defined by different error thresholds.Source: 1) Generalist policy with automated… · p. 3

Specialist policy finetuning

  • Specialist policies are fine-tuned from the generalist policy for targeted motion groups rather than trained from scratch.This warm start supports higher-precision adaptation while retaining exposure to diverse motions and disturbances.Source: 2) Specialist policy with finetuning… · p. 4

B. Policy Objective and Architecture

Exbody2 trains a privileged-information teacher policy in simulation, then distills it into a deployable student using real-world-aligned observations. The student is trained from historical observations to reproduce the teacher’s control actions.

Policy Objective and Architecture

  • The teacher policy is trained with PPO using privileged simulator information, while the student replaces that information with observations aligned with real-world deployment.Privileged information includes ground-truth robot and environment states unavailable outside simulation.Source: B. Policy Objective and Architecture · p. 4, 1) Teacher Policy Training · p. 5
  • The humanoid control problem is formulated as an MDP in which the teacher receives privileged states, proprioception, and motion targets and outputs joint-position targets.The action is a 23-dimensional target for joint proportional-derivative controllers.Source: 1) Teacher Policy Training · p. 4
  • Motion targets include desired joints and 3D keypoints together with target root velocity and root pose.These targets support whole-body motion tracking under joystick control.Source: 1) Teacher Policy Training · p. 5
  • The student encodes a history of observations and the motion target, then learns to reproduce the teacher’s actions with mean squared error.The student therefore operates without the teacher’s privileged information.Source: 2) Student Policy Training · p. 5
  • Student training uses iterative rollouts in simulation, with the teacher providing oracle actions for visited states until the distillation loss converges.This procedure follows the DAgger strategy and accumulates training data from the student’s own states.Source: 2) Student Policy Training · p. 5

3) Motion-velocity Decoupled Control Strategy:

Exbody2 decouples local keypoint motion imitation from velocity control to avoid failures caused by drifting global keypoints. The evaluation compares this strategy with prior whole-body tracking methods using velocity, keypoint-position, and joint-position errors.

Motion-velocity decoupling

  • Exbody2 maps global keypoints into the robot’s current coordinate frame and uses velocity-based global tracking instead of direct global-keypoint tracking.This separates movement guidance from key body tracking for motion imitation.Source: 3) Motion-velocity Decoupled Control… · p. 5

Limitations of prior tracking

Experimental comparison

Method comparison

  • Exbody2 uses local keypoint tracking, data curation, and training techniques intended to improve motion fidelity and sim-to-real transfer.Source: A. Experimental Setup · p. 6

Evaluation metrics

  • The evaluation measures root linear velocity error, keybody-position error, and joint-position error, including separate upper- and lower-body metrics.The reported metrics assess velocity tracking, keypoint tracking, and joint tracking across body regions.Source: A. Experimental Setup · p. 6

B. Generalist Policy Performance

Exbody2’s generalist policy outperforms prior baselines across simulation and real-world whole-body tracking, while automated filtering improves stability and precision. The filtering results show that moderate thresholds best balance motion diversity with feasibility, although filtered training slightly increases velocity error relative to unfiltered training.

Simulation performance

Trade-off

  • Filtered training slightly increases velocity tracking error compared with unfiltered training, but the reported stability and precision gains outweigh this trade-off.The comparison is reported under simulation conditions.Source: B. Generalist Policy Performance · p. 6

Real-world performance

  • Exbody2 achieves higher upper- and lower-body tracking accuracy than baselines in real-world experiments on diverse motions including walking, squatting, and dancing.Source: B. Generalist Policy Performance · p. 6

Automatic data curation

  • Low thresholds limit generalization through overly simple motion data, whereas high thresholds introduce unstable motions and training noise that reduce tracking accuracy.Source: C. Impact of Automatic Data Curation · p. 7
  • The moderate threshold πτ=0.15 achieves the best trade-off between feasibility and diversity and yields the lowest overall tracking error.It retains variability for generalization while excluding excessively difficult or unstable motions.Source: C. Impact of Automatic Data Curation · p. 7

D. Specialist Policy finetuning

Specialist policies are created by fine-tuning a filtered generalist policy on task-specific datasets rather than training from scratch. Fine-tuning improves specialized motion fidelity, especially as motion difficulty increases, while the comparison spans easy, moderate, hard, and out-of-distribution datasets.

Training strategies

Evaluation design

Results by difficulty

  • The finetuned policy achieves the best performance across easy, moderate, and hard datasets, with a larger advantage over scratch training on more challenging motions.Source: D. Specialist Policy finetuning · p. 8

Specialization

Overall conclusion

V. CONCLUSION

Exbody2 combines automated dataset filtering, generalist-specialist training, and decoupled keypoint-velocity tracking to improve expressive whole-body control. Its main limitation is that separately fine-tuned specialists cannot be seamlessly recombined, constraining transitions between motion types.

  • Exbody2 integrates automated dataset filtering, generalist-specialist training, and decoupled keypoint-velocity tracking for humanoid whole-body control.Source: §V · p. 9
  • The framework balances feasibility and diversity in dataset selection while improving robust tracking and generalization across diverse motion tasks.Source: §V · p. 9
  • Specialist fine-tuning refines performance for high-precision tasks within a structured pretrain-finetune pipeline.Source: §V · p. 9
  • Separately fine-tuned specialist policies cannot be seamlessly recombined, limiting flexibility when switching between motion groups during one tracking session.Source: §VI · p. 9
  • Dynamic blending or switching of specialists could unify broad generalist coverage with specialists’ high-accuracy tracking.Source: §VI · p. 9

APPENDIX

The appendix specifies Exbody2’s robot interface, observations, action space, baselines, and teacher-student training setup. The student policy uses longer histories and omits privileged information, while motion targets include keypoints, joint positions, and root movement.

E. Reward Design

The dataset study supports a feasibility-diversity principle: motion data should be diverse, especially in upper-body actions, while avoiding lower-body movements beyond robot limits. D250 provides the strongest balance, outperforming simpler or noisier alternatives across broader evaluations.

  • A useful humanoid tracking dataset must be diverse enough for generalization and feasible enough to avoid motions beyond mechanical limits.Source: E. Reward Design · pp. 14–15
  • D50 contains mostly fundamental static actions, making it feasible but insufficiently diverse in upper- and lower-limb motions.Source: E. Reward Design · p. 15
  • D250 adds upper-limb variations and moderately dynamic lower-body actions while excluding extreme motions difficult for the robot to replicate.Source: E. Reward Design · p. 15
  • The full 1,919-sequence CMU dataset includes extreme actions such as push-ups, rolling, and somersaults that introduce training noise.Source: E. Reward Design · p. 15
  • On D50, D50-trained policies achieve the highest in-distribution tracking accuracy, while DCMU-trained policies show a substantial drop.Source: E. Reward Design · p. 15
  • On DCMU and DACCAD, D250-trained policies achieve the best tracking accuracy, outperforming policies trained on D50 or full DCMU.Source: E. Reward Design · p. 15
  • Figure 7 compares diverse motions across SMPL, simulation, and real-world environments, including clapping, punching, crouching, and defensive poses.Source: Figure 7 · p. 16
  • D250 offers the best balance between feasible lower-body motions and diverse upper-body actions for robust, expressive whole-body control.Source: E. Reward Design · p. 16

F. Ablation on Policy Training

Ablations show that student history length and teacher-student distillation materially affect policy training. A history length of 10 performs best, while removing DAgger-style distillation severely degrades dynamic-motion tracking.

History length

Teacher–student distillation

G. Distribution-Guided Threshold Selection

The paper selects filtering thresholds from the base policy’s empirical error distribution rather than using arbitrary cutoffs. These thresholds define subsets that trade off motion feasibility against behavioral diversity.

Distribution analysis

  • The empirical CDF of base-policy error e(s) ranks motion sequences from lowest to highest error and supports data-driven threshold selection.The percentile axis spans 0% to 100%, while the vertical axis reports e(s).Source: G. Distribution-Guided Threshold… · p. 16

Threshold selection

  • Thresholds τ = 0.075, 0.10, 0.125, 0.15, 0.175 partition the dataset into subsets with varying feasibility and diversity.The thresholds are derived directly from the empirical error distribution.Source: G. Distribution-Guided Threshold… · p. 16

Feasibility-diversity trade-off

  • Smaller thresholds retain mostly lower-body motions with limited dynamics, whereas higher thresholds admit more dynamic behaviors but may include excessive-error samples.Samples with excessive errors could degrade policy learning.Source: G. Distribution-Guided Threshold… · p. 16

H. Real-world Results Visualization

ExBody2 is visualized across simulated, real-world, interactive, and generated-motion settings. The results show high-fidelity tracking, stable operation with monocular pose input, and support for extended motion sequences.

Real-world visualization

  • Figure 7 aligns reference SMPL poses, simulated robot poses, and real deployment snapshots to visualize whole-body tracking fidelity.The comparison includes lower-body poses important for balance.Source: H. Real-world Results Visualization · p. 16

Motion sources

  • ExBody2 supports motion capture, RGB-based real-time mimicry, motion synthesis, and possible VR or IMU streams as motion sources.These sources extend the policy toward interactive and long-horizon tasks.Source: H. Real-world Results Visualization · p. 16

Real-time mimicry

  • Monocular RGB tracking extracts 3D poses, retargets them to the robot, and feeds them into ExBody2 for stable lower-body tracking under partial or noisy signals.The pipeline uses HybrIK for per-frame 3D pose extraction.Source: §I · p. 17
  • Monocular pose estimates are lighter than full-body motion capture or multi-camera setups but can be less accurate than multi-view solutions.The policy is described as remaining stable under potential keypoint noise.Source: §I · p. 17

Motion synthesis

  • The CVAE generates motion segments conditioned on past poses, using continuity and smoothness objectives to avoid abrupt transitions.Generated frames are retargeted like regular motion-capture clips, enabling extended varied sequences without a fixed clip database.Source: J. Motion Synthesis for Extended… · p. 17

Threshold visualization

  • The empirical CDF visualization uses motion-sequence percentile on the horizontal axis and error e(s) on the vertical axis, with dashed lines marking filtering thresholds.The figure illustrates how feasible and infeasible motions are separated.Source: Figure 8 · p. 17

Related papers

Loading 2412.13196v2…