Source-linked AI summary
ExBody2: Advanced Expressive Humanoid Whole-Body Control
Mazeyu Ji, Xuanbin Peng, Fangchen Liu, Jialong Li, Ge Yang, Xuxin Cheng, Xiaolong Wang
TL;DR
Humanoid robots still struggle to imitate expressive whole-body human motion while remaining stable and robust across the robot–human dynamic and kinematic gap. Exbody2 addresses this with automated motion curation, generalist-to-specialist training, teacher–student distillation, and decoupled velocity and landmark tracking. It achieves broad, robust motion coverage and higher targeted fidelity, while specialist policies remain difficult to recombine smoothly.
Problem
Humanoid robots lack a reliable way to achieve human-like whole-body expressiveness and versatility while maintaining stability and control across their dynamic and kinematic gap with biological bodies.
Method
Exbody2 retargets human motion, filters infeasible lower-body sequences, trains a generalist policy, fine-tunes specialists, and uses decoupled motion–velocity control with teacher–student distillation.
Results
Exbody2 achieves higher fidelity than four state-of-the-art baselines in simulation and real-world tests, while fine-tuning further improves targeted motion quality and expressiveness.
Takeaways & Limitations
The generalist–specialist pipeline balances broad adaptability with focused accuracy for expressive humanoid whole-body tracking.
Takeaways & Limitations
Specialist policies cannot yet be seamlessly recombined, limiting smooth and efficient transitions between different motion types within one tracking session.
Abstract
from arXiv · showhide
This paper tackles the challenge of enabling real-world humanoid robots to perform expressive and dynamic whole-body motions while maintaining overall stability and robustness. We propose Advanced Expressive Whole-Body Control (Exbody2), a method for producing whole-body tracking controllers that are trained on both human motion capture and simulated data and then transferred to the real world. We introduce a technique for decoupling the velocity tracking of the entire body from tracking body landmarks. We use a teacher policy to produce intermediate data that better conforms to the robot's kinematics and to automatically filter away infeasible whole-body motions. This two-step approach enabled us to produce a student policy that can be deployed on the robot that can walk, crouch, and dance. We also provide insight into the trade-off between versatility and the tracking performance on specific motions. We observed significant improvement of tracking performance after fine-tuning on a small amount of data, at the expense of the others.
I. INTRODUCTION
Exbody2 addresses the challenge of producing expressive, versatile humanoid whole-body motions while maintaining stability across the robot–human dynamic and kinematic gap. Its generalist–specialist pipeline, automated data curation, and decoupled motion–velocity control improve adaptability, targeted fidelity, and robustness in simulation and real-world tests.
- Motivation: Whole-body humanoid control remains difficult because robots must reconcile high-dimensional nonlinear dynamics with the competing demands of expressiveness, versatility, stability, and control.Traditional model-based approaches require accurate system identification, physical modeling, and intensive online computation for real-time stability.
- Core approach: Exbody2 combines automated data curation, generalist–specialist policies, and decoupled motion–velocity control for expressive whole-body humanoid tracking.The framework is trained through a sim-to-real pipeline using retargeted human motion and simulated data before deployment on humanoid robots.
- Generalist policy: Automated curation removes infeasible lower-body motions while preserving upper-body diversity, balancing physical feasibility with broad expressive coverage.This addresses the mismatch between human motion datasets and robot capabilities while retaining diverse behaviors for generalist training.
- Specialist policy: Specialist policies fine-tune the generalist policy for targeted motion groups, improving precision while retaining priors learned from broad motion exposure.Motion labels or action recognition can support dynamic selection of an appropriate specialist policy.
- Motion–velocity control: Decoupling local keypoint tracking from velocity control lets velocity guide movement while keypoint tracking emphasizes expressive motion imitation.A teacher policy uses privileged simulation information, and DAgger-style student distillation infers that information from historical observations for real-world deployment.
- Evaluation: Exbody2 achieves higher fidelity than four state-of-the-art baselines across simulation and real-world Unitree G1 tests, with further quality gains from task-specific fine-tuning.The reported evaluations emphasize a robust single generalist policy for diverse motions and improved expressiveness for targeted behaviors such as dancing.
1) Generalist policy with automated data curation:
Exbody2 builds a generalist policy by filtering motion data according to tracking errors, preserving diversity while excluding infeasible motions. The resulting policy can then be fine-tuned into specialists for higher-precision motion categories.
- Generalist policy with automated data curation: The initial policy is trained on a diverse dataset containing many infeasible motions, then evaluated per sequence using lower-body tracking errors.Lower-body errors are emphasized because the lower body is central to dynamic feasibility and balance.
- Generalist policy with automated data curation: The filtering error combines lower-body keybody-position error and joint-angle tracking error to rank motion sequences.The keybody term helps prevent extreme deviations such as flipping or rolling, while the joint-angle term measures precision.
- Generalist policy with automated data curation: Moderate filtering thresholds produce the best-performing policy by balancing motion diversity against feasibility.Very small thresholds limit generalization, whereas very large thresholds retain infeasible motions that degrade training effectiveness.
- Generalist policy with automated data curation: The selected optimal threshold generalizes to other motion datasets, supporting robust training and improved performance.The threshold is selected by evaluating policies trained on subsets defined by different error thresholds.
- Specialist policy finetuning: Specialist policies are fine-tuned from the generalist policy for targeted motion groups rather than trained from scratch.This warm start supports higher-precision adaptation while retaining exposure to diverse motions and disturbances.
B. Policy Objective and Architecture
Exbody2 trains a privileged-information teacher policy in simulation, then distills it into a deployable student using real-world-aligned observations. The student is trained from historical observations to reproduce the teacher’s control actions.
- Policy Objective and Architecture: The teacher policy is trained with PPO using privileged simulator information, while the student replaces that information with observations aligned with real-world deployment.Privileged information includes ground-truth robot and environment states unavailable outside simulation.
- Policy Objective and Architecture: The humanoid control problem is formulated as an MDP in which the teacher receives privileged states, proprioception, and motion targets and outputs joint-position targets.The action is a 23-dimensional target for joint proportional-derivative controllers.
- Policy Objective and Architecture: Motion targets include desired joints and 3D keypoints together with target root velocity and root pose.These targets support whole-body motion tracking under joystick control.
- Policy Objective and Architecture: The student encodes a history of observations and the motion target, then learns to reproduce the teacher’s actions with mean squared error.The student therefore operates without the teacher’s privileged information.
- Policy Objective and Architecture: Student training uses iterative rollouts in simulation, with the teacher providing oracle actions for visited states until the distillation loss converges.This procedure follows the DAgger strategy and accumulates training data from the student’s own states.
3) Motion-velocity Decoupled Control Strategy:
Exbody2 decouples local keypoint motion imitation from velocity control to avoid failures caused by drifting global keypoints. The evaluation compares this strategy with prior whole-body tracking methods using velocity, keypoint-position, and joint-position errors.
- Motion-velocity decoupling: Exbody2 maps global keypoints into the robot’s current coordinate frame and uses velocity-based global tracking instead of direct global-keypoint tracking.This separates movement guidance from key body tracking for motion imitation.
- Limitations of prior tracking: Global keypoint tracking can accumulate errors over time when reference keypoints drift, hindering learning and producing suboptimal or failed behavior.
- Experimental comparison: The experiments compare Exbody2 with Exbody, whole-body Exbody, and OmniH2O using simulation and real-world evaluations on the Unitree G1.
- Method comparison: Exbody2 uses local keypoint tracking, data curation, and training techniques intended to improve motion fidelity and sim-to-real transfer.
- Evaluation metrics: The evaluation measures root linear velocity error, keybody-position error, and joint-position error, including separate upper- and lower-body metrics.The reported metrics assess velocity tracking, keypoint tracking, and joint tracking across body regions.
B. Generalist Policy Performance
Exbody2’s generalist policy outperforms prior baselines across simulation and real-world whole-body tracking, while automated filtering improves stability and precision. The filtering results show that moderate thresholds best balance motion diversity with feasibility, although filtered training slightly increases velocity error relative to unfiltered training.
- Simulation performance: Exbody2 surpasses Exbody, Exbody†, and OmniH2O* across all reported simulation metrics, with motion filtering providing additional upper- and lower-body gains.
- Simulation performance: Lower-body accuracy improves global stability and consequently enables more precise upper-body control.
- Trade-off: Filtered training slightly increases velocity tracking error compared with unfiltered training, but the reported stability and precision gains outweigh this trade-off.The comparison is reported under simulation conditions.
- Real-world performance: Exbody2 achieves higher upper- and lower-body tracking accuracy than baselines in real-world experiments on diverse motions including walking, squatting, and dancing.
- Automatic data curation: Low thresholds limit generalization through overly simple motion data, whereas high thresholds introduce unstable motions and training noise that reduce tracking accuracy.
- Automatic data curation: The moderate threshold πτ=0.15 achieves the best trade-off between feasibility and diversity and yields the lowest overall tracking error.It retains variability for generalization while excluding excessively difficult or unstable motions.
D. Specialist Policy finetuning
Specialist policies are created by fine-tuning a filtered generalist policy on task-specific datasets rather than training from scratch. Fine-tuning improves specialized motion fidelity, especially as motion difficulty increases, while the comparison spans easy, moderate, hard, and out-of-distribution datasets.
- Training strategies: The Specialist policy fine-tunes the pretrained Generalist policy, whereas Scratch trains directly on the same task-specific datasets.
- Evaluation design: The comparison evaluates Generalist, Specialist, and Scratch strategies across Deasy, Dmoderate, Dhard, and the OOD DACCAD dataset.
- Results by difficulty: The finetuned policy achieves the best performance across easy, moderate, and hard datasets, with a larger advantage over scratch training on more challenging motions.
- Specialization: Task-specific finetuning captures fine-grained motion details while maintaining stability.
- Overall conclusion: The pretrain-finetune paradigm combines broad generalist coverage with task specialization and shows benefits on challenging and out-of-distribution scenarios.
V. CONCLUSION
Exbody2 combines automated dataset filtering, generalist-specialist training, and decoupled keypoint-velocity tracking to improve expressive whole-body control. Its main limitation is that separately fine-tuned specialists cannot be seamlessly recombined, constraining transitions between motion types.
- Exbody2 integrates automated dataset filtering, generalist-specialist training, and decoupled keypoint-velocity tracking for humanoid whole-body control.
- The framework balances feasibility and diversity in dataset selection while improving robust tracking and generalization across diverse motion tasks.
- Specialist fine-tuning refines performance for high-precision tasks within a structured pretrain-finetune pipeline.
- Separately fine-tuned specialist policies cannot be seamlessly recombined, limiting flexibility when switching between motion groups during one tracking session.
- Dynamic blending or switching of specialists could unify broad generalist coverage with specialists’ high-accuracy tracking.
APPENDIX
The appendix specifies Exbody2’s robot interface, observations, action space, baselines, and teacher-student training setup. The student policy uses longer histories and omits privileged information, while motion targets include keypoints, joint positions, and root movement.
- Training observations include proprioceptive states, privileged information, and motion-tracking targets; the student uses longer history because it lacks privileged information.
- Motion-tracking targets contain keypoint positions, joint positions, and root movement information.
- The action space consists of 23-dimensional target joint positions for proportional-derivative controllers on the Unitree G1.
- Exbody tracks upper-body keypoints and joint positions, whereas Exbody† tracks full-body keypoints and joint positions.
- OmniH2O* relies on global tracking during training and does not use robot velocity as privileged information.
- Exbody2 trains a privileged-information teacher with PPO and a longer-history student with DAgger.
E. Reward Design
The dataset study supports a feasibility-diversity principle: motion data should be diverse, especially in upper-body actions, while avoiding lower-body movements beyond robot limits. D250 provides the strongest balance, outperforming simpler or noisier alternatives across broader evaluations.
- A useful humanoid tracking dataset must be diverse enough for generalization and feasible enough to avoid motions beyond mechanical limits.
- D50 contains mostly fundamental static actions, making it feasible but insufficiently diverse in upper- and lower-limb motions.
- D250 adds upper-limb variations and moderately dynamic lower-body actions while excluding extreme motions difficult for the robot to replicate.
- The full 1,919-sequence CMU dataset includes extreme actions such as push-ups, rolling, and somersaults that introduce training noise.
- On D50, D50-trained policies achieve the highest in-distribution tracking accuracy, while DCMU-trained policies show a substantial drop.
- On DCMU and DACCAD, D250-trained policies achieve the best tracking accuracy, outperforming policies trained on D50 or full DCMU.
- Figure 7 compares diverse motions across SMPL, simulation, and real-world environments, including clapping, punching, crouching, and defensive poses.
- D250 offers the best balance between feasible lower-body motions and diverse upper-body actions for robust, expressive whole-body control.
F. Ablation on Policy Training
Ablations show that student history length and teacher-student distillation materially affect policy training. A history length of 10 performs best, while removing DAgger-style distillation severely degrades dynamic-motion tracking.
- History length: A history length of 10 yields the best student-policy results among tested non-zero histories.
- History length: Longer histories make privileged-information fitting harder and ultimately reduce tracking performance.
- Teacher–student distillation: Removing DAgger-style distillation severely degrades performance because the student must learn velocity tracking directly from raw observations.
- Teacher–student distillation: Without privileged velocity guidance, the student has greater difficulty tracking fast or dynamic motions accurately.
G. Distribution-Guided Threshold Selection
The paper selects filtering thresholds from the base policy’s empirical error distribution rather than using arbitrary cutoffs. These thresholds define subsets that trade off motion feasibility against behavioral diversity.
- Distribution analysis: The empirical CDF of base-policy error e(s) ranks motion sequences from lowest to highest error and supports data-driven threshold selection.The percentile axis spans 0% to 100%, while the vertical axis reports e(s).
- Threshold selection: Thresholds τ = 0.075, 0.10, 0.125, 0.15, 0.175 partition the dataset into subsets with varying feasibility and diversity.The thresholds are derived directly from the empirical error distribution.
- Feasibility-diversity trade-off: Smaller thresholds retain mostly lower-body motions with limited dynamics, whereas higher thresholds admit more dynamic behaviors but may include excessive-error samples.Samples with excessive errors could degrade policy learning.
H. Real-world Results Visualization
ExBody2 is visualized across simulated, real-world, interactive, and generated-motion settings. The results show high-fidelity tracking, stable operation with monocular pose input, and support for extended motion sequences.
- Real-world visualization: Figure 7 aligns reference SMPL poses, simulated robot poses, and real deployment snapshots to visualize whole-body tracking fidelity.The comparison includes lower-body poses important for balance.
- Motion sources: ExBody2 supports motion capture, RGB-based real-time mimicry, motion synthesis, and possible VR or IMU streams as motion sources.These sources extend the policy toward interactive and long-horizon tasks.
- Real-time mimicry: Monocular RGB tracking extracts 3D poses, retargets them to the robot, and feeds them into ExBody2 for stable lower-body tracking under partial or noisy signals.The pipeline uses HybrIK for per-frame 3D pose extraction.
- Real-time mimicry: Monocular pose estimates are lighter than full-body motion capture or multi-camera setups but can be less accurate than multi-view solutions.The policy is described as remaining stable under potential keypoint noise.
- Motion synthesis: The CVAE generates motion segments conditioned on past poses, using continuity and smoothness objectives to avoid abrupt transitions.Generated frames are retargeted like regular motion-capture clips, enabling extended varied sequences without a fixed clip database.
- Threshold visualization: The empirical CDF visualization uses motion-sequence percentile on the horizontal axis and error e(s) on the vertical axis, with dashed lines marking filtering thresholds.The figure illustrates how feasible and infeasible motions are separated.