Source-linked AI summary
GMT: General Motion Tracking for Humanoid Whole-Body Control
Zixuan Chen, Mazeyu Ji, Xuxin Cheng, Xuanbin Peng, Xue Bin Peng, Xiaolong Wang
TL;DR
General whole-body motion tracking for real-world humanoids is difficult because diverse motions must be handled under partial observability, hardware constraints, imbalanced data, and limited policy expressiveness. GMT combines Adaptive Sampling with a Motion Mixture-of-Experts architecture in a single unified policy, achieving state-of-the-art performance across diverse real-world motions while remaining limited in contact-rich skills such as getting up or rolling.
Problem
Unified general motion tracking remains challenging because real-world humanoids face partial observability, hardware limitations, imbalanced motion data, and insufficient model expressiveness.
Method
GMT trains a single unified policy using Adaptive Sampling to address uneven motion frequencies and Motion MoE to improve model expressiveness and specialization.
Results
GMT achieves state-of-the-art performance as a single unified policy across diverse motions in simulation and real-world deployment.
Takeaways & Limitations
The resulting general controller can serve as a foundation for future whole-body algorithm development on humanoid robots.
Takeaways & Limitations
GMT does not currently support contact-rich skills such as getting up from a fallen state or rolling on the ground.
Abstract
from arXiv · showhide
The ability to track general whole-body motions in the real world is a useful way to build general-purpose humanoid robots. However, achieving this can be challenging due to the temporal and kinematic diversity of the motions, the policy's capability, and the difficulty of coordination of the upper and lower bodies. To address these issues, we propose GMT, a general and scalable motion-tracking framework that trains a single unified policy to enable humanoid robots to track diverse motions in the real world. GMT is built upon two core components: an Adaptive Sampling strategy and a Motion Mixture-of-Experts (MoE) architecture. The Adaptive Sampling automatically balances easy and difficult motions during training. The MoE ensures better specialization of different regions of the motion manifold. We show through extensive experiments in both simulation and the real world the effectiveness of GMT, achieving state-of-the-art performance across a broad spectrum of motions using a unified general policy. Videos and additional information can be found at https://gmt-humanoid.github.io.
1 Introduction
GMT addresses the open problem of unified whole-body motion tracking for real-world humanoids, where partial observability, hardware mismatches, imbalanced motion data, and limited model expressiveness create distinct challenges.
- Motivation: Humanoid robots need general whole-body controllers spanning basic locomotion and dynamic skills to support broad everyday task repertoires.Such controllers could support high-level planners that select and sequence skills autonomously.
- Challenges: Partial observability prevents real-world robots from accessing full state information, including linear velocities and global root positions important for motion tracking.This missing information makes policy training more challenging.
- Challenges: Hardware limitations make some human motions infeasible and can prevent robots from matching the torque, speed, and dynamics of walking or running.These mismatches require special handling during controller training.
- Challenges: AMASS contains highly unbalanced motion categories, with walking and in-place activities dominating while complex dynamic motions are scarce.This distribution can hinder mastery of less frequent but critical skills.
- Challenges: Simple MLP policies often lack the capacity to model complex temporal dependencies and distinguish diverse motion categories in large mocap datasets.The resulting limitations can reduce tracking performance and generalization across skills.
- GMT: GMT jointly addresses data distribution and model problems while incorporating designs for partial observability and hardware issues to train one unified policy.Its Adaptive Sampling strategy targets uneven motion frequencies, while Motion MoE enhances expressiveness and generalizability.
2 Related Works
Prior work includes model-based and learning-based whole-body control, simulated general motion tracking, and real-robot imitation, but real-world unified control remains constrained by observability and coordination challenges.
- Whole-Body Control: Traditional model-based whole-body controllers can provide robust locomotion but are labor-intensive and require gait planning and dynamics modeling.The paper contrasts these approaches with learning-based methods for broader motion repertoires.
- Whole-Body Control: Manually rewarded policies have enabled locomotion, jumping, and fall recovery, but are typically task-specific and require separate policies for different tasks.Walking policies are not easily transferable to jumping or manipulation.
- Motion Imitation: Human motion data has supported high-quality general tracking and diverse skills in simulated characters.These results motivate transferring motion-data-driven control to humanoid robots.
- GMT: GMT is positioned as a unified framework intended to overcome the limitations of separate task policies and prior real-robot motion-imitation approaches.The supplied overview identifies motion targets, proprioceptive observations, and privileged information as GMT inputs.
- Humanoid Motion Imitation: Real-robot whole-body imitation is difficult because of partial observability, and some methods decouple upper and lower bodies to trade expressiveness for balance.HumanPlus and OmniH2O enabled whole-body imitation but produced unnatural lower-body movements, while ExBody2 improved tracking.
3 Learning General Motion Tracking Controllers
GMT combines two-stage teacher-student training with Adaptive Sampling and a Motion Mixture-of-Experts architecture to train general motion-tracking controllers from curated mocap data.
- Training Framework: GMT first trains a privileged teacher with PPO and then trains a student with DAgger to imitate the teacher’s outputs.This follows a two-stage teacher-student framework used in prior work.
- Adaptive Sampling: Adaptive Sampling addresses category imbalance and composite sequences that otherwise oversample easy motion segments when harder segments cause tracking failures.The method is introduced specifically to increase attention to difficult portions of long motions.
- Adaptive Sampling: Random Clipping divides motions longer than 10 seconds into sub-clips of at most 10 seconds, using offsets up to 2 seconds and periodic re-clipping.These choices diversify sampled sub-clips and reduce transition artifacts.
- Adaptive Sampling: Tracking Performance-based Probabilities log each motion’s completion level, terminate episodes after excessive error, and normalize sampling levels into probabilities.The completion level starts at 10, decays by 0.99 after completion, and has a minimum of 1.
- Adaptive Sampling: Adaptive Sampling focuses training on harder motions with higher tracking errors instead of repeatedly sampling easy segments.The strategy is applied from the start of training.
- Motion Mixture-of-Experts: Motion MoE uses expert networks and a gating network that combine expert action distributions according to gating probabilities.The experts and gate receive robot observations and motion targets, while the final action is a weighted combination of expert outputs.
- Data Curation: The policy is trained on curated AMASS and LAFAN1 data after filtering infeasible motions that could hamper learning.The curation removes motions such as crawling, fallen states, and extremely dynamic movements that exceed hardware constraints.
- Motion Inputs: GMT stacks future motion frames covering approximately two seconds, compresses them into z_t ∈R128, and combines that representation with the immediate target frame.This supplies both long-term motion trends and the immediate tracking target.
4 Experiments
GMT is evaluated in simulation using large-scale randomized training and tested on AMASS and LAFAN1, with comparisons and ablations performed on privileged policies.
- Simulation Setup: Each simulation policy uses approximately 6.8 billion samples with domain randomization and action delay on filtered AMASS and LAFAN1 data.Training uses IsaacGym with 4096 parallel environments and evaluation includes AMASS test and LAFAN1 datasets.
- Evaluation Protocol: The simulation experiments evaluate policies on AMASS test data and LAFAN1, while baseline comparisons and ablations focus on privileged policies.The supplied passage states the evaluation setup but does not report numerical results.
4.2 Baselines
GMT is compared with ExBody2 in simulation using the filtered dataset, outperforming it across local and global tracking measures.
- GMT outperforms ExBody2 on both local and global tracking performance in simulation.The comparison uses an ExBody2 reimplementation trained on GMT’s filtered dataset.
- Local tracking is evaluated with Empkpe and Empjpe.
- Global tracking is evaluated with Evel and Eyaw vel.
4.3 Ablation Studies
Ablations examine Motion MoE, Adaptive Sampling, and motion-input configurations, showing improved performance from expert specialization, adaptive training, and suitable temporal inputs.
- Motion MoE: Motion MoE improves tracking especially on challenging motions and yields expert transitions across distinct phases of composite motion sequences.The gating network activates different experts across standing, kicking, backward walking, and standing phases.
- Adaptive Sampling: Adaptive Sampling improves tracking on both datasets, particularly for challenging motions.Without Adaptive Sampling, the policy fails to learn a tested segment with high quality and struggles to balance, preventing real-world deployment.
- Motion Inputs: Increasing the motion-input window length improves tracking accuracy.
- Motion Inputs: GMT-L2 degrades significantly, indicating that the immediate next frame is crucial alongside future motion frames.Future-frame windows capture upcoming motion tendencies but can lose detailed information.
4.4 Real-World Deployment
GMT is deployed on a real-world humanoid robot to reproduce a broad range of human motions with high fidelity and state-of-the-art performance.
- GMT reproduces stylized walking, high kicking, dancing, spinning, crouch walking, and soccer kicking on a real-world humanoid robot.The deployment also demonstrates many other human motions.
- The deployed policy achieves high fidelity and state-of-the-art performance across these motions.
4.5 Applications - Tracking MDM-Generated Motions
GMT is tested on motions generated by a motion diffusion model in MuJoCo sim-to-sim experiments, where it performs well on text-prompted motions.
- GMT performs well on MDM-generated motions in MuJoCo sim-to-sim settings.
- The motions are generated from text prompts by a motion diffusion model.
- The results indicate potential application to other downstream tasks.
5 Conclusion
GMT is a general, scalable framework that trains one unified policy to imitate diverse humanoid motions in the real world. Experiments and deployment show broad tracking capability and state-of-the-art performance compared with prior work.
- GMT trains a single unified policy for diverse real-world humanoid motion imitation.
6 Limitations
GMT remains limited in contact-rich skills and challenging terrain settings. The current framework does not support several recovery and ground-contact behaviors and lacks terrain observations for slopes and stairs.
- Contact-rich skills such as getting up after falling and rolling are not currently supported.The limitation reflects added simulation complexity and hardware constraints.
- The policy is not designed to imitate motions on challenging terrains such as slopes and stairs.It is trained without terrain observations; future work targets flat and challenging terrains.
Appendix
The appendix defines the goal-conditioned reinforcement-learning formulation, sim-to-real adjustments, training pipeline, observations, actions, rewards, simulation setup, and domain randomization used by GMT.
- 6.1 Goal-Conditioned Reinforcement Learning: Motion tracking is formulated as goal-conditioned reinforcement learning, with policy π mapping state and goal inputs to actions that produce rewards over trajectories.The objective is to maximize expected discounted return over the time horizon.
- 6.1 Goal-Conditioned Reinforcement Learning: The expected return uses trajectory likelihood, a time horizon T, and discount factor γ.The appendix identifies p(τ|π) as trajectory likelihood, T as the time horizon, and γ as the discount factor.
- 6.2 Sim-to-Real Transfer: Sim-to-real transfer combines domain randomization with simulator armature set to k^2I to approximate reduction-drive inertia.Here k is the reduction ratio and I is the reduction-drive moment of inertia.
- 6.3 Training Framework: Training uses a privileged teacher optimized with PPO followed by a deployable student trained from proprioceptive history through DAgger.The teacher observes privileged information, while the student is supervised by the teacher policy.
- 6.4 Observations and Actions: The student receives proprioceptive history and motion targets, and the policy outputs target joint positions.Teacher and student observation contents are specified separately, with the student using a sequence of past observations.
- 6.5 Simulation Setup: Experiments use IsaacGym with 4096 parallel environments, 500Hz simulation, 50Hz control, and Mujoco validation before real-robot deployment.The privileged policy trains for around 3 days and the student for around 1 day on an RTX4090 GPU.