Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking

Zekun Qi; Xuchuan Chen; Dairu Liu; Chenghuai Lin; Yunrui Lian; Sikai Liang; Zhikai Zhang; Yu Guan; Jilong Wang; Wenyao Zhang; Xinqiang Yu; He Wang; Li Yi

Source-linked AI summary

Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking

Zekun Qi, Xuchuan Chen, Dairu Liu, Chenghuai Lin, Yunrui Lian, Sikai Liang, Zhikai Zhang, Yu Guan, Jilong Wang, Wenyao Zhang, Xinqiang Yu, He Wang, Li Yi

arXiv:2606.03985v1Jun 2, 2026cs.ROcs.AIcs.CV

TL;DR

Humanoid motion trackers have been limited by shallow models and small motion corpora, leaving robust whole-body generalization to unseen tasks insufficiently addressed. Humanoid-GPT scales data and model capacity with a 2B-frame corpus, causal Transformer tracking, and diversity-balanced training, achieving improved agility and zero-shot generalization while tracking dynamic motions.

Problem
Humanoid motion tracking has relied on shallow models and small corpora, limiting evidence for robust whole-body generalization to unseen tasks, styles, and environments.
Method
Humanoid-GPT combines a 2B-frame retargeted motion corpus, GPT-style causal Transformer tracking, and diversity-balanced sampling based on Harmonic Motion Embedding.
Results
Humanoid-GPT substantially improves agility and zero-shot generalization while tracking highly dynamic motions across unseen tasks.
Takeaways & Limitations
Scaling data, model capacity, and motion diversity balance establishes a performance frontier for general-purpose whole-body control.

Abstract

from arXiv · show

We introduce Humanoid-GPT, a GPT-style Transformer with causal attention trained on a billion-scale motion corpus for whole-body control. Unlike prior shallow MLP trackers constrained by scarce data and an agility-generalization trade-off, Humanoid-GPT is pre-trained on a 2B-frame retargeted corpus that unifies all major mocap datasets with large-scale in-house recordings. Scaling both data and model capacity yields a single generative Transformer that tracks highly dynamic behaviors while achieving unprecedented zero-shot generalization to unseen motions and control tasks. Extensive experiments and scaling analyses show that our model establishes a new performance frontier, demonstrating robust zero-shot generalization to unseen tasks while simultaneously tracking highly dynamic and complex motions.

1 Introduction

Humanoid motion tracking faces an agility–generalization trade-off because existing shallow MLP trackers rely on small corpora and mismatched training designs. Humanoid-GPT addresses this with scaled data, GPT-style causal structure, diversity-balanced sampling, and systematic scaling analysis.

Motivation: Existing trackers trained on small motion corpora trade off agile in-domain tracking against zero-shot generalization to unseen movements.BeyondMimic and ASAP track agile motions well but lack zero-shot generalization, whereas TWIST and UniTracker generalize better but struggle with highly dynamic actions.
Contribution: Humanoid-GPT is a universal online humanoid motion tracker built around scaling data, model structure, and training design.The paper frames large-scale tracking around data processing, online-compatible structure, and stable training as datasets grow from millions to billions of frames.
Science of Scale: 2B G1-retargeted motion frames/tokens are assembled from widely available mocap sources and internal recordings after filtering, segmentation, and augmentation.The corpus is over 200× larger than prior tracker training sets and includes Lafan1, AMASS, Motion-X++, PHUMA, and MotionMillion.
Modern Structure for Online Tracking: A GPT-style Transformer uses causal temporal attention to predict per-joint PD targets, matching online deployment constraints while scaling with data and model size.The causal design avoids access to future observations during test-time tracking.
Balanced Diversity Matters: Harmonic Motion Embedding measures motion diversity and enables diversity-aware, distribution-balanced sampling to prevent common styles from overwhelming rare behaviors.The paper emphasizes that increasing data alone does not automatically improve generalization when large corpora contain imbalanced style distributions.
Results and Scaling Laws: Humanoid-GPT improves agility and zero-shot generalization while deriving a scaling law relating tracking performance to data scale and model capacity.The system trains on a curated 2B-frame corpus and characterizes how data scale, model scale, and diversity balance jointly govern zero-shot agile tracking on real humanoid hardware.

2 Related work

Related work progresses from increasingly diverse motion datasets and physics-based tracking toward generalizable controllers. Existing approaches remain limited by motion scale, MLP capacity, or training procedures, motivating Humanoid-GPT’s causal Transformer sequence modeling on 2B frames.

Motion datasets: Large-scale datasets evolved from studio-constrained motions toward diverse activities, styles, subjects, and multimodal supervision through video reconstruction and synthetic generation.Recent work also provides physically consistent motions with contact modeling.
Physics-based tracking: Physics-based tracking couples imitation with contact-aware stability to produce temporally coherent, dynamically feasible whole-body control, with later pipelines targeting real-world deployment.These approaches have been demonstrated on specific platforms.
Generalizable tracking: GMT and UniTracker improve coverage through adaptive sampling and CVAE-based teacherstudent learning, but remain constrained by limited motion scale.GMT uses Mixture-of-Experts, while UniTracker adopts a CVAE-based teacherstudent framework.
Generalizable tracking: SONIC scales to 100M frames with an MLP controller, but MLP capacity saturates as data grows; HumanPlus uses a Transformer with standard PPO, missing Transformer parallelism.These limitations illustrate tradeoffs between data scaling, controller capacity, and training procedure.
Humanoid-GPT: Humanoid-GPT reframes tracking as GPT-style sequence modeling, distilling hundreds of RL experts into a causal Transformer trained on 2B frames for strong zero-shot generalization.The paper contrasts this approach with similarly sized MLPs that plateau.

3 Scaling up Humanoid Motion Data

The section presents a pipeline for building a high-quality, diverse humanoid motion corpus: curate and unify human motion data, retarget it into humanoid joint space, filter incompatible interactions, and cluster motions for efficient training.

Motion data curation: Large-scale motion curation targets fidelity and diversity because existing datasets have limited motion categories, physical-plausibility inconsistencies, and spatial-alignment problems.These limitations constrain generalization to complex whole-body tracking scenarios.
Humanoid retargeting: Retargeting maps each human sequence into the Unitree-G1 humanoid’s 29-DoFs joint space.Sequences involving explicit object interactions, including sitting, swimming, or stair climbing, are filtered for compatibility with plain-scene actuation.
Motion clustering: Harmonic Motion Embedding aggregates per-joint periodic amplitudes and frequencies into compact sequence vectors for latent-space clustering.Periodic Autoencoders extract the harmonic features before mean and standard-deviation aggregation.
Motion clustering: Roughly 300 motion clusters, each containing about 1k–2k sequences, balance motion coverage and training efficiency while preserving intra-cluster consistency.K-Means clustering uses pairwise distances between HME embeddings.

4 Scalable Generative Tracker

Humanoid-GPT uses RL-trained motion experts followed by Transformer distillation to produce a single policy that tracks arbitrary human motions without finetuning. Its causal sequence modeling and multi-timestep DAgger supervision support stable, physically consistent control with limited episode history.

Framework: Humanoid-GPT uses a two-stage pipeline of RL-trained motion experts and Transformer distillation to track arbitrary human motions without finetuning.The distillation stage consolidates all expert behaviors into a single unified policy.
Motion Experts: Multiple motion experts are trained to collectively cover the dataset’s dynamic motion distribution, with each PPO policy tracking sequences within one cluster.Policies map reference joints and proprioceptive observations to low-level motor actions.
Motion Experts: The experts optimize keypoint-level position, orientation, and velocity consistency while enforcing balance, dynamic stability, self-contact avoidance, and smoothness.The reward combines body-keypoint tracking terms with penalties that promote globally accurate and locally stable motion.
Transformer Distillation: DAgger distills all expert behaviors into a single Transformer generalist tracker that consumes proprioceptive state and reference-pose token histories under a temporal causal mask.The model captures long-horizon dependencies and temporal consistency across trajectories.
Transformer Distillation: Parallel sequence supervision and autoregressive temporal prediction enable stable, physically consistent control even at episode beginnings with scarce historical information.During inference, the current control target is taken from the final position of a maximal-length history-token queue.

5 Experiments

Experiments in MuJoCo and on the 29-DoF Unitree-G1 evaluate how motion-data diversity, Transformer capacity, and architecture affect zero-shot humanoid tracking. Humanoid-GPT scales consistently with data and model size, transfers difficult unseen motions to hardware, and runs with under 1.5ms inference latency on an NVIDIA RTX 4090 GPU.

Diversity and Generalization: The curated dataset shows approximately 4−5 higher log-volume than AMASS, indicating broader latent coverage and richer motion diversity.Diversity is measured using HME embeddings, geometric mean standard deviation, and covariance-ellipsoid log-volume with 10,000 uniformly sampled embeddings per dataset.
Evaluation Setup: Experiments use MuJoCo for controlled quantitative evaluation and the 29-DoF Unitree-G1 for real-world tracking.The real-world pipeline continuously retargets MoCap actor motion into the G1’s joint space.
Scaling Law in Humanoid Tracking: Humanoid-GPT exhibits consistent gains in tracking accuracy and stability as both motion-corpus size and Transformer capacity increase.The largest Humanoid-GPT-L model trained on 2B tokens achieves the best performance across nearly all metrics, while MLP and TCN baselines show data-scaling saturation.
Real-World Generalization: On the real Unitree-G1, Humanoid-GPT tracks entirely unseen, highly dynamic dancing sequences in real time without task-specific fine-tuning.Motor-sensor analysis confirms tracking stability and smooth torque regulation, while live MoCap teleoperation reproduces diverse actions while maintaining balance and fluid transitions.
Deployment Efficiency: Under 1.5ms end-to-end inference latency is achieved on a single NVIDIA RTX 4090 GPU after TensorRT and C++ streaming optimizations.The deployment pipeline reduces communication latency as well as computing and memory-access costs.

6 Scaling Laws

Formal scaling analysis quantifies how training-data scale and model architecture affect generalization. Increasing tokens improves zero-shot performance, while the Transformer scales more effectively than a comparable MLP, whose gains saturate early.

Data Scaling: Training on 2M, 20M, 200M, and 2B tokens reveals improving zero-shot performance as data scale increases.The subsets are sampled without overlap from the 2B-frame corpus using Humanoid-GPT-B.
Data Scaling: Marginal gains decrease slightly between 200M and 2B tokens, suggesting the current model capacity is entering a data-limited regime.
Model Scalability: With 2B tokens and comparable parameter size, the Transformer continues improving steadily as training progresses, whereas the MLP saturates early.This comparison demonstrates the scalability of Humanoid-GPT.

7 Conclusion & Future Work

Humanoid-GPT is a GPT-style humanoid motion tracker that scales data and model capacity to unify agility, stability, and zero-shot generalization. Simulation and real Unitree-G1 experiments demonstrate reliable realtime whole-body imitation without fine-tuning, while future work targets richer modalities and broader extensions.

Conclusion: Humanoid-GPT is a GPT-style humanoid motion tracker built by scaling both data and model capacity.
Conclusion: Billion-frame motion curation, clustered expert training, and Transformer-based distillation enable unified agility, stability, and zero-shot generalization.
Conclusion: Simulation and real Unitree-G1 experiments show strong transfer without fine-tuning and reliable realtime whole-body imitation.
Future Work: Future work includes incorporating contacts, vision, or language and extending the system further.

A Summary of Contributions

The paper contributes a 2B-frame, zero-shot motion tracker built with scalable causal Transformer structure and diversity-aware, distribution-balanced sampling. It shows that large-scale training, causal modeling, and balanced diversity are central to general tracking.

Science of Scale: The tracker is the first with zero-shot ability trained on 2BFrame data, using a dataset over 200× larger than prior trackers.Scaling by two orders of magnitude required redesigning the reward and retuning key hyperparameters.
Modern Structure: A scalable causal Transformer fits online tracking’s inability to access future observations and scales better than MLP and non-causal variants.The causal design matches the online tracking constraint.
Balanced Diversity Matters: HME Representation Learning applies diversity-aware, distribution-balanced sampling to motion tracking.The method is designed to support a general tracker.
Balanced Diversity Matters: Diversity and balance are both critical for a general tracker.This finding motivates the use of diversity-aware, distribution-balanced sampling.

B Additional Ablation Studies … B.3 Environment Number for DAgger Rollout

The ablations show that motion-expert granularity, Transformer history length, and DAgger environment count materially affect training and tracking. The selected configuration uses roughly 384 experts, 32 historical frames, and 32K environments to balance fidelity, compute, and generalization.

B.1 Number of Experts and Cluster Granularity: Extremely coarse clustering produces heterogeneous experts that harm teacher tracking fidelity.The example given is 128 experts.
B.1 Number of Experts and Cluster Granularity: Overly fine clustering increases training cost and creates conflicting student guidelines.The example given is 1024 experts.
B.1 Number of Experts and Cluster Granularity: Roughly C ≈384 experts offers the best balance among motion diversity, per-cluster coherence, and compute.This configuration was identified as the preferred balance in the ablation.
B.2 History Length of Transformer: Transformer performance continues improving with sequence history up to 64 frames.The experiment uses a Base-sized model under a controlled single-factor setting.
B.2 History Length of Transformer: The model adopts 32 historical frames by default because computation increases quadratically with sequence length.The default sacrifices longer-history gains to limit computational cost.
B.3 Environment Number for DAgger Rollout: As training data scales, DAgger requires more environments, leading to an adopted total of 32K environments.The authors hypothesize that too few environments can cause overfitting and forgetting when reference motions are numerous.

C Implementation and Reproducibility Details

This section provides implementation details omitted from the main paper, covering reinforcement-learning hyperparameters, DAgger schedules, and complete compute accounting.

Implementation Details: The section reports reinforcement-learning hyperparameters that were omitted from the main paper.These details expand the method description.
Implementation Details: It specifies the DAgger schedules used in the method.The schedules are included among the expanded method details.
Reproducibility Details: It provides full compute accounting for reproducibility.The accounting supplements details that could not fit in the main paper.

C.1 RL Expert Training Details … C.5 Deployment Details and Latency Measurements

The appendix details robust PPO expert training, DAgger distillation, compute usage, feature-distribution coverage, and real-time deployment configuration. The pipeline uses randomized simulation environments, discards the expert library after distillation, and runs the deployed policy at 50 Hz.

C.1 RL Expert Training Details: PPO experts train in MuJoCo with randomized 600–1200-frame episodes and a 50 Hz control loop matched by downsampled reference motions.Episodes terminate on falls, excessive joint-limit violations, or timeout.
C.1 RL Expert Training Details: Domain randomization perturbs terrain, external forces, and physical properties across all PPO environments and the DAgger stage.Physical perturbations include DoF friction, armature, torso center of mass and mass, and per-DoF position jitter; force intervals and magnitudes use uniform distributions.
C.2 DAgger Distillation Schedule: DAgger alternates student rollouts, expert action queries on the same state, student action matching, and environment updates using the student’s executed action.The maximal history length H is fixed to 32, with buffers for teacher actions and student observations.
C.2 DAgger Distillation Schedule: Behaviour Cloning uses a batch size no smaller than the number of experts to reduce mode collapse when experts cover partial behavior distributions.The same randomized environment configuration is used during DAgger so demonstrations capture the expert’s dynamic variations.
C.3 Compute Cost Breakdown: The reported total compute budget is roughly 15,000 GPU hours, divided between expert training and Transformer distillation.After training, only the distilled Humanoid-GPT policy is required at deployment; the expert library can be discarded.
C.4 T-SNE feature distribution: t-SNE visualization shows that the dataset covers a substantially broader feature-distribution region than AMASS+LAFAN1.The visualization is provided in Fig. 11.
C.5 Deployment Details and Latency Measurements: Deployment uses a single NVIDIA RTX 4090 GPU with an Intel Core i9-14900KF CPU, FP32 CUDA ONNX weights, and TensorRT optimization.The TensorRT engine uses optimized kernels for causal attention and fused MLPs.
C.5 Deployment Details and Latency Measurements: The complete closed-loop deployment runs at 50 Hz, including sensor read, inference, PD computation, and actuation commands.Configuration files and scripts will be released for reproducible real-time control on G1-like humanoids.