Source-linked AI summary
HumanTracker: Towards Comprehensive and Human-Aligned Motion Tracking Benchmark
Dairu Liu, Zekun Qi, Jiayu Zeng, Ruixi Yu, Yu Guan, Yintianrun Zhang, Xuchuan Chen, Sikai Liang, Zekai Li, Chenghuai Lin, Xinqiang Yu, Wenyao Zhang, He Wang, Li Yi
TL;DR
Humanoid tracking evaluation can diverge from human perception because per-frame kinematic metrics miss contact and stability artifacts, while common test suites lack diverse, challenging motions. HumanTracker addresses this with a categorized optical-motion benchmark and HumanScore, a preference-aligned metric that better captures perceptual tracking quality and exposes contact-related failures.
Problem
Kinematic metrics and small evaluation suites inadequately capture contact, stability, and diverse long-horizon humanoid tracking behavior.
Method
HumanTracker combines approximately 153 hours of labeled optical trajectories from professional performers across four motion families with HumanScore, a preference-aligned reward model trained from human comparisons.
Results
HumanScore better predicts human preferences and reveals contact and stability failures that kinematic metrics often miss across representative state-of-the-art trackers.
Takeaways & Limitations
Category-level evaluation distinguishes overall tracking reliability from tracker strengths under particular contact regimes and evaluation criteria.
Takeaways & Limitations
HumanScore should be read alongside success and kinematic error because those diagnostics remain valuable for locating specific error sources.
Abstract
from arXiv · showhide
Humanoid motion tracking is central to teleoperation and whole-body imitation, yet evaluation often disagrees with what people perceive in videos. Kinematic errors average per-frame pose differences but miss the physical artifacts that matter most, particularly unstable support and incorrect contacts such as foot skating and mistimed touch-downs. Meanwhile, widely used test suites are small and lack the diversity needed to stress contact-rich, long-horizon behaviors. We introduce HumanTracker to make humanoid tracking evaluation both perceptually aligned and scalable. The HumanTracker benchmark contains approximately 153 hours of optical motion trajectories from multiple professional performers, organized into four motion families with text labels for fine-grained diagnosis. We further propose HumanScore, a preference-aligned metric trained on 12K motion pairs containing 24K motions. Across representative state-of-the-art trackers, HumanScore better predicts human preferences and reveals contact and stability failures that kinematic metrics often miss.
1 Introduction
HumanTracker addresses two limitations in humanoid tracking evaluation: kinematic metrics can miss perceptually important contact and stability failures, while common test suites lack diversity and category-level diagnosis. It introduces a human-aligned metric, a large categorized benchmark, and a standardized evaluation protocol.
- Evaluation gaps: Kinematic metrics can disagree with human judgments because they miss foot sliding, mistimed contacts, support, balance, and accumulated closed-loop errors.Rollouts with similar pose errors can differ substantially in stability and perceived quality.
- Evaluation gaps: The commonly used AMASS evaluation suite contains only 140 sequences and under-represents challenging contact transitions, asymmetric balancing, and complex recoveries.Aggregate reporting without motion-category breakdowns also makes tracker failures difficult to diagnose.
- Proposed approach: HumanScore is a preference-aligned reward model trained on synchronized tracking-video comparisons to target human preferences, perceptual stability, and realistic physical contacts.The metric is designed to capture nuanced perceptual qualities beyond rule-based diagnostics such as foot slip or root drift.
- Proposed approach: HumanTracker provides approximately 153 hours of optical trajectories from 24 professional performers, organized into four motion families with category and text labels.The benchmark enables per-family metrics and fine-grained diagnosis of failure modes.
- Contributions: The paper establishes a rigorous, standardized protocol intended to make humanoid-tracking progress measurable and meaningful.Its contributions combine human-aligned evaluation, diverse categorized motion data, and reproducible comparison.
2 Related Work
Prior work has expanded humanoid tracking, motion repositories, and physically grounded references, but reported performance remains difficult to compare. HumanTracker responds by treating evaluation as a controlled experiment with diverse, physically plausible, diagnostically categorized data.
- Humanoid tracking: Reference-conditioned policy learning underpins modern physics-based character control and humanoid motion tracking, including systems such as GMT, SONIC, Humanoid-GPT, and UniTracker.Related systems emphasize broader motion coverage and control robustness through different mechanisms.
- Evaluation methodology: Reported tracker performance depends on the reference set, simulator, action representation, initialization, and termination rule, complicating direct comparisons.HumanTracker therefore frames evaluation as a controlled experiment.
- Motion data: Large motion repositories expand training diversity and support semantic retrieval and conditional generation, but dataset size alone does not define a useful humanoid benchmark.Useful references must also be retargeted, physically plausible around contacts, and difficult enough to expose controller failures.
- Motion data: HumanTracker reports approximately 153 hours and 25K clips across four complementary tracking regimes.The benchmark is organized around distinct regimes rather than relying only on aggregate scale.
3 HumanTracker: Benchmark and Preference-Aligned Evaluation
HumanTracker combines a diverse motion benchmark, standardized tracker evaluation, and preference-aligned HumanScore to assess humanoid tracking beyond kinematic error. Its pipeline uses expert comparisons of synchronized rollouts and trajectory features that include reference, simulated-state, control, and contact information.
- Benchmark construction: 153 hours of optical trajectories from 24 professional performers form the HumanTracker benchmark, with motion coverage organized into diagnostic families.The performers include dance, fitness, tennis, and motion-capture professionals; trajectories are retargeted to the benchmark humanoid.
- Benchmark construction: Four motion families expose distinct failure regimes, including steady-state drift, rapid support switching, contact transitions, asymmetric balancing, and complex recoveries.The taxonomy is designed to diagnose where trackers fail rather than reduce evaluation to one aggregate score.
- Standardized tracker evaluation: A common MuJoCo entry point standardizes the motion list, robot model, reference indexing, rollout accounting, and metric implementation while preserving each tracker's native interface.The evaluator records trajectories at 50 Hz and stores state, actions, contacts, forces, and keypoint kinematics for shared downstream analysis.
- Preference data construction: Human preference data compare aligned 250-frame, 5-second rollout windows sampled uniformly across the training distribution, with source motions kept within one partition.Six doctoral researchers judge completion, balance, jitter, foot sliding, locomotion consistency, and whole-body naturalness; display order is randomized.
- HumanScore: HumanScore maps simulator trajectory segments to rewards using current reference and simulated-state features, temporal Transformer processing, masked mean pooling, and an MLP scalar reward.The 539-dimensional frame features include control, measured contact dynamics, root motion, and keypoint kinematics, without future reference residuals.
- HumanScore: HumanScore aggregates bounded window rewards by the number of actual frames represented, while right-padding contributes no weight to the reported average.At evaluation, rollouts are divided into windows of at most 250 actual frames, with validity masks applied to short final windows.
4 Experiments
The experiments evaluate representative trackers under a standardized protocol and show that HumanScore captures human-perceived trajectory quality, including contact and stability, beyond conventional metrics. Sensitivity analyses show that contact features and longer temporal context improve alignment, while family-balanced evaluation exposes criterion-specific strengths.
- Evaluation protocol: The standardized evaluation applies the same retargeted references, evaluator, whole-body metric, and HumanTracker test split to GMT, TWIST2, SONIC, and Humanoid-GPT.Trackers retain their native observation and action-processing stacks, and none is trained or fine-tuned on HumanTracker.
- Results: Humanoid-GPT is the strongest overall tracker, leading most comparisons and all three metrics on Daily and Highly Dynamic, while SONIC leads selected Interaction and Ground comparisons.SONIC has the highest completion rate on Interaction and highest HumanScore on Ground, where its rollouts are perceived as more natural and stable.
- Evaluation protocol: Align Rate is computed within each motion family and then averaged across four families, preventing frequent categories from obscuring performance on less frequent regimes.Balanced tracker pairings further reduce the possibility that agreement is explained by a dominant family or matchup.
- Human preference alignment: HumanScore agrees with human preferences more consistently than any individual analytic diagnostic.Conventional metrics isolate pose, velocity, or contact fidelity, whereas human judgments also reflect smoothness, stability, human-like quality, and error development over time.
- Sensitivity analysis: Removing measured contact features degrades performance most clearly on Ground, while adding future reference information performs slightly worse than the baseline.The result indicates that contact information matters for complex contact transitions and that future-reference signals offer limited benefit.
- Sensitivity analysis: Alignment improves steadily as available context grows from one to five seconds because longer windows reveal sliding, jitter, drift, and recovery that isolated poses miss.HumanScore therefore integrates complementary temporal evidence rather than evaluating motion frame by frame.
5 Conclusion
HumanTracker combines a large, diverse optical-motion benchmark with HumanScore, a human-aligned metric for humanoid tracking evaluation. The framework supports reproducible tracker comparison and failure analysis while identifying weaknesses in highly dynamic and ground-contact motions.
- Conclusion: HumanTracker contains approximately 153 hours of optical motion from 24 professional performers across four motion families.The benchmark is designed for diverse, categorized tracking evaluation.
- Conclusion: HumanScore provides a human-aligned metric for evaluating humanoid motion tracking.The benchmark and metric together support standardized comparison and failure analysis.
- Conclusion: The benchmark reveals persistent weaknesses in highly dynamic and ground-contact motions.Future work extends the framework to cross-embodiment settings, real-world hardware, and reward optimization.
A Implementation details
HumanScore represents each frame with reference and simulated-rollout features, handles short windows with masked padding, and aggregates bounded rewards using actual-frame weighting. Its implementation explicitly includes contact, force, acceleration, and gravity-aligned keypoint information.
- Sequence handling: Short segments are right-padded to 250 × 539 inputs, while validity masks exclude padded frames from attention and temporal pooling.Window rewards are weighted by actual frame counts when combined across a rollout.
- Frame features: Each frame token contains 70 current-reference dimensions and 469 simulated-rollout dimensions.Rollout features include floor contacts and forces, foot acceleration, and gravity-aligned keypoints for 14 bodies.
- Frame features: The input-feature design combines current reference information with simulated robot state, including contact and force measurements.These features support trajectory-level evaluation rather than relying only on local pose fidelity.
- Feature ablations: Removing the measured rollout contact block reduces the input from 539 to 519 dimensions, while the Kinematics Only variant has 516 dimensions.Both variants retain the current-reference foot-contact target.
- Model and scoring: A bidirectional Transformer and three-layer reward head produce window rewards that sigmoid normalization bounds between zero and one.HumanScore is 100 times the mean reward weighted by the number of actual frames in each window.
B.1 Evaluation from human preferences
HumanTracker applies preference learning to compare the physical quality of robot rollouts following the same reference motion. This setting differs from prior approaches that assess generated motion, human likeness, task completion, or policy optimization.
- Preference learning: Pairwise human judgments train a reward model when structured output quality is easier to recognize than to specify with a fixed analytic objective.This preference-learning strategy is adapted here to humanoid tracking evaluation.
- Scope: Unlike methods evaluating generated motion or general human likeness, HumanTracker compares the physical quality of robot rollouts following the same reference.The comparison targets tracking quality rather than motion-generation quality.
- Scope: Related robotic reward models primarily support task completion or policy optimization, whereas HumanScore evaluates trajectory quality in humanoid motion tracking.This distinction separates the benchmark’s evaluation objective from reward-learning systems for robotics.
C Preference data statistics
The preference catalogue pairs tracker rollouts for human judgment, while released trajectories are partitioned and documented at substantial scale with fixed schemas.
- Preference catalogue: The catalogue compares paired tracker rollouts selected from aligned clips across source motions and motion families.A balanced schedule assigns tracker combinations to selected clips, producing paired comparisons.
- Preference labels: The label set includes strict preferences, Similar judgments, and Cannot compare judgments, with Cannot compare excluded from reward optimization.
- Data partitioning: The data split is 80/20 by source motion_id, preventing clips from one motion from crossing partitions.The split balances motion family, tracker pair, label, annotator, clip length, and sampled clips per source motion.
- Released scale: 22,495 training trajectories and 2,500 test trajectories are released at 50 Hz, forming a 9:1 trajectory-count split.The manifests contain 24,793,129 training frames and 2,687,461 test frames.
- Data schema: Each NPZ archive stores robot states, keypoint and global motion quantities, and bilateral foot contact in a common temporal schema.The schema spans 24,995 released archives and includes qpos, qvel, poses, velocities, and foot_contact arrays.
E Analysis of motion coverage
HumanTracker characterizes motion coverage with trajectory-derived kinematic descriptors across four diagnostic families, while explicitly separating these descriptors from contact and stability evaluation.
- Motion families: Four motion families organize the benchmark: Daily, Highly Dynamic, Interaction, and Ground-level movements.
- Coverage descriptors: Coverage descriptors include horizontal root path, root height range, and the 95th-percentile absolute speed of 29 actuated joints.Statistics are reported as medians and quartiles across source motions.
- Family differences: Daily motions are longer and travel farther horizontally, while Interaction motions are spatially localized.
- Family differences: Ground motions span a broader upper half of the root-height distribution, whereas Highly Dynamic contains many short skills.
- Scope of descriptors: These kinematic descriptors do not measure contact timing, support, or recovery and cannot replace Succ, MPJPE, or HumanScore.
F Analysis of preference alignment
Preference-alignment uncertainty is estimated with bootstrap resampling that preserves family weighting and clusters comparisons by source motion.
- Uncertainty estimation: 20,000 bootstrap draws estimate uncertainty for Align Rate using source motions sampled with replacement within each motion family.
- Uncertainty estimation: The procedure recomputes four family rates and their equal-weight mean while retaining all comparisons associated with sampled source motions.
- Reported uncertainty: Table 9 reports 95% intervals for preference alignment, stratified by motion family and clustered by source motion.
G Discussion and Limitations
The discussion argues that contact-rich events explain perceptual differences beyond pose error, while positioning HumanScore as complementary to analytic diagnostics and bounded by data, embodiment, and observability limits.
- Discussion: Contact regime changes tracker behavior: a method reliable on upright Daily motion can fail during Ground-level transitions.
- Discussion: HumanScore gains most of its advantage when contact-related state is observed over several seconds.
- Discussion: Perceptual failures often occur as events such as slides, impacts, support switches, or recoveries rather than isolated poses.
- Practical interpretation: HumanScore summarizes perceived trajectory quality, while success and kinematic error remain useful for locating specific error sources.
- Limitations: HumanScore generalizes to unseen motions but not necessarily to an entirely unseen robot, simulator, or controller family.
- Limitations: Applying HumanScore to real-world trajectories requires observable features or a separately validated state estimator.
- Limitations: The benchmark uses imbalanced motion families and one 29-DoF humanoid embodiment in MuJoCo, limiting population and hardware-robustness claims.