Source-linked AI summary
MimicX: Policy-in-the-Loop Supervision Refinement for Video-Driven Humanoid Motion Tracking
Shuaijun Liu, Chenglong Zhang, Xuhao Liu, Feiyang You, Yifan Liao, Shuyang Hao, Chaozhe Zhang, Chengyu Wu, Zhen Sun, Ningxin Su
TL;DR
Physics-based execution can fail repeatedly on plausible video-derived motion references, motivating supervision that responds to policy failures. MimicX diagnoses those failures, revises tracking objectives and practice, and verifies candidate updates; across four core video skills, it improves tracking and execution horizon over Fixed Reference.
Problem
Plausible video-derived motion references can still produce persistent failures during physics-based humanoid execution, and execution feedback is needed to identify where supervision should change.
Source: §1 · p. 2Method
MimicX diagnoses difficult transitions and affected body regions, coordinates tracking-objective and reset-curriculum updates, and repeatedly verifies candidate policies.
Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 4Results
Across four core video skills, MimicX improves body-tracking error and Robust Execution Horizon over Fixed Reference, with task-averaged 25.7% reduction in body error and 255.6% increase in horizon.
Source: §5.1 · p. 6Takeaways & Limitations
MimicX turns humanoid execution failures into actionable supervision and selects policy updates through execution-guided refinement.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Human videos provide rich motion targets for humanoid learning, yet visually plausible references can still produce persistent failures under physics-based execution. These failures reveal where training supervision should change. We present MimicX, a policy-in-the-loop framework that uses execution feedback to refine video-driven humanoid motion tracking. Starting from reconstructed and retargeted motion, MimicX localizes difficult transitions and affected body regions, then jointly adapts tracking objectives and the reset curriculum for policy continuation. Repeated rollout verification selects execution-priority improvements subject to tracking guards. Across four core video tasks, MimicX consistently improves tracking accuracy and Robust Execution Horizon relative to the Fixed Reference baseline. Task-averaged results show a 25.7% reduction in body-tracking error and a 255.6% increase in execution horizon. Additional video, motion-reference, and collision-scene studies evaluate the method beyond the core tasks, while MimicX-HLoop accelerates feedback through heterogeneous execution. Overall, MimicX turns policy failure into actionable supervision for deciding what to refine and which refinement to retain.
1 Introduction
Video-derived references can look plausible yet repeatedly fail during physics-based execution because robots differ from humans in geometry and dynamics. MimicX uses these failures to localize supervision changes and retain updates only when repeated rollouts verify reliable improvement.
Motivation
- Repeated failures can occur at the same transition despite plausible contact timing, root motion, or limb coordination, reflecting differences in robot geometry and dynamics.The paper argues that execution should inform the supervision used to learn the motion.Source: §1 · p. 2
Approach
- MimicX diagnoses when execution becomes unreliable, which body regions deviate, and how local, global, and dynamic errors develop together.It uses diagnosis to condition objective and curriculum updates before policy continuation.Source: §1 · p. 2
Contributions and results
- Across four core video skills, MimicX improves tracking error and worst-case execution horizon over Fixed Reference, including full-horizon Tennis success in nine of nine evaluations versus one of nine.The paper also describes repeated verification that retains the current policy when a proposal fails its execution-priority checks.Source: §1 · p. 2
2 Related Work
Prior work supports video reconstruction, motion tracking, and adaptive whole-body execution through diverse representations and control strategies. MimicX distinguishes itself by making measured execution failure an explicit input to supervision revision and validating updates through repeated rollouts.
Video reconstruction and motion tracking
- Video reconstruction and tracking methods use human geometry, world-grounded motion, robot references, physics-based imitation, motion priors, and contact-aware retargeting.The passage contrasts MimicX with SMPL-X, GVHMR, GMR, BeyondMimic, SONIC, and MeshMimic using their stated roles.Source: §2 · p. 2
MimicX in relation to prior work
- MimicX makes measured execution failure an explicit input to supervision revision and uses diagnosis–proposal–verification to select updates through execution.This distinguishes its stated approach from the related methods reviewed in these passages.Source: §2 · pp. 2–3
Adaptive whole-body execution
- Adaptive whole-body methods address contact-aware tracking, timing adaptation, dynamics alignment, task goals, long-horizon control, and end-effector skills.These methods include separating motor learning from physical refinement and composing skills.Source: §2 · pp. 2–3
3 Policy-in-the-Loop Supervision Refinement
MimicX uses a motion reference and current policy as inputs, then diagnoses execution failures to revise tracking objectives and reset practice. Repeated evaluation selects a retained policy, while heterogeneous scheduling can shorten feedback without changing the evidence used for selection.
3 Policy-in-the-Loop Supervision Refinement
- The outer loop takes a motion reference and current policy, uses execution errors to revise tracking supervision, and returns the policy retained after repeated verification.A video is reconstructed into world-grounded SMPL-X motion and retargeted to a humanoid reference; the policy drives joint-position commands in MuJoCo/MjLab.Source: §3 · p. 3
3.1 Localizing execution-critical transitions
- Diagnosis uses the first scheduled evaluation repeat, focusing on the first flagged termination or, after completion, the step with maximum tracked-body error.Within the interval, channel magnitudes and body identities map errors to local limb precision, global pelvis/torso stabilization, or dynamics and contact supervision; the interval initializes practice.Source: §3.1 · p. 3
3.2 Coordinating tracking objectives and practice
- Tracking objectives combine local limb, global anchor/core, and dynamic velocity/contact terms; diagnosed failures condition bounded candidates that adjust reward weights, smoothing, and reset distributions.A lower-body failure can strengthen position, height, velocity, and stance terms together, while a root failure emphasizes anchor and core consistency. Reset sampling mixes motion-start states with transition-window states; the window candidates use ρ = 0.45 and 0.65, while consolidation uses ρ = 0.Source: §3.2 · p. 3
3.3 Selecting updates through repeated execution
- The gate accepts the first candidate that improves execution priority while satisfying the body-error guard, or retains the current policy if none passes.MimicX-HLoop overlaps ready GPU rollouts with CPU diagnosis and joins reports before selection, preserving the evidence supplied to the gate while changing feedback latency.Source: §3.3 · p. 4, Figure 3
4 Experimental Setup
The experiments test tracking feedback, component contributions, verification, and execution efficiency across four video-derived humanoid skills. They compare successive supervision configurations under defined continuation and evaluation protocols, with additional video and collision-scene studies.
Tasks and learner
- The core evaluation uses Tennis Swing, Football Juggling, Dance Sequence, and Kung Fu Sequence on a 29-DoF Unitree G1, with 48 task–method–seed trials and 144 final-policy rollouts.Four methods use three continuation seeds, and each final policy receives three evaluation repeats.Source: §4 · p. 4
Compared supervision
- Fixed Reference, Failure Curriculum, Task-Aware Refinement, and full MimicX receive 250 continuation iterations, while full MimicX evaluates three proposed continuations through repeated verification.Tennis and Football share original references across methods; Dance and Kung Fu use prepared repaired references for MimicX, and errors are measured against each method’s reference.Source: §4 · p. 5
Execution and fidelity
- Evaluation uses deterministic inference, full-start initialization, fixed termination tests, and task-macro effects averaged after within-task continuation-seed averaging.Robust Execution Horizon is the earliest failure across repeats, assigning T + 1 on success; body error is the temporal mean of maximum tracked-body position error.Source: §4 · p. 5
Collision-scene extension
- Five additional video-derived motions are evaluated in fixed collision scenes using equal 3,000-update continuations from shared checkpoints.The comparison uses one training seed and three evaluation seeds.Source: §4 · p. 5
Evaluation measures
- Figure 5 organizes evidence as twelve task-matched trials across seven channels, task-wise mean and p95 error reductions, full-horizon completion, and worst-case execution horizon.Its caption identifies Fixed Reference, Failure Curriculum, and Task-Aware Refinement, and defines Tail as body-error p95 and Ankle as ankle-height error.Source: Figure 5 · p. 6
5 Results
Across the core tasks, MimicX improves tracking and execution horizon over Fixed Reference; the results also report complementary benefits from practice allocation, tracking objectives, and policy verification. Additional studies extend evaluation to other video, motion-reference, and collision-scene tasks, while HLoop shortens feedback time.
5.1 Tracking Quality
- MimicX improves body-tracking error and Robust Execution Horizon in all twelve task-matched comparisons against Fixed Reference, averaging 25.7% lower body error and 255.6% higher horizon.Tennis reaches full-horizon success in all nine evaluations.Source: §5.1 · p. 6
5.2 Refinement and Verification
- Failure Curriculum yields full-horizon Tennis execution and the lowest task-macro velocity errors, Task-Aware Refinement the lowest anchor error and highest normalized horizon, and the complete loop the lowest mean body error while matching best task-macro success.Table 3 reports unweighted means across four tasks after averaging three continuation seeds within each task.Source: §5.2 · p. 6, Table 3 · p. 8
5.1 Tracking Quality
- On identical Tennis and Football references, MimicX reduces joint RMSE and root-local body error relative to BeyondMimic in both tasks and every paired continuation seed.The comparisons cover complete source intervals without failure resets.Source: §5.1 · p. 6, Figure 6 · p. 7
5.2 Refinement and Verification
- Verification accepts six of twelve refinements and protects six incumbent policies; in Kung Fu, two reward-improving candidates are rejected because they shorten repeated-evaluation horizon.Reward can break an execution-and-tracking tie but cannot reverse an earlier comparison.Source: §5.2 · p. 7, Table 5 · p. 8, Table 4 · p. 8
5.3 Task Breadth and Efficiency
- Across five collision-scene tasks, strict completion rises from 9/15 to 12/15 rollouts, while additional video and motion-reference studies report gains on Soccer and three other videos and lower reference-averaged body error on LaFAN1 and AMASS/ACCAD.The LaFAN1 and AMASS/ACCAD reductions are 15.81% and 19.84%, respectively; Forest Traversal improves from consistent early failure to 3/3 complete executions.Source: §5.3 · p. 8, Table 6 · p. 9
- MimicX-HLoop reduces median rollout–diagnosis–selection time from 241.2 s to 124.0 s, a 1.95× speedup over sequential execution, with identical selection outcomes across paired repetitions.The benchmark overlaps GPU simulation with CPU diagnosis while holding checkpoints and workload fixed.Source: §5.3 · pp. 9–10, Figure 11 · p. 10, Table 7 · p. 10
A.1 Reference and Policy Interface
- MimicX uses a video-derived robot reference and policy rollouts to locate difficult transitions and affected body channels, then proposes coordinated tracking-objective and reset-curriculum changes.When no termination occurs, localization uses the step with the largest tracked-body position error.Source: A Notation and Algorithmic Details · p. 13, §A.1 · p. 13, §A.2 · p. 13
A.5 Repeated Execution and Protected Selection
- The finite-budget loop continues PPO from a shared accepted checkpoint, evaluates candidates under common verification, and retains the first guarded strict improvement; execution feedback thus determines which supervision update is kept.If no candidate passes, the incumbent is retained; the conclusion frames the loop as coupling video-derived references with policy learning through execution-guided refinement.Source: §6 · p. 10, §A.4 · p. 14, §A.5 · p. 15
B.2 Protected Execution-Priority Selection
Protected selection prioritizes execution while guarding tracking error; objective and curriculum changes target different parts of learning, and heterogeneous execution preserves selection decisions while accelerating feedback.
B.2 Protected Execution-Priority Selection
- Verified selection prioritizes full-horizon success, then worst-case execution horizon and tracking coordinates, while a body-error guard can reject candidates independently of reward.In the Kung Fu case, both proposals increased reward but shortened the worst execution horizon, so the stronger execution record was retained.Source: §B.2 · p. 16
B.3 Complementarity of Objective and Curriculum
- Mixing full-start and window-initialized resets raises practice-window visit probability to ρ + (1 − ρ)pW; the exposure gain is largest when the current policy rarely reaches the transition.When pW = 0.1, ρ values of 0.45 and 0.65 yield visit probabilities of 0.505 and 0.685; full-start verification separately measures successful execution through the window.Source: §B.3 · pp. 16–17
- Objective refinement changes sensitivity to local, global, and dynamic tracking errors, whereas curriculum refinement changes exposure to those errors during learning.For fixed nonzero error, sensitivity is not strengthened indefinitely by reducing a reward scale, motivating a bounded joint search over weights and scales.Source: §B.3 · p. 17
B.4 Decision-Preserving Heterogeneous Execution
- Dependency-respecting schedules preserve semantic outputs and selection decisions when node outputs are schedule-independent and reports use fixed identity ordering.The measured workload had identical selection outcomes and a median 1.945× speedup over sequential execution; independent rollout and diagnosis work can overlap.Source: §B.4 · pp. 17–18
C.1 Tasks, Robot, and Learning Backend
- The core study tracks four motions on a 29-DoF Unitree G1 in MuJoCo/MjLab, using 14 tracked bodies and PPO at 50 Hz.The evaluation horizons are 800, 455, 784, and 1,062 control steps for Tennis Swing, Football Juggling, Dance Sequence, and Kung Fu Sequence, respectively.Source: §C.1 · p. 18
C.2 Controlled Continuation Protocol
- The controlled continuation compares Fixed Reference, Failure Curriculum, Task-Aware Refinement, and MimicX from shared warmstarts; MimicX evaluates three candidates, with repaired references for Dance and Kung Fu.The first three methods each use 6,144,000 new transitions; MimicX uses 18,432,000 candidate transitions plus verification.Source: §C.2 · pp. 18–19
C.3 Verification and Metric Reduction
Across four core tasks, MimicX reduces body-tracking error and increases worst-case execution horizon relative to Fixed Reference. The evaluation also defines repeated-rollout metrics and examines broader video, motion-reference, and heterogeneous-execution studies.
C.3 Verification and Metric Reduction
- Core verification uses nine rollouts per task–method cell, with full-start initialization, deterministic inference, and active termination tests; tracking summaries pool steps across repeats.Mean, nearest-rank 95th percentile, and peak are reported, while task summaries average training-seed values and success counts all nine repeats.Source: §C.3 · p. 19
C.4 Breadth and External Evaluation
- The broader evaluation covers four additional videos, 14 LaFAN1 and AMASS/ACCAD references, direct-tracker comparisons, and five collision-scene motions.The direct tracker study compares BeyondMimic, SONIC, Fixed Reference, and MimicX on Tennis and Football; Dance and Kung Fu also receive common-reference comparisons.Source: §C.4 · p. 19
C.5 Heterogeneous Execution Benchmark
- The HLoop timing study uses a fixed 17-job workload with eight GPU rollouts, eight dependent CPU diagnoses, and one selection job, reporting median wall time and selection outcomes.The sequential, bulk-synchronous, and dependency-driven modes use the same graph; parallel modes use two GPU slots and at most eight CPU workers.Source: §C.5 · p. 19, Table 9 · p. 20
D.1 Closing the loop improves execution and fidelity
- 25.72% lower body error and 255.57% higher execution horizon are the task-macro effects of MimicX versus Fixed Reference across four core video tasks.Both measures improve in all twelve task-matched continuation-trial pairs; MimicX achieves full-horizon success in all nine Tennis evaluations, while all methods record 0/9 on the other three tasks.Source: §D.1 · p. 20
D.2 Cumulative components address different execution errors
Across selected continuations, Task-Aware Refinement leads macro horizon AUC while MimicX has the lowest final body error. The component and learning-dynamics analyses show how supervision changes affect execution and tracking over time.
D.2 Cumulative components address different execution errors
- Failure-conditioned sampling makes difficult segments more influential and brings Tennis to full-horizon execution; task-aware supervision further extends Football horizon while reducing root-and-anchor error.Source: §D.2 · p. 20
D.3 Learning dynamics under selected supervision
- Twelve 250-update Tennis continuations trace an initial body-error transient, subsequent tracking improvement, reward development, and episode-length evolution.The curves share a 250-update continuation axis.Source: §D.3 · pp. 20–21
D.4 Evaluation milestones across all core tasks
- Across 48 selected policy trajectories followed through 240 milestone evaluations, Task-Aware Refinement provides the highest macro horizon AUC, while MimicX provides the lowest final body error.Source: §D.4 · p. 21
D.5 Temporal diagnostics connect fidelity to dynamics
- Temporal diagnostics align orientation, acceleration, and angular velocity; Table 11 retains eight metrics and distinguishes logged aligned bodywise mean from evaluation maximum.The aligned traces exclude restart-crossing derivatives.Source: §D.5 · pp. 21–22, Table 11 · p. 22
E Gate and Rejection Case Studies
The execution gate accepts or protects policies based on verified outcomes, not reward alone. Its Kung Fu rejection case shows candidates with higher reward but shorter worst-case horizons, alongside time-aligned motion diagnostics.
E Gate and Rejection Case Studies
- Across 12 full-loop trials, six proposals are accepted and six current policies protected; every Tennis and Football trial accepts a proposal, while Dance and Kung Fu retain their current policies.Candidate exploration preserves stronger verified execution when a proposed continuation regresses.Source: E Gate and Rejection Case Studies · p. 22
- Figure 12 aligns orientation, acceleration, and angular velocity across 800 Tennis states, using raw traces and 11-sample means while excluding derivatives across resets.Source: Figure 12 · p. 23
- Kung Fu proposals raise reward to 0.6275 and 0.5838 but reduce worst-case horizon from 879 to 677 and 689, so both are rejected.The current policy has reward 0.5707 and worst-case horizon 879.Source: E Gate and Rejection Case Studies · p. 23
F.1 Direct same-motion tracker comparison
Matched-reference evaluations compare articulation across Tennis and other motions, while additional video, motion-reference, collision-scene, and developmental studies test refinement in distinct settings. Outcomes include broad gains as well as task-specific exceptions.
F.1 Direct same-motion tracker comparison
- On matched Tennis and Football references, MimicX improves joint RMSE and root-local body error over BeyondMimic in every seed pair; Dance and Kung Fu have metric-specific winners.The articulation comparisons use a common joint order and G1 kinematic model, with full-interval non-reset playback.Source: §F.1 · pp. 23–24, Table 16 · p. 24
F.2 Additional videos and motion-reference inputs
- Additional videos show mixed outcomes: Soccer improves in body tracking and full-horizon success, while Basketball keeps the same success rate with higher body error; supplied-motion cases include 14 references.The additional video tasks exercise the reconstruction-to-policy path, while supplied-motion cases test the policy-side loop with existing references.Source: §F.2 · p. 24, Table 17 · p. 24, Figure 13 · p. 25
F.3 Collision-scene video extension
- Across collision-scene tasks, strict completion rises from 60% to 80%; Forest Traversal shifts from 0/3 to 3/3 completions while body and root prefix errors fall 79.15% and 89.04%.Parkour reaches 220 of 255 steps in both conditions, with improved prefix errors.Source: §F.3 · p. 25, Figure 14 · p. 26
F.4 Developmental refinement cases
- Selected developmental runs remove Tennis terminations before wrist-error refinement while preserving zero terminations, and separately illustrate Football continuation and Dance reference repair.These are selected single-rollout mechanism cases, separate from controlled multi-seed results.Source: §F.4 · p. 26, Table 19 · p. 26
G HLoop Execution Analysis
MimicX-HLoop shortens the rollout-feedback workload while preserving selection outcomes. In the related-work context, MimicX is distinguished by using measured policy failure to revise supervision and verify continued policies through execution.
G HLoop Execution Analysis
- MimicX-HLoop reduces median wall time from 241.22 s to 124.02 s, a 1.945× speedup over sequential execution, while preserving selection outcomes across repetitions.Bulk-synchronous execution has a 122.26 s median; dependency-driven scheduling diagnoses completed rollouts while other simulations continue.Source: G HLoop Execution Analysis · p. 26
H Extended Related Work
- MimicX adds rollout-failure feedback to video reconstruction and retargeting, using failures to locate where reference-conditioned objectives and training distributions should change.Source: H Extended Related Work · pp. 26–27
- Across physics-based tracking, adaptive execution, and scene-aware imitation, MimicX distinguishes itself by revising supervision from measured policy failure and verifying continuation through repeated execution.Source: H Extended Related Work · p. 27
I Reproducibility and Artifact Organization
The section documents a versioned refinement record and visual materials tracing reconstructed motion through policy execution, comparing task-critical events, and presenting recorded tracking in reconstructed scenes.
I Reproducibility and Artifact Organization
- The versioned record links references, proposals, policy continuations, and repeated evaluations; only fully verified candidates update the accepted policy.Interrupted work resumes from the last accepted state, while studies retain their own sampling units and evaluation protocols.Source: I Reproducibility and Artifact… · p. 27
J.1 Human reconstruction and learned execution
- Sequences connect original video frames to recovered human motion and recorded policy states across Tennis, Football, Dance, and additional running and stair tasks.Figure 4 shows distinct pipeline stages for two tasks; the sequences use recovered source-phase mapping for Tennis and Football and nominal timestamps for Dance.Source: §J.1 · p. 27, Figure 21 · p. 30
J.2 Task-critical events reveal execution differences
- Task-event plates compare solid policy executions with translucent references, using gray for Fixed Reference and coral for MimicX at selected moments.The accompanying numerical tables summarize every declared trial.Source: §J.2 · p. 27, Figure 18 · p. 29, Figure 19 · p. 29
J.3 A continuous scene makes motion progression readable
- Figure 22 separates seven recorded states spatially and uses increasing opacity for chronology; translucent robots denote earlier policy states, not reference overlays.The Tennis scene includes a racket and registered ball, while Football ball placement follows observations at recovered source phases.Source: §J.3 · p. 30, Figure 22 · p. 31
J.4 Scene reconstruction connects the rendered policy to the input
- The Tennis presentation registers the recovered camera to a Gaussian background while preserving the robot’s recorded pose, with median reprojection error of 0.647 source pixels.The presentation provides both an unobstructed simulation view and original-scene context; foreground contours remain inside the output canvas.Source: §J.4 · p. 30
J.5 Visual breadth beyond the four core tasks
Additional video-derived tasks show positive tracking results, and motion-reference examples demonstrate the same diagnosis, proposal, and verification interfaces across different reference-construction routes.
J.5 Visual breadth beyond the four core tasks
- Two additional video-derived tasks show positive tracking results beyond the core tasks.Figure 23 presents a separate Soccer task, and Figure 24 presents the Uniandes Dance task.Source: §J.5 · p. 31, Figure 23 · p. 31, Figure 24 · p. 32
- The motion-reference plates illustrate the shared refinement interface for provided retargeting and a second retargeting of the same human motion.The two executions use different robot references derived from the common human source motion.Source: §J.5 · p. 31, Figure 25 · p. 32, Figure 26 · p. 33