BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion

Introduction

• Humanoid control is challenging because dozens of joints and actuators create a high-dimensional problem, floating-base balancing is difficult, and kinematic and dynamic surrogates can yield unnatural motions (13–15).Source: Introduction · pp. 2–3
• BeyondMimic learns diverse human-level skills with a compact, motion-agnostic tracking formulation and transfers human-like behaviors zero-shot to real humanoids.The formulation uses classical-mechanics-based actuation modeling, limited domain randomization, and three regularization terms plus a unified task reward, while sharing model parameters, rewards, and hyperparameters across skills and physics engines.Source: Introduction · pp. 4–5
• The framework achieves agile, natural motions outdoors across varied conditions that surpass prior humanoid-robotics work, as shown in Fig. 1 and Movie S1.The introduction presents this as evidence that the approach produces human-like performance under real-world deployment conditions.Source: Introduction · p. 4
• BeyondMimic provides a unified real-world controller that handles diverse downstream tasks through online planning alone, without task-specific training, fine-tuning, or policy optimization.It synthesizes atomic skills learned by reinforcement learning into novel sequences for zero-shot, task-specific control after deployment.Source: Introduction · pp. 4–5
• A state-action co-diffusion model with classifier guidance captures multimodal skill distributions and enables gradient-based optimization toward arbitrary differential objectives at test time.Unlike VAEs or AMPs, diffusion models learn gradient fields of the data distribution (49), allowing costs on both predicted future states and actions in a predictive-control formulation.Source: Introduction · p. 5
• The unified controller solves unseen tasks with unlabeled data while preserving the natural style and dynamic quality of original human motions.Examples include waypoint navigation, joystick teleoperation, and obstacle avoidance.Source: Introduction · p. 5
• BeyondMimic combines scalable human-motion learning with diffusion-based online optimization to bridge specialized motion tracking and general-purpose task adaptability.The framework is designed to let humanoid robots learn, adapt, and compose skills seamlessly.Source: Introduction · p. 5

Results

• Learned skills can be flexibly composed and repurposed at inference time for unseen locomotion, navigation, and motion-completion tasks, with smooth transitions without retraining.Diffusion-based online optimization supports command-conditioned locomotion, scene-aware navigation, and keyframe-based motion completion.Source: Results · p. 6

Scalable Learning from Human Motions

• The framework learns a broad repertoire of agile, stylish, human-like skills using one motion-tracking formulation and shared hyperparameters without motion-specific tuning.This demonstrates scalable learning from diverse human motions.Source: Scalable Learning from Human Motions · p. 6

• The system trained on approximately 2.5 hours of diverse human motions, validated all motions in high-fidelity simulation, and transferred 30 representative clips totaling 15 minutes reliably to the physical robot.The sim-to-real deployment is reported in Fig. 2.Source: Diverse Skills · p. 6
• Validated behaviors covered static balance-critical motions, dynamic jumps, kicks, spins, and cartwheels, plus expressive walking, dance, and sport movements with recognizable human-like quality.Source: Diverse Skills · p. 6
• Many challenging clips were trained jointly with multiple other skills in reference motions exceeding three minutes, while retaining agility and stylistic details.Source: Diverse Skills · p. 6
• These results indicate that the unified formulation scales across diverse human motions without motion-specific tuning.Source: Diverse Skills · p. 6

• The robot completed complex acrobatic and martial-arts-inspired sequences on soft, unstable, and uneven outdoor terrain absent from training.These unseen deformable-contact challenges included soft soil, decaying leaves, and uneven ground.Source: Human-level Agility · p. 7
• Aerial cartwheels reached 31 m/s2 peak acceleration and 20 rad/s peak pelvic angular velocity, versus 7.75 rad/s average in skilled human aerial motions (50).The robot’s mean pelvic angular velocity was 7.01 rad/s.Source: Human-level Agility · p. 7
• Clean aerial-cartwheel landings required minimal recovery, indicating accurate airborne posture control and highly dynamic human-level agility.Source: Human-level Agility · p. 7
• Contact-rich skills included consecutive cartwheels, crawling, ground-to-standing jumps, and five repeated celebration jump-turns without losing stability or style, whereas prior work (33) reported one execution.Source: Human-level Agility · p. 7

Natural, Human-like Behaviors

• Across walking and running, the framework demonstrated human-like natural behaviors and smoothly resumed walking after gentle external disturbances without stiff or exaggerated reactions.When held, the robot remained compliant, paused, and stabilized before resuming after release.Source: Natural, Human-like Behaviors · pp. 7–8
• The robot matched human ground-reaction-force shapes, showing walking double peaks and running single peaks with aligned loading and push-off timing.Walking peaks were sharper because the robot lacked a toe joint, whereas the limitation was less pronounced in running’s shorter, single-contact phase.Source: Natural, Human-like Behaviors · p. 7
• BeyondMimic was preferred over Unitree’s native controller overall (70.8% vs. 29.2%, p< .001, Cohen’s h= 0.859), including walking and running.Walking preference was 57.0% vs. 43.0%, while running preference was 84.7% vs. 15.3%, both with p< .001.Source: Natural, Human-like Behaviors · p. 8

Versatile Humanoid Control

• Diffusion guidance synthesizes agile skills for unseen tasks without retraining by iteratively refining predictions toward specified velocity costs using an acquired feasible motor-skill prior.This avoids the numerous regularization and behavior-shaping terms typically required by model-based or learning-based methods.Source: Versatile Humanoid Control · p. 8
• The framework supports command-conditioned locomotion, keyframe-based inpainting for agile skill composition, and flexible task composition at inference time.These capabilities are illustrated in Figures 5 and 6 and Movies S5 and S6.Source: Versatile Humanoid Control · p. 8

• Zero-shot commanded locomotion supported joystick velocity commands and waypoint goals, with smooth trajectories reaching targets from varying initial positions.Joystick control produced smooth omnidirectional walking and reliably tracked the commanded direction.Source: Command-conditioned Locomotion · pp. 8–9
• The controller remained robust to large external perturbations, including kicks, while continuing to pursue task objectives.Source: Command-conditioned Locomotion · p. 9
• 12.14% and 13.65% average velocity tracking errors were recorded for walking and running, respectively, in simulation.The robot also ran continuously for over 50 m, indicating stable control over extended distances.Source: Command-conditioned Locomotion · p. 9
• Similar commands produced multimodal, human-like, dynamically stable gaits, including walking or light jogging at low velocity.Source: Command-conditioned Locomotion · p. 9
• Velocity input alone elicited smooth walking-to-running transitions that were rare and unlabeled in the data but inferred and reproduced naturally when required.This enabled context- and intention-like skill transitions without explicit skill specification.Source: Command-conditioned Locomotion · p. 9

• Motion inpainting replaces weak velocity or position cues with sparse future keyframes that guide smooth intermediate motions for online skill composition.This formulation supports insertion, transition, and composition of agile motor skills.Source: Motion Inpainting and Task Transitioning · p. 9
• Sparse keyframes were inpainted into temporally coherent trajectories, smoothly transitioning from joystick-controlled walking into cartwheels and back to command-conditioned walking.Desired cartwheel keyframes were injected at 0.2 s intervals, and the diffusion policy generated continuous motion around them.Source: Motion Inpainting and Task Transitioning · p. 9
• The model smoothly transitioned from command-conditioned locomotion into diverse human-tracked skills, including lying down, standing up, spin-kicks, and flip-kicks.These transitions covered both contact-rich behaviors and highly dynamic gaits.Source: Motion Inpainting and Task Transitioning · pp. 9–10
• Online optimization enabled three consecutive cartwheels stitched with walking and running, demonstrating robust, smooth long-horizon mode switching.The execution switched freely between velocity following and motion inpainting before, between, and after cartwheels.Source: Motion Inpainting and Task Transitioning · p. 10

Task Composition

• The framework composes multiple inference-time objectives without enumerating goal combinations, enabling unseen task compositions without retraining.Simple task-specific costs can be flexibly summed without extensive objective tuning.Source: Task Composition · p. 10
• Combining waypoint tracking with SDF-based obstacle avoidance detoured the robot around an obstacle to reach its target waypoint, while joystick tracking mitigated collisions under moderate off-goal inputs.The signed distance field supplies distance gradients across the prediction horizon during denoising, steering trajectories away from obstacles.Source: Task Composition · p. 10

Discussion

• BeyondMimic learns hundreds of diverse, agile, naturalistic motions with one RL formulation and shared hyperparameters, transferring the same policy directly to hardware without motion- or robot-specific tuning.Moderate domain randomization and careful system implementation replace extensive sim-to-real heuristics while preserving agility and natural appearance.Source: Discussion · p. 11
• The unified model composes skills learned from unlabeled human motion into coherent sequences that solve novel downstream tasks through denoising-time guidance.Inference-time trajectory adjustment couples planning and control, supporting obstacle navigation, joystick velocity commands, and recovery from external disturbances while preserving smooth, human-like behavior.Source: Discussion · p. 11
• Task-agnostic, unlabeled training lets new motion data expand the skill set without redefining objectives or retraining per task, supporting generalization across motions and environments.The work positions this predictive paradigm as a step toward foundational humanoid behavior models that unify human-like behavior with control and planning.Source: Discussion · p. 11

Limtations and Future Works

• State-estimation errors propagate directly into generated trajectories, motivating sensor fusion or learned estimators despite some robustness to noisy observations.Improved perception remains a future direction for enhanced performance.Source: Limtations and Future Works · p. 12
• The 0.64-second prediction horizon supports reactive control and local obstacle avoidance but is insufficient for long-horizon planning toward distant goals or obstacles.History stabilizes predictions but can trap the model in repetitive patterns; larger guidance weights may destabilize denoising during mode switching or high-variance states.Source: Limtations and Future Works · p. 12
• Guided diffusion is stable after establishing a gait orbit but tends to stumble at motion boundaries, while guidance works better for coarse than fine-grained objectives and still requires weight tuning.Future work could reduce history dependence and use supervised fine-tuning (55,56) or adapter-style control layers (57–59) for fine-grained composable trajectory control.Source: Limtations and Future Works · p. 12

Overview

• The framework combines shared-hyperparameter RL motion tracking with latent state–action diffusion to preserve agile, natural motion and compose skills for unseen scenarios.Classifier guidance enables online task optimization, while the latent state–action representation provides foresight by capturing how actions influence future states.Source: Overview · p. 13

Scalable Human Motion Tracking via RL

• The motion-tracking pipeline trains separate policies for each motion under one shared formulation and setup, producing sim-to-real-ready policies that scale to hundreds of skills without manual tuning.It formulates tracking as an MDP and solves it with RL using the same MDP and hyperparameters across motions.Source: Scalable Human Motion Tracking via RL · p. 14

• The objective is to reproduce a human-retargeted reference motion on a real humanoid with high fidelity while tolerating global drift from perturbations and sim-to-real mismatch.Source: Tracking Objective · p. 14
• Reference motion is represented by per-frame generalized positions and velocities, with forward kinematics providing each robot body's pose for tracking.Positions include base position, orientation, and joint angles; velocities include linear, angular, and joint velocities, with n_jnt denoting the number of joints.Source: Tracking Objective · p. 14
• A compact target-body subset, including the end-effectors, is selected to avoid redundant tracking of closely spaced links.Source: Tracking Objective · p. 14
• Because training perturbations and sim-to-real mismatch cause global drift, the policy tracks relative rather than global body poses to preserve motion style.Source: Tracking Objective · p. 14
• An anchor body, typically the root or torso, defines the anchor-centered frame in which desired poses of tracked bodies are expressed.Source: Tracking Objective · p. 14

𝑏, 𝑇anchor

• Yaw-aligned, height-preserving anchoring preserves motion style while allowing benign global drift, improving robustness.Desired twists remain unchanged under the transform.Source: 𝑏, 𝑇anchor · pp. 14–15

Rewards

• The reward uses a unified task-space tracking term plus only three lightweight penalties for joint limits, action smoothness, and self-collisions.Tracking covers position, orientation, linear velocity, and angular velocity across target bodies, while the penalties encourage physical consistency.Source: Rewards · p. 15

Observation and Action

• A continuous robot-centric observation without temporal stacking combines motion phase, anchor pose error, proprioception, and previous actions, while normalized joint-position setpoints shape torques through low-level PD control.The design supports balance, drift correction, push recovery, foot timing, stabilization, and jitter suppression without treating setpoints as high-precision position plans.Source: Observation and Action · p. 16

Domain Randomization

• Compact domain randomization improves scalable sim-to-real training by randomizing friction and restitution, default joint positions, torso center of mass, and training-time velocities.The approach avoids excessive randomization that can make behaviors conservative and training unnecessarily difficult.Source: Domain Randomization · p. 17

Adaptive Sampling

• Adaptive sampling prioritizes trajectory segments with higher empirical failure rates, then gradually returns toward uniform sampling after difficult segments are mastered.This addresses uneven segment difficulty in long motion sequences, where uniform sampling can oversample easy portions and slow or prevent convergence.Source: Adaptive Sampling · p. 17

Versatile Humanoid Control via Guided Diffusion

• The framework jointly models states and actions in a causally consistent latent diffusion model, predicting future trajectories whose guided states produce actions through receding-horizon control.This bridges state-space objectives and action-space policies without relying on high-dimensional, real-time forward dynamics models.Source: Versatile Humanoid Control via Guided… · p. 18

Latent Diffusion Models

• The latent diffusion model applies standard diffusion operations in a lower-dimensional VAE latent space, encoding data into z = E(x) and decoding it as x̂ = D(z).The VAE objective balances reconstruction accuracy with latent-space regularization through the ELBO.Source: Latent Diffusion Models · p. 18

LDMs for Trajectory Modeling

• For trajectory modeling, the system learns state–action distributions from motion-tracking policies, using smooth latent motion representations to avoid unstable direct action diffusion and inference latency.A VAE encodes motion intents, a state–latent diffusion model learns unlabeled trajectories, and the decoder converts denoised latents into dynamically consistent actions using current observations.Source: LDMs for Trajectory Modeling · pp. 19–20

Online Optimization via Guidance

• Classifier guidance transforms the diffusion score with differentiable task costs, enabling inference-time, training-free optimization for unseen joystick control, obstacle avoidance, and motion inpainting.The approach uses Bayes’ rule and associates desired trajectories with p(τ*|τ) ∝ exp(−G(τ)), without retraining the model.Source: Online Optimization via Guidance · p. 21

Validation of the method

• The validation ablations test motion-tracking MDP parameters and adaptive reset sampling, using repeated martial-arts trials with motion-capture ground truth to measure local and global tracking errors.These experiments assess design choices under challenging sim-to-real conditions and motivate the reported tracking and training changes.Source: Validation of the method · p. 22, MDP Design and Parameters · p. 22

Adpative Sampling

• Adaptive sampling is critical for learning difficult segments: without it, three out of four motions fail in challenging segments even after 30k training iterations, while easier Motion 4 requires 2k rather than 4k iterations.Source: Adpative Sampling · p. 23

Latent Diffusion

• 5% versus 95% success in sim-to-sim cartwheel transfer favors latent diffusion over the non-latent baseline, with performance transferring to the physical robot.The comparison uses full-cartwheel completion without falling as the success criterion, and the improvement is attributed to robustness to variations and noise.Source: Latent Diffusion · p. 23
• The ablation evaluates orientation representation, history length, armature, deployment delay, and adaptive sampling through motion-tracking and training-performance comparisons.Figure 8 organizes these tests into sim-to-real tracking and adaptive-sampling analyses.Source: Figure 8 · p. 31, Latent Diffusion · p. 31
• Ablations identify continuous orientations, no observation history, correct armature settings, and minimal deployment delay as critical for sim-to-real transfer.These factors determine motion-tracking performance in the evaluated setup.Source: Latent Diffusion · p. 31, Figure 8 · p. 31
• Without adaptive sampling, difficult motion segments remain unsolved even after 30k iterations, whereas adaptive sampling focuses early training on failures before spreading across the motion.The challenging segments include two cartwheel portions that cause the baseline to fail.Source: Latent Diffusion · p. 31

Armature

• Reflected rotor inertia, multiplied by the square of the gear ratio, contributes a large—often dominant—fraction of effective joint-axis inertia and changes the robot’s apparent inertia and dynamic response.It is a real physical contribution to joint inertia, not merely a numerical stability knob.Source: Armature · p. 44

Reward

• The tracking formulation averages position, orientation, linear-velocity, and angular-velocity errors across target bodies, normalizes them with Gaussian-shaped exponential functions, and penalizes action changes for smoothness.Source: Reward · p. 45

Observation

• Asymmetric actor–critics improve training efficiency by giving the critic per-body poses relative to the anchor, while linear anchor and root-velocity observations can be omitted when drift compensation or reliable state estimation is unavailable.The privileged critic can estimate body-wise Cartesian tracking errors directly.Source: Observation · p. 45

Heuristically Designed Paramters

• Joint stiffness and damping are heuristically set as kp,j = Ijω2 and kd,j = 2Ijζω using reflected inertia, with ζ = 2 (overdamped) because motor-only inertia typically underestimates apparent link inertia.Reflected inertia is used because subtree inertia varies with configuration, and exact precision is not required.Source: Heuristically Designed Paramters · pp. 45–46

Domain Randomization

• Domain randomization independently samples episode-level contact friction and restitution, calibration offsets, torso center-of-mass shifts, and periodic velocity disturbances to improve robustness.Static friction is sampled from U(0.3, 1.6), dynamic friction from U(0.3, 1.2), and restitution from U(0.0, 0.5).Source: Domain Randomization · p. 46

Termination and Adaptive Sampling

• Episodes terminate when tracking errors become unrecoverably large, at |e_p,z,b| > 0.25 m or ||e_R,anchor|| > 0.8 rad, while resets adaptively sample motion phases to balance difficulty.The smoothed failure rate is updated as ¯f_s ← 0.999 ¯f_s + 0.001 f_s to mitigate short-term fluctuations.Source: Termination and Adaptive Sampling · pp. 46–47

Complete List of Validated Motions

• The framework is validated across datasets from prior work, Unitree-retargeted LAFAN1, and online animation data, with the complete simulation and hardware motion list reported in Table S8.The supplementary materials identify broad applicability through multiple motion sources, including the Reallusion “MD Panther Lady” pack.Source: Complete List of Validated Motions · p. 47

Extra Ablation on PD Gains

• A moderate natural frequency yields the best global tracking performance and stronger local tracking than 5 Hz and ASAP; 25 Hz slightly lowers local pose error but causes high-frequency oscillation and loud actuator-gearbox sound.The study adopts 10 Hz in all experiments.Source: Extra Ablation on PD Gains · p. 48

Diffusion State

• The diffusion state uses a hybrid character-frame representation that makes root motion invariant to global translation and yaw while retaining detailed local body structure.Root features are expressed relative to the current character frame, whereas body features use instantaneous local root frames; an emphasis projection further weights root poses and twists before diffusion and pseudoinverse reconstruction.Source: Diffusion State · pp. 48–49

Diffusion Dataset Collection

• For offline-trained diffusion models, augmenting clean VAE rollouts with temporally correlated Ornstein–Uhlenbeck policy perturbations forms an error band that the policy learns to recover from during deployment.Source: Diffusion Dataset Collection · p. 49

Task Costs

• Task-specific costs cover joystick steering, waypoint navigation, and obstacle avoidance through velocity matching, position-to-velocity objectives, and signed-distance barrier penalties.The obstacle cost uses body positions, collision radii, and a relaxed barrier function.Source: Task Costs · pp. 49–50

Motion Tracking

• Motion-tracking policies run onboard Unitree G1 robots without external motion capture, using 500 Hz state estimation and ONNX Runtime inference under 1.0 ms per step.Extreme contact-rich behaviors may use LiDAR-inertial odometry or omit state-estimation-dependent observations.Source: Motion Tracking · pp. 50–51

Diffusion

• The diffusion policy uses a roughly 19.8M-parameter Transformer backbone and trains and deploys tracking policies at 25 Hz, with guiding-cost gradients computed automatically using CppAD.Source: Diffusion · p. 51
• Sagittal-symmetry augmentation trains on original and mirrored state–action pairs, effectively doubling the skill repertoire with minimal additional data.Both the VAE and diffusion model use the two data copies during training.Source: Diffusion · p. 51
• The supplementary inertia table defines subtree, armature, and effective inertia quantities for the left arm and left leg.The listed quantities include joint-axis subtree inertia, reflected motor/gear inertia, and armature contribution to effective inertia.Source: Table S4 · p. 55, Diffusion · p. 55
• Supplementary tables report motion-tracking, VAE, and diffusion-policy hyperparameters, including entropy coefficient 0.005 and value-loss coefficient 1.0.The motion-tracking hyperparameters are explicitly listed alongside separate VAE and diffusion-policy hyperparameter tables.Source: Table S5 · p. 56, Diffusion · p. 56, Table S6 · p. 57
• The evaluation covers walking, dancing, falling and getting up, running, jumping, and fighting or sports motions across specified subjects and temporal segments.Table S8 lists the motion segments tested in simulation and on real hardware, including full sequences and selected time intervals.Source: Table S8 · p. 59, Diffusion · p. 59