PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Humanoid Motion Representations for Physics-Based Control

Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, Weipeng Xu

arXiv:2310.04582v2cs.CVcs.GRcs.RO

TL;DR

Prior humanoid motion representations often cover specialized datasets or narrow movement styles, limiting their use across complex control tasks. PULSE distills a large-scale motion imitator into a variational latent space with a proprioception-conditioned prior, then reuses the frozen representation for hierarchical control. It outperforms alternative latent spaces across generative tasks and improves motion tracking, while still lagging training from scratch on some VR-controller tracking settings.

  • Problem

    Existing physics-based motion latent spaces often cover specialized datasets and limited movement styles, restricting their coverage of diverse motor skills.

  • Method

    PULSE distills a large-scale motion imitator into an encoder-decoder latent space with a variational bottleneck and a proprioception-conditioned learnable prior.

  • Results

    PULSE retains most of the imitator’s motor skills, outperforms latent-space baselines on generative tasks, and significantly improves tracking error over other latent models.

  • Takeaways & Limitations

    A single frozen motion representation supports generative control and free-form motion tracking across a broad range of humanoid tasks.

  • Takeaways & Limitations

    The variational bottleneck is lossy, and VR-controller tracking can lag behind training from scratch.

Abstract

from arXiv · show

We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. Due to the high dimensionality of humanoids and the inherent difficulties in reinforcement learning, prior methods have focused on learning skill embeddings for a narrow range of movement styles (e.g. locomotion, game characters) from specialized motion datasets. This limited scope hampers their applicability in complex tasks. We close this gap by significantly increasing the coverage of our motion representation space. To achieve this, we first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator. This is achieved by using an encoder-decoder structure with a variational information bottleneck. Additionally, we jointly learn a prior conditioned on proprioception (humanoid's own pose and velocities) to improve model expressiveness and sampling efficiency for downstream tasks. By sampling from the prior, we can generate long, stable, and diverse human motions. Using this latent space for hierarchical RL, we show that our policies solve tasks using human-like behavior. We demonstrate the effectiveness of our motion representation by solving generative tasks (e.g. strike, terrain traversal) and motion tracking using VR controllers.

2 RELATED WORK

Physics-based motion latent spaces reuse motor skills for humanoid control, but existing methods often cover specialized datasets or limited movement styles. Related approaches include physics-based tracking, adversarial learning, kinematics-based representations, and knowledge transfer.

  • Physics-based Humanoid Motion Latent Space: Adversarial physics-based representations such as ASE and CALM learn realistic behavior from random noise but may fail to cover diverse motor skills.Their reported strengths are on small, specialized datasets such as locomotion and sword strikes.
  • Physics-based Humanoid Motion Latent Space: Explicit motion-tracking methods learn latent spaces from data and have been applied to tasks including point-goal navigation, maze traversal, and boxing.ControlVAE, NPMP, PhysicsVAE, and NCP follow this general methodology, while some require adaptation layers for uneven terrains.
  • Kinematics-based Human Motion Latent Space: Kinematics-based motion representations are learned directly from motion data through supervised learning for motion generation and pose estimation.These methods compress multi-step motion into latent codes without involving physics simulation.
  • Physics-based Humanoid Motion tracking: Physics-based motion tracking has progressed from single-clip imitation to large-scale motion datasets using reinforcement learning.Mixture-of-experts, differentiable simulation, and external forces have been used to improve imitation quality.
  • Knowledge Transfer and Policy Distillation: Policy distillation transfers motor skills from a pretrained teacher, while kickstarting and DAgger-style procedures combine student exploration with teacher supervision.The paper applies this teacher-annotation perspective when distilling skills from a pretrained imitator.

3 PRELIMINARIES

The preliminaries define humanoid motion, task-conditioned reinforcement learning, and the physics-based control setup. Humanoid states combine full-body pose and velocities, while goals vary across imitation, trajectory following, and VR tracking tasks.

  • Motion Representation: Full-body pose q_t consists of 3D joint rotations and positions for all J humanoid links, using a 6-DOF rotation representation.The notation distinguishes ground-truth MoCap quantities from physics-simulation values.
  • Goal-conditioned Reinforcement Learning: Goal-conditioned policies are trained in a Markov Decision Process defined by states, actions, transition dynamics, rewards, and a discount factor.Physics simulation determines the state transitions, and PPO is used for policy training.
  • Goal-conditioned Reinforcement Learning: Proprioception s^p_t contains the humanoid’s body pose q_t and velocity q̇_t, while the goal state s^g_t depends on the current task.Goals include reference-motion imitation, 2D trajectory following, and 3D VR-controller tracking.
  • Motion Representation Pipeline: Figure 2’s pipeline distills a pretrained motion imitator into a latent representation whose decoder and prior are later frozen for downstream tasks.The representation uses a variational information bottleneck conditioned on proprioception.
  • Humanoid Model: The humanoid follows SMPL’s kinematic structure with 24 joints, 23 actuated joints, and actions specifying proportional-derivative controller targets.The action space is R^23×3.

4 PHYSICS-BASED UNIVERSAL HUMANOID MOTION LATENT SPACE

PULSE builds a universal humanoid motion latent space by distilling a large-scale motion imitator through a variational encoder-decoder and a proprioception-conditioned prior. The frozen latent controller is then used as the action space for hierarchical downstream policies.

  • 4 PHYSICS-BASED UNIVERSAL HUMANOID MOTION LATENT SPACE: PULSE first trains a motion imitator on a large-scale dataset, distills its motor skills into a latent model, and reuses that model for downstream tasks.The pipeline consists of motion imitation, online distillation, and hierarchical control.
  • 4.1 PHC+: ONE POLICY TO IMITATE THEM ALL: PHC+ reaches a 100% success rate on AMASS training data after modifications to the original PHC training procedure.The original PHC achieves 98.9%; PULSE uses PHC+ as its pretrained imitator.
  • 4.2 LEARNING MOTION REPRESENTATION VIA ONLINE DISTILLATION: The PHC+ imitator is distilled online with an encoder-decoder and conditional variational information bottleneck that models motor skills from proprioception.The decoder produces actions directly, while the encoder produces latent codes and the prior models their conditional distribution.
  • 4.2 LEARNING MOTION REPRESENTATION VIA ONLINE DISTILLATION: The training objective combines action matching, prior regularization, and KL divergence, while temporal regularization encourages nearby transitions to have similar latent codes.The regularizer improves continuity and is reported as crucial for downstream exploration.
  • 4.3 HIERARCHICAL CONTROL FOR DOWNSTREAM TASKS: For downstream control, the frozen decoder and simulation form a new dynamics system whose actions are high-level latent codes rather than joint torques.A task policy samples latent actions using the learned proprioception-conditioned prior.
  • 4.3 HIERARCHICAL CONTROL FOR DOWNSTREAM TASKS: Residual actions relative to the prior mean improve downstream performance, while omitting this formulation causes a significant performance drop.The prior supplies coherent motion as a starting point for exploration.

5 EXPERIMENTS

Experiments evaluate PULSE on motion imitation, generative control, VR tracking, and random motion generation. Across these settings, the representation retains imitation skills, supports human-like behavior, and improves downstream training and performance.

  • 5.2 DOWNSTREAM TASKS: Experiments cover speed, striking, reaching, terrain traversal, and VR controller tracking using AMASS and real-world evaluation data.Generative tasks include locomotion-related speed control, target striking, 3D reaching, and traversal of slopes, stairs, uneven surfaces, and obstacles.
  • 5 EXPERIMENTS: The experiments use a 32-dimensional latent space and evaluate metrics including normalized return, joint-position error, acceleration, velocity, and success rate.The latent code is approximately half the humanoid’s 69 degrees of freedom; ablations examine the prior, prior-action, regularization, and joint RL training.
  • 5.1 MOTION IMITATION: PULSE retains most motor skills from PHC+ after variational-bottleneck distillation, preserving accurate imitation across AMASS motion data.Without the variational bottleneck, distillation reaches 100% success; the observed degradation resembles non-zero VAE reconstruction error.
  • 5.2 DOWNSTREAM TASKS: PULSE outperforms latent-space baselines and training from scratch on generative tasks, with faster convergence even for unseen obstacles and stairs.The comparison includes ASE, CALM, and scratch-trained policies; qualitative results report human-like obstacle navigation without style or adversarial rewards.
  • 5.2 DOWNSTREAM TASKS: PULSE improves VR tracking error relative to other latent-space models while achieving tracking comparable to training from scratch.The representation can trade some success rate for precise tracking, sometimes favoring stability; Figure 5 reports faster convergence than scratch training.
  • 5.3 RANDOM MOTION GENERATION: Random sampling from the learned prior produces realistic, diverse motion with controllable smoothness or energy through input-noise variance.Compared with kinematics-based HuMoR and physics-based ASE and CALM, the representation is described as having broader coverage and more natural motion.

6 DISCUSSION AND FUTURE WORK

PULSE provides a universal motion representation for physics-based humanoid control, but its variational bottleneck and downstream use still have limitations. Future work targets more interpretable representations and richer interaction modeling.

  • Limitations: PULSE does not achieve a 100% imitation success rate because its variational information bottleneck lossy-compresses motor skills.An unconstrained latent space can reach 100% success through online distillation.
  • Limitations: For VR controller tracking, PULSE can lag behind training from scratch, while generated motion may become stuck in fallen or standing states.Increasing system noise may jolt the humanoid out of these states.
  • Future work: PULSE can serve as a generative motion prior for downstream generative and estimation tasks without additional modification or adaptive layers.Future work includes human-interpretable representations, scene information, human-object interactions, and articulated fingers.

A INTRODUCTION

The supplementary material expands the method and reports qualitative evaluations of imitation, generation, and downstream tasks. These results indicate that PULSE retains core imitator abilities while producing diverse, physically plausible motion.

  • Supplementary material: The supplement adds details on PHC+, PULSE architecture, training, and downstream task configurations, with code and models planned for release.These materials are organized in supplementary sections covering PHC+ modifications and PULSE implementation.
  • Qualitative results: PULSE largely retains PHC+ abilities after online distillation and generates long-term, diverse, human-like motions from its prior and decoder.Input-noise variance can control behaviors during random generation.
  • Qualitative results: Compared with HuMoR, PULSE motion remains physically plausible, while its random generation appears more diverse than ASE and CALM.The supplement also visualizes generative and estimation/tracking downstream tasks against state-of-the-art methods.

B.1 DATA CLEANING

The data-cleaning process addresses dynamic motions and corrupted motion-capture sequences that can cause imitation failures or teach undesirable behavior. The resulting dataset is filtered for motion imitation training and testing.

  • Failure analysis: Dynamic motions such as cartwheeling and consecutive back flips are identified as major sources of imitation failure.Motion-capture sequences can also contain discontinuities and penetrations caused by optimization or fitting failures.
  • Control and rewards: The controller action targets each of the humanoid’s 69 actuators through a proportional-derivative controller.Joint torques are capped at 500 N-m.
  • Control and rewards: The motion-tracking reward measures translation, rotation, linear velocity, and angular velocity across 23 rigid bodies.The reward also includes adversarial motion-prior and energy terms; the energy penalty regulates the policy and prevents high-frequency jitter.

B.3 MODEL ARCHITECTURE AND ABLATIONS

PHC+ uses larger SiLU multilayer perceptrons and progressive training to improve tracking and fail-state recovery. Ablations attribute gains to the progressive procedure and activation-function change.

  • Architecture: PHC+ primitives and composers use six-layer SiLU MLPs, replacing ReLU and expanding the network beyond the prior three-layer design.The authors report a non-trivial tracking-performance boost from SiLU and use three primitives for fail-state recovery.
  • Ablations: 100% success rate is achieved for fail-state recovery using three primitives.This result is reported for the PHC+ training configuration.
  • Ablations: 3 × 10^9 samples are used to train each primitive.The ablations evaluate progressive training and the activation-function change on a single primitive P.
  • Ablations: Progressive training substantially improves success rate, while changing ReLU to SiLU improves tracking performance and Empjpe.The progressive procedure updates Q̂hard during each primitive’s training.

C.1 TRAINING PROCEDURE

PULSE training uses progressive primitive formation and online distillation from the pretrained PHC+ imitator, with the KL coefficient annealed during training.

  • Training procedure: Progressive training updates Q̂hard during primitive learning rather than waiting for convergence before initializing a new primitive.This is the meaning assigned to “Progressive” in the ablation table caption.
  • Training procedure: Online distillation queries πPHC+ during training because πPULSE and πPHC+ share the same state and action spaces.The KL coefficient β is annealed from 0.01 to 0.001 between 2.5 × 10^9 and 5 × 10^9 samples, then held constant.

C.2 COMPARISON TO TRAINING SCRATCH WITHOUT DISTILLATION

The scratch-training comparison finds that variational and alternative latent spaces can imitate effectively, but random sampling fails to produce coherent generative motion without an appropriate prior.

  • Comparison to training scratch without distillation: Training PULSE from scratch with a variational bottleneck and imitation objective becomes unstable, motivating online distillation from the trained imitator.The authors contrast this with prior work that trained a policy from scratch using reinforcement learning.
  • Comparison to training scratch without distillation: Random samples from the spherical and quantized spaces do not generate coherent motion, while quantization additionally introduces high-frequency jitter artifacts.The quantized representation requires an additional prior and switches between discrete codes.

C.4 DOWNSTREAM TASKS

The downstream policies apply the representation across speed, reach, striking, terrain traversal, and VR tracking, using PPO and task-specific observations and rewards.

  • Downstream task setup: Each downstream policy uses a three-layer MLP and PPO, while VR tracking uses a six-layer MLP; training requires approximately 2 × 10^9 samples for simple tasks and 1 × 10^10 for terrain traversal.The value function matches the policy architecture.
  • Generative tasks: Speed targets sample velocities from 0 to 5 m/s, matching the approximate maximum running speed represented in AMASS.The task reward compares target velocity with the humanoid root velocity.
  • Generative tasks: Strike requires the right hand to knock over a target while terminating if another body part contacts it.The reward measures target uprightness using the global and target up vectors.
  • Generative tasks: Reach samples a 3D point from a 2-meter box centered at (0, 0, 1), rewarding proximity of the right hand to that point.The stated reach reward uses an exponential distance penalty.
  • Generative tasks: Terrain traversal follows random trajectories across stairs, slopes, uneven surfaces, and obstacles using a local height map and the next ten trajectory steps.The task combines trajectory following with an energy penalty and does not require an additional adversarial reward for human-like behavior.
  • Motion tracking: VR tracking infers full-body motion from three 6DOF poses—headset and two hand controllers—using synthetic training data and a full-body imitation reward.The policy tracks only the head and hands as controller proxies while using progressive training.
Loading 2310.04582v2…