Source-linked AI summary
Neural probabilistic motor primitives for humanoid control
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, Nicolas Heess
TL;DR
The paper asks how one motor module can flexibly control high-dimensional simulated humanoids across many behaviors. It proposes an offline-trained latent-variable motor architecture that compresses thousands of expert policies, enabling one-shot imitation and skill reuse, while LFPC provides an experience-efficient cloning alternative.
Problem
Flexible humanoid control requires a shared representation that can generalize, sequence, and compose many skills rather than reproduce only isolated behaviors.
Method
The paper trains neural probabilistic motor primitives offline from expert policies and compares behavioral cloning with linear feedback policy cloning for skill transfer.
Results
The module performs robust one-shot imitation, including unseen trajectories, and LFPC transfers expert behavior using single rollouts, though it performs less well than noisy-rollout cloning in the full-scale model.
Takeaways & Limitations
The learned motor primitive module is interpretable and reusable, supports behavior transitions, and can be repurposed for new tasks with relatively naturalistic movement.
Takeaways & Limitations
The work is restricted to non-object-interacting motor behaviors with the full behavior set available in advance.
Abstract
from arXiv · showhide
We focus on the problem of learning a single motor module that can flexibly express a range of behaviors for the control of high-dimensional physically simulated humanoids. To do this, we propose a motor architecture that has the general structure of an inverse model with a latent-variable bottleneck. We show that it is possible to train this model entirely offline to compress thousands of expert policies and learn a motor primitive embedding space. The trained neural probabilistic motor primitive system can perform one-shot imitation of whole-body humanoid behaviors, robustly mimicking unseen trajectories. Additionally, we demonstrate that it is also straightforward to train controllers to reuse the learned motor primitive space to solve tasks, and the resulting movements are relatively naturalistic. To support the training of our model, we compare two approaches for offline policy cloning, including an experience efficient method which we call linear feedback policy cloning. We encourage readers to view a supplementary video ( https://youtu.be/CaDEf-QcKwA ) summarizing our results.
1 INTRODUCTION
The paper addresses flexible humanoid control by compressing many specialized skills into a shared motor module that can imitate, sequence, and reuse behaviors. It introduces neural probabilistic motor primitives and offline policy-cloning methods to support this transfer at scale.
- Humanoid control must handle high-dimensional bodies, balance, locomotion, and the persistent idiosyncrasies of learned behaviors.
- Motion-capture data enables humanlike movement, but existing expert policies typically reproduce individual clips narrowly.
- Flexible control requires representing, recruiting, sequencing, and composing many individual skills in task-directed ways.
- Neural probabilistic motor primitives represent and generate many behaviors through a shared embedding space designed for one-shot imitation and controlled composition.
- The system transfers thousands of expert skills offline using behavioral cloning or linear feedback policy cloning, reducing reliance on closed-loop reinforcement learning.
- One-shot imitation lets a trained system watch an example behavior at test time and reproduce it, while related movement-primitive approaches parameterize and learn robot trajectories.
2 TRANSFER AND COMPRESSION OF EXPERT BEHAVIORS
The paper develops an offline-trained latent-variable motor module that compresses many humanoid expert behaviors into a shared representation for imitation, transfer, and composition. It also introduces LFPC to reduce the expert rollouts needed for policy cloning while preserving behavior around nominal trajectories.
- Neural probabilistic motor primitives: The system targets a motor primitive module that can deploy, sequence, and interpolate diverse skills without manually aligning or processing raw expert trajectories.The representation should encode behavioral modes, support behavior indexing, and place similar behaviors near one another for reliable transitions.
- Neural probabilistic motor primitives: The proposed architecture is an autoregressive latent-variable inverse model conditioned during training on short look-ahead snippets of reference trajectories.The encoder represents q(z_t|z_t−1, x_t), while the decoder models π(a_t|z_t, s_t).
- Offline training: The model is trained offline by optimizing an ELBO with a tunable β weighting the latent prior, using supervised behavioral cloning from noisy rollouts.For β = 1, the objective is the variational lower bound on log p(a_1:T|s_1:T).
- Neural probabilistic motor primitives: The conditional information bottleneck compresses future-trajectory information into latent motor representations, while an autocorrelated prior encourages temporally nearby latent states to remain close.The model uses an AR(1) prior, with α = 0.95 in the reported experiments unless otherwise stated.
- Offline training: Behavioral cloning can require many rollouts because the training states must adequately cover the state distribution later experienced by the student.This motivates small-sample estimates of expert state distributions for scalable compression of thousands of experts.
- Offline training: LFPC records an expert’s action-state Jacobian and action along one nominal trajectory, then uses linear corrections to provide target actions in nearby perturbed states.Its purpose is to match behavioral cloning performance while requiring considerably fewer expert rollouts.
3 EXPERIMENTS
Experiments validate offline transfer and compression of humanoid behaviors, then examine one-shot imitation, latent-space optimization, skill reuse, and sparse-reward control.
- 3.1 VALIDATION: TRANSFER OF SINGLE-BEHAVIOR POLICIES: Linear feedback policy cloning approximates expert behavior reasonably well in single-skill transfer experiments.
- 3.2 CORE RESULTS: COMPRESSING THOUSANDS OF EXPERTS: The trained module executes open-loop noisy latent trajectories, indicating that its decoder stabilizes latent-conditioned behavior.
- 3.2 CORE RESULTS: COMPRESSING THOUSANDS OF EXPERTS: Among model variations, cloning with 100 trajectories per expert and medium regularization (β = 0.1) performs best, while LFPC performs less well but similarly.
- 3.3 ANALYSIS OF THE TRAINED MODEL: Optimization improves failed one-shot imitations, raising median relative expert performance from 43% to 78% among clips initially below 50%.
- 3.3 ANALYSIS OF THE TRAINED MODEL: Latent-variable trajectories can sometimes be concatenated to produce seamless transitions between behaviors, such as sequences of turns.
- 3.3 ANALYSIS OF THE TRAINED MODEL: A high-level controller can use the learned primitive space for sparse-reward locomotion, producing humanlike task-directed movement and improving with more regularized modules.
4 DISCUSSION
The paper demonstrates offline transfer and compression of control policies into a reusable motor primitive module for simulated humanoids. LFPC enables transfer from a single rollout, though it underperforms noisy-rollout cloning in the full-scale model and remains limited in scope.
- The neural probabilistic motor primitive system represents and executes motor behaviors for simulated humanoid control through offline policy transfer and compression.
- LFPC achieves robust one-shot imitation using only a single rollout from each expert.
- LFPC performs less well than cloning from noisy rollouts in the full-scale model.
- The authors identify costly rollouts as a setting where LFPC may be useful, while noting that its parameters, especially the marginal noise distribution ∆, were not carefully tuned.
- The work is restricted to motor behaviors without object interactions and assumes that the full set of behaviors is available in advance.
A MOTION CAPTURE EXPERTS
The expert policies are produced by training time-indexed controllers to imitate motion-capture reference clips using a per-timestep reward derived from the SAMCON energy function.
- Expert policies track and imitate motion-capture reference clips using a time-indexed policy trained with a per-timestep SAMCON energy reward.
- The expert-generation approach largely follows the procedure of Peng et al. (2018).
B ARCHITECTURE AND TRAINING DETAILS
The implementation uses a Gaussian MLP decoder, reparameterization-based optimization, and subsequence sampling procedures for training, including perturbed sequences for LFPC.
- The decoder is a three-layer MLP with 1024 hidden units that takes the state and latent variable as input.
- The decoder outputs a multivariate Gaussian with fixed standard deviation 0.1, which made preventing overfitting significantly easier.
- Training uses the reparameterization trick and ADAM stochastic gradient descent with learning rate 0.0001.
- For models trained on 100 trajectories per expert, minibatches contain 512 subsequences of length 30.
- LFPC samples 32 subsequences of length 30 and generates five perturbed state sequences per subsequence.
C RELATIONSHIP TO OTHER KNOWLEDGE TRANSFER IDEAS
The proposed transfer approach focuses on matching expert responsiveness around each state rather than merely matching actions or activation distributions at individual states.
- The proposal emphasizes matching the expert policy’s responsivity in a neighborhood around each state.
- This differs from activation matching or KL matching, which focus on matching action or activation distributions for a particular state.
- Blindly reproducing expert actions despite input perturbations removes local feedback and produces open-loop-like behavior.
D VISUALIZATION OF STATIONARY POLICY BEHAVIOR
LFPC generalizes running behavior from three gait cycles despite noise: cloned-policy rollouts deviate from the reference but return to the limit cycle.
- Three gait cycles of running behavior were used to test LFPC, without requiring the student policy to be time-indexed.The demonstrations were time-indexed, but the student policy was not.
- Noisy cloned-policy rollouts are considerably noisier than the reference trajectory and consistently deviate from it in PCA pose space.The stationary cloned policy trajectories are shown in blue, while projected reference data are shown in green.
- Despite these deviations, the cloned-policy trajectories return to the locomotion limit cycle.