Source-linked AI summary
Learning human behaviors from motion capture by adversarial imitation
Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, Nicolas Heess
TL;DR
Pure reinforcement-learning rewards often produce stereotyped, non-humanlike movement, motivating a method that can learn from demonstrations instead. The paper extends GAIL to partial, action-free observations and mismatched bodies, then uses learned low-level skills within higher-level controllers; it reports reusable humanlike behaviors and robust multi-behavior transitions from small, noisy motion-capture datasets.
Problem
The central gap is learning reusable humanlike humanoid behaviors without relying on difficult hand-designed reward functions or complete matching demonstrations.
Method
The method extends GAIL with partial state features, no demonstrator actions, body-transfer capability, and context labels for simultaneous multi-behavior training.
Results
The approach produces reusable humanlike skills from very small quantities of noisy motion-capture data and supports robust transitions between learned behaviors.
Takeaways & Limitations
Learned low-level skills can be embedded in larger control systems where a higher-level controller learns by reinforcement learning to solve new tasks.
Takeaways & Limitations
The demonstrated scope remains limited to a restricted set of behaviors with limited reuse, and behavior quality must currently be assessed largely through human judgment.
Abstract
from arXiv · showhide
Rapid progress in deep reinforcement learning has made it increasingly feasible to train controllers for high-dimensional humanoid bodies. However, methods that use pure reinforcement learning with simple reward functions tend to produce non-humanlike and overly stereotyped movement behaviors. In this work, we extend generative adversarial imitation learning to enable training of generic neural network policies to produce humanlike movement patterns from limited demonstrations consisting only of partially observed state features, without access to actions, even when the demonstrations come from a body with different and unknown physical parameters. We leverage this approach to build sub-skill policies from motion capture data and show that they can be reused to solve tasks when controlled by a higher level controller.
1 Introduction
The paper targets humanlike, reusable humanoid movement by combining motion-capture imitation with reinforcement-learning control. Its pipeline trains low-level skills from demonstrations and uses a high-level controller to sequence them for new tasks.
- Motivation: Pure reinforcement learning with simple rewards tends to produce non-humanlike, overly stereotyped movement and requires repeated reward engineering for new behaviors.Motion capture offers a more direct alternative, but existing approaches can violate physical constraints or require specialized trajectory controllers and engineered parameterizations.
- Approach: GAIL avoids requiring an explicit hand-designed similarity metric by training a discriminator to distinguish demonstration state-action data from imitation data.The policy receives reward for fooling the discriminator, paralleling the generator-discriminator structure of GANs.
- Contribution: The paper extends GAIL to learn humanlike behaviors from limited motion-capture demonstrations without requiring demonstrator actions or matching body dynamics.The demonstrations may contain noisy, physically inconsistent motion and originate from bodies with different physical parameters.
- Approach: The approach first trains low-level controller policies by imitation learning, then trains a high-level controller by reinforcement learning to reuse those skills on new tasks.This structure supports movement patterns that can be reused, refined, and flexibly sequenced.
- Contribution: The resulting skills are reported as reusable humanlike behaviors learned from very small quantities of noisy motion-capture data without extensive domain engineering.The paper also reports that robust transitions between multiple behaviors emerge during training.
2 Methods
The method adapts GAIL for partial, action-free observations, multiple behaviors, and transfer across bodies. Policies are trained through alternating rollouts, discriminator-based rewards, and discriminator updates in MuJoCo environments using noisy motion-capture data.
- Extensions: The original GAIL setting used complete state-action trajectories from matching bodies and policy architectures, whereas this work targets partial observations and body transfer.The authors motivate these extensions as necessary for motion-capture imitation when actions and matching physical parameters are unavailable.
- Transfer: The discriminator can support transfer when observed features are sufficiently invariant across bodies, without requiring the demonstrator's physical parameters or dynamics.Validation covers partial state observations and imitation across bodies, including two-link and three-link arms.
- Extensions: The GAIL variant excludes actions from discriminator features and adds a context label so one policy can learn multiple behaviors and transition between them.Demonstrations and generated trajectories are compared in a shared feature space, which may contain only selected state features.
- Training procedure: Each training iteration collects policy data, updates the policy using discriminator rewards, and updates the discriminator using demonstration and generated data.The policy is stochastic and produces Gaussian action distributions for actuators; policy optimization uses trust-region policy optimization among possible update methods.
- Experimental setting: Experiments use MuJoCo bodies including walkers, planar arms, and a custom humanoid, while motion-capture trajectories may remain dynamically inconsistent after limited preprocessing.The custom humanoid has 56 joint-angle degrees of freedom plus a 6-DOF root joint.
3 Results
The authors validate adversarial imitation without actions and with partial observations, then apply it to humanoid motion capture and reuse learned skills for higher-level tasks. Motion-capture initialization, invariant features, contextual modulation, and fine-tuning support more natural, transferable, and task-capable behavior.
- Humanoid motion capture: Motion-capture pose initialization produced a more natural, though still awkward, gait than randomized initialization.The randomized-initialization policy exhibited an extremely unnatural sideways skip.
- Validation: State-only adversarial imitation was sufficient in the planar walker, with discriminator action inputs providing no training benefit.The interpretation is that actions can often be inferred from successive states and body-environment constraints.
- Validation: A three-link arm imitated a two-link arm when the discriminator observed only end-effector displacement from the target.The two-link imitator performed roughly equivalently with full observations or the restricted end-effector feature.
- Humanoid motion capture: Using egocentric sensors and root-to-end-effector vectors dramatically improved imitation quality for humanoid walking.The discriminator received velocimeter, gyroscope, accelerometer, and vectors to the feet, hands, and head rather than non-egocentric features.
- Contextual modulation: A contextual low-level controller switched among straight running and left or right veering using only 286 time-resampled frames.The context was externally set at runtime, enabling interactive modulation and higher-level control.
- Task-based control: Fine-tuning preserved flat-ground locomotion while enabling the running policy to ascend and descend stairs with two discrete slopes.Without tuning, the running policy fell quickly on the stairs.
- Task-based control: A two-layer neural higher-level controller used a local top-down depth camera to send context signals for autonomous navigation along a novel procedurally generated course.Its reward corresponded to forward movement along a linear 3D track.
4 Relationships with related work
Related approaches include kinematic sequence models, specialized gait or foot-placement controllers, and recent deep RL or motion-capture systems. The authors position their method as more general, with fewer assumptions and less controller and reward structure.
- Existing approaches: Kinematic sequence models generate plausible motion trajectories without physical control, while physical animation methods often require substantial domain expertise.Examples include specialized gait controllers, explicit foot-placement systems, and curated motion capture.
- Recent deep learning: Recent deep RL and motion-capture work reduced domain knowledge but prioritized aesthetically pleasing motion over generic reusable tools.The cited work includes 2D locomotion, 3D humanoid control, and specialized neural architectures for motion-capture sequence modeling.
- Positioning: The authors claim fewer assumptions about motion-capture tracking, a more general learning approach, and less controller and reward-function structure.Their stated contrast is with approaches designed primarily to produce aesthetically pleasing motions.
5 Discussion
The discussion frames imitation learning as a response to the difficulty of specifying complex behaviors with objective reward functions. It also identifies human judgment, restricted behavior coverage, and limited skill reuse as current boundaries.
- Demonstrations offer a straightforward way to communicate complex behaviors to artificial agents.
- The quality of produced behaviors remains difficult to assess because objective functions for complex behaviors are lacking, requiring human judgment.
- The demonstrated behavior repertoire is diverse but restricted, and skill reuse has so far been limited.
A Additional motion capture details
Motion-capture data are drawn from the CMU database and converted into observations for a programmatically generated humanoid body. The data-processing pipeline resamples poses and logs body-state features.
- The CMU database supplies subject-specific body segment lengths and per-frame joint-angle poses.ASF files provide segment lengths, while AMC files provide joint-angle poses.
- The humanoid model is generated from subject 8's segment lengths with asserted values for other proportions and actuator strengths.
- AMC motion is spline-interpolated from 120Hz to 30ms timesteps, then body-state features are logged as observations.
B Additional experimental details
Training uses input and reward preprocessing that differs between reinforcement learning and imitation learning. Policy action variance is initialized within a specified range.
- RL training applies online z-filtering to network inputs and rewards.
- Imitation-learning rewards are not filtered but are clipped at a maximum value of 10, with advantages normalized per iteration.
- Action variance sigma is generally initialized between e−1 ≈.37 and 1.
Experiments using walker:
Walker experiments expose body-state observations to both the policy and discriminator, while Figure A1 depicts equispaced frames of a baseline RL-trained walker running.
- Policy and discriminator observations include vertical body position, joint angles, joint angular velocities, and horizontal and vertical velocity.
- Walker experiments use 30,000 samples per iteration, and adversarial imitation from RL demonstrations uses 100 demonstration episodes.
- Figure A1 presents equispaced frames of a baseline RL-trained walker running.
Experiments using arms:
Arm policies use joint-angle trigonometric features, joint angular velocities, and target-relative end-effector information. The experiments also specify demonstration scale and discriminator hyperparameter sweeps.
- Observations include the sine and cosine of each joint angle, the joints’ angular velocities, and the vector from the end-effector to the target center.
- Arm experiments used 20,000 samples per iteration.
- Adversarial imitation from reinforcement-learning demonstrations used 100 demonstration episodes, while discriminator learning rates were swept over 1e-4, 1e-3, and 1e-2.
Experiments using complex humanoid:
The complex humanoid experiments combine physically simulated bodies, reinforcement-learning locomotion objectives, and imitation-learning training. Higher-level navigation controllers use depth-map and egocentric state information with generic neural-network policies.
- Simulation and control: Humanoid environments used a 5ms simulation timestep and a 30ms control timestep, with neural-network actions bounded in [-1,1].Joint-angle torque generators assign different gains to action dimensions, including higher gains for legs and smaller gains for toes.
- Locomotion reinforcement learning: The locomotion task placed the humanoid on a plane and rewarded forward velocity with a small control cost; falling terminated episodes early.This termination affects the reward and biases visited states toward those expected to be more reasonable.
- Training duration: Humanoid reinforcement-learning training took 1,000–1,500 iterations to saturate, while motion-capture imitation ran for up to 3,000 iterations.All humanoid experiments used 100,000 samples per iteration, and imitation in simpler environments took roughly as long as reinforcement learning.
- Imitation learning: Imitation policies used a three-hidden-layer neural network with sizes 300, 200, and 100 and hyperbolic-tangent nonlinearities.Walking imitation also used early episode termination when the body fell.
- Higher-level navigation: Navigation used an approximate TRPO method, with a higher-level controller observing a top-down depth map extending up to 15m ahead.The controller also received egocentric heading and lateral position, but not position along the track.