PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills

Xue Bin Peng, Pieter Abbeel, Sergey Levine, Michiel van de Panne

arXiv:1804.02717v3cs.GRcs.AIcs.LG

TL;DR

Physics-based character animation needs to combine motion examples with controllers that can handle diverse behaviors, recoveries, and goals. This paper combines reference-motion imitation with goal-directed reinforcement learning, producing robust, natural skills across characters, environments, tasks, and composite multi-skill policies, while retaining limitations in timing adaptation and scalability to large motion libraries.

  • Problem

    Physics-based character animation has limited ability to simulate diverse real-world behaviors while generalizing to new skills and situations and achieving directable task goals.

  • Method

    The framework directly rewards policies for resembling reference motion data while also achieving task objectives, and integrates multiple clips through max-reward training, user-triggered multi-skill policies, or value-based sequencing.

  • Results

    The method produces a broad range of challenging, natural, robust skills, supports retargeting across characters, environments, and tasks, and enables composite policies executing multiple skills.

  • Takeaways & Limitations

    Reference-motion imitation combined with goal-directed reinforcement learning provides a flexible basis for physics-based characters that perform diverse skills and interactive tasks.

  • Takeaways & Limitations

    Policies require a linearly advancing phase variable synchronized with the reference motion, multi-clip integration has only been demonstrated for small clip sets, and training is time consuming.

Abstract

from arXiv · show

A longstanding goal in character animation is to combine data-driven specification of behavior with a system that can execute a similar behavior in a physical simulation, thus enabling realistic responses to perturbations and environmental variation. We show that well-known reinforcement learning (RL) methods can be adapted to learn robust control policies capable of imitating a broad range of example motion clips, while also learning complex recoveries, adapting to changes in morphology, and accomplishing user-specified goals. Our method handles keyframed motions, highly-dynamic actions such as motion-captured flips and spins, and retargeted motions. By combining a motion-imitation objective with a task objective, we can train characters that react intelligently in interactive settings, e.g., by walking in a desired direction or throwing a ball at a user-specified target. This approach thus combines the convenience and motion quality of using motion clips to define the desired style and appearance, with the flexibility and generality afforded by RL methods and physics-based animation. We further explore a number of methods for integrating multiple clips into the learning process to develop multi-skilled agents capable of performing a rich repertoire of diverse skills. We demonstrate results using multiple characters (human, Atlas robot, bipedal dinosaur, dragon) and a large variety of skills, including locomotion, acrobatics, and martial arts.

1 INTRODUCTION

DeepMimic combines reference-motion imitation with task-directed reinforcement learning to produce physically realistic, robust character behaviors. It also supports multiple clips and diverse skills while addressing limitations of prior controllers and deep RL.

  • Physics-based character simulation struggles to reproduce diverse behaviors and generalize across new skills and situations.Manually designed controllers can produce compelling results but depend on human insight and are limited in generalization.
  • Prior deep reinforcement learning methods generated complex behaviors, but their motion quality lagged behind kinematic methods and manually designed controllers.The resulting controllers could exhibit severe motion artifacts.
  • Reference-tracking controllers have limited ability to modify motions for plausible recoveries or task goals that substantially deviate from the kinematic motion.Such methods are also complex to implement.
  • DeepMimic directly rewards learned controllers for resembling reference animations while achieving additional task objectives.The framework also demonstrates max-based multi-clip rewards, user-triggered multi-skill policies, and value-function-based sequencing of single-clip policies.
  • The framework combines goal-directed reinforcement learning with motion-capture or keyframed data to produce physics-based character skills.The authors report motion quality and robustness that substantially exceed prior work, with natural perturbation recoveries requiring no human engineering.

2 RELATED WORK

Related work spans kinematic motion synthesis, manually designed physics-based controllers, reinforcement learning, and reference-motion imitation. DeepMimic is positioned near systems that reproduce diverse dynamic skills while seeking greater flexibility for task objectives.

  • Kinematic Models: Kinematic methods use motion datasets to select clips, learn latent representations, or generate motions with deep models.With high-quality data, these methods often produce higher-quality motions than most simulation-based approaches.
  • Physics-based Models: Physics-based character controllers have often relied on human insight and task-specific strategies, especially for locomotion.Many use simplified models combined with optimization processes.
  • Reinforcement Learning: Reinforcement learning reduces task-specific control structure, but learned behaviors generally appear less natural than manually engineered counterparts.A stated challenge is specifying reward functions for natural movement.
  • Motion Imitation: Reference-motion imitation has been applied to bipedal locomotion and 3D humanoid tracking through policy-search and model-based methods.The related work establishes motion imitation as a longstanding animation approach.
  • Sampling-Based Controllers: SAMCON reproduces a diverse corpus of dynamic and acrobatic motions, but its complex design and low-dimensional state representation constrain extension to task objectives.The text specifically questions extension to objectives involving significant sensory input.

3 OVERVIEW

DeepMimic takes a character model, reference motions, and a reward-defined task, then learns a neural controller that imitates the motions while satisfying task objectives.

  • The system inputs a character model, kinematic reference motions, and a task specified by a reward function.
  • A policy maps the character state and task-specific goal to actions that specify target joint angles for PD controllers.The PD controllers convert those targets into joint torques in the simulation.
  • The final policy combines an imitation reward from reference motions with a task-specific reward defined by the goal.Example goals include striking a target or running in a desired direction over irregular terrain.
  • Neural-network policies are trained with proximal policy optimization to imitate reference behaviors while fulfilling task objectives.

4 BACKGROUND

The paper frames character control as reinforcement learning over trajectories generated by a policy interacting with simulated dynamics. Policy-gradient optimization increases the likelihood of actions with higher-than-expected returns, and DeepMimic uses PPO with value and advantage estimation methods.

  • At each timestep, the policy observes a state, samples an action, and receives a successor state from the environment dynamics.
  • A trajectory is a sequence of states and actions induced by the policy, with its return defined as the discounted sum of rewards over the episode horizon.The discount factor γ can ensure the return remains finite.
  • The policy-gradient estimator weights log-policy gradients by the advantage of each action in its state.The advantage measures whether an action performs better or worse than expected.
  • Policy gradients increase the likelihood of actions producing higher-than-expected returns and decrease the likelihood of actions producing lower-than-expected returns.REINFORCE is given as a classic policy-gradient algorithm.
  • DeepMimic trains policies with PPO, trains value functions using TD(λ) multi-step returns, and computes advantages with GAE(λ).

5 POLICY REPRESENTATION

The policy maps character state and task goals to actions while combining reference-motion imitation with task rewards. Its inputs can include phase, goals, and terrain heightmaps, and its reward decomposes imitation into motion-matching terms.

  • Policy inputs and actions: State features encode local link positions, joint rotations, linear and angular velocities, and a phase variable ϕ indicating progress through the reference motion.For task-directed policies, the goal д is provided alongside the state; cyclic motions reset ϕ after each cycle.
  • Policy inputs and actions: The policy maps character state s and task goal д to an action distribution, using a Gaussian with network-defined mean and fixed diagonal covariance.The action specifies target joint orientations for PD controllers, and the policy is queried at 30Hz.
  • Network architecture: For vision-based tasks, a terrain heightmap H is processed with convolutional layers, concatenated with state and goal features, and passed through fully connected layers to produce the policy output.Without a heightmap, the network uses only the state-and-goal processing layers; hidden layers use ReLU activations.
  • Reward design: The step reward combines imitation and task objectives with weights ωI and ωG, encouraging reference-motion matching while satisfying task-specific goals.The imitation objective is decomposed into terms for characteristics such as joint orientations and velocities.
  • Reward design: Imitation rewards measure deviations in joint orientations, local joint velocities, end-effector positions, and center-of-mass position relative to the reference motion.End-effectors include the left foot, right foot, left hand, and right hand; joint orientation differences use quaternion operations.

6 TRAINING

Training uses reference-motion states and carefully chosen episode termination to improve exploration and learn difficult physics-based skills. Policies are optimized with PPO using imitation and task-related objectives.

  • Training procedure: PPO trains separate policy and value networks from episodic rollouts initialized by states sampled uniformly from the reference motion.The value function uses TD(λ), while policy advantages are computed with GAE(λ).
  • 6.1 Initial State Distribution: Fixed initial states can make sequential motions and difficult recoveries hard to learn because later phases provide reward only after earlier phases are mastered.The paper highlights backflips, where unsuccessful landings can prevent useful returns from the jump.
  • 6.1 Initial State Distribution: Reference state initialization exposes the agent to desirable states along a motion before it can reach them through its own actions.This provides a richer initial-state distribution for exploration in motion imitation tasks.
  • 6.2 Early Termination: Early termination ends an episode when specified links, such as the torso or head, contact the ground and assigns zero reward thereafter.This also shapes rewards against undesirable behaviors and curates the training-state distribution.
  • 6.2 Early Termination: Without early termination, training data can be dominated by futile ground states, creating a class-imbalance-like problem that consumes model capacity.Terminating failure states mitigates this imbalance.

7 MULTI-SKILL INTEGRATION

The framework integrates multiple motion clips through flexible rewards, user-controlled skill selection, or runtime composition of independently trained policies. These methods support sequencing diverse skills without always requiring new training.

  • Multi-Clip Reward: A multi-clip reward uses the maximum imitation objective across reference clips, allowing the policy to select and switch clips as appropriate.The objective avoids requiring a manually crafted kinematic planner for clip selection.
  • Skill Selector: A skill-selector policy receives a one-hot user goal and learns to imitate diverse skills while executing arbitrary requested sequences.Random skill goals sampled at cycle starts require transitions among the clips.
  • Composite Policy: A composite policy combines independently trained skill policies and uses their value functions to determine which policy should act at runtime.This divide-and-conquer strategy avoids requiring one network to learn all skills jointly.
  • Composite Policy: A temperature parameter makes policies with larger expected values more likely to be selected by the composite policy.Repeated sampling enables sequences of skills from a diverse motion library without additional training.

8 CHARACTERS

The experiments use several articulated simulated characters with substantially different morphologies and actuators, and evaluate traversal across varied terrains.

  • Characters: The character set includes a 3D humanoid, Atlas robot, T-Rex, and dragon represented as articulated rigid bodies.Most links use 3-degree-of-freedom spherical joints, while knees and elbows use 1-degree-of-freedom revolute joints.
  • Environments: Traversal environments include mixed obstacles, dense gaps, a winding balance beam, and stairs.Figure 4 depicts the character center-of-mass trajectory across these terrain types.
  • Characters: Humanoid and Atlas morphologies and actuators differ significantly, with Atlas being almost four times the humanoid’s mass.PD gains and torque limits also differ between the characters.

9 TASKS

The framework extends motion imitation with task objectives for target striking, throwing, heading control, and terrain traversal. It also uses progressive learning to adapt policies from flat-terrain imitation to irregular environments.

  • Goal-directed control: Task objectives let policies preserve reference-motion style while pursuing goals such as striking targets or moving in a desired direction.The task goal is provided to the policy, and target heading can be manually specified at runtime.
  • Goal-directed control: Target-heading objectives penalize travel below the requested speed along the target direction but do not penalize exceeding that speed.The target direction varies during training and can be manually specified at runtime.
  • Goal-directed control: Strike policies target randomly placed spherical targets using specified character links, with a hit registered when the link center is within 0.2m.The target location and hit-memory variable are reset at the start of each cycle.
  • Goal-directed control: Throw policies release a ball attached to the character’s hand and train it to reach a target positioned 2.5–3.5m away.The policy state includes the ball’s position, rotation, and linear and angular velocity.
  • Terrain traversal: Terrain-traversal policies address obstacle-filled environments, including mixed obstacles, dense gaps, winding balance beams, and stairs.Training first imitates motions on flat terrain, then augments the networks with heightmaps and convolutional layers for irregular environments.

10 RESULTS

The framework learns physics-based policies that imitate diverse motion clips while adapting to task objectives, multiple skills, characters, dynamics, and environments. The resulting policies show natural motion, robust recovery, and successful task-directed deviations from reference motions.

  • Individual Skills: Policies learn natural locomotion, acrobatics, contact-rich motions, vaulting, and coordinated environmental interactions from motion-capture clips.The humanoid also learns backflips and spinkicks with long flight phases, crawling, rolling, and vaulting.
  • Task Objectives: 75% versus 5% throwing success and 99% versus 19% striking success show that combining imitation and task objectives substantially improves goal fulfillment.The comparisons are between policies trained with both objectives and policies trained only to imitate the reference motion.
  • Multi-Skill Integration: A multi-clip reward lets a humanoid switch among forward-walking and turning clips as the desired heading changes.The approach works best when clips represent similar motion types; more diverse clips can cause subset imitation, for which composite policies are more effective.
  • Multi-Skill Integration: Composite policies transition among skills using value functions and activate an appropriate getup policy after falls without manual scripting.The individual policies are not explicitly trained for transitions.
  • Environment Retargeting: A flat-ground landing clip is adapted to jumping from a 2m ledge, while heightmap input supports obstacle traversal, winding-path walking, and stair climbing.The Atlas stair policy still exhibits an awkward gait.
  • Physics Retargeting: Under moon gravity of 1.622m/s2, spinkick and cartwheel policies achieve returns of 0.792 and 0.688, respectively.These results test robustness to discrepancies between motion-capture dynamics and simulation dynamics.
  • Robustness: The run policy survives 720N ×0.2s forward pushes, while the spin-kick policy survives 600N ×0.2s perturbations in both directions.Perturbations are applied to the pelvis halfway through a motion cycle, and no external perturbations are used during training.

11 DISCUSSION AND LIMITATIONS

The framework combines data-driven imitation with goal-directed reinforcement learning to produce broad, robust skill repertoires. Its flexibility is demonstrated across characters, environments, tasks, and composite multi-skill policies, but several practical limitations remain.

  • Discussion: The framework combines goal-directed reinforcement learning with motion-capture or keyframed data for physics-based character animation.
  • Discussion: Policies retarget skills across characters, environments, and tasks, while composite policies execute multiple skills.
  • Limitations: A linearly advancing phase variable limits timing adaptation, multi-clip integration is untested on large libraries, and training can require several days per skill.PD-controller tuning, manually defined state similarity, and reward-term weighting also require character-specific insight or care.

SUPPLEMENTARY MATERIAL

The supplementary material describes return estimators, importance sampling, and PPO’s clipped surrogate objective for improving policy-gradient efficiency and stability. These estimators trade bias against variance, while clipping constrains policy updates relative to the previous policy.

  • Return Estimation: Monte-Carlo returns are unbiased but high variance, whereas n-step returns reduce variance at the cost of bias.
  • Return Estimation: The λ-return exponentially averages n-step returns, with λ = 1 recovering Monte-Carlo returns and intermediate λ values balancing bias and variance.
  • Advantage Estimation: GAE(λ) estimates advantage by subtracting the value function from the λ-return, and TD(λ) updates the value function using the same return.
  • Off-Policy Learning: Importance sampling reuses off-policy samples from an older policy to obtain an unbiased policy-gradient estimate and enable multiple updates from one batch.
  • Policy Optimization: PPO replaces TRPO’s hard trust-region constraint with a clipped surrogate loss that discourages likelihood ratios from moving more than ε from 1.

D LEARNING ALGORITHM

The learning algorithm trains policies with PPO using batches of simulated experience, TD(λ) value targets, and clipped policy updates. Learning curves report normalized mean returns for individual clips, multiple characters, and task-conditioned policies.

  • Learning Procedure: Each training iteration collects m = 4096 samples, then performs gradient updates using minibatches of n = 256 samples.
  • Hyperparameters: Training uses γ = 0.95, λ = 0.95, ε = 0.2, αv = 10^-2, and policy step sizes of 5 × 10^-5 for humanoid and Atlas.Dragon and T-Rex use απ = 2 × 10^-5.
  • Learning Procedure: The algorithm initializes policy and value networks, samples initial states from reference motions, simulates actions, records transitions, and updates both networks.
  • Policy Update: Policy updates use likelihood-ratio clipping, with wi(θ) computed as the current-policy action probability divided by the old-policy probability.
  • Evaluation: Learning curves measure normalized mean return over 32 episodes for humanoid skills, with three random-seed runs reported for backflip and run.Curves are also provided for Atlas, T-Rex, dragon, and policies combining imitation with task objectives.
Loading 1804.02717v3…