Source-linked AI summary
Learning Agile Robotic Locomotion Skills by Imitating Animals
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, Sergey Levine
TL;DR
Robotic systems struggle to reproduce animals’ agile locomotion without laborious controller and objective design. This paper uses reference animal motions for imitation learning, combines simulation training with sample-efficient adaptation, and demonstrates diverse gaits, hops, and turns on a quadruped robot.
Problem
Robotic locomotion controllers require time-consuming development and skill-specific expertise to reproduce diverse agile animal behaviors.
Method
The framework retargets animal reference motions, trains imitation policies in simulation, and adapts them for real-world deployment.
Results
The system learns diverse locomotion skills, including gaits, dynamic hops, and turns, with real-world policies showing adaptive advantages across skills.
Takeaways & Limitations
A common reference-motion-based approach can synthesize multiple agile quadruped behaviors without separately designing each skill’s controller.
Takeaways & Limitations
The learned behaviors are not as stable as the best manually designed controllers, and the system has not learned large jumps or runs.
Abstract
from arXiv · showhide
Reproducing the diverse and agile locomotion skills of animals has been a longstanding challenge in robotics. While manually-designed controllers have been able to emulate many complex behaviors, building such controllers involves a time-consuming and difficult development process, often requiring substantial expertise of the nuances of each skill. Reinforcement learning provides an appealing alternative for automating the manual effort involved in the development of controllers. However, designing learning objectives that elicit the desired behaviors from an agent can also require a great deal of skill-specific expertise. In this work, we present an imitation learning system that enables legged robots to learn agile locomotion skills by imitating real-world animals. We show that by leveraging reference motion data, a single learning-based approach is able to automatically synthesize controllers for a diverse repertoire behaviors for legged robots. By incorporating sample efficient domain adaptation techniques into the training process, our system is able to learn adaptive policies in simulation that can then be quickly adapted for real-world deployment. To demonstrate the effectiveness of our system, we train an 18-DoF quadruped robot to perform a variety of agile behaviors ranging from different locomotion gaits to dynamic hops and turns.
I. INTRODUCTION
Animal locomotion is more agile and diverse than current robotic systems, while manually designed and reinforcement-learning controllers require substantial expertise. The paper proposes imitation learning from animal motions to reduce skill-specific design effort and support diverse behaviors.
- Manual locomotion controllers require lengthy development and substantial expertise in both the system and desired skills.
- Reinforcement learning can automate controller development but may produce unnatural, dangerous, or infeasible behaviors in the real world.
- Reference motions provide feasible-skill priors, reducing the need to design skill-specific reward functions for each behavior.
- The framework uses sample-efficient adaptation to transfer policies trained in simulation toward real-world deployment.
- The system learns diverse dynamic locomotion skills with a Laikago quadruped, including gaits, hops, and turns.
III. OVERVIEW
The framework converts animal motion data into robot control policies through motion retargeting, motion imitation, and domain adaptation. Inverse kinematics maps source motions to the robot morphology before simulated policy training and transfer.
- The three-stage pipeline comprises motion retargeting, motion imitation, and domain adaptation.
- Animal motion data is retargeted to the robot morphology through inverse kinematics.
- The retargeted motion trains a simulated robot policy before domain adaptation facilitates real-world transfer.
- Paired source and target body keypoints, including feet and hips, define the retargeting correspondence.
- A regularization term keeps retargeted poses near a default pose using joint-specific coefficients.
V. MOTION IMITATION
Motion imitation is formulated as reinforcement learning in which a policy receives robot state and a motion goal, then learns to reproduce reference poses and motion features.
- The policy maximizes expected return by repeatedly mapping observed states and goals to actions.
- The trajectory model combines the initial-state distribution, system dynamics, and policy action probabilities.
- The policy input combines recent robot poses and actions with a goal specifying the motion to imitate.
- The policy is queried at 30Hz to generate a new action at each timestep.
- The reward tracks reference joint rotations, velocities, end-effector positions, and root pose and velocity.
VI. DOMAIN ADAPTATION
Simulation-to-real transfer is difficult because simulated and real-world dynamics differ, motivating a sample-efficient adaptation technique.
- Differences between simulated and real-world dynamics can make simulation-trained policies perform poorly on physical robots.
A. Domain Randomization
Domain randomization varies environment dynamics during training so policies can learn strategies that function across different dynamics, but robustness in simulation may not ensure physical-world success.
- Domain randomization varies dynamics during training rather than using one fixed environment.
- The strategy encourages policies to remain functional across different dynamics.
- Robustness to simulated dynamics can still fail in the real world because some effects are unmodeled.
B. Domain Adaptation
The framework randomizes dynamics, encodes them into a latent variable, and regularizes how much information the policy uses, balancing robustness with adaptability.
- Dynamics parameters are randomized in simulation and encoded into a stochastic latent embedding z for policy conditioning.
- The policy can become brittle if its strategy depends on z accurately representing the true dynamics.
- The mutual-information constraint is approximated with a variational upper bound based on the KL divergence between the encoder and prior.
- The constrained objective is converted into an information-regularized objective with β ≥0 as a Lagrange multiplier.
- The encoder is modeled as a Gaussian distribution with mean m(µ) and standard deviation Σ(µ), while the prior is unit Gaussian.
- The regularized objective trades robustness against adaptability by limiting the information the policy accesses from dynamics parameters.
C. Real World Transfer
Real-world adaptation searches for a latent encoding that maximizes physical-system return, using sampled episodes and advantage-weighted regression while avoiding premature convergence.
- Real-world adaptation directly searches for an encoding z that maximizes return on the physical system.
- AWR samples encodings, executes policy-conditioned episodes, records returns, and stores encoding-return pairs in a replay buffer.
- The search distribution starts from a unit Gaussian prior and is updated using returns from sampled real-world trials.
- The return baseline is the average of all replay-buffer samples, while α controls the update temperature.
- Testing uses a larger range of dynamics parameters than training to evaluate generalization to unfamiliar dynamics.
- Incremental gradient updates are used because the analytic Gaussian solution can prematurely converge to a suboptimal solution.
VII. EXPERIMENTAL EVALUATION
The experiments evaluate an 18-DoF Laikago quadruped on diverse skills using simulated training, real-world adaptation, and neural policies conditioned on state, goals, and dynamics encodings.
- The Laikago is an 18-DoF quadruped with three actuated degrees of freedom per leg and six under-actuated root degrees of freedom.
- The motion dataset combines dog mocap clips with artist-generated animations, retargeted to Laikago through inverse kinematics.
- Figure 4 compares the reference motion with the simulated robot and the real robot.
- Each policy uses about 200 million simulated samples for PPO training and approximately 50 real-world trials for adaptation.
- The policy network receives state, goal, and dynamics encoding as inputs and outputs the mean of a Gaussian action distribution.
- The value function receives the state, goal, and dynamics parameters.
B. Learned Skills
The framework learns diverse quadruped locomotion skills from reference motions and transfers adaptive policies across simulated and real-world dynamics. Adaptive policies generally outperform non-adaptive alternatives, especially for dynamic behaviors.
- Learned skills: The Laikago learns pacing, trotting, turning, spinning, backwards gaits, hops, and other skills from different reference motions.The system also imitates artist-generated motions, including a dynamic 90-degree Hop-Turn.
- Real-world performance: Adaptive policies outperform non-adaptive policies on most real-world skills.Performance is measured as average normalized return over 15 trials per method.
- Adaptive policy: The adaptive-policy architecture encodes dynamics parameters into a latent variable supplied to the policy alongside state and goal.Separate networks model the policy and value function.
- Domain adaptation: Adaptive policies achieve higher returns under more diverse simulated dynamics than non-adaptive policies.For Dog Pace, adaptive policies exceed return 0.6 in 50% of environments, versus 38% for the robust policy.
- Domain adaptation: Adaptive policies generally adapt to new simulated environments within relatively few episodes.Learning curves evaluate adaptation across five environments for each skill.
D. Information Bottleneck
The information bottleneck trades off robustness before adaptation against adaptability afterward. In the experiments, β = 10^-4 provides a good balance, while information-constrained policies generally outperform unconstrained ones before and after adaptation.
- Information bottleneck: Larger β values produce more robust policies with higher pre-adaptation performance but smaller post-adaptation improvements.These policies access fewer bits of information about dynamics parameters.
- Information bottleneck: Smaller β values produce less robust but more adaptive policies, with lower initial performance and larger improvements after adaptation.The trade-off arises because behavior depends less or more strongly on latent variables.
- Information bottleneck: Information-constrained policies generally outperform unconstrained No IB policies both before and after adaptation.No IB denotes training without an information bottleneck.
APPENDIX
The appendix reports training settings and performance statistics for simulation and real-world deployment. Simulation performance is similar across methods, whereas adaptive policies outperform non-adaptive policies on most real-world skills.
- Training details: PPO hyperparameters are reported for simulation training, and AWR hyperparameters are reported for real-world domain adaptation.Adam is used for gradient-descent updates.
- Performance statistics: Real-world performance is measured as average normalized return between 0 and 1 across policies initialized with different random seeds.Each method is evaluated over 15 trials per skill in the reported table description.
- Performance statistics: Adaptive policies outperform non-adaptive policies on most skills when deployed on the real robot.Robust policies suffice for simpler skills but are prone to falling on more dynamic motions.
- Performance statistics: Simulation performance is similar across the different methods under canonical dynamics parameters.The real-world comparison instead favors adaptive policies on most skills.