Source-linked AI summary
Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions
Alejandro Escontrela, Xue Bin Peng, Wenhao Yu, Tingnan Zhang, Atil Iscen, Ken Goldberg, Pieter Abbeel
TL;DR
Under-specified rewards can drive physically infeasible locomotion, while complex hand-designed rewards require laborious tuning and may not generalize. The paper learns style rewards from motion-capture data using Adversarial Motion Priors and combines them with task rewards, producing energy-efficient locomotion, natural gait transitions, and transfer to a real quadruped.
Problem
Under-specified rewards can produce aggressive, physically infeasible behaviors, while hand-designed style rewards require substantial tuning and may not generalize across platforms and tasks.
Method
Adversarial Motion Priors learn a style reward from reference motion data that is combined with an auxiliary task reward to train naturalistic policies.
Results
AMP policies track desired velocities, transfer to a real robot, achieve lower Cost of Transport than competing baselines, and produce natural gait transitions across speeds.
Takeaways & Limitations
A few seconds of German Shepherd motion capture can provide an energy-efficient motion prior while allowing policies to deviate from reference motions when task completion requires it.
Abstract
from arXiv · showhide
Training a high-dimensional simulated agent with an under-specified reward function often leads the agent to learn physically infeasible strategies that are ineffective when deployed in the real world. To mitigate these unnatural behaviors, reinforcement learning practitioners often utilize complex reward functions that encourage physically plausible behaviors. However, a tedious labor-intensive tuning process is often required to create hand-designed rewards which might not easily generalize across platforms and tasks. We propose substituting complex reward functions with "style rewards" learned from a dataset of motion capture demonstrations. A learned style reward can be combined with an arbitrary task reward to train policies that perform tasks using naturalistic strategies. These natural strategies can also facilitate transfer to the real world. We build upon Adversarial Motion Priors -- an approach from the computer graphics domain that encodes a style reward from a dataset of reference motions -- to demonstrate that an adversarial approach to training policies can produce behaviors that transfer to a real quadrupedal robot without requiring complex reward functions. We also demonstrate that an effective style reward can be learned from a few seconds of motion capture data gathered from a German Shepherd and leads to energy-efficient locomotion strategies with natural gait transitions.
I. INTRODUCTION
Under-specified rewards can produce aggressive, physically implausible locomotion that transfers poorly to real robots, while hand-designed alternatives require substantial tuning and may not generalize. The paper proposes learning style rewards from small motion-capture datasets with Adversarial Motion Priors to support natural, energy-efficient task behavior and real-robot deployment.
- Motivation: Under-specified forward-velocity rewards can induce flailing, high-impulse contacts, and simulator-exploiting behaviors that are unsafe or ineffective on real robots.These behaviors may exceed actuator limits and risk damaging the robot.
- Motivation: Hand-designed style rewards and custom action spaces achieve strong locomotion results but require substantial domain knowledge, delicate tuning, and often remain platform-specific.
- Approach: Adversarial Motion Priors learn a style reward from reference motions, allowing a simple task reward to be combined with dataset-matching behavior.The learned reward encourages the policy trajectory distribution to minimize Pearson divergence from reference trajectories.
- Contributions: The proposed framework uses small amounts of motion capture data to produce policies that can be deployed on a real robot while satisfying an auxiliary task objective.The experiments use 4.5 seconds of motion-capture data.
- Contributions: AMP policies achieve lower Cost of Transport than competing baselines and produce natural gait transitions that improve energy efficiency across speeds.The paper compares AMP with complex style rewards and no style reward.
II. RELATED WORK
Prior work develops controllers through trajectory optimization, reinforcement learning, motion tracking, and adversarial imitation. These approaches differ in specialization and flexibility: explicit tracking reproduces reference sequences, whereas adversarial methods match broader behavior distributions.
- Controller Learning: Trajectory-optimization controllers are often specialized to particular tasks, limiting generalization across diverse tasks or environments.
- Reinforcement Learning: Deep reinforcement learning can synthesize policies for specified objectives but may produce jerky, unnatural behaviors unsuitable for real-robot deployment.
- Motion Imitation: Motion-tracking methods explicitly imitate target pose sequences and have reproduced complex dynamic motor skills in simulation.
- Adversarial Imitation: Adversarial imitation learns policies that match the state or trajectory distribution of demonstrations rather than explicitly tracking individual motion clips.A discriminator distinguishes policy behavior from demonstrations and supplies the style reward.
III. METHOD
The method models legged locomotion as reinforcement learning with a velocity-tracking task reward, then regularizes the under-specified objective using a data-driven motion prior. This combination preserves controllability while encouraging more natural gait behavior across commanded speeds.
- Background and Problem Formulation: Legged locomotion is formulated as an MDP in which a policy maps states to actions and maximizes expected discounted return.The MDP includes state and action spaces, dynamics, rewards, an initial-state distribution, and a discount factor.
- Task Reward: The task reward encourages tracking desired forward, lateral, and yaw velocities expressed in the robot base frame.
- Task Reward: Velocity commands are sampled across specified ranges, giving the controller a high degree of movement controllability at different speeds.
- Motion Prior: Using only the task reward can produce violent vibrations because the objective is under-specified, so the method adds a data-driven motion prior learned through adversarial imitation.
- Motion Prior: AMP enables gait changes with commanded velocity, including pacing at lower speeds, trotting at intermediate speeds, and cantering at higher speeds.
B. Adversarial Motion Priors as Style Rewards
Adversarial Motion Priors learn a style reward from reference transitions and combine it with a user-specified task reward. Training alternates policy and discriminator optimization so behaviors remain stylistically consistent while completing tasks.
- Adversarial Motion Priors as Style Rewards: The style reward is learned from reference motion transitions, while the task reward is specified by the user.A discriminator distinguishes dataset transitions from agent-produced transitions.
- Adversarial Motion Priors as Style Rewards: The discriminator is trained with a least-squares GAN objective and a gradient penalty to distinguish reference and policy transitions and improve training stability.The objective is described as minimizing Pearson divergence, while the penalty mitigates oscillations and discriminator overshoot.
- Adversarial Motion Priors as Style Rewards: The style and task rewards are combined into a composite reward used to optimize the policy’s discounted return.The style reward is offset and scaled to lie in the range [0, 1].
- Adversarial Motion Priors as Style Rewards: During training, each policy-generated transition produces both style and task rewards before the combined signals update the policy and discriminator.The transition is evaluated by the discriminator and environment task-reward computation.
C. Motion Capture Data Preprocessing
The motion-capture dataset contains short German Shepherd locomotion clips that are retargeted to the A1 quadrupedal robot and converted into state transitions for discriminator training.
- Motion Capture Data Preprocessing: The dataset contains 4.5 seconds of German Shepherd motion spanning pacing, trotting, cantering, and turning in place.The raw data is a time series of keypoints across motion frames.
- Motion Capture Data Preprocessing: The German Shepherd motion is retargeted to the A1 robot using inverse kinematics for joint angles and forward kinematics for end-effector positions.Joint, linear, and angular velocities are computed using finite differences.
- Motion Capture Data Preprocessing: Sampled state transitions from the processed dataset serve as real discriminator samples and initialize simulated episodes from randomly selected reference states.The dataset states include joint angles, velocities, base velocities, and angular velocities.
D. Model Representation
The policy is represented by a shallow multilayer perceptron that outputs a distribution over target joint angles at 30 Hz, while PD controllers compute motor torques.
- Model Representation: The policy uses a shallow MLP with hidden dimensions [512, 256, 128] and exponential linear unit activation layers.It outputs both the mean and standard deviation of the target-joint-angle distribution.
- Model Representation: Adversarial Motion Priors allow the policy to deviate from reference motions while tracking velocity commands through sharp-turn routes.The figure describes naturalistic locomotion during careful navigation.
- Model Representation: The policy is conditioned on observations containing joint angles, joint velocities, orientation, and previous actions.The observation is derived from the robot state.
- Model Representation: At 30Hz, sampled target joint angles are fed to PD controllers that compute the motor torques.The policy’s output distribution has standard deviation initialized to σ_i = 0.25.
E. Domain Randomization
Domain randomization is applied during simulation training to support transfer of learned behavior to the real world by varying physical and control conditions.
- Domain Randomization: Simulation randomization varies terrain friction, base mass, PD controller gains, and base velocity perturbations.The perturbations are applied at random intervals during training.
- Domain Randomization: The randomized variables are sampled from uniform-distribution ranges listed in Table I.The passage identifies Table I as the source of those ranges.
F. Training
The experiments compare policies trained with different style-reward formulations for velocity tracking, energy efficiency, and real-world qualitative performance. Policies without style rewards are evaluated only in simulation because their behaviors are too violent for real deployment.
- Experimental Setup: The study compares complex hand-designed rewards, Adversarial Motion Priors, and no style reward across velocity tracking, energy efficiency, and real-robot behavior.The AMP style reward is learned from 4.5 seconds of German Shepherd motion-capture data.
- Experimental Setup: Policies trained without style rewards are analyzed solely in simulation because their exhibited behaviors are too violent for real-robot evaluation.The experiments specifically investigate whether AMP policies retain task performance while improving energy efficiency and real-world behavior.
A. Task Completion and Energy Efficiency in Simulation
In simulation, AMP policies track commanded velocities while using less mechanical energy than the compared baselines. They avoid the jittery, high-torque behavior of policies trained without style rewards and can adapt gait to changing speeds.
- Task Completion and Energy Efficiency in Simulation: Cost of Transport is used as a dimensionless measure for comparing the energy efficiency of different controllers across speeds.Mechanical COT is defined from actuator power, robot weight, and velocity.
- Task Completion and Energy Efficiency in Simulation: AMP tracks desired forward velocity commands with COT ranging from 0.93 to 1.12, below the hand-designed style reward’s 1.37 to 1.65.The no-style policy has extremely high COT because of high torques and motor velocities generated by jittery behavior.
- Task Completion and Energy Efficiency in Simulation: The no-style policy moves by violently vibrating its feet with high motor velocities and torques, producing high-impulse ground contacts that cannot be deployed on the real robot.Despite high tracking accuracy, the resulting control strategy is infeasible because of its violent motions and damage risk.
- Task Completion and Energy Efficiency in Simulation: AMP’s energy efficiency is attributed to extracting energy-efficient motion priors from the reference data and transitioning between gaits as velocity changes.The policy changes from pacing at 1 m/s to cantering at 2 m/s, with cantering offering a more energy-efficient option at high speeds.
- Task Completion and Energy Efficiency in Simulation: AMP can track commanded linear and angular velocities even when the 4.5-second German Shepherd dataset lacks motions at those particular velocities.The policy captures the essence of reference motions while deviating enough to complete the specified task.
C. Qualitative Performance of Policies in Real
Real-robot evaluation shows AMP policies using motion-prior-based locomotion and changing gait with velocity commands. The paper concludes that adversarially learned style rewards support naturalistic, energy-efficient behavior without complex hand-designed style rewards.
- Qualitative Performance of Policies in Real: On the real robot, AMP changes from a pace at 1 m/s to a canter at 2 m/s in response to the velocity command.Pacing alternates left and right swing and stance phases, whereas cantering alternates front and hind-foot placement and includes a flight phase.
- Qualitative Performance of Policies in Real: AMP style rewards encourage reference-grounded behavior, enable deviation from demonstrations for task completion, and support transfer without complex hand-designed style rewards.The paper links energy-efficient locomotion to both the dataset’s motion prior and transitions between gaits suited to commanded velocities.