Source-linked AI summary
Learning Symmetric and Low-energy Locomotion
Wenhao Yu, Greg Turk, C. Karen Liu
TL;DR
Learning natural locomotion without motion examples, contact planning, or morphology-specific guidance is difficult because learned motions can be jerky, energetic, and asymmetric. This paper combines a mirror symmetry loss with curriculum-based physical assistance, producing symmetric, low-energy, speed-appropriate gaits across multiple morphologies. The authors also identify limits in motion quality and evaluation scope.
Problem
Existing locomotion methods often rely on motion data, finite-state machines, or morphology-specific knowledge, while DRL can produce energetic and asymmetric motions.
Method
The paper combines a mirror symmetry loss in the policy objective with curriculum learning that provides and gradually relaxes physical assistance for balance and forward movement.
Results
The method produces symmetric, relatively low-energy, speed-appropriate locomotion without motion examples or contact planning and generalizes across biped, humanoid, quadruped, and hexapod morphologies.
Takeaways & Limitations
Different gait patterns emerge for different target velocities using nearly identical reward functions and without additional engineering effort.
Takeaways & Limitations
Motion quality remains below methods using real-world data, and evaluation is limited to terrestrial locomotion with articulated rigid-body characters.
Abstract
from arXiv · showhide
Learning locomotion skills is a challenging problem. To generate realistic and smooth locomotion, existing methods use motion capture, finite state machines or morphology-specific knowledge to guide the motion generation algorithms. Deep reinforcement learning (DRL) is a promising approach for the automatic creation of locomotion control. Indeed, a standard benchmark for DRL is to automatically create a running controller for a biped character from a simple reward function. Although several different DRL algorithms can successfully create a running controller, the resulting motions usually look nothing like a real runner. This paper takes a minimalist learning approach to the locomotion problem, without the use of motion examples, finite state machines, or morphology-specific knowledge. We introduce two modifications to the DRL approach that, when used together, produce locomotion behaviors that are symmetric, low-energy, and much closer to that of a real person. First, we introduce a new term to the loss function (not the reward function) that encourages symmetric actions. Second, we introduce a new curriculum learning method that provides modulated physical assistance to help the character with left/right balance and forward movement. The algorithm automatically computes appropriate assistance to the character and gradually relaxes this assistance, so that eventually the character learns to move entirely without help. Because our method does not make use of motion capture data, it can be applied to a variety of character morphologies. We demonstrate locomotion controllers for the lower half of a biped, a full humanoid, a quadruped, and a hexapod. Our results show that learned policies are able to produce symmetric, low-energy gaits. In addition, speed-appropriate gait patterns emerge without any guidance from motion examples or contact planning.
1 INTRODUCTION
The paper targets natural locomotion learned with minimal prior guidance, addressing energetic and asymmetric motions produced by existing reinforcement-learning methods. It combines a symmetry loss and curriculum learning to produce low-energy, symmetric, speed-appropriate gaits across morphologies.
- Motivation: Existing DRL agents can move without falling, but their locomotion often contains jerky, high-frequency movements.Motion examples or reward objectives can mitigate these artifacts but reduce the minimalist character of the approach.
- Motivation: The paper identifies excessive energy use and asymmetric motion as two problems in learned locomotion.These problems occur even when the agent has symmetrical morphology, contrary to biomechanical observations of statistically symmetrical gaits.
- Approach: A curriculum-learning method enables a high energy penalty by progressively assisting balance and forward movement during training.The supplied passage describes the curriculum as guidance for learning tasks with conflicting goals, including balance, propulsion, acceleration, and energy conservation.
- Approach: A mirror symmetry loss penalizes paired limbs for learning different control strategies.The loss is introduced in the policy objective rather than the reward function.
- Results: The evaluation reports symmetric, speed-appropriate, relatively low-energy locomotion without motion examples, contact planning, or morphology-specific reward terms.The method is applied across bipeds, quadrupeds, and hexapods; removing the curriculum causes failures to move or balance, while removing symmetry produces slower, asymmetric learning.
2 RELATED WORK
Prior character-control methods use finite-state structures, motion data, reward engineering, or policy search to generate physically plausible motion. The paper seeks locomotion synthesis with minimal prior knowledge about the character, target motion, and system dynamics.
- Physically-based Character Control: Finite-state-machine methods can generate plausible motions but are difficult to generalize to non-biped morphologies or arbitrary tasks.They connect pose keyframes or switch among objective terms to produce locomotion behaviors.
- Physically-based Character Control: Motion data can produce high-fidelity character motion, but its requirement limits application to arbitrary morphologies and novel control tasks.The related work passage cites videos and motion-capture data as examples of this approach.
- Physically-based Character Control: Reward engineering and trajectory optimization have been used to generate physically based character motions and interactive locomotion controllers.These methods include breaking motion into windows, optimizing trajectories, and training policies from generated data.
- Reinforcement Learning: Reinforcement-learning policy search has been applied to complex motor skills, while sampling-based and gradient-based methods trade robustness, sample requirements, variance, and scalability.CMA-ES does not require gradients but scales in sample and memory demands; gradient methods fit stochastic optimization but can have high gradient variance.
- Curriculum Learning: Curriculum learning presents training problems in increasing complexity and has been applied to navigation, games, manipulation, grasping, and motor skills.The paper positions its approach within this broader use of curricula for learning increasingly difficult behaviors.
3 BACKGROUND: POLICY LEARNING
The paper formulates locomotion as a continuous-control Markov decision process and learns a torque policy with a generic reward for velocity, balance, survival, and minimal actuation. PPO optimizes a clipped policy objective using advantage estimates and policy-ratio constraints.
- MDP Formulation: Locomotion is modeled as an MDP with state and action spaces, reward, initial-state distribution, transitions, and discount factor.The policy parameters are optimized to maximize expected long-term reward.
- MDP Formulation: The locomotion state contains joint positions and velocities, end-effector contacts, and the target forward velocity; actions are actuator-generated joint torques.The target velocity is included to condition the policy on the desired center-of-mass motion.
- Reward Design: The generic reward combines desired velocity, upright orientation, forward alignment, alive status, and minimal actuation.Velocity error uses the average velocity over the most recent 2 seconds, while excessive torques are penalized as energy use.
- Policy Optimization: PPO is used to learn the locomotion policy because the paper describes it as providing data-efficiency and learning-performance advantages over alternative algorithms.The method is also described as applicable to TRPO and DDPG.
- Policy Optimization: PPO estimates action advantage as Qπ(s,a) − Vπ(s) and uses a clipped policy-ratio objective to limit changes from the previous policy.The importance-sampling ratio allows data from the old policy to estimate the current-policy expectation.
4 LOCOMOTION CURRICULUM LEARNING
The paper uses physical assistance as a curriculum for learning locomotion’s competing demands, then gradually removes that assistance through adaptive or predefined schedules. The assistant supports forward movement and lateral balance while policies learn sagittal balance and low energy.
- Locomotion must reconcile competing goals, including maintaining balance while moving forward and conserving energy while accelerating.
- The virtual assistant applies pelvis-based forces for lateral balance and forward propulsion, allowing the learner to focus on sagittal balance and energy consumption.A proportional-derivative controller regulates lateral position and forward velocity using curriculum parameters x = (kp,kd).
- The assistant is designed for morphology-independent use without pre-training, with smooth state-space assistance and unconditional stability at large controller gains.
- Curriculum scheduling gradually reduces assistance from an initial lesson x0 to the origin, balancing the risks of overly aggressive and overly conservative schedules.
- Learner-centered curriculum: Learner-centered curriculum adapts the next lesson to current skill, using five directional line searches to select the largest step preserving 60% of the original return.The next lesson is chosen closer to the origin while retaining policy proficiency.
- Environment-centered curriculum: Environment-centered curriculum follows predefined lessons while exposing the learner to ranges of assistance, including overlapping lessons that help bootstrap new skills.The lesson range is updated after policy evaluation when both performance conditions are met.
5 MIRROR SYMMETRY LOSS
The paper encourages gait symmetry by comparing actions in mirrored states rather than using a delayed full-cycle symmetry reward. This symmetry loss is optimized alongside the reinforcement-learning objective.
- Full-cycle gait symmetry measurements create delayed rewards that make policy learning harder.
- Action symmetry avoids the potential issue of delayed reward because it can be measured without waiting for an entire trajectory.
- The method enforces symmetry on actions: an action in one pose should mirror the action taken in the corresponding mirrored pose.Mirror mappings are defined for both actions and states, while the policy represents the mean action.
- The differentiable symmetry loss can be combined with the standard reinforcement-learning objective and optimized using gradient-based reinforcement-learning algorithms.The implementation uses 20,000 simulation samples per iteration.
- The mirror-symmetry loss is included in the policy objective rather than the MDP reward because it depends explicitly on policy parameters.Its gradient is calculated separately from the policy-gradient objective and then combined for optimization.
6 RESULTS
The method learns symmetric locomotion across biped, humanoid, quadruped, and hexapod morphologies, with gait patterns adapting to target speed. Environment-centered curriculum learning improves training efficiency, while mirror symmetry loss supports faster and more symmetric policies.
- Locomotion of Different Morphologies: Four characters with different morphologies and degrees of freedom are evaluated using the same general locomotion methodology.The experiments include simplified biped, humanoid, quadruped, and hexapod models.
- Locomotion of Different Morphologies: At 1m/s the simplified biped walks, while at 5m/s it runs with an emergent flight phase.The gait transition arises from changing the target velocity.
- Locomotion of Different Morphologies: Quadrupeds consistently trot at 2m/s and learn either trotting or galloping at 7m/s, depending on the policy’s initial random seed.The model has 13 links and 22 degrees of freedom.
- Curriculum Learning Comparison: Environment-centered curriculum learning produces a successful simplified-biped policy with about half the data required by learner-centered curriculum learning.Both curriculum methods train symmetric walking and running at target velocities.
- Comparison with Baseline Methods: Without curriculum learning policies typically fall slowly or stand still, whereas removing mirror symmetry loss yields slower training and asymmetric gaits.A high-torque vanilla PPO baseline can walk but appears jerky and uses significantly more torque.
7 DISCUSSION
The discussion examines how curriculum design, symmetry, and actuator limits shape learned locomotion, while showing that the approach can be refined and adapted across gait styles.
- Policy refinement: Warm-starting from learned policies enabled high-knee biped walking and large-stride humanoid running, with refinement taking only 200 iterations.The starting policies already balanced and moved forward before refinement learning began.
- Gait emergence: Different target velocities produced speed-appropriate gaits using nearly identical reward functions and no additional engineering effort.The paper contrasts this with prior approaches that obtained gait differences through contact planning or constraint enforcement.
- Sensitivity: Resulting motion is particularly sensitive to the torque range allowed for each actuator, which the authors set heuristically based on perceived joint strength.The paper notes that a comprehensive sensitivity analysis is beyond its scope.
- Sensitivity: Greatly increasing humanoid leg strength produced hopping behavior, showing that actuator settings can change the learned locomotion gait.This observation accompanies the reported sensitivity to torque-range settings.
- Assistance design: Wall-based balance assistance encouraged the character to lean on the wall, while a virtual treadmill failed to provide sufficient forward-movement assistance.As the wall gap widened, the character leaned more until falling; treadmill learning remained unsuccessful even in the initial lesson.
8 CONCLUSION
The paper combines virtual physical assistance with an additional symmetry loss to learn low-energy, symmetric, speed-appropriate locomotion across varied articulated body plans.
- Method: The approach combines virtual assistance for balance and target-speed reaching with an additional loss term encouraging symmetric behavior.These two techniques are presented as the central components of the locomotion-controller method.
- Results: Controllers learned natural-looking locomotion gaits for bipeds, humanoids, quadrupeds, and hexapods without prior motion knowledge.The tested body plans include the lower half of a biped, a full humanoid, a quadruped, and a hexapod.
- Limitations: The method remains below the motion quality of computer-animation approaches that exploit real-world data.The authors identify motion-capture integration, biological modeling, and policy refinement as areas needing further investigation.
- Limitations: Evaluation covered terrestrial locomotion with articulated rigid-body characters, leaving swimming, flying, and soft-body locomotion as future directions.This defines the reported scope boundary of the experiments.
Learning Symmetric and Low-Energy Locomotion • 144:11
The cited related work spans physics-based animation, reinforcement learning, curriculum learning, trajectory optimization, and motion-data-driven control for character locomotion.
- Curriculum learning: Curriculum learning has been applied to robot manipulation, navigation, grasping, and motor-skill training.The cited examples use increasingly difficult initial states, improved learning efficiency, or decomposed subtasks.
- Locomotion systems: Other cited studies address data-driven biped control, many-muscle humanoids, terrain-adaptive locomotion, hierarchical locomotion, and animal gait optimization.The references cover both motion-data-based and optimization- or learning-based locomotion systems.
- Reinforcement learning: The related work includes deep reinforcement-learning methods for continuous control, policy gradients, generalized advantage estimation, and proximal policy optimization.These references establish the reinforcement-learning foundations surrounding locomotion control.