Source-linked AI summary
Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots
Zipeng Fu, Ashish Kumar, Jitendra Malik, Deepak Pathak
TL;DR
Fixed gait libraries do not capture the contrast between regular locomotion on ideal terrain and irregular locomotion in natural settings. The paper learns quadruped movement by minimizing energy, finding natural gait patterns across speeds and structured or unstructured terrain conditions. These results support energy minimization as a principle underlying emergent locomotion patterns in robots.
Problem
Pre-programmed gaits limit general locomotion, while animal locomotion is regular on ideal flat terrain but often irregular in diverse natural settings.
Method
The paper uses analysis-by-synthesis with model-free reinforcement learning, an energy-based reward, and uneven terrain training to learn locomotion without predefined motion heuristics.
Results
Energy minimization produces walk, trot, and gallop-like bouncing at increasing speeds on flat terrain, and unstructured gaits on uneven complex terrain in a real quadruped.
Takeaways & Limitations
Energy minimization can support the emergence of natural-looking, speed-dependent locomotion patterns across flat and complex terrains without pre-programmed gaits.
Takeaways & Limitations
The training framework relies on natural constraints including an energy-motivated reward and uneven terrain policies as sufficient ingredients for the demonstrated results.
Abstract
from arXiv · showhide
Legged locomotion is commonly studied and expressed as a discrete set of gait patterns, like walk, trot, gallop, which are usually treated as given and pre-programmed in legged robots for efficient locomotion at different speeds. However, fixing a set of pre-programmed gaits limits the generality of locomotion. Recent animal motor studies show that these conventional gaits are only prevalent in ideal flat terrain conditions while real-world locomotion is unstructured and more like bouts of intermittent steps. What principles could lead to both structured and unstructured patterns across mammals and how to synthesize them in robots? In this work, we take an analysis-by-synthesis approach and learn to move by minimizing mechanical energy. We demonstrate that learning to minimize energy consumption plays a key role in the emergence of natural locomotion gaits at different speeds in real quadruped robots. The emergent gaits are structured in ideal terrains and look similar to that of horses and sheep. The same approach leads to unstructured gaits in rough terrains which is consistent with the findings in animal motor control. We validate our hypothesis in both simulation and real hardware across natural terrains. Videos at https://energy-locomotion.github.io
1 Introduction
Pre-programmed gaits work in ideal conditions but limit general locomotion, which is irregular on diverse terrain. The paper proposes energy minimization as a principle that can produce both structured and unstructured quadruped locomotion without predefined motion heuristics.
- Motivation: Pre-programmed gait patterns limit legged systems across diverse terrains and speeds, whereas natural locomotion can be irregular or organized into regular gaits under ideal conditions.Animal studies describe locomotion as bouts of steps in natural settings but identify consistent walk, trot, canter, and gallop patterns on flat terrain.
- Motivation: Energy minimization offers a way to explain gait transitions with speed, but prior biomechanics studies mainly considered ideal flat-terrain motion and treated gaits as given.The paper extends this perspective beyond straight-line locomotion in ideal terrain.
- Approach: The paper uses analysis-by-synthesis to show that minimizing energy can produce structured gaits on flat terrain and unstructured gaits on complex terrain.This approach avoids relying on demonstrations or predefined motion heuristics.
- Results: On a real quadruped, energy-based training produces walk, trot, and gallop-like bouncing as speed increases, with gait selection linked to energetic efficiency.Trotting is reported as energy-efficient around 0.9m/s, while walking or bouncing becomes more efficient outside that speed range.
- Results: The same framework produces unstructured gaits on uneven complex terrain, consistent with animal locomotion in diverse conditions.The authors also report successful deployment of the complex-terrain policy in the real world.
- Contributions: The contributions include smooth velocity-conditioned gait transitions and validation of emergent behaviors, robustness, and gait patterns in simulation and on a real quadruped.The paper presents a distillation-based pipeline for changing target speeds.
2 Method
The method learns locomotion policies with model-free reinforcement learning, using rewards for target-speed tracking, energy minimization, and survival. It combines energy-based training with terrain variation and adaptation for simulation-to-real deployment.
- Learning Locomotion Policy: The policy maps the current state and previous action to the next action, with predicted joint positions converted to torque through a PD controller.The policy receives a 30-dimensional state and 12-dimensional previous action, then outputs desired positions for 12 joints.
- Learning Locomotion Policy: RL maximizes an expected trajectory return generated while executing the policy.The trajectory distribution depends on the policy being learned.
- Energy Consumption-Based Reward: Training uses bio-energetic work minimization and uneven terrain as natural constraints instead of artificial simulation noise or additional robustness rewards.The authors report that these constraints are sufficient to achieve the demonstrated gaits and results.
- Energy Consumption-Based Reward: The reward combines straight-line motion at the specified speed, an energy-consumption penalty, and a survival bonus.The method estimates energy per timestep by summing torque–joint-velocity power across the 12 motors.
- Simulation-to-Real Transfer: RMA adapts the learned policy during deployment by estimating environmental parameters online from state history.The adaptation module is trained in simulation after the base policy is learned with true extrinsic parameters.
3 Experimental Setup and Training Details
Experiments use a Unitree A1 quadruped and its simulated model, with policies trained under normal or aggressive environmental perturbations. The setup evaluates regular and unstructured gait emergence across terrain conditions.
- Hardware and Simulation: The hardware platform is Unitree’s A1 robot, simulated with its A1 URDF in RaiSim.Terrain and environment setup details are provided separately by the paper.
- State-Action Space: The policy uses a 30-dimensional state and a 12-dimensional target-joint-position action space controlled through a PD controller.The state includes joint positions, joint velocities, torso roll and pitch, and binary foot-contact indicators.
- Environmental Variations: Normal environmental perturbations support regular gait emergence, whereas aggressive perturbations with sharper fractal terrain support unstructured walking.The perturbation ranges are summarized in Table 1.
- Policy Learning: The policy is a three-layer multilayer perceptron trained with PPO from the current state and previous action to output 12 target joint angles.Training runs for 15,000 iterations and 1.5 billion samples.
4 Results and Analysis
Energy-minimizing policies produce speed-dependent gaits, remain robust under disturbances and payloads, and adapt from structured patterns on flat terrain to unstructured stepping on complex terrain.
- Complex terrain: On rocky terrain and unstable planks, terrain interactions interrupt periodic contacts, producing unstructured walking while the robot maintains stability and crosses successfully.The front-right foot destabilizes a plank, yet the robot remains stable; contact patterns vary with terrain complexity.
- Energy and robustness: The method is more energy efficient than convex MPC for walking, trotting, and bouncing across the evaluated target speeds.Table 2 compares actual speed and energy consumed per meter, while the walking policy is reported as 50% more energy efficient than MPC.
- Emergent gaits: At 0.375, 0.9, and 1.5 m/s target speeds, walking, trotting, and bouncing emerge and transfer from simulation to hardware without fine-tuning.The same foot-contact sequences are observed in simulation and the real world.
- Emergent gaits: 0.396 m/s, 0.914 m/s, and 1.714 m/s are the average real-world speeds for 0.375, 0.9, and 1.5 m/s targets, respectively.The reported averages closely match the target speeds across three trials per target.
- Energy and robustness: A 1 kg payload outside the training perturbation range leaves the demonstrated gait patterns robust through sim-to-real adaptation.Bouncing was not tested with the payload because it hindered rapid operator intervention at high speed.
- Ablation studies: Removing energy minimization produces high-frequency tapping, while removing fractal terrain produces unnatural and unstable gaits.Energy minimization also reduces artificial penalties as a byproduct, and fractal terrain improves foot clearance and deployment robustness.
5 Related Works
Model-based gait controllers commonly optimize stance-leg behavior around predefined contacts and swing motions, whereas this work studies gaits emerging from energy minimization across speeds.
- Model-based optimization: Model-based controllers optimize stance legs for given foot-contact sequences and often use predefined swing-leg motions for stable real-robot gait generation.Contact-implicit optimization additionally optimizes contact forces and sequences.
6 Conclusion
The work uses energy minimization in real quadruped robots to generate both regular laboratory gaits and unstructured complex-terrain gaits, with robustness and smooth transitions across speeds.
- Conclusion: Energy consumption plays a key role in the emergence of natural locomotion patterns in real quadruped robots.The analysis-by-synthesis approach produces straight-line laboratory gaits and unstructured complex-terrain gaits.
- Conclusion: The robot’s gaits are most similar to those of horses and sheep in Froude-number metric space.The conclusion links this similarity to the animals’ proximity to the robot under the Froude-number metric.
A Experimental Setup and Training Details
The experiments use a Unitree A1 quadruped and RaiSim simulation, with randomized environmental factors, energy-based rewards, and hardware-oriented sensing and control details.
- Hardware: The Unitree A1 platform has 18 degrees of freedom, 12 actuated joints, and an approximately 12 kg mass.Joint states come from motor encoders, torso roll and pitch from an IMU, and foot contacts from pressure sensors.
- Simulation: RaiSim uses the A1 URDF with fractal terrains for structured gait emergence on flat terrain and unstructured gait emergence on uneven terrain.The setup specifies distinct fractal frequencies and amplitudes for flat and uneven terrain conditions.
- Environment randomization: The environmental vector contains 19 dimensions covering payload, motor strength, friction, and linear and yaw speeds.The environmental information is used by the adaptation system for simulation-to-real transfer.
- State and action spaces: The state has 30 dimensions, while the action outputs 12 target joint positions with bounded delta commands.The state includes joint positions and velocities, torso roll and pitch, and binarized foot-contact indicators.
- Reward: Energy consumption is the primary reward, with optional extra penalties on uneven terrain to reduce hardware-wear risks.Those additional penalties are not responsible for unstructured gait emergence.
- Training: PPO and Adam train the base policy and environmental-factor encoder for 15,000 epochs using batches of 100,000 state-action transitions.Each batch is split into four mini-batches, each reused four times for optimization.
- Robustness: Walking remains robust under a 1 kg payload outside the normal simulation perturbation range.Supplementary results show the walking gait under the added payload.
B.1 Emergence of Walking, Trotting and Bouncing
As target speed increases, the robot exhibits walking, trotting, and bouncing or galloping gaits, with a transition from galloping to bouncing at high speed.
- 0.375 m/s produces walking, 0.9 m/s produces trotting, and 1.5 m/s produces bouncing or galloping.
- At high speed, galloping begins with the front legs leaving and contacting the ground first.
- After several seconds at high speed, the robot switches to bouncing, with all four legs contacting the ground approximately together.
B.2 Robustness Tests
The walking gait remains robust under a 1 kg payload outside the normal range of simulated training perturbations.
- The walking gait remains robust with a 1 kg payload outside the normal range of environment perturbations used during simulation training.
C Additional Ablation Studies
Terrain variation is important for producing stable, deployable gaits: flat-ground training without fractal perturbations yields unstable patterns, while fractal terrain improves foot clearance and robustness.
- Figure 11 presents two tilted, unstable failure gaits produced by flat-ground training without fractal variations.
- All 10 trials trained on flat terrain without fractal perturbations converge to unnatural and unstable gaits.
- Adding fractal terrain facilitates larger foot clearance and robustness, improving real-world hardware deployment.
- The convex MPC baseline uses separately fine-tuned gait-generation parameters for walking, trotting, and bouncing.
- The velocity-conditioned policy is bootstrapped from demonstrations collected from fixed-velocity walking, trotting, and bouncing experts.