Fast and Efficient Locomotion via Learned Gait Transitions

1 Introduction

• Gait timing on quadruped robots is often hand-engineered, while transitions between gaits rely on ad-hoc user commands.The paper asks whether robots can learn energy-efficient gaits and natural transitions automatically.Source: §1 · p. 1
• The framework separates gait generation from motor control: an ES-trained high-level policy specifies gait parameters, and a low-level convex MPC controller computes motor commands.The policy is trained as an MDP with a reward based on velocity tracking and energy efficiency.Source: §1 · p. 1
• The learned policy produces slow walking, mid-speed trotting, and fast fly-trotting, and deploys on a Unitree A1 without additional data collection or fine-tuning.The paper reports deployment in environments including carpet, grass, and a short obstacle.Source: §1 · p. 2
• The contributions include automatic gait switching across locomotion speeds and robust real-world deployment in varied environments.The paper presents the hierarchical learning framework as combining reinforcement learning with optimal control.Source: §1 · p. 2

2 Related Work

• Model-based controllers commonly assume predefined contact sequences, while contact-implicit optimization jointly handles contact forces and sequences but is too computationally costly for real-time use.Heuristic transition methods were also reported, including approaches limited to 0.6m/s or 1 body length/s.Source: §2 · p. 2
• Prior reinforcement-learning locomotion policies often require extra sim-to-real work, whereas this paper reports zero-shot transfer and robustness across real-world environments.Examples of transfer efforts include more accurate simulation, dynamic randomization, motion imitation, and meta learning.Source: §2 · p. 2
• Unlike hierarchical gait control that selects from fixed primitives, this approach searches a continuous range of gaits with arbitrary gait changes and reports faster walking.The authors describe their setup as similar to Da et al. [35] but with a more flexible high-level policy.Source: §2 · p. 2

3.1 Overview

• A high-level gait generator feeds leg phases to a low-level convex MPC controller, with a state estimator supplying torso velocity to both.The gait generator runs at 20Hz and the low-level controllers at 500Hz.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.2 High-Level Gait Generation

• Each leg’s phase tracks progress through a gait cycle, switching from swing to stance at a threshold; the policy sets frequency, swing ratio, and phase offsets to determine contact schedules.The phase advances by 2πfΔt, and the swing threshold is φswing = 2πpswing.Source: §3.2 · pp. 3–4

3.3 Low-Level Convex MPC Control

• In stance, convex MPC optimizes ground reaction forces to track a reference trajectory under torque and friction constraints, then converts forces to torques.The controller re-optimizes every 2ms and applies only the first command in its optimized sequence.Source: §3.3 · p. 4
• In swing, a quadratic foot trajectory connects lift-off, mid-air, and landing positions; inverse kinematics and PD control produce motor torques to track it.The landing position uses Raibert Heuristics, and the mid-air position ensures minimum ground clearance.Source: §3.3 · p. 4

4 Learning Gait Policies for Fast and Efficient Locomotion

• Evolutionary Strategies discover energy-efficient gaits at different speeds, addressing the difficulty of optimizing discrete gait contacts alongside continuous forces.The framework formulates learning as a Markov Decision Process.Source: §4 · p. 4

4.1 Preliminaries

• An MDP policy maps states to actions to maximize expected cumulative reward over an episode of length T.The MDP includes states, actions, transition probabilities, a reward function, and an initial-state distribution.Source: §4.1 · pp. 4–5

4.2 MDP Formulation

• The MDP learns only the high-level gait policy, treating the phase integrator, convex MPC controller, and robot dynamics as the environment.The policy outputs gait parameters and receives reward based on energy consumption and speed tracking.Source: §4.2 · p. 5
• The state contains desired and actual base linear velocity, while the five-dimensional action specifies gait frequency, swing phase, and three phase offsets.The low-level controller handles balance and locomotion using proprioceptive information omitted from the state.Source: §4.2 · p. 5
• The reward combines survival bonus, speed-tracking penalty, and Cost of Transport energy penalty, with fixed weights c = 3, wv = 1, and we = 0.37.The energy estimate uses power across all 12 motors and divides by robot mass and gravity; commanded speed rises to 2.5 m/s at 1 m/s².Source: §4.2 · p. 5
• Episodes terminate early when the robot falls, avoiding further exploration of infeasible gaits.Falling is defined by substantial orientation deviation from upright or robot height becoming too low.Source: §4.2 · p. 5

4.3 Policy Representation and Training

• CMA-ES trains a compact, one-hidden-layer policy network with 256 tanh units, exploring policy parameters without requiring accurate value estimates at every step.The authors choose this architecture for expressiveness and efficient optimization in the hierarchical system.Source: §4.3 · p. 5

5.1 Experiment Setup

• Experiments used a 15-kg Unitree A1 in PyBullet, with the gait generator running at 20 Hz and convex MPC at 500 Hz on an M1 Mac Mini.Source: §5.1 · p. 6

5.2 Emergence of Energy-Efficient Gaits

• At most tested speeds, the learned policy achieves the lowest CoT among the four hand-tuned gaits.The hand-tuned baselines are Walk, Slow Trot, Rapid Trot, and Fly Trot; the learned result is averaged over five random seeds.Source: §5.2 · p. 6, Figure 4 · p. 6
• The policy walks below 0.9 m/s, trots from 0.9 to 1.8 m/s, and fly-trots at 1.8 m/s or above, with airborne phases.The fly-trot has phases with all legs off the ground, characterized by pswing > 0.5.Source: §5.2 · pp. 6–7, Figure 3 · p. 6
• On this robot, the reproduced galloping gait was 30% less efficient than flying trot, unlike animals; morphology, kinematics limits, and joint actuation were proposed explanations.Source: §5.2 · p. 7

5.3 Validation on the Real Robot

• The simulation-trained policy transferred to the real robot without additional data collection or fine-tuning, switching gaits as it accelerated to 2.5 m/s.The robot reached 5 body lengths per second.Source: §5.3 · p. 7, Figure 5 · p. 7
• In unseen scenarios, the robot remained stable through abrupt acceleration and braking, crossed an 8-cm step, and walked on grass.The authors attribute this generalization to the hierarchical setup and robust low-level convex MPC controller.Source: §5.3 · p. 7, Figure 6 · p. 7

5.4 Comparisons with Non-Hierarchical Policies

• E2E learned only to stand, while PMTG used three legs and tracked desired speed less well than the hierarchical framework.The authors suggest flat-policy optimization can converge to bad local optima; reward shaping might improve the baselines.Source: §5.4 · pp. 7–8, Figure 7 · p. 8

5.5 Comparisons with Different Learning Algorithms

• CMA-ES and ARS significantly outperformed other tested algorithms; PPO and SAC failed with the original observation space, while extended-observation SAC walked but tracked speed poorly and used more energy.The proposed explanation is that the low-level convex MPC is a black box to the RL agent, making the environment less Markovian.Source: §5.5 · p. 8, Table 2 · p. 8

6 Conclusion

• Distinctive gaits and natural transitions emerge automatically, and the simulation-trained policy deploys successfully on a real robot.The framework combines a high-level gait generator with low-level convex MPC and trains the gait policy using evolutionary strategies and a simple reward.Source: §6 · p. 8
• Robotic observations agree with prior biomechanical studies showing that quadrupeds switch gaits at different speeds to lower energy expenditure.Source: §6 · p. 8
• The framework could also modulate foot placement and desired base pose; the authors plan bi-level optimization and applications to other platforms.Source: §6 · p. 8

A Details of the Low-Level Convex MPC Controller

• The low-level controller uses separate controllers for stance and swing legs.Source: Figure 8 · p. 12

A.1 Stance Leg Controller

• Stance MPC optimizes ground-reaction forces to track a desired base trajectory using a simplified centroidal dynamics model.The desired trajectory sets linear velocity to the commanded speed and orientation and angular velocity to zero; optimized forces are converted to motor torques.Source: §A.1 · pp. 12–13
• The stance model assumes massless legs and a rigid base, and simplifies dynamics for small angular velocity and near-upright posture.The resulting linear, time-varying dynamics are discretized for the MPC formulation.Source: §A.1 · pp. 12–13

A.2 Swing Leg Control

• Swing control fits a quadratic foot trajectory through lift-off, mid-air, and landing points, then tracks it with a PD controller.The Raibert heuristic estimates the landing point from projected center-of-mass velocity and expected stance duration.Source: §A.2 · pp. 13–14

B.1 DC Motor Model

• The DC motor model expresses battery power as mechanical power plus circuit heat loss, with consumption lower-bounded at zero.The zero lower bound reflects the battery management system's lack of regenerative braking.Source: §B.1 · p. 14

B.2 Power Consumption of A1 motors

• At 4 Nm, A1 motor output power is approximately 400 W with approximately 50% efficiency, implying rk^2 is approximately 25.Source: §B.2 · p. 14, Figure 10 · p. 15
• The 9.1 gear-reduction ratio is used to convert motor torque and velocity into joint torque and velocity for the power expression.Source: §B.2 · pp. 14–15
• The characteristic curve reports angular velocity and output torque at motor level, before gear reduction.Source: Figure 10 · p. 15

C.1 Comparison with Different Learning Algorithms

• CMA-ES and ARS use one-hidden-layer, 256-unit tanh policies, while PPO and SAC use two-hidden-layer, 64-unit tanh actor and critic networks.Source: §C.1 · p. 15
• CMA-ES and ARS each reached 1.5 million environment steps in 2 hours, versus 12 hours for PPO and 40 hours for SAC.ES-based methods update less frequently and require no back-propagation for policy updates, explaining their wall-clock efficiency compared with PPO.Source: §C.1 · p. 16
• CMA-ES and ARS consistently outperformed other algorithms; PPO and SAC with original observations failed to complete the task, while extended-observation SAC walked forward but had lower return.Source: §C.1 · p. 16

• Reducing TG’s energy penalty weight from 0.37 to 0.037 keeps rewards positive when its cost of transport is higher, avoiding early episode termination as a return-maximizing strategy.Source: §C.2 · p. 16
• E2E reward shaping adds height and orientation terms to encourage stable, upright walking without TG’s cyclic trajectory priors.The paper says E2E otherwise struggles to balance and often walks forward in an unstable, tilted pose.Source: §C.2 · p. 16
• E2E retains the speed and energy penalties, while adding IMU-based orientation and desired-height penalties for the robot’s center of mass.The desired height is 0.26; the E2E orientation and height weights are 10 and 200, respectively.Source: §C.2 · pp. 16–17
• Figure 12 reports non-hierarchical-policy learning curves averaged over five random seeds, with error bars showing one standard deviation.Source: Figure 12 · p. 17
• Both non-hierarchical policies use 256-unit tanh neural networks trained with CMA-ES under the hierarchical policy’s setup.Source: §C.2 · p. 17