Source-linked AI summary
Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition
Jonah Siekmann, Yesh Godse, Alan Fern, Jonathan Hurst
TL;DR
The paper addresses how to specify rewards that reliably produce diverse bipedal gaits without constraining policies to reference trajectories. It composes probabilistic periodic costs on forces and velocities, then demonstrates individual gait transfer and continuous multi-gait transitions on Cassie.
Problem
Reward design for real-robot learning of diverse bipedal gaits must be specific enough to produce characteristic behavior without overly constraining adaptation, while reference-free and trajectory-based approaches have important shortcomings.
Method
The framework composes reward components over probabilistic periodic intervals, using gait phases and physical measurements such as foot forces and velocities.
Results
The framework transfers walking, running, galloping, skipping, and hopping from simulation to Cassie and supports a policy that continuously transitions among two-beat gaits.
Takeaways & Limitations
Reference-free periodic reward composition enables exploration beyond arbitrary reference trajectories while retaining sufficient gait structure for individual behaviors and continuous multi-gait control.
Abstract
from arXiv · showhide
We study the problem of realizing the full spectrum of bipedal locomotion on a real robot with sim-to-real reinforcement learning (RL). A key challenge of learning legged locomotion is describing different gaits, via reward functions, in a way that is intuitive for the designer and specific enough to reliably learn the gait across different initial random seeds or hyperparameters. A common approach is to use reference motions (e.g. trajectories of joint positions) to guide learning. However, finding high-quality reference motions can be difficult and the trajectories themselves narrowly constrain the space of learned motion. At the other extreme, reference-free reward functions are often underspecified (e.g. move forward) leading to massive variance in policy behavior, or are the product of significant reward-shaping via trial-and-error, making them exclusive to specific gaits. In this work, we propose a reward-specification framework based on composing simple probabilistic periodic costs on basic forces and velocities. We instantiate this framework to define a parametric reward function with intuitive settings for all common bipedal gaits - standing, walking, hopping, running, and skipping. Using this function we demonstrate successful sim-to-real transfer of the learned gaits to the bipedal robot Cassie, as well as a generic policy that can transition between all of the two-beat gaits.
I. INTRODUCTION
The paper frames bipedal gait learning as a reward-design problem: rewards must specify periodic gait characteristics without overconstraining adaptation. It proposes probabilistic periodic reward composition and demonstrates sim-to-real learning of individual and multi-gait behaviors on Cassie.
- Learning all common bipedal gaits on a real robot remains an unsolved reinforcement-learning problem.
- Reference trajectories specify gait behavior but capture limited variation and can be difficult to derive for a desired gait.
- Reference-free rewards are often underspecified, requiring tedious and unreliable heuristic adjustments or remaining specific to one behavior.
- The framework composes periodic rewards on foot forces and velocities, penalizing force while allowing velocity during swing to encourage foot lifting.
- Policies trained with this framework learn walking, running, galloping, skipping, and hopping on hardware without motion-capture data or reference trajectories.
II. BACKGROUND
Prior locomotion-synthesis work spans kinematic, deep-learning, physics-based, and reinforcement-learning approaches. Reinforcement-learning methods commonly either match reference trajectories or optimize objectives without reference trajectories, with trade-offs in behavioral realism and flexibility.
- Character-animation methods include kinematics-based synthesis using motion capture and newer deep-learning methods for adaptation and action blending.
- Physics-based character animation controls character poses through physical actuation rather than purely kinematic synthesis.
- Many reinforcement-learning locomotion methods use trajectory-matching rewards to imitate reference motions under simulated physics.
- Reference-free reinforcement-learning approaches synthesize locomotion without trajectories but may emphasize objective maximization over subjective behavioral quality.
A. Reinforcement Learning Framework
The paper formulates locomotion learning as reinforcement learning in a continuous-state, continuous-action Markov Decision Process. Training occurs through trial-and-error in simulation, after which the learned policy is used in the real world.
- The environment is modeled as a discrete-time Markov Decision Process with continuous states, continuous actions, transition dynamics, and reward function R(s,t).
- A control policy maps states to actions and the reinforcement-learning objective is to maximize expected T-horizon discounted return.
- The policy is learned from trial-and-error training experience, following the sim-to-real paradigm in which training is performed in simulation before real-world use.
B. Periodic Reward Composition
The framework represents gait rewards as periodic compositions whose components activate phase-specific measurements with probabilistic timing. Expected rewards replace sampled stochastic rewards without changing any policy’s expected cumulative return.
- The periodic reward R(s,φ) uses normalized cycle time φ, making the non-stationary reward periodic rather than tied to absolute time.
- Each reward component combines a phase coefficient, binary phase indicator, and real-valued measurement such as foot-force norm.
- The coefficient sign determines whether an active phase measurement contributes positively or negatively, while phase indicators encode uncertain start and end times with Von Mises distributions.
- Uncertainty in phase boundaries smooths reward transitions and was found to encourage more stable and consistent learning.
- Figure 3 visualizes swing and stance intervals on a polar plot alongside the expected summed force phase coefficient Cfrc(φ).
- For any policy π, expected cumulative return J(π) is unchanged when stochastic rewards are replaced by their expectations.
C. Describing Bipedal Gaits
Bipedal gait rewards are formed by applying periodic swing and stance costs to both feet with relative cycle offsets. Two phase offsets describe walking, galloping, and hopping, while four phases extend the framework to skipping.
- Single-foot swing penalizes foot force while allowing velocity, whereas stance penalizes foot velocity while ignoring force.
- The framework summarizes swing and stance behavior through expected sums of phase indicators and coefficients for foot force and speed.
- Bipedal gaits use the same phase sequence for both feet, shifted by cycle offsets θleft and θright.
- Walking offsets the feet by half a period, while hopping synchronizes their swing and stance phases.
- Two phase offsets define walking, galloping, and hopping, while four phases define skipping.
IV. METHOD
The method conditions recurrent locomotion policies on gait timing, velocity commands, and estimated robot state while training with auxiliary costs and dynamics randomization. This setup supports single-gait learning and sim-to-real-oriented policy training.
- Network Architecture and Action Space: Policies use an LSTM with two recurrent hidden layers of 128 units and a 30-dimensional output for joint positions and PD gains.
- State Space: The policy state includes cycle time, left and right cycle offsets, and phase timing information to represent the periodic reward.
- State Space: Phase timing is encoded through ratios representing each phase’s share of the total period; for example, times 0.3 to 0.7 yield a ratio of 0.4.
- State Space: Policy inputs include desired velocity, phase ratios, clock inputs, and estimated pelvis, joint, and velocity states.
- Reward Formulation: Auxiliary costs constrain exploration and target orientation, forward speed, sidespeed, pelvis motion, action changes, and joint torque.
- Dynamics Randomization: Dynamics randomization varies execution rate, joint mass and damping, ground friction and slope, and encoder noise during training.
- Proximal Policy Optimization: Training uses recurrent PPO with trajectory batches, fixed exploration noise, and a mirror loss intended to promote symmetric locomotion.
V. EXPERIMENTAL RESULTS
Simulation training produced standalone policies for several bipedal gaits, while additional transition penalties supported a generic multi-gait policy. Hardware experiments showed transitions and disturbance responses across varied outdoor settings.
- Training Details: 24–36 hours per policy were required for training with PPO in the Cassie MuJoCo simulator.Training used up to 150,000,000 samples per policy.
- Single-Gait Policy Results: Standalone policies learned hopping, walking, running, and skipping by holding gait ratios and cycle offsets constant during training.The paper reports these behaviors as straightforward to learn with the probabilistic framework.
- Multi-Gait Policy Results: Transition penalties enabled a generic controller to distinguish commanded behaviors when varying both cycle offsets and phase ratios proved challenging.Without these penalties, policies could produce asymmetric walking or fused, undesirable behaviors.
- Outdoor Experiments: The multi-gait policy was tested hopping on and off sidewalks, walking with one foot elevated on a curb, and running over small bumps.These outdoor tests evaluated responses to environmental disturbances.
- Force Profiles: The policies’ expected Cfrc values distinguish refraining from applying foot forces: values near -1 indicate no foot forces, while values near 0 indicate applied forces.Figure 5 compares these expected values with mean measured simulation ground reaction forces.
- Outdoor Experiments: The policy smoothly transitioned among all 2-beat gaits while moving on a crowned road and turning on turf, and descended five stairs while running.Foot slipping and compliance on small ledges suggested priority on force profiles rather than position alone.
VI. CONCLUSION
The paper introduced a probabilistic reward framework for learning common bipedal gaits without reference trajectories. It demonstrated individual gait learning and continuous transitions among 2-beat behaviors, while identifying broader morphology and aperiodic-motion applications as open questions.
- Contribution: The probabilistic reward framework learns common animal-like bipedal gaits without reference trajectories.The authors state that omitting references permits exploration of a richer interaction space without constraining policies to an arbitrary trajectory.
- Results: Policies learned walking, running, hopping, galloping, and skipping individually, and a single policy transitioned continuously among all 2-beat behaviors while moving.These results are presented as the framework’s demonstrated scope.
- Open Questions: Open questions include applying the framework to quadrupedal gaits, other bipedal morphologies, and modified formulations for aperiodic one-off behaviors.These are explicitly identified as unanswered or possible future applications.
Full Reward Functions for Generic 2-Beat Policies
The generic 2-beat reward is extended to standing by identifying a standing region and activating stillness-related costs there. A coefficient gates these additional costs according to the swing-phase ratio.
- Hopping Symmetry: Stuttering can occur when learning the continuum of 2-beat gaits, so a hop symmetry cost penalizes large inter-foot distances when leg cycle offsets closely match.This addresses the failure mode of stuttering instead of hopping with feet together.
- Standing: Standing is represented by a swing-to-stance phase ratio close to 0, and the reward is modified to keep the policy very still under standing commands.The standing region is defined in gait-parameter space.
- Standing: ω = (1+exp(−50(rswing −0.15)))−1 is near one during normal locomotion and near zero during standing.The coefficient uses the swing-phase ratio to distinguish locomotion from standing.
- Full Reward: The full generic reward includes the standing-specific terms alongside command and smoothness rewards.The displayed formulation is introduced as the full reward for a generic policy that includes standing.
- Reward Terms: The reward formulation defines current speed, pelvis orientation, actions, torque, pelvis rotation, and acceleration terms, with some terms multiplied by ω.These variables specify the physical and control quantities used by the reward.