PaperlayerView exact revision on arXiv

Source-linked AI summary

CPG-RL: Learning Central Pattern Generators for Quadruped Locomotion

Guillaume Bellegarda, Auke Ijspeert

arXiv:2211.00458v1cs.ROcs.AIcs.LGcs.SYeess.SY

TL;DR

Quadruped locomotion research seeks robust, omnidirectional control while clarifying how learned modulation and sensory feedback can coordinate rhythmic movement. CPG-RL uses deep reinforcement learning to modulate oscillator parameters, and transfers to the Unitree A1 with minimal feedback, including contact booleans. The authors report robustness to unseen disturbances, including a dynamically added load equal to 115% of nominal robot mass.

  • Problem

    Direct proprioception-to-joint-command learning remains challenging, including specifying desired motions such as foot swing height and achieving sim-to-real transfer.

    Source: 4) Action Space and Phase in DRL · p. 2

  • Method

    CPG-RL uses deep reinforcement learning to modulate each limb oscillator’s time-varying intrinsic amplitude and frequency, with no explicit coupling between oscillators.

    Source: B. Contribution · p. 2

  • Results

    The Unitree A1 trotted at 0.8 m/s with 13.75 kg dynamically added, representing 115% of nominal robot mass.

    Source: 5) Added Mass · p. 6

  • Takeaways & Limitations

    The authors report sim-to-real transfer without domain randomization and with only contact booleans as environmental feedback, alongside oscillator states.

    Source: 2) Observation Space Effects · p. 5, §V · p. 7

Abstract

from arXiv · show

In this letter, we present a method for integrating central pattern generators (CPGs), i.e. systems of coupled oscillators, into the deep reinforcement learning (DRL) framework to produce robust and omnidirectional quadruped locomotion. The agent learns to directly modulate the intrinsic oscillator setpoints (amplitude and frequency) and coordinate rhythmic behavior among different oscillators. This approach also allows the use of DRL to explore questions related to neuroscience, namely the role of descending pathways, interoscillator couplings, and sensory feedback in gait generation. We train our policies in simulation and perform a sim-to-real transfer to the Unitree A1 quadruped, where we observe robust behavior to disturbances unseen during training, most notably to a dynamically added 13.75 kg load representing 115% of the nominal quadruped mass. We test several different observation spaces based on proprioceptive sensing and show that our framework is deployable with no domain randomization and very little feedback, where along with the oscillator states, it is possible to provide only contact booleans in the observation space. Video results can be found at https://youtu.be/xqXHLzLsEV4.

I. INTRODUCTION

Robotics has advanced through biological, model-based, and learning-based approaches, but how animals learn and produce motion—especially how higher brain regions interact with spinal CPGs—remains unclear. These approaches also differ in how they use biological inspiration and dynamics models to control locomotion.

I. INTRODUCTION

  • The paper combines biology-inspired oscillators with deep learning to address open questions about how animals learn and produce motion.The authors particularly identify how higher brain regions interact with spinal CPGs as unclear.Source: §I · p. 1

1) Biology-Inspired Control:

  • Animal-inspired CPG control combines oscillators with sensory input, reflexes, and mechanical design to support adaptive walking on irregular terrain.Prior work also reports gait generation and transitions through simple force feedback without explicit oscillator coupling,.Source: 1) Biology-Inspired Control · p. 1

2) Model-Based Control:

  • Model-based controllers commonly use online optimization with simplified dynamics models, requiring domain knowledge and potentially limiting generalization to environments not considered during development.Pre-determined swing trajectories and Raibert heuristics can also bias resulting motions.Source: 2) Model-Based Control · pp. 1–2

3) Learning-Based Control:

Learning-based quadruped control has used proprioception and vision, while action-space design and phase encodings address challenges in learning direct joint commands. This paper proposes CPG-RL, which learns time-varying oscillator parameters and targets transfer, minimal feedback, and user-specified motion quantities.

3) Learning-Based Control:

  • DRL quadruped policies have achieved locomotion on flat and rough terrain using proprioception, while vision-based methods support obstacle avoidance and rough-terrain crossing.Other learning-based work has also explored animal-motion imitation and gait emergence through energy minimization.Source: 3) Learning-Based Control · p. 2

4) Action Space and Phase in DRL:

  • Direct proprioception-to-joint-command learning faces challenges including specifying swing-foot height and sim-to-real transfer, motivating structured action spaces and phase encodings.Prior work also uses DRL to learn parameters of CPGs or trajectory generators for locomotion –[37].Source: 4) Action Space and Phase in DRL · p. 2

B. Contribution

  • CPG-RL uses deep reinforcement learning to learn time-varying intrinsic amplitude and frequency for each limb oscillator, without explicit interoscillator coupling.The authors also identify body-height and swing-foot-height selection, minimal sensory feedback, and robustness to unseen disturbances as method highlights.Source: B. Contribution · p. 2

II. BACKGROUND

The background introduces reinforcement learning as policy learning in an MDP and describes CPG oscillators whose parameters and couplings shape rhythmic output.

A. Reinforcement Learning

  • An MDP specifies states, actions, transitions, and rewards; the agent learns a policy that maximizes expected return.The transition function gives the probability of the next state given the current state and action.Source: A. Reinforcement Learning · pp. 2–3

B. Central Pattern Generators

  • The paper uses amplitude-controlled phase oscillators, with intrinsic amplitude and frequency and optional coupling weights and phase biases.CPG design choices include oscillator count and whether outputs map directly to joint commands or through task-space trajectories and inverse kinematics,,.Source: B. Central Pattern Generators · p. 3

III. LEARNING CENTRAL PATTERN GENERATORS

CPG-RL uses reinforcement learning to modulate each leg’s oscillator parameters and coordinate locomotion. It removes explicit interoscillator coupling and adds a trajectory-orientation state, enabling the agent to learn limb coordination and direct motion in any direction.

III. LEARNING CENTRAL PATTERN GENERATORS

  • The framework learns to modulate each leg’s oscillator parameters using sensory feedback and the current CPG state.The paper frames this as an approximation of higher brain centers sending modulation signals to spinal CPG circuits.Source: §III · p. 3

A. Action Space

  • Removing explicit coupling and adding φ lets the agent learn limb coordination and orient motion in any direction.The authors identify coupling as a potential source of unnatural rhythms and two-state motions as restricted to a single plane.Source: A. Action Space · p. 3
  • The policy directly adjusts each leg’s amplitude, frequency, and phase-related parameters online, rather than using only fixed oscillator settings.These parameters can be adjusted, for example, to increase step length or accelerate a leg during swing.Source: A. Action Space · p. 3
  • Desired foot positions are converted to joint positions through inverse kinematics before control.The framework maps oscillator states to foot positions, then calculates desired joint positions.Source: A. Action Space · p. 3
  • Randomly varying robot height and swing-clearance parameters during training lets users specify those quantities at deployment.The agent receives no explicit observation of the sampled parameters.Source: A. Action Space · p. 3

B. Observation Space

The paper compares full, medium, and minimal observation spaces, keeping CPG states in each. The minimal space uses only contact booleans and CPG states as observations.

B. Observation Space

  • The policy selects CPG parameters at 100 Hz, while the other architecture blocks operate at 1 kHz.The policy input includes velocity commands, selected proprioceptive measurements, and current CPG states.Source: Figure 2 · p. 4
  • Full observation combines velocity commands, body and joint state, contact booleans, the previous action, and CPG states.The CPG states are always known, unlike proprioceptive measurements that are subject to sensor noise.Source: Figure 3 · p. 4
  • Medium observation removes joint state and the previous action, retaining velocity commands, body state, contact booleans, and CPG states.This space tests locomotion without joint information.Source: Figure 3 · p. 4
  • Minimal observation contains only contact booleans and CPG states, so foot contacts provide its only environmental feedback.For this observation space, the task is forward motion at a desired velocity.Source: Figure 3 · p. 4

C. Reward Function

The reward tracks commanded body-frame velocities and yaw rate while penalizing unwanted body motion and work. Training uses simulation with sampled commands and parameter randomization, although the paper notes randomization is not strictly needed for sim-to-real transfer.

C. Reward Function

  • The reward tracks commanded body-frame x and y velocities and yaw rate, while penalizing vertical and roll/pitch motion and work.The authors describe these terms as a simple way to specify desired behavior without adding foot-swing-time terms.Source: C. Reward Function · p. 4

D. Training Details

  • Training uses Isaac Gym, [44] with PhysX to simulate 4096 Unitree A1 quadrupeds in parallel on one NVIDIA RTX 3070 GPU, and uses PPO to train the policy.Episodes last at most 20 seconds, and the terrain is always flat during training.Source: D. Training Details · pp. 4–5
  • Domain randomization is not strictly required for sim-to-real transfer, but training typically randomizes friction, limb mass, added base mass, and external pushes.The push is applied in a random direction every 15 seconds; no observation noise is added.Source: D. Training Details · p. 5
  • The policy runs at 100 Hz, torques update at 1 kHz, and sim-to-real proprioceptive information comes from Unitree sensors.The oscillator equations are also integrated at 1 kHz.Source: D. Training Details · p. 5

IV. EXPERIMENTAL RESULTS AND DISCUSSION

The experimental section evaluates how observation-space richness relates to locomotor performance, sim-to-real transfer, policy interpretability, and robustness to disturbances. It points to figure and video results for the experiments discussed.

IV. EXPERIMENTAL RESULTS AND DISCUSSION

  • The experiments assess whether locomotion requires full, medium, or minimal sensory observations.The section frames observation-space richness as a question of locomotor performance.Source: §IV · p. 5
  • The section evaluates the difficulty of transferring learned controllers from simulation to the robot.Source: §IV · p. 5
  • The evaluation also considers policy interpretability and robustness to disturbances not seen during training.The section directs readers to Figure 1 and the supplementary video for examples.Source: §IV · p. 5

1) CPG State Modulation:

CPG-RL produces an adaptable trot by coordinating oscillator amplitude and phase, and transfers across observation spaces from full sensing to contact-only feedback. Users can also change body height and foot clearance online while the robot continues following velocity commands.

1) CPG State Modulation:

  • At 0.8 m/s, the policy produces an approximate trot with faster swing than stance and greater amplitude during stance to propel the quadruped forward.The phase-amplitude coordination is also visible in the CPG state plot.Source: 1) CPG State Modulation · p. 5, Figure 4 · p. 5

2) Observation Space Effects:

  • Contact-only feedback supports sim-to-real transfer without simulator noise or randomization, while removing contact feedback prevents learning fixed-velocity locomotion.The contact-only policy tracks 0.5 m/s forward with other velocities at zero; policies using full or medium observations also transfer with and without joint state information.Source: 2) Observation Space Effects · p. 5

3) Body Height and Swing Foot Height Online Modulation:

  • After training, body height changes from 0.30 m to 0.19 m and back, while foot clearance changes from 0.03 m to 0.12 m and back during locomotion.The height adjustment lets the robot crawl under a ledge; the clearance reaches almost half the nominal robot height.Source: 3) Body Height and Swing Foot Height… · p. 6

4) Uneven Terrain:

On hardware, CPG-RL remains robust to light debris and large added loads, including loads beyond those used in simulation. In sim-to-sim comparisons, it produces more natural gaits and offers direct swing-clearance control, unlike the Joint PD baseline.

4) Uneven Terrain:

  • The policy continued walking when soft foam or hard styrofoam caught or impeded swing feet, although training used flat terrain.The authors note the tested materials were light and easy for the A1 to kick or crumple.Source: 4) Uneven Terrain · p. 6

5) Added Mass:

  • The A1 trotted at 0.8 m/s with a dynamically added 13.75 kg load, 115% of nominal mass, despite simulation randomization adding only up to 5 kg.The authors report no noticeable performance drop across tested loads and velocities of 0.3–0.8 m/s.Source: 5) Added Mass · p. 6

B. Sim-to-Sim Comparison with Learning in Joint Space

  • CPG-RL produced natural-looking, interpretable gaits and allowed swing-clearance control; Joint PD with the same reward produced unnatural gaits, while its special reward reduced tracking accuracy.The sim-to-sim evaluation compares Isaac Gym policies transferred to Gazebo; the special-reward Joint PD policy also lacked direct swing-clearance control.Source: B. Sim-to-Sim Comparison with Learning… · pp. 6–7

1) Sim-to-sim Forward Locomotion:

In sim-to-sim load tests, CPG-RL policies continued forward locomotion under loads up to 15 kg, including policies trained without noise. For forward velocity tracking, CPG-RL with full and medium observations performed best among the compared methods.

1) Sim-to-sim Forward Locomotion:

  • CPG-RL with full and medium observations achieved the best forward-velocity tracking; its higher mean duty factor and gait period suggest greater stability.Unlike Joint PD policies, all CPG-RL policies could vary body height and foot clearance online.Source: 1) Sim-to-sim Forward Locomotion · p. 7

2) Sim-to-sim with Added Loads:

  • CPG-RL policies trained without noise still locomoted with 15 kg added, while policies trained with noise of up to 5 kg made some forward progress under loads up to 9 kg.The 15 kg load represents 250% of nominal robot mass; performance was not guaranteed at higher loads, with falls or stopping increasingly likely.Source: 2) Sim-to-sim with Added Loads · p. 7

3) Sim-to-sim Omnidirectional Commands:

CPG-RL closely tracks forward, lateral, and angular commands, including combinations, while Joint PD policies face transfer difficulties or fail to learn with minimal observations. The authors conclude that contact or limb loading is important sensory information and that stable locomotion can occur with weak or absent oscillator coupling.

3) Sim-to-sim Omnidirectional Commands:

  • CPG-RL policies closely track desired forward, lateral, and angular velocities and their combinations, even when trained without noise.Joint PD omnidirectional policies track less accurately and have a narrow, seed-inconsistent sim-to-sim transfer window.Source: 3) Sim-to-sim Omnidirectional Commands · p. 7

4) Joint PD Observation Spaces:

  • Joint PD with medium observations can learn omnidirectional locomotion but cannot transfer, while Joint PD with minimal observations cannot learn locomotion.The transferring medium-observation policies develop high-frequency, small-step gaits that exploit simulator dynamics.Source: 4) Joint PD Observation Spaces · p. 7

V. CONCLUSION

  • The authors conclude that stable locomotion can use weak or nonexistent oscillator coupling, and that limb contact or loading is among the most important sensory information.They relate the contact-feedback finding to a neuromechanical simulation of cat locomotion.Source: §V · p. 7
Loading 2211.00458v1…