PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Evolutionary Trajectory Generator: A General Approach for Quadrupedal Locomotion

Haojie Shi, Bo Zhou, Hongsheng Zeng, Fan Wang, Yueqiang Dong, Jiangyong Li, Kang Wang, Hao Tian, Max Q. -H. Meng

arXiv:2109.06409v2cs.ROcs.AI

TL;DR

Learning effective quadrupedal gaits from scratch remains difficult in nonlinear dynamics, especially for challenging tasks such as balance-beam walking. The paper combines an evolutionary foot trajectory generator with reinforcement-learning policy updates, and reports success in challenging simulation tasks and real-world scenarios on a 12-DoF robot.

  • Problem

    Learning effective quadrupedal locomotion from scratch remains challenging because of nonlinear dynamics, and a unified framework for versatile skills in difficult environments such as balance-beam walking remains an open problem.

    Source: §I · p. 2

  • Method

    The method evolves task-specific foot trajectories and trains a residual-control policy with reinforcement learning, alternating their optimization and reusing sampled trajectories in the policy's replay buffer.

    Source: §I · p. 2, B. Policy learning via RL · p. 5, C. Dual Training · p. 5

  • Results

    The approach solves all reported tasks while significantly outperforming baselines in all but Terrain in simulation, and the transferred controller traverses six real-world environments on a 12-DoF robot.

    Source: A. Simulation Experiments · p. 7, C. Physical Experiments · p. 7

  • Takeaways & Limitations

    The reported simulations and physical trials show that the framework can learn challenging quadrupedal gaits, including balance-beam walking, and transfer the controller to real-world tasks.

    Source: C. Physical Experiments · p. 7, §VII · p. 8

Abstract

from arXiv · show

Recently reinforcement learning (RL) has emerged as a promising approach for quadrupedal locomotion, which can save the manual effort in conventional approaches such as designing skill-specific controllers. However, due to the complex nonlinear dynamics in quadrupedal robots and reward sparsity, it is still difficult for RL to learn effective gaits from scratch, especially in challenging tasks such as walking over the balance beam. To alleviate such difficulty, we propose a novel RL-based approach that contains an evolutionary foot trajectory generator. Unlike prior methods that use a fixed trajectory generator, the generator continually optimizes the shape of the output trajectory for the given task, providing diversified motion priors to guide the policy learning. The policy is trained with reinforcement learning to output residual control signals that fit different gaits. We then optimize the trajectory generator and policy network alternatively to stabilize the training and share the exploratory data to improve sample efficiency. As a result, our approach can solve a range of challenging tasks in simulation by learning from scratch, including walking on a balance beam and crawling through the cave. To further verify the effectiveness of our approach, we deploy the controller learned in the simulation on a 12-DoF quadrupedal robot, and it can successfully traverse challenging scenarios with efficient gaits.

I. INTRODUCTION

Manual quadrupedal controllers require expert knowledge and tuning, while learning versatile skills from scratch remains difficult in nonlinear dynamics, especially in challenging environments. The paper adapts foot trajectories to tasks and reports results in challenging simulation tasks and on a real quadrupedal robot.

I. INTRODUCTION

  • Fixed foot trajectory generators may not fit specific tasks, leaving manual effort to provide motion priors and potentially impairing policy performance.The paper identifies an open need for a unified framework for versatile quadrupedal skills, particularly in challenging environments such as a narrow balance beam.Source: §I · p. 2
  • The method evolves foot trajectories in trajectory space with evolutionary strategies while reinforcement learning trains the policy, then alternates optimization of the two components.The policy outputs residual control signals, and alternating updates freeze one component while training the other.Source: §I · p. 2
  • The authors report that their approach outperforms prior learning-based proposals on challenging simulation tasks, including balance-beam walking and jumping across plank intervals.They state that most compared proposals cannot make much progress on those tasks.Source: §I · p. 2
  • The approach is evaluated on challenging simulation tasks and transferred to a 12-DoF robot, where it demonstrates stair climbing and balance-beam walking.The authors describe the real-world results as the first learning-based controller to walk over a balance beam.Source: §I · p. 2

II. RELATED WORK

  • Prior trajectory-generator and learning-based approaches include fixed reference trajectories, whereas this work adapts generator outputs to the task.The cited related-work passage describes the controller architecture as following [24] while improving its generator to optimize task-specific trajectories.Source: §II · pp. 2–3

III. PRELIMINARIES

The section frames quadrupedal control as a Markov decision process and describes reinforcement learning as finding a controller that maximizes reward. It introduces actor-critic learning, with a value function and policy optimization forming the learning setup.

III. PRELIMINARIES

  • Quadrupedal control is formulated as an MDP with states, actions, transition probabilities, rewards, and an initial-state distribution.The controller selects an action from the current state, receives a reward, and moves to a state determined by the transition function.Source: A. Deep reinforcement learning for… · p. 3

A. Deep reinforcement learning for quadrupedal locomotion

  • The controller's objective is to find a policy that maximizes its average reward, with γ discounting accumulated reward.The paper adopts actor-critic reinforcement learning and defines a neural-network state-action value function to estimate expected future reward.Source: A. Deep reinforcement learning for… · p. 3
  • The value function is approximated by minimizing temporal-difference error, while the policy is optimized toward maximizing expected future reward.Experience replay stores transitions, and two target networks maintain slow copies of the value function and policy for TD-error computation.Source: A. Deep reinforcement learning for… · p. 3

B. Problem setting

The robot state combines inertial, joint, contact, and estimated velocity measurements; actions specify desired joint angles tracked by a joint-level PD controller.

B. Problem setting

  • The state includes IMU roll, pitch, and yaw; each leg joint's angle and angular velocity; four ground-contact indicators; and estimated body velocity.These signals constitute the controller's observation of the robot.Source: 1) State and action space · p. 3
  • Control signals specify the desired angle of each joint, which the joint-level PD controller tracks.The action is expressed as desired joint angles.Source: 1) State and action space · p. 3
  • Each leg contributes a binary ground-contact value to the state, alongside its joint angles and angular velocities.The passage specifies four binary contact values, one for each leg.Source: 1) State and action space · p. 3

1) State and action space:

The section describes a reward balancing directional progress and energy use, alongside CPG- and RBF-based foot-trajectory generation.

2) Reward function:

  • The reward encourages fast movement in the desired direction while penalizing unnecessary energy consumption.λ sets the importance of energy consumption in the reward.Source: 2) Reward function · p. 3

C. CPG for locomotion controller

IV. METHODOLOGY

The methodology pairs evolutionary search over foot trajectories with reinforcement learning for quadrupedal locomotion.

IV. METHODOLOGY

  • ETG-RL improves the foot-trajectory generator to provide diversified reference trajectories, while a neural-network policy learns through reinforcement learning in simulation.Source: §IV · p. 4
  • The method searches directly in trajectory space, which is described as more efficient than parameter-space search and allows desired foot trajectories to specify motion priors directly.Source: §IV · p. 4
  • The controller architecture combines an open-loop trajectory generator with a neural-network policy to learn gaits for challenging tasks.Source: §IV · p. 4

A. Evolutionary Trajectory Generator

The evolutionary trajectory generator searches over foot trajectories to provide task-specific motion priors for policy learning.

A. Evolutionary Trajectory Generator

  • Direct trajectory-space search supports specifying a desired foot trajectory as a motion prior, unlike setting RBF-network parameters directly.The passage describes trajectory-space search as a way to provide diversified reference trajectories.Source: A. Evolutionary Trajectory Generator · p. 4
  • The desired trajectory is represented by n control points, which are used to solve for RBF parameters that initialize the network.Figure 3 depicts the desired trajectory as a dotted line with red stars.Source: A. Evolutionary Trajectory Generator · p. 4, Figure 3 · p. 5
  • At each iteration, ES samples trajectories by adding Gaussian noise to control points and updates RBF parameters according to sampled-trajectory performance.Searching trajectory space is described as lower-dimensional than parameter space and avoids samples that may deviate unexpectedly from the current trajectory.Source: A. Evolutionary Trajectory Generator · pp. 4–5
  • ETG evaluates noisy candidate trajectories, aggregates their performance-weighted parameter distances, and updates the trajectory parameters until convergence.Source: A. Evolutionary Trajectory Generator · p. 5

B. Policy learning via RL

The policy learns residual controls alongside an open-loop foot-trajectory generator, with its inputs including both sensor data and generator signals.

B. Policy learning via RL

C. Dual Training

Dual training alternates optimization of the trajectory generator and policy to stabilize learning and reuse exploratory trajectories. The sim-to-real section addresses the simulation–robot gap, and the training curves report results across nine tasks.

C. Dual Training

  • Alternating between trajectory-generator and policy optimization stabilizes training by avoiding simultaneous changes to behavior distributions.Each component is frozen while the other is optimized.Source: C. Dual Training · p. 5
  • ETG-sampled trajectories are added to the replay buffer, reusing exploratory data for policy optimization and improving sample efficiency.The algorithm presents the ETG phase before the RL phase.Source: C. Dual Training · p. 5

V. SIM-TO-REAL TRANSFER

  • Sim-to-real transfer is treated as a challenge because simulation and the real robot differ.The section introduces methods to address this gap.Source: §V · p. 5
  • Figure 4 plots average training scores and standard deviations for ETG-RL and baselines across nine tasks, using four random seeds per experiment.The tasks correspond to scenarios S1–S9 in Figure 1.Source: Figure 4 · p. 6

A. Behavior Cloning

The transfer procedure addresses noisy real-world velocity sensing through behavior cloning, simulation parameter calibration, and noisy-observation training. The student policy imitates a teacher with full velocity information and performs similarly in simulation.

A. Behavior Cloning

  • A student policy learns control signals by imitating a teacher with velocity input, avoiding direct reliance on noisy real-world body-velocity measurements.In simulation, the student achieves performance similar to the original policy.Source: A. Behavior Cloning · p. 6

B. Domain Adaptation

  • Domain adaptation calibrates simulation dynamics by black-box optimization to minimize the average joint-angle difference from the real robot.The robot is driven with predefined control signals while joint angles are collected.Source: B. Domain Adaptation · p. 6

C. Observation Randomization

  • Adding sensor noise during behavior cloning encourages the student policy to adopt robust behaviors for real-world deployment.The approach follows prior work [19].Source: C. Observation Randomization · p. 6

VI. EXPERIMENTS

ETG-RL is evaluated against CPG-RBF and SAC across nine challenging simulation tasks, with ablations examining trajectory optimization and dual training. It is the only method reported to solve every task, while the ablations show that both trajectory search and dual training matter across most tasks.

VI. EXPERIMENTS

  • The evaluation compares ETG-RL with CPG-RBF and SAC across nine PyBullet tasks, including stairs, uneven terrain, a narrow balance beam, plank gaps, and a cave.The experiments also include real-world evaluation and an ablation study.Source: §VI · pp. 6–7, A. Simulation Experiments · p. 7

A. Simulation Experiments

  • ETG-RL is the only algorithm reported to solve all nine tasks and significantly outperforms baselines except on Terrain.The comparison is presented in Table I; SAC also solves Terrain, where the robot is less prone to fall.Source: A. Simulation Experiments · p. 7, Table I · p. 7

B. Ablation Study

  • ETG-RL’s performance drops significantly without dual training in almost all tasks except Terrain and Balance, where the trajectory generator’s motion prior performs decently.The ablation comparison is presented in Table II.Source: B. Ablation Study · p. 7, Table II · p. 7
  • Replacing ETG with CPG degrades performance; trajectory-space search is less likely than parameter-space sampling to produce trajectories that deviate unexpectedly.The comparison evaluates variants across nine tasks.Source: B. Ablation Study · p. 7, Table II · p. 7

C. Physical Experiments

The transferred controller successfully traverses six real-world environments on a 12-DoF quadrupedal robot, including stairs, slopes, and a balance beam. The conclusion also reports learning challenging simulation gaits and transferring the controller to the robot.

C. Physical Experiments

  • The transferred controller successfully traverses all six real-world environments, including climbing stairs and slopes and walking over a balance beam.The experiments use a 12-DoF Unitree quadrupedal robot; the paper claims this is the first learning-based controller to walk over a real-world balance beam.Source: C. Physical Experiments · p. 7

VII. CONCLUSIONS

  • The conclusion reports that ETG-RL learns efficient gaits in challenging simulation environments, including walking over a narrow balance beam.The controller is also validated on a 12-DoF quadrupedal robot with sim-to-real transfer.Source: §VII · p. 8
Loading 2109.06409v2…