Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion

Gwanghyeon Ji; Juhyeok Mun; Hyeongjun Kim; Jemin Hwangbo

Source-linked AI summary

Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion

Gwanghyeon Ji, Juhyeok Mun, Hyeongjun Kim, Jemin Hwangbo

arXiv:2202.05481v2Feb 11, 2022cs.ROcs.LGeess.SY

TL;DR

The paper addresses unreliable state estimation and dependence on accurate state inputs for robust legged locomotion on challenging terrain. It concurrently trains a control policy and interpretable state estimator in simulation, then transfers the networks to a real robot. The resulting system traverses diverse terrains and reaches 3.75 m/s on flat ground and 3.54 m/s on a slippery plate with µ = 0.22.

Problem
Existing locomotion approaches rely on accurate state estimates, but estimation can become unreliable on challenging terrains and some methods require gait patterns or dedicated contact sensing.
Method
The framework concurrently trains a policy network for desired joint positions and a state estimation network for linear velocity, foot height, and contact probability in simulation before real-robot transfer.
Results
3.75 m/s is achieved on normal outdoor ground and 3.54 m/s on a slippery plate with µ = 0.22, while traversing hills and bumpy roads.
Takeaways & Limitations
The framework supports dynamic and robust locomotion without requiring an advanced control algorithm or an accurate state estimation algorithm.

Abstract

from arXiv · show

In this paper, we propose a locomotion training framework where a control policy and a state estimator are trained concurrently. The framework consists of a policy network which outputs the desired joint positions and a state estimation network which outputs estimates of the robot's states such as the base linear velocity, foot height, and contact probability. We exploit a fast simulation environment to train the networks and the trained networks are transferred to the real robot. The trained policy and state estimator are capable of traversing diverse terrains such as a hill, slippery plate, and bumpy road. We also demonstrate that the learned policy can run at up to 3.75 m/s on normal flat ground and 3.54 m/s on a slippery plate with the coefficient of friction of 0.22.

I. INTRODUCTION

Existing quadrupedal locomotion methods depend on state estimates that can become unreliable on challenging terrain, while many alternatives require gait patterns or costly and damage-prone contact sensing. The paper addresses these shortcomings with concurrent policy and state-estimator training, demonstrating dynamic locomotion across slippery terrain, slopes, and other environments.

Existing quadrupedal control approaches rely on accurately estimated state inputs.
State estimation can become unreliable on ice and sand, while contact sensing is either computationally costly or vulnerable to landing damage.
3.74 m/s is reported for highly dynamic locomotion across indoor and outdoor environments, alongside robust behavior on slippery plates, bumpy asphalt, and hills.
The proposed framework concurrently trains a control policy and a learning-based state estimation network.
The trained networks demonstrate dynamic locomotion on slippery terrains and slopes.

II. METHOD

The method concurrently trains estimator, actor, and critic networks in simulation, using randomized environments, curriculum commands, and reward shaping before real-robot deployment.

Framework: The framework uses estimator, actor, and critic networks trained in RaiSim, with the estimator feeding state estimates to the actor.PPO trains the actor and critic, while supervised learning trains the estimator; all three networks are updated after trajectory collection.
Framework: Concurrent updates let the policy adapt to the estimator’s performance characteristics, including unreliable estimates from slippery foot contacts.
Training in Simulation: Training uses 800 flat-terrain environments with randomized initial states, plus uneven terrain and slopes up to ±10°.A 25% probability of reusing the previous episode’s final state helps address sudden real-world velocity-command changes.
Training in Simulation: A velocity-command curriculum expands x-direction commands from U1(−0.5,1.0) m/s to U1(−1.75,3.5) m/s over training.Ten percent of trajectories use zero velocity commands to learn standing still.
Reward Functions: The reward combines command tracking with efficient, natural locomotion, emphasizing foot clearance, swing-stance timing, and collision avoidance.Body contact with the environment terminates an episode and incurs a −10 reward, while foot clearance supports sim-to-real transfer.

B. Network Architecture

The framework combines actor, critic, and estimator networks, with the estimator providing robot-state estimates to the actor for desired joint-position control. These networks use MLP architectures and operate synchronously at 100 Hz.

B. Network Architecture: The actor and critic use [512×256×64] MLPs, while the estimator uses a [256×128] MLP.MLPs were selected for computational efficiency over memory-based networks such as RNNs.
B. Network Architecture: The architecture contains actor, critic, and estimator networks, with the estimator predicting robot states from observations and feeding them to the actor.The estimator predicts variables such as base linear velocity, foot height, and contact probability.
B. Network Architecture: The system retains analytical estimates of the body-frame gravity vector and orientation, while joint velocities are obtained by finite differences on joint positions.These estimates complement the learned state-estimation network.
B. Network Architecture: Desired joint positions are computed at 100 Hz and converted to joint torques by a PD controller running at 40 kHz on the real robot.The controller uses Kp=17 N·m·rad−1 and Kd=0.4 N·m·s·rad−1.
B. Network Architecture: The estimator is trained with supervised learning on simulation data, while actor and critic training uses reinforcement-learning methods.The figure caption specifies MSE supervision for the estimator and PPO for the actor and critic.
B. Network Architecture: Training with frequent foot slips allows the estimator to infer linear velocity from other sensor information and prior observations, while the policy receives imperfect information.This design targets robustness to possible slippages.

C. Dynamics Randomization

The training procedure uses dynamics and observation randomization to reduce discrepancies between simulation and the real robot. Randomized factors include geometry, friction, motor characteristics, controller gains, and sensor measurements.

C. Dynamics Randomization: Dynamics randomization addresses the reality gap caused by mismatches in leg length, actuator positions, and center of mass between simulation and the robot.Without dynamics randomization, the deployed controller exhibited shaky motions on the real robot.
C. Dynamics Randomization: Foot positions, collision geometry, and ground friction are randomized at the start of each episode or iteration.These parameters are included among the randomized simulation components.
C. Dynamics Randomization: Observation noise models joint-velocity errors from numerical differentiation and occasional 2-millisecond logging delays.Joint-position and joint-velocity measurements are randomized during simulation training.
C. Dynamics Randomization: PD gains are randomized by U1(-2, 2) N·m·rad−1 for position gains and U1(-0.1, 0.1) N·m·s·rad−1 for velocity gains.The randomization mitigates motor-friction and damping effects.
C. Dynamics Randomization: Foot-position observations receive noise ranges of U1(-10, 10) mm longitudinally, U1(-5, 5) mm laterally, and U1(-20, 20) mm along leg length.Foot sphere radii are randomized to U1(6, 10) mm.

III. RESULTS

The results section notes that some reported results are also available in the accompanying video.

III. RESULTS: Some results from this section are also presented in the accompanying video.

A. Controller Descriptions

The evaluation compares the concurrent framework with implicit, sequential, MPC, and RL-LKF controllers, and includes slope-trained and state-estimator ablations. These settings support analysis of how network components affect performance metrics.

A. Controller Descriptions: The evaluation tests multiple controller settings to analyze how each component affects different performance metrics.
A. Controller Descriptions: The Implicit baseline replaces the explicit state estimator with an implicit estimator modeled after prior work.
A. Controller Descriptions: The Sequential setting trains a policy with ground-truth states before training a state estimator to replace those inputs.The two phases are trained sequentially rather than concurrently.
A. Controller Descriptions: The comparison includes a built-in MPC controller and an RL-LKF policy using an LKF state estimator on the real robot.
A. Controller Descriptions: The Concurrent model is trained on relatively flat ground, whereas Concurrent+Slope is trained on randomly generated slopes.
A. Controller Descriptions: Simulation ablations remove the linear-velocity, foot-height, contact, or all estimators to measure their individual contributions.
A. Controller Descriptions: The Implicit model uses a 20-step history and three 1D-CNN layers in its adaptation module, with encoder and adaptation outputs of dimension 11.
A. Controller Descriptions: The total training time for phases 1 and 2 together is 7 hours.

B. Evaluation of the Performance in Simulation

Simulation comparisons show that concurrent estimation improves learning outcomes, with linear velocity estimation most influential and contact estimation mainly improving learning stability.

After 2500 iterations, all controllers converged to stable expected returns following approximately 800 million samples and 4 hours of real-time training.
The Concurrent model converged to the highest rewards, while omitting the estimator produced the lowest performance.
Omitting linear velocity estimation significantly reduced total, linear velocity, and foot clearance rewards.
Explicit foot-height estimation improved foot clearance and increased the corresponding reward.
Contact probability estimation had the least effect on final performance but stabilized and accelerated learning.
The evaluation randomized velocity commands over specified linear and angular ranges while maintaining flat-ground friction at 0.6.

2) Tracking and Estimation Error:

Tracking experiments found that concurrent estimation reduced steady-state velocity errors and outperformed sequential training, while foot clearance was separately evaluated for rough-terrain robustness.

Tracking and Estimation Error: The Concurrent model achieved the smallest RMS errors when following commanded linear velocity.
Tracking and Estimation Error: The estimated states helped stabilize the robot, whereas the implicit estimator provided tracking performance similar to omitting an estimator.
Tracking and Estimation Error: The concurrently trained model performed better than the sequentially trained model, whose performance degraded slightly.
Tracking and Estimation Error: Concurrent training uses one training dataset, making it more efficient than sequential training.
Tracking and Estimation Error: Rough-terrain fall resistance was evaluated using average time to fall on terrain with z-scale 0.525 under commands with relatively small foot clearance.

3) Locomotion on rough terrains:

Rough-terrain and computational evaluations compare controller robustness, learning curves, and estimator cost across simulated and real-robot settings.

Locomotion on rough terrains: The Concurrent model showed the strongest rough-terrain performance and resisted falls through increased foot clearance.
Locomotion on rough terrains: Implicit-estimator and single-policy models lacked sufficient foot clearance and were easier to make fall.
Computational Cost: The estimator network required 7 µs per forward pass, compared with 34 µs for the Mini Cheetah linear Kalman filter.
Computational Cost: The implicit estimator with 20 history inputs required 20 µs, about three times longer than the explicit estimator.
Locomotion on rough terrains: Learning curves report total reward, linear velocity reward, and foot clearance reward for models estimating linear velocity, foot height, and contact probability.
Locomotion on rough terrains: Real-robot evaluation measured command following, state estimation, maximum running speed, traversable slope angle, and foot clearance.
Locomotion on rough terrains: Real-environment comparisons included Built-in MPC, RL-LKF, Concurrent, and Concurrent+Slope controllers.
Locomotion on rough terrains: The slippery-plate tests used boric acid powder and a friction coefficient of 0.22, lower than in training environments.

2) Command Following and State Estimation:

The concurrently trained controllers maintain robust locomotion across normal, slippery, uneven, and sloped terrains while achieving high running speeds. The Concurrent+Slope model improves hill climbing, and explicit state estimation supports foot clearance and robustness to estimation errors.

Command Following and State Estimation: 3.75 m/s was the Concurrent+Slope model’s maximum outdoor speed.Controllers were tested repeatedly until the robot fell.
Command Following and State Estimation: 3.54 m/s was achieved on a slippery plate with µ = 0.22, including recovery from large foot slippages near maximum speed.The controller also formed a stable stopping pose when the command was suddenly set to zero.
Command Following and State Estimation: The Concurrent model outperformed Built-in MPC and RL-LKF on normal and slippery ground.Built-in MPC stayed below 1.7 m/s on normal ground and fell below 1.3 m/s on slippery ground, while RL-LKF remained below 2.2 m/s.
Command Following and State Estimation: Training with randomized slope angles and friction coefficients significantly improved climbing performance.Concurrent+Slope climbed normal hills up to 19.1°, beyond the ±10° training slopes, and slippery hills up to 9.0°.
Command Following and State Estimation: Other controllers could not climb normal hills steeper than 12.4°.Their training used nearly flat terrains, limiting climbing performance.
Command Following and State Estimation: Explicit foot-clearance estimation improved performance because lower clearance hindered stable locomotion on highly uneven terrain.Concurrent and Concurrent+Slope had higher real-robot foot clearance than RL-LKF and w/o Estimator at 1.0 m/s.

5) Foot Clearance:

The estimator identifies foot contacts from proprioceptive motion, with detected contacts delayed relative to actual contacts. Concurrent training remains robust despite estimation errors and produces interpretable estimates usable with other controllers.

Foot Clearance: 0.04 seconds is the reported delay between estimated and actual contacts.The delay corresponds to three control steps, excluding 0.01 seconds of communication delay.
Foot Clearance: The estimator detects contact when the joint abruptly stops after impact with the ground.Contact ends with upward joint motion, and the estimated contact is shifted by 0.04 seconds from the actual contact.
Foot Clearance: Concurrent training requires neither an advanced control algorithm nor an accurate state-estimation algorithm.The framework is reported to outperform implicit estimation and sequential training in command tracking, rough-terrain robustness, and training time.
Foot Clearance: The learned state estimator outputs interpretable estimates that may also support motion analysis and MPC-based controllers.This potential use is stated independently of the control policy.