PaperlayerView exact revision on arXiv

Source-linked AI summary

RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control

Siddhant Gangapurwala, Mathieu Geisert, Romeo Orsolino, Maurice Fallon, Ioannis Havoutis

arXiv:2012.03094v3cs.ROcs.LG

TL;DR

The paper addresses robust quadrupedal locomotion over uneven terrain despite the limitations of simplified models and simulation-based RL transfer. It combines RL footstep planning with model-based motion control and auxiliary adaptive policies, achieving reliable complex-terrain traversal and transfer from ANYmal B to ANYmal C without retraining.

  • Problem

    Quadrupedal locomotion over rough terrain remains difficult because simplified models can lose accuracy, while simulation-trained RL policies can suffer from mismatched real-world dynamics.

  • Method

    RLOC combines an RL footstep planning policy with a model-based motion controller, domain-adaptive tracking, and recovery control policies.

  • Results

    90.2% success on the bricks environment, compared with 71.5% for the baseline perceptive controller, with reliable dynamic traversal over uneven terrain.

  • Takeaways & Limitations

    The modular framework supports perceptive, dynamic locomotion and transfers policies trained with ANYmal B to ANYmal C without retraining.

Abstract

from arXiv · show

We present a unified model-based and data-driven approach for quadrupedal planning and control to achieve dynamic locomotion over uneven terrain. We utilize on-board proprioceptive and exteroceptive feedback to map sensory information and desired base velocity commands into footstep plans using a reinforcement learning (RL) policy. This RL policy is trained in simulation over a wide range of procedurally generated terrains. When ran online, the system tracks the generated footstep plans using a model-based motion controller. We evaluate the robustness of our method over a wide variety of complex terrains. It exhibits behaviors which prioritize stability over aggressive locomotion. Additionally, we introduce two ancillary RL policies for corrective whole-body motion tracking and recovery control. These policies account for changes in physical parameters and external perturbations. We train and evaluate our framework on a complex quadrupedal system, ANYmal version B, and demonstrate transferability to a larger and heavier robot, ANYmal C, without requiring retraining.

I. INTRODUCTION

Legged locomotion offers mobility over inaccessible terrain but requires complex dynamic stability and contact control. RLOC addresses limitations of model-based and model-free approaches by combining reinforcement learning with optimal control for terrain-aware, adaptive locomotion.

  • Legged robots can traverse terrains inaccessible to similarly sized wheeled or tracked robots, but must maintain dynamic stability, select contacts, and balance actively.
  • Simplified models lose descriptive accuracy on complex geometry, while contact-force formulations add computational delays that reduce reactive control.
  • Model-predictive controllers for uneven terrain often rely on approximated system models and require exhaustive tuning of planning and control parameters.
  • RL methods provide low-overhead mappings from sensory information to control signals, but simulation training is needed to avoid physical damage and remains vulnerable to simulation-to-real dynamics mismatch.
  • RLOC combines a perceptive RL footstep planner with a model-based dynamic motion controller and adaptive tracking to support locomotion over uneven terrain.
  • The framework demonstrates transfer from simulation-trained ANYmal B policies to the larger ANYmal C without retraining.

B. Motion Parameterization

The motion is parameterized with quintic splines for center-of-mass and foot trajectories, using contact states to distinguish swing, stance, and uncertainty during tracking.

  • Center-of-mass motion: Center-of-mass motion is represented as a sequence of quintic splines with position, velocity, and acceleration obtained from the spline basis.Each spline occupies a time interval and uses coefficient vectors to define the motion.
  • Foot motion: Foot motion is parameterized by a four-element contact-state vector whose values encode open contact, uncertain contact, and closed contact.Planning uses only open and closed states, while the uncertain state adapts joint PD gains during tracking.
  • Foot motion: Swing phases correspond to open contacts, whereas stance phases correspond to closed contacts for each foot.The phase descriptions include lift-off, touch-down, and clearance quantities.
  • Reinforcement-learning formulation: The RL formulation models sequential decisions with an MDP comprising states, actions, rewards, transition probabilities, and an initial-state distribution.The policy maps states to action distributions and is optimized for expected discounted cumulative return.

III. METHODOLOGY

The methodology combines an RL footstep planner with model-based motion control and terrain encoding, using modular components to support real-time operation and transfer across robots.

  • Framework architecture: The framework uses proprioceptive and exteroceptive information to generate desired footholds, which are updated after each completed gait stride.The dynamic-gaits controller then generates and tracks the corresponding center-of-mass and foot motion plans.
  • Framework architecture: The modular architecture includes a footstep planner, motion controller, domain-adaptive tracker, and recovery controller.The modules are described as separately trained components with distinct planning, tracking, and recovery roles.
  • Framework architecture: Separate modules facilitate tuning for different tasks and enable zero-shot transfer from simulated ANYmal B to physical ANYmal C.The transfer is motivated by adapting the motion controller or domain-adaptive tracker for different robot properties.
  • Elevation-map encoding: The elevation-map pipeline addresses depth-camera occlusions and noisy physical measurements through interpolation and denoising.Preprocessing also avoids repeated elevation-map processing by the multiple modules that use exteroceptive terrain information.
  • Elevation-map encoding: Terrain elevation is encoded into a 96-dimensional latent representation using a denoising convolutional autoencoder.The encoder is trained to reconstruct normalized elevation maps after artificial noise is introduced.
  • Elevation-map encoding: The autoencoder decoder maps the latent embedding back to an approximate normalized elevation map, and encoder and decoder are trained jointly with reconstruction loss.Training uses noisy inputs and terrain patches generated from stairs, waves, bricks, planks, and unstructured ground.

B. Motion Controller

The motion controller extends dynamic-gait planning to uneven terrain by aligning commands with estimated terrain geometry and tracking planned footholds through whole-body control.

  • Terrain-aware planning: The framework replaces direct desired-footstep generation with an RL footstep planner while retaining the motion controller for terrain-aware gait execution.This design separates high-level foothold selection from model-based motion generation and tracking.
  • Terrain-aware planning: The controller expresses velocity commands in a terrain-aligned control frame whose vertical axis follows the estimated terrain-plane normal.Desired heading, lateral velocity, and yaw rate are represented in this frame.
  • Foothold and gait generation: A quadratic-programming foothold planner generates desired touchdown positions, while terrain height supplies the vertical touchdown component.The contact scheduler defines lift-off and swing and stance timings for each gait stride.
  • Whole-body tracking: A hierarchical whole-body controller optimizes generalized accelerations and contact forces to produce feed-forward joint torques.Desired joint positions and velocities are extracted from the planned foot motions using inverse kinematics.
  • Whole-body tracking: Adaptive impedance gains use contact-state information to increase stiffness for legs associated with uncertain or slipping contacts.This approach is applied to whole-body tracking of the reference motion plans.

C. Footstep Planner

The footstep planner maps proprioceptive and exteroceptive robot information to desired foothold positions, which are passed to a dynamic-gaits controller. Its sequential MDP rewards velocity tracking, stability, low energy use, and safe foothold placement.

  • State and Action: The neural footstep planner maps robot state and terrain information to desired positions for all four feet.The desired foothold positions are forwarded to the dynamic-gaits motion controller.
  • State and Action: The planner executes sequentially after each gait stride, updating footholds when the feet enter stance.This formulation treats footstep planning as an infinite-state, infinite-action MDP.
  • State and Action: The state includes base orientation and velocities, joint-state history, desired base velocity, and an embedded robocentric elevation map.Joint-state history helps infer the active gait, whose stability margin affects foothold selection.
  • State and Action: The action contains horizontal-frame foothold locations for four feet, while foothold heights are extracted directly from the elevation map.The resulting positions are transformed to the world frame before controller execution.
  • Reward Function: The reward combines velocity tracking, joint-torque cost, foot-slippage penalty, stability margin, nominal footholds, terrain-edge distance, and foot height.A curriculum factor increases during training to refine behavior.

3) Episode Termination Criteria:

The framework terminates footstep-planning episodes when terrain inclination, collisions, or self-collision violate safety conditions. It uses stochastic SAC training and domain-adaptive tracking to compensate for model mismatch and support transfer across dynamics.

  • Episode Termination Criteria: Episodes terminate when the terrain normal deviates beyond its threshold from the base vertical axis.This criterion prevents continued training under excessive terrain inclination.
  • Episode Termination Criteria: Episodes also terminate after non-foot terrain collisions or robot self-collision, followed by resetting the robot for a new episode.These criteria enforce contact-safety constraints during training.
  • Policy Training: The footstep policy uses a bounded Gaussian distribution with state-dependent mean and log standard deviation, trained using entropy-regularized SAC.State-dependent stochasticity supports exploration during training.
  • Domain Adaptive Tracker: The tracker addresses dynamics mismatch, enabling motion-controller deployment across systems with different dynamics without retuning control parameters.The authors connect this reformulation to zero-shot transfer from ANYmal B to ANYmal C.
  • Domain Adaptive Tracker: The domain adaptive tracker receives robot state and tracking-error history and outputs corrective joint torques.Its action is the corrective torque δτj introduced into the impedance controller.

2) Reward Function:

The framework combines tracking-oriented rewards for domain adaptation with a recovery controller that stabilizes perturbed robots. Recovery emphasizes stability margin while penalizing slippage, energy, and excessive motion, and it can resist substantially larger lateral forces than the motion controller.

  • Reward Function: The domain adaptive tracker is formulated as imitation learning that generates corrective feedforward torques to track reference motion plans.Its rewards penalize base-position, base-velocity, joint-position, and joint-velocity tracking errors.
  • Policy Parameterization: The tracker represents its policy as a deterministic state-to-action mapping with bounded corrective torque actions and execution noise for exploration.It uses a discrete probability representation and clips actions to [−40,40] Nm.
  • Recovery Control: The recovery controller adds a foothold-optimization cost that balances desired-velocity tracking against stability during external disturbances.The optimizer generates footholds in the perturbation direction, with the weighting determining the stability-versus-tracking trade-off.
  • Recovery Control: The recovery controller activates when tuned robot-state criteria exceed limits and deactivates after the criteria return within deactivation limits.It operates at 400 Hz during recovery.
  • Recovery Control: More than twice the lateral-force magnitude: the recovery controller resists stronger perturbations than the dynamic-gaits motion controller.This robustness result is reported from the evaluation in Fig. 13.
  • Recovery Control: The recovery objective maximizes stability margin while penalizing foot slippage and energy consumption, with velocity tracking included for directed recovery motion.Additional terms penalize excessive joint velocity, joint acceleration, and foot acceleration.

3) Episode Termination Criteria:

Training uses fast physics simulation, many procedurally generated terrains, actuator modeling, terrain and dynamics randomization, and limited external perturbations. These choices target robust policies while acknowledging constraints in terrain generation and controller robustness.

  • Policy Training: The recovery controller uses the same termination criteria as the domain adaptive tracker and is trained with PPO using a Gaussian policy.The recovery state includes terrain information alongside robot and motion variables.
  • Training Setup: The policies are trained in RaiSim, whose fast contact dynamics and online heightmap reloading support terrain-aware simulation.The setup uses 10k terrains to promote generalization across terrain conditions.
  • Training Setup: An actuator network trained by supervised learning approximates physical-system joint torques to reduce the simulation-to-real gap.Its inputs include PD gains because the controller uses different gains across leg contact states.
  • Training Setup: The terrain-generation tool omits object collision detection, producing overlapping objects and some locally impassable height deviations.These artifacts constrain the realism and traversability of generated terrains.
  • Training Setup: Domain randomization varies system properties within limits for which the motion controller remains robust, complementing terrain randomization.The footstep planner is trained to adapt to controller behavior under changed dynamic properties such as added mass.
  • Training Setup: External forces are sampled from N(0,10) N, clipped to [−30,30] N, and applied for durations from 1 to 4 s.Gait randomization selects trot with probability 0.7, crawl with 0.2, and amble and pace with 0.05 each.

5) Reinforcement Learning:

The footstep planning policy is trained through guided learning and SAC-based exploration, with increasing terrain complexity and domain randomization to improve robustness and transferability.

  • Training procedure: The training alternates interactive guided learning with SAC policy optimization for footstep planning.Guided learning uses expert dynamic-gait samples, while SAC explores uneven terrain and learns footholds that avoid edges and slopes.
  • Training procedure: Terrain difficulty increases during training by scaling the maximum terrain elevation up to hmax = 2 m.The curriculum updates terrain elevation as policy iterations progress.
  • Training procedure: A second training session uses SAC without guided learning to improve robustness to modeled system-dynamics changes and support deployment on ANYmal C.The same footstep planner trained with ANYmal B is used for ANYmal C without retraining the planner from scratch.
  • Deployment: The footstep planner uses robot observations available on the physical platforms, while elevation-map processing runs in parallel with the 400 Hz motion controller.Deployment observes an approximately 35 ms elevation-map extraction and encoding delay.
  • Domain adaptation: The domain adaptive tracking policy is trained using randomized parameters and curriculum expansion of their sampling limits.The policy is trained with the terrain generator, actuator network, and Session 1 footstep planner.

2) Reinforcement Learning:

The ancillary policies use constrained reinforcement learning to improve tracking and recovery under terrain variation, perturbations, and changing robot states.

  • Domain adaptive tracking: TD3 trains the domain adaptive tracking policy with constrained exploration to avoid excessive corrective torques.The corrective action is scaled before being added to the whole-body controller torque.
  • Recovery control: The recovery policy is trained with PPO and external perturbations to stabilize the robot and recover its state.Training transitions from reference velocity tracking to recovery control.
  • Recovery control: The deterministic recovery policy maps its state to desired joint positions, which are forwarded to the actuator.The resulting effective actuation torques are generated through the actuator model.
  • Deployment: The recovery controller runs at 400 Hz and deactivates the footstep planner, motion controller, and domain adaptive tracker while active.A forward pass through the recovery policy requires approximately 42 µs.
  • Evaluation: Footstep planning is evaluated over 1000 runs for each planner, gait, and terrain configuration.The comparison includes an RL planner, a perceptive baseline, and a blind baseline across stairs, bricks, wave, and unstructured terrain.
  • Footstep-planning results: The RL planner favors wider, more level foot placement, with brick-terrain mean foot-height deviation of 0.057 m.It fails when brick-terrain maximum elevation exceeds 0.12 m, corresponding to a maximum foot-height deviation of 0.24 m.

B. Domain Adaptive Tracking

Domain adaptive tracking expands the range of physical-parameter variation handled by the whole-body controller, including transfer to ANYmal C without retraining.

  • ANYmal B: Base-mass scaling beyond [0.94,1.06] destabilizes the motion controller, whereas DAT supports scaling over [0.84,1.17] without increased oscillation amplitude.The unadapted controller exhibits increased oscillations outside its narrower mass range.
  • ANYmal C: For ANYmal C, DAT extends stable base-mass scaling from [0.91,1.10] to [0.81,1.19] without retraining.The test tracks a forward velocity command of 0.3 m/s.
  • ANYmal C: On ANYmal C using ANYmal B description parameters, DAT tracks randomly generated velocity commands but produces limb oscillations.Stable tracking fails beyond ±0.4 m/s heading velocity, ±0.3 m/s lateral velocity, and ±0.5 rad/s yaw rate.
  • Corrective actions: DAT generates corrective joint torques for base-mass and link-length scaling during flat-ground locomotion.Figure 12 reports the mean and standard deviation of the absolute corrective torques.

C. Recovery Controller

The recovery controller improves robustness to perturbations and terrain failures, while RLOC transfers across simulators, terrains, and ANYmal platforms with clear limits on steep inclines and perception quality.

  • Recovery evaluation: Recovery-control experiments estimate stabilization success across external-force magnitude and duration along heading and lateral axes.The results use kernel-density estimates of recovery rate and success probability.
  • Velocity tracking: DAT improves velocity tracking over even and uneven terrain, while removing DAT produces performance similar to the motion controller.Tracking performance is evaluated across terrain types and in Gazebo with different actuation dynamics.
  • Recovery evaluation: Adding the recovery controller improves success rates on brick and wave terrains during trotting.Table IX reports success rate, recovery rate, and the number of episodes in which recovery was activated.
  • Real-robot evaluation: On real ANYmal B, elevation-map distortion from state-estimation drift, sensor noise, and occluded-region interpolation sometimes shifts footholds toward brick edges.The distortion is illustrated during brick-terrain experiments.
  • Platform transfer: RLOC transfers to real ANYmal C for indoor and outdoor locomotion with only robot-parameter and impedance-gain changes.The robot traverses grassy, muddy, and sloped terrain, with higher torque limits aiding steeper-slope ascent.
  • Limitations: Heading-velocity tracking fails on steep staircases beyond approximately π/4 rad base pitch for ANYmal B and π/3 rad for ANYmal C.The limitation is attributed partly to joint-torque limits and a ZMP-based center-of-mass planner assuming constant base height and attitude.
  • Overall results: Brick-terrain success increases from 71.5% to 90.2% versus the baseline perceptive controller, with at least 90% success over 4 m at approximately 0.25 m/s on the real robot.The hybrid approach achieves this despite a model-based controller tuned primarily for quasi-flat terrain.

APPENDIX

The motion controller adapts foothold planning, contact scheduling, terrain geometry, and center-of-mass optimization for uneven terrain. The RL footstep planner replaces the conventional foothold optimizer while terrain-aware checks and conservative planning support robust execution.

  • Motion controller: The dynamic-gaits motion controller generates center-of-mass plans through nonlinear SQP optimization over a finite horizon.
  • Motion controller: The adapted controller fits the terrain plane through current and planned footholds and optimizes base height instead of using a fixed reference.It also uses ZMP constraints and a path regularizer to limit base drift relative to reference footholds.
  • Contact scheduling: The contact scheduler determines lift-off and touchdown timings, supports smooth gait transitions, and can slow gait execution when touchdown is delayed.
  • Foothold planning: The RL footstep planner replaces the conventional foothold optimizer and generates desired touchdown positions for locomotion over uneven terrain.
  • Terrain adaptation: Terrain-aware footholds use elevation-map terrain normals, swing-foot clearance based on local maximum elevation, and checks that fall back to vertical friction-cone estimates when terrain data is unreliable.Clearance includes a 0.05 m offset and is clipped to limit uncertainty from sensor noise and occlusions.
  • Implementation: Training uses a fixed 50 Hz optimization frequency, while deployment returns to the original variable optimization frequency.The fixed frequency makes planning independent of computation time and supports parallel simulation during RL training.

B. Stability Margin

The stability margin measures how far the instantaneous capture point lies from the feasible-region boundary and is incorporated into optimization and RL rewards. Maximizing it favors wider, less aggressive motions and improves the resulting margin.

  • Definition: The stability margin is the distance between the instantaneous capture point and the edges of the feasible region.Positive values indicate a stable state, while negative values indicate instability; bounded negative values can still permit recovery.
  • Reward design: The stability-margin reward discourages unnecessary linear and angular accelerations while encouraging timely, well-placed footholds and larger support regions.
  • Validation: 0.0051 m mean absolute prediction error and 0.0018 m standard deviation were observed on 50k test samples for the stability-margin network.The network also supplies partial derivatives with respect to robot states.
  • Trajectory optimization: 0.025 m average stability-margin improvement was obtained when TOWR trajectories optimized the learned stability-margin cost.The improvement occurred during both triple- and quadruple-stance phases, with reduced unnecessary base motion and a larger stance.
  • RL behavior: Increasing the stability-margin reward coefficient made the RL footstep planner prefer wider stances, especially during unstable motion phases when nominal-foot penalties were increased.

D. Elevation Map Encoding

The elevation-map encoder is designed to retain terrain information within the robot’s control-time budget, while the training pipeline models actuation and selects RL algorithms for different auxiliary policies. Recovery control is activated when robot-state thresholds are exceeded.

  • Elevation-map encoding: The encoder must capture terrain information while keeping its real-robot forward pass below the 2.5 ms control-step time.
  • Elevation-map encoding: 1.7 ms is required for an encoder forward pass on the real robots, leaving time for state updates and the RL planning and control networks.Reducing the latent representation below 96 degraded reconstruction of unstructured terrain.
  • Training objective: No additional regularization is needed because reconstructing from noisy inputs prevents the autoencoder from memorizing the training dataset.The encoder instead maps inputs to a lower-dimensional manifold of relevant terrain features.
  • Recovery control: The recovery controller replaces planning, motion control, and tracking when robot-state parameters exceed activation thresholds, then returns control after stabilizing the robot.
  • Terrain generation: Terrain training uses staircases, planks, bricks, unstructured sub-terrain, and wave sub-terrain represented as elevation maps.
  • RL training: SAC was selected for footstep planning because preliminary experiments required approximately one-fourth as many policy iterations as PPO for similar training parameters.
  • RL training: TD3 replaced SAC and PPO for domain-adaptive tracking because their stochastic exploration required precise tuning and could produce unstable tracking behavior.The replacement used uncorrelated Gaussian-noise exploration with a curriculum that increased noise during training.
  • RL training: PPO was used for recovery control because prior work supported reproducible training steps and policy updates were faster than with SAC or TD3.
Loading 2012.03094v3…