Source-linked AI summary
DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learning
Vassilios Tsounis, Mitja Alge, Joonho Lee, Farbod Farshidian, Marco Hutter
TL;DR
Quadrupedal locomotion over difficult non-flat terrain requires planning footholds and maintaining balance despite terrain geometry that is hard to model. DeepGait independently trains planner and controller policies using DRL, replacing physics-based MDP transitions with LP-based dynamic-feasibility checks. The resulting policies succeed across challenging simulated terrain scenarios, including a 40 cm gap that the comparison method cannot traverse.
Problem
Autonomous quadrupeds must select appropriate footholds and maintain balance in complex rigid non-flat terrain.
Method
The method independently trains neural-network gait-planner and gait-controller policies with DRL, using LP-based transition-feasibility checks instead of direct physics interaction.
Results
Only DeepGait’s policies traversed the 40 cm gap in comparison trials against Free-Gait.
Takeaways & Limitations
The approach supports terrain-aware quadrupedal locomotion across challenging simulated scenarios while combining foothold planning with base-motion execution.
Abstract
from arXiv · showhide
This paper addresses the problem of legged locomotion in non-flat terrain. As legged robots such as quadrupeds are to be deployed in terrains with geometries which are difficult to model and predict, the need arises to equip them with the capability to generalize well to unforeseen situations. In this work, we propose a novel technique for training neural-network policies for terrain-aware locomotion, which combines state-of-the-art methods for model-based motion planning and reinforcement learning. Our approach is centered on formulating Markov decision processes using the evaluation of dynamic feasibility criteria in place of physical simulation. We thus employ policy-gradient methods to independently train policies which respectively plan and execute foothold and base motions in 3D environments using both proprioceptive and exteroceptive measurements. We apply our method within a challenging suite of simulated terrain scenarios which contain features such as narrow bridges, gaps and stepping-stones, and train policies which succeed in locomoting effectively in all cases.
I. INTRODUCTION
The paper targets autonomous quadrupedal locomotion over complex rigid terrain, combining model-based feasibility checks with independently trained learning policies for planning and control.
- Non-flat locomotion requires selecting terrain-appropriate footholds while maintaining balance during multi-contact motion.
- Existing model-based methods can be computationally intensive online, while kinostatic approaches simplify terrain or decouple foothold selection from motion optimization.
- End-to-end DRL introduces challenges in suppressing undesirable emergent behavior and reducing sample complexity.
- The proposed method combines a terrain-aware planner with a foothold and base-motion controller, both implemented as neural-network stochastic policies optimized with DRL.
- Trajectory-optimization feasibility checks allow planner and controller policies to be trained independently, reducing their interaction with physics and lowering sample complexity.
- Experiments in challenging simulated locomotion scenarios show terrain generalization, reference-foothold tracking, balance maintenance, and comparison against a model-based approach.
II. PRELIMINARIES
The paper formulates locomotion decisions as a discounted MDP and represents quadrupedal motion through robot states, terrain sensing, and timed support phases.
- The locomotion problem is modeled as a discounted infinite-horizon MDP with states, actions, transition dynamics, rewards, initial-state distribution, and discount factor γ.
- The robot has an unactuated floating base, four articulated legs, actuated joints, and joint-torque control.
- Terrain elevation is measured locally through a robocentric mapping with 4 cm resolution.
- A gait is parameterized as a sequence of support phases containing base pose and velocity, foot positions, contact states, and phase timing variables.
III. METHODOLOGY
The control architecture is a two-level hierarchy in which a slower terrain-aware planner proposes support phases and a faster controller converts them into joint references.
- A high-level Gait Planner runs at roughly 2Hz using exteroceptive and proprioceptive measurements to generate finite support-phase plans.
- A low-level Gait Controller runs at 100Hz, uses proprioceptive sensing and the phase plan, and outputs joint-position references.
A. Gait Planning
Gait planning uses a neural policy to propose support-phase transitions, while an LP-based feasibility criterion checks terrain, contact, and kinodynamic constraints without direct physics interaction.
- A. Gait Planning: The planner observes robot state and local height-map measurements, then generates a distribution over candidate support-phase transitions.
- A. Gait Planning: Support phases encode contact timing through tE and tS around the base motion associated with phase Φs.
- A. Gait Planning: Transition feasibility is evaluated with a convex LP based on CROC, using centroidal dynamics, contact constraints, foot-motion limits, and force restrictions.
- A. Gait Planning: The MDP terminates transitions when foothold obstacles or gaps, base collisions, or CROC feasibility checks indicate failure.
- A. Gait Planning: The planner directly proposes phase transitions, while the MDP checks their feasibility and otherwise ends the episode.
- A. Gait Planning: The reward encourages goal-directed footholds, goal-facing orientation, low kinematic effort, and foot lifting to avoid prolonged stance phases.
- A. Gait Planning: Multiplicative rewards provide automatic scaling that preserves useful gradients while retaining exploration during training.
- A. Gait Planning: The planner policy is a Gaussian neural-network policy using exteroceptive and proprioceptive inputs and trained with a PPO variant using clipped loss and GAE.
B. Gait Control
The gait controller executes planner-generated support phases by tracking foothold and base targets while maintaining balance. Its reinforcement-learning formulation uses physics-based transitions and rewards that emphasize tracking and recovery.
- Controller role: The GC executes the GP’s support-phase sequence by tracking foothold and base positions while maintaining balance.Training can use any feasible target phase sequence; the GP is used only for convenience in this work.
- Target extraction: The GP’s rolled-out phase plan supplies target base positions, contact states, and foothold positions for all legs.Foothold references look ahead in the phase plan so swing and stance legs have valid targets.
- Controller MDP: The controller MDP incorporates system physics, joint-space PD control, and zero-order holding between 100Hz policy outputs and 400Hz torque commands.Termination occurs when the base attitude exceeds 60° or the base contacts terrain.
- Reward design: The controller reward combines foothold and contact tracking with penalties for foot slip, swing-leg velocity, torque, base motion, and excessive attitude.The formulation explicitly emphasizes target foothold positions and contact states while inhibiting aggressive motions.
- Policy representation: The GC policy is a Gaussian distribution whose mean comes from a two-layer tanh network, while its standard deviations come from observation-independent parameters.The architecture is shown in Fig. 3(b).
A. Experimental Setup
The experiments use simulated terrains ranging from a flat baseline to randomly elevated stairs and a composite ascent course. Separate software environments support the planner and controller evaluations.
- Terrain scenarios: The evaluation suite includes Flat-World as a baseline, Random-Stairs with randomly elevated 1 × 1 m2 regions, and Temple-Ascent with gaps, stepping stones, stairs, and flat regions.Random-Stairs covers a 20 × 20 m2 area with an effective diagonal inclination.
- Simulation environments: The GP environment uses a custom CROC implementation in C++, whereas the GC environment uses the RaiSim multi-body physics engine.These environments realize the planner and controller MDPs separately.
- Implementation: The deep-reinforcement-learning algorithms are implemented using the TensorFlow C/C++ API.
B. Gait Planner
The Gait Planner learns terrain-aware support-phase sequences using randomized training and feasibility-based transitions, achieving near-perfect goal-reaching while adapting gait structure to terrain features.
- The planner is evaluated with Episodic Success Rate, counting episodes that reach within 0.5 m of the goal before the maximum duration.
- Randomization of initial states is crucial because policies often fail to generalize to unseen terrain when training does not expose them to all terrain aspects.GP training required approximately 82 hours in each terrain scenario.
- The conservative centroidal dynamics model and restriction to three- and four-contact states limit the transitions the planner can generate.The contact-state restriction reduces problem complexity, while extension to two- and single-contact configurations remains future work.
- Average ESR nears 100.0% when trained GP policies generate valid support-phase sequences that guide the robot to goals with their respective controllers.
- The planner produces mostly cyclic support phases on flat regions but switches to acyclic phases on stepping-stones, gaps, bridges, and sharp point-turns.
- 18-fold effective sample-complexity reduction is estimated by replacing physics simulation with transition-feasibility evaluation during training.Episodes averaged 130 s of simulated time, while the physics simulator would require approximately 24,000 steps for that duration.
C. Gait Controller
The Gait Controller is trained to track randomized foothold targets in planar and three-dimensional terrain, with performance assessed by tracking accuracy and foothold success.
- The controller is trained on randomized initial states and target footholds, including randomized foot positions and terrain-consistent target heights.
- Foothold Tracking Error Rate measures mean tracking error over an episode using desired contacts, desired footholds, and measured contacting-foot positions.
- Foothold Tracking Score is the ratio of successfully tracked footholds, counting contacts within 5 cm of each target in the xy plane.
- 58 hours of training were required in Flat-World, compared with approximately 116 hours in Random-Stairs and Temple-Ascent.The longer duration was attributed to increased physics-engine contact-evaluation costs for terrain meshes and the multibody system.
- Training in Random-Stairs and Temple-Ascent extended foothold tracking to 3D, but the stair-climbing agent had worse MER and FTS than the Flat-World-trained agent.The paper attributes this difference to difficult valid-state sampling and repetitive foothold targets.
- The controller was evaluated across Temple-Ascent terrain scenarios and test-time perturbations including a 25% base-mass increase and ±10% shank-length changes.ESR, FTS, and FTER were recorded over five runs of 100 episodes each.
D. Comparison to Existing Approaches
The paper contrasts online kinodynamic planning with existing approaches that are either computationally intensive, kinostatic, or dependent on terrain modeling. In a gap-crossing experiment, only the proposed policies traversed the 40 cm gap.
- Existing approaches: Optimization-based methods can be kinodynamic but are not feasible online, while sampling-based methods remain kinostatic.Kinostatic methods cannot fully exploit system dynamics and often decouple foothold selection from base and foot-motion optimization.
- Proposed approach: The proposed approach uses a minimal height-map representation for online, gait-free kinodynamic planning that jointly selects footholds and optimizes base and swing-foot motions.This contrasts with methods requiring terrain modeling or qualification and with approaches that separate foothold selection from motion optimization.
- Gap-crossing comparison: The evaluation compares the proposed method with Free-Gait on trials involving 30 cm and 40 cm gaps.Free-Gait jointly optimizes body poses and footholds using a Sequential Quadratic Programming solver.
- Gap-crossing comparison: Only the proposed policies traversed the 40 cm gap.The planner identified valid footholds across the gap, while the controller oriented the base to extend leg reach while retaining balance.
V. CONCLUSION & DISCUSSION
The paper presents a two-layer hierarchy of independently trained neural-network policies for terrain-aware quadrupedal locomotion. It combines model-free deep reinforcement learning with model-based feasibility optimization to plan footholds and execute motions without relying on physical simulation for gait-planner transitions.
- Conclusion & Discussion: The method decomposes terrain-aware locomotion into two independently trained model-free DRL policies connected through a parameterization of quadrupedal gaits.The hierarchy separates gait planning from gait execution.
- Conclusion & Discussion: The gait planner uses convex linear-program feasibility criteria to model MDP transitions and reduce training sample complexity without direct physics interaction.The feasibility criteria encode kinodynamic, robot-limit, and contact constraints for proposed phase transitions.
- Conclusion & Discussion: The upper-level DRL policy selects footholds and instantaneous center-of-mass motion, while the lower-level model-based LP optimizes transitions between support phases.The LP produces center-of-mass trajectories between support phases proposed by the higher-level policy.