Source-linked AI summary
Learning Quadrupedal Locomotion over Challenging Terrain
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, Marco Hutter
TL;DR
Quadrupedal robots have lacked robust dynamic locomotion across diverse, challenging natural environments. This paper trains a proprioceptive neural controller in simulation and shows zero-shot robustness on physical robots across deformable terrain, dynamic footholds, and overground impediments.
Problem
Published controllers had not demonstrated dynamic locomotion across diverse challenging natural environments, where robots often suffer slippage, imbalance, and failure.
Method
The controller uses a temporal convolutional network over proprioceptive history and an adaptive terrain curriculum for reinforcement-learning training in simulation.
Results
The controller generalized zero-shot from rigid simulated terrain to physical environments including mud, snow, rubble, vegetation, and gushing water.
Takeaways & Limitations
The results indicate that training on simpler rigid simulated terrains can produce robust locomotion across diverse field conditions.
Takeaways & Limitations
Because locomotion is blind and proprioceptive, the controller cannot detect hazards such as cliffs and therefore adopts a fairly conservative gait.
Abstract
from arXiv · showhide
Some of the most challenging environments on our planet are accessible to quadrupedal animals but remain out of reach for autonomous machines. Legged locomotion can dramatically expand the operational domains of robotics. However, conventional controllers for legged locomotion are based on elaborate state machines that explicitly trigger the execution of motion primitives and reflexes. These designs have escalated in complexity while falling short of the generality and robustness of animal locomotion. Here we present a radically robust controller for legged locomotion in challenging natural environments. We present a novel solution to incorporating proprioceptive feedback in locomotion control and demonstrate remarkable zero-shot generalization from simulation to natural environments. The controller is trained by reinforcement learning in simulation. It is based on a neural network that acts on a stream of proprioceptive signals. The trained controller has taken two generations of quadrupedal ANYmal robots to a variety of natural environments that are beyond the reach of prior published work in legged locomotion. The controller retains its robustness under conditions that have never been encountered during training: deformable terrain such as mud and snow, dynamic footholds such as rubble, and overground impediments such as thick vegetation and gushing water. The presented work opens new frontiers for robotics and indicates that radical robustness in natural environments can be achieved by training in much simpler domains.
1. INTRODUCTION
Quadrupedal locomotion could extend robotics into challenging natural environments where conventional controllers fail, but robust proprioceptive control remains difficult. The paper presents a simulation-trained controller using temporal sequence modeling, privileged learning, and adaptive terrain curricula to achieve zero-shot robustness.
- Motivation: Quadrupedal robots can access challenging terrain beyond wheeled and tracked machines by selecting safe footholds and rapidly adapting their kinematic state.Wheeled and tracked machines can become unstable on challenging terrain, while quadrupedal animals reach remote areas.
- Challenge: Existing controllers often fail on irregular, deformable, slippery, and obstructed terrain, where robots experience foot slippage, balance loss, and catastrophic failure.Successful control requires high-resolution proprioception and rapid whole-body responses to unexpected contact, deformation, and slipping.
- Contribution: Prior reinforcement-learning approaches largely remained confined to laboratory settings, motivating a radically robust controller for blind quadrupedal locomotion.Earlier work focused mainly on flat or moderately textured surfaces, including prior end-to-end learning and recovery demonstrations on flat ground.
- Method: The approach adds a temporal convolutional network, privileged teacher-student learning, and an automated curriculum that adaptively synthesizes terrains during training.These ingredients address limitations of snapshot-based MLP policies, sparse supervision on rough terrain, and the need to increase terrain difficulty as the controller improves.
- Result: Despite simulation training on rigid terrains and limited procedural profiles, the controller remains effective in zero-shot tests on previously unseen physical conditions.The cited training terrains include hills and steps, while deployment evaluates conditions never encountered during training.
2. RESULTS
The controller was deployed on two generations of ANYmal robots across challenging natural, indoor, and underground environments. It maintained robust locomotion against disturbances, unstable terrain, payload-induced model mismatch, and slippery surfaces, outperforming or surpassing the baseline in reported evaluations.
- Robot deployments: The trained controller was deployed on ANYmal-B and ANYmal-C robots, which differ in kinematics, inertia, and actuators.The deployments are summarized in Movie 1 and Figures 2–3.
- Natural environments: The controller traversed steep mountain trails, running-water creeks, mud, thick vegetation, loose rubble, snow-covered hills, and damp forest despite these terrains being absent from training.These environments deform, crumble, and disturb the legs through vegetation, rubble, and sticky mud.
- Natural environments: The controller never failed in forest experiments, whereas the baseline frequently failed on loose branches, thick vegetation, and mud.The baseline traversed only flat, unobstructed patches.
- Quantitative comparison: The presented controller achieved higher locomotion speed than the baseline in moss, mud, and vegetation, with performance compared using mechanical cost of transport.The evaluation measured locomotion speed and energy efficiency.
- Field deployment: Two ANYmal-B robots completed four 60-minute DARPA Subterranean Challenge missions with a zero failure rate, including traversal of a steep staircase.The controller replaced the team’s previous model-based controller and operated without physical assistance.
- Robustness tests: The controller stably locomoted over shifting debris, developed a proprioceptive foot-trapping reflex, traversed 13.4 cm steps with a 10 kg payload, and adapted to slippery terrain.The payload was 22.7 % of the robot’s total weight and was never simulated during training; the baseline traversed no steps with it.
3. DISCUSSION
The work shows that robust locomotion can be trained in a simplified rigid-terrain simulation and deployed across diverse field conditions. The controller remains limited by its trot-only gait and reliance on proprioception, while the methodology broadens legged robotics’ deployment and training possibilities.
- Discussion: Controllers trained only on rigid terrain successfully handled diverse deployment conditions despite lacking simulated compliance and overground obstructions.The training environment included no compliance or vegetation, yet controllers met the diversity of field conditions encountered during deployment.
- Discussion: The methodology suggests that simulating the astonishing variety of the physical world may not be necessary for robust field deployment.This challenges the hypothesis that simulation’s inability to represent physical complexity fundamentally constrains training.
- Discussion: The controller exhibits only the trot gait, narrower than the range of gait patterns discovered by quadrupeds in nature.The authors hypothesize that training protocols and objectives emphasizing diversity could elicit additional gaits.
- Discussion: The controller relies solely on proprioception, reducing assumptions about the sensor suite and vulnerability when exteroception fails.Blind locomotion remains inherently limited, including the inability to avoid walking off a cliff when commanded to do so.
- Discussion: The results expedite deployment of legged machines in environments beyond wheeled and tracked robots’ reach and dangerous or inaccessible to humans.The methodology also supports training complex robotic systems in simulation before deployment in the physical world’s full richness and complexity.
4. MATERIALS AND METHODS · A. Overview · Motion synthesis
The method trains a privileged terrain-aware teacher in simulation and distills it into a proprioceptive student for direct physical deployment. Its controller synthesizes periodic leg motions from proprioceptive inputs, modulates trajectory-generator outputs, and tracks foot targets through analytic inverse kinematics and joint-position control.
- A. Overview: The controller receives a command and proprioceptive measurements, without cameras, depth sensors, haptic sensors, or handcrafted terrain features.Inputs include base velocity, orientation, and joint states.
- A. Overview: The command specifies horizontal movement direction and turning direction rather than a target base speed, whose feasible range varies across challenging terrain.The text gives downhill versus uphill walking speed as an example.
- A. Overview: The two-stage training process first optimizes a terrain-informed teacher with reinforcement learning in simulation, then trains a proprioceptive student by imitation.The teacher accesses terrain properties and contacts; the student receives a sequence of N proprioceptive observations and is deployed on physical machines.
- A. Overview: Training uses procedurally generated terrains whose difficulty is adapted to policy skill through a traversability measure, sampling, and particle filtering.The terrain-parameter distribution is maintained to select appropriate difficulty during training.
- A. Overview: The PMTG architecture supplies motion-generation priors while the neural policy modulates leg phases and motion primitives through residual position commands.This architecture is used to synthesize residual commands rather than replace trajectory generation entirely.
- Motion synthesis: Each leg uses a periodic phase variable φ_i ∈[0.0, 2π), with contact for φ_i ∈[0.0, π) and swing for φ_i ∈[π, 2π).The phase evolves from an initial phase using a common base frequency and a leg-specific frequency offset.
- Motion synthesis: The controller expresses foot targets in horizontal foot frames, converts them to joint targets using analytic inverse kinematics, and tracks them with joint-position PD controllers.Analytic inverse kinematics is selected for computational efficiency and reuse of position-control actuator models in sim-to-real transfer.
- Motion synthesis: Motion generation comprises four identical foot trajectory generators, with the policy outputting leg-frequency offsets and target-foot-position residuals added to each generator’s output.Each generator maps leg phase φ to a three-dimensional foot-position target.
Teacher policy · Student policy
The teacher policy formulates locomotion as an MDP using privileged terrain and contact information, while the student policy reconstructs teacher representations and actions from proprioceptive history. Teacher actions are 16-dimensional, and student training uses a temporal convolutional encoder with supervised DAgger data aggregation.
- Teacher policy: Teacher training models locomotion as a Markov Decision Process with state, action, transition probability, and reward components.The teacher assumes full environmental observability and uses reinforcement learning to solve the control problem.
- Teacher policy: The teacher state combines robot measurements ot with privileged information xt that is generally unavailable in the real world.ot includes commands, orientation, base twist, joint states, foot-related signals, and short-term joint histories.
- Teacher policy: 16-dimensional teacher actions contain leg frequencies and foot position residuals.The action vector is denoted ¯at.
- Teacher policy: The teacher reward favors faster advancement toward the goal, and its policy is trained with TRPO.Reward details and training hyperparameters are provided in the supplementary sections and tables.
- Teacher policy: Two MLP blocks encode xt into terrain- and contact-related latent features before combining them with ot to compute actions.The latent representation is hypothesized to support adaptive behaviors such as terrain-dependent foot clearance.
- Student policy: The student policy observes only ot and is hypothesized to recover teacher latent features from a time series of proprioceptive observations ht.ht excludes fo, joint history, and previous foot position targets.
- Student policy: A TCN encoder processes H = {ht−1, ..., ht−N−1} using three dilated causal convolutional layers and strided layers that reduce dimensionality.The architecture supports controllable and potentially long input histories and is intended to be robust to hyperparameter settings.
- Student policy: The student is trained by supervised learning to match teacher actions and latent features, using DAgger trajectories for data aggregation.The teacher supplies target outputs for states visited during student-policy rollouts, with the loss defined over action and latent-feature errors.
Adaptive terrain curriculum
The method adaptively trains a single generalist policy by modifying terrain-parameter distributions according to terrain traversability. A particle filter targets terrains with mid-range traversability, promoting continuous skill improvement and generalization to nearby environments without separate evaluation steps.
- Curriculum objective: The curriculum gradually modifies environmental-parameter distributions so one policy can continuously improve locomotion skills and generalize to new environments.Unlike POET’s evolving population of specialized agents, the method seeks a single generalist agent.
- Particle-filter curriculum: A particle filter tracks a distribution of high-desirability terrain parameters during training by approximating the target distribution with finite samples.The target distribution consists of terrain parameters satisfying Tr(cT, π) ∈[0.5, 0.9].
- Terrain evaluation: Terrain difficulty is evaluated by traversability—the success rate of traversing generated terrains—instead of the multi-objective reward function.The authors consider traversability more intuitive because the reward combines multiple objectives that are often unbounded.
- Terrain evaluation: The terrain generator targets parameter settings with traversability Tr(cT, π) ∈[0.5, 0.9], synthesizing terrains that are neither too easy nor too difficult.The fixed thresholds 0.5 and 0.9 define minimum and maximum traversability.
- Particle-filter curriculum: The same trajectories used to estimate measurement probabilities also train the policy, so advancing the terrain curriculum requires no additional evaluation steps.Resampling follows normalized importance weights, while terrain parameters evolve through a bounded random walk in parameter space.
Validation of the method
Ablations show that proprioceptive memory, privileged training, and adaptive terrain curricula are important for robust locomotion. Longer memory improves step traversal and disturbance resistance, while removing privileged training or adaptive curricula substantially degrades performance.
- Proprioceptive memory: Longer proprioceptive memory enables controllers to traverse higher steps, especially when hind legs encounter obstacles.Longer-memory controllers adapt hind-leg trajectories to provide higher foot clearances, whereas limited-memory controllers have particularly high failure rates.
- Proprioceptive memory: Memory length had little effect on uniform slopes but strongly affected step traversal and disturbance robustness.The tested memory range was N=1, corresponding to 20 ms, through N=100, corresponding to 2 s; N=100 was the deployed default.
- Proprioceptive memory: 35.5% lower deviation under lateral disturbance was achieved by TCN-100 than by TCN-1.Controllers were tested with a 50 N lateral force applied to the base for 5 s during straight walking.
- Privileged training: Without privileged training, the TCN-20 baseline failed diagnostic tests, including slope locomotion and step traversal.The baseline was trained directly with TRPO, using the same reward and hyperparameters as teacher training, and was compared with privileged-learning policies.
- Privileged training: The directly trained baseline did not reach rewards comparable to privileged-information teacher or privileged-learning TCN-20 policies.The comparison used the same TCN-20 architecture for the baseline and the proprioceptive policy trained via privileged learning.
- Adaptive terrain curriculum: Teachers trained without the adaptive terrain curriculum achieved significantly lower success rates on testing terrains.The baseline used randomly generated terrains uniformly sampled from C, while adaptive-curriculum training used hills, steps, and stairs.
Further analysis of emergent behavior
The proprioceptive policy encodes privileged environmental information and uses temporal proprioceptive history to adapt during a foot-trapping reflex. Decoder reconstructions and saliency analysis show that it updates terrain estimates and attends to foot-trapping measurements to execute the subsequent step-up.
- Privileged-information decoding: A decoder reconstructs privileged information from an intermediate TCN-policy layer, revealing encoded contact states, terrain shape, and external disturbances without training the policy itself.The policy parameters remain fixed during decoder training, so the decoder only probes information encoded after training.
- Foot-trapping reflex: During the foot-trapping reflex, estimated terrain elevations and normal vectors adapt to the encountered step while terrain uncertainty increases at collision and remains after stepping up.When the LF foot collides with the step, estimated elevation in front of the front legs rises and uncertainty grows; estimates then adapt during the reflex.
- Temporal proprioceptive integration: The policy uses a history of proprioceptive measurements to assess sensitivity to observations while overcoming the step.The input H ∈R60×N stacks N measurements, with history length defined as 0.02 s × N.
- Temporal proprioceptive integration: Saliency remains high at the foot-trapping event, and the policy attends to LF leg joint states measured there during the following swing phase.The policy directly accesses measurements at foot-trapping and leverages them while stepping up.
SUPPLEMENTARY MATERIALS
The supplementary materials provide implementation details, training components, terrain evaluations, student-policy analyses, and supporting algorithms.
- Supplementary Materials: The supplement includes nomenclature, implementation details, and a foot trajectory generator.These sections define terminology and describe core locomotion-control components.
- Supplementary Materials: It documents teacher-policy training through the reward function, parameterized terrains, and qualitative evaluation of the adaptive terrain curriculum.Together, these sections cover training objectives and terrain-generation or curriculum procedures.
- Supplementary Materials: It analyzes privileged-information reconstruction, the recurrent neural network student policy, and the latent-representation-loss ablation for student training.The supplement also contains Algorithm S1 for teacher training.
SUPPLEMENTARY MATERIALS
The supplementary materials detail the simulation, actuator modeling, domain randomization, procedural terrain generation, adaptive curriculum, and reward design used to train the locomotion controller.
- Simulation and training: RaiSim simulates rigid-body and contact dynamics, while robot-specific actuator networks model Series Elastic Actuators from six-dimensional histories of joint errors and velocities.The actuator input includes current and two past states at t−0.01 s and t−0.02 s.
- Simulation and training: Training randomizes disturbances, foot–terrain friction coefficients, and observation noise, with the teacher policy accessing randomized physical quantities.The randomization is intended to improve policy robustness.
- Reward design: The reward combines velocity, angular velocity, base motion, foot clearance, body collision, target smoothness, and torque terms to promote stable and efficient locomotion.Torque penalties are intended to protect actuators and reduce energy consumption, while clearance rewards encourage feet to clear surrounding terrain.
- Terrain generation: Procedural terrain generation varies terrain parameters across Hills, Steps, and Stairs environments to expose policies to slopes, slippage, elevation changes, and foot-trapping.Hills use Perlin noise and induce smooth slopes and foot slippage; Steps create discrete elevation changes and foot-trapping.
- Adaptive curriculum: The adaptive curriculum identifies desirable terrain difficulty by relating traversability and desirability, while a particle filter fits the latent distribution of desirable terrains.Undesirable terrains are either too easy or too difficult.
S7. Reconstruction of the privileged information in different situations
The decoded privileged information tracks changing terrain friction during a slippery traversal, while controller comparisons show GRU performance between TCN-20 and TCN-100 and faster TCN training than GRU.
- Privileged-information reconstruction: The estimated foot–terrain friction coefficient decreases when slipping begins, stays low on the wet whiteboard, and rises about 2 s after returning to normal ground.This reconstruction is illustrated for traversal of a wet, slippery whiteboard.
- Controller comparison: GRU-controller performance falls between TCN-20 and TCN-100 overall, matching TCN-100 on slopes but failing to match it in step experiments.The comparison uses diagnostic settings presented in Fig. 5A.
- Training efficiency: TCN training is much faster than GRU training, with computation times reported in Table S1.The TCN is identified as having the chief advantage in training efficiency.
S9. Ablation of the latent representation loss for student training
The ablation removes the student-training latent representation loss, leaving a policy that imitates the teacher’s output; performance remains comparable on uniform slopes and external disturbances but declines on steps.
- Loss ablation: The ablated baseline removes the squared-error loss on the latent vector l_t from the student policy’s training objective.The baseline uses a loss that simply imitates the teacher’s output.
- Performance: TCN-100 naive IL performs comparably in the uniform slope setting and under external disturbances.This model is a TCN-100 network trained without the latent representation loss.
- Performance: The ablated version has lower success rates on steps.The comparison is reported in Fig. S3 across models trained with different random seeds.