PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning robust perceptive locomotion for quadrupedal robots in the wild

Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, Marco Hutter

arXiv:2201.08117v1cs.RO

TL;DR

Reliable use of exteroception with proprioception for quadrupedal locomotion remains a challenge when terrain observations are incomplete and noisy. The paper integrates both modalities in a recurrent encoder and reports improved speed and obstacle traversability over [4] while maintaining robustness.

  • Problem

    Reliably integrating exteroceptive and proprioceptive perception for legged robots traversing challenging real-world terrain remains a grand challenge.

    Source: §1 · p. 1

  • Method

    An end-to-end trained recurrent encoder combines proprioceptive and exteroceptive data into an integrated belief state, falling back to proprioception when exteroception is unreliable.

    Source: §1 · p. 3

  • Results

    The controller substantially improves locomotion speed and obstacle traversability over [4] while maintaining exceptional robustness.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present the approach as a general framework for robust deployment of complex autonomous machines in the wild.

    Source: §1 · p. 3

Abstract

from arXiv · show

Legged robots that can operate autonomously in remote and hazardous environments will greatly increase opportunities for exploration into under-explored areas. Exteroceptive perception is crucial for fast and energy-efficient locomotion: perceiving the terrain before making contact with it enables planning and adaptation of the gait ahead of time to maintain speed and stability. However, utilizing exteroceptive perception robustly for locomotion has remained a grand challenge in robotics. Snow, vegetation, and water visually appear as obstacles on which the robot cannot step~-- or are missing altogether due to high reflectance. Additionally, depth perception can degrade due to difficult lighting, dust, fog, reflective or transparent surfaces, sensor occlusion, and more. For this reason, the most robust and general solutions to legged locomotion to date rely solely on proprioception. This severely limits locomotion speed, because the robot has to physically feel out the terrain before adapting its gait accordingly. Here we present a robust and general solution to integrating exteroceptive and proprioceptive perception for legged locomotion. We leverage an attention-based recurrent encoder that integrates proprioceptive and exteroceptive input. The encoder is trained end-to-end and learns to seamlessly combine the different perception modalities without resorting to heuristics. The result is a legged locomotion controller with high robustness and speed. The controller was tested in a variety of challenging natural and urban environments over multiple seasons and completed an hour-long hike in the Alps in the time recommended for human hikers.

1. INTRODUCTION

Legged robots need reliable terrain foresight to traverse challenging environments at animal-like speed, but real-world perception is incomplete and noisy. This paper presents a quadrupedal controller that integrates exteroception and proprioception to use visual foresight when reliable and fall back to robust proprioceptive locomotion when needed.

Problem

  • Depth and elevation-map perception can fail under difficult textures, lighting, reflective surfaces, vegetation, pose error, occlusion, or dynamic objects, making onboard terrain information unreliable.Conventional methods generally assume map information and its uncertainty are accurate, while existing outdoor controllers often avoid visual input or add heuristic reflex rules.Source: §1 · pp. 1, 3

Related work

  • Prior learning-based locomotion largely omitted vision, while a learning-based foothold planner with perception was limited to rigid, mostly flat terrain and depended on map quality.More broadly, prior approaches focused on foothold selection and trajectory generation from accurate terrain information, leaving diverse perception failures in the wild unresolved.Source: §1 · p. 3

Method

  • A recurrent encoder learns an integrated belief state from proprioceptive and exteroceptive inputs, using terrain foresight when reliable and falling back to proprioception when needed.Trained end-to-end without heuristics, it captures terrain information from exteroceptive observations that may be incomplete, biased, or noisy.Source: §1 · p. 3

Training and deployment

  • Privileged learning trains a teacher with ground-truth terrain information, then an imitation-trained student predicts its actions from partial, noisy observations available on the robot.The deployed policy requires no fine-tuning, uses a robot-centric elevation map, and can work with different depth sensors or continue walking when mapping fails or sensors break.Source: §1 · p. 3

Contribution

  • The approach substantially improves locomotion speed and obstacle traversability over the state of the art [4] while maintaining robustness to exteroceptive failures.The authors report extensive hardware experiments demonstrating robust traversal across varied environments.Source: §1 · p. 3

2. RESULTS

Results show robust locomotion across varied environments, a successful alpine hike, and gains from exteroception combined with fallback to proprioception when terrain maps are unreliable.

Fast and robust locomotion in the wild

  • Across alpine, forest, underground, and urban deployments, the controller had zero falls and used advance terrain perception for fast, smooth motion, including stair traversal in arbitrary orientations.It also handled snow-covered stairs despite slippery ground and erroneous depth maps, with zero failures.Source: Fast and robust locomotion in the wild · pp. 3, 5

A hike in the Alps

  • On the 2.2 km Etzel hike, ANYmal finished without failure, reaching the summit in 31 minutes versus the sign’s 35-minute estimate and completing the route in 78 versus 76 minutes.The route included inclines up to 38%, rocky and wet ground, slippery forest terrain, and tree-root obstacles; the robot finished without human help or a fall.Source: Figure 2 · p. 4, A hike in the Alps · p. 5

Exteroceptive challenges

  • Despite missing, noisy, or misleading elevation maps, the belief-state estimator assessed exteroceptive reliability, using it when informative and falling back to robust proprioceptive locomotion otherwise.Challenges included reflective surfaces and snow, overhangs and soft terrain, odometry drift, and sensor occlusion.Source: Exteroceptive challenges · p. 5

Evaluating the contribution of exteroception

  • 1.2 m/s vs 0.6 m/s on flat ground and 3 vs 0.6 rad/s turning, our controller outperformed the proprioceptive baseline and traversed steps up to 30.5 cm.It followed an obstacle-course path smoothly without assistance, while the baseline got stuck on all three obstacles and required human lifting and pushing.Source: Evaluating the contribution of… · pp. 5, 8

Evaluating robustness with belief state visualization

  • In ambiguous terrain, the recurrent controller revised terrain beliefs on contact, retained updates, adapted after detecting slip, and relied on proprioception when covered sensors or map drift made exteroception uninformative.It learned from contact with deformable foam and an unseen transparent step; with sensors covered, it successfully climbed stairs after contact and stayed balanced on descent.Source: Evaluating robustness with belief… · p. 8

3. DISCUSSION

The discussion presents a learned controller that combines exteroception’s anticipatory gait adaptation with proprioceptive robustness when sensing is unreliable. It reports successful operation in a difficult Alpine hike and DARPA subterranean missions, while identifying limits and future extensions.

3. DISCUSSION

  • The end-to-end learned controller anticipates terrain to adapt gait before contact, then smoothly relies on proprioception when exteroception is misleading, incomplete, or missing.The authors describe it as the first rough-terrain legged locomotion controller combining vision-based speed and grace with proprioceptive robustness, without hand-coded rules or heuristics.Source: §3 · p. 8
  • The robot completed an hour-long difficult Alps hike in the human-recommended time without assistance except shoe reattachment and battery swaps.The route was rated “difficult.”Source: §3 · p. 8
  • The controller was the default for team Cerberus’s DARPA Subterranean Challenge missions [50, 51], where four ANYmals explored over 1700m across three course types without a fall.The team won first prize in the finals [52]; missions involved rough terrain and degraded sensing from dust, fog, water, and smoke [53].Source: §3 · p. 8

Possible extensions

  • Explicit uncertainty estimation could help prevent stepping off occluded cliffs or stepping stones, potentially by prompting foot-probing or learned occlusion-aware behavior.The current policy uses uncertainty only implicitly and may assume a continuous surface when the elevation map lacks information behind an occlusion.Source: Possible extensions · p. 8
  • The elevation-map intermediate makes the model sensor-independent, but omits raw-sensor material and texture detail and relies on pose estimation not jointly trained with the controller.LiDAR and stereo cameras were used in different deployments without retraining or fine-tuning; integrating raw sensory processing could further improve speed and robustness.Source: Possible extensions · p. 8
  • The current controller cannot complete maneuvers far from normal walking, such as freeing a leg stuck in a narrow hole or climbing onto a high ledge.The authors identify these as locomotion-task limitations.Source: Possible extensions · p. 8

4. MATERIALS AND METHODS

The method trains a neural-network policy in simulation for zero-shot transfer to the real robot, using a three-stage pipeline whose first stage trains a teacher with reinforcement learning.

Overview

  • The neural-network policy is trained in simulation and transferred to the real robot without additional training.Source: Overview · p. 8
  • The method consists of three stages, illustrated in Figure 6.Source: Overview · p. 8
  • The teacher learns to follow random target velocities over randomly generated terrain while experiencing random disturbances.Source: Overview · p. 8

1. Teacher policy training

Training proceeds in two stages: a privileged-information teacher is trained with reinforcement learning, then a student learns from noisy observations and is deployed zero-shot on the robot.

Teacher policy training

  • The teacher policy is trained with reinforcement learning using privileged simulation information, including noiseless terrain measurements, ground friction, and introduced disturbances.The student is then distilled from the teacher and deployed zero-shot using height samples from a robot-centric elevation map.Source: Figure 5 · p. 10, §1 · p. 11

Student policy training

  • The student uses a recurrent encoder to form a belief state from non-privileged observations and is trained with behavior-cloning and state-reconstruction losses.Behavior cloning imitates the teacher’s actions, while reconstruction encourages an informative internal representation.Source: Figure 5 · p. 10, §1 · p. 11

Deployment

  • For real-world deployment, onboard depth data is integrated into an elevation map, whose sampled heights are combined with proprioceptive input to generate actuator commands.The student policy is transferred to the physical robot and deployed zero-shot using these onboard sensor inputs.Source: Figure 5 · p. 10, §1 · p. 11

Problem formulation

The control problem is modeled as a discrete-time process in which actions produce observations, state transitions, and rewards. Because terrain and external forces may be unobservable, the problem is a POMDP, with a policy trained to maximize expected discounted reward.

Problem formulation

  • Actions generate observations and rewards as the environment transitions between states under an observation model and transition probability.The policy acts at time t, receives observation o_t, and the environment transitions to s_{t+1} and returns r_{t+1}.Source: Problem formulation · p. 11
  • Unobservable external forces and full terrain information make locomotion a POMDP rather than a fully observable MDP.When observations equal states, the process is an MDP; otherwise, it is partially observable.Source: Problem formulation · p. 11
  • The RL objective is to find a policy that maximizes expected discounted reward over future trajectories.Source: Problem formulation · p. 11
  • POMDP methods construct a belief state from observation history, using stacked observations [54], RNNs [55, 56], or TCNs [4, 57].These approaches aim to capture unobserved state information.Source: Problem formulation · p. 11
  • To avoid training a sequential policy naively from scratch, the approach trains a privileged-information teacher and distills it into a student through supervised learning [45].Source: Problem formulation · p. 11

Training environment

Training uses parallel RaiSim simulations of ANYmal-C robots on randomly generated rough terrain, with an integrated actuator model. Terrain, robot dynamics, disturbances, and episode termination are designed to support training and constrain behavior.

Training environment

  • Parallel RaiSim simulations train multiple ANYmal-C robots on randomly generated rough terrain with an integrated actuator model [58, 3].Source: Training environment · p. 11

Terrain

  • Parameterized height-map terrain includes slopes, steps, and four box-built stair types, avoiding nonvertical risers that policies exploited and that harmed sim-to-real transfer.The stair types are standard, open, ledged, and random.Source: Terrain · p. 11

Domain randomization

  • Episodes randomize robot masses, initial joint and body states, external forces and torques, and occasionally lower foot friction to introduce slippage.Source: Domain randomization · p. 11

Termination

  • Episodes end after ground collision, excessive body tilt, or actuator torque-limit violations, shaping motion toward constraint-satisfying behavior.Source: Termination · p. 11

Teacher policy training

The teacher policy is trained to make ANYmal follow randomly generated velocity commands over random terrain using privileged information. Its observations, CPG-inspired actions, architecture, reward shaping, and curricula define the training setup.

Teacher policy training

  • The teacher policy targets commanded locomotion over randomly generated terrain using perfect privileged information; it is modeled as a Gaussian policy and trained with PPO.Commands specify longitudinal and lateral velocity and yaw velocity in the robot’s body frame.Source: Teacher policy training · p. 11

Observation and Action

  • Observations combine proprioceptive history, terrain height samples, and privileged contact and disturbance state; CPG-inspired actions adjust leg phase and residual joint targets.The proprioceptive history includes body and joint state, action history, and leg phase; height samples are taken around each foot at five radii.Source: Observation and Action · p. 11

Policy architecture

  • The teacher uses exteroceptive and privileged encoders to compress their inputs into latent representations, alongside a main network; the compact encoders facilitate component reuse by the student.The policy is modeled as an MLP.Source: Policy architecture · pp. 11–12

Rewards

  • Rewards encourage command-velocity tracking while penalizing orthogonal and rotational velocity, and shaping terms discourage poor posture, aggressive motion, foot slip, and leg collisions.Reward terms were tuned by evaluating simulated traversal and locomotion smoothness.Source: Rewards · p. 12

Curriculum

  • Two curricula increase difficulty as performance improves: adaptive terrain selection keeps terrain challenging but achievable [4], while logistic schedules ramp domain randomization and selected reward terms toward full magnitude [3].The scheduled reward terms include joint velocity and acceleration, orientation, slip, and thigh and shank contact.Source: Curriculum · p. 12

Student policy training

The student policy distills a privileged teacher into a robot-observable controller that uses recurrent belief estimation to combine noisy terrain perception with proprioception. Training randomizes height samples to model exteroceptive failures and uses a gated encoder to regulate terrain information.

Student policy training

  • The student distills a privileged teacher into a robot-observable policy that estimates unobservable states from sequential exteroceptive and proprioceptive inputs.Supervised training combines behavior cloning and reconstruction losses, and samples are generated by rolling out the student policy [60, 61].Source: Student policy training · p. 12

Height sample randomization

  • Gaussian height-sample noise shifts scan points or perturbs heights, with variance applied per scan point, per foot, or per episode.Scan-point and foot noise are resampled each time step, whereas episode noise remains constant across scan points.Source: Height sample randomization · p. 12
  • Three episode-level mapping conditions model nominal quality, large map offsets, and missing terrain information, sampled in a 60%, 30%, and 10% ratio.Cell-dependent height offsets simulate transitions such as vegetation to deep snow, and noise magnitude increases linearly during the training curriculum.Source: Height sample randomization · p. 12

Belief state encoder

  • A gated recurrent encoder combines proprioceptive and noisy exteroceptive features into a belief state, adaptively controlling terrain information [62, 63] [64–66].A decoder gate reconstructs privileged information and height samples during training, encouraging the belief state to capture veridical environmental information.Source: Height sample randomization · p. 13, Belief state encoder · p. 14

Deployment

The controller was deployed zero-shot on ANYmal C with either LiDAR or depth-camera sensors. A GPU-based elevation-mapping pipeline supported fast terrain mapping while the policy ran at 50 Hz.

Deployment

  • The controller ran zero-shot on ANYmal C with either two Robosense Bpearl LiDAR sensors or four Intel RealSense D435 depth cameras.A robot-centric 2.5D elevation map was built at 20 Hz; the policy ran at 50 Hz and used randomly sampled heights when map information was unavailable at a query location.Source: Deployment · p. 14
  • A GPU pipeline parallelized point-cloud processing and used Kalman-filter map updates, drift compensation, and ray casting to keep mapping rates high enough for fast locomotion.The update approach was similar to Fankhauser et al. [17].Source: Deployment · p. 14

SUPPLEMENTARY MATERIALS

The supplementary materials index sections on evaluation, training, terrain, observations and actions, architecture, rewards, noise, and belief-encoder evaluation, and define the paper’s notation.

SUPPLEMENTARY MATERIALS

  • Sections S2–S9 cover exteroception evaluation, training details, terrain generation, observations and actions, network architecture, reward function, height-sample noise, and belief-encoder evaluation.Source: SUPPLEMENTARY MATERIALS · p. 14
  • The supplementary materials list controller and encoder comparisons, PPO and student-training hyperparameters, observations, teacher–student action differences under two noise conditions, and four movies.Source: SUPPLEMENTARY MATERIALS · p. 14, Table S4 · p. 14

S1. Nomenclature

  • The nomenclature defines symbols for state, observation, belief and hidden states, latent features, motion and joint quantities, CPG variables, training losses, and privileged, target, proprioceptive, and exteroceptive quantities.Source: S1. Nomenclature · p. 17

S2. Evaluating the importance of exteroception: Additional experiments in simulation

Simulation experiments report that the controller traverses a wider range of terrain than a proprioceptive baseline, while the gated recurrent encoder uses exteroception when reliable and limits its influence when noisy.

S3. Training details

  • Teacher training used PPO with 1,000 parallel environments, while student training used 300 environments and a curriculum that gradually increased height-sample noise.The student used 400 trajectory steps per iteration and trained its RNN with truncated backpropagation through time; its loss was Lbc + 0.5 · Lre.Source: S3. Training details · p. 17

S4. Terrain generation

  • Training terrain included randomized rough surfaces, steps, boxes, grid steps, and four stair types, with terrain-specific parameters controlling heights, widths, and depths.Large steps used two height levels up to 0.4 m, while boxes reached 0.25 m maximum height.Source: S4. Terrain generation · p. 17

S5. Observation and action

  • The policy combined proprioceptive commands, joint and body state, and leg-phase information with height samples around each foot, and output phase offsets and residual joint targets.Exteroception used circular samples at five radii, from 0.08 to 0.48 m, with 6 to 16 points per ring.Source: S5. Observation and action · p. 17, Table S3 · p. 18

S6. Network architecture

  • A two-layer GRU belief encoder with an exteroceptive gate produced a 120-dimensional belief state and a 96-dimensional attention vector.The gate filters exteroceptive features in the belief encoder.Source: S6. Network architecture · p. 18

S7. Reward function

  • The reward combined command tracking with penalties for foot clearance, non-foot collisions, joint motion, torque, and foot slip.The torque penalty was included to reduce energy consumption, while the slip penalty targeted contact-foot velocity.Source: S7. Reward function · pp. 18–19

S8. Height sample noise

  • Height-sample perturbations included per-point and per-foot noise each time step, per-foot noise each episode, and intermittent large outlier noise; a condition was randomly selected at trajectory beginning and middle with probabilities 60%, 30%, and 10%.Source: S8. Height sample noise · p. 19

S9. Ablation study of attention gate in belief encoder

  • The gated encoder improved performance when exteroception was reliable and remained robust when it was noisy; GRU gate performed best in step traversal under both noise conditions.The GRU outperformed the feed-forward MLP, which failed to reconstruct privileged information; the gate improved both architectures, and the advantage over GRU without a gate was larger with small noise.Source: S9. Ablation study of attention gate… · pp. 19–20, Table S4 · p. 21, Table S5 · p. 21

S2. Evaluating the importance of exteroception: Additional experiments in simulation

A Learning curves of student training

The student-training curves favor the GRU encoder and attention gate over their alternatives; the initial rise in losses and fall in reward are attributed to the curriculum.

A Learning curves of student training

  • GRU outperformed MLP in all tested settings, and the attention gate outperformed no gate with both encoder types.The ablation compared GRU gate, GRU no gate, MLP gate, and MLP no gate.Source: Figure S2 · p. 22
  • At the beginning of training, losses increased and reward decreased because of the curriculum.Source: Figure S2 · p. 22

Related papers

Loading 2201.08117v1…