PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Terrain Generalization for Visual Locomotion Policies

Alejandro Escontrela, George Yu, Peng Xu, Atil Iscen, Jie Tan

arXiv:2011.05513v1cs.RO

TL;DR

Legged-robot controllers often generalize poorly beyond a single terrain type, limiting their usefulness across varied environments. This paper trains an end-to-end visual-locomotion policy using terrain types as multi-task reinforcement-learning tasks and raw simulated LiDAR input. The learned controller demonstrates zero-shot generalization across the tested simulated environments, including 13 distinct environments.

  • Problem

    Many locomotion policies succeed on a single terrain type with limited variations, leaving generalization to new or unseen environments as an open challenge.

    Source: §I · p. 1

  • Method

    The system treats terrain types as multi-task reinforcement-learning tasks and uses procedural terrain generation with an end-to-end policy that maps perception and proprioception to motor commands.

    Source: §I · p. 1, B. Terrain Parameterization and… · p. 3, C. Visual-Locomotion Policy Architecture · p. 3

  • Results

    The learned system demonstrates zero-shot generalization to unseen testing environments within a suite of 13 realistic simulated environments.

    Source: §I · p. 1

  • Takeaways & Limitations

    The results show that multi-task training and a strong gait prior are associated with increased generalization performance across simulated terrains.

    Source: §V · p. 6

Abstract

from arXiv · show

Legged robots have unparalleled mobility on unstructured terrains. However, it remains an open challenge to design locomotion controllers that can operate in a large variety of environments. In this paper, we address this challenge of automatically learning locomotion controllers that can generalize to a diverse collection of terrains often encountered in the real world. We frame this challenge as a multi-task reinforcement learning problem and define each task as a type of terrain that the robot needs to traverse. We propose an end-to-end learning approach that makes direct use of the raw exteroceptive inputs gathered from a simulated 3D LiDAR sensor, thus circumventing the need for ground-truth heightmaps or preprocessing of perception information. As a result, the learned controller demonstrates excellent zero-shot generalization capabilities and can navigate 13 different environments, including stairs, rugged land, cluttered offices, and indoor spaces with humans.

I. INTRODUCTION

The paper targets the limited terrain generalization of locomotion policies by training a single controller across terrain tasks. Its visual-locomotion policy combines perception and locomotion, and the system demonstrates zero-shot generalization in simulated environments unseen during training.

I. INTRODUCTION

  • Single-terrain policies with limited variation leave robots less able to generalize to new or unseen environments, a crucial feature of useful locomotion controllers.Source: §I · p. 1
  • The learning system formulates locomotion as a multi-task POMDP and uses procedural terrain generation to support learning across diverse terrain tasks.The approach also introduces an end-to-end neural network architecture for perception and locomotion.Source: §I · p. 1
  • The visual-locomotion policy takes LiDAR and proprioceptive information as input and outputs low-level motor commands.It embeds PMTG to generate cyclic, smooth actuation patterns and facilitate robust locomotion-policy learning.Source: §I · p. 1
  • The system demonstrates zero-shot generalization in eight testing environments not encountered during training, within a suite of 13 realistic simulated environments.The visual-locomotion policy parameterization is reported to perform far better than commonly used reactive policies.Source: §I · pp. 1–2

II. RELATED WORK

Prior locomotion controllers and reinforcement-learning policies have faced challenges in adapting across terrain types. Multi-task reinforcement learning offers a route to a single policy for varied tasks, which this paper applies to legged locomotion.

A. Legged Locomotion

  • Hand-engineered controllers can generalize to some degree, but adapting them to different terrains often requires expertise and manual tuning.Source: A. Legged Locomotion · p. 2
  • Prior RL policies often target one environment or variations of one terrain type; Lee et. al. achieved zero-shot real-world transfer, while this work adds exteroception for cluttered indoor navigation.The cited prior work combined ActuatorNet, PMTG, curriculum learning, and “learning by cheating.”Source: A. Legged Locomotion · p. 2

B. Multi-Task Reinforcement Learning

  • Multi-task reinforcement learning has supported single policies across diverse tasks, and this paper applies it to legged locomotion across environments.Examples include a single policy evaluated on 57 Atari games and a policy completing varied robotic manipulation tasks.Source: B. Multi-Task Reinforcement Learning · p. 2

III. METHODS

The method frames legged locomotion as multi-task reinforcement learning, with each terrain type treated as a task. Procedural terrain generation supplies diverse training environments, while an end-to-end visual-locomotion policy maps robot observations to motor commands.

III. METHODS

  • Each terrain type is a task, and procedural terrain generation creates diverse training environments for an end-to-end policy mapping exteroceptive and proprioceptive observations to motor commands.Source: §III · p. 2

A. Multi-Task Reinforcement Learning Formulation

The formulation models terrain tasks as partially observable decision processes and trains a policy over randomly sampled tasks. The policy uses onboard sensor observations and target information to select actions, with reward tied to progress toward the target.

A. Multi-Task Reinforcement Learning Formulation

  • Each terrain task is a POMDP, and the policy maximizes expected accumulated reward over an episode while training tasks are randomly sampled.Source: A. Multi-Task Reinforcement Learning… · p. 2
  • The robot’s observation combines LiDAR distances, simulated IMU and joint-encoder measurements, target distance and heading, the previous action, and trajectory-generator parameters.The policy does not receive an explicit task ID, because identifying tasks automatically in the real world is challenging.Source: A. Multi-Task Reinforcement Learning… · p. 2
  • The reward encourages movement toward a target and can be interpreted as the robot’s speed toward it; the task completes when its center of mass is within a threshold distance.Source: A. Multi-Task Reinforcement Learning… · p. 2

B. Terrain Parameterization and Procedural Task Generation

The terrain generator creates diverse training terrains from a pillar-based height field, with terrain types defined by parameter ranges. The authors note that evaluation uses a simulated robot because physical-robot access was limited.

B. Terrain Parameterization and Procedural Task Generation

  • The terrain generator builds an m×n pillar environment whose heights form a height field, providing diverse training terrains.The table summarizes parameterization and generation for selected terrain types.Source: B. Terrain Parameterization and… · pp. 2–3, Table I · p. 3
  • The authors use a simulated robot because access to the physical robot was limited during COVID-19.They aimed to make the simulation, including sensor measurements, faithful to the real robot.Source: B. Terrain Parameterization and… · p. 2
  • Each terrain type has a parameter vector that sets sampling bounds; generated terrains within a task share that terrain type.For example, rugged terrain uses height-field bounds, while stairs use step height and length.Source: B. Terrain Parameterization and… · p. 3

C. Visual-Locomotion Policy Architecture

The policy processes LiDAR and proprioceptive inputs through separate encoders, then uses PMTG to produce smooth, cyclic locomotion that can adapt to terrain.

C. Visual-Locomotion Policy Architecture

  • Separate encoders process LiDAR and proprioceptive inputs, whose learned features are joined with target information before locomotion control.PMTG generates leg trajectories, with learned gait parameters and a residual action combined into the final action.Source: C. Visual-Locomotion Policy Architecture · p. 3
  • PMTG promotes smooth, cyclic locomotion while allowing the policy to modulate gait parameters and add a residual action.The parameters include gait frequency, swing height, and stride length; the final action combines trajectory-generator output and residual action.Source: C. Visual-Locomotion Policy Architecture · p. 3
  • The architecture separates basic locomotion skills from terrain perception, allowing locomotion to adapt to surrounding environments.Source: C. Visual-Locomotion Policy Architecture · p. 3

IV. EXPERIMENTAL RESULTS

The experiments are designed to test whether the system generalizes to unseen terrains and whether LiDAR input and PMTG parameterization improve that generalization.

IV. EXPERIMENTAL RESULTS

  • The experiments assess whether the system learns visual locomotion policies that generalize zero-shot to new terrains.Source: §IV · p. 3
  • The evaluation is designed to validate generalization to terrains not encountered during training.Source: §IV · p. 3
  • A second question tests whether LiDAR input and PMTG parameterization improve generalization on unseen terrains.Source: §IV · p. 3

A. Experiment Details

Experiments use a simulated Laikago quadruped with noisy LiDAR observations and high-rate actuator and physics simulation. Policies are evaluated on unseen simulated environments using task completion rate.

A. Experiment Details

  • The simulated Laikago weighs approximately 22 kg, has 12 motors, and uses a simulated Velodyne VLP-16 LiDAR with added Gaussian measurement noise.The sensor has 16 channels and 360° horizontal and 30° vertical fields of view; distance readings are normalized and flattened.Source: A. Experiment Details · pp. 3–4
  • The policy acts at 100 Hz, while the PD controller and rigid-body dynamics run at 1 kHz.The actuator model computes ten internal 100 μs steps per action-repeat interval.Source: A. Experiment Details · p. 4
  • Training uses distributed PPO, with separate neural networks for the value function and policy.The value network has dimensions (512, 256), the policy network (256, 128), and evaluation executes the policy’s Gaussian mean.Source: A. Experiment Details · p. 4
  • Testing uses environments not encountered during training, and task completion rate measures how close the robot gets to the target relative to its starting position.A rate of 1 indicates successful navigation; a rate near zero means the robot cannot navigate across the terrain.Source: A. Experiment Details · p. 4

B. The Impact of MTRL on Generalization

The multi-task policy generalizes to unseen terrain environments, with stronger mountain-task completion than policies trained on a single terrain. The evaluation also reports successful navigation in new unseen environments.

B. The Impact of MTRL on Generalization

C. Ablation Studies

The ablations report benefits from PMTG, LiDAR input, and multi-task training rather than sequential task training. Together, the components support zero-shot generalization to new environments.

C. Ablation Studies

  • PMTG-parameterized policies perform 28%–218% better than a pure reactive locomotion component.The authors report that PMTG produces smoother actions and improves zero-shot generalization to new terrains.Source: C. Ablation Studies · p. 5
  • Removing LiDAR input reduces generalization, while rugged-terrain foot placement adapts to terrain geometry to maintain balance.The trajectory visualizations show foot Z positions for all four feet in obstacle-filled and rugged terrain.Source: C. Ablation Studies · pp. 5–6, Figure 5 · p. 6, Figure 6 · p. 6
  • Training on one task for a fixed number of steps before switching performs poorly, which the authors attribute to catastrophic forgetting.Source: C. Ablation Studies · p. 6
  • The ablations identify PMTG, exteroceptive input, and multi-task POMDP training as components of policies that generalize to new environments.Source: C. Ablation Studies · p. 6

V. CONCLUSION

The conclusion presents a multi-task reinforcement-learning system combining visual locomotion and procedural terrain generation. Simulated evaluations support improved generalization, while real-world robot evaluation remains future work.

V. CONCLUSION

  • The system combines a multi-task reinforcement-learning formulation, a visual-locomotion policy that encourages smooth actions and uses perception, and procedural terrain generation for diverse simulated training data.Source: §V · p. 6
  • Results on simulated environments show increased generalization from treating locomotion as a multi-task POMDP, with a gait prior further enhancing generalization to unseen terrains.Source: §V · p. 6
  • Evaluation on a real-world robot is planned as future work.Source: §V · p. 6
Loading 2011.05513v1…