Source-linked AI summary
Emergence of Locomotion Behaviours in Rich Environments
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller, David Silver
TL;DR
The paper addresses whether complex locomotion can emerge from simple rewards rather than carefully designed reward functions. It trains agents with scalable policy-gradient reinforcement learning across diverse terrains and obstacles, finding robust skills such as jumping, crouching, and turning. The results suggest that richer environments can produce non-trivial locomotion behaviors, although the approach is not claimed to suffice in every case.
Problem
The paper asks whether reinforcement learning agents can acquire rich locomotion behaviors without brittle, carefully engineered reward functions.
Method
Agents are trained with simple rewards in diverse, procedurally generated terrains using a scalable distributed policy-gradient algorithm and separated proprioceptive and exteroceptive policy subnetworks.
Results
Training on diverse terrain produces robust locomotion skills, including jumping, crouching, and turning, without directly incentivizing each behavior.
Takeaways & Limitations
Environmental richness can help agents develop non-trivial locomotion skills for which sensible reward design is difficult.
Takeaways & Limitations
Humanoid learning remains sensitive to the algorithm, exploration strategy, reward, termination condition, and weight initialization.
Abstract
from arXiv · showhide
The reinforcement learning paradigm allows, in principle, for complex behaviours to be learned directly from simple reward signals. In practice, however, it is common to carefully hand-design the reward function to encourage a particular solution, or to derive it from demonstration data. In this paper explore how a rich environment can help to promote the learning of complex behavior. Specifically, we train agents in diverse environmental contexts, and find that this encourages the emergence of robust behaviours that perform well across a suite of tasks. We demonstrate this principle for locomotion -- behaviours that are known for their sensitivity to the choice of reward. We train several simulated bodies on a diverse set of challenging terrains and obstacles, using a simple reward function based on forward progress. Using a novel scalable variant of policy gradient reinforcement learning, our agents learn to run, jump, crouch and turn as required by the environment without explicit reward-based guidance. A visual depiction of highlights of the learned behavior can be viewed following https://youtu.be/hx_bgoTF7bs .
1 Introduction
The paper asks whether diverse environments can produce robust locomotion from simple rewards, avoiding brittle hand-designed reward functions. It uses varied terrains and obstacles to encourage skills that generalize across challenges.
- Hand-designed locomotion rewards are brittle and can produce unexpected behavior when modified slightly.
- The paper proposes that rich, diverse environments can promote robust behaviors from simple reward functions.Different challenge levels may guide learning toward solutions that are difficult to discover in limited settings, while reducing overfitting to one setting.
- Agents are trained on procedurally generated obstacle courses for Quadruped, Planar Walker, and Humanoid bodies.Each episode presents a different instance of the task, with obstacles varying in difficulty.
- Terrain difficulty provides an implicit curriculum that supports increasingly sophisticated locomotion skills.Explicitly increasing difficulty can also improve learning speed by presenting easier obstacles before harder ones.
- The experiments use a scalable reinforcement learning algorithm to learn effectively in rich and challenging domains.The algorithm combines robust policy-gradient methods with distributed computation over many parallel instances.
- The paper reports that simple rewards yield effective locomotion, while curriculum terrain and environmental diversity improve progress and robustness.
2 Large scale reinforcement learning with Distributed PPO
The paper develops Distributed PPO for stable, scalable reinforcement learning in continuous-control environments. Across benchmark tasks, DPPO matches TRPO, scales with workers, supports recurrent networks, and is faster than the compared A3C implementation.
- Policy-gradient methods directly maximize expected reward but can suffer from high variance and hyperparameter sensitivity.
- PPO approximates TRPO using first-order gradients and an adaptive regularization term that limits policy changes.
- DPPO distributes data collection and gradient calculation across workers, with synchronous gradient averaging performing better in practice.
- DPPO uses truncated backpropagation and K-step returns to support recurrent networks with batch updates and variable-length episodes.
- The benchmark covers Planar Walker, Humanoid, and Memory Reacher continuous-control tasks with MuJoCo physics.Memory Reacher requires an RNN because the target disappears before the arm can move.
- DPPO achieves performance equivalent to TRPO, scales with worker count, supports recurrent networks, and is faster than A3C at equal worker counts.
3 Methods: environments and models
The study trains torque-controlled bodies in procedurally generated courses containing varied obstacles and difficulty levels. Policies separate body-local locomotion information from terrain-related information while using simple, consistent rewards across terrains.
- Terrain and obstacles: Courses are procedurally generated so each episode presents a new terrain instance containing hurdles, gaps, variable terrain, slalom walls, or platforms.Obstacle parameters such as height, width, steepness, unevenness, and gap distance vary across courses.
- Bodies: Three torque-controlled bodies are studied: a 9-DoF planar walker, a 12-DoF quadruped, and a 28-DoF humanoid with 21 actuated dimensions.The bodies increase in control complexity from planar to three-dimensional locomotion.
- Rewards: The reward combines forward x-axis velocity with a small torque penalty, plus body-specific pose, track-centering, or time-based terms.The walker retains pose box constraints, quadruped and humanoid policies are penalized for deviations from the track center, and the humanoid receives an additional reward per time step.
- Terrain and obstacles: Training includes single-type courses, mixtures of terrain types, mixed courses, stationary courses, and curriculum courses with gradually increasing difficulty.These settings vary whether terrain types are interleaved and whether challenge difficulty changes along the course.
- Observations: Observations comprise egocentric proprioceptive features and task-relevant exteroceptive features describing the terrain ahead.Proprioception includes body-state sensing, while exteroception provides environment information used for terrain perception and navigation.
- Policy parameterization: The policy uses separate subnetworks for proprioceptive locomotion information and exteroceptive terrain information, increasing learning speed relative to a fully connected network.The architecture is designed to separate basic locomotion skills from terrain perception and navigation.
4 Results
Across simulated bodies and challenging terrains, simple forward-progress rewards produced diverse locomotion skills, while terrain structure affected learning speed and robustness. Humanoid learning remained sensitive and variable, but successful agents acquired obstacle-navigation behaviours and showed robustness to some novel challenges.
- Simple rewards produced sophisticated locomotion across bodies and terrains, including jumping, crouching, turning, climbing, and obstacle avoidance.These behaviours emerged without separate shaping rewards for each skill.
- Planar Walker: Planar Walker policies learned robust gaits and spontaneously jumped over hurdles and gaps or crouched beneath platforms.At the end of learning, the walker jumped over hurdles nearly as tall as its own body.
- Quadruped: Quadruped policies navigated obstacles reliably, using jumping, turning, climbing, or walking around obstacles despite receiving reward only for forward motion.The policy also handled variable terrain that was poorly suited to the quadruped’s short legs.
- Terrain structure: Policies trained on gradually increasing hurdle difficulty improved faster than policies trained on stationary hurdle statistics.Learning was evaluated on both easy shallow-hurdle and hard tall-hurdle validation terrains.
- Terrain structure: Training on hurdles showed a trend toward greater robustness than flat-terrain training under unobserved changes in friction, surface, body model, and ground incline.The comparison involved both Planar Walker and Quadruped policies.
- Humanoid: Humanoid results varied substantially across random seeds, with differences in learning speed and asymptotic performance sometimes suggesting local optima.Despite this variability, several agents achieved effective gaits and moderate robustness on altered courses and perturbations.
5 Related work
The paper builds on physics-based animation and reinforcement-learning approaches to locomotion, but combines simple rewards with curriculum training and limited policy structure to produce adaptive behavior in challenging environments.
- Physics-based character animation has produced impressive locomotion and movement skills, but many approaches rely on substantial prior knowledge or demonstrations.Related work includes skill sequencing, terrain-adaptive behavior, and optimization-based movement generation.
- Prior reinforcement-learning work demonstrated end-to-end locomotion and terrain adaptation, but often imposed considerable structure on the solution or policy.
- The present work combines simple reinforcement-learning rewards and curriculum training to produce adaptive locomotion in challenging environments with limited policy and behavior structure.
6 Discussion
The discussion reports that rich, diverse environments can produce non-trivial locomotion behaviors from simple rewards, while acknowledging that environmental variation may not always be sufficient.
- Training on diverse terrain produced jumping, crouching, and turning without directly incentivizing these behaviors through the reward function.
- The experiments suggest that richer environments and broader task spectra may improve the quality, robustness, and learnability of behaviors.
- The authors do not claim that environmental variations will always be sufficient to produce the desired behaviors.
A.1 Algorithm details
The algorithm details describe distributed PPO, in which workers collect trajectory data and asynchronously provide policy and baseline gradients while shared normalization and trust-region safeguards stabilize updates.
- Distributed PPO uses W workers, gradient-availability threshold D, policy and baseline sub-iterations M and B, and K-step data collection before updates.
- The chief updates global policy and baseline parameters after receiving at least W − D gradients from workers.
- Workers run the current policy for K timesteps, estimate truncated returns and advantages, store partial trajectories, and send policy and baseline gradients to a chief process.
- Training normalizes observations, scales rewards by a running standard deviation, and normalizes advantages per batch.
- Workers share normalization statistics and the time-varying regularization parameter λ, while KL-based penalties and early stopping constrain excessive policy changes.
A.2 Algorithm comparison
The algorithm comparison evaluates whether DPPO matches TRPO while scaling distributed computation, using fixed experimental settings and acknowledging that implementation differences complicate wall-clock comparisons.
- The comparison tests whether DPPO achieves results comparable to TRPO and whether distributing computation across workers accelerates data-intensive experiments.
- DPPO experiments use a coarse learning-rate search, shared learning rates across Section 2.1 experiments, and body-specific batch sizes.
- DPPO networks use tanh nonlinearities and body-specific architectures, including recurrent units for Memory Reacher.
- A3C uses a coarse search over learning rate and entropy cost, with matching hidden-unit counts but non-identical network architectures.
- Like-for-like algorithm comparison is difficult because separate code bases and distributed implementation choices affect wall-clock performance.
B Additional experimental details
Terrain and platform geometry are encoded through heightfield samples positioned around each agent. The sampling layout and density vary by body to provide forward-looking, spatially structured exteroceptive input.
- Terrain and platform heights are represented as heightfields sampled at points relative to the agent.Each heightfield pixel indicates terrain or platform height within a small region.
- The planar walker observes 50 equally spaced terrain samples along the x-axis.The samples extend from 2m behind the agent to 8m ahead.
- The planar walker separately samples platform height and receives body-height and sampling-grid offset features.The offset helps resolve aliasing from the piece-wise constant terrain representation.
- Quadruped and Humanoid observations use a two-dimensional variable-resolution grid spanning 1.2m behind to 5.6m ahead and 4m laterally.Sampling density decreases with distance from the body to reduce input dimensionality.
B.2 Rewards
The reward functions combine forward velocity with body-specific posture, height, falling, track-position, and torque terms. The supplied formulations define these components for the planar walker, quadruped, and humanoid.
- The planar walker reward combines forward velocity, upright torso orientation, target torso height, a low-height penalty, and torque cost.The orientation term uses nz, while the low-height penalty activates when ∆h < 0.3.
- The quadruped reward is forward velocity plus a small upright-orientation term minus torque cost.The orientation value nz ranges from 1.0 when upright to -1.0 when upside down.
- The humanoid reward uses cutoff forward velocity, quadratic velocity and control penalties, and a constant offset.The velocity cutoff vmax is usually set to 4m/s.