Learning to Walk via Deep Reinforcement Learning

Tuomas Haarnoja; Sehoon Ha; Aurick Zhou; Jie Tan; George Tucker; Sergey Levine

Source-linked AI summary

Learning to Walk via Deep Reinforcement Learning

Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, Sergey Levine

arXiv:1812.11103v3Dec 26, 2018cs.LGcs.AIcs.ROstat.ML

TL;DR

Real-world deep RL for legged locomotion is hindered by poor sample complexity and sensitivity to hyperparameters. The paper develops maximum-entropy deep RL with automated temperature adjustment and an asynchronous learning system, then trains a Minitaur gait directly on hardware in about two hours while demonstrating robustness and strong simulated benchmark performance.

Problem
Real-world deep RL for robotic locomotion requires many trials and sensitive hyperparameter tuning, making learning on physical legged robots difficult.
Method
The paper extends soft actor-critic with entropy-constrained dual optimization that automatically adjusts temperature, integrated into an asynchronous end-to-end robotic learning framework.
Results
About 400 rollouts, or two hours, produced a Minitaur walking gait from scratch on a real robot, with generalization to unseen terrains and state-of-the-art simulated benchmark performance using one hyperparameter set.
Takeaways & Limitations
The results support direct, model-free real-world learning of quadrupedal locomotion controllers without prior simulation or robot dynamics knowledge.

Abstract

from arXiv · show

Deep reinforcement learning (deep RL) holds the promise of automating the acquisition of complex controllers that can map sensory inputs directly to low-level actions. In the domain of robotic locomotion, deep RL could enable learning locomotion skills with minimal engineering and without an explicit model of the robot dynamics. Unfortunately, applying deep RL to real-world robotic tasks is exceptionally difficult, primarily due to poor sample complexity and sensitivity to hyperparameters. While hyperparameters can be easily tuned in simulated domains, tuning may be prohibitively expensive on physical systems, such as legged robots, that can be damaged through extensive trial-and-error learning. In this paper, we propose a sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies. We apply this method to learning walking gaits on a real-world Minitaur robot. Our method can acquire a stable gait from scratch directly in the real world in about two hours, without relying on any model or simulation, and the resulting policy is robust to moderate variations in the environment. We further show that our algorithm achieves state-of-the-art performance on simulated benchmarks with a single set of hyperparameters. Videos of training and the learned policy can be found on the project website.

I. INTRODUCTION

The paper targets real-world legged locomotion, where conventional controller design requires expertise and modeling while deep RL is sample- and tuning-intensive. It proposes maximum-entropy deep RL with automated temperature adjustment and demonstrates real-world and simulated performance.

Deep RL could automate end-to-end locomotion control without prior gait knowledge, explicit dynamics, system identification, or manual engineering.
Tens of thousands of trials and sensitive hyperparameters make existing deep RL difficult and potentially costly to apply on physical legged robots.
The proposed method extends soft actor-critic by optimizing temperature toward a target expected entropy, eliminating manual per-task temperature tuning.
About 400 rollouts, or two hours of real-world training, produced a Minitaur walking gait that generalized from flat terrain to unseen terrains and obstacles.The robot succeeded in repeated tests involving slopes, wooden blocks, and stairs.
The algorithm achieved state-of-the-art simulated benchmark performance using exactly the same hyperparameters across tasks.

III. ASYNCHRONOUS LEARNING SYSTEM

The learning system separates robot data collection, reward computation, and neural-network training across asynchronously running machines. It combines maximum-entropy reinforcement learning with replay-buffer training while addressing entropy-temperature sensitivity.

Three asynchronous jobs collect robot experience, compute motion-capture rewards, and update neural networks across different machines.
The data-collection job executes the latest policy onboard, records state-action trajectories, and sends rollouts to the training job.
The motion-capture system evaluates rewards and appends experience to a replay buffer, from which training batches are sampled randomly.
Asynchronous subsystems can be paused or restarted independently after hardware or communication errors, and the design can scale by adding data-collection jobs.
Maximum-entropy RL balances expected return and policy entropy, but reward scaling and the temperature parameter can substantially affect performance.

V. AUTOMATING ENTROPY ADJUSTMENT FOR MAXIMUM ENTROPY RL

The method replaces manually tuned temperature selection with an entropy-constrained maximum-entropy RL objective. Dual optimization automatically adjusts temperature while allowing entropy to vary across states.

Reward scale changes across tasks and during training, making a manually selected temperature difficult to transfer or maintain.
The entropy constraint's Lagrangian relaxation recovers a maximum-entropy objective in which the dual variable serves as the temperature.
The method constrains minimum expected policy entropy rather than enforcing the same entropy independently at every state.This preserves stochastic behavior where actions are uncertain and permits deterministic behavior where one action is clearly preferable.
For a fixed temperature, the resulting objective can be optimized with an off-the-shelf maximum-entropy RL algorithm, specifically approximate dynamic programming corresponding to soft actor-critic.
Temperature updates use dual gradient optimization, alternating with policy optimization to satisfy the entropy constraint.

B. Practical Algorithm

The practical algorithm implements automatic entropy adjustment with function approximators and stochastic optimization. Training alternates between collecting environment data and jointly optimizing the policy, Q-functions, and temperature.

The implementation uses a Gaussian policy, two parameterized Q-functions, replay-buffer minibatches, and stochastic gradient descent.
The policy update follows soft actor-critic with an explicit dynamic temperature α.
Temperature learning uses a truncated dual-gradient procedure that alternates single gradient steps rather than optimizing primal variables to convergence.Global convergence is not guaranteed, although the authors report that the approach worked well in practice.
Training alternates between data collection and optimization phases, jointly updating all objectives and using delayed target Q-function networks.
The temperature is parameterized as αt = exp(βt) to keep it non-negative during optimization.

VI. EVALUATION ON SIMULATION ENVIRONMENTS

The paper evaluates the proposed algorithm in simulation against several continuous-locomotion benchmarks and frames the study around efficiency, robustness, entropy regulation, and generalization.

The simulation study asks whether the method achieves state-of-the-art data efficiency, reduces hyperparameter sensitivity, regulates entropy, and generalizes to unseen situations.

A. Evaluation on OpenAI Benchmark Environments

The benchmark comparison evaluates the proposed method against tuned soft actor-critic and several established deep RL algorithms using matched network architectures.

The method is compared with soft actor-critic using an environment-specific fixed temperature, plus DDPG, PPO, and TD3.
All compared algorithms use the same network architecture in the benchmark evaluation.

1) Comparative Evaluation:

Across standard locomotion benchmarks, the method performs comparably to or better than competing algorithms, despite requiring less hyperparameter tuning.

Our method achieves practically identical or better performance than standard SAC across the evaluated environments.The SAC baseline was tuned separately for each environment.
The benchmark learning curves compare performance across standard tasks while other algorithms underwent dense hyperparameter tuning.
The method’s performance is evaluated against average normalized returns over the final 100k samples across multiple environments.
The method performs better or comparably to standard SAC, DDPG, TD3, and PPO.

2) Sensitivity Analysis:

The analysis compares standard SAC’s reward-scale sensitivity with the proposed method’s target-entropy sensitivity, showing greater robustness for the proposed method.

Reward scale affects maximum-entropy RL by changing the trade-off between reward maximization and entropy maximization.
Figure 5 compares entropy and temperature trajectories on HalfCheetah using a target entropy of -13 for the proposed method.
Our method is substantially less sensitive to its target-entropy hyperparameter than standard SAC is to reward scale.The comparison uses average normalized performance over the last 100k samples across environments.

3) Validation of Entropy Control:

The experiments examine entropy regulation, simulated locomotion efficiency and robustness, and whether learned policies generalize beyond training conditions.

The method matches the target entropy in relatively few steps, whereas fixed-temperature SAC’s entropy slowly decreases as its Q-function increases.
The simulated Minitaur evaluation measures data efficiency, perturbation robustness, and states unavailable on the physical robot without transferring a simulated policy to reality.
The method is the most data efficient and requires fewer episodes to train a good policy than the compared algorithms.Tuning SAC’s temperature required sweeping candidate values, equivalent to collecting an order of magnitude more samples.
Maintaining higher policy entropy helps the algorithm escape the local minimum associated with “diving forward.”This failure mode commonly causes falling and early episode termination.
The simulated robot withstands lateral pushes up to 220N and recovers to normal walking, compared with 130N for the best PPO-trained policy selected from 1000 trials.
The real-world evaluation asks whether the method trains efficiently without tuning and generalizes to unseen situations.

A. Experiment Setup

On a real Minitaur robot, the study uses an end-to-end reinforcement-learning setup with direct hardware training and evaluates gait performance, perturbation recovery, and terrain generalization.

A. Experiment Setup: The quadrupedal robot’s observation includes motor angles, roll and pitch, and angular velocities, while actions specify leg swing and extension mapped to motor positions.Yaw is excluded because it drifts quickly, and low PD gains provide compliant motion.
A. Experiment Setup: The policy and value functions use randomly initialized fully connected networks with two hidden layers of 256 neurons each.Actions were smoothed during the first 50 episodes to reduce jerky early motion.
B. Results: 160k control steps, approximately 400 rollouts, are sufficient for the method to learn walking on the real robot.Each rollout can last up to 500 steps, or 10 seconds, and may terminate when the robot falls.
B. Results: Training takes about two hours, with slightly lower real-world performance than in simulation, potentially because fewer samples are collected.
B. Results: The trained robot walks at 0.32m/s, approximately 0.8 body lengths per second, using a learned periodic and synchronized gait.The gait emerges without an explicit trajectory generator, symmetry constraint, or periodicity constraint.
B. Results: The learned gait has similar frequency and swing amplitude to the manufacturer’s trotting controller but substantially different joint trajectories and foot placement.The learned policy uses approximately 2Hz frequency and 0.7 Rad swing amplitudes.
B. Results: After untrained backward and lateral perturbations, the policy recovered to periodic gait in all 10 repeated tests.
B. Results: After flat-terrain training, the robot successfully navigated slopes, wooden blocks, and stairs in all 10 repeated tests.These obstacles were absent during training.

VIII. CONCLUSION

The paper presents an end-to-end, entropy-constrained RL system that learns quadrupedal locomotion directly on a real robot without prior modeling. A walking gait emerged in two hours, while the system’s remaining barriers include manual resets and missing safety mechanisms for larger robots.

The entropy-constrained algorithm automatically adjusts the temperature hyperparameter, producing sample-efficient and hyperparameter-stable learning.This dual formulation targets expected entropy during training rather than requiring manual temperature tuning.
A walking gait emerged from scratch on a real-world quadruped in two hours without prior task knowledge or a robot dynamics model.The result demonstrates direct end-to-end learning on physical hardware rather than relying on simulation pretraining.
Learning directly on hardware may be more practical than sim-to-real methods when accurate models are difficult and expensive to obtain.The paper specifically identifies diverse-terrain walking and deformable-object manipulation as settings where modeling is challenging.
The current system depends heavily on manual episode resets and lacks a safety layer for training larger robots.The authors identify automatic resets and safety-aware learning as future-work priorities.