Source-linked AI summary
A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning
Laura Smith, Ilya Kostrikov, Sergey Levine
TL;DR
Sample inefficiency has kept deep reinforcement learning applications focused largely on simulation. This paper combines existing model-free RL methods with careful controller and task design for real-world quadruped learning, reporting walking acquisition in about 20 minutes across terrains.
Problem
Sample inefficiency has kept deep reinforcement learning applications primarily focused on simulated environments, leaving efficient real-world locomotion learning as a challenge.
Source: §I · p. 1Method
The authors combine standard actor-critic algorithms and regularization strategies with careful controller and task design for direct real-world quadruped training.
Source: §I · p. 2, §VII · p. 7Results
About 20 minutes of training produced walking on a range of real-world terrains, including indoor and outdoor locations.
Source: §I · p. 2, §VII · p. 7, B. Results · p. 7Takeaways & Limitations
The results suggest real-world locomotion training can be more practical than previously believed using careful combinations of current RL practices.
Source: §VII · p. 7
Abstract
from arXiv · showhide
Deep reinforcement learning is a promising approach to learning policies in uncontrolled environments that do not require domain knowledge. Unfortunately, due to sample inefficiency, deep RL applications have primarily focused on simulated environments. In this work, we demonstrate that the recent advancements in machine learning algorithms and libraries combined with a carefully tuned robot controller lead to learning quadruped locomotion in only 20 minutes in the real world. We evaluate our approach on several indoor and outdoor terrains which are known to be challenging for classical model-based controllers. We observe the robot to be able to learn walking gait consistently on all of these terrains. Finally, we evaluate our design decisions in a simulated environment.
I. INTRODUCTION
The introduction frames real-world sample efficiency as a key challenge and reports that careful implementation lets existing model-free RL learn quadruped walking in under 20 minutes. Related work contrasts direct real-world learning with simulation transfer, higher-level control, and computationally intensive training strategies.
I. INTRODUCTION
- Real-world sample demands have led many robotic RL studies to rely on simulation training and transfer rather than learning directly in deployment settings.Prior work also explored real-world training, but sometimes required extensive experience, including 1.2 · 10^9 samples for an A1 controller.Source: §I · p. 1
- Under 20 minutes, existing deep RL methods learned quadruped locomotion directly in the real world through careful task setup and implementation.The evaluation spans four locations—one indoor and three outdoor—and includes simulated analysis of design choices.Source: §I · pp. 1–2
- Table I compares prior works' hardware, action types, reset access, training data, and collection time for tasks resembling forward walking.Its training statistics exclude evaluation data and distinguish simulation from real-world collection.Source: Table I · p. 2
II. RELATED WORK
- Simulation-trained policies remain bounded by their training experiences and may not generalize perfectly to sufficiently different, unexpected real-world conditions.The authors note that anticipating every situation in complex, unstructured environments is difficult.Source: §II · p. 2
- Earlier direct-learning approaches used higher-level actions or fine-tuning, whereas this work learns from scratch with low-level control in multiple real-world environments.Some earlier methods shaped exploration and safety with high-level actions, which limits the skills that can be learned.Source: §II · pp. 2–3
- Prior locomotion studies used parallel simulation training or asynchronous pipelines, while this implementation supports synchronous per-step training on a single-GPU laptop.Earlier work sometimes treated asynchronous training as necessary because computation can constrain real-world training.Source: §II · p. 3
III. FAST AND SIMPLE RL FOR REAL-WORLD ROBOTS
The method uses a standard Q-function actor-critic framework, building most directly on DroQ. The authors emphasize careful implementation and task setup rather than a novel algorithmic component.
III. FAST AND SIMPLE RL FOR REAL-WORLD ROBOTS
- The framework uses standard Q-function actor-critic methods, building most directly on DroQ, which extends SAC with dropout and layer normalization.For this task, the authors report significant improvement from layer normalization and a higher update-to-data ratio.Source: §III · p. 3
A. Preliminaries
The preliminaries formalize locomotion as an infinite-horizon MDP and describe actor-critic learning through interleaved evaluation and improvement. They also contrast per-step updates with updating after collecting a trajectory.
A. Preliminaries
- Locomotion is modeled as an infinite-horizon MDP, with the policy seeking to optimize expected discounted cumulative return.The MDP specifies state and action spaces, initial-state and transition distributions, reward, and discount factor.Source: A. Preliminaries · p. 3
- Actor-critic learning interleaves policy evaluation, where a critic estimates discounted returns from a state and action, with policy improvement.The critic is then used to improve the policy by maximizing an objective.Source: A. Preliminaries · p. 3
- Off-policy continuous-control algorithms commonly update the critic and policy after each environment step, while some real-world systems collect a trajectory before updating.Trajectory-based updates can accommodate update times that exceed the robot's control frequency.Source: A. Preliminaries · p. 3
B. Efficient Model-Free RL
Regularization and normalization allow actor-critic methods to take more critic updates per sample, improving sample efficiency but increasing computation. The implementation targets synchronous real-time training to avoid delays associated with asynchronous or between-trial updates.
B. Efficient Model-Free RL
- Regularization or normalization lets off-policy actor-critic methods take more critic updates per environment step, increasing sample-efficient learning.The described methods can use up to 20 times as many critic updates, but computation can become the primary training bottleneck.Source: B. Efficient Model-Free RL · p. 4
- Asynchronous or between-trial training adds delay between interaction and learning, so the authors target real-time synchronous training.They use JAX just-in-time compilation to optimize execution.Source: B. Efficient Model-Free RL · p. 4
IV. SYSTEM DESIGN
The system is designed for fast learning in unstructured environments using low-level actions and onboard proprioception. Its state, action, and controller choices support training without motion-capture infrastructure.
IV. SYSTEM DESIGN
- The policy uses proprioceptive feedback and directly outputs joint targets rather than relying on predefined gaits or trajectory generators.The design is intended to allow training anywhere without an instrumented motion-capture system.Source: §IV · p. 4, A. State and Action Spaces · p. 4
A. State and Action Spaces
- The state includes root orientation and velocities, joint angles and velocities, binary foot contacts, and the previous action.Root orientation includes roll and pitch; angular velocity includes roll, pitch, and yaw.Source: A. State and Action Spaces · p. 4
- Actions are 12-joint PD position targets applied at 20 Hz within bounded offsets from default motor angles.The real-world default angles differ from the simulated settings; the authors report that the real-world setting promotes safer exploration with fewer failures.Source: A. State and Action Spaces · p. 4
B. Reward Function
The reward is based on forward speed relative to a target, with velocity projected onto the ground plane to avoid rewarding a forward dive. For outdoor training, the system uses onboard velocity estimates despite their lower accuracy than VR estimates.
B. Reward Function
- The reward is constant within a target-velocity interval and linearly decays to zero outside it.The target velocity is v_t, and v_x denotes forward linear velocity in the robot frame.Source: B. Reward Function · pp. 4–5
- Projecting forward velocity onto the ground plane avoids rewarding a forward dive caused by a fall onto the robot’s head.The authors found that local-frame forward velocity could be high during such a fall.Source: B. Reward Function · p. 5
- The outdoor setup replaces indoor HTC Vive reward estimates with a Kalman filter-based velocity estimator that is less accurate and systematically underestimates velocity.Despite the resulting difference between simulated and real-world rewards, the authors found effective real-world gaits feasible to learn.Source: B. Reward Function · p. 5
- Training continues with zero reward when no foot contacts are detected during a lift, and terminates when roll or pitch exceeds 30 degrees.The open-source reset policy from Smith et al. is used to reset the robot.Source: B. Reward Function · p. 5
V. SIMULATION ANALYSIS
Simulation tests examine controller damping, task setup, policy-update timing, and SAC regularization choices. The results highlight the importance of constrained actions and show that several regularization or normalization methods can support higher update-to-data ratios.
V. SIMULATION ANALYSIS
- Controller damping strongly affects learning, and constraining the action space enables stable training while preserving progress toward target velocity.The authors use damping set to 10 for the remaining ablations; unconstrained actions do not yield progress in simulation.Source: §V · p. 5
- Regularization or normalization methods can similarly improve SAC performance by enabling higher update-to-data ratios, rather than requiring one specific technique.Dropout alone performs similarly to REDQ, while layer normalization without dropout performs even better in the reported comparison.Source: Figure 3 · p. 5, §V · p. 6
- Updating after every environment step rather than between episodes leads to more stable training and faster convergence.The more frequent updates must be inexpensive enough to run between the environment’s 20 steps per second.Source: Figure 3 · p. 5, §V · p. 6
- A low-pass filter on PD targets degrades learning efficiency, which the authors attribute to action-history dependence violating the Markovian assumption.The caption identifies the evaluated comparisons, including damping, task setup, and regularization and normalization methods.Source: §V · p. 5, Figure 3 · p. 5
- About 30k samples are needed for SAC with a 20 update-to-data ratio to reach target velocity; REDQ reduces this by roughly 5k time-steps.The authors report that the reduction corresponds to about five minutes of wall-clock time.Source: §V · p. 5
VI. LEARNING IN THE REAL WORLD
The real-world experiments assess how quickly and consistently model-free RL can teach walking, including outside the lab. The evaluation repeats training in a controlled setting and tests additional terrains, most of them outdoors.
VI. LEARNING IN THE REAL WORLD
- The experiments ask how quickly and consistently model-free RL can teach the robot to walk in the real world.One experiment trains the robot four times in the same controlled environment to examine variation across seeds and small changes in conditions.Source: §VI · p. 6
- Four additional terrains are used to test walking beyond flat laboratory ground, including three outdoor locations.The evaluation asks whether the robot can learn to walk in the wild as well as in the lab.Source: §VI · p. 6
- The section also discusses applying design decisions tuned in simulation to real-world training.Source: §VI · p. 6
A. Setup
The experiments span five terrains, from flat solid ground to deformable and irregular outdoor surfaces. Training begins with random actions, followed by synchronous policy updates between robot actions.
A. Setup
- The five test terrains are flat solid ground, memory foam, mulch, lawn, and a hiking trail.The flat-ground setup uses dense foam tiles; the other surfaces include a 5 cm memory-foam mattress, deep bark mulch, cushioned lawn, and an irregular trail.Source: A. Setup · p. 6
- The surfaces pose distinct challenges: the robot sinks into mulch, must avoid dragging the mattress cover, and faces a low-friction, deformable lawn.The hiking trail is compact, dry, slightly inclined, and irregular.Source: A. Setup · p. 6
- Each experiment starts with about 1 minute of uniformly random actions, then updates the policy between robot actions every 0.05 seconds.Each update comprises 20 critic updates and an actor update.Source: A. Setup · p. 6
- The final implementation runs training at 120 Hz, while the best PyTorch implementation permits only a 7.5 Hz effective control frequency.The final implementation batches 20 critic updates per environment step.Source: A. Setup · p. 6
B. Results
The robot learned to traverse each tested terrain in about 20 minutes, with terrain-specific behaviors emerging during training. Learning curves track average policy velocity against wall-clock time across the real-world experiments.
B. Results
- About 20,000 samples (roughly 20 minutes) sufficed to traverse each tested terrain; flat-ground runs learned to walk in under 20,000 samples.For flat ground, the paper reports roughly 17 minutes of data and 20 minutes of wall-clock time, including overhead.Source: Figure 5 · p. 7, B. Results · p. 7
- Figure 5 plots average stochastic-policy velocity against wall-clock time; 1,000 time-steps correspond to roughly 1 minute of training.The robot continues running while training occurs between actions, and reported sample counts exclude periods when it is lifted and reoriented.Source: B. Results · p. 6, Figure 5 · p. 7
- After 15 minutes on flat ground, the robot adopted more conservative behavior to walk while maintaining balance.The flat-ground average and standard deviation are reported across four runs.Source: B. Results · p. 7
- On memory foam, the robot escaped its initially sunk position within about 5 minutes; on thick wood mulch, it learned in the first ten minutes to kick up its front feet to dig forward.On memory foam, its feet often caught on the fabric cover, and it learned to walk without dragging the fabric.Source: B. Results · p. 7
VII. CONCLUSION
Deep RL trained legged robots to walk on real-world terrains in about 20 minutes using a high-quality implementation of existing algorithmic ideas. Simulation comparisons found several design choices worked well, and the resulting approach learned successfully across varied terrains.
VII. CONCLUSION
- About 20 minutes of deep RL training enabled legged robots to walk on grass, loose ground, forest trails, and mattresses.The conclusion describes these as real-world settings.Source: §VII · p. 7
- Combining standard actor-critic algorithms with different regularization strategies supported efficient real-world learning without significant departures from existing RL practice.The authors emphasize careful implementation and combination of current best practices.Source: §VII · p. 7
- Simulation comparisons showed that a variety of design choices can work well, while the approach learned successfully across a range of real-world terrains.Source: §VII · p. 7