Source-linked AI summary
Learning to Walk in the Real World with Minimal Human Effort
Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, Jie Tan
TL;DR
Reliable legged locomotion is difficult to learn in the real world because on-robot training requires automated data collection and safety. The paper introduces multi-task learning and safety-constrained RL to address these bottlenecks, and demonstrates autonomous locomotion learning on multiple terrains with minimal human intervention.
Problem
Real-world deep-RL locomotion requires scalable training with minimal human effort, including automatic collection, workspace containment, resets, and safe operation.
Method
The system combines multi-task learning across walking directions with a safety-constrained RL framework for autonomous real-world locomotion learning.
Results
The Minitaur learns distinct locomotion skills on flat ground, a soft mattress, and a doormat with crevices in a few hours, with minimal human intervention.
Takeaways & Limitations
Four directional policies can be trained simultaneously and composed into an interactive navigation controller, while flat-ground training requires zero manual resets in the reported case.
Takeaways & Limitations
The current system relies on a manually designed robust stand-up controller, although future work aims to learn recovery policies from real-world experience.
Abstract
from arXiv · showhide
Reliable and stable locomotion has been one of the most fundamental challenges for legged robots. Deep reinforcement learning (deep RL) has emerged as a promising method for developing such control policies autonomously. In this paper, we develop a system for learning legged locomotion policies with deep RL in the real world with minimal human effort. The key difficulties for on-robot learning systems are automatic data collection and safety. We overcome these two challenges by developing a multi-task learning procedure and a safety-constrained RL framework. We tested our system on the task of learning to walk on three different terrains: flat ground, a soft mattress, and a doormat with crevices. Our system can automatically and efficiently learn locomotion skills on a Minitaur robot with little human intervention. The supplemental video can be found at: \url{https://youtu.be/cwyiq6dCgOc}.
1 Introduction
The paper develops an autonomous deep-RL system for learning legged locomotion in real-world environments with minimal human effort. It addresses automated data collection and safety through multi-task learning and safety-constrained reinforcement learning.
- Motivation: Real-world locomotion learning must reduce human supervision while handling varied environments that are difficult to represent accurately in simulation.The system is intended to scale across robots and real-world terrains with minimal intervention.
- Approach: The system addresses workspace departure, dangerous falls, and trial resets through multi-task learning and a safety-constrained learner.Different walking directions help keep the robot within the training area, while balancing reward and safety reduces falls.
- Contribution: The main contribution is an autonomous real-world reinforcement-learning system that learns multiple locomotion skills for a quadrupedal robot across varied surfaces.The tested terrains are flat ground, a soft mattress, and a doormat with crevices.
- Results: The system learns distinct, effective gaits on three terrains in a few hours with minimal human effort.It also trains four directional policies that can be composed into an interactive walking controller.
- Results: On flat ground, the system requires zero manual resets, whereas prior work required over a hundred manual resets in the simple walking case.The comparison concerns manual resets during training.
2 Related Work
Prior legged-robot controllers commonly use modular model-based designs, while deep RL has mostly been demonstrated in simulation. Real-robot RL avoids the sim-to-real gap but remains difficult to scale because of repeated resets, safety requirements, and human intervention.
- Model-based control: Traditional legged-robot control decomposes locomotion into modules including state estimation, foot-step planning, trajectory optimization, and model-predictive control.Examples include state machines, impulse scaling, inverted-pendulum footstep planning, and whole-body control.
- Deep reinforcement learning: Deep RL offers a general framework for acquiring control policies but has been used for locomotion mostly in simulation, where sim-to-real transfer can degrade performance.The cited applications span driving, navigation, manipulation, and locomotion.
- Real-robot learning: Direct RL on real robots avoids the sim-to-real gap, but underactuated legged robots require repeated resets after hundreds or thousands of roll-outs.Prior solutions include statically stable robots, external reset devices, and manual resets.
- Safe reinforcement learning: Safety during learning can be formulated as a constrained Markov Decision Process and addressed through Lagrangian relaxation.The related work frames safety as maintaining constraints while improving rewards.
- This paper: This paper targets autonomous and safe legged-robot learning with minimal human intervention, including challenging terrains and simultaneous policy learning.The contrast is with prior work requiring hundreds of human interventions during training.
3 Background: Reinforcement Learning
The paper formulates learning to walk as a reinforcement-learning problem represented by a Markov Decision Process. The objective is to find a policy that maximizes expected cumulative returns over trajectories.
- MDP formulation: The walking task is modeled as a Markov Decision Process with state and action spaces, transitions, rewards, and initial-state distribution.A policy generates trajectories of states and actions under the transition dynamics.
- Objective: The learning objective is to find an optimal policy that maximizes the sum of expected returns.Expected returns are accumulated along trajectories induced by the policy.
4 Automated Learning in the Real World
The system automates real-world locomotion learning by combining multi-task scheduling with safety-constrained reinforcement learning. These mechanisms reduce manual resets and protect the robot during repeated training rollouts.
- Automated Unattended Learning via Multi-Task RL: Multi-task learning schedules locomotion in multiple directions so the robot can remain within the training workspace.Tasks are defined by desired walking direction relative to each episode’s initial position and orientation.
- Automated Unattended Learning via Multi-Task RL: The scheduler selects the task whose desired direction points toward the workspace center, rather than using simple round-robin scheduling.This accounts for tasks potentially being learned at different rates.
- Automated Unattended Learning via Multi-Task RL: Counter-tasks can return the robot from a workspace boundary, assuming partially trained policies move slightly in their desired directions.The authors report that these assumptions generally hold, except when the robot becomes stuck at workspace corners.
- Automated Unattended Learning via Multi-Task RL: Each task has a separate policy-learning instance without shared actors, critics, or replay buffer.The authors report no clear performance gain from sharing weights or data across discrete tasks.
- Safety-Constrained RL: Early termination prevents occasional out-of-workspace failures when the robot approaches and moves toward a boundary.Unlike falls, these terminations require special treatment of return calculation because the robot remains upright.
- Safety-Constrained RL: Safety-constrained reinforcement learning limits dangerous falls while optimizing locomotion reward.The safety constraints are designed to prevent falls that could damage servo motors, using pitch and roll tilt limits of π/12 and π/6.
- Safety-Constrained RL: Dual gradient descent alternates optimization of the policy and the Lagrangian multiplier.The method trains value functions for both the regular reward and safety term before updating the actor and multiplier.
5 Experiments
The experiments evaluate autonomous real-world locomotion learning on flat ground and challenging surfaces, alongside simulation analyses of automation and safety. The system learns multiple gaits and policies with substantially reduced failures and limited human intervention.
- 5.1 Learning on Flat Terrain: In two of three flat-ground runs, the system required zero human interventions, while the third required only two manual resets.The prior method required hundreds of human interventions during one training run.
- 5.1 Learning on Flat Terrain: The system trained forward and backward policies in 1.5 hours using approximately 60k steps per policy, compared with 2 hours or 160k steps for one prior policy.The authors attribute the efficiency to updating the policy at every real-world execution step.
- 5.1 Learning on Flat Terrain: The learned forward and backward policies used distinct gaits because of the robot’s asymmetric leg structure.Forward walking used higher clearance and resembled pacing, whereas backward walking could drag the feet and resembled high-frequency bounding.
- 5.1 Learning on Flat Terrain: A four-task configuration learned forward, backward, left-turning, and right-turning policies, including complete 360° in-place turns within ten seconds.The learned policies were integrated with a remote controller for real-time gait switching and navigation.
- 5.2 Learning to Walk on Varied Surfaces: The system learned forward and backward locomotion on both a soft mattress and a doormat with crevices, adapting gait behavior to each surface.Mattress training required 200k steps over 5.5 hours, doormat training required 150k steps over 4.5 hours, and each challenging surface still required 20 to 30 manual resets.
- 5.3 Analysis: In simulation, multi-task learning reduced out-of-workspace failures to 5%–10% of the single-task baseline across three workspace sizes.The large-workspace multi-task failure count also matched empirical real-world results.
- 5.3 Analysis: The safety-constrained method achieved a successful policy with approximately 40 falls, versus over 100 falls for SAC without safety constraints.Its fall count was higher than optimally tuned SAC with fixed weight = 1.0, but it avoided careful hyperparameter tuning.
6 Conclusion and Future Work
The paper presents an autonomous real-world system that addresses locomotion-learning automation and safety through multi-task learning and a safety-constrained MDP. It reduces manual resets, learns gaits on challenging surfaces, and still relies on a manually designed stand-up controller.
- The system uses multi-task learning to keep the robot within the training area and a safety-constrained MDP to reduce falls and breakage during training.The safety-constrained learner does so without additional hyper-parameter tuning.
- It reduces manual resets by more than an order of magnitude compared with the prior state-of-the-art on-robot training system.
- The system learns successful gaits on challenging surfaces, including a soft mattress and a doormat with crevices.These surfaces are settings where accurate simulation models are expensive to acquire.
- Future work: The current system requires a robust stand-up controller designed manually for varied situations.The authors intend to learn recovery policies using the proposed framework in future work.
Appendix - Experiment Details
The experiments use a Minitaur quadruped with onboard sensing and optimize a safety-constrained SAC learner. Hardware additions manage tether cables and filter exploratory actions to reduce tangling and motor wear.
- Robot: Minitaur is a roughly 7 kg quadruped with eight direct-drive servo motors, joint-position encoders, and an IMU measuring torso roll and pitch dynamics.Its legs use nonsymmetric four-bar linkages and move in the sagittal plane.
- Learning algorithm: The safety-constrained MDP is solved with Soft Actor-Critic while dual gradient descent optimizes entropy and safety multipliers α and λ.The policy and value functions use fully connected two-hidden-layer networks with 256 neurons per layer.
- Hardware adaptations: A cable management system hangs power, communication, and motion-capture cables above the robot to prevent tether tangling during walking and turning.The cables run through a 1.2 m rod mounted 2.5 m high.
- Hardware adaptations: A first-order low-pass Butterworth filter with a 5 Hz cutoff reduces motor wear caused by jerky random exploration.