Source-linked AI summary
Real-World Humanoid Locomotion with Reinforcement Learning
Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, Koushil Sreenath
TL;DR
Humanoid locomotion controllers are difficult to generalize and adapt to diverse environments, limiting autonomous operation in the physical world. The paper uses a causal transformer trained with large-scale reinforcement learning in randomized simulation and deployed zero-shot, achieving robust outdoor walking and contextual adaptation. The results support learning-based control for complex humanoid locomotion in the physical world.
Problem
Classical humanoid controllers can perform impressively but are challenging to generalize and adapt to new environments.
Method
A causal transformer predicts actions from proprioceptive observation-action history and is trained in large-scale randomized simulation before zero-shot real-world deployment.
Results
The policy enables reliable outdoor walking without falls, resists external disturbances, traverses different terrains, carries varying-mass payloads, and adapts its behavior to context.
Takeaways & Limitations
The results suggest that simple, general learning-based controllers can perform complex, high-dimensional humanoid control in the physical world.
Takeaways & Limitations
The paper notes a limitation involving conditions that can cause the robot to fall.
Abstract
from arXiv · showhide
Humanoid robots that can autonomously operate in diverse environments have the potential to help address labour shortages in factories, assist elderly at homes, and colonize new planets. While classical controllers for humanoid robots have shown impressive results in a number of settings, they are challenging to generalize and adapt to new environments. Here, we present a fully learning-based approach for real-world humanoid locomotion. Our controller is a causal transformer that takes the history of proprioceptive observations and actions as input and predicts the next action. We hypothesize that the observation-action history contains useful information about the world that a powerful transformer model can use to adapt its behavior in-context, without updating its weights. We train our model with large-scale model-free reinforcement learning on an ensemble of randomized environments in simulation and deploy it to the real world zero-shot. Our controller can walk over various outdoor terrains, is robust to external disturbances, and can adapt in context.
Introduction
The paper addresses the challenge of generalizing humanoid locomotion beyond controlled settings with a fully learning-based controller. It trains a causal transformer in randomized simulation and deploys it zero-shot to demonstrate robust, adaptive real-world walking.
- Motivation and challenge: Humanoid locomotion remains a considerable control challenge despite progress from full-sized robots and classical methods.
- Approach: The paper proposes a learning-based approach for real-world humanoid locomotion using a causal transformer.
- Approach: The controller uses proprioceptive observation-action history to predict the next action.
- Training and deployment: Large-scale reinforcement learning trains the model across thousands of randomized simulation environments before zero-shot real-world deployment.
- Results: The policy enables reliable outdoor walking without falls, traverses different terrains, carries varying-mass payloads, and resists external disturbances.
- Results: The controller adapts behavior to context, including gradual gait changes across terrain and rapid responses to sudden obstacles.
Results
The learning-based controller performs robust humanoid locomotion across outdoor and indoor conditions, including disturbances, varied terrain, payloads, and commanded motions. It also exhibits emergent context-dependent adaptation, recovering from foot-trapping and changing gait based on terrain.
- Outdoor deployment: The controller was deployed zero-shot and walked reliably across outdoor terrains not encountered during training without observed falls over one week of full-day testing.The tested environments included plazas, walkways, sidewalks, tracks, and grass fields with varied materials and wet or dry conditions.
- External forces: The controller stabilized the robot against sudden disturbances including a yoga-ball impact, stick push, and pulling while walking.These tests required rapid reactions to avoid falling.
- Rough terrain: The controller traversed rubbers, cloths, cables, bubble wraps, and slopes up to 8.7% grade, with greater robustness at 0.2 m/s on steeper slopes.The training simulation included slopes up to 10% grade, while testing slopes reached 8.7%.
- Payloads: The robot completed walking routes with five payload types, including a loaded trash bag attached to its arm despite reliance on arm swing for balance.The results support adaptation to payload mass, shape, and center-of-mass changes.
- Comparison to the state of the art: The controller outperformed the company controller in simulation across three settings, with larger gains on steps and unstable ground; on steps, the company controller shut off after foot-trapping while ours recovered.The step scenario was replicated on hardware with consistent recovery behavior, but the controllers were not evaluated on that terrain in the real world because of potential hardware damage.
- Natural and adaptive locomotion: The policy followed forward, backward, turning, and other velocity commands, achieved commanded velocity from rest within 1 s, and exhibited coordinated contralateral arm swing.It also changed from natural walking to small steps on a downward slope and back again, while hidden-state projections clustered by terrain type.
- Emergent recovery: After foot-trapping, the robot lifted its legs higher and faster on subsequent attempts and recovered consistently, although this behavior was not pre-programmed or encouraged during training.Changes and spikes in transformer hidden-state activity correlated with foot-trapping events.
Discussion
The paper presents a learning-based transformer controller for full-sized humanoid locomotion, trained in simulation and deployed zero-shot to the real world. It demonstrates robust, adaptable behavior across terrains and disturbances, while retaining practical limitations in symmetry, velocity tracking, and extreme disturbances.
- Contribution and approach: A causal transformer controller uses histories of past observations and actions to predict future actions for full-sized humanoid locomotion.The controller is presented as a general transformer model for learning-based locomotion.
- Contribution and approach: Large-scale simulation training supports zero-shot deployment of the model to the real world.The model is trained using large-scale simulation before real-world deployment.
- Demonstrated capabilities: The policy enables reliable outdoor walking without falls, resists external disturbances, traverses different terrains, and carries payloads of varying mass.These capabilities are reported across real-world and simulated experiments.
- Demonstrated capabilities: The controller exhibits natural walking behaviors, including following commands, high-speed locomotion, and emergent arm swing motion.These behaviors are reported as part of the policy’s locomotion outcomes.
- Adaptation: At test time, the controller adapts its behavior to context, including terrain-dependent gait changes and recovery from foot-trapping.The paper links these adaptations to novel scenarios encountered during testing.
- Limitations: The approach remains limited by asymmetric motion, imperfect commanded-velocity tracking, and falls under excessive external disturbances.The authors identify better lateral movement to the left than the right and falls caused by very strong cable pulls.
- Future directions: The transformer architecture may be easier to scale with additional data and compute and to extend with additional input modalities.These are proposed future-work directions rather than demonstrated results in this passage set.
Materials and Methods
The method trains a transformer-based observation policy through staged teacher supervision and reinforcement learning, using randomized simulation before real-world transfer. The controller models histories of observations and actions to support causal, in-context locomotion decisions.
- Simulation and transfer: Training uses massively parallel randomized simulation, followed by validation in a manufacturer-provided high-fidelity simulator and transfer to the real robot.The simulation framework supports closed kinematic chains for the underactuated Digit humanoid robot.
- Problem formulation: The control problem is modeled as a POMDP because real-world agents receive noisy observations rather than the environment’s true state.The observation function represents uncertainty in mapping states to observations.
- Model architecture: The causal transformer consumes a fixed-length history of observation-action pairs and predicts the next action.Observation-action pairs are embedded as tokens with positional encodings, while causal attention restricts access to preceding tokens.
- Policy learning: Training first optimizes a fully observable teacher state policy, then distills it into an observation policy using teacher imitation and reinforcement learning.The state policy is trained in simulation before supervision is transferred through KL divergence.
- Joint optimization: The joint objective combines reinforcement-learning loss with KL-divergence supervision, while the supervision weight is gradually annealed to zero.This addresses the mismatch between state-space supervision and observation-space decision-making; the joint objective outperforms either term alone.