Source-linked AI summary
Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World
Laura Smith, J. Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, Sergey Levine
TL;DR
The paper tackles the difficulty of making locomotion policies work in deployment environments that cannot all be anticipated during training. It presents a system that combines simulation pre-training with autonomous real-world reinforcement-learning fine-tuning. With modest real-world training, an A1 quadruped improves multiple locomotion skills across outdoor lawn and indoor terrains.
Problem
Locomotion policies may fail when deployment conditions differ substantially from the environments represented during training.
Method
The system pre-trains motion-imitation and recovery policies in simulation, then continues off-policy reinforcement-learning updates using onboard sensing, autonomous resets, and real-world data.
Results
In less than 2 hours on a lawn and less than 2.5 hours on each indoor terrain, the robot learned to execute pacing or sidestepping consistently with few failures.
Takeaways & Limitations
The system enables an A1 quadruped to autonomously fine-tune multiple locomotion skills across grass, carpet, memory foam, and a creviced doormat.
Abstract
from arXiv · showhide
Legged robots are physically capable of traversing a wide range of challenging environments, but designing controllers that are sufficiently robust to handle this diversity has been a long-standing challenge in robotics. Reinforcement learning presents an appealing approach for automating the controller design process and has been able to produce remarkably robust controllers when trained in a suitable range of environments. However, it is difficult to predict all likely conditions the robot will encounter during deployment and enumerate them at training-time. What if instead of training controllers that are robust enough to handle any eventuality, we enable the robot to continually learn in any setting it finds itself in? This kind of real-world reinforcement learning poses a number of challenges, including efficiency, safety, and autonomy. To address these challenges, we propose a practical robot reinforcement learning system for fine-tuning locomotion policies in the real world. We demonstrate that a modest amount of real-world training can substantially improve performance during deployment, and this enables a real A1 quadrupedal robot to autonomously fine-tune multiple locomotion skills in a range of environments, including an outdoor lawn and a variety of indoor terrains.
I. INTRODUCTION
The paper addresses the difficulty of building locomotion controllers that generalize across unpredictable deployment environments. It proposes autonomous real-world fine-tuning so robots can continue improving after deployment.
- Legged robots can traverse diverse terrains, but robust controllers require substantial engineering and precise system-dynamics modeling.
- Training-time environment coverage remains a fundamental limitation because policies may fail when deployment conditions differ substantially from those seen during training.
- The proposed system fine-tunes locomotion policies rapidly in real-world conditions without external state estimation or human assistance.
- The robot uses onboard sensors for reward, repeatedly attempts tasks, and improves with each trial, including through learned recovery from falls.
- The system combines simulation pre-training with continued real-world learning and demonstrates autonomous fine-tuning of pacing and side-stepping skills across outdoor and indoor terrains.
II. RELATED WORK
Prior locomotion controllers rely on manual modeling or training-time adaptation mechanisms, but real-world learning systems often require extensive supervision and instrumentation.
- Conventional locomotion controllers use footstep planning, trajectory optimization, and model-predictive control, requiring robot-dynamics characterization and manual design.
- Even robust learned policies may fail to generalize because deployment conditions are difficult to anticipate completely.
- Adaptive policies can perform few-shot adaptation, but their flexibility remains limited by the variability represented during training.
- Existing real-world locomotion training often depends on heavy environmental instrumentation or manual intervention for reward supervision and resets.
III. FINE-TUNING LOCOMOTION IN THE REAL WORLD
The system combines simulation pre-training, off-policy reinforcement learning, motion imitation, and learned resets to support autonomous real-world fine-tuning with limited human intervention.
- The framework uses randomized ensembled double Q-learning to improve sample-efficient real-world reinforcement learning.
- Training collects trajectories with each skill policy, stores them in replay buffers, and updates critics and policies through repeated gradient steps.
- The example system alternates forward or backward pacing with automatic resets, then updates policies using data collected during real-world execution.
- A learned reset policy stitches episodes together by enabling autonomous recovery after falls.
- Motion imitation provides a common way to learn different skills by changing the reference motion used by the reward function.
- Policies are first trained to convergence in simulation, then continue training on real-world data, with skill selection repeated indefinitely.
IV. SYSTEM DESIGN
The implementation uses an A1 quadruped with simulation-based motion references and separate neural policies trained with standard deep-learning infrastructure.
- The system uses Unitree’s A1 robot and PyBullet simulation, with dog-pacing mocap and artist-generated side-step motions as references.
- Each policy is modeled as a separate fully connected neural network and trained with Adam using a learning rate of 10^-4 and batch size 256.
A. State and Action Spaces
The policy uses a short history of proprioceptive state and future reference poses as input, producing joint-level PD targets at 33 Hz. Real-world state estimation combines onboard sensing with filtering, while target commands are low-pass filtered for smoothness.
- The state contains three timesteps of root orientation, joint angles, and previous actions, together with four future target poses.The latest target pose is approximately one second ahead.
- Actions are PD position targets for all 12 joints, applied at 33 Hz.
- A low-pass filter processes PD targets before they are sent to the robot to smooth the motions.
B. Reward Function
The reward function trains motion imitation by combining terms for joint rotations, joint velocities, end-effector positions, root pose, and root velocity. During real-world training, filtered onboard state estimates support these reward components.
- The reward weights are wp = 0.5, wv = 0.05, we = 0.2, wrp = 0.15, and wrv = 0.1.
- The reward encourages the robot to match joint rotations with those of the reference motion.
- Additional terms encourage matching joint velocities and end-effector positions.
- Root pose and root velocity rewards encourage tracking the reference root motion.
- A Kalman filter estimates linear root velocity from IMU acceleration and orientation, correcting it with foot-contact sensors.
C. Reset Controller
The reset controller is trained in simulation from diverse fallen states and learns to return the robot to a standing pose. A single motion-imitation-based policy handles both self-righting and stabilization across test terrains without fine-tuning.
- The reset policy is trained from initial states generated by dropping the robot from random heights and orientations.
- Its objective is to recover the robot to a default standing pose after a fall.
- A single reset policy rewards rolling upright and then applies motion imitation toward a standing pose.
- The learned controller can roll and jump upright after falls or stabilize the robot when it is already upright.
- The reset policy transferred successfully to all test terrains, so it required no fine-tuning.
V. EXPERIMENTS
The experiments evaluate the fine-tuning method against prior simulated and real-world adaptation approaches, examine design choices affecting feasibility, and measure autonomous online improvement across real-world settings.
- The experiments compare fine-tuning with prior approaches based on simulated training, including methods that perform real-world adaptation.
- The study examines how design decisions affect the feasibility of real-world training.
- The experiments measure how much autonomous online fine-tuning improves robotic skills across real-world settings.
A. Simulation Experiments
Simulation experiments show that fine-tuning can continue improving locomotion when test environments differ substantially from training conditions, whereas prior learned adaptation strategies degrade under such shifts.
- Comparison to prior work: The comparison adapts a learned forward pacing gait from simulated pre-training to flat ground, randomized rugged terrain, and low-friction surfaces.All methods are pre-trained with dynamics randomization before deployment for adaptation.
- Comparison to prior work: RMA relies on a pre-trained encoder, while the latent-space method relies on suitable strategies already existing in its latent space.These assumptions limit adaptation when test environments differ sufficiently from training environments.
- REDQ versus SAC: REDQ is used for policy optimization and is compared against a vanilla SAC ablation for both pre-training and fine-tuning.The figure caption reports that REDQ improves over SAC.
- Comparison to prior work: Fine-tuning continuously improves and eventually succeeds on uneven or extremely slippery terrains where prior adaptive policies exhibit significant performance drops.Prior methods perform well when test environments resemble training environments but struggle when deployed on substantially different terrain.
B. Real-World Experiments
Real-world experiments evaluate autonomous fine-tuning across four challenging domains and show substantial improvement with modest real-world data, including reliable locomotion on difficult surfaces.
- Performance across domains: Fine-tuning substantially improves performance in all four real-world domains, with especially pronounced gains on the lawn and memory foam mattress.The experiments cover an outdoor grassy lawn, carpet, a creviced doormat, and memory foam.
- Pacing on lawn: After less than 2 hours of operation, the robot learns to pace forward and backward consistently and stably on the lawn with very few failures.The pre-trained forward policy made little progress, while the backward policy tended to trip and fall.
- Side-stepping indoors: In less than 2.5 hours of training on each indoor terrain, the robot learns to execute side-stepping consistently without stumbling.The pre-trained side-stepping policy tended to twitch violently and fall on carpet, memory foam, and the creviced doormat.
- Autonomous resets: The learned recovery controller achieved 100% successful resets across experiments and was faster than the built-in controller on hard surfaces.The built-in controller was less reliable on memory foam, while occasional workspace drift still required human repositioning.
VI. CONCLUSION
The paper presents a system combining autonomous data collection and data-efficient model-free reinforcement learning to fine-tune locomotion policies in real-world environments. Fine-tuning substantially improves diverse skills, while lifelong adaptation remains future work.
- Conclusion: The system combines autonomous data collection with data-efficient model-free reinforcement learning for real-world locomotion fine-tuning.It provides automated recovery from falls and reward calculation through onboard-sensor state estimation.
- Conclusion: Example rollouts show pacing policies progressing without falling after fine-tuning and side-step policies reliably completing the skill across three domains.Before fine-tuning, the pacing and side-step policies struggled or failed to complete their tasks.
- Conclusion: Fine-tuning enables substantial performance gains and high proficiency even when the initial gait frequently stumbles and falls.The conclusion summarizes results across environments including grass, carpets, doormats, and mattresses.
- Conclusion: The experiments fine-tune in each environment separately, while lifelong learning across complex, diverse, and changing environments is identified as future work.The proposed future system would continue adapting in new environments and gradually perfect skills in familiar ones.