Source-linked AI summary
Data Efficient Reinforcement Learning for Legged Robots
Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Tingnan Zhang, Jie Tan, Vikas Sindhwani
TL;DR
Physical legged-robot locomotion requires data-efficient learning that remains accurate over long horizons and safe during exploration. The paper learns dynamics for model-predictive control, adds multi-step modeling, latency compensation, and trajectory-based action constraints, and achieves walking from 4.5 minutes of data while supporting new locomotion tasks.
Problem
Learning locomotion on physical robots is limited by the large data demands of model-free methods and the risks of random exploration for actuators.
Method
The framework learns long-horizon dynamics and uses MPC with multi-step loss, latency compensation, and trajectory-based smooth-action exploration.
Results
36 rollouts, corresponding to 4.5 minutes or 45,000 control steps, produce walking, while the learned model generalizes to new tasks without additional data or fine-tuning.
Takeaways & Limitations
The learned dynamics model can be reused for different locomotion behaviors, and the approach is more than an order of magnitude more sample efficient than model-free methods.
Takeaways & Limitations
Future work must address model misspecification, predictive uncertainty, online adaptation, and integration with perceptual modules for more agile and autonomous behaviors.
Abstract
from arXiv · showhide
We present a model-based framework for robot locomotion that achieves walking based on only 4.5 minutes (45,000 control steps) of data collected on a quadruped robot. To accurately model the robot's dynamics over a long horizon, we introduce a loss function that tracks the model's prediction over multiple timesteps. We adapt model predictive control to account for planning latency, which allows the learned model to be used for real time control. Additionally, to ensure safe exploration during model learning, we embed prior knowledge of leg trajectories into the action space. The resulting system achieves fast and robust locomotion. Unlike model-free methods, which optimize for a particular task, our planner can use the same learned dynamics for various tasks, simply by changing the reward function. To the best of our knowledge, our approach is more than an order of magnitude more sample efficient than current model-free methods.
1 Introduction
The paper addresses the data and safety challenges of learning locomotion on physical legged robots. Its model-based framework combines long-horizon modeling, latency-aware planning, and safe exploration to learn efficient walking from limited robot data.
- Motivation: Model-free locomotion learning often requires too much physical-robot data, while random exploration can stress actuators and cause mechanical failures.These challenges motivate a more data-efficient and safer learning framework.
- Approach: The framework uses a multi-step loss to reduce long-horizon prediction error, asynchronous control to compensate for planning latency, and trajectory-based exploration for smooth actions.These components target model accuracy, real-time control, and actuator safety.
- Results: 36 rollouts provide 4.5 minutes of data, or 45,000 control steps, for a Minitaur robot to learn walking from scratch.The full robot experimentation time is approximately 10 minutes, including setup overhead.
- Results: The method is at least an order of magnitude more sample efficient than the compared on-robot model-free method and generalizes to new tasks without additional data or fine-tuning.The same learned dynamics model supports task changes through planning.
2 Model-learning and Model-Predictive Control Loop
The system formulates locomotion as an MDP and learns a deterministic dynamics model from collected trajectories. The planner uses this model to optimize cumulative reward while replay-buffer retraining addresses mismatch between training and deployment states.
- Model-based formulation: The learned deterministic dynamics model approximates state transitions from collected state-action trajectories.It predicts the next state from the current state-action pair.
- Model-based formulation: An action planner uses the learned model to optimize cumulative reward in the locomotion MDP.The MDP includes state and action spaces, transition dynamics, an initial-state distribution, and a reward function.
- Model updates: The model is periodically retrained on all trajectories stored in a replay buffer to address distribution mismatch between training data and planner-selected states.This reduces reliance on only the most recently collected trajectories.
3 Model-based Learning for Locomotion
The locomotion controller combines multi-step dynamics learning with MPC, efficient smooth-action planning, parallel execution, and latency compensation. Together, these choices improve long-horizon planning and real-time control on the robot.
- 3.1 Accurate Dynamics Modeling with Multi-step Loss: The dynamics model predicts state differences with a neural network and is trained using recursively predicted multi-step trajectories.Recursive predictions measure error accumulation rather than only one-step accuracy.
- 3.1 Accurate Dynamics Modeling with Multi-step Loss: As the prediction horizon n increases, the multi-step loss emphasizes accuracy over multiple steps for long-horizon planning.When n = 1, the loss reduces to the single-step loss.
- 3.2 Efficient Planning of Smooth Actions: MPC repeatedly replans from the latest robot state, reducing sensitivity to model inaccuracies while making planning speed critical.Only the current planning cycle needs to be optimized before execution continues.
- 3.2 Efficient Planning of Smooth Actions: CEM samples action sequences, fits a normal distribution to the best samples, and iteratively resamples to optimize actions without model gradients.The method is derivative-free, parallelizable, and able to handle non-smooth rewards.
- 3.2 Efficient Planning of Smooth Actions: Time-correlated exploration noise favors smooth, periodic action sequences instead of jerky independently sampled motions.The sampled actions retain the desired normal distribution while correlating noise across consecutive timesteps.
- 3.3 Online Replanning in the Presence of Latency: Parallel planning and execution decouple planning frequency from control frequency, allowing actions to be applied at a higher control rate.The execution thread applies actions from the previous planning step while planning continues in the background.
- 3.3 Online Replanning in the Presence of Latency: Asynchronous control predicts the future state at planning completion and plans from that state, aligning the planner's input with the state where actions execute.This compensates for latency T and improves plan quality during delayed execution.
4 Safe Exploration with Trajectory Generators
Trajectory generators provide smooth, periodic leg-extension trajectories for safe exploration, while the planner adds residuals and independently adjusts phases to coordinate more complex gaits.
- Trajectory generators: Trajectory generators output periodic leg-extension trajectories that encourage smooth actions and reduce the risk of damaging the robot.They are used instead of directly commanding motor angles, which can cause abrupt changes and large torque outputs.
- Trajectory generators: Each trajectory generator maintains a phase and alternates between stance and lift modes with distinct amplitudes.The resulting leg-extension function is continuous as the phase is rescaled.
- Planner interaction: The planner uses a 12-dimensional action space combining eight leg swing-and-extension residuals with four trajectory-generator phase scales.Residuals complement trajectory-generator outputs, while phase scales propagate each leg’s phase independently.
- Planner interaction: Independent phase control lets the controller synchronize arbitrary leg pairs and coordinate varied gait patterns.The planner can modulate trajectory-generator phases while adding residual corrections for more complex behaviors.
5.1 Experimental Setup
Experiments use a Minitaur quadruped with sensor-based state estimation, a 6 ms control timestep, and CEM-based MPC configured for fast replanning over a 450 ms horizon.
- Hardware and control: The experiments use a Minitaur robot whose desired leg trajectories are converted to motor positions and tracked by a PD controller.The controller operates with a timestep of 6 ms.
- State and dynamics: The 18-dimensional state combines trajectory-generator state, base velocity, IMU readings, and motor positions.The dynamics model receives a history of four observations to account for hardware latency and partial observability.
- MPC configuration: CEM plans over 75 control steps, corresponding to a 450 ms horizon, using 5 iterations and 400 samples per iteration.The implementation runs on a Nvidia GTX 1080 GPU and replans every 72 ms.
- Training task: Training episodes require the robot to follow a forward speed profile rising from zero to 0.66 m/s within 3 seconds.The reward penalizes speed-tracking error, lateral deviation, and base roll, pitch, and yaw.
5.2 Learning on Hardware
On hardware, the learned model produces forward walking from limited real-robot data, tracks the target speed, develops coordinated periodic gaits, and remains upright on an unseen slope.
- Learning performance: 36 episodes, or 45,000 control steps, suffice to learn a forward-walking gait from real-robot data.This corresponds to approximately 10 minutes of robot time, while the full process including offline model training takes less than one hour.
- Learning performance: The robot tracks a desired speed of 0.66 m/s, equivalent to 1.6 body lengths per second.The reported speed is twice the fastest speed achieved by the cited baseline.
- Online model improvement: Interleaving data collection with model updates improves planning as the learned dynamics become more accurate in the planner’s utilized state space.Initially, a model trained only on random trajectories yields inaccurate predictions and slow forward motion.
- Gait development: MPC develops a periodic gait by swinging the four legs in succession while trajectory generators provide the underlying leg-extension motion.MPC controls swing angles and coordinates individual-leg phases.
- Robustness: On an unseen slope, the robot maintains a periodic gait and remains upright most of the time despite slight pitch perturbations.The slope includes both uphill and downhill walking, although it was not used during training.
5.3 Generalization to Unseen Tasks
A dynamics model trained only for forward walking can support MPC on new reward-defined tasks, including backward walking and turning, without additional model training.
- Task generalization: The learned dynamics model is tested on new tasks by changing the reward function while retaining the model trained for forward walking.The turning task uses a reward based on tracking a desired turning rate while penalizing other terms.
- Task generalization: The model supports zero-shot generalization to related tasks, including walking backwards and turning.MPC plans these behaviors using the dynamics model learned from forward walking data.
5.4 Ablation Study
The ablations show that long-horizon modeling, latency-aware planning, trajectory generators, and tuned CEM parameters each matter for effective locomotion control. The comparison also places the method's sample efficiency well beyond model-free baselines.
- Generalization to Unseen Rewards: The learned dynamics model supports backward walking and turning under new cost functions despite training only on forward walking.Figure 5 evaluates backward-speed tracking and turning left or right at 15 degrees per second while retaining the existing dynamics model.
- Dynamics Modeling with Multi-step Loss: Multi-step loss improves long-horizon dynamics prediction and enables higher-reward MPC control; n = 20 balances model accuracy and training time.Increasing propagated timesteps makes predicted velocity increasingly match the ground truth, while omitting multi-step loss prevents accurate long-horizon tracking.
- Asynchronous CEM Controller: Without asynchronous control, MPC tracks desired speed only up to approximately 0.4m/s, limiting fast locomotion.Planning must use an accurate state because robot states can change rapidly within a few timesteps at higher speeds.
- CEM Hyperparameters: CEM requires at least 5 iterations for optimal performance, while excessive action smoothing can make legs overly compliant and overly long horizons expose model imperfections.The ablation evaluates average return over five episodes using a shared dynamics model.
- Role of Trajectory Generators: Trajectory generators smooth motor actions and make leg behavior periodic, while removing them makes real-robot learning unstable and quickly overheats the motors.With and without trajectory generators, rollouts achieve similar total reward, but trajectory generators improve action regularity and learning stability.
- Comparison with Model-free Algorithms: PPO requires nearly 1000 times more samples for similar reward, while SAC requires an order of magnitude more samples and has a less stable learning curve.The comparison uses learning curves against model-free implementations obtained from TF-Agents.
6 Perspectives and Future Work
The approach achieves fast, robust quadruped gaits from only 4.5 minutes of real-world data and reuses learned dynamics for new locomotion behaviors. Future work targets model misspecification, uncertainty-aware exploration, greater agility, and perceptual autonomy.
- Perspective: 4.5 minutes of real-world data collection induces robust and fast quadruped gaits, with learning efficiency more than an order of magnitude superior to model-free methods.The learned dynamics model can also be reused to induce new locomotion behaviors.
- Future Work: Future work must address model misspecification, predictive uncertainty, real-time online adaptation, and integration of vision and contact sensing.These directions are presented as necessary for more agile, complex behaviors and greater autonomy.