PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, Vincent Vanhoucke

arXiv:1804.10332v2cs.ROcs.AI

TL;DR

Agile quadruped locomotion traditionally requires extensive expertise, while simulation-trained policies often fail to transfer to real robots. The paper presents a deep-RL system with user-controllable gait guidance, improved simulation, and robust policies, successfully deploying trotting and galloping after simulation training.

  • Problem

    Designing agile quadruped locomotion requires extensive expertise and tedious manual tuning, while simulation-trained controllers often face a reality gap when transferred to robots.

  • Method

    The system learns policies in simulation using simple rewards, optionally follows open-loop reference gaits, and narrows the reality gap through improved simulation and robust-controller training.

  • Results

    The system successfully deploys simulation-trained trotting and galloping policies on a real quadruped robot.

  • Takeaways & Limitations

    Deep RL can automatically produce agile locomotion gaits that operate on real robots without further physical-system training.

  • Takeaways & Limitations

    The demonstrated policies use a simple flat-ground running task and do not yet address dynamic speed and direction changes or complex terrain with vision.

Abstract

from arXiv · show

Designing agile locomotion for quadruped robots often requires extensive expertise and tedious manual tuning. In this paper, we present a system to automate this process by leveraging deep reinforcement learning techniques. Our system can learn quadruped locomotion from scratch using simple reward signals. In addition, users can provide an open loop reference to guide the learning process when more control over the learned gait is needed. The control policies are learned in a physics simulator and then deployed on real robots. In robotics, policies trained in simulation often do not transfer to the real world. We narrow this reality gap by improving the physics simulator and learning robust policies. We improve the simulation using system identification, developing an accurate actuator model and simulating latency. We learn robust controllers by randomizing the physical environments, adding perturbations and designing a compact observation space. We evaluate our system on two agile locomotion gaits: trotting and galloping. After learning in simulation, a quadruped robot can successfully perform both gaits in the real world.

I. INTRODUCTION

The paper addresses the difficulty of designing agile quadruped locomotion and the reality gap that often prevents simulation-trained policies from working on real robots. It presents a deep-RL system that learns or user-guides gaits in simulation, improves transfer, and demonstrates trotting and galloping on hardware.

  • Agile quadruped locomotion is difficult to control, and classical methods often require extensive expertise and tedious manual tuning.
  • Simulation-trained controllers often perform poorly on real robots because of discrepancies between simulated and physical dynamics.
  • The proposed system learns policies in simulation, supports controllability from fully learned to user-specified gaits, and deploys them on real robots.
  • System identification, faithful actuator and latency models, dynamics randomization, perturbations, and compact observations are used to narrow the reality gap.
  • The system evaluates trotting and galloping and reports automatically learned gaits that are more energy efficient than expert-handcrafted gaits at the same running speed.
  • The learned trotting and galloping gaits can work directly on robots without further physical-system training.

II. RELATED WORK

Prior locomotion optimization and deep-RL work reduced manual expertise but largely remained limited to simulation or struggled with high-dimensional control. This paper emphasizes real-robot evaluation and methods for narrowing the simulation-to-reality gap.

  • Black-box and Bayesian optimization reduce manual tuning, but high-dimensional locomotion control remains difficult to scale without feature engineering or controller design.
  • The paper uses PPO because it is described as stable, on-policy, and easily parallelized.
  • Earlier reinforcement-learning studies addressed varied locomotion tasks, including quadruped, bipedal, dog, and hierarchical control, primarily in simulation.
  • This work tests simulation-trained policies on real robots and explores approaches to narrow the reality gap.
  • Reality-gap approaches include system identification, improved actuator and latency models, noise or perturbation injection, and dynamics randomization.
  • Other transfer strategies combine simulation and real-world data or adapt simulated policies, including fine-tuning and domain adaptation.

III. ROBOT PLATFORM AND PHYSICS SIMULATION

The system uses the Minitaur quadruped and a PyBullet simulation, with onboard hardware supporting neural-network inference. Locomotion is formulated as a partially observable control problem with compact observations and a leg-space action representation.

  • The Minitaur is a quadruped with eight direct-drive actuators, two sagittal-plane actuators per leg, motor encoders, and an IMU.
  • An Nvidia Jetson TX2 was added because the original microcontroller could not execute deep-RL neural-network policies.
  • The physics simulation is built in PyBullet, which integrates articulated-body dynamics while enforcing contacts, joint limits, and actuator models.
  • Locomotion is formulated as a POMDP because the robot lacks direct measurements of some states, including base position and foot-contact forces.
  • The observation space includes base roll, pitch, angular velocities, and eight motor angles, while excluding drifting yaw and noisy motor velocities.
  • Actions specify each leg’s swing and extension in leg space rather than eight motor angles, allowing simple bounds that exclude invalid self-colliding configurations.

C. Reward Function

The reward balances forward progress against energy consumption, using base displacement along the desired direction and motor power expenditure. Training uses a fixed energy weight and terminates after a time limit or loss of balance.

  • The reward encourages faster forward running while penalizing high energy consumption.
  • Forward progress is measured from consecutive base positions along the desired direction, while energy expenditure uses motor torques and velocities.
  • The energy-weight parameter is fixed at w = 0.008, and episodes end after 1000 steps or when the base tilts beyond 0.5 radians.
  • The leg pose is represented by swing and extension, where extension changes leg length and swing changes overall leg rotation.

D. Policy Representation

The policy combines user-provided open-loop gait references with learned feedback, allowing controllability from fully specified gaits to policies learned from scratch while addressing the simulation-to-reality challenge.

  • Controllability: The hybrid policy gives users control over gait details such as style or ground clearance while learning handles remaining control requirements.This separates manually specified gait characteristics from feedback behaviors that are tedious to design manually.
  • Policy structure: The controller decouples an open-loop reference trajectory from feedback that adjusts leg poses using observations.The open-loop component is typically periodic, while the feedback component learns adjustments such as balance control.
  • Controllability: Users can continuously vary policy controllability from fully user-specified to entirely learned from scratch.Setting the feedback bounds to zero yields a user-specified policy, while setting the open-loop signal to zero and widening feedback outputs enables learning from scratch.
  • Learning algorithm: The feedback component is represented by a neural network and optimized with Proximal Policy Optimization.The network uses two fully connected hidden layers whose size is selected through hyperparameter search.
  • System goal: The system addresses both controllable locomotion-policy learning and transfer of policies from simulation to the physical system.The paper frames these as the two main challenges of its complete agile-locomotion learning system.

A. Improving Simulation Fidelity

The system improves simulation fidelity by identifying robot parameters, modeling actuator behavior more faithfully, and reproducing sensing and control latency that can destabilize real-world feedback policies.

  • System identification: System identification measures robot geometry, mass, centers of mass, and motor friction to reduce discrepancies between simulated and real dynamics.Inertia is estimated from shape and mass under a uniform-density assumption when direct measurement is difficult.
  • Actuator model: The simulator’s constraint-based motor model can remain stable under large gains even when corresponding real motors oscillate.The discrepancy arises because the simulated constraint enforces end-of-step angle and velocity, unlike a PD servo using current state.
  • Actuator model: The actuator model follows ideal DC-motor dynamics and adds a piece-wise linear torque-current relation to represent torque saturation.The model uses PWM-derived current and actuator parameters such as torque constant and armature resistance.
  • Validation: The actuator model’s simulated motor trajectory agrees with ground truth in a sine-trajectory validation experiment.The comparison is reported in Figure 4.
  • Latency: Latency matters because instantaneous simulated feedback makes controller stability appear larger than it is on hardware.The paper links this mismatch to simulated policies oscillating, diverging, and failing in the real world.
  • Latency: Latency is modeled by interpolating delayed observations from a history of measurements before the controller receives them.The physical system has different measured delays for the microcontroller and the Nvidia Jetson TX2.

B. Learning Robust Controllers

The system seeks robust transfer by exposing policies to varied dynamics and perturbations while limiting observation complexity to reduce overfitting to simulation-specific details.

  • Robustness strategies: Robust-controller training uses randomized dynamic parameters, random perturbation forces, and a compact observation space.These three approaches target robustness to model error and improve transferability to the real world.
  • Dynamics randomization: Dynamic parameters are randomly sampled within specified ranges at the beginning of each training episode.The randomized parameters and ranges are summarized in Table I.
  • Dynamics randomization: Randomization trades optimality for robustness, so parameter ranges are selected carefully to avoid overly conservative running gaits.The paper uses more conservative ranges for measured mass and friction than for uncertain inertia estimates and time-varying quantities.
  • Perturbations: Training applies random base perturbations every 200 simulation steps, lasting ten steps with forces ranging from 130N to 220N.These disturbances knock the simulated robot out of balance so it must learn recovery behaviors.
  • Observation design: A compact observation space leaves less opportunity for policies to overfit unimportant simulation details.The paper presents a detailed analysis of observation-space choice in Section VI-B.
  • Evaluation setup: The paper evaluates trotting and galloping learning curves alongside task-specific learning parameters.Figures 5 and Table II provide the corresponding task-level evaluation materials.

VI. EVALUATION AND DISCUSSION

The evaluation trains locomotion policies in simulation for trotting and galloping using neural-network policy and value functions and task-specific rollout-based updates.

  • Tasks: The system is tested on two locomotion tasks: galloping and trotting.The learned gaits are evaluated after training in simulated Minitaur environments.
  • Learning setup: Policy and value functions use fully connected neural networks with two hidden layers.Network sizes are selected through hyperparameter search.
  • Learning setup: Each policy-update iteration collects simulated experience from 25 roll-outs of up to 1000 steps each.The rollout data are used during the simulation-based learning process.

A. Locomotion Tasks

The system learns galloping from scratch and uses open-loop signals to guide trotting. Learned gaits transfer to real robots and use less power than expert-handcrafted gaits at comparable speeds.

  • Galloping: 1.34 m/s in simulation and 1.18 m/s on the real robot were achieved by automatically learned galloping.These correspond to 2.48 and 2.18 body lengths per second, respectively.
  • Trotting: Open-loop signals let users specify trotting, while feedback control maintains balance and keeps the learned gait similar to that preference.The open-loop controller alone cannot produce forward movement in the real world.
  • Energy comparison: 35% and 23% reductions in power consumption were obtained for galloping and trotting, respectively, versus expert-handcrafted gaits.The learned gaits were reported to be as fast as the carefully tuned expert gaits.

B. Narrowing the Reality Gap

The study evaluates how simulation fidelity, randomization, and observation-space design affect sim-to-real transfer. Accurate simulation and robust, compactly observed controllers reduce the reality gap, while randomization trades peak performance for robustness.

  • Measuring the Reality Gap: The reality gap is measured as the difference between expected returns in simulation and real experiments.The expected return averages performance over nine runs, while success rate alone would not capture speed and energy consumption.
  • Simulation Fidelity: Improved simulation with random perturbations transferred comparably between simulation and the robot, unlike baseline simulation groups.The improved setup used an accurate simulator, whereas baseline groups retained a large simulation-to-robot performance gap.
  • Simulation Fidelity: Accurate actuator modeling and latency simulation were both necessary for learned controllers to work on the real robot.Removing either component prevented successful real-robot operation.
  • Randomization: Controllers trained without randomization achieved higher peak performance but could degrade substantially when actual body inertia differed from training.Randomization improved performance consistency across different physical environments.
  • Randomization: Randomization reduced mean return and standard deviation across test environments, revealing a robustness–optimality trade-off.Lower mean reflected conservative behavior, while lower standard deviation indicated less performance variation across environments.
  • Observation Space: All nine runs of the top three randomized controllers with compact observations trotted more than three meters and balanced for the entire episode.Randomization helped narrow the reality gap regardless of observation choice, with the compact observation space producing the best real-world results.
  • Observation Space: Large observation spaces performed better in simulation but worse on the robot, widening the reality gap because observation distributions differed between domains.The small observation space was 4D and contained IMU information; the large space was 12D and also included motor angles.

VII. CONCLUSION

The paper presents deep reinforcement learning as a way to learn agile quadruped locomotion automatically, either from scratch or with user guidance. With an accurate physical model and robust controllers, the learned trotting and galloping policies transfer to real robots, while broader navigation capabilities remain future work.

  • VII. CONCLUSION: Deep RL learns agile quadruped locomotion automatically from scratch while allowing users to guide the learning process.The system develops trotting and galloping gaits for a quadruped robot.
  • VII. CONCLUSION: Accurate physical modeling and robust controllers enable deployment of policies learned in simulation on real robots.The conclusion links successful transfer to both simulator accuracy and controller robustness.
  • VII. CONCLUSION: The current system uses a simple reward and flat-ground environment focused on transferable locomotion policies.The paper identifies dynamic speed and direction changes, complex terrain, and vision-based sensing as future directions.
Loading 1804.10332v2…