Source-linked AI summary
Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion
Zhaoming Xie, Xingye Da, Michiel van de Panne, Buck Babich, Animesh Garg
TL;DR
Because legged-robot policies are commonly trained in simulation and the role of dynamics randomization has produced conflicting findings, this paper studies the issue with Laikago experiments. It finds successful transfer without randomization in its settings and argues that randomization should be reserved for identified problems.
Problem
Legged-robot policies are often trained in simulation, where the reality gap can hinder transfer, and prior findings conflict on whether dynamics randomization is needed.
Source: §I · p. 1Method
The authors study dynamics randomization and policy-design choices through sim-to-sim and sim-to-real experiments with the Unitree Laikago.
Source: §I · p. 1Results
Dynamics randomization was not necessary for successful transfer across multiple gaits and speeds in the authors’ settings, while unnecessary randomization could yield conservative policies with limited robustness gains.
Source: §I · p. 1, C. Summary · p. 6Takeaways & Limitations
The authors recommend identifying high-sensitivity sim-to-real bottlenecks and applying modeling or randomization only to relevant parameters.
Source: §VII · p. 6
Abstract
from arXiv · showhide
Understanding the gap between simulation and reality is critical for reinforcement learning with legged robots, which are largely trained in simulation. However, recent work has resulted in sometimes conflicting conclusions with regard to which factors are important for success, including the role of dynamics randomization. In this paper, we aim to provide clarity and understanding on the role of dynamics randomization in learning robust locomotion policies for the Laikago quadruped robot. Surprisingly, in contrast to prior work with the same robot model, we find that direct sim-to-real transfer is possible without dynamics randomization or on-robot adaptation schemes. We conduct extensive ablation studies in a sim-to-sim setting to understand the key issues underlying successful policy transfer, including other design decisions that can impact policy robustness. We further ground our conclusions via sim-to-real experiments with various gaits, speeds, and stepping frequencies. Additional Details: https://www.pair.toronto.edu/understanding-dr/.
I. INTRODUCTION
The paper revisits when dynamics randomization is useful for Laikago locomotion, using sim-to-sim and sim-to-real experiments. It reports successful transfer without randomization in its settings and argues for targeting identified sources of failure.
I. INTRODUCTION
- Prior work reports both successful sim-to-real transfer with dynamics randomization and cases where it was unnecessary, leaving its role unsettled.The paper situates this question within legged-robot RL, where simulation is commonly used because hardware training is impractical and unsafe exploration is a concern.Source: §I · p. 1
- Dynamics randomization was not necessary for successful Laikago sim-to-real transfer across multiple gaits and speeds in the authors’ settings.The policies also remained robust to common perturbations, despite prior work reporting failure for the same robot model and motion class.Source: §I · p. 1
- The study evaluates whether randomization is needed and analyzes design choices that can prevent successful transfer even when randomization is used.It reports that unnecessary randomization can produce conservative policies with limited robustness gains, while randomizing genuinely problematic parameters can bridge the reality gap.Source: §I · p. 1
B. Sim-to-real for Robot Control
Prior robot-control work uses dynamics randomization to improve robustness to modeling errors, but also reports transfer without it. This paper studies the Laikago and describes a control policy based on robot state and reference tracking.
B. Sim-to-real for Robot Control
- Dynamics randomization varies simulated parameters such as link masses, inertial moments, latency, and actuator response to seek robustness to modeling errors.Prior work also reports successful transfer without randomization through appropriate design choices.Source: B. Sim-to-real for Robot Control · p. 2
C. Robust Control
- Other robust-control approaches include model ensembling, optimization perturbations, and feedback tracking of an open-loop trajectory.The cited approaches obtain robustness through different control-synthesis strategies.Source: C. Robust Control · p. 2
III. CONTROL POLICIES
- The study uses the Laikago to examine dynamics randomization and the importance of appropriate design choices.The policy state comprises the robot’s base position and orientation, twelve joint angles, and their velocities.Source: C. Robust Control · p. 2, §III · p. 2
A. Training Environment
The training setup uses desired gait trajectories to shape a tracking reward, while the policy receives robot state and user commands and outputs residual joint targets for PD control.
A. Training Environment
- For quadrupedal gait policies using a reference trajectory χ of desired robot states, the timestep-t reward is r_t = 0.4rj_t + 0.3rp_t + 0.3ro_t.The trajectory χ = {x̂0, x̂1, …} specifies desired robot states at each timestep.Source: A. Training Environment · p. 2
- The policy is trained to maximize cumulative discounted reward.The policy input omits x and y coordinates and includes gait phase, gait type, frequency, and desired speed.Source: A. Training Environment · p. 2
- The policy receives robot state and user commands, then outputs residual joint targets added to reference targets for a joint PD controller.The controller uses gains kp = 40 and kd = 0.5; the policy runs at 40 Hz and the PD controller updates torque at 500 Hz.Source: A. Training Environment · p. 2, Figure 2 · p. 2
B. Gaits
Gait-specific reference trajectories vary contact phases, speed, and phase duration to produce different motions. Policies are trained with PPO in a GPU-accelerated simulator using many parallel robots.
B. Gaits
- Swing-leg reference angles vary with phase time and desired forward speed, while stance legs use fixed reference angles.The swing trajectories are designed for high foot clearance and forward or backward motion to adapt to velocity.Source: B. Gaits · pp. 2–3
- Walking, trotting, and pacing reference trajectories vary desired speed and phase duration to cover different stepping frequencies.The study demonstrates transfer with all three gaits but focuses evaluation on trotting and pacing.Source: B. Gaits · p. 3
C. Training Setup
- Policies use actor-critic training with proximal policy optimization and a two-layer feedforward network with 128 hidden units.Actions are stochastic during training and deterministic at test time.Source: C. Training Setup · p. 3
- Training uses Isaac Gym with 1600 simulated robots in parallel and up to 2.4 × 10^8 samples per policy.Training takes 6 to 8 hours on one GeForce RTX 2080 Ti GPU.Source: C. Training Setup · p. 3
D. Physical Robot Setup
The paper describes the physical-robot state estimation and control setup, then tests how policies trained without dynamics randomization handle modeling errors and perturbations. In the authors’ setting, the default policy already copes with modeling errors larger than prior randomization ranges.
D. Physical Robot Setup
- Physical-robot state estimates use an onboard-sensor Kalman filter, with velocity and yaw offsets applied to counter drift and initialization error.The offsets can also command sideways or turning motion; the authors attribute these implicit control abilities to encountering similar states during stochastic training.Source: D. Physical Robot Setup · p. 3
- Target joint angles update every 4 ms on hardware rather than every 26 ms in training, then pass through a low-pass filter with λ = 0.2.The faster hardware update is possible because policy queries typically take 2–3 ms; filtering smooths target angles before the PD controller.Source: D. Physical Robot Setup · p. 3
IV. DYNAMIC RANDOMIZATION IS NOT NECESSARY
- Policies trained without dynamics randomization coped in simulation with modeling errors larger than prior sim-to-real randomization ranges, and transferred successfully to the physical robot.The paper tests balance under mass, gain, latency, push, and slope perturbations.Source: §IV · p. 3
A. Types of Perturbation
- Robustness tests record the maximum perturbation the robot withstands, counting balance for over 10 seconds as a pass rather than measuring command-following reward.Tests cover mass, proportional gain, latency, lateral pushes, and slopes; physical push testing is qualitative because prescribed-force hardware is unavailable.Source: A. Types of Perturbation · pp. 3–4
B. Robustness Tests in Simulation
Policies trained without dynamics randomization handled perturbations beyond ranges typically used in related work, and their tested robustness covered plausible modeling errors. The authors evaluated trotting and pacing across repeated trials.
B. Robustness Tests in Simulation
- Up to twice the body mass could be carried during pacing, compared with the 20 percent randomization typically used in related work.The manufacturer-measured body mass typically has less than 5 percent error.Source: B. Robustness Tests in Simulation · p. 4
- Policies across trotting and pacing already handled perturbation ranges larger than those typically used in related sim-to-real work.The reported ranges also cover possible modeling error; latency is typically less than 4 ms, significantly below what the policy can handle in simulation.Source: B. Robustness Tests in Simulation · p. 4
- Three policies per gait were tested, with mean and sample standard deviation computed for the perturbation thresholds.The measured threshold was the maximum perturbation at which the robot could still complete the task.Source: B. Robustness Tests in Simulation · p. 4
C. Tests on the Physical Robot
Policies transferred directly to the physical robot without adaptation, although the sim-to-real robustness gap differed by gait. The paper relates transfer success to design choices and notes that dynamics randomization alone may not close the gap.
C. Tests on the Physical Robot
- Policies transferred to the physical robot without adaptation, including trotting and pacing policies shown on slopes and under forceful pushes.The policies had not encountered slopes or pushes during training.Source: C. Tests on the Physical Robot · p. 4, Figure 3 · p. 4
- Trotting robustness results were similar in simulation and on hardware, while pacing typically performed worse on hardware; both nevertheless handled a large perturbation range.The authors note that the reality gap depends on the motion, so results may differ for more dynamic motions such as gallops or jumps.Source: C. Tests on the Physical Robot · p. 4
A. Design Choices
- The paper investigates observation and proportional-gain design differences as explanations for contrasting transfer outcomes, noting that its setup otherwise resembles prior work in trajectories, motion classes, and robot model.In, observations are raw sensory measurements; this work uses a state estimator and includes body velocity and joint velocity. The PD gains are kp = 220 in and kp = 40 in this work.Source: §V · p. 4, A. Design Choices · pp. 4–5
B. Simulation Test
The simulation ablation tests whether dynamics randomization improves robustness across design choices and perturbation dimensions. It generally did not, with latency randomization as an exception.
B. Simulation Test
- The ablation trained pacing policies without velocity feedback and trotting policies with kp = 160, each with and without dynamics randomization.Three policies were trained under each setting, and lateral pushes and slopes were left unrandomized to test cross-correlations in robustness.Source: B. Simulation Test · p. 5
- The high-gain trot used kp = 160 rather than kp = 220 because the higher gain can cause instability on the robot.The authors used the lower gain while testing whether robustness on randomized dimensions influenced unrandomized dimensions.Source: B. Simulation Test · p. 5
- Dynamics randomization did not improve observed robustness in most tests, including along randomized dimensions; latency randomization was the exception.The authors conclude that much of the tested randomization was unnecessary or potentially harmful.Source: B. Simulation Test · p. 5
C. Robot Test
In the robustness tests, policies without velocity feedback or with kp = 160 fail sim-to-real transfer and generally perform worse than the default. The robot tests show that missing lateral velocity feedback and high proportional gain are problematic design choices.
C. Robot Test
- Alternative design choices fail physical-robot transfer even with dynamics randomization.This matches prior results that used additional on-robot adaptation to cross the reality gap.Source: C. Robot Test · p. 5
- Policies without lateral velocity feedback cannot control lateral velocity, often fall sideways, and remain unsuccessful when trained with lateral pushes.Push training does not compensate for the missing velocity feedback.Source: C. Robot Test · p. 5
D. Comparison of Proportional Gain
- With kp = 160, trotting motion is stiffer and more contact-timing-sensitive than with kp = 40, sometimes causing backward kicks before stance contact.The comparison associates kp = 40 with larger tracking errors and torque-controller-like behavior, and kp = 160 with smaller errors and position-controller-like behavior.Source: C. Robot Test · p. 5, D. Comparison of Proportional Gain · p. 5, Figure 4 · p. 5
VI. EFFECT OF DYNAMICS RANDOMIZATION
- Policies without velocity feedback or with kp = 160 fail sim-to-real tests and generally perform worse in robustness tests than the default.The table compares robustness tests across training setups and reports attempted sim-to-real transfer.Source: Table III · p. 6
A. Dynamics Randomization Produces Conservative Policies
Dynamics randomization can make policies overly conservative when it targets parameters that do not matter, but randomizing a parameter behind a significant modeling error can help cross the reality gap. The authors recommend limiting randomization or additional modeling to relevant parameters.
A. Dynamics Randomization Produces Conservative Policies
- Dynamics randomization can produce overly conservative policies, with unnecessary latency robustness compromising performance and robustness on other dimensions.The physical robot's estimated latency is less than 4 ms.Source: A. Dynamics Randomization Produces… · p. 6
- Trotting policies reach a maximum speed of 0.9 m/s with randomization and 1.1 m/s without randomization, in simulation and on the physical robot.The authors interpret the lower maximum speed as evidence that randomization can produce conservative policies.Source: A. Dynamics Randomization Produces… · p. 6
B. Randomize Parameters that Matter
- Policies trained with randomized latency handle latency up to 32 ms, whereas policies without randomization fail above 17 ms.The randomized-latency policy was trained with latency up to 20 ms.Source: B. Randomize Parameters that Matter · p. 6
C. Summary
- The authors recommend randomizing or modeling only parameters that matter when significant modeling errors are present; superfluous randomization measurably harms performance and may add no robustness.Actuator modeling errors are another potential sim-to-real challenge.Source: C. Summary · p. 6
VII. DISCUSSION AND FUTURE WORK
The discussion concludes that commonly used dynamics randomization often provides negligible robustness gains in this quadrupedal setting, while proportional gain and state-observation choices also affect transfer. The authors note that the findings may not extend to more dynamic motions or other robot morphologies.
VII. DISCUSSION AND FUTURE WORK
- Commonly used dynamics randomization often offers negligible robustness improvements in this quadrupedal-locomotion setting.The study also evaluates proportional-gain stiffness and state-observation choices that may have been overlooked despite their role in transfer success.Source: §VII · p. 6
- Other tasks or robot morphologies may require additional modeling or randomization; examples include actuator modeling for ANYmal and latency randomization for Ghost Minitaur [1].The authors recommend identifying high-sensitivity sim-to-real bottlenecks and targeting the relevant parameters.Source: §VII · p. 6
- The reality gap may be more pronounced for dynamic motions such as running or jumping, and empirically tuned parameters such as proportional gains can vary across robots.Source: §VII · p. 6
- State-history or recurrent policies can address challenging terrain without exteroceptive sensing, but training them requires randomizing the external environment rather than robot parameters.The authors treat this as consistent with randomizing only a minimal parameter set.Source: §VII · p. 6