Source-linked AI summary
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel
TL;DR
Simulation-trained control policies often fail to transfer because simulated and real dynamics differ, while physical training is costly and raises safety concerns. The paper randomizes simulator dynamics to train adaptive policies, demonstrating direct deployment on a real robot for object pushing with comparable performance and robustness to altered contact dynamics.
Problem
Policies learned in simulation can be specific to simulator characteristics, while real-world reinforcement-learning training is limited by sample complexity and safety concerns.
Method
The paper trains recurrent policies exclusively in simulation while randomizing dynamics parameters to promote adaptation to unfamiliar physical dynamics.
Results
The simulation-only policies transfer directly to a real robot for object pushing, with success rates of 0.91 ± 0.04 under altered contact dynamics and 0.89 ± 0.06 without the attachment.
Takeaways & Limitations
Dynamics randomization enables real-robot deployment without physical-system training and supports adaptation to contact-dynamics changes and calibration error.
Abstract
from arXiv · showhide
Simulations are attractive environments for training agents as they provide an abundant source of data and alleviate certain safety concerns during the training process. But the behaviours developed by agents in simulation are often specific to the characteristics of the simulator. Due to modeling error, strategies that are successful in simulation may not transfer to their real world counterparts. In this paper, we demonstrate a simple method to bridge this "reality gap". By randomizing the dynamics of the simulator during training, we are able to develop policies that are capable of adapting to very different dynamics, including ones that differ significantly from the dynamics on which the policies were trained. This adaptivity enables the policies to generalize to the dynamics of the real world without any training on the physical system. Our approach is demonstrated on an object pushing task using a robotic arm. Despite being trained exclusively in simulation, our policies are able to maintain a similar level of performance when deployed on a real robot, reliably moving an object to a desired location from random initial configurations. We explore the impact of various design decisions and show that the resulting policies are robust to significant calibration error.
I. INTRODUCTION
Deep reinforcement learning offers rich simulated control capabilities, but transferring them to physical systems is hindered by sample complexity, safety concerns, and the simulation–reality gap. The paper shows that low-fidelity simulation with dynamics randomization can train policies for direct real-robot deployment.
- DeepRL enables agents to perform diverse and challenging control tasks in simulation.
- High sample complexity and real-world safety concerns often preclude directly training modern DeepRL algorithms on physical systems.Simulation alleviates some of these obstacles but introduces a mismatch between simulated and real environments.
- Dynamics randomization using low-fidelity simulations can develop policies transferable directly to the real world.
- A policy trained exclusively in simulation successfully performs an object-pushing task with a real robot without additional physical-system training.
B. Domain Randomization
Domain randomization treats source-domain discrepancies as variability during simulation training, supporting adaptation across environments and dynamics. The paper positions memory-based policies as better suited than memoryless policies for handling mismatch and calibration error.
- Domain randomization models discrepancies between source and target domains as variability in the source simulation.It has been applied to visual properties and system dynamics for transfer without requiring real images or physical fine-tuning.
- Randomized dynamics can transfer manipulation policies directly to physical robots without additional fine-tuning, but memoryless policies have limited adaptation to mismatch.
- Memory-based policies cope with greater training variability and generalize better to real-world dynamics, including significant calibration error.
- Pushing positions and orients objects that are too large or heavy to grasp, but complex contact dynamics make the skill difficult to model accurately.Friction variation and sensitivity to initial contact configurations complicate planning and motivate data-driven or simulation-based approaches.
- In reinforcement learning, environment dynamics determine the consequences of actions and the behaviors that can be realized.
A. Policy Gradient Methods
A universal policy conditions actions on both the current state and a sampled goal, while policy-gradient methods optimize parametric policies by ascending an estimate of expected-return gradients.
- A universal policy receives a goal g in addition to the state, with the pushing-task goal specifying an object’s target location.
- Policy-gradient methods learn parametric policies by estimating the objective gradient and using gradient ascent to maximize expected return.
B. Hindsight Experience Replay
Hindsight Experience Relay addresses the difficulty of learning from sparse binary rewards by replaying trajectories with goals they actually achieved. The paper combines this strategy with simulated dynamics modeling to improve transfer toward real-world dynamics.
- Hindsight Experience Relay: Sparse binary rewards are challenging because unsuccessful trajectories provide the same −1 reward at every timestep.This gives the agent little information for adjusting its actions.
- Hindsight Experience Relay: HER reassigns a trajectory’s goal to one satisfied in its final state, then recomputes rewards under that new goal.A trajectory unsuccessful under its original goal can become successful under the replayed goal.
- Simulation-to-Real Transfer: The training objective uses an approximate physics-simulation dynamics model because sampling real-world dynamics can be prohibitive.The approximate model is intended to represent the real-world transition dynamics despite modeling and calibration error.
- Simulation-to-Real Transfer: Training policies across variable dynamics is proposed to improve generalization to real-world dynamics.The approach targets policies that adapt to variability rather than relying on a single simulated dynamics setting.
A. Tasks
The experiments use a 7-DOF Fetch Robotics arm for a puck-pushing task. Each episode requires moving a randomly initialized puck to a randomly specified target on the table.
- Task Setup: The task requires pushing a puck to a random target position on the table.The target is sampled for each episode.
- Task Setup: The experimental platform is a 7-DOF Fetch Robotics arm evaluated in both real and simulated MuJoCo settings.Figure 2 presents the real robot and simulated model.
- Initialization: Episodes begin with the arm in a default pose and the puck randomly placed within a fixed table bound.The initial puck location varies across episodes.
- State and Action Spaces: The policy observes a 52D state and produces actions in a 7D space of relative target joint-angle offsets.The state includes arm, gripper, and puck positions, orientations, and velocities.
C. Dynamics Randomization
Dynamics randomization samples simulator parameters at the start of each episode and holds them fixed throughout that episode. The randomized parameters cover robot, puck, environment, controller, timing, and sensing properties.
- Randomization Procedure: A random dynamics-parameter set µ is sampled from ρµ at each episode’s start and held fixed for its duration.This creates episodes with distinct but internally consistent simulator dynamics.
- Randomized Parameters: The randomized robot parameters include each link’s mass and each joint’s damping.These parameters alter the simulated robot’s physical response.
- Randomized Parameters: The puck’s mass, friction, and damping are randomized alongside the table height.Both object properties and the task environment vary during training.
- Randomized Parameters: Controller gains, action timestep, and observation noise are also randomized.The action timestep models controller latency, while observation noise models sensor uncertainty.
- Randomized Parameters: Observation noise contributes to a total of 95 randomized parameters.Noise is implemented as independent Gaussian noise on each state feature.
D. Adaptive Policy
The adaptive policy addresses manipulation’s dependence on unknown physical properties by inferring dynamics from state-action history. Training combines recurrent policies, HER, and an omniscient critic using randomized simulator dynamics.
- Adaptive Policy: Pushing depends strongly on physical properties such as mass, friction, and actuator characteristics.A successful controller therefore needs information about the environment’s dynamics.
- Adaptive Policy: When deployed on a real robot, the policy cannot directly access dynamics parameters and must infer them from past states and actions.This motivates using history-dependent policy representations.
- Recurrent Policy and Critic: The policy is recurrent, while the value function receives simulator dynamics µ as an additional training-time input.The value function is called an omniscient critic because it knows the dynamics parameters unavailable to the policy.
- Training Procedure: RDPG supplies off-policy training for recurrent policies, enabling HER-generated replay data to be used.HER augments rollout data with trajectories relabeled using replayed goals.
- Training Procedure: Algorithm 1 samples goals and dynamics, generates rollouts, stores them with rewards in a replay buffer, and updates the policy and value function.The procedure may relabel sampled episodes with HER before computing recurrent memories, targets, and updates.
F. Network Architecture
The policy and value networks use separate feedforward and recurrent branches, with an LSTM recurrent branch inferring environment dynamics from state-action history. The LSTM policy is deployed on the Fetch arm, including under modified puck contact dynamics.
- Network Architecture: The recurrent branch uses an LSTM to infer environment dynamics from past observations, while the feedforward branch processes other inputs.The recurrent branch receives information such as the current state and previous action.
- Network Architecture: The policy and value networks are organized as separate feedforward and recurrent branches.Figure 4 schematically illustrates the policy network and value network.
- Network Architecture: The LSTM policy was deployed on the Fetch arm with modified puck contact dynamics created by attaching a packet of chips underneath.The modified contact dynamics provide a physical variation for deployment.
V. EXPERIMENTS
The experiments train and evaluate dynamics-randomized policies using simulation, sensor and action perturbations, and comparisons between simulated and real robot behavior. The setup includes limited calibration and evaluates deployment on a physical Fetch arm.
- Training and Evaluation: Each training episode samples a random set of dynamics parameters and holds them fixed for that episode.The randomized parameters are drawn according to ρ_µ.
- Training and Evaluation: The simulator randomizes physical parameters, action timing, observation noise, and Gaussian action exploration noise.Action timing varies by step, while its exponential rate parameter remains fixed within an episode.
- Physical Evaluation: The physical evaluation uses a 0.2kg puck with a 0.065m radius, and considers the goal satisfied within 0.07m of the target.The puck location is tracked with the PhaseSpace motion-capture system.
- Physical Evaluation: The same target trajectories are executed on simulated and real robots, with joint trajectories recorded for the Fetch shoulder, elbow, and wrist.These recordings illustrate discrepancies between simulated and real robot dynamics.
- Physical Evaluation: Little calibration was performed, although more extensive calibration was expected to improve performance.The experiment tests whether policies can adapt despite poor simulation calibration.
A. Comparison of Architectures
The experiments compare LSTM, feedforward, and history-augmented policies under randomized dynamics and on the real Fetch arm. LSTM policies learn faster, achieve higher simulated success, and generalize better to physical dynamics, while several randomization ablations reduce adaptation.
- Comparison of Architectures: The LSTM learns faster and converges to a higher success rate than the feedforward models in randomized simulation.Figure 6 reports learning curves from four policies per architecture, trained with different random seeds and evaluated over 100 episodes.
- Comparison of Architectures: A feedforward policy trained without randomization cannot cope with unfamiliar dynamics, while randomized memoryless training improves robustness but remains inconsistent.The comparison isolates both the effect of dynamics randomization and the effect of recurrent memory.
- Comparison of Architectures: Although LSTM and FF + Hist policies perform comparably in simulation, the LSTM generalizes better to the real Fetch arm.The policies are trained exclusively with simulation data, and the evaluation randomizes puck targets and initial locations within a 0.3m × 0.3m bound.
- Ablation: Disabling randomization of action timestep, observation noise, link mass, or friction impairs adaptation to the physical environment.The largest reported drops occur when action timestep and observation noise are not randomized, highlighting controller latency and sensor noise.
C. Robustness
The policies remained robust when deployed on a real robot under altered contact dynamics and calibration conditions. They also developed fine-adjustment strategies during simulation training.
- 0.91 ± 0.04 success rate with a packet of chips attached to the puck was comparable to 0.89 ± 0.06 without the attachment.The attachment reduced friction and further altered contact dynamics.
- The real-robot robustness evaluation included policies trained with subsets of parameters held fixed.Table III reports performance for these parameter-randomization settings.
- Fine-adjustment strategies emerged using only a sparse binary reward, including partially upending, top-side manipulation, and overshoot correction.These behaviors positioned the puck over the target after coarse movement.
- The policies adapted to unfamiliar dynamics at runtime and could be deployed despite poor calibrations.These properties were demonstrated through recurrent policies trained with randomized dynamics in simulation.