Source-linked AI summary
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, Dieter Fox
TL;DR
Policies trained for sim-to-real transfer face costly real-world data collection and mismatches between simulated and real dynamics. The paper adapts simulation-parameter randomization using a few real-world roll-outs interleaved with policy training, and reports reliable transfer across two robots and two articulated tasks.
Problem
Simulation-based policy transfer is limited by the reality gap, while real-world data collection is prohibitively laborious and expensive.
Method
SimOpt iteratively adapts a simulation-parameter distribution using discrepancies between real-world and simulated policy observations, with sampling-based distribution optimization.
Results
Policies trained with SimOpt reliably transferred to ABB Yumi and Franka Panda on swing-peg-in-hole and cabinet-drawer opening tasks.
Takeaways & Limitations
Adapting simulation randomization with real-world data can produce parameter distributions suited to successful policy transfer without exact replication of the real environment.
Abstract
from arXiv · showhide
We consider the problem of transferring policies to the real world by training on a distribution of simulated scenarios. Rather than manually tuning the randomization of simulations, we adapt the simulation parameter distribution using a few real world roll-outs interleaved with policy training. In doing so, we are able to change the distribution of simulations to improve the policy transfer by matching the policy behavior in simulation and the real world. We show that policies trained with our method are able to reliably transfer to different robots in two real world tasks: swing-peg-in-hole and opening a cabinet drawer. The video of our experiments can be found at https://sites.google.com/view/simopt
I. INTRODUCTION
Simulation-based policy transfer addresses the cost of real-world data collection but is hindered by the reality gap. SimOpt adapts simulation randomization with real-world roll-outs to improve transfer without requiring exact scene replication.
- Real-world data collection for continuous-control learning is laborious and expensive, motivating policy training in faster, more diverse simulators.
- Simulation policies often fail to transfer directly because simulation models and scenes do not faithfully replicate the real world.This mismatch is known as the reality gap.
- Domain randomization improves transfer by training policies across diverse simulated scenarios, but selecting its parameter distributions requires tedious expert tuning.
- SimOpt uses real-world data to adapt simulation randomization so simulated and real policy behavior better match, without exact scene replication.
- Prior approaches address sim-to-real transfer through system identification, representation learning, generative models, policy fine-tuning, dynamics ensembles, or broad scenario variation.
- Unlike related iterative approaches, SimOpt uses partial real-world observations and does not require estimating real-world rewards.
III. CLOSING THE SIM-TO-REAL LOOP
The framework represents simulator uncertainty as a distribution over simulation parameters and trains policies under the resulting probabilistic dynamics. It targets distributions that support learning while avoiding overly broad, infeasible randomization.
- A simulation parameter distribution pφ(ξ) induces probabilistic simulator dynamics used for policy training.The simulator is deterministic, while the parameter distribution introduces probabilistic dynamics through reparameterization.
- Domain randomization seeks parameter distributions under which policies trained in simulation perform well on real-world dynamics.
- Manual design of domain-randomization distributions requires significant expertise and tedious fine-tuning.
- SimOpt updates the distribution using discrepancies between real and simulated observations after policy training and roll-outs.
- Overly wide parameter distributions can include infeasible scenarios that hinder policy learning or produce excessively conservative policies.
B. Learning simulation randomization
SimOpt iteratively learns a simulation-parameter distribution by matching policy-induced simulated observations to real observations. A KL-divergence trust region keeps each update near the distribution used to train the current policy.
- The objective is to find a simulation-parameter distribution whose policy-induced observations or partial observations are closer to real-world observations.
- The discrepancy compares real observation trajectories with simulated trajectories sampled under the current policy and parameter distribution.The policy inputs and observations used for discrepancy computation need not be the same.
- Evaluating every candidate distribution would require extensive reinforcement-learning training and real-robot trials, so SimOpt uses an iterative approximation.
- Each iteration trains a policy on the previous distribution, samples real-world observations, and optimizes the next simulation-parameter distribution.
- A KL-divergence step ε constrains updates to remain within the trust region of the policy trained on the old distribution.
C. Implementation
The implementation combines parallel PPO training, Gaussian simulation-parameter distributions, discrepancy-based costs, and sampling-based black-box optimization. Its components can be exchanged for differentiable simulators, alternative discrepancies, discriminators, or richer distributions.
- Policies are trained with parallelized PPO on a GPU simulator and multi-GPU cluster, while simulation parameters are modeled with a Gaussian distribution.
- The discrepancy uses importance-weighted observation differences with ℓ1 and ℓ2 norm terms, plus Gaussian filtering for trajectory misalignment.
- A sampling-based, gradient-free relative-entropy policy-search method optimizes the distribution while bounding each KL-divergence update.The simulator is therefore treated as a black box using sampled parameters and their discrepancy costs.
- Sampling parameters and policy roll-outs is highly parallelizable, enabling evaluation of large numbers of simulation samples.
- The framework can substitute gradient descent with a differentiable simulator, alternative discrepancy functions, learned discriminators, or higher-dimensional multimodal distributions.
IV. EXPERIMENTS
The experiments compare SimOpt with standard and wide-distribution randomization, then test transfer across real robots and manipulation tasks. SimOpt supports successful real-world transfer using few real robot trials.
- The experiments ask whether SimOpt improves over standard domain randomization, handles wide parameter distributions, and transfers across tasks and robots.
- Updating simulation parameter distributions enables successful policy transfer, whereas leaving an initial distribution unchanged does not.
- Policies trained with SimOpt transfer successfully to ABB Yumi and Franka Panda on cabinet opening and swing-peg-in-hole tasks.The tasks include articulated contact dynamics and non-rigid rope dynamics.
A. Tasks
The paper evaluates two manipulation tasks: inserting a rope-suspended peg into an angled hole and opening a cabinet drawer. The tasks use different 7-DoF robots and task-specific observations and controls.
- The evaluation covers cabinet drawer opening and swing-peg-in-hole manipulation tasks.
- Swing-peg-in-hole: Swing-peg-in-hole requires placing a rope-attached peg into a hole at a 45 degrees angle using a 7-DoF ABB Yumi robot.Soft-rope motion creates swinging dynamics; observations include seven joint configurations and the peg’s 3D position.
- Swing-peg-in-hole: The swing-peg-in-hole simulation reward combines peg distance, hole-angle alignment, and a binary task-solving reward.
- Drawer opening: Drawer opening requires grasping and pulling a cabinet handle with a 7-DoF Franka Panda arm under contact dynamics.Its observation space contains seven joint angles and the drawer handle’s 3D position.
- The method updates simulation parameters without requiring full real-world state information such as rope diameter or compliance.The policy outputs seven joint velocity commands and an additional gripper command for drawer opening.
2) Drawer opening:
Drawer-opening experiments show that standard domain randomization is highly sensitive to cabinet-position variance, while SimOpt progressively adapts a conservative source distribution toward the target.
- Wide simulation-parameter distributions can contain many infeasible swing-peg-in-hole instances, making policy training difficult.
- Standard domain randomization: Standard domain randomization opens the drawer only at a 2cm cabinet-position standard deviation and becomes conservative as variance increases.At larger variances, policies reach the handle but fail to open the drawer; the tested range extends to 10cm.
- Standard domain randomization: A 15cm or 22cm target cabinet offset would require at least 10cm standard deviation for naive randomization, which fails to open the drawer.
- SimOpt: The cabinet-position distribution gradually shifts from the source environment toward the target distribution over five SimOpt iterations.
- SimOpt: SimOpt starts from a conservative source distribution and changes it using target-scene roll-outs until source and target policy behavior matches.
D. Real robot experiments
Real-robot experiments show that SimOpt adapts simulation distributions using a few roll-outs, enabling successful transfer in swing-peg-in-hole and drawer opening.
- Experimental setup: SimOpt used 7-DoF Franka Panda and ABB Yumi robots, with simulation training and parameter sampling run on 64 GPUs.The real-world system tracked task objects with DART and initialized corresponding simulation positions.
- Drawer opening: Drawer-opening transfer improved as the source distribution was adjusted, with successful task completion reached by the fourth SimOpt iteration.The figure reports performance across iterations in the target drawer-opening environment.
- Drawer opening: For drawer opening, SimOpt adjusted robot and drawer parameters; after updating, the robot controlled gripper orientation accurately enough to open the drawer.The covariance updates also showed correlations between robot joint compliance and damping.
- Swing-peg-in-hole: After two SimOpt iterations, swing-peg-in-hole succeeded in 90% of 20 real-world trials.Each iteration included 100 RL iterations, three real-robot roll-outs, and distribution updates using 9,600 simulation samples per update.
- Swing-peg-in-hole: SimOpt adjusted rope, peg, and robot parameter distributions, with the largest changes affecting rope dynamics and robot policy behavior.Robot parameters included action scaling, while rope parameters influenced its dynamical behavior.
2) Drawer opening:
SimOpt adapts robot and cabinet simulation parameters for drawer opening, improving gripper control and transfer across scene offsets, while using uni-modal distributions.
- Method: The drawer-opening policy learned a Gaussian distribution over robot and cabinet simulation parameters.During each iteration, training used 200 RL iterations, three real-robot roll-outs, and 20 distribution-update steps with 9,600 samples each.
- Results: After one SimOpt update, the robot opened the drawer in all 20 trials while keeping the gripper orthogonal to the handle.Before updating, excessive force on the handle caused the gripper fingers to open.
- Scope: The experiments applied SimOpt to uni-modal simulation parameter distributions, with multi-modal distributions and richer sensor modalities left for future work.Planned modalities include vision and touch for policy observations and simulation randomization.
APPENDIX
The appendix describes closed-loop and open-loop simulation handling and documents task-specific simulation distributions and their reported parameters.
- Comparison to trajectory-based parameter learning: Closed-loop simulated policies can produce meaningful parameter updates even when the simulator cannot be continuously initialized from real-world states.This remained effective when some observations, such as cabinet X-position, were excluded from SimOpt cost computation.
- Simulation parameter distributions: Tables I and II report initial means, initial covariance diagonals, and final Gaussian-distribution means for drawer opening and swing-peg-in-hole.The tables summarize distributions optimized with SimOpt for the two tasks.
C. SimOpt parameters
The appendix lists SimOpt update settings for both tasks, including REPS, discrepancy-function, observation-weight, and PPO-training parameters.
- SimOpt parameters: Tables III and IV provide task-specific SimOpt parameters for swing-peg-in-hole and drawer opening.The tables include REPS settings and discrepancy-function parameters.
- SimOpt parameters: The reported settings include observation-dimension weights and parallelized PPO training parameters with task reward weights.These parameters accompany the SimOpt distribution-update configuration.