Source-linked AI summary
Robot Learning from Randomized Simulations: A Review
Fabio Muratore, Fabio Ramos, Greg Turk, Wenhao Yu, Michael Gienger, Jan Peters
TL;DR
Robot learning needs scalable data, but simulation-to-real transfer is limited by imperfect models and the resulting reality gap. This review synthesizes sim-to-real robotics research, focusing on domain randomization and its taxonomy, transferability measures, practical requirements, and research directions. It concludes that randomized simulations can support transfer, while simulator imperfections, the lack of a common benchmark, and adaptive-randomization requirements remain important boundaries.
Problem
Real-world data collection is expensive to scale, while imperfect simulators create a reality gap that can prevent learned policies from transferring to robots.
Method
The paper reviews sim-to-real robotics research, focusing on domain randomization, its taxonomy, transferability measures, practical requirements, and connections to related fields.
Results
The review reports that static domain randomization has bridged the reality gap in several cases, while adaptive approaches generally achieve better final target-domain performance.
Takeaways & Limitations
Domain randomization is a useful framework for learning policies across varied environments, but sim-to-real research still requires task-specific approach selection and further study.
Takeaways & Limitations
Adaptive randomization commonly requires intermediate policy executions on the physical device, which are a likely point of failure.
Abstract
from arXiv · showhide
The rise of deep learning has caused a paradigm shift in robotics research, favoring methods that require large amounts of data. Unfortunately, it is prohibitively expensive to generate such data sets on a physical platform. Therefore, state-of-the-art approaches learn in simulation where data generation is fast as well as inexpensive and subsequently transfer the knowledge to the real robot (sim-to-real). Despite becoming increasingly realistic, all simulators are by construction based on models, hence inevitably imperfect. This raises the question of how simulators can be modified to facilitate learning robot control policies and overcome the mismatch between simulation and reality, often called the 'reality gap'. We provide a comprehensive review of sim-to-real research for robotics, focusing on a technique named 'domain randomization' which is a method for learning from randomized simulations.
1 INTRODUCTION
Robot learning increasingly relies on simulation because real-world data collection is costly, but imperfect models create a reality gap that can prevent transfer. This review examines domain randomization as a way to improve sim-to-real learning by perturbing simulation conditions.
- Real-robot data collection is too expensive to scale, whereas simulation offers faster, cheaper, safer, and more diverse data generation.
- Simulator approximation errors create a reality gap between learned policies and physical robots.
- Policies may overfit to simulator-specific features, including exploitable physics-engine behavior absent from the real world.
- Increasing simulator accuracy alone is not expected to bridge the gap; domain randomization instead perturbs simulator parameters, observations, or actions.
- The review develops fundamentals, surveys sim-to-real approaches and transferability measures, connects related research, and proposes a domain-randomization taxonomy.
2 PROBLEM FORMULATION AND NOMENCLATURE
The formulation models robot learning as reinforcement learning across environments whose parameters are random. The objective therefore evaluates policy performance over a distribution of domains rather than a single fixed instance.
- The system evolves through parameter-dependent transitions from an initial-state distribution while a policy selects actions from states.
- A domain is characterized by parameters such as masses, friction coefficients, time delays, and surface appearance properties.
- Reinforcement learning adapts policy parameters to maximize expected discounted return, a numeric measure of policy performance.
- Experience is collected as trajectories of state-action-reward tuples, or rollouts.
- With partial observability, the policy receives observations produced by an environment-specific mapping from states rather than the states themselves.
- Domain randomization maximizes expected discounted return over a distribution of domain parameters, enabling policies intended to work across environments.
3 FOUNDATIONS OF SIM-TO-REAL TRANSFER
Sim-to-real research builds on stochastic simulators, domain randomization, and transferability measures, while recognizing that simulators remain imperfect approximations. Key design choices concern which parameters to randomize, when and how to randomize them, and how to assess transferability.
- Constructing stochastic simulators: Parametric simulators are central to sim-to-real methods, but their randomization details are crucial and often underreported.Simulators model how bodies or particles evolve through parameterized interactions, while transferability assessment helps predict target-domain performance.
- Constructing stochastic simulators: Even accurately identified domain parameters cannot eliminate the simulator’s fundamental status as an imperfect approximation of reality.This limits what parameter estimation alone can achieve in sim-to-real transfer.
- Randomizing a simulator: Randomization design requires deciding which parameters matter, when to vary them, and how to balance physical validity, numerical stability, and learnability.Essential parameters differ by task; episodic randomization is common, whereas per-step changes can create excessive variance or artificial motion.
- Randomizing a simulator: Physics engines can differ substantially in coordinate representations, numerical solvers, friction, and contact models, affecting transfer especially in manipulation.Renderer quality and randomized distractors or backgrounds also correlate with transferability in vision-based tasks.
- Measuring and predicting the reality gap: Simulation Optimization Bias estimates transferability from source domains to a target domain and decreases in expectation as the number of domains increases.Its upper confidence bound can indicate when training should stop after predicted transferability exceeds a threshold.
- Measuring and predicting the reality gap: Physics-engine evaluations find accurate modeling of control and kinematics but deficits during dynamic robot-object interactions.The reported accuracy uses accumulated mean-squared error between motion-capture positions and simulator predictions.
4 RELATION OF SIM-TO-REAL TO OTHER FIELDS
Sim-to-real research overlaps with fields that schedule task difficulty, transfer knowledge across domains, adapt controllers, identify system parameters, and infer simulator parameters from observations. These connections clarify shared goals while also exposing differences in assumptions and objectives.
- Curriculum learning: Curriculum learning schedules training from easier to harder tasks, and can be viewed as updating the domain-parameter distribution when task parameters form part of the domain.Its sim-to-real use faces challenges because configuration difficulty may be unknown and curriculum learning does not specifically target robustness to model uncertainty.
- Meta learning: Meta learning leverages prior experience from conceptually similar tasks to accelerate learning of new tasks.The reviewed connection places sim-to-real within broader efforts to reuse knowledge across tasks.
- Transfer learning and domain adaptation: Transfer learning uses knowledge from a source domain to solve a target-domain task, with domain adaptation covering cases where target-domain ground truth is available.The field distinguishes settings by task correspondence and the availability of labeled data.
- Knowledge distillation: Knowledge distillation compresses larger networks into smaller policies to reduce evaluation time when high-frequency control makes forward-pass latency critical.This is particularly relevant for ensembles of neural-network policies.
- System identification and adaptive control: System identification estimates simulator or system parameters from observed data, commonly by minimizing prediction-dependent losses such as mean-squared error.In sim-to-real, identified parameters can support simulator construction and policy conditioning.
- Simulation-based inference: Simulation-based inference treats the simulator and its parameters as a statistical model and often infers simulation parameters from real-world time-series data.Likelihood-free methods are useful when the simulator is an implicit, nondifferentiable black box and only samples plus an observation-likelihood measure are available.
5 DOMAIN RANDOMIZATION FOR SIM-TO-REAL TRANSFER
Domain randomization methods vary by whether their parameter distributions are fixed, adapted during learning, or shaped adversarially. Static approaches avoid real-world data and have bridged the reality gap, while adaptive and adversarial methods introduce distinct trade-offs.
- Taxonomy: Static and adaptive methods sample domain parameters at the beginning of each simulated rollout, differing in whether the distribution is updated during learning.Static schemes keep the distribution fixed; adaptive schemes update it during training.
- Adaptive methods: Adaptive randomization requires target-domain data, typically from the real robot, making it substantially more expensive to obtain.
- Adversarial methods: Adversarial randomization can produce exceptionally robust policies, but its adversary must be constrained to prevent unlearnable scenarios.Balancing adversarial power introduces a sensitive hyper-parameter.
- Distribution representations: Independent parameter distributions reduce modeling complexity but severely limit expressiveness by failing to represent correlations between parameters.Likelihood-free inference methods instead estimate the complete posterior.
- Static methods: Static randomization targets sim-to-real transfer without real-world data and has bridged the reality gap in several cases.Its final target-domain policy performance is usually inferior to adaptive approaches, despite its speed and convenience.
Randomizing Dynamics without Using Real-World Data at Runtime
Early and recent studies randomized simulator conditions to improve robustness without requiring real-world data at runtime. Applications span locomotion, object detection, manipulation, and visuomotor control.
- Early randomized simulation: Wang et al. randomized initial states, disturbances, goals, and actuator noise, improving robustness of simulated locomotion controllers.The evaluation was sim-to-sim rather than on a physical robot.
- Robust policy objectives: Risk-averse optimization targeted a lower return quantile, but was evaluated only on simulated MuJoCo tasks.The approach was among the first to draw on a Bayesian perspective.
- Locomotion: Cully et al. learned many hexapod policies before physical execution and queried a policy based on behavioral descriptors when performance dropped.The approach could respond to damage without retraining or fine-tuning.
- Vision and manipulation: Fixed-distribution visual randomization trained a grasping detector exclusively on simulated RGB images and supported simulated-to-real manipulation transfer.Other work added distracting shapes and structured noise while mapping pixels directly to motor velocities.
- Visuomotor control: Domain randomization was used for visuomotor manipulation of deformable objects and for aerial robotics.Matas et al. scaled sim-to-real control to three variations of tissue folding.
Randomizing Dynamics, Randomizing Visual Appearance, and Configurations
Research extends randomization across visual appearance, dynamics, and adaptive parameter estimation. Adaptive schemes can use target-domain interactions, while source-only updates are less flexible and generally less effective.
- Visual appearance: GAN-based pixel-level adaptation transformed simulated monocular RGB images toward real counterparts and reduced the real-world samples needed for robotic grasping.Physical-robot evaluation found both domain randomization and pixel-level adaptation important for successful transfer.
- Adaptive configurations: Static domain randomization assumes knowledge of the true parameter mean or accepts biased samples, limiting its flexibility.Adaptive schemes can narrow or widen the search distribution against selected criteria.
- Adaptive configurations: Adaptive methods commonly update parameter distributions using data collected from intermediate policies on the physical device.Executing those intermediate policies is identified as the most likely failure point.
- Dynamics estimation: Online system identification can estimate unknown domain parameters from observed rollouts, enabling adaptation to sudden environmental changes.A policy conditioned on state and domain parameters uses a second neural network to regress those parameters.
The Bilevel Optimization Perspective
Bilevel optimization separates the search for a domain-parameter distribution from policy learning in that randomized source domain. Related Bayesian and adversarial approaches address robustness, posterior inference, and the difficulty of controlling perturbation strength.
- Bilevel optimization: Bilevel adaptive randomization searches for a domain-parameter distribution whose simulated training produces maximal real-world return, while the lower level learns the policy.Bayesian optimization handles the upper level and model-free reinforcement learning handles the lower level.
- Bilevel optimization: A related meta-algorithm updates the domain-parameter distribution with REINFORCE, but most evaluation used simulated synthetic data without dynamics-dependent real-world interaction.
- Posterior inference: Likelihood-free inference estimates an unrestricted posterior over simulator parameters, allowing correlated and multimodal explanations of observed trajectories.Policies trained with the inferred posterior were more robust to model uncertainties in sim-to-sim evaluation.
- Posterior inference: Neural inference with policy optimization in the loop improved results over BayesSim and Bayesian linear regression on an underactuated swing-up and balancing task.
- Adversarial robustness: Adversarial perturbations can significantly degrade reinforcement-learning policy performance, motivating careful control of adversary strength.
Adversary Available Analytically
Adversarial domain randomization perturbs simulations to improve transfer across domain configurations. Approaches range from physically plausible disturbances to learned hybrid-simulator identification, with mixed but generally favorable comparisons against static randomization.
- Adversarial randomization applies physically plausible perturbations and slightly improves generalization across domain configurations compared with static randomization.The comparison was conducted in sim-to-sim evaluation on four MuJoCo tasks.
- Game-theoretic formulations assign opposing objectives to policy and model players, framing domain randomization as an interaction between control and model adaptation.
- Learned hybrid simulators can outperform static domain-randomization baselines on sim-to-real quadruped locomotion.The method augments an analytical physics simulator with a neural network trained to match simulated and real trajectories.
- The hybrid-simulator approach adds behavioral-policy noise to obtain diverse target-domain trajectories, which may be dangerous.
6 DISCUSSION AND OUTLOOK
The discussion weighs practical choices for sim-to-real transfer, highlights alternatives to domain randomization, and identifies open problems in automation, inference, data costs, and task structure. It also stresses that method selection lacks a common benchmark and must account for task requirements and assumptions.
- Choosing a Suitable Domain Randomization Approach: No common benchmark currently compares sim-to-real methods by target-domain performance, and designing one could encourage overfitting to a narrow scenario set.
- Choosing a Suitable Domain Randomization Approach: Practitioners should check whether methods have been tested on at least two sim-to-real tasks and whether their assumptions can be met.Adaptive randomization may require operating the physical device to collect real-world data.
- Alternatives to Domain Randomization: Domain randomization is useful for transferring simulated control policies, but it is not the only viable transfer strategy.
- Real-to-Sim-to-Real Transfer: Automated real-to-sim-to-real pipelines could reduce environment-construction workload and initialize domain distributions from recorded real-world data.RGBD reconstruction and parameter inference are proposed as components of such pipelines.
- Towards Dual Control via Neural Likelihood-Free Inference: Neural likelihood-free inference with normalizing flows is proposed for identifying multimodal simulator distributions under mild assumptions.
- Towards Dual Control via Neural Likelihood-Free Inference: Including domain parameters in inference could connect adaptive randomization with dual control.
- Accounting for the Cost of Information Collection: Future work should schedule simulated and real experiments under a computational budget and determine when additional information no longer justifies its collection cost.
- Structured and Sequential Tasks: Most related studies use nonsequential continuous-control tasks, whereas many real-world tasks consist of disconnected segments.
SELECTION OF REFERENCES
The review selected references to cover major sim-to-real milestones while diversifying across subfields and research groups. Additional references came from alerts, reverse citation searches, and citation notifications.
- Reference selection prioritized coverage of milestones in sim-to-real robotics research.
- The authors diversified references across subfields and research groups.
- Google Scholar alerts, reverse searches from influential publications, and citation notifications supplied additional references.
FUNDING
The paper acknowledges financial and institutional support from Honda Research Institute Europe, the European Union’s Horizon 2020 programme, and the NSF.
- Fabio Muratore acknowledges financial support from Honda Research Institute Europe.
- Jan Peters received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 640554.
- Wenhao Yu and Greg Turk were supported by NSF award IIS-1514258.
CONFLICT OF INTEREST STATEMENT
The authors disclose an employment collaboration involving Fabio Muratore and report no other potential commercial or financial conflicts among the remaining authors.
- Fabio Muratore is employed by the Technical University of Darmstadt in collaboration with the Honda Research Institute Europe.
- The remaining authors declare no commercial or financial relationships that could constitute a potential conflict of interest.