PaperlayerView exact revision on arXiv

Source-linked AI summary

Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey

Wenshuai Zhao, Jorge Peña Queralta, Tomi Westerlund

arXiv:2009.13303v2cs.LGcs.RO

TL;DR

Real-world DRL training is costly, data-intensive, and potentially unsafe, while simulation introduces a gap that can degrade transferred policies. This survey synthesizes sim-to-real transfer methods, applications, and challenges. It identifies domain randomization as widely adopted and highlights policy distillation and meta-learning as promising alternatives.

  • Problem

    Real-world DRL data are costly and limited, and differences between simulation and reality can degrade policies transferred to robots.

  • Method

    The survey reviews and categorizes sim-to-real approaches including domain randomization, domain adaptation, imitation learning, meta-learning, and knowledge distillation.

  • Results

    Domain randomization is the most widely adopted method, while policy distillation supports multi-task learning and smaller networks and meta-learning supports wider task variability.

  • Takeaways & Limitations

    The survey points to integrating methods and using incremental complexity learning, continual learning, and reward shaping as promising directions.

  • Takeaways & Limitations

    The field still lacks wider theoretical and empirical studies and comprehensive analyses of how well existing results generalize.

Abstract

from arXiv · show

Deep reinforcement learning has recently seen huge success across multiple areas in the robotics domain. Owing to the limitations of gathering real-world data, i.e., sample inefficiency and the cost of collecting it, simulation environments are utilized for training the different agents. This not only aids in providing a potentially infinite data source, but also alleviates safety concerns with real robots. Nonetheless, the gap between the simulated and real worlds degrades the performance of the policies once the models are transferred into real robots. Multiple research efforts are therefore now being directed towards closing this sim-to-real gap and accomplish more efficient policy transfer. Recent years have seen the emergence of multiple methods applicable to different domains, but there is a lack, to the best of our knowledge, of a comprehensive review summarizing and putting into context the different methods. In this survey paper, we cover the fundamental background behind sim-to-real transfer in deep reinforcement learning and overview the main methods being utilized at the moment: domain randomization, domain adaptation, imitation learning, meta-learning and knowledge distillation. We categorize some of the most relevant recent works, and outline the main application scenarios. Finally, we discuss the main opportunities and challenges of the different approaches and point to the most promising directions.

I. INTRODUCTION

Real-world DRL training is costly, data-hungry, and potentially unsafe, motivating simulation-based training and methods that transfer policies across the simulation–reality gap. This survey reviews transfer approaches, applications, challenges, and future directions.

  • Real-robot reinforcement learning requires many episodes, faces limited experience variability, and can produce dangerous or unexpected behaviors.
  • Domain randomization perturbs simulator parameters such as colors, textures, and dynamics to produce more robust policies.
  • Simulation provides low-cost data but mismatches real sensing, actuation, dynamics, and novel experiences encountered after deployment.
  • The survey reviews methods for transferring simulation-trained knowledge to real robots while excluding specific simulators and direct real-world learning techniques.
  • It emphasizes DRL policy transfer in robotics, covering end-to-end approaches and sensing-related transfer of deep-learning vision algorithms.

II. BACKGROUND

The background frames sim-to-real transfer within reinforcement learning and related fields, emphasizing simulation’s usefulness for DRL and the challenge of transferring its success to reality.

  • Sim-to-real intersects with transfer learning, robust reinforcement learning, and meta learning across robotics and classic machine-vision tasks.
  • The paper relates deep reinforcement learning, knowledge distillation, transfer, adaptation, and meta learning to sim-to-real transfer.
  • A reinforcement-learning task models an agent interacting with an environment through discrete state transitions, actions, transition probabilities, and rewards.
  • DRL has achieved significant success in simulated environments, motivating efforts to transfer that success to reality.

B. Sim-to-Real Transfer

Sim-to-real transfer adapts simulation-trained DRL policies for reality, where both actuation variability and sensing differences create distinct transfer challenges.

  • End-to-end DRL policies map raw sensor data directly to actuation commands, linking sensing and control within one transfer problem.
  • Closing the actuation gap requires more accurate simulators that account for variability in agent dynamics.
  • The sensing gap includes real-world situations that did not appear during simulation, extending beyond simulator accuracy alone.
  • Transfer-learning methods reduce dependence on target-domain data by moving knowledge from related source domains.
  • Domain adaptation uses sufficient labeled source data for the same task while target-domain data are absent or scarce.

D. Knowledge Distillation

Knowledge distillation compresses DRL knowledge into smaller student networks, while meta-learning develops adaptation across task distributions for potentially unseen tasks.

  • Knowledge Distillation: Policy distillation trains a smaller, more efficient student network to maintain an expert-level policy using teacher-generated data.
  • Knowledge Distillation: DisCoRL distills knowledge from multiple teachers into a single student network for continual learning across multiple tasks.
  • Meta Learning: Meta learning trains adaptation ability across varied training tasks and optimizes performance over a task distribution that may include unseen test tasks.
  • Meta Learning: MetaRL commonly uses an LSTM policy whose hidden states track trajectory characteristics from past rewards and actions.

F. Robust RL and Imitation Learning

Robust RL explicitly addresses disturbances and modeling errors, while imitation learning replaces manually specified rewards with expert demonstrations or inferred reward functions. These approaches appear among the representative methods reviewed for sim-to-real transfer.

  • Robust RL: Robust RL considers input disturbances and modeling errors by optimizing performance against a bad or adversarial model.The approach explicitly incorporates uncertainty in the model and inputs.
  • Imitation Learning: Imitation learning trains agents from expert demonstrations or trajectories instead of manually constructing a fixed reward function.Its two broad categories are behaviour cloning and inverse reinforcement learning.
  • Imitation Learning: Behaviour cloning learns an observation-to-action mapping from demonstrations, whereas inverse reinforcement learning estimates a reward function describing those demonstrations.The two categories differ in whether they directly learn actions or infer the underlying reward.
  • Research Context: The survey summarizes representative sim-to-real methods amid a several-orders-of-magnitude increase in publications.The review places robust RL and imitation learning within a broader set of transfer approaches.
  • Research Context: Domain randomization is identified as the most widely used transfer method, alongside policy distillation, system identification, and metaRL.The surveyed learning algorithms include PPO, TRPO, MPO, A3C, SAC, and DDPG, among others.
  • Transfer Strategies: Direct or zero-shot transfer applies a model trained with realistic or sufficiently abundant simulated experience directly to real-world settings.System identification and domain randomization are discussed as one-shot transfer techniques.

B. System Identification

System identification seeks to make simulators more realistic through precise physical modeling and calibration, but realistic simulation remains difficult. The section also situates visual and dynamics randomization as alternatives for covering real-world variability.

  • System Identification: System identification builds a precise mathematical model of a physical system so the simulator more faithfully represents reality.Careful calibration is necessary to improve simulator realism.
  • System Identification: High-quality rendered images are difficult to produce, and physical parameters can vary with temperature, humidity, positioning, and wear.These factors complicate the construction of realistic simulators and system identification.
  • Domain Randomization: Domain randomization highly varies simulation parameters to cover the real-world data distribution despite bias between the model and reality.This avoids carefully modeling every real-world parameter.
  • Survey Scope: Table I classifies relevant publications in sim-to-real transfer.The table is presented as a classification of representative works rather than a quantitative comparison.
  • Domain Randomization: Domain-randomization methods are divided into visual and dynamics randomization according to which simulator components are randomized.Visual randomization targets variability in simulated visual parameters during training.
  • Domain Randomization: A related approach translates randomized simulated and real images into canonical simulated images before training a vision-based closed-loop grasping agent.The reported application trains the agent in simulation.

D. Domain Adaptation Methods

Domain adaptation transfers knowledge from a source domain to a data-scarce target domain by addressing differences in their feature spaces. In reinforcement-learning settings, the survey formalizes source and target domains and summarizes three common adaptation strategies.

  • Domain Adaptation: Domain adaptation uses source-domain data to improve a learned model in a different target domain where data are less available.Its central objective is transfer across domains with limited target data.
  • Domain Adaptation: Domain adaptation seeks to unify source and target feature spaces because the domains usually have different representations.This feature-space alignment is the main spirit of the approach.
  • Robotics Applications: The survey focuses on domain adaptation for reinforcement-learning and robotics tasks, building on vision-based adaptation work used before control-policy construction.Purely vision-related tasks serve as priors for subsequent reinforcement-learning or controlling tasks.
  • RL Formulation: In the RL formulation, the source domain is D_S ≡ (S_S, A_S, P_S, R_S) and the target domain is D_T ≡ (S_T, A_T, P_T, R_T).The survey formalizes adaptation scenarios using source and target Markov decision-process domains.
  • RL Formulation: Source and target states may differ because of the perceptual-reality gap, while action spaces, transitions, and reward functions retain approximate or structural similarity.The stated relationships are S_S ≠ S_T, A_S ≈ A_T, P_S ≈ P_T, and R_S ≈ R_T.
  • Adaptation Strategies: Three common adaptation strategies are discrepancy-based, adversarial-based, and reconstruction-based methods.Discrepancy-based methods align feature spaces using predefined statistical distances between source and target features.

E. Learning with Disturbances

Learning with disturbances introduces simulation perturbations to reduce agents’ susceptibility to simulation–reality mismatches. Transfer performance also depends on simulator choice and problem-specific simulation design.

  • Learning with Disturbances: Domain and dynamics randomization perturb simulation environments to make agents less susceptible to simulation–reality mismatches.The same idea extends to other perturbations, including noisy rewards and agent-specific environmental effects.
  • Learning with Disturbances: Agent-specific environmental perturbations matter when multiple real agents are deployed or trained with a common policy.The perturbations can affect different concurrently learning agents differently.
  • Simulation Environments: Simulation choice is a key aspect of sim-to-real transfer, and more realistic simulation is expected to produce better results.The survey names Gazebo, Unity3D, PyBullet, and MuJoCo among widely used simulators.
  • Simulation Environments: Gazebo integrates widely with ROS and suits more complex scenarios, whereas PyBullet and MuJoCo integrate more broadly with deep-learning and reinforcement-learning libraries and train faster.These simulator trade-offs concern robotics-stack integration, scenario complexity, and training speed.
  • Combined Approaches: Domain randomization and domain adaptation are often used separately but can also be applied together.Figure 3 illustrates both methods as widely used sim-to-real approaches.
  • Simulation Design: For one-shot transfer based on system identification, researchers often build or customize simulations around problem-specific requirements and constraints.The simulation is tailored to the particular task rather than treated as a general-purpose environment.

IV. APPLICATION SCENARIOS

DRL sim-to-real applications include navigation and dexterous manipulation, where simulation supports varied robotic tasks but imperfect physical modeling can hinder transfer.

  • Dexterous Robotic Manipulation: DRL supports manipulation tasks ranging from peg-in-hole and deformable-object manipulation to multi-fingered-hand control and force-control policies.Force control is especially relevant because excessive force can damage real objects, while insufficient force can cause grasping failure.
  • Dexterous Robotic Manipulation: Domain randomization enables learning for deformable-object manipulation, despite simulation difficulties in representing object deformability.The real robot was unable to grasp stiffer objects when deformability was not properly simulated.
  • Dexterous Robotic Manipulation: Excessive domain randomization can reduce real-robot performance when the number of texture colors becomes too large.This result highlights a practical trade-off in choosing randomization distributions for transfer.

B. Robotic Navigation

Robotic navigation is an emerging sim-to-real application with sparse literature and limited standardization, prompting customized simulation and varied transfer strategies.

  • B. Robotic Navigation: Compared with manipulation, navigation has a sparse sim-to-real literature and lacks standard simulation environments.Navigation tasks require task-specific environments and sensor suites.
  • B. Robotic Navigation: Navigation studies use custom simulators or worlds built with Unity, Unreal Engine, or Gazebo.The choice reflects the specific environmental and sensing requirements of different navigation tasks.
  • B. Robotic Navigation: Sim-to-real transfer has been applied to six-legged robots, depth-based mapless navigation, and soccer-competition robots.These applications demonstrate transfer across distinct robotic navigation settings.
  • B. Robotic Navigation: Notable navigation transfer strategies include curriculum learning, incremental environment complexity, continual learning, and policy distillation.The reviewed approaches vary how training complexity and transferred knowledge are introduced.
  • B. Robotic Navigation: Other emerging sim-to-real applications include plasma-jet control, tactile sensing, and multiagent manipulation.These examples extend beyond the navigation and manipulation scenarios discussed in greater depth.
  • B. Robotic Navigation: Domain adaptation commonly assumes identical source and target feature spaces, an assumption that may fail in many navigation applications.The survey calls for methods that transfer knowledge without this limitation.
  • B. Robotic Navigation: Promising directions include combining domain randomization with domain adaptation and using incremental complexity, continual learning, and reward shaping for complex tasks.These directions target more efficient transfer and multi-step or multiagent settings.

VI. CONCLUSION

The survey reviews sim-to-real transfer approaches in DRL for robotics, identifies domain randomization as the most widely adopted method, and highlights continuing theoretical and generalization gaps.

  • VI. CONCLUSION: The survey focuses on approaches for sim-to-real transfer in DRL for robotics, motivated by reliance on simulated data for vast labeled experiences.It presents itself as the first survey focused specifically on this topic.
  • VI. CONCLUSION: Domain randomization is the most widely adopted method for increasing simulation realism and preparing policies for the real world.The conclusion also identifies policy distillation and meta-learning as promising alternatives.
  • VI. CONCLUSION: Policy distillation supports multi-task learning and smaller, more efficient networks, while meta-learning supports wider task variability.These outcomes are presented as examples of promising alternative research directions.
  • VI. CONCLUSION: The field still lacks broader theoretical and empirical studies explaining how these techniques affect learning.The survey also identifies insufficient comprehensive analysis for generalizing existing results.
Loading 2009.13303v2…