PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Domain Randomization

Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J. Pal, Liam Paull

arXiv:1904.04762v2cs.LGcs.AIcs.RO

TL;DR

Zero-shot domain transfer requires policies to generalize across unseen domains, but uniform domain randomization can be inefficient and yield suboptimal policies. The paper introduces Active Domain Randomization, which learns to sample informative, difficult environments from rollout discrepancies. Across simulated and real-robot settings, ADR improves generalization while producing more robust and consistent policies.

  • Problem

    Uniform domain randomization may be sample-inefficient and can fail to identify which environment variations are informative for zero-shot policy generalization.

  • Method

    Active Domain Randomization learns a sampling policy that focuses training on difficult parameter settings using discrepancies between randomized and reference-environment rollouts.

  • Results

    ADR delivers better generalization than traditional domain randomization across simulated environments and produces robust policies for real-world transfer.

  • Takeaways & Limitations

    Training more frequently on informative environments can produce more robust and lower-variance policies than uniform randomization.

  • Takeaways & Limitations

    The discriminator reward assumes that a reference environment Eref = S(ξref) is provided with the original task definition.

Abstract

from arXiv · show

Domain randomization is a popular technique for improving domain transfer, often used in a zero-shot setting when the target domain is unknown or cannot easily be used for training. In this work, we empirically examine the effects of domain randomization on agent generalization. Our experiments show that domain randomization may lead to suboptimal, high-variance policies, which we attribute to the uniform sampling of environment parameters. We propose Active Domain Randomization, a novel algorithm that learns a parameter sampling strategy. Our method looks for the most informative environment variations within the given randomization ranges by leveraging the discrepancies of policy rollouts in randomized and reference environment instances. We find that training more frequently on these instances leads to better overall agent generalization. Our experiments across various physics-based simulated and real-robot tasks show that this enhancement leads to more robust, consistent policies.

1 Introduction

Zero-shot transfer seeks policies that generalize from a source domain to an unseen target without fine-tuning, but uniform domain randomization can be inefficient and produce suboptimal transfer. Active Domain Randomization learns to focus training on difficult, informative environment variations, improving generalization and reducing variance.

  • Zero-shot domain transfer evaluates a policy in a previously unseen target domain without fine-tuning, especially when target-world rollouts are inaccessible, complex, or expensive.
  • Uniformly randomizing environment parameters can suffer exponential sample complexity and offers little guidance for diagnosing failed policy transfer.
  • Active Domain Randomization replaces uniform sampling with a learned strategy that searches for difficult environment parameters and allocates more training to them.
  • ADR uses rollout discrepancies between randomized and reference environments to learn a reward for its sampling policy, whose particles propose challenging parameter settings.
  • Across simple and complex simulated environments and real-robot settings, ADR produces superior generalization and lower overall variance than Uniform Domain Randomization.

2 Preliminaries

The paper frames domain randomization within reinforcement learning and explains how randomized simulator parameters generate varied MDPs for training. Standard Uniform Domain Randomization samples these parameters uniformly, aiming to promote cross-domain policy generalization.

  • A reinforcement-learning task is modeled as an MDP with state and action spaces, transitions, rewards, and a discount factor; the agent learns a policy maximizing expected discounted reward.
  • SVPG learns an ensemble of policies using maximum-entropy reinforcement learning and Stein Variational Gradient Descent.
  • SVPG balances exploitation of high-reward regions with exploration through repulsion among similar policies.
  • Domain randomization samples prescribed simulation parameters from bounded ranges in a randomization space before generating training environments.
  • Randomization can alter much of an MDP while keeping reward and discount factor constant, producing environments with substantially different difficulty.
  • Uniform Domain Randomization generates environment instances by uniformly sampling the randomization space and trains the policy on their rollouts.

3 Method

The method asks whether all uniformly sampled MDPs are equally useful, then develops ADR to search the randomization space for informative environments. ADR learns this search with reinforcement learning and a discriminator-based reward derived from randomized-versus-reference rollouts.

  • Problem Formulation: The central question is whether all MDPs generated by uniform randomization are equally useful for training.
  • Toy Experiments: In LunarLander-v2, the experiment varies main engine strength (MES) to test whether some environment instances improve generalization more efficiently than others.
  • Toy Experiments: Lower engine strengths generate harder MDPs, and focusing uniform sampling on these harder ranges improves generalization even outside the training distribution.
  • Toy Experiments: In LunarLander, ADR approaches expert-level generalization while Baseline and UDR fail on lower-MES tasks.
  • Problem Formulation: ADR is motivated by the difficulty of identifying informative environments beforehand, particularly in high-dimensional or noninterpretable randomization spaces.
  • Active Domain Randomization: ADR formulates environment selection as reinforcement learning, with states as randomization configurations and actions as continuous parameter changes.
  • Active Domain Randomization: ADR rolls out policies in randomized and reference environments, then uses a discriminator-based reward to guide SVPG particles toward informative configurations.

4 Results

Across simulated tasks and zero-shot robot transfer, ADR improves generalization by concentrating training on difficult or informative environment variations. Compared with uniform domain randomization, it produces stronger, less variable, and more consistent policies.

  • Experiment Details: The experiments cover LunarLander-v2, Pusher3Dof-v0, ErgoReacher-v0, and real-robot analogs, using five seeds and five random resets per simulated evaluation point.This totals 25 independent trials per evaluation point, with mean performance and one standard deviation reported.
  • Toy Experiments: ADR approaches expert-level generalization on LunarLander, whereas UDR fails to generalize over lower main-engine-strength ranges.The comparison varies only the main engine strength parameter.
  • Toy Experiments: On hard LunarLander settings with ξMES ∼U[8, 11], ADR significantly outperforms both the single-instance baseline and UDR in performance.The baseline trains only at MES 13, while UDR trains over ξMES ∼U[8, 20].
  • Toy Experiments: ADR shifts sampling toward currently deficient regions, then redirects attention as the policy improves across the main-engine-strength range.It samples approximately uniformly for the first 650K steps, emphasizes higher MES between 650K and 800K steps, and later rewards lower-MES instances.
  • Randomization in High Dimensions: In Pusher3Dof-v0, ADR prioritizes the harder purple region, outperforms UDR in all but one test region, and produces lower-variance policies.Only 25% of environments in the training region produce sliding puck dynamics, identified as the hardest environments there.
  • Randomization in Uninterpretable Dimensions: On held-out difficult Pusher3Dof-v0 and ErgoReacher-v0 settings, ADR outperforms UDR in performance and variance while retaining better performance across environments.Uniform sampling can cause UDR to unlearn good behaviors, whereas ADR adjusts quickly and retains stronger performance.
  • Sim2Real Transfer Experiments: In zero-shot real-robot transfer, ADR achieves overall better or similar performance to UDR and consistently shows lower spread across environments.The policies are transferred without fine-tuning while torque strength or puck friction is manually varied.

5 Related Work

Related work positions ADR among approaches for simulation transfer, robust reinforcement learning, active learning, and generalization. ADR differs by searching for useful environments without target-domain rewards and by using a learned discriminator-based signal.

  • Dynamic and Adversarial Simulators: Simulation-parameter randomization is an established strategy for improving policy generalization and zero-shot transfer from simulation to real robots.The paper connects this line of work to earlier evolutionary-robotics and deep-reinforcement-learning studies.
  • Dynamic and Adversarial Simulators: Learnable-simulation methods adapt parameters to a target environment but require the target domain for reward calculation, whereas ADR requires only a reference domain and parameter ranges.The paper identifies target-domain access as a distinction between these approaches.
  • Dynamic and Adversarial Simulators: Unlike Robust Adversarial Reinforcement Learning, ADR removes zero-sum game dynamics and produces reusable SVPG-based sampling and discriminator components.The paper relates removal of the zero-sum game to training stability and notes reuse for training new agents.
  • Active Learning: ADR resembles active learning by searching for environments useful to the agent, but reinforcement learning lacks an oracle or supervised-loss signal.ADR instead learns a proxy signal through a discriminator.
  • Generalization in Reinforcement Learning: The paper relates ADR to reinforcement-learning generalization work showing that deep policies can overfit to details of their training environments.ADR explicitly searches for and varies environment aspects to which policies may have overfit.

6 Conclusion

The paper concludes that ADR searches randomization spaces for informative environments through reinforcement learning, rather than sampling uniformly. Across simulated environments and real-world transfer, it reports better generalization, scalability to high-dimensional spaces, and more robust policies.

  • 6 Conclusion: ADR searches the randomization parameter space for the most informative environments on which to train a reinforcement-learning agent.The method poses this search as an RL problem and optimizes a learned reward with multiple policies.
  • 6 Conclusion: Across simulated environments, ADR efficiently trains agents with better generalization than traditional domain randomization and extends to high-dimensional parameter spaces.The conclusion presents this as a broad result across the paper’s evaluated simulated environments.
  • 6 Conclusion: ADR produces more robust policies for transfer to the real world.This is the conclusion’s stated sim-to-real outcome.

Appendix A Architecture Walkthrough

ADR generates parameterized randomized environments, trains the agent on them, and compares randomized rollouts with reference-environment rollouts to update its sampler.

  • Sampler: SVPG parameterizes the ADR sampler, whose particles propose environment settings while preserving parallelized environment variety.The learned reward lets the sampler focus on problematic MDP instances rather than sampling uniformly.
  • Environment generation: Each proposed parameter setting is converted into a randomized environment instance before agent training.
  • Trajectory generation: The agent is trained on randomized instances, then rolled out in both randomized and default reference environments to collect paired trajectories.These trajectories are stored for scoring parameter settings and updating the discriminator.
  • Scoring: A discriminator scores randomized trajectories by predicting whether they came from reference or randomized environments, and the score updates each SVPG particle.
  • Discriminator update: The randomized and reference trajectories subsequently train the discriminator, closing the active sampling loop.

Appendix C Interpretability Benefits of ADR

ADR can reveal when randomization ranges conflict with task solvability and support focused, pre-transfer diagnosis after range adjustment.

  • Range incompatibility: ADR learned a degenerate LunarLander-v2 policy when engine strength ranged from 6 to 20.Values below approximately 8 made the task almost impossible because the lander hit the ground too quickly and was penalized.
  • Range adjustment: After narrowing the range, ADR produced a better distribution while still preferring harder environments with lower engine strength.
  • Pre-transfer diagnosis: ADR enables focused analysis of problematic simulation regions before expensive experiments such as real-robot transfer.Uniform domain randomization would train equally on these degenerate settings.

Appendix D Bootstrapping Training of New Agents

ADR’s learned sampler and discriminator can be reused to train new agents, although bootstrapping slows training on the default environment.

  • Bootstrapping: ADR’s learned sampling strategy and discriminator are reused to train new agents from scratch after replacing the original policy initialization.
  • Procedure: The bootstrapping procedure extracts the SVPG particles and discriminator from a LunarLander-v2 ADR run before retraining a newly initialized agent.
  • Observed effects: The bootstrapped agent achieves better generalization than an agent trained with ADR from scratch, but trains relatively more slowly on the default environment.

F.1 Catastrophic Failure States in ErgoReacher

This appendix presents a catastrophic-failure progression for ErgoReacher-v0 and identifies the held-out target setting as an extremely low-torque, low-gain environment.

  • Target setting: The held-out simulated target environment uses extremely low torque and gain values.
  • Failure progression: The ErgoReacher-v0 progression depicts an agent moving toward a catastrophic failure state in a hard environment.

Appendix H Network Architectures and Experimental Hyperparameters

The appendix specifies the architectures, optimization settings, and training protocol used for the agent, SVPG particles, and discriminator-based reward generator.

  • Experimental settings: Experiments use fixed network architectures and hyperparameters across environments, except for the SVPG particle count N.LunarLander-v2 uses N = 10, while Pusher-3DOF-v0 and ErgoReacher-v0 use N = 15.
  • Training protocol: Training runs for 1 million agent timesteps in randomized environments, with episode limits of 1000 steps for LunarLander-v2 and 100 steps for the other environments.The specified DDPG settings include a learning rate of 0.001, target update coefficient of 0.005, discount factor γ = 0.99, and batch size 1000.
  • Network architectures: The agent policy uses DDPG with actor and critic networks having hidden layers of 400 and 300 ReLU-activated neurons.The discriminator-based rewarder uses two 128-neuron layers with tanh activations and sigmoid output.
  • Network architectures: SVPG particles use two-layer actor-critic networks with 100 neurons per layer and A2C gradient estimates.The particles operate in a continuous randomization space with each axis bounded between [0, 1].
  • Reward generation: The discriminator reward generator classifies randomized versus reference trajectories using binary cross entropy and averages tuple-level outputs into trajectory rewards.Each trajectory is split into (s_t, a_t, s_t+1) tuples, and rewards are computed before those examples train the discriminator.
Loading 1904.04762v2…