PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Reinforcement Learning (AutoRL): A Survey and Open Problems

Jack Parker-Holder, Raghu Rajan, Xingyou Song, André Biedenkapp, Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, Frank Hutter, Marius Lindauer

arXiv:2201.03916v2cs.LG

TL;DR

RL agents are highly sensitive to training design choices, making manual tuning tedious and error-prone. This survey unifies Automated Reinforcement Learning by presenting a taxonomy of methods for automating the RL training pipeline, while highlighting promising directions and open problems.

  • Problem

    RL agents are sensitive to hyperparameters, architectures, implementations, and other design choices, making manual optimization tedious, expensive, and potentially error-prone.

  • Method

    The survey presents AutoRL as methods that automate varying degrees of the RL training pipeline and provides a taxonomy spanning diverse approaches.

  • Results

    The survey identifies a variety of promising AutoRL directions for improving RL training and applications.

  • Takeaways & Limitations

    AutoRL offers a starting point for researchers seeking to improve RL algorithms in open-ended research and practical real-world applications.

  • Takeaways & Limitations

    RL evaluations are stochastic and noisy, and even aggregate metrics can be unreliable for ranking optimization methods.

Abstract

from arXiv · show

The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents. However, the success of RL agents is often highly sensitive to design choices in the training process, which may require tedious and error-prone manual tuning. This makes it challenging to use RL for new problems, while also limits its full potential. In many other areas of machine learning, AutoML has shown it is possible to automate such design choices and has also yielded promising initial results when applied to RL. However, Automated Reinforcement Learning (AutoRL) involves not only standard applications of AutoML but also includes additional challenges unique to RL, that naturally produce a different set of methods. As such, AutoRL has been emerging as an important area of research in RL, providing promise in a variety of applications from RNA design to playing games such as Go. Given the diversity of methods and environments considered in RL, much of the research has been conducted in distinct subfields, ranging from meta-learning to evolution. In this survey we seek to unify the field of AutoRL, we provide a common taxonomy, discuss each area in detail and pose open problems which would be of interest to researchers going forward.

1. Introduction

AutoRL addresses RL’s sensitivity to many costly manual design choices by surveying methods that automate varying portions of the training pipeline. The survey unifies work across subfields with a taxonomy while emphasizing RL-specific challenges such as non-stationarity and environment design.

  • Motivation: RL agents are highly sensitive to hyperparameters, architectures, objectives, and update rules, making manual optimization tedious, expensive, and error-prone.These sensitivities can cause heavily tuned results to fail beyond their intended domains.
  • AutoRL: AutoRL is presented as a suite of methods that automate a varying degree of the RL training pipeline.
  • Motivation: AutoRL is motivated both by resource-constrained practitioners facing unseen problems and by efforts to improve agents when more computation is available.The introduction cites RNA design and AlphaGo as examples of these settings.
  • Survey scope: The survey provides a taxonomy of AutoRL methods to connect research across areas including AutoML, evolution, and meta-learning.The authors aim to encourage cross-pollination and introduce RL researchers to techniques for improving their algorithms.
  • RL-specific challenges: RL creates challenges beyond standard AutoML because training data depend on the current policy and automation can include environment and algorithm design.

2. Preliminaries and Notation

AutoRL formalizes the automation of underspecified RL training components through an outer optimization over ζ and an inner optimization over policy parameters θ. Evaluation must account for task objectives, resource budgets, algorithm classes, and substantial training instability.

  • Problem setting: The formal background represents tasks as MDPs or POMDPs, while environment parameterizations may themselves be learned or automated when fixed task specifications are insufficient.POMDPs add observations and an observation-probability function; environment dynamics may be sampled from parameterized distributions.
  • AutoRL formulation: AutoRL optimizes underspecified training-pipeline components ζ, ranging from individual hyperparameters to entire algorithms, while θ denotes policy parameters optimized in the inner loop.The outer loop selects ζ, and the inner loop optimizes training performance through J(θ; ζ) or an equivalent loss.
  • Evaluation objectives: The inner-loop objective commonly uses cumulative reward, but may instead evaluate generalization across task distributions or efficiency measures such as inference speed and floating-point operations.Multi-objective optimization can combine reward with efficiency-related objectives.
  • Evaluation methods: Outer-loop performance is commonly measured by the best value found within a trial or compute budget, with alternatives including wall-clock time, cumulative regret, and time to reach a threshold.Budgets may count trials, inner-loop timesteps, or total wall-clock time.
  • Evaluation methods: Training instability makes single-run cumulative reward unreliable, so AutoRL evaluations aggregate multiple seeds and increasingly use robust statistics such as the interquartile mean and optimality gap.Even mean or median results can be unreliable for ranking optimization methods when outcome distributions are wide.
  • Inner-loop optimization: RL inner-loop optimizers include model-free and model-based methods, with policy gradients optimizing J(θ) and value-based methods learning action-value functions such as Qθ.Q-learning updates values toward rt + γ max_a∈A Qθ(st+1, a), while DQN is a canonical deep value-based method.

3. What Needs to be Automated?

AutoRL targets decisions across the RL pipeline, from environment and reward design to algorithm, architecture, and hyperparameter selection. These choices can strongly affect performance, while their interactions make manual design difficult.

  • Scope: AutoRL can automate environment definitions, algorithms, architectures, and hyperparameters across the RL training pipeline.The tunable object ζ may range from a single hyperparameter to an entire algorithm.
  • How Do We Design Tasks?: Reward functions and observations are difficult environment-design choices whose differences can substantially affect policy optimization.Examples include sparse versus intermediate rewards and preprocessing pixel observations.
  • Which Algorithm Do We Use?: Choosing an RL algorithm is difficult for new problems because brittle methods and limited expertise or compute can produce sub-optimal solutions.The survey discusses meta-learning, learned objectives, and algorithm selection as possible approaches.
  • What about Architectures?: Architectural choices such as depth, density, nonlinearities, and normalization can improve RL methods across manipulation and locomotion tasks.The survey notes that conceptual understanding of these benefits remains limited.
  • What about Hyperparameters?: RL hyperparameters are highly sensitive, with performance differences depending on loss functions, architectures, initialization schemes, and their interactions.The survey also highlights exploration–exploitation as an important hyperparameter-related trade-off.
  • What about Hyperparameters?: Jointly searching architectures and hyperparameters can avoid the chicken-and-egg problem that architecture search requires meaningful hyperparameters.The survey cites joint search as one way to side-step this dependency.

4. Methods for Automating Reinforcement Learning

AutoRL methods commonly combine an inner RL loop with an outer configuration-optimization loop, using gradient-based or black-box optimization. The survey organizes methods by broad classes while highlighting trade-offs and the difficulty of non-stationary optimization.

  • Framework: Most AutoRL methods use an inner RL pipeline and an outer loop that optimizes the agent configuration.The outer loop can send architectures, hyperparameters, or algorithms, while the inner loop returns objectives, rewards, or gradients.
  • Taxonomy: The taxonomy spans methods with different scopes and trade-offs, from tuning one hyperparameter to learning rewards or entire algorithms.The survey emphasizes that classes may overlap and that sample and compute efficiency can conflict.
  • Random/Grid Search Driven Approaches: Random Search samples configurations while Grid Search evaluates fixed grid points, making both general but commonly focused on hyperparameters.They can also select basic architectural choices such as layer counts or widths.
  • Random/Grid Search Driven Approaches: Fixed configurations can become sub-optimal when the RL loss landscape changes during training, because early and later stages may favor different hyperparameters.Classical search methods therefore optimize performance averaged over the run rather than adapting to changing conditions.
  • Random/Grid Search Driven Approaches: Hyperband accelerates Random Search through adaptive resource allocation and early stopping.It frames configuration evaluation as a pure-exploration infinite-armed bandit problem with predefined resources.
  • Random/Grid Search Driven Approaches: Random and Grid Search scale poorly because they explore configurations without exploiting information about promising regions of the search space.This limitation becomes more pronounced as the search space grows.

4.2 Bayesian Optimization

Bayesian Optimization tunes RL configurations by modeling uncertainty, balancing exploration and exploitation, and selecting promising trials. Extensions use multiple fidelities or RL-specific information to reduce cost, while empirical studies report improvements across RL tasks.

  • 66.5% win rate was achieved in AlphaGo self-play after Bayesian Optimization tuned MCTS hyperparameters and time-control settings, up from 50%.
  • Bayesian Optimization models uncertainty with a regressor, constructs an acquisition function, and evaluates its argmax as the next trial.
  • Multi-Fidelity Algorithms: BOHB combines Bayesian Optimization with Hyperband to select configurations and fidelities, achieving faster searches and better PPO cartpole results than vanilla BO in some cases.
  • Exploiting additional information: BOIL models training curves and selects both a hyperparameter configuration and evaluation duration, enabling more inexpensive experiments.
  • Empirical results: Bayesian Optimization has been effective across RL case studies, including noisy hyperparameter tuning and offline-RL agent improvement.
  • Open Problems: Most BO-based approaches perform static tuning, while methods that account for RL’s temporal optimization structure remain scarce.

4.3 Evolutionary Approaches

Evolutionary approaches optimize RL by updating populations through mutation, crossover, selection, or population-based copying. They can learn changing hyperparameter schedules and achieve substantial speedups, but often require extensive computation and remain difficult to scale efficiently.

  • Evolutionary algorithms update entire populations using mechanisms such as crossover between genomes or mutation of individual genomes.
  • Neuroevolution: Neuroevolution evolves neural-network weights and architectures, with NEAT and HyperNEAT evaluated on balancing and robotic-control problems.
  • Multi-Fidelity Algorithms: DEHB was up to 1000x faster than random search for neural-network hyperparameter optimization and 5x faster for seven PPO hyperparameters.
  • Population Based Training: Population Based Training periodically replaces weaker agents with copies of stronger ones while inheriting their parameters and hyperparameters.
  • Population Based Training: Evolutionary and PBT-style methods can learn hyperparameter schedules, which are particularly effective in RL because the problem is non-stationary.
  • Open Problems: Evolutionary approaches can be inefficient, often requiring thousands of CPU cores, limiting practicality for smaller groups and applications.
  • Open Problems: PBT-style methods are data-inefficient beyond small hyperparameter spaces, and population size remains an influential, insufficiently understood meta-parameter.

4.4 Meta-Gradients for Online Tuning

Meta-gradient methods tune differentiable RL hyperparameters online through inner and outer optimization loops, achieving strong benchmark improvements while retaining important limitations.

  • Meta-gradient methods: Meta-gradients optimize differentiable hyperparameters such as bootstrapping, discounting, and learning rate through an inner agent-training loop.The selected meta-parameters are optimized while the agent trains with them fixed in the inner loop.
  • Meta-gradient methods: 30%–80% improvement over the baseline agent on the full Atari suite was reported for the original meta-gradient method.The range depends on the evaluation protocol used.
  • Meta-gradient methods: STAC and STACX further improved performance on Atari and robotic benchmarks, with STACX integrating meta-gradients with auxiliary tasks.These methods also learned non-trivial hyperparameter schedules that were not smooth or static.
  • Meta-gradient methods: Bootstrapped Meta-Learning extended the meta-learning horizon and achieved state-of-the-art performance for model-free agents on Atari by tuning multiple hyperparameters online.The method first bootstraps a target from the meta-learner before matching it in a meaningful space.
  • Meta-gradient methods: Meta-gradients have been extended to discover auxiliary tasks, options, and online RL objectives, including FRODO’s strong asymptotic performance.Xu et al. found meta-learning the target more effective than meta-learning the full update rule.
  • Open problems: Meta-gradient methods require well-initialized meta-parameters and cannot currently tune non-differentiable choices such as optimizers or activation functions.The survey nevertheless describes them as among the most efficient approaches.

4.5 Blackbox Online Tuning

Blackbox online-tuning methods adapt RL hyperparameters without requiring differentiability, using adaptive schedules, random search, and bandit feedback.

  • Online tuning: Blackbox online tuning includes approaches that adapt hyperparameters on the fly when those hyperparameters are not necessarily differentiable.The section covers single-agent methods ranging from blackbox optimization to online-learning approaches.
  • Adaptive methods: Earlier adaptive methods selected λ schedules or policy-gradient hyperparameters using weighting schemes, Bayesian model averaging, state-dependent policies, and random search.HOOF periodically selects new hyperparameters using off-policy data.
  • Bandit methods: Bandit-based methods adapt behavioral hyperparameters, exploration, optimism, or population diversity, with Agent57 reaching human-level performance on all 57 Atari games.Agent57 adaptively selects among several exploration policies.
  • Open problems: Bandit approaches face limited search spaces because the algorithm must explore all arms, while independent-arm assumptions can discard useful relationships.These constraints can reduce search efficiency.

4.6 Learning Reinforcement Learning Algorithms

Learning RL algorithms treats losses, update rules, or algorithm choices as objects to optimize, using meta-learning, neural or symbolic search, and algorithm selection.

  • Meta-RL: Learned loss functions parameterize L(θ; ζ) and optimize the expected reward J(θ; ζ) through an inner agent-training process and an outer search over ζ.This reframes loss design as an object that can be learned from data.
  • Neural loss functions: Neural loss-function methods use outer-loop evolution or gradients to optimize a learned loss, while MetaGenRL models the policy update with an LSTM.Gradient-based approaches require second-order information, although Jacobian-vector products are typically sufficient.
  • Learning update rules: Learned update rules can transfer between qualitatively different tasks, and incorporating symmetries can improve transfer to unseen action and observation spaces.These results come from Learned Policy Gradient and subsequent extensions.
  • Symbolic loss functions: Symbolic loss-function search represents objectives as directed acyclic graphs over predefined primitives, with one reported search space containing 52,000 valid programs.Pruning includes training in cheap environments and predicting algorithm performance.
  • Algorithm selection: Algorithm Selection for RL uses a UCB bandit-style meta-algorithm to choose among off-policy algorithms and improved Atari performance by a wide margin.The selection occurs at the episode level to maximize expected return.
  • Open problems: Open problems include explaining neural loss functions, understanding their generalization, and determining how symbolic search spaces and search algorithms affect results.The survey identifies these issues as unresolved analytical and empirical questions.

4.7 Environment Design

Automated environment design changes environment components or generates curricula to improve RL training, spanning rewards, observations, actions, and task distributions.

  • Overview: Environment design methods generate curricula or synthetic environments to speed RL learning, with approaches spanning curriculum learning, environment generation, and learned transition-reward models.The survey organizes methods by the POMDP component they automatically learn.
  • Reward function: Reward-design methods learn intrinsic or shaped rewards, and automated reward search becomes more useful as task difficulty increases.Simple sparse objectives in MuJoCo produced results close to those from complex hand-tuned rewards.
  • Action space: Action-space automation simplifies training by repeating actions or constructing macro-actions and options from basic actions.These methods reduce decision points or create temporally extended choices.
  • Observation space: Observation-space methods select transformations such as crops, rotations, or flips and train agents to learn policies and value functions invariant to them.A UCB bandit selects the image transformation.
  • Multiple components: POET and related methods generate diverse environments for curricula, while PAIRED selects environments by maximizing protagonist–antagonist regret.PAIRED encourages environments that the protagonist cannot yet solve and provides theoretical guarantees.
  • Multiple components: Robotic experiments with automatically generated curricula found a significant increase in robustness.Lee et al. applied the approach to quadrupedal robot control.
  • Open problems: The field lacks unified frameworks and shared benchmarks, making it unclear which environment-design approaches yield the largest or complementary gains.The survey proposes combining methods and evaluating them on shared benchmarks.

4.8 Hybrid Approaches

Hybrid AutoRL methods combine multiple approach classes to exploit complementary strengths, but their broader search spaces create substantial efficiency and scalability challenges.

  • Hybrid methods combine more than one approach class, allowing them to exploit complementary strengths within a single AutoRL pipeline.
  • Evolutionary search combined with reward shaping can jointly select reward functions and neural architectures, producing policies and hyperparameters that generalize better to unseen environments.
  • PBT-based bi-level optimization can jointly learn reward-shaping coefficients and hyperparameters while an inner loop trains independent RL agents.
  • PB2 uses a batch GP-bandit explore step with an upper confidence bound to select configurations and was more sample efficient than PBT on several RL problems.Its performance remains untested on larger problems and spaces with more than a handful of hyperparameters.
  • Open Problems: Hybrid approaches require efficient pruning because combining methods and searching over their possible combinations can demand extremely large amounts of computation.

5. Benchmarks

AutoRL lacks standardized benchmarks, while existing evaluations vary in task complexity, generalization demands, and hyperparameter or architecture search spaces.

  • Procedurally generated benchmarks such as Procgen, CoinRun, MiniGrid, NetHack, MiniHack, Griddly, and MineRL support evaluation across environment variations.
  • Configurable environments with latent causal structure enable low-cost experiments, while CARL supports environment distributions defined through potentially observable context features in more complex physics domains.
  • Meta-World and Alchemy provide targeted compositional or meta-distribution benchmarks, but AutoRL for generalization remains understudied because defining the outer-loop task is challenging.
  • There are no established standardized benchmarks for AutoRL, and methods often reuse conventional continuous-control or discrete-environment evaluation suites.
  • Comparing AutoRL methods is complicated because evaluation tasks, generalization requirements, and configuration spaces differ across methods and environments.

6. Future Directions

Future AutoRL research should address dynamic and cross-environment configuration, interactions among design choices, broader RL settings, and efficient multi-objective or multi-agent optimization.

  • It remains unclear which hyperparameters should be optimized dynamically rather than statically and whether their effects remain consistent during training.
  • Most AutoRL optimization methods consider individual environments, leaving cross-environment configuration and generalization insufficiently explored.
  • A rigorous understanding of how RL design choices interact could guide more compatible combinations and better search-space construction.
  • Future work could treat the on-policy versus off-policy choice as an AutoRL hyperparameter because it has significant algorithmic implications.
  • Offline RL introduces evaluation challenges because real-world policy returns are difficult to obtain, often requiring off-policy evaluation for AutoRL.
  • The survey largely addresses single-agent RL, leaving multi-agent parameterization, reward design, and algorithm design as important extensions.
  • Multi-objective AutoRL may optimize trade-offs such as performance, memory usage, and wall-clock time through Pareto-front methods or constrained design choices.

7. Conclusion

AutoRL must ultimately address interacting pipeline components and noisy stochastic evaluation, but the survey identifies promising directions as the field matures.

  • Because RL is a closed-loop system, its pipeline components influence one another, motivating holistic end-to-end AutoRL solutions.
  • Stochastic and potentially noisy RL evaluations make automatic tuning more difficult than in supervised learning.
  • The survey presents promising directions for overcoming these challenges and concludes that AutoRL is maturing as a field.
Loading 2201.03916v2…