Automated Reinforcement Learning (AutoRL): A Survey and Open Problems

Motivation

• RL performance is highly sensitive to hyperparameters, architectures, objectives, and update rules, making manual optimization tedious, expensive, error-prone, and difficult to generalize across domains.RL applications are additionally challenging because environments and algorithms are diverse and the learning problem is nonstationary.Source: §1 · p. 2

AutoRL motivation

• AutoRL automates varying degrees of the RL pipeline to improve performance and enable RL applications where manually finding strong configurations is prohibitively expensive.Applications include RNA design, while increased algorithmic flexibility can improve performance for practitioners with sufficient compute, as demonstrated by AlphaGo’s Bayesian Optimization improvement.Source: §1 · p. 2

Survey contribution

• The survey unifies methods from AutoML, evolution, and meta-learning by providing a taxonomy of AutoRL techniques and introducing researchers to tools for improving RL algorithms.The surveyed methods range from early AutoRL algorithms to modern approaches inspired by AutoML, evolving neural networks, and meta-learning.Source: §1 · pp. 2–3

Open problems

• AutoRL has unique challenges because training data depends on the current policy, while the problem also includes RL-specific environment and algorithm design.The survey therefore outlines open problems and aims to attract broader AutoML research to AutoRL.Source: §1 · p. 3

Paper organization

• The paper formalizes AutoRL, surveys automatable environments, algorithms, hyperparameters, and architectures, reviews current methods and benchmarks, and discusses future directions.Its structure covers background and evaluation, a taxonomy of methods, publicly available benchmarks and applications, and future work.Source: §1 · p. 3

2. Preliminaries and Notation

• AutoRL optimizes underspecified training components ζ in a bi-level problem: inner-loop agent training maximizes J(θ; ζ), while the outer loop maximizes evaluation performance f(ζ, θ*), subject to an evaluation budget.ζ may range from one hyperparameter to an entire algorithm, and the budget is commonly a maximum trial count N.Source: §2 · pp. 4–5

2.1 Evaluation Methods

• Evaluation may target training or validation reward, efficiency, or multiple objectives, but RL instability makes single-run metrics unreliable and motivates aggregation across seeds and robust statistics.Outer-loop performance can also be measured by best-so-far value, wall-clock or multi-fidelity budgets, cumulative regret, and threshold-reaching trial count.Source: §2.1 · pp. 5–6

2.2 Inner Loop Optimization

• Inner-loop RL methods include model-free policy-gradient, value-based, and actor-critic algorithms, alongside model-based methods that learn or use environment models for policy training or planning.Model-free systems commonly organize learning around learners, replay buffers, and actors, while model-based methods may optimize prediction losses and use planning procedures such as MCTS or MPC.Source: §2.2 · pp. 6–7

2.3 Non-Stationarity of the Optimization Problem

• RL amplifies hyperparameter non-stationarity because agents continually generate training data dependent on their current state, so AutoRL includes dynamically selecting configurations over training.Dynamic learning-rate adaptation has benefited both model-free and model-based RL, while static settings can cause slow or diverging learning.Source: §2.3 · p. 8

3. What Needs to be Automated?

• AutoRL encompasses tunable pipeline components ranging from complete algorithms to individual hyperparameters, organized around the sequence of decisions made when applying RL to a problem.These components include learner algorithms and hyperparameters, actor networks and weights, and environments.Source: §3 · p. 8, Figure 1 · p. 9, §3.1 · p. 9

3.1 How Do We Design Tasks?

• Task design requires choosing reward functions, observations, action spaces, preprocessing, and training environments, each of which can materially change learning difficulty or policy-optimization success.Reward specification is often non-trivial in real-world problems, while raw visual inputs can increase computation and memory demands and large action spaces hinder initial learning.Source: §3.1 · pp. 8–9

3.2 Which Algorithm Do We Use?

• Choosing an RL algorithm is difficult because agents are brittle even on standard benchmarks and arbitrary choices can produce sub-optimal solutions on new problems.Potential solutions include meta-learning across related tasks, learning proxy objectives, or algorithm selection from prior performance and environment features, although selectors require substantial evaluation data.Source: §3.2 · pp. 9–10

3.3 What about Architectures?

• Neural architecture design is underexplored in RL, despite evidence that deeper, denser, normalized networks and alternative nonlinearities can improve performance across manipulation, locomotion, and generalization tasks.The conceptual reasons for these architectural effects remain unclear, including whether supervised-learning theories explain overparameterization and batch-normalization benefits.Source: §3.3 · pp. 10–11

3.4 Last but not Least: What about Hyperparameters?

• Hyperparameter and implementation choices strongly affect RL performance, with sensitivity spanning discounting, exploration, losses, architectures, initialization, regularization, normalization, codebases, and random seeds.Architecture and hyperparameter searches can also create a chicken-and-egg problem, motivating joint optimization of both.Source: §3.4 · pp. 11–12

4. Methods for Automating Reinforcement Learning

• AutoRL methods combine an outer loop that optimizes agent configurations with an inner loop implementing a standard RL pipeline.The loops exchange specifications, objectives, rewards, and gradients.Source: §4 · p. 12, Figure 2 · p. 12
• Outer and inner loops can each use blackbox or gradient-based optimization, except gradient-based outer optimization cannot operate with a blackbox inner loop.AutoRL classes trade off properties such as sample and compute efficiency and range from tuning one hyperparameter to learning rewards or complete algorithms.Source: §4 · p. 12

4.1 Random/Grid Search Driven Approaches

• Random Search samples hyperparameter configurations randomly, whereas Grid Search evaluates fixed grid points, enabling general configuration selection by exhaustive evaluation.These approaches can also select basic architectural choices, such as network depth or layer width, although they are usually applied to hyperparameters.Source: §4.1 · p. 13
• Grid Search remains common in RL hyperparameter tuning, but its simplicity does not make it the most efficient approach.Both methods are broadly applicable because they evaluate candidate configurations and select the best observed one.Source: §4.1 · p. 13
• Random and grid search overlook changing optimization landscapes because they keep hyperparameters fixed throughout a run, potentially producing sub-optimal configurations at different stages.Averaging performance can hide changing hyperparameter importance and opposing preferences across training, while final performance also conceals valuable information.Source: Figure 3 · p. 14, §4.1 · pp. 14–15
• Hyperband improves Random Search through adaptive resource allocation and early stopping, repeatedly retaining the top 50% of configurations while doubling their training budget.It uses Successive Halving to allocate resources to randomly sampled configurations and eliminate lower-performing candidates.Source: §4.1 · p. 15
• Random/grid search scales poorly because it explores without exploiting information about promising search-space regions as configurations accumulate.This limitation motivates informed methods when the hyperparameter search space becomes large.Source: §4.1 · p. 15

Empirical results

• AlphaGo’s BO-tuned hyperparameters increased self-play win rate from 50% to 66.5% (Chen et al., 2018).The tuning covered core Monte Carlo Tree Search hyperparameters and time-control settings.Source: §4.2 · p. 15

Bayesian Optimization

• Standard BO uses a regressor for uncertainty estimates, an acquisition function for exploration-exploitation, and its maximum to select the next trial.Because full black-box evaluations can take hours or days, multi-fidelity methods use seeds, epochs, or runtimes to reduce optimization overhead.Source: §4.2 · p. 15, Figure 4 · p. 16

Multi-Fidelity Algorithms

• BOHB was orders of magnitude faster than vanilla BO, found dramatically better PPO results on CartPole swing-up, and produced an RNA design method over 1000 times faster than prior state of the art (Falkner et al., 2018).BOHB combines BO-selected configurations with Hyperband-selected fidelities and can exploit parallel resources.Source: §4.2 · p. 16

Exploiting additional information

• BOIL models training curves and selects both hyperparameter settings and evaluation epochs, enabling more cheap, high-utility experiments and greater sample efficiency than traditional BO for deep RL (Nguyen et al., 2020).It augments intermediate training-curve information for the underlying Gaussian-process model.Source: §4.2 · pp. 16–17

Empirical results

• BO with a noise-robust acquisition function was the best choice among successive halving, random search, and BO for RL hyperparameter optimization.BO also improved offline-RL agent performance when tuning hyperparameters (Lu et al., 2021).Source: §4.2 · p. 17

Open Problems

• BO-based approaches usually perform static tuning, while methods accounting for RL’s temporal optimization structure remain scarce.This temporal mismatch is identified as an open problem for BO-based AutoRL.Source: §4.2 · p. 17

Neuroevolution

• Neuroevolution evolves network weights or architectures, from NEAT and HyperNEAT to WANNs, which solve RL tasks by evolving topology with one random weight.These methods have been evaluated on control and Atari tasks, including work predating DQN.Source: §4.3 · p. 18

Hyperparameter Optimization

• Evolutionary hyperparameter optimization has tuned SARSA(λ), DDPG, and other RL agents using genetic algorithms, whale optimization, restarting, and parallel competition.OMPAC successfully tuned agents learning Tetris and Atari games.Source: §4.3 · p. 18

Multi-Fidelity Algorithms

• DEHB combined Hyperband with Differential Evolution, running up to 1000x faster than random search and 5x faster for tuning seven PPO hyperparameters.DEHB uses multi-fidelity evaluations to accelerate evolutionary hyperparameter optimization.Source: §4.3 · p. 18

Population Based Training

• PBT trains agents in parallel, exploits stronger weights, explores hyperparameters, and can learn schedules suited to RL’s non-stationarity.PBT can match a single training run’s wall-clock time with parallel resources, while later variants improve experience sharing, backtracking, and exploitation.Source: §4.3 · p. 19

Open Problems

• Evolutionary methods often require thousands of CPU cores, while PBT can be data-inefficient and sensitive to population size, limiting broader adoption.Few large-scale methods jointly optimize hyperparameters ζ and neural-network parameters θ, although ES-ENAS combines evolutionary algorithms toward this goal.Source: §4.3 · pp. 19–20

• Meta-gradients optimize selected differentiable hyperparameters through inner-loop agent updates and outer-loop loss minimization, with each loss pairing defining a distinct algorithm.Typical meta-parameters include bootstrapping, discount factor, and learning rate; the formulation is inspired by MAML (Finn et al., 2017).Source: §4.4 · p. 20
• Meta-gradient IMPALA improved full-suite Atari performance by 30%–80% over baseline, depending on evaluation protocol, with STAC and STACX providing further gains [Xu et al. (2018); Zahavy et al. (2020)].STAC extended tuning to all differentiable IMPALA hyperparameters, while STACX integrated meta-gradients with auxiliary tasks.Source: §4.4 · p. 20
• Bootstrapped Meta-Learning achieved state-of-the-art model-free Atari performance by tuning multiple hyperparameters online [Flennerhag et al. (2021)].It bootstraps a target from the meta-learner and matches it in a meaningful space, extending the meta-learning horizon.Source: §4.4 · p. 21
• Meta-gradients discover auxiliary tasks, options, and online RL objectives, while differentiable architecture search finds effective architectures in challenging environments [Akinola et al., 2021].FRODO showed that meta-learning the target was more effective than learning the full update rule and achieved strong asymptotic performance.Source: §4.4 · p. 21
• Meta-gradient methods require well-initialized meta-parameters and cannot tune non-differentiable choices such as optimizers or activation functions, despite their efficiency and improvement potential.Initialization itself becomes a hyperparameter optimization problem.Source: §4.4 · p. 21

Blackbox methods

• Blackbox online tuning includes random search with off-policy data, Bayesian selection, and confidence-based switching between TD and MC policy evaluation.These methods adapt λ, policy-gradient hyperparameters, or evaluation procedures without requiring differentiability.Source: §4.5 · pp. 21–22

Bandit-based adaptation

• Bandit methods adapt behavioral hyperparameters, exploration, off-policy optimism, and population diversity, with Agent57 achieving human-level performance on all 57 Arcade Learning Environment games.Agent57 adaptively selects among several exploration policies using learning progress as feedback.Source: §4.5 · pp. 21–22

Open problems

• Bandit approaches have limited search scope because they must explore all arms, and assuming independent arms can reduce efficiency by discarding known relationships.These constraints are identified as open problems for blackbox online tuning.Source: §4.5 · p. 22

Meta-RL formulation

• Meta-RL treats the loss function as a parameterized object and optimizes its parameters to maximize expected reward, distinguishing neural and symbolic representations.The parameterization is L(θ; ζ), with ζ encoding either neural-network parameters or computational graphs.Source: §4.6 · p. 22

Neural loss functions

• Neural loss methods optimize learned update rules through evolution or meta-gradients, using inner-loop policy learning and outer-loop evaluation of performance.Meta-gradient approaches require second-order information, typically computed through Jacobian-vector products.Source: §4.6 · pp. 22–23
• MetaGenRL and Learned Policy Gradient demonstrate that learned policy updates or targets can transfer across tasks, with symmetry incorporation further improving transfer to unseen spaces and environments.MetaGenRL models the inner loss with an LSTM, while Learned Policy Gradient manually defines meta and task losses around learned targets.Source: §4.6 · p. 23

Symbolic loss functions

• Symbolic-loss approaches search DSL-defined computational graphs for curiosity modules or complete RL losses, using exhaustive evaluation with pruning or Regularized Evolution.Alet et al. search 52,000 valid programs, whereas Co-Reyes et al. evolve DAG-based losses for value-based agents.Source: §4.6 · p. 23

Algorithm selection

• Algorithm Selection for RL uses a UCB bandit-style meta-algorithm to choose among off-policy algorithms episode by episode, substantially improving Atari performance.The method selects the algorithm expected to maximize return for the next episode.Source: §4.6 · p. 23

Open problems

• Open problems include explaining neural loss functions and their generalization, while determining effective search spaces and search algorithms remains unclear for symbolic losses.The section identifies empirical tools as important for understanding why learned losses work.Source: §4.6 · p. 23

Overview

• Environment-design methods are organized by the POMDP component they automatically learn, including reward, action, observation, and multiple environment components.Figure 7 depicts a policy–environment feedback loop with controllable reward, action, and observation components.Source: §4.7 · p. 24, Figure 7 · p. 24

Reward Function, R

• Bi-level optimization, meta-gradients, evolutionary search, and transfer methods automatically learn reward shaping or intrinsic rewards to improve policy training.Faust et al. (2019) found that reward-search utility correlates with task difficulty, simple sparse objectives can match hand-tuned rewards, and reward search can outperform hyperparameter search under fixed budgets.Source: §4.7 · pp. 24–25

Action Space, A

• Action-space design simplifies training by learning action repetitions, macro-actions, or options with learned policies and termination conditions.Biedenkapp et al. (2021) conditioned repetition on the predicted action, while Bacon et al. (2017) jointly learned an option policy and the options themselves.Source: §4.7 · p. 25

Observation Space, O

• Observation-space design can select image transformations with a UCB bandit and train policies and value functions invariant to those transformations.Raileanu et al. (2020) passed both original and augmented observations to a regularized actor-critic agent.Source: §4.7 · p. 25

Multiple Components, Unsupervised

• Unsupervised multi-component methods generate or curate diverse environments as curricula, improving robustness or targeting environments that expose current agent weaknesses.POET uses mutations or agent-ranking changes to generate environments, whereas PAIRED maximizes protagonist–antagonist regret; PLR curates randomly sampled levels while retaining PAIRED’s theoretical properties and showing stronger empirical performance.Source: §4.7 · pp. 25–26

Multiple Components, Supervised

• Supervised curriculum methods progressively increase task difficulty according to agent performance, using value estimates, task generators, or self-play to reach specific goals.OpenAI et al. (2019) demonstrated progressive Rubik’s-cube manipulation, while Asymmetric Self-Play (Sukhbaatar et al. (2018)) has two agents create and solve increasingly difficult tasks.Source: §4.7 · pp. 26–27

Synthetic Environment Learning

• Synthetic environment learning can optimize a learned transition and reward model so agents train more efficiently on a target environment.Ferreira et al. (2021) reported that synthetic environments can proxy expensive target environments and reduce the number of training steps.Source: §4.7 · p. 27

Open Problems

• Open problems include determining which methods produce the largest or complementary gains and establishing unified frameworks with shared benchmarks.The survey calls for combined-method studies and efforts such as Romac et al. (2021) to improve cooperation and comparability.Source: §4.7 · p. 27

Definition

• Hybrid methods are defined as approaches that use more than one class of methods from the survey’s taxonomy.They aim to exploit the strengths of different approaches.Source: §4.8 · p. 27

Evolutionary search and reward shaping

• Chiang et al. (2019) combined evolutionary search and reward shaping to jointly select reward functions and neural architectures, producing policies and hyperparameter configurations that generalize better to unseen navigation environments.The method learned a more robust policy for navigation.Source: §4.8 · p. 27

Population-based hybrid systems

• Jaderberg et al. (2019) used bi-level PBT with an evolutionary outer loop and IMPALA inner loop to learn reward signals, representations, and hyperparameters, achieving human-level Quake III Arena Capture the Flag performance from pixels and game points.Agents trained concurrently across thousands of parallel matches in randomly generated environments.Source: §4.8 · p. 27

PBT and Bayesian optimization

• PB2 combines PBT with Bayesian optimization by formulating configuration exploration as a batch GP-bandit problem using UCB, and was shown more sample efficient than PBT on several RL problems.PB2 remains untested on larger problems or with more than a handful of hyperparameters.Source: §4.8 · pp. 27–28

Open Problems

• Hybrid approaches must efficiently prune the search space because combining methods and searching over their combinations can require extremely large computational resources.Further challenges include determining how best to combine individual approaches.Source: §4.8 · p. 28

Benchmark landscape

• AutoRL has no established standardized benchmarks, and prior studies commonly tune baseline RL components while reusing OpenAI Gym or Arcade Learning Environment tasks.These environments typically emphasize continuous control or discrete tasks.Source: §5 · p. 28

Generalization benchmarks

• Generalization benchmarks increasingly evaluate agents across environment distributions, especially procedurally generated settings such as OpenAI Procgen, CoinRun, MiniGrid, NetHack, MiniHack, Griddly, and MineRL.OpenAI Procgen has been used to evaluate AutoRL methods.Source: §5 · p. 28

Structured and compositional benchmarks

• Configurable latent-causal environments and CARL support inexpensive or complex-distribution experiments, while Meta-World and Alchemy target compositional, robotic, and meta-distribution learning.These distributions can improve robustness to variation and enable few-shot learning, but AutoRL for generalization remains understudied because defining the outer-loop task is difficult.Source: §5 · p. 28

Open Problems

• Reliable AutoRL comparisons require protocols that standardize RL environment factors and AutoRL-specific hyperparameter or architecture configuration spaces, while Brax may accelerate experimentation through massively parallel single-GPU simulation.Existing tasks are not AutoRL-specific and vary in complexity and generalization needs, complicating comparisons across methods.Source: §5 · p. 29

• AutoRL needs methods that determine which hyperparameters should be optimized dynamically or statically and whether their effects transfer across environments (Eimer et al., 2021a).Existing hyperparameter-effect analyses were developed for static configuration procedures (Hutter et al., 2014; Fawcett & Hoos, 2016; Biedenkapp et al., 2017, 2018; van Rijn & Hutter, 2018) and remain rarely used in RL.Source: §6 · p. 29
• Cross-environment algorithm configuration should seek robust settings, use multi-environment evaluation protocols, and benchmark performance improvement, data efficiency, and generalization capability.Nearly all surveyed methods optimize individual environments, while algorithm configuration can find robust hyperparameters for environment sets; proposed protocols include Jordan et al. (2020), Kirk et al. (2021), and Patterson et al. (2021).Source: §6 · p. 29
• Broader AutoRL should address per-instance algorithm configuration, multi-agent parameterization, and multi-objective optimization over reward, memory, wall-clock time, or cross-task trade-offs.Per-instance configuration could select among portfolio configurations for each environment (Xu et al. (2010), Kadioglu et al. (2010)); multi-agent settings require centralized or decentralized training, reward design, and algorithms capturing agent interactions, while MORL can optimize Pareto fronts or constrain the design space.Source: §6 · pp. 30–31
• AutoRL needs empirical and theoretical analyses of component interactions, including hyperparameter-importance tools from AutoML, to guide compatible combinations and dependency-aware search spaces.Foundational studies include Fedus et al. (2020), Obando-Ceron and Castro (2020), and Andrychowicz et al. (2021), but substantial work remains.Source: §6 · p. 30
• Future configuration work should treat the choice between on-policy and off-policy algorithms as an AutoRL hyperparameter, despite methods such as SEARL being restricted to off-policy learning.This choice affects algorithms, including V-trace for IMPALA (Espeholt et al., 2018).Source: §6 · p. 30
• Offline RL introduces challenges including scarce real-world policy returns and reliance on off-policy evaluation, potentially requiring new AutoRL approaches.The field is nascent but methodologically diverse, leaving open whether traditional AutoML methods are sufficient.Source: §6 · p. 30

Conclusion

• AutoRL spans diverse methods for automating the RL training pipeline, whose interdependent components ultimately require holistic end-to-end solutions.Unlike supervised learning’s typically open-loop one-step process, RL is a complete closed-loop system in which each component can influence the others.Source: §7 · p. 31
• Stochastic, noisy RL evaluations—arising from sources including policies and environments—make automatic tuning especially challenging.These evaluations are generally more variable than those in supervised learning.Source: §7 · p. 31
• The survey identifies promising directions for overcoming AutoRL’s challenges and concludes that the field is maturing, with exciting possibilities ahead.The authors anticipate continued improvements in the coming years.Source: §7 · p. 31