Population Based Training of Neural Networks

1 Introduction

• PBT is motivated by costly hyperparameter tuning and potentially non-stationary learning dynamics, which can make ideal hyperparameters change during training.The paper notes that the schedule for such settings may be impossible to set in advance, particularly in deep reinforcement learning.Source: §1 · p. 1
• Parallel search runs many trainings concurrently, while sequential optimisation uses information from earlier runs but requires multiple sequential trainings that may be infeasible.Grid and random search exemplify parallel search; hand tuning and Bayesian optimisation exemplify sequential optimisation.Source: §1 · p. 1
• PBT shares model parameters and hyperparameters across a concurrently running population, adapting them online to tune hyperparameter schedules and direct more computation to promising models.The method is decentralised and asynchronous, with minimal overhead and infrastructure; its wall-clock runtime is no greater than one optimisation process and it avoids the sequential runs required by sequential optimisation.Source: §1 · p. 2
• Across three reinforcement-learning settings, machine translation, and GAN training, the paper reports faster learning and higher performance, BLEU matching or exceeding tuned schedules, and stable GAN training with large gains over a tuned baseline.The experiments cover UNREAL on DeepMind Lab, Feudal Networks on Atari, A3C agents for StarCraft II, WMT 2014 English-to-German translation with Transformers, and GANs optimised for Inception score.Source: §1 · p. 2

Related work

• Most automatic tuners sequentially use each run to guide the next; Bayesian variants use early stopping and performance prediction, but parallel variants still require sequential optimizations.Bayesian methods model promising hyperparameters, while some approaches use intermediate losses or model time- and data-dependent optimization to reduce training runs and improve exploration.Source: §2 · pp. 2–3
• Genetic hyperparameter evolution also entails sequential model optimizations, while parallel Hyperband can require impractical initial computational resources.PBT builds on random search, using partially trained models to explore promising hyperparameter regions without sequential optimization runs.Source: §2 · p. 3
• PBT evaluates hyperparameters on partially trained models within one training process, reusing strong models’ weights to focus exploration and enable adaptive schedules.Its population periodically replaces underperforming models with better-performing ones, modifies the inherited hyperparameters, and continues training.Source: §2 · p. 3, Figure 1 · p. 3
• PBT resembles self-adaptive evolutionary strategies but trains model parameters with standard optimization rather than evolving them.Related approaches inherit parameters while evolving hyperparameters, evolve network settings while learning parameters, or combine evolution and learning; related applications include architecture search, feature selection, parameter learning, and program search.Source: §2 · p. 4

3 Population Based Training

• PBT jointly optimizes model weights θ and hyperparameters h against the metric Q of interest, and its evaluation function need not be differentiable or match the update objective if the two are correlated.Source: §3 · p. 4
• Hyperparameter choices can make optimization fail or converge poorly, while the exponentially growing schedule space makes searching for effective settings difficult.Training can take days, weeks, or months; common alternatives are a constant setting or a predefined schedule, each requiring a search over settings.Source: §3 · p. 4
• PBT trains a population in parallel, then uses exploit to adopt a more promising solution and explore to propose new hyperparameters before iterative training resumes.Workers exploit and explore when ready, for example after a minimum number of steps or a performance threshold; Algorithm 1 describes the procedure.Source: §3 · p. 5
• In the described applications, exploit copies another member’s weights and hyperparameters, while explore perturbs or resamples hyperparameters after enough gradient steps for learning to occur.The update step uses gradient descent, and evaluation uses mean episodic return or validation-set performance.Source: §3 · p. 5
• PBT combines local gradient-based optimization with periodic model selection and hyperparameter refinement in an asynchronous two-timescale system that requires no population synchronization.Population members access globally available performance information, weights, and hyperparameters.Source: §3 · pp. 5–6
• In the toy example, PBT converges to the global optimum; exploitation provides most of the performance gain, with exploration adding a smaller improvement.Source: Figure 2 · p. 6

4.1.1 PBT for RL

• For RL, PBT tunes learning rate plus task-specific entropy, unroll-length, or intrinsic-reward costs, and exploits via t-test or truncation selection before perturbing or resampling settings.The tuned parameters differ by task: UNREAL uses entropy cost and unroll length, FuN uses entropy and intrinsic reward costs, and StarCraft II A3C uses learning rate only.Source: §4.1.1 · p. 7

4.1.2 Results

• PBT beats equal-worker random search on all three RL domains, raising UNREAL from 93% to 106% human performance, FuN from 147% to 181%, and StarCraft II A3C from 36% to 39%.UNREAL’s results average across DeepMind Lab levels, FuN’s across four Atari games, and StarCraft II A3C’s across six levels.Source: Figure 3 · p. 7, §4.1.2 · pp. 8–9

4.2.2 Results

• PBT raises Transformer BLEU from 23.71 to 24.23 on newstest2012 and from 22.30 to 22.65 on WMT 2014 English-to-German test, using a small model with reduced batch size.PBT adapts dropout rates and discovers a learning-rate schedule resembling the hand-tuned baseline; the paper also reports faster training.Source: Figure 3 · p. 7, §4.2.2 · p. 10

4.3.2 Results

• GAN PBT separately optimizes generator and discriminator learning rates, and learns dynamic irregular decay schedules; truncation selection slightly outperforms binary tournament.The discriminator’s learned schedule closely tracks the best baseline’s, while the generator’s schedule substantially deviates from it.Source: Figure 4 · p. 8, Figure 5 · p. 9, §4.3.1 · p. 11
• PBT improves GAN CIFAR Inception score from 6.39 to 6.80 and corresponding ImageNet Inception score from 6.45 to 6.89, selecting models with CIFAR score.CIFAR Inception score is used for model selection to avoid directly optimizing the final ImageNet performance metric.Source: Figure 3 · p. 7, §4.3.2 · p. 12

4.4 Analysis

• On FuN, populations of 20–40 yield strong, consistent improvements, with diminishing returns at larger sizes; truncation selection works better than binary tournament for GANs and was also promising for FuN.Very small populations (10 or fewer) tend to have higher variance and poorer results; the authors describe truncation selection as a robustly useful choice while noting that further study is needed.Source: Figure 5 · p. 9, §4.4 · pp. 12–13
• For UNREAL, combining hyperparameter optimization with model selection performs best, and full PBT benefits from adapting hyperparameters during training rather than fixing their final values.The ablations compare PBT on hyperparameters alone or model weights alone against full PBT; the adaptivity comparison uses the hyperparameters found at the end of training as a fixed set.Source: Figure 5 · p. 9, §4.4 · p. 13
• PBT discovers non-standard schedules in GAN and translation, while phylogenetic trees show final GAN and Atari agents descending from one initial agent after multiple promising lineages persisted.Translation dropout first rises almost an order of magnitude, falls to its initial value, then climbs again; GAN schedules differ between generator and discriminator. Atari exploration can cause late reward drops, unlike GAN’s mostly parent-relative score gains.Source: Figure 6 · p. 10, Figure 7 · p. 11, §4.4 · p. 13

5 Conclusions

• PBT delivers consistent improvements in accuracy, training time, and stability across domains, including state-of-the-art results on deep reinforcement learning, hierarchical reinforcement learning, machine translation, and GANs.The improvements come from jointly optimizing model weights and hyperparameters.Source: §5 · pp. 13–14
• Unlike conventional hyperparameter optimization, PBT discovers adaptive schedules rather than a fixed set of hyperparameters.Source: §5 · p. 14
• PBT can enhance optimization for unfamiliar models, adapt to non-stationary learning problems, and incorporate indirect performance metrics and auxiliary tasks.The authors describe these capabilities as a platform for future research.Source: §5 · p. 14

A.1 Practical implementations

• A shared key-value store or file system is the main addition needed beyond parallel hyperparameter search to implement PBT on a commodity platform.Population members need the ability to read from and write to the shared data store.Source: §A.1 · p. 18
• Population members use the data store to update their current performance and query the recent performance of other members.Source: §A.1 · p. 18
• At periodic checkpoints, members write their performance to shared storage, making their weights available for other members to restore.Source: §A.1 · p. 18
• During exploit-and-explore, a member may restore another member’s checkpoint weights, modify the restored hyperparameters, and continue training.Source: §A.1 · p. 18
• Semi-serial or partially synchronous PBT takes longer than fully asynchronous parallel execution but retains performance-per-total-compute benefits over random search.This mode may suit researchers using cheaper but less reliable preemptible or spot compute tiers.Source: §A.1 · p. 18
• A partial-synchrony procedure can advance members asynchronously for fixed steps, then exploit-and-explore once all or most have advanced, and repeat.Partial locking accommodates population members running at different speeds because of preemption events.Source: §A.1 · p. 18

FuN - Atari

• Results are means across three runs of each condition, using the single best-performing population member at the end of training.The table reports a per-game breakdown for FuN with and without PBT at different population sizes.Source: FuN - Atari · p. 18, Table 1 · p. 18
• Training lasted 2.5 × 10^8 agent-environment interactions, or 1 × 10^9 environment steps with action-repeat of four.Source: FuN - Atari · p. 18
• The experiments used Truncation for exploitation and Perturb for exploration.Source: FuN - Atari · p. 18
• Population hyperparameter ranges matched the original FuN settings, except intrinsic-reward cost was sampled from [0.25, 1] rather than [0, 1].The authors excluded very low intrinsic-reward cost weights because they rarely found them effective.Source: FuN - Atari · p. 18
• Worker and manager discounts were fixed at 0.95 and 0.99, respectively, based on their good performance in the original FuN paper for these games.Source: FuN - Atari · p. 18
• The games were selected because FuN’s hierarchical abstractions had shown particularly strong gains over a flat A3C baseline.Source: FuN - Atari · p. 18

• Both methods sampled learning rate from [0.00001, 0.005] and entropy cost from [0.0005, 0.01] on log-uniform ranges, plus unroll length uniformly from 5 to 50 steps.For PBT-UNREAL, these sampled hyperparameters were subsequently perturbed.Source: UNREAL - DM Lab · pp. 18–19
• Other experimental details followed Jaderberg et al. (2016), except pixel control weight was set to 0.1.Source: UNREAL - DM Lab · p. 19
• Table 2 presents per-game results for UNREAL with and without PBT on DM Lab.Source: UNREAL - DM Lab · p. 18, Table 2 · p. 19

• Table 3 reports per-game A3C results on StarCraft II with and without PBT.Source: Table 3 · p. 19

A.3 Detailed Results: GAN

• The GANs used a smaller DCGAN-like generator, 4 × 4 convolutions, generator-only batch normalisation, and Adam with β1 = 0.5, β2 = 0.999, and ϵ = 10−8.The batch size was 128; the architecture draws on DCGAN (Radford et al., 2016) and Gulrajani et al. (2017).Source: §A.3 · p. 19
• The selected GAN baseline used exponential learning-rate decay after 1 × 10^5 steps, then continued to 10^6 total steps with final rates held constant.The search compared exponential decay to 10^-2 of the initial rate and linear decay to zero, with N = {0.5, 1, 2} × 10^5; the passage notes GAN performance can degrade with additional training.Source: §A.3 · pp. 19, 21
• Figure 8 shows CIFAR samples from the best-performing baseline GAN and PBT-GAN generators.Source: §A.3 · p. 19, Figure 8 · p. 20
• Table 4 reports CIFAR and ImageNet GAN Inception scores for truncation selection and binary-tournament strategies, including best-generator and peak ImageNet measures.The supplied passage does not include the table’s score values.Source: Table 4 · p. 21

A.4 Detailed Results: Machine Translation

• Machine translation used the Transformer base single-GPU setup, searched learning rate and dropout values, and evaluated tokenized BLEU on newstest2014 after beam-width-4 decoding.PBT used newstest2012 and newstest2013 for evaluation and monitoring, respectively; other hyperparameters and implementation settings were unchanged.Source: §A.4 · p. 21
• 21.23 BLEU was the original machine-translation configuration’s score, lower than both the random-search baseline and PBT.The score was obtained by training the original configuration for the same number of steps.Source: §A.4 · p. 21