Source-linked AI summary
Population Based Training of Neural Networks
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, Koray Kavukcuoglu
TL;DR
Neural-network training is sensitive to hyperparameters whose optimisation can be costly and whose ideal values may change during learning. The paper introduces asynchronous Population Based Training, which jointly adapts models and hyperparameters through online population-based selection. Across reinforcement learning, machine translation, and GANs, PBT discovers adaptive schedules and improves reported training outcomes, while the paper notes instability in complex reinforcement-learning settings and an evaluation-function assumption.
Problem
Neural-network optimisation can take days, weeks, or months and is highly sensitive to hyperparameter sequences, whose possible schedules grow exponentially with training time.
Method
PBT asynchronously trains a population, evaluates members, copies weights and hyperparameters from better performers, and perturbs or resamples hyperparameters for continued learning.
Results
PBT produces consistent improvements in accuracy, training time, and stability across reinforcement learning, machine translation, and GANs, while discovering adaptive hyperparameter schedules.
Takeaways & Limitations
PBT provides a practical way to jointly optimise neural-network weights and hyperparameters while adapting to non-stationary learning problems and indirect performance metrics.
Takeaways & Limitations
Complex reinforcement-learning experiments can still show late exploration steps with sharp drops in cumulative episode reward relative to the parent.
Abstract
from arXiv · showhide
Neural networks dominate the modern machine learning landscape, but their training and success still suffer from sensitivity to empirical choices of hyperparameters such as model architecture, loss function, and optimisation algorithm. In this work we present \emph{Population Based Training (PBT)}, a simple asynchronous optimisation algorithm which effectively utilises a fixed computational budget to jointly optimise a population of models and their hyperparameters to maximise performance. Importantly, PBT discovers a schedule of hyperparameter settings rather than following the generally sub-optimal strategy of trying to find a single fixed set to use for the whole course of training. With just a small modification to a typical distributed hyperparameter training framework, our method allows robust and reliable training of models. We demonstrate the effectiveness of PBT on deep reinforcement learning problems, showing faster wall-clock convergence and higher final performance of agents by optimising over a suite of hyperparameters. In addition, we show the same method can be applied to supervised learning for machine translation, where PBT is used to maximise the BLEU score directly, and also to training of Generative Adversarial Networks to maximise the Inception score of generated images. In all cases PBT results in the automatic discovery of hyperparameter schedules and model selection which results in stable training and better final performance.
1 Introduction
Neural-network performance depends on jointly tuning model structure, data, and optimisation details, but conventional approaches make this costly or sequential. Population Based Training combines concurrent training with online parameter and hyperparameter adaptation, and is evaluated across reinforcement learning and other domains.
- Motivation: Neural-network success depends on jointly tuning model structure, data, and optimisation details, making hyperparameter selection computationally expensive.The number of hyperparameters grows as neural-network systems become more complicated.
- Existing approaches: Parallel search uses more computational resources, whereas sequential optimisation requires multiple sequential training runs and can be infeasible for lengthy processes.These approaches trade concurrently used resources against the time required to achieve optimal results.
- Population Based Training: PBT bridges parallel search and sequential optimisation through asynchronous information sharing and online transfer of parameters and hyperparameters.It has wall-clock time no greater than one optimisation process and does not require sequential runs.
- Evaluation: The method is evaluated on deep reinforcement learning, supervised machine translation, and generative modelling.The experiments cover UNREAL, Feudal Networks, A3C agents, machine translation, and GANs.
- Population Based Training: PBT automatically selects hyperparameters, performs online model selection, and adapts hyperparameters during training to discover complex schedules.These mechanisms are identified as the sources of the empirical improvements reported by the paper.
- Paper scope: The paper introduces PBT, presents its general algorithm, evaluates domain-specific implementations, and concludes with results across different domains.The paper structure covers related work, the method, experiments, and conclusions.
2 Related Work
Prior hyperparameter optimisation methods either use sequential information across runs or parallelise independent searches, but both retain important resource or latency costs. PBT instead uses a population of partially trained models to explore promising hyperparameter regions while sharing model state.
- Sequential optimisation: Most automatic hyperparameter tuning mechanisms use sequential optimisation, often updating Bayesian models from the outcomes of earlier training runs.Examples include GP-UCB, TPE, Spearmint, and SMAC.
- Search paradigms: Sequential optimisation trades longer optimisation time for fewer concurrent resources, while parallel search takes one training-run duration but uses more resources.The comparison motivates methods that share information during concurrent training.
- Sequential optimisation: Parallelised Bayesian and genetic approaches can accelerate updates, but still require multiple sequential model optimisations.Parallel training may also introduce bias in Bayesian approaches.
- Parallel search: Hyperband incorporates parallelisation but cannot practically run within a single training optimisation process because of its initial computational requirements.This limits its use when the available process cannot support the required resources.
- Population-based search: PBT uses partially trained models to explore promising regions identified by random search, including by copying their weights.It requires a single training optimisation process.
- Evolutionary connections: PBT resembles evolutionary strategies with self-adaptive hyperparameters, but trains model parameters rather than evolving them.Its hyperparameters are adjusted more slowly than the model parameters, paralleling self-adaptive evolutionary methods.
3 Population Based Training
Population Based Training jointly adapts model parameters and hyperparameters while optimizing the metric of interest. It trains a population asynchronously, combining gradient updates with periodic model selection and hyperparameter exploration.
- Core objective: PBT jointly optimizes model parameters θ and hyperparameters h using the actual metric Q of interest.Q may differ from the surrogate objective used for iterative parameter updates.
- Motivation: Hyperparameter schedules can be difficult to optimize because performance depends on the sequence h=(h_t)^T_{t=1}, whose possible values grow exponentially with training time.This sensitivity motivates simple constant settings or predefined schedules in conventional practice.
- Population mechanism: A population of models is trained in parallel with different hyperparameters, sharing partially trained solutions to perform meta-optimization.PBT adapts both weights and hyperparameters according to population-wide performance.
- Population mechanism: Each worker alternates gradient-based steps and evaluations with exploit-based copying of promising solutions and explore-based generation of new hyperparameters.Workers act when ready, such as after enough optimization steps or reaching a performance threshold.
- Algorithmic properties: PBT combines local gradient optimization, periodic model selection, and hyperparameter refinement in an asynchronous two-timescale learning system.The method does not require population synchronization.
- Toy example: In the toy quadratic example, fixed grid or random search reaches approximately Q(θ) ≈ 0.4 because it explores too few hyperparameter settings.PBT is introduced as an alternative that can continue adapting hyperparameters during partially completed training.
4 Experiments
PBT is evaluated across reinforcement learning, machine translation, and GAN training, where it improves performance and discovers adaptive hyperparameter schedules. Ablations indicate that its gains depend on combining online hyperparameter adaptation with model selection.
- Deep Reinforcement Learning: PBT increases final performance over random search across DeepMind Lab, Atari, and StarCraft II when trained for the same number of steps.The method is evaluated on UNREAL, Feudal Networks, and A3C agents across these domains.
- DeepMind Lab: 93% to 106% human performance: PBT improves UNREAL on DeepMind Lab using 40 workers, averaged across all levels.PBT adapts hyperparameters over time, including learning-rate annealing and increasing unroll length.
- Atari: 147% to 181% human-normalised performance: PBT improves FuN across four Atari levels.On Ms. Pac-Man, performance rises from 6506 to 9001.
- StarCraft II: 36% to 39% human performance: PBT improves A3C across six StarCraft II levels through online learning-rate adaptation and model selection.
- Machine Translation: 23.71 to 24.23 validation BLEU and 22.30 to 22.65 test BLEU: PBT exceeds the tuned Transformer baseline on WMT 2014 English-to-German.The reported models use a small Transformer with a reduced batch size and are not representative of state-of-the-art performance.
- GANs: 6.39 to 6.80 CIFAR Inception score: PBT outperforms the GAN baseline, with corresponding ImageNet scores of 6.45 and 6.89.CIFAR Inception score is used for model selection to avoid potentially overfitting the ImageNet test metric.
- Ablations: Population sizes of 20 to 40 produce strong, consistent improvements, while larger populations show diminishing returns and populations of 10 or below have higher variance.Truncation selection works better than binary tournament for GAN training, and combining hyperparameter and weight adaptation performs best.
5 Conclusions
PBT augments neural-network training by jointly optimising model weights and hyperparameters, discovering adaptive schedules rather than fixed hyperparameter sets. The paper reports improvements across reinforcement learning, machine translation, and GANs.
- PBT jointly optimises neural-network weights and hyperparameters as an augmentation to standard model training.
- PBT improves accuracy, training time, and stability across deep reinforcement learning, machine translation, and GAN tasks.
- PBT discovers adaptive hyperparameter schedules instead of a fixed set used throughout training.
A.1 Practical implementations
PBT can be added to a standard parallel hyperparameter-training platform by enabling population members to share performance and checkpoints through a data store. Its asynchronous execution can also be adapted to partially synchronous or semi-serial settings.
- Implementing PBT requires population members to read and write through a shared data store.
- Members update stored performance and query recent results from other population members.
- Checkpoint-based exploitation restores another member’s weights, after which exploration modifies the restored hyperparameters.
- PBT may run semi-serially or with partial synchrony, but fully asynchronous parallel execution provides shorter training times.
FuN - Atari
The FuN Atari experiments compare training with and without PBT across games and population sizes, using the original study’s hyperparameter ranges with one modified intrinsic-reward-cost range.
- The initial hyperparameter ranges were matched to the original Feudal Networks study except for intrinsic-reward cost.
- Intrinsic-reward cost was sampled from [0.25, 1] rather than [0, 1].
- Table 1 reports per-game FuN Atari results with and without PBT for different population sizes.
UNREAL - DM Lab
The UNREAL DeepMind Lab experiments report per-level comparisons between UNREAL and PBT-UNREAL. They randomly sample three hyperparameters, perturb them with PBT, and replicate the original experimental setup with one changed weight.
- The experiments randomly sampled learning rate, entropy cost, and unroll length, then perturbed them for PBT-UNREAL.
- Learning rate was sampled from [0.00001, 0.005], entropy cost from [0.0005, 0.01], and unroll length uniformly from 5 to 50 steps.
- Other experimental details followed Jaderberg et al. (2016), except that pixel control weight was set to 0.1.
- Table 2 reports per-level results for UNREAL with and without PBT on the DeepMind Lab domain.
A3C - StarCraft II
The supplied passages identify per-game A3C results on StarCraft II and describe a separate UNREAL evaluation on DM Lab with and without PBT.
- The UNREAL DM Lab breakdown compares results with and without PBT across different population sizes.The baseline uses feed-forward network agents, with learning rates sampled log-uniformly from [0, 0.001].
- Table 3 reports per-game A3C results on StarCraft II with and without PBT.
A.3 Detailed Results: GAN
The GAN experiments compare generators trained with and without PBT using CIFAR and ImageNet Inception scores. They use a DCGAN-like architecture, selected learning-rate-annealing baselines, and two PBT exploitation strategies.
- Architecture and hyperparameters: The GAN generator and discriminator follow DCGAN-like designs with 4 × 4 convolutions and a smaller generator.Batch normalisation is used in the generator but not the discriminator, and optimisation uses Adam with β1 = 0.5, β2 = 0.999, and ϵ = 10−8.
- Baseline selection: The baseline was selected by searching learning-rate annealing strategies and choosing the strongest CIFAR Inception score.Strategies kept learning rates fixed initially, then applied exponential decay toward 10−2 times the initial setting or linear decay to 0.
- Figure 8 shows CIFAR samples from the best-performing Baseline GAN and PBT-GAN generators.
- Table 4 compares CIFAR and ImageNet Inception scores for GANs trained with and without PBT using TS and BT exploitation.ImageNet Inception is reported both for the CIFAR-selected best generator and for the peak score reached by any generator during training.
- Baseline selection: Baseline training continued for the remaining 8 × 10^5 steps with constant learning rates fixed to their final exponentially decayed values.