Source-linked AI summary
CMA-ES for Hyperparameter Optimization of Deep Neural Networks
Ilya Loshchilov, Frank Hutter
TL;DR
The paper addresses expensive and difficult-to-parallelize hyperparameter optimization for deep neural networks by evaluating CMA-ES as an alternative to Bayesian optimization. In a 30-GPU MNIST experiment, CMA-ES was competitive, particularly with parallel evaluations, with its latest solutions reaching validation errors below 0.4%.
Problem
DNN hyperparameter evaluations are expensive, making sequential optimization infeasible, while Bayesian optimization is sequential by nature and difficult to parallelize perfectly.
Method
The study applies CMA-ES, which iteratively samples candidate hyperparameter settings, evaluates them sequentially or in parallel, and adapts its sampling distribution toward good solutions.
Results
With 30 GPUs and 30-minute evaluations, CMA-ES improved so that most latest solutions had validation error below 0.4% and was competitive especially in parallel evaluations.
Takeaways & Limitations
CMA-ES is a powerful, computationally cheap alternative to include among methods for DNN hyperparameter optimization, particularly when evaluations are parallel.
Takeaways & Limitations
The study did not employ noise-reduction techniques or surrogate models, and the authors call for broader comparisons across more test problems and algorithm modifications.
Abstract
from arXiv · showhide
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions. We provide a toy example comparing CMA-ES and state-of-the-art Bayesian optimization algorithms for tuning the hyperparameters of a convolutional neural network for the MNIST dataset on 30 GPUs in parallel.
1 SUPPLEMENTARY MATERIAL
The supplementary material includes figures analyzing hyperparameter likelihoods and validation-error distributions, plus a table documenting the MNIST experiment hyperparameters.
- Supplementary figures: Figure 3 compares early and late hyperparameter-value likelihoods for CMA-ES and TPE with Gaussian priors over 1000 MNIST evaluations.Dotted lines represent the first 30 evaluations, while bold lines represent the last 100; likelihoods were estimated using diffusion-based kernel density estimation.
- Supplementary figures: Figure 4 presents estimated validation-error likelihoods on MNIST for different algorithms and separately reports cases exceeding 70% error.The additional percentage identifies divergence or near-divergence results because the density estimator poorly fits outliers near 90% error.
- Supplementary figures: Figure 5 reports preliminary CIFAR-10 validation errors for CMA-ES and TPE with Gaussian priors under 60- and 120-minute training budgets.These results use Adam without data augmentation and with ZCA whitening; the hyperparameter ranges differ from the MNIST table because the network is deeper.
- Supplementary tables: Table 1 documents the descriptions, pseudocode transformations, and ranges of hyperparameters used in the MNIST experiments.The table covers the hyperparameters used for the MNIST experiments.