Source-linked AI summary
Practical Bayesian Optimization of Machine Learning Algorithms
Jasper Snoek, Hugo Larochelle, Ryan P. Adams
TL;DR
Hyperparameter tuning is expensive and involves practical choices about covariance functions, experiment duration, and parallelism. The paper develops Bayesian optimization practices and algorithms addressing these issues, finding better-than-expert CIFAR-10 error and faster LDA optimization than comparison methods.
Problem
Bayesian optimization for machine-learning hyperparameters needs practical choices for covariance models, variable experiment durations, and multi-core parallelism.
Method
The paper studies Bayesian optimization practices, including fully Bayesian kernel treatment, cost-aware experiment selection, and parallel experimentation.
Results
Parallelized GP EI MCMC found better LDA minima in less time, while its CIFAR-10 configuration achieved 14.98% test error, over 3% better than the expert and state of the art.
Takeaways & Limitations
These Bayesian optimization procedures can improve hyperparameter search efficiency and exceed expert or grid-search baselines on machine-learning tasks.
Takeaways & Limitations
Cost-aware optimization assumes that the objective and duration functions are independent, although coupled Gaussian-process models may capture their relationship.
Abstract
from arXiv · showhide
Machine learning algorithms frequently require careful tuning of model hyperparameters, regularization terms, and optimization parameters. Unfortunately, this tuning is often a "black art" that requires expert experience, unwritten rules of thumb, or sometimes brute-force search. Much more appealing is the idea of developing automatic approaches which can optimize the performance of a given learning algorithm to the task at hand. In this work, we consider the automatic tuning problem within the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). The tractable posterior distribution induced by the GP leads to efficient use of the information gathered by previous experiments, enabling optimal choices about what parameters to try next. Here we show how the effects of the Gaussian process prior and the associated inference procedure can have a large impact on the success or failure of Bayesian optimization. We show that thoughtful choices can lead to results that exceed expert-level performance in tuning machine learning algorithms. We also describe new algorithms that take into account the variable cost (duration) of learning experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization on a diverse set of contemporary algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks.
1. Introduction.
The section frames hyperparameter tuning as expensive black-box optimization of generalization performance and motivates Bayesian optimization as a model-based approach. It highlights variable evaluation costs and argues that robust results depend critically on fully Bayesian treatment of GP kernel parameters.
- Hyperparameter tuning can be automated by treating generalization performance as an unknown black-box function whose evaluations are expensive.
- GP-based Bayesian optimization models observed generalization performance across hyperparameter settings and can select experiments using expected improvement or UCB.
- Machine-learning evaluations have variable duration and may incur differing economic costs, such as when larger models require more training time or memory.
- The work identifies good practices for Bayesian optimization, emphasizing fully Bayesian inference over GP kernel parameters for robust results and examining kernel choice.
- Bayesian optimization builds a probabilistic model of the objective and uses uncertainty and prior observations to choose the next evaluation.
2. Bayesian Optimization with Gaussian Process Priors.
The method models the objective with a flexible, tractable Gaussian process prior and selects new evaluations through acquisition functions derived from the posterior. It focuses on expected improvement because it is analytically tractable, empirically well-behaved, and requires no additional tuning parameter.
- Bayesian optimization choices: Bayesian optimization requires choosing both a function prior and an acquisition function for selecting the next evaluation.The acquisition function converts the model posterior into a utility function and chooses x_next = argmax_x a(x).
- Gaussian process prior: A Gaussian process prior assigns multivariate Gaussian distributions to every finite set of function values, enabling closed-form marginals and conditionals.Its function-space behavior is determined by a mean function m and positive definite covariance function K.
- Posterior and acquisition: Noisy observations under the GP prior induce a posterior over functions that acquisition functions use to determine the next point.The observations satisfy y_n ∼ N(f(x_n), ν), where ν is the observation-noise variance.
- Acquisition criteria: Probability of improvement, expected improvement, and GP-UCB provide alternative acquisition criteria under the Gaussian process.Probability of improvement and expected improvement have closed forms, while GP-UCB uses a tunable κ to balance exploitation and exploration.
- Expected improvement: The work focuses on expected improvement because it is better-behaved than probability of improvement and avoids the additional tuning parameter required by GP-UCB.The authors report that expected improvement performs well in minimization problems, while noting that regret formalization may suit many settings better.
3. Practical Considerations for Bayesian Optimization of Hyperparameters.
Practical Bayesian optimization requires choosing a realistic GP covariance, integrating uncertainty over GP hyperparameters, and accounting for variable evaluation costs and pending parallel experiments. The section introduces Matérn 5/2 modeling, integrated acquisition functions, expected improvement per second, and fantasy-based parallel selection.
- Covariance functions: The default ARD squared exponential kernel produces unrealistically smooth sample functions, so the authors propose an ARD Matérn 5/2 kernel.The proposed kernel yields twice-differentiable samples without imposing squared-exponential smoothness.
- GP hyperparameters: Uncertainty in GP hyperparameters is handled by marginalizing acquisition functions, using posterior samples to obtain a Monte Carlo estimate of integrated expected improvement.The samples can be acquired efficiently using slice sampling.
- Variable evaluation cost: Expected improvement per second selects points that are likely to be good and quick to evaluate, extending naturally to other budgeted resources such as reagents or money.The motivation is optimizing progress in wallclock time rather than merely counting function evaluations.
- Variable evaluation cost: Because evaluation duration is unknown, the method models ln c(x) alongside the objective f(x) with an independent Gaussian process.The authors note that coupled modeling could instead use GP variants of multi-task learning.
- Parallel experimentation: For parallel optimization, pending evaluations are represented by fantasies, and acquisition functions are averaged over their possible outcomes to choose the next non-duplicate point.The approach supports selecting new evaluations while a set of points is still running.
4. Empirical Analyses.
The empirical analyses evaluate Bayesian optimization across benchmark functions and challenging machine-learning tasks, including online LDA, M3E models, and convolutional neural networks. The proposed strategies improve efficiency, benefit from appropriate covariance choices and parallelization, and achieve better results than established baselines in these settings.
- Standard benchmarks: The study compares GP EI MCMC with standard approaches and Tree Parzen Algorithm on the Branin-Hoo benchmark and MNIST logistic regression.The MNIST task requires selecting four hyperparameters.
- Online LDA: Parallelized GP EI MCMC finds the best online LDA parameters in significantly less time and achieves a better minimum than Hoffman et al.’s grid search with fewer experiments.GP EI MCMC is most efficient by function evaluations, while parallelization improves walltime performance.
- M3E protein motif finding: Bayesian optimization strategies are considerably more efficient than randomized grid search for M3E protein motif finding in both wallclock time and function evaluations.The comparison includes GP EI MCMC, GP EI per Second, and their 3x parallelized versions, all constrained to the same grid points.
- M3E protein motif finding: Covariance-function selection significantly affects GP EI MCMC performance on M3E optimization, making length-scale estimation critical.The squared-exponential covariance’s infinite differentiability assumption is described as too restrictive for the underlying function.
- Convolutional neural networks: 14.98% test error was achieved by GP EI MCMC on CIFAR-10, over 3% better than the expert and the state of the art.The result uses hyperparameters optimized over nine parameters on a withheld validation set, with five randomly initialized runs reported.