Source-linked AI summary
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, Frank Hutter
TL;DR
Large-dataset hyperparameter optimization is limited by evaluations that can take hours, days, or weeks. FABOLAS models loss and evaluation time as functions of dataset size, selecting cheaper subset evaluations to learn about the full-dataset optimum; across SVM and deep-network experiments, it often finds good configurations 10 to 100 times faster than competing methods.
Problem
Evaluations of machine-learning configurations on entire large datasets can take hours, days, or weeks, limiting the number of trials and achievable optimization performance.
Method
FABOLAS extends Bayesian optimization by modeling loss and evaluation time as functions of dataset size and selecting configuration–subset-size pairs using information gained per unit cost.
Results
10 to 100 times faster: FABOLAS often found good configurations than state-of-the-art Bayesian optimization methods and Hyperband across SVM and deep-neural-network experiments.
Takeaways & Limitations
FABOLAS often accelerates hyperparameter optimization by using smaller, cheaper dataset evaluations to gather information about good settings for the full dataset.
Takeaways & Limitations
Entropy Search requires several approximations because its primary numerical computation and integral are intractable.
Abstract
from arXiv · showhide
Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for large datasets, training and validating a single configuration often takes hours, days, or even weeks, which limits the achievable performance. To accelerate hyperparameter optimization, we propose a generative model for the validation error as a function of training set size, which is learned during the optimization process and allows exploration of preliminary configurations on small subsets, by extrapolating to the full dataset. We construct a Bayesian optimization procedure, dubbed Fabolas, which models loss and training time as a function of dataset size and automatically trades off high information gain about the global optimum against computational cost. Experiments optimizing support vector machines and deep neural networks show that Fabolas often finds high-quality solutions 10 to 100 times faster than other state-of-the-art Bayesian optimization methods or the recently proposed bandit strategy Hyperband.
1 Introduction
Machine-learning hyperparameter optimization becomes difficult when evaluations on large datasets take days or weeks, limiting the number of configurations that can be tested. FABOLAS addresses this by actively selecting dataset sizes during Bayesian optimization and extrapolating from smaller subsets to the full dataset.
- Hyperparameter choices strongly affect prediction error in support vector machines and neural networks, while naïve grid search scales poorly with dimensionality.Examples include SVM regularization and kernel parameters, alongside neural-network learning rates, dropout, and weight decay.
- Days- or weeks-long evaluations on entire datasets permit few trials, limiting the quality of the best configuration found.Experts instead often inspect subsets first, and this strategy can outperform contemporary Bayesian optimization methods.
- FABOLAS treats randomly subsampled dataset size as an additional optimization input and actively chooses it alongside each hyperparameter configuration.Dataset size is used during exploration but the objective remains good performance on the full dataset.
- FABOLAS differs from discrete multi-task approaches by exploiting regularity across dataset sizes to generalize to the full dataset without evaluating it directly.Earlier approaches use manually chosen subset sizes or fixed resource schedules, including Hyperband’s early discarding of poorly performing configurations.
- Experiments with SVMs and deep neural networks show FABOLAS often identifies good hyperparameter settings 10 to 100 times faster than full-dataset Bayesian optimization and Hyperband.The method is introduced as a Bayesian optimization procedure that trades information about the full-dataset optimum against computational cost.
2 Bayesian optimization
Bayesian optimization combines probabilistic models with acquisition functions to select evaluations, balancing exploration and exploitation under expensive objectives. Gaussian processes support tractable posterior modeling, while Entropy Search and Multi-Task Bayesian optimization provide information-based approaches with computational limitations and task assumptions.
- Bayesian optimization seeks a globally minimizing input by iteratively selecting evaluations through a prior model and an acquisition function.Each iteration optimizes the acquisition function, evaluates the selected point, adds the observation, and updates the posterior and acquisition function.
- 2.1 Gaussian Processes: Gaussian processes model the objective with a mean function and positive-definite covariance function, yielding analytically tractable posterior predictions.The posterior remains a Gaussian process after conditioning on observations.
- 2.1 Gaussian Processes: The Matérn 5/2 kernel with automatic relevance determination models hyperparameter relationships using a Mahalanobis distance and less restrictive smoothness assumptions.Its parameters are internal to the optimizer’s surrogate model, distinct from the target algorithm’s hyperparameters.
- 2.2 Acquisition functions: Acquisition functions trade off exploration and exploitation; the experiments use Expected Improvement and Entropy Search.Expected Improvement favors points with small predicted mean and/or large variance, whereas Entropy Search targets information about the optimum.
- 2.2 Acquisition functions: Entropy Search measures expected information gain about the function minimum, but computing the updated minimum distribution and integral is numerically intractable without approximations.Its information-based objective can nevertheless be traded against evaluation cost.
- 2.3 Multi-Task Bayesian optimization: Multi-Task Bayesian optimization transfers knowledge across correlated tasks using a Gaussian-process product kernel and information gain per unit cost.Using discrete dataset sizes as tasks can exploit cheaper subsets, but the approach requires modeling correlations among those tasks.
3 Fast Bayesian optimization for large datasets
FABOLAS extends Bayesian optimization with dataset size as a continuous input, modeling loss and computational cost to select informative, affordable evaluations. It extrapolates performance to the full dataset while automatically choosing subset sizes and tracks the best predicted full-data configuration.
- Method: FABOLAS models loss and computational cost as functions of hyperparameters and dataset size, allowing optimization across subset sizes.Dataset size is an additional input, while the objective remains performance on the full dataset.
- Method: The acquisition rule selects each hyperparameter–subset-size pair by trading off information about the full-data optimum against evaluation cost.It automatically determines how much data is needed to extrapolate usefully to the full dataset.
- Empirical motivation: 1/128 of MNIST was sufficient to locate a reasonable SVM configuration, with good configurations remaining broadly stable across subset sizes.The illustrative experiment evaluated 400 SVM configurations across relative subset sizes from 1/512 to 1.
- Method: A factorized kernel combines a stationary hyperparameter kernel with a finite-rank covariance over dataset size, using separate basis functions for loss and cost.The loss basis enforces decreasing loss with an extremum at full dataset size, while the cost model uses log-cost to fit polynomial scaling and ensure positive predictions.
- Method: The algorithm fits Gaussian processes for loss and cost, chooses evaluations by the acquisition function, measures outcomes and cost, and selects the incumbent using predicted loss at full dataset size.Its initial design is biased toward small, cheap datasets, and the method accounts for Bayesian-optimization reasoning overhead in computational cost.
4 Experiments
FABOLAS was evaluated against Bayesian optimization, MTBO, Hyperband, and related baselines on SVM and neural-network tasks. Across these experiments, it reached good configurations substantially faster, while speedups varied with dataset and model scaling.
- Experimental setup: The evaluation tracked wall-clock time, including optimization overhead and function evaluations, and compared incumbents through offline full-dataset testing.Most experiments used 10 independent runs with medians and interquartile error bars; the grid experiment used 30 runs.
- SVM grid on MNIST: An MTBO auxiliary task at relative dataset size s = 1/4 performed best, whereas performance degraded as the number of auxiliary tasks increased.The authors attribute the degradation mainly to discrete task-kernel parameters, insufficient MCMC sampling, and added computational overhead.
- SVM grid on MNIST: 105 seconds: EI and ES found the MNIST grid’s best configuration with error 0.014, while MTBO required around 2 × 10^4 seconds to find the global optimum.Random search was roughly five times slower than EI and ES; FABOLAS found good configurations roughly an order of magnitude faster than MTBO and reached the optimum at the same time.
- SVMs on various datasets: 10 to 1000 times faster: FABOLAS found good SVM configurations on MNIST, vehicle, and covertype than the other methods.These experiments also compared FABOLAS with Hyperband under realistic training costs on datasets containing more than 50000 data points.
- SVMs on various datasets: Hyperband’s first recommendation was already very good and arrived sooner than standard Bayesian optimization, but FABOLAS typically returned configurations of the same quality another order of magnitude faster.The comparison concerns methods’ time to produce good settings rather than only their eventual incumbents.
- Convolutional neural networks: 10 times faster: FABOLAS found good CNN configurations than vanilla Bayesian optimization on CIFAR10 and SVHN, although its speedup was smaller than for SVMs.CNNs scale linearly with the number of datapoints; Hyperband was slower than vanilla Bayesian optimization to make a recommendation but found good settings given enough time.
- Residual neural network: 10 times faster: FABOLAS found reasonable-performance residual-network configurations than ES and MTBO on CIFAR10.Hyperband was not run because one iteration exceeded a day, although the authors note benchmark-specific parameter adaptation could improve its runtime.
5 Conclusion
FABOLAS accelerates hyperparameter optimization by modeling dataset size, evaluation time, and information gain to prioritize inexpensive evaluations that inform full-dataset performance. The authors report substantial speedups, while noting that Gaussian-process complexity may limit practical scalability.
- FABOLAS treats dataset size as a continuous input and uses entropy search to identify informative, inexpensive evaluations.It models evaluation time and targets points yielding the most information per unit time about the globally best full-dataset hyperparameters.
- 10 to 100 times faster: FABOLAS often found good configurations than Multi-Task Bayesian optimization, Hyperband, and standard Bayesian optimization.
- Gaussian-process cubic complexity may become the limiting factor because FABOLAS replaces expensive evaluations with more numerous cheaper evaluations.
- Future extensions will model variables such as image resolution, number of classes, and number of epochs to seek additional speedups.