Source-linked AI summary
Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates
Ilija Ilievski, Taimoor Akhtar, Jiashi Feng, Christine Annette Shoemaker
TL;DR
Deep-learning hyperparameter optimization is difficult because evaluations are expensive and probabilistic surrogates become costly with many hyperparameters. The paper introduces HORD, a deterministic RBF-surrogate method with dynamic coordinate search, and reports stronger efficiency and final errors across DNN benchmarks, especially at higher dimensions.
Problem
Deep-learning hyperparameter optimization requires searching an expensive validation-error function, while probabilistic surrogates need covariance statistics and many evaluations as dimensionality grows.
Method
HORD uses a deterministic radial basis function surrogate and dynamic coordinate search to optimize mixed integer hyperparameters with fewer function evaluations.
Results
HORD performs better on average across the tested DNN problems and reaches the 19-CNN validation error achieved by SMAC after 200 evaluations in 54 evaluations, or 27%.
Takeaways & Limitations
HORD is especially effective for higher-dimensional deep-learning hyperparameter optimization, where its deterministic surrogate and candidate-generation strategy improve efficiency.
Takeaways & Limitations
The authors identify future extensions to parallelization and mixture surrogate models as ways to improve HORD's efficiency.
Abstract
from arXiv · showhide
Automatically searching for optimal hyperparameter configurations is of crucial importance for applying deep learning algorithms in practice. Recently, Bayesian optimization has been proposed for optimizing hyperparameters of various machine learning algorithms. Those methods adopt probabilistic surrogate models like Gaussian processes to approximate and minimize the validation error function of hyperparameter values. However, probabilistic surrogates require accurate estimates of sufficient statistics (e.g., covariance) of the error distribution and thus need many function evaluations with a sizeable number of hyperparameters. This makes them inefficient for optimizing hyperparameters of deep learning algorithms, which are highly expensive to evaluate. In this work, we propose a new deterministic and efficient hyperparameter optimization method that employs radial basis functions as error surrogates. The proposed mixed integer algorithm, called HORD, searches the surrogate for the most promising hyperparameter values through dynamic coordinate search and requires many fewer function evaluations. HORD does well in low dimensions but it is exceptionally better in higher dimensions. Extensive evaluations on MNIST and CIFAR-10 for four deep neural networks demonstrate HORD significantly outperforms the well-established Bayesian optimization methods such as GP, SMAC, and TPE. For instance, on average, HORD is more than 6 times faster than GP-EI in obtaining the best configuration of 19 hyperparameters.
Introduction
Deep-learning hyperparameter selection is crucial but difficult because many architectural and training choices define an expensive, complex validation-error function. HORD addresses this problem with a deterministic RBF surrogate and dynamic coordinate search for mixed integer and continuous hyperparameters.
- Deep-learning adoption is hampered by numerous hyperparameter choices, including network architectures, activation functions, and learning rates.
- Hyperparameter optimization treats validation error as an expensive black-box function of D configurable hyperparameters and learned network parameters.
- HORD uses a deterministic radial basis function surrogate instead of covariance-based probabilistic modeling, reducing the number of expensive function evaluations required.
- Dynamic coordinate search enables HORD to optimize both integer and continuous hyperparameters while seeking near-optimal configurations with few evaluations.
- HORD is evaluated against GP-EI, GP-PES, TPE, and SMAC on neural networks using MNIST and CIFAR-10 benchmarks.
- Extensive evaluations report fewer evaluations, lower final validation error, and 3.7- to 6-fold speedups over other algorithms for 19-dimensional problems.
Related Work
Surrogate-based optimization replaces expensive black-box evaluations with inexpensive surrogate-guided searches. Bayesian methods use probabilistic models and acquisition functions, while prior evidence motivates comparing GP, TPE, and SMAC with HORD across dimensionalities.
- Surrogate-based optimization seeks near-optimal solutions to constrained expensive black-box functions using as few evaluations as possible.
- Bayesian optimization models the error distribution probabilistically and selects evaluations by optimizing an acquisition function.
- GP-EI and GP-PES use Gaussian processes with expected improvement and predictive entropy search, respectively.
- TPE and SMAC address GP scalability differently, using tree-structured Parzen estimators and random forests to estimate error density.
- Prior empirical results associate GP approaches with strong low-dimensional performance and tree-based approaches with strong high-dimensional performance.
- Earlier surrogate work modeled validation error for comparing optimizers but did not unify the surrogate model with an optimization algorithm.
- A neural-network surrogate used fewer computational resources than GP methods at large evaluation counts but did not outperform GP-based or tree-based algorithms.
Description of The Proposed Method HORD
HORD combines an RBF interpolation surrogate with dynamic coordinate-based candidate generation for mixed integer global optimization. It balances predicted error and distance from evaluated points when selecting each expensive evaluation.
- HORD is a hyperparameter optimizer using an RBF-based surrogate and dynamic coordinate search.
- The surrogate model: The surrogate interpolates validation errors from n configurations in a D-dimensional hyperparameter space.
- The surrogate model: The model uses a cubic spline RBF with Euclidean distances and a linear polynomial tail; its parameters are obtained from a linear system.
- Algorithm: HORD initializes with Latin hypercube samples, repeatedly updates the surrogate, generates candidate points, evaluates the selected point, and returns the lowest-error configuration.
- Candidate hyperparameter generation and selection: Candidates perturb coordinates of the current best point, with perturbation probability decreasing across iterations and integer coordinates rounded when required.
- Candidate hyperparameter generation and selection: HORD limits the initial average number of perturbed coordinates to fewer than 20 and adjusts perturbation variance after improvement or stagnation.
- Candidate hyperparameter generation and selection: The acquisition score combines surrogate-estimated value and distance from evaluated points using cyclic weights, and the lowest-scoring candidate is evaluated next.
Experimental Setup
The evaluation spans four DNN hyperparameter problems from 6 to 19 dimensions on MNIST and CIFAR-10, comparing HORD with GP, TPE, and SMAC under a limited evaluation budget. HORD-ISP additionally tests manually supplied starting points in the two highest-dimensional CNN problems.
- DNN problems: The experiments include 6-MLP and 8-CNN problems on MNIST, followed by 15-CNN and 19-CNN problems with higher-dimensional spaces.
- DNN problems: The four problems contain 6, 8, 15, and 19 hyperparameters, including both continuous and integer variables.
- DNN problems: The 19-CNN task uses CIFAR-10 and adds four dropout layers to the CNN architecture used in the 15-CNN problem.
- Baseline algorithms: HORD is compared with GP-EI, GP-PES, TPE, and SMAC.
- Evaluation budget, trials and initial points: Each experiment uses at most 200 function evaluations, with one evaluation comprising full DNN training and evaluation, and at least five random-seed trials.
- Evaluation budget, trials and initial points: HORD-ISP incorporates a manually supplied initial point for the 15-CNN and 19-CNN tasks, using the same initial point as SMAC.
Numerical Results and Discussion
Across the tested neural-network problems, HORD and HORD-ISP generally achieved strong validation and test errors while requiring substantially fewer evaluations, especially in higher-dimensional spaces. The results also indicate that HORD’s dynamic coordinate-search strategy contributes to its efficiency.
- Evaluation protocol: The evaluation compared methods using validation error for optimization and test-set error for final performance, with a 200-function-evaluation budget.Each function evaluation involved a full DNN training and evaluation, and experiments used at least five trials.
- Evaluation efficiency: HORD reached the mean best validation error obtained by SMAC after 200 evaluations using only 54 evaluations, or 27% of SMAC’s budget, on 19-CNN.The comparison concerns the mean best validation error over five trials.
- Overall performance: After 200 evaluations, HORD and HORD-ISP achieved the best average test-set results across all tested problems.The comparison used at least five trials with different random seeds.
- Lower-dimensional problems: HORD outperformed other algorithms on average for 6-MLP and 8-CNN after 200 evaluations, while GP-EI remained statistically comparable in these lower-dimensional cases.For 6-MLP, HORD’s result was not statistically distinguishable from GP-EI; for 8-CNN, the reported GP-EI p-value was 0.391.
- Higher-dimensional problems: Higher-dimensional problems were where HORD-ISP performed especially well: it continued to perform strongly against SMAC and TPE as GP-based performance degraded.The comparison covered the 15-CNN and 19-CNN problems.
- Search behavior: After the initial n0 evaluations, HORD and HORD-ISP concentrated proposals in lower-validation-error regions, unlike the other methods’ broader search behavior.For 19-CNN, their proposed values were mostly in the 20–30% validation-error range, and they soon reached errors below the other methods’ final lowest errors.
- Evaluation efficiency: In 19-CNN, HORD-ISP matched GP-EI’s 200-evaluation result in 33 evaluations, making it about 6 times faster, while HORD was about 3.7 times faster than SMAC.The reported speedups are based on 200 divided by 33 and 200 divided by 54, respectively.
- Search behavior: HORD uses DYCORS to perturb only a fraction of dimensions, combining this trial-point strategy with surrogate search to improve efficiency in higher-dimensional problems.The paper contrasts this strategy with the candidate-generation procedures used by the baseline methods.
Conclusion
HORD is presented as an effective hyperparameter optimization method, with reported speed advantages on higher-dimensional problems. The authors identify parallelization and mixture surrogate models as future extensions for improving efficiency.
- HORD and HORD-ISP can be up to 6 times faster than the best competing methods on higher-dimensional problems.
- HORD's efficiency is attributed to its deterministic radial basis surrogate and candidate generation that increasingly concentrates perturbations near the current best solution.
- Parallelization and mixture surrogate models are planned extensions intended to improve HORD's efficiency.