Source-linked AI summary
An effective algorithm for hyperparameter optimization of neural networks
Gonzalo Diaz, Achille Fokoue, Giacomo Nannicini, Horst Samulowitz
TL;DR
The paper addresses the costly, heuristic selection of neural-network structure and training hyperparameters. It formulates the task as derivative-free optimization with bounded and integer-constrained variables, using RBFOpt and asynchronous parallel evaluations. Empirically, the method finds configurations with higher accuracy than domain-expert choices and results comparable to or better than popular alternatives, although its scope and assumptions are limited.
Problem
Selecting neural-network structure and training hyperparameters is time-consuming because evaluating a single configuration may require hours and manual tuning commonly relies on heuristics.
Source: Introduction · p. 1Method
The paper casts hyperparameter selection as a derivative-free, box-constrained optimization problem and uses RBFOpt’s surrogate-based search with asynchronous parallel evaluations.
Source: Introduction · p. 2, Derivative-free optimization: overview · p. 4, Parallelization of RBFOpt · p. 6Results
The method yields higher prediction accuracy than domain-expert configurations and is at least comparable, sometimes better, than popular existing methods on benchmark instances.
Source: Conclusion · p. 12Takeaways & Limitations
The open-source methodology provides an automated approach that can identify strong neural-network hyperparameters without relying solely on manual heuristic tuning.
Source: Introduction · p. 2, Conclusion · p. 12The study fixes the number of training epochs, treats validation accuracy as deterministic, and reports a limited-scope numerical evaluation.
Source: Introduction · p. 2, The hyperparameter optimization problem · p. 3
Abstract
from arXiv · showhide
A major challenge in designing neural network (NN) systems is to determine the best structure and parameters for the network given the data for the machine learning problem at hand. Examples of parameters are the number of layers and nodes, the learning rates, and the dropout rates. Typically, these parameters are chosen based on heuristic rules and manually fine-tuned, which may be very time-consuming, because evaluating the performance of a single parametrization of the NN may require several hours. This paper addresses the problem of choosing appropriate parameters for the NN by formulating it as a box-constrained mathematical optimization problem, and applying a derivative-free optimization tool that automatically and effectively searches the parameter space. The optimization tool employs a radial basis function model of the objective function (the prediction accuracy of the NN) to accelerate the discovery of configurations yielding high accuracy. Candidate configurations explored by the algorithm are trained to a small number of epochs, and only the most promising candidates receive full training. The performance of the proposed methodology is assessed on benchmark sets and in the context of predicting drug-drug interactions, showing promising results. The optimization tool used in this paper is open-source.
Introduction
The paper targets the time-consuming, heuristic task of selecting neural-network hyperparameters and introduces an automated optimization methodology. It combines derivative-free optimization with parallel evaluation through an extension of the open-source RBFOpt library.
- Neural-network hyperparameters include architecture choices such as hidden-layer number and size, plus learning and dropout rates.Source: Introduction · p. 1
- Evaluating one hyperparameter configuration can take several hours, making manual heuristic tuning long and tedious.Source: Introduction · p. 1
- The paper applies rigorous derivative-free optimization to hyperparameter selection, which the authors identify as novel for this task.Source: Introduction · p. 2
- The methodology transforms neural-network hyperparameter optimization into a box-constrained problem and extends RBFOpt for parallel, asynchronous evaluations.Source: Introduction · p. 2
- The authors report numerical evidence that the approach can provide significant benefits and outperform random search.Source: Introduction · p. 2
- The paper fixes the number of training epochs, treats validation accuracy as deterministic, and evaluates the method on a limited numerical scope.Source: Introduction · p. 2
The hyperparameter optimization problem
The paper formulates hyperparameter selection as optimizing validation performance over valid assignments that may mix continuous, integer, and other parameter domains. Because objective evaluations require training and testing a model, the resulting function is computationally expensive and may lack a closed-form expression.
- Hyperparameters govern a neural network’s structure and training, including hidden-layer number and size, learning rate, and dropout rate.Source: The hyperparameter optimization problem · pp. 2–3
- The feasible set X contains all valid assignments of hyperparameter values, which may belong to different domains such as positive reals and positive integers.Source: The hyperparameter optimization problem · p. 3
- For a configuration x, the objective f(x, D) estimates neural-network performance on dataset D using validation data or k-fold cross-validation.Source: The hyperparameter optimization problem · p. 3
- The objective function is computable but may have no known closed form, and evaluating one point can require extensive model training and validation.Source: The hyperparameter optimization problem · p. 3
Derivative-free optimization: overview
Derivative-free optimization is appropriate when the objective lacks an analytical form and evaluations are expensive. The reviewed approach uses a surrogate model to balance exploration and exploitation while restricting the search to bounded variables, including integer-constrained ones.
- Derivative-free optimization uses only function values, avoiding derivative information that is unavailable or impractical to compute for expensive objectives.Source: Derivative-free optimization: overview · pp. 3–4
- Because each evaluation may require several hours of CPU time, the method seeks good solutions with few objective evaluations.Source: Derivative-free optimization: overview · p. 4
- The optimization problem uses lower and upper bounds for variables and permits selected variables to take integer values.Source: Derivative-free optimization: overview · p. 4
- RBFOpt builds a radial-basis-function surrogate and selects evaluations by balancing exploration of unknown regions with exploitation of promising regions.Source: Derivative-free optimization: overview · p. 4
- After selecting a candidate through the surrogate and distance information, the algorithm evaluates the original objective, adds the point to the sample set, and iterates until stopping.Source: Derivative-free optimization: overview · p. 5
Derivative-free optimization applied to hyperparameter optimization
Applying derivative-free optimization to neural-network hyperparameters requires handling architectural validity constraints beyond simple box bounds. The paper also addresses the high and variable cost of neural-network evaluation through asynchronous parallelization.
- Valid neural-network architectures impose constraints beyond box bounds, including restrictions that prevent empty hidden layers between two layers.Source: Derivative-free optimization applied… · p. 5
Hyperparameter optimization of NNs as a box-constrained problem
The paper maps neural-network hyperparameter optimization into a box-constrained formulation by explicitly bounding the number of nonempty hidden layers. This formulation is intended to explore configurations more uniformly and achieves similar or better accuracy with less computation than a naive alternative.
Hyperparameter optimization of NNs as a box-constrained problem
- The formulation uses an upper bound on the number of hidden layers to map NN hyperparameter optimization into a box-constrained problem.The upper bound can be set large, but practitioners can typically provide a reasonable small bound.Source: Hyperparameter optimization of NNs as… · p. 5
- An additional variable explicitly controls the number of nonempty hidden layers, avoiding the dense networks encouraged by naive bounds.The naive formulation rarely tests configurations with empty layers because optimization spreads points across the design space.Source: Hyperparameter optimization of NNs as… · pp. 5–6
- The proposed formulation is likely to yield higher accuracy for similar or shorter computing times than the naive formulation, unless dense networks are more effective.The computational evaluation supports this claim.Source: Hyperparameter optimization of NNs as… · p. 6
Parallelization of RBFOpt
Because evaluating a neural network can take several hours and training times are nondeterministic, the paper extends RBFOpt for parallel, asynchronous objective-function evaluations.
Parallelization of RBFOpt
- Training and assessing one neural network can take several hours, making objective-function evaluation the dominant cost of hyperparameter optimization.The objective function is the performance of the tested neural network.Source: Parallelization of RBFOpt · p. 6
- The extension uses a task queue that assigns objective evaluations and new search-point computations to available processors asynchronously.Objective-evaluation tasks receive priority because they take longer.Source: Parallelization of RBFOpt · p. 6
Computational experiments
The empirical study covers a standard MNIST benchmark and a drug-related application evaluated on public WordNet and FreeBase datasets as well as non-public datasets.
Computational experiments
- The study first evaluates the methodology on the publicly available MNIST benchmark dataset.MNIST is used as the standard benchmark dataset.Source: Computational experiments · p. 6
- A specific application is evaluated on publicly available WordNet and FreeBase datasets and on non-public datasets.This application uses proprietary code.Source: Computational experiments · p. 7
Experiments on the MNIST dataset
On MNIST, RBFOpt is compared with random search and SMAC under matched hyperparameter spaces and repeated random seeds. The reported results show that RBFOpt and SMAC outperform random search early, while the proposed formulation achieves comparable accuracy with substantially lower optimization time than the naive formulation.
Experiments on the MNIST dataset
- The MNIST case study compares RBFOpt, random search, and SMAC on convolutional neural networks using the same hyperparameter space.Each algorithm is run with 10 different random seeds, using the same seeds across methods.Source: Experiments on the MNIST dataset · p. 7
- After 50 iterations, both RBFOpt and SMAC perform better than random search.The results are summarized using test-set accuracy, validation-accuracy snapshots, repeated runs, and statistical testing.Source: Experiments on the MNIST dataset · p. 8
- The naive and proposed formulations each achieve higher validation accuracy in 5 out of 10 runs after 100 iterations.The comparison uses RBFOpt under the same experimental framework.Source: Experiments on the MNIST dataset · p. 8
- The naive formulation requires an average CPU time 3.6 times larger than the proposed formulation.The larger time requirement is attributed to the naive formulation mostly exploring dense neural networks.Source: Experiments on the MNIST dataset · p. 8
Description of the application to drug-drug interaction
The paper applies neural-network hyperparameter optimization to drug-drug interaction prediction within the Tiresias system, while also evaluating the framework on benchmark knowledge-graph tasks.
- Tiresias receives drug-related data and knowledge and produces drug-drug interaction predictions, addressing a problem associated with preventable adverse drug reactions.Source: Description of the application to… · p. 9
- The application uses a multi-layer perceptron to predict DDIs directly from the fingerprints of two drugs.Source: Description of the application to… · p. 9
- The optimized hyperparameters include network depth, hidden-unit counts, activation functions, pre-training, and several learning parameters.Source: Description of the application to… · p. 9
- The authors also test Tiresias on publicly available multi-relational datasets to enable comparison with reported state-of-the-art results.Source: Description of the application to… · pp. 9–10
Numerical results using Tiresias
Experiments show that Tiresias with RBFOpt can find competitive or superior configurations across benchmark and DDI tasks, while asynchronous parallelism substantially reduces reported runtime.
- Within fewer than 30 search iterations, WN performance shows a clear jump, followed by another improvement at about 70 iterations.The search alternates between exploration and exploitation, producing oscillations in the progress curves.Source: Numerical results using Tiresias · p. 10
- RBFOpt + Tiresias is significantly better than the previously reported best results on WN for both mean rank and hits@10.Source: Numerical results using Tiresias · pp. 10–11
- On FB15k raw results, RBFOpt + Tiresias achieves mean rank 168 versus 243 and hits@10 47.9% versus 34.9% for TransE.Source: Numerical results using Tiresias · p. 11
- Parallel search reaches 76.6% accuracy and 290 mean rank, comparable to sequential search, while reducing runtime from approximately 92 hours to 19 hours.The parallel run evaluated approximately 160 configurations, compared with approximately 130 for the serial run.Source: Numerical results using Tiresias · p. 11
- For DDI prediction, the optimized configuration reaches 53.59% test accuracy versus 47.35% for a domain-expert configuration.The optimized configuration uses two hidden layers, whereas the domain-expert configuration uses one.Source: Numerical results using Tiresias · p. 12
Conclusion
The paper presents an open-source derivative-free methodology for neural-network hyperparameter optimization and reports empirically competitive or improved prediction accuracy across benchmark and application tasks.
- The methodology casts hyperparameter optimization as a derivative-free optimization problem and provides a corresponding software implementation.Source: Conclusion · p. 12
- Empirically, the method finds hyperparameters with higher prediction accuracy than those determined by a domain expert.Source: Conclusion · p. 12
- On benchmark instances, the results are at least comparable to and sometimes better than random search and sequential model-based algorithm configuration.Source: Conclusion · p. 12
- The authors identify choosing the number of training epochs as future work because the current methodology does not optimize that quantity.Source: Conclusion · p. 12