Source-linked AI summary
Hyperparameter Search in Machine Learning
Marc Claesen, Bart De Moor
TL;DR
Machine learning algorithms rely on hyperparameters whose values can substantially affect model performance, yet manual and grid-based selection is difficult to reproduce and scale. The paper formalizes hyperparameter search as loss minimization and surveys its costly, stochastic, and complex optimization challenges, concluding that it is crucial for fully automated learning strategies.
Problem
Hyperparameter values strongly affect model performance, while manual and grid-based search are difficult to reproduce and impractical for large numbers of hyperparameters.
Method
The paper formalizes hyperparameter search as selecting algorithm parameters that minimize a chosen loss and discusses its optimization challenges and current approaches.
Results
The paper identifies costly and stochastic evaluations, large mixed-type spaces, and conditional hyperparameters as central challenges for automated search.
Takeaways & Limitations
Hyperparameter search is a crucial element in pursuing fully automated, self-configuring learning strategies.
Takeaways & Limitations
Because objective evaluations are stochastic, the empirically best hyperparameter tuple is not necessarily the true optimum of interest.
Abstract
from arXiv · showhide
We introduce the hyperparameter search problem in the field of machine learning and discuss its main challenges from an optimization perspective. Machine learning methods attempt to build models that capture some element of interest based on given data. Most common learning algorithms feature a set of hyperparameters that must be determined before training commences. The choice of hyperparameters can significantly affect the resulting model's performance, but determining good values can be complex; hence a disciplined, theoretically sound search strategy is essential.
1 Introduction
Machine learning methods use hyperparameters to configure algorithms, and their values can substantially affect model performance. Hyperparameter search is therefore formalized as selecting values that minimize loss, while manual approaches remain difficult to reproduce and scale.
- Machine learning methods capture structures or predict discrete and continuous targets from given data.
- Hyperparameters configure learning algorithms and can have widely varying effects on the resulting model and its performance.
- Manual rules-of-thumb and grid search are difficult to reproduce and impractical when many hyperparameters must be chosen.
- 1.2 Formalizing hyperparameter search: Hyperparameter search seeks values λ⋆ that produce a model minimizing the chosen loss on test data.
- 1.2 Formalizing hyperparameter search: The objective function maps hyperparameters to loss, with training and test data, learning algorithm, and loss function determining the search problem.
2 Challenges in hyperparameter search
Hyperparameter search is a demanding optimization problem because evaluations can be costly and stochastic, while search spaces may be large, mixed-type, and conditional. These properties complicate both efficient evaluation and reliable identification of good hyperparameters.
- Hyperparameter search is typically non-differentiable, single-objective optimization over a mixed-type, constrained domain.
- 2.1 Costly objective function evaluations: Model evaluations can take minutes, days, or weeks, creating pressure for methods that require few objective-function evaluations.
- 2.1 Costly objective function evaluations: Hyperparameter choices can alter training and testing time through model architecture, ensemble size, regularization, or kernel complexity.
- 2.2 Randomness: Randomness from learning procedures and finite samples makes objective values stochastic, while mitigating it can dramatically increase evaluation time.
- 2.2 Randomness: The empirically best hyperparameter tuple is not necessarily the true optimum of interest.
- 2.3 Complex search spaces: Search spaces can contain hundreds of hyperparameters, although only a handful may strongly affect performance and identifying them beforehand is difficult.
- 2.3 Complex search spaces: Some hyperparameters exist only conditionally on others, as when neural-network layer sizes depend on the number of hidden layers.
3 Current approaches
Hyperparameter search uses a broad range of optimization methods and increasingly dedicated software. Bayesian approaches are prominent, while metaheuristic methods and random sampling also serve as important alternatives or baselines.
- Methods used for hyperparameter search include particle swarm optimization, genetic algorithms, coupled simulated annealing, and racing algorithms.
- Random sampling was established relatively recently as a baseline for comparing optimization methods.
- Dedicated software packages commonly implement Bayesian methods and also offer metaheuristic approaches alongside machine-learning libraries.
4 Conclusion
The paper presents hyperparameter search as a crucial element in pursuing fully automated, self-configuring learning strategies. It argues that the field may benefit from techniques and lessons from metaheuristic optimization.
- Hyperparameter search is described as a crucial element in pursuing fully automated, self-configuring learning strategies.
- Metaheuristic optimization techniques and lessons may benefit automated hyperparameter search.