PaperlayerView exact revision on arXiv

Source-linked AI summary

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice

Li Yang, Abdallah Shami

arXiv:2007.15745v3cs.LGstat.ML

TL;DR

Manual hyper-parameter selection is difficult because ML models use heterogeneous, interacting settings and expensive evaluations, while optimization methods have different strengths and drawbacks. This survey reviews HPO techniques, model-specific hyper-parameters, tools, challenges, and benchmark comparisons. It concludes that method suitability depends on configuration space, resources, and model characteristics, with important limitations remaining for costly or poorly representative evaluations.

  • Problem

    HPO is needed because ML hyper-parameters affect performance, yet their heterogeneous domains, interactions, nonconvex objectives, and costly evaluations make manual or traditional optimization difficult.

  • Method

    The paper surveys common ML hyper-parameters, state-of-the-art HPO techniques, libraries, frameworks, open challenges, and practical comparisons across ML models and datasets.

  • Results

    The survey reports that HPO method suitability varies by configuration space and resources: BO-TPE, BOHB, and PSO often work well for large spaces, while BO-GP suits small spaces and GA suits large spaces.

  • Takeaways & Limitations

    Selecting an HPO method should account for the ML model, hyper-parameter types, configuration-space size, computational resources, and evaluation behavior.

  • Takeaways & Limitations

    HPO remains constrained by costly objective evaluations and by assumptions such as small-budget evaluations being representative of full training.

Abstract

from arXiv · show

Machine learning algorithms have been used widely in various applications and areas. To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. It often requires deep knowledge of machine learning algorithms and appropriate hyper-parameter optimization techniques. Although several automatic optimization techniques exist, they have different strengths and drawbacks when applied to different types of problems. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization. This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.

1. Introduction

Hyper-parameter optimization addresses the difficulty of manually configuring machine-learning models by comparing optimization techniques and explaining their suitability for different problems. The survey also reviews practical tools, challenges, and algorithm-selection considerations.

  • Manual hyper-parameter tuning is ineffective for many problems because models can have many parameters, costly evaluations, and nonlinear parameter interactions.
  • HPO automates hyper-parameter tuning to reduce human effort, improve model performance, and support more reproducible comparisons.
  • Grid search exhaustively evaluates fixed hyper-parameter values, whereas random search samples configurations independently within available resources.
  • Bayesian optimization uses previous evaluations to select subsequent hyper-parameter values, reducing unnecessary evaluations relative to grid and random search.
  • Multi-fidelity methods and metaheuristics address resource limits or large search spaces, with Hyperband, genetic algorithms, and particle swarm optimization serving distinct practical roles.
  • The survey reviews common ML algorithms, HPO techniques, libraries, frameworks, open challenges, and practical algorithm-selection considerations.

2. Mathematical Optimization and Hyper-parameter Optimization Problems

Machine-learning HPO is a constrained optimization problem over heterogeneous, often nonconvex hyper-parameter spaces. The paper defines its components and explains why methods designed for simpler optimization settings may be unsuitable.

  • Optimization problems specify decision variables, an objective function to minimize or maximize, and constraints defining feasible regions.
  • HPO commonly requires constrained optimization because model configurations and resources impose bounds on feasible solutions.
  • Most ML and HPO problems are nonconvex, so inappropriate optimization methods may return a local rather than global optimum.
  • An HPO process combines an estimator and objective function, a search space, an optimization method, and an evaluation function.
  • Hyper-parameters may be continuous, discrete, binary, categorical, or conditional, producing complex constrained search spaces.
  • The HPO objective is to find an optimal or near-optimal configuration within performance and time budgets.

3. Hyper-parameters in Machine Learning Models

Selecting hyper-parameters requires identifying which settings govern each machine-learning algorithm and dataset. The paper frames this identification as an initial step in applying HPO effectively.

  • The first step in applying HPO is identifying the key hyper-parameters that must be tuned for a particular model, problem, or dataset.
  • Supervised learning maps input features to targets using labeled data, while unsupervised learning models unlabeled data.

3.1. Supervised Learning Algorithms

Supervised-learning models use hyper-parameters to determine predictive structures, losses, regularization, neighborhood rules, and training behavior. Their relevant settings vary substantially across algorithm families.

  • Supervised learning seeks a predictive function that minimizes loss over labeled training examples within the model family allowed by its hyper-parameters.
  • Linear regression usually requires no hyper-parameter tuning, while ridge and lasso regression tune regularization strength α.
  • Logistic regression tunes the penalty, regularization coefficient C, and solver, whose settings can be correlated.
  • KNN primarily tunes the neighbor count k, with weighting and distance choices also depending on the problem.
  • Naïve Bayes often requires no tuning or only tuning the continuous smoothing parameter α; Complement NB suits imbalanced data, while Bernoulli NB requires binary features.
  • Deep-learning hyper-parameters include hidden-layer and neuron counts, loss and activation functions, optimizer type, mini-batch size, and epochs.

3.2. Unsupervised Learning Algorithms

Unsupervised learning includes clustering and dimensionality-reduction methods, each with algorithm-specific hyper-parameters. The number of clusters or extracted components is especially important to tune.

  • Unsupervised learning comprises clustering and dimensionality-reduction algorithms for identifying patterns in unlabeled data.
  • Clustering Algorithms: The number of clusters is the most important hyper-parameter in k-means, EM, and hierarchical clustering.
  • Clustering Algorithms: K-means minimizes squared errors using k cluster centroids, with n clusters specifying the required number of clusters.
  • Clustering Algorithms: K-means performance is also slightly affected by centroid initialization, n init, and max iter.
  • Clustering Algorithms: DBSCAN replaces a preset cluster count with eps and min samples, which jointly define cluster density.
  • Dimensionality Reduction Algorithms: PCA and LDA primarily tune n components, the number of features extracted during dimensionality reduction.

4. Hyper-parameter Optimization Techniques

The section contrasts manual, grid, and random search for hyper-parameter optimization. Manual tuning is difficult at scale, while grid search suffers from dimensionality and random search uses a fixed evaluation budget.

  • Manual Tuning: Manual tuning becomes infeasible with many hyper-parameters, complex models, costly evaluations, and nonlinear hyper-parameter interactions.
  • Grid Search: Grid search exhaustively evaluates the Cartesian product of user-specified hyper-parameter values.
  • Grid Search: Grid search cannot exploit well-performing regions automatically, so narrowing the search space based on results must be performed manually.
  • Grid Search: O(n^k) complexity makes grid search inefficient when k hyper-parameters each have n distinct values.
  • Random Search: Random search samples a fixed number of configurations, exploring a larger search space than grid search under a limited budget.
  • Random Search: O(n) complexity and independent evaluations make random search easy to parallelize and resource-allocate.

4.2. Gradient-based Optimization

Gradient-based optimization follows gradients toward an optimum and can converge faster than earlier methods. Its use for HPO is limited by gradient availability and non-convex objectives.

  • Gradient-based Optimization: Gradient descent moves opposite the largest gradient to identify a promising direction toward an optimum.
  • Gradient-based Optimization: Gradient-based algorithms can converge faster to a local optimum than the previously presented optimization methods.
  • Gradient-based Optimization: They require calculable hyper-parameter gradients and are efficient mainly for convex functions, such as neural-network learning-rate optimization.

4.3. Bayesian Optimization

Bayesian optimization iteratively uses a surrogate model and acquisition function to choose evaluations from prior results. It can be efficient across varied HPO objectives, but sequential dependence limits parallelization and poor exploration–exploitation balance can yield local optima.

  • Bayesian Optimization: Bayesian optimization selects future evaluations from previous results using a surrogate model and an acquisition function.
  • Bayesian Optimization: Its workflow builds a surrogate, selects promising hyper-parameters, evaluates them on the objective, updates the surrogate, and repeats.
  • Bayesian Optimization: Surrogate-model evaluation is often cheaper than evaluating the full objective, making Bayesian optimization more efficient than grid and random search.
  • BO-GP: BO-GP mainly targets continuous variables and has O(n^3) time and O(n^2) space complexity.
  • BO-RF / SMAC: SMAC supports continuous, discrete, categorical, and conditional hyper-parameters, with fitting and prediction complexities of O(nlogn) and O(logn).
  • BO-TPE: TPE retains conditional dependencies through tree-structured density models and has O(nlogn) time complexity.
  • Bayesian Optimization: Bayesian optimization handles stochastic, non-convex, and non-continuous objectives, but sequential dependence makes it difficult to parallelize.

4.4. Multi-fidelity Optimization Algorithms

Multi-fidelity methods reduce HPO cost by evaluating configurations with different resource budgets and eliminating poor performers early. Hyperband and BOHB improve this strategy through adaptive allocation and Bayesian-guided sampling, but BOHB depends on low-budget evaluations being representative.

  • Larger configuration spaces and datasets can make HPO take hours or days, motivating multi-fidelity methods that use reduced datasets or feature subsets.
  • Successive Halving: Successive halving starts with n configurations at budget b = B/n, removes the poorer half after each evaluation, and doubles budgets for survivors.
  • Successive Halving: Successive halving is more efficient than random search but must trade off the number of configurations against the budget allocated to each.
  • Hyperband: Hyperband dynamically chooses configuration counts and applies successive halving across random configurations to balance exploration and resource allocation.
  • BOHB: BOHB replaces Hyperband’s random search with Bayesian optimization, and it has outperformed many techniques on SVM and deep-learning tuning.Its limitation is that small-budget rankings must represent full-training-set rankings; otherwise, convergence can be slower than standard Bayesian optimization.

4.5. Metaheuristic Algorithms

Metaheuristic HPO methods search complex spaces through evolving populations or cooperative particle swarms. Genetic algorithms are flexible but sequential and costly, whereas PSO is simpler and more parallelizable but sensitive to initialization.

  • Overview: Metaheuristics address non-convex, non-continuous, and non-smooth optimization problems, while population-based methods can evaluate individuals in parallel.
  • Genetic Algorithm: Genetic algorithms represent hyper-parameter configurations as chromosomes and iteratively use fitness evaluation, selection, crossover, and mutation to produce new generations.
  • Genetic Algorithm: Random initialization makes genetic algorithms usable without expert-designed initializations because evolutionary operations reduce the chance of missing the global optimum.
  • Genetic Algorithm: Genetic algorithms add configuration choices such as population size and mutation rate, are difficult to parallelize, and have time complexity O(n^2).These properties can produce low convergence speed and inefficiency.
  • Particle Swarm Optimization: PSO updates particles using individual-best and global-best information, then moves them according to updated velocities until convergence or termination.
  • Particle Swarm Optimization: PSO is simpler than genetic algorithms, has computational complexity O(nlogn), and supports parallelization through particle-wise information exchange.Its main limitation is that poor initialization can lead to a local rather than global optimum, especially for discrete hyper-parameters.

5. Applying Optimization Techniques to Machine Learning Algorithms

The paper recommends matching HPO methods to hyper-parameter structure and computational constraints. Bayesian, bandit-based, and metaheuristic methods serve different configuration types, while no single algorithm is universally suitable.

  • Optimization Techniques Analysis: Grid search is time-consuming, suffers from the curse of dimensionality, and cannot reliably find continuous optima from a finite predefined value set.
  • Optimization Techniques Analysis: Random search is more efficient than grid search and supports all hyper-parameter types, but ignores previous evaluations and may waste trials.
  • Optimization Techniques Analysis: Hyperband improves resource use over random search, but grid search, random search, and Hyperband ignore correlations among conditional hyper-parameters.
  • Optimization Techniques Analysis: Gradient-based methods support only continuous hyper-parameters and may find local rather than global optima in non-convex HPO problems.
  • Choosing Methods: BOHB is recommended when low-budget evaluations preserve full-budget rankings; otherwise, standard Bayesian optimization or other methods may be more efficient.
  • Model-Specific Choices: For one continuous hyper-parameter, BO-GP is recommended, while BO-TPE or SMAC suit conditional configurations and PSO suits large mixed-type spaces requiring parallel execution.
  • Conclusion: The suitable HPO algorithm should be selected according to hyper-parameter properties to achieve high model performance with low computational cost.

6. Existing HPO Frameworks

The paper surveys open-source HPO libraries covering grid and random search, Bayesian optimization, Hyperband, evolutionary methods, and specialized machine-learning integrations. Their capabilities differ in supported algorithms, variable types, parallelism, and framework compatibility.

  • General-Purpose Libraries: Scikit-learn provides GridSearchCV and RandomizedSearchCV, using cross-validation to evaluate exhaustive or randomly selected configurations.
  • Bayesian Optimization: Bayesian libraries include Spearmint and BayesOpt with Gaussian-process surrogates, while SMAC uses random forests and supports categorical, continuous, and discrete variables.
  • Bayesian Optimization: Hyperopt combines random search and BO-TPE, models hierarchical hyper-parameters, and supports parallelism through a central MongoDB database.
  • Multi-Fidelity Libraries: BOHB combines Bayesian optimization with Hyperband and uses TPE to trade off model performance against available budgets.
  • Integrated Frameworks: Optunity, Skopt, Talos, Sherpa, and Osprey provide combinations of search, Bayesian, probabilistic, or Hyperband methods for general or framework-specific workflows.
  • Evolutionary and Auto-ML Tools: Evolutionary and auto-ML tools include DEAP, TPOT, and Nevergrad, offering GA, PSO, genetic programming, or broad optimizer collections for machine-learning problems.

7. Experiments

The experiments compare eight HPO methods across KNN, SVM, and RF models on MNIST classification and Boston-housing regression, using standardized search spaces and repeated cross-validation. Results show trade-offs between computational time, configuration-space size, and optimization performance.

  • Experimental design: The study evaluates eight HPO approaches on KNN, SVM, and RF models using MNIST and Boston-housing benchmark datasets.MNIST supports classification evaluation, while Boston-housing supports regression evaluation.
  • Experimental design: Experiments use 3-fold cross-validation, accuracy for classification, and mean squared error for regression.The comparisons also report computational time.
  • Experimental design: All methods share the same hyper-parameter configuration spaces and fixed iteration budgets, with ten random-seed repetitions to reduce randomness.The iteration limits are 50 for RF and SVM and 10 for KNN; regression results are averaged and classification results use majority voting.
  • Performance comparison: Default hyper-parameter configurations do not produce the best performance, while grid search is often substantially slower than other methods.Random search is faster than grid search at the same search-space size, but neither guarantees near-optimal configurations, especially for RF and SVM.
  • Performance comparison: Bayesian optimization and multi-fidelity methods outperform grid and random search overall, with BO-TPE and BOHB often finding optimal or near-optimal configurations quickly.BO-GP performs well on small continuous spaces but has higher computational cost, whereas Hyperband is fast but may miss the highest accuracy or lowest MSE.
  • Performance comparison: GA and PSO often achieve stronger classification accuracy and lower regression MSE, but they generally require more computation than BO-TPE and multi-fidelity methods.GA is especially costly because it does not support parallel execution; for large configuration spaces, BO-TPE, BOHB, and PSO often work well.

8. Open Issues, Challenges, and Future Research Directions

The paper identifies computational cost, scalability, generalization, stochasticity, and comparability as major unresolved HPO challenges. It proposes benchmarks, combined methods, budget-aware optimization, and continuously adaptable approaches as future directions.

  • Costly Objective Function Evaluations: Objective-function evaluations can require minutes, hours, days, or longer, while some hyper-parameters also directly increase execution time.Evaluation cost depends on data scale, model complexity, and available computational resources.
  • Costly Objective Function Evaluations: Bayesian optimization reduces evaluations but has poor parallelization, whereas multi-fidelity methods may remain ineffective for highly complex models and massive datasets.ImageNet is cited as an unresolved example because of its scale and CNN complexity.
  • Model Complexity and Resource Budgets: HPO becomes increasingly difficult as the number of hyper-parameters and configurations expands the search space, creating substantial resource demands for complex models.Deep learning models are highlighted as resource-intensive black-box optimization settings under limited practical budgets.
  • Comparability of HPO Methods: No single optimization approach consistently outperforms others across datasets, metrics, and hyperparameter types, motivating standardized benchmarks and unified metrics.A reliable platform covering common HPO approaches is still described as unavailable.
  • Future Method Design: Future HPO methods should combine complementary models, balance model performance with time budgets, and scale across continuous, discrete, categorical, and conditional hyper-parameters.BOHB is given as an example combining Bayesian optimization and Hyperband.
  • Generalization and Practical Deployment: Finite evaluations can overfit validation data, while cross-validation may improve stability at the cost of several-fold longer execution times.The paper also identifies stochastic objectives, limited distributed-framework support, and changing data as generalization and deployment challenges.

9. Conclusion

The conclusion surveys state-of-the-art HPO research and its application to ML models, emphasizing that hyperparameter types should guide method selection. It recommends BOHB, Bayesian optimization, or PSO under different dataset and search-space conditions.

  • Conclusion: The survey covers state-of-the-art HPO research, practical application to ML models, and the role of hyperparameter types in selecting an optimization method.The conclusion frames automatic optimization as increasingly important because manual tuning is computationally expensive for growing real-world datasets.
  • Conclusion: BOHB is recommended when randomly selected subsets represent the dataset well because it can efficiently optimize all hyper-parameter types.The recommendation is conditional on subset representativeness.
  • Conclusion: Bayesian optimization is recommended for small hyper-parameter configuration spaces, while PSO is usually recommended for large configuration spaces.These recommendations are presented as configuration-space-dependent choices rather than a universal ranking.
Loading 2007.15745v3…