Source-linked AI summary
Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges
Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix, Difan Deng, Marius Lindauer
TL;DR
Manual hyperparameter selection is costly, biased, error-prone, and irreproducible, motivating automated HPO. The paper reviews major HPO methods and develops practical guidance, resources, and examples for applying them. It emphasizes choosing methods, evaluation procedures, search spaces, pipelines, and computational strategies appropriately, while noting risks such as meta-overfitting and exploration failures.
Problem
Manual hyperparameter trial-and-error is time-consuming, biased, error-prone, and computationally irreproducible, while users lack guidance on HPO methods and search spaces.
Method
The paper formally introduces HPO, reviews grid and random search, evolutionary algorithms, Bayesian optimization, multifidelity methods, and related practical choices.
Results
The paper provides a general overview of HPO concepts and algorithms, practical recommendations, appendices with software and search spaces, and notebooks demonstrating HPO concepts.
Takeaways & Limitations
HPO can optimize complete predictive pipelines, including preprocessing, model selection, and post-processing, using structured search spaces.
Takeaways & Limitations
Nested resampling provides unbiased outer evaluation, but extensive HPO can select configurations affected by inner-resampling stochasticity or overfitting, known as meta-overfitting.
Abstract
from arXiv · showhide
Most machine learning algorithms are configured by one or several hyperparameters that must be carefully chosen and often considerably impact performance. To avoid a time consuming and unreproducible manual trial-and-error process to find well-performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods, e.g., based on resampling error estimation for supervised machine learning, can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods such as grid or random search, evolutionary algorithms, Bayesian optimization, Hyperband and racing. It gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with ML pipelines, runtime improvements, and parallelization. This work is accompanied by an appendix that contains information on specific software packages in R and Python, as well as information and recommended hyperparameter search spaces for specific learning algorithms. We also provide notebooks that demonstrate concepts from this work as supplementary files.
1 Introduction
Hyperparameters strongly affect machine-learning behavior and performance, while manual trial-and-error is costly, biased, error-prone, and irreproducible. The paper introduces HPO and provides practical guidance, software resources, search spaces, and notebooks to make automated tuning more accessible.
- Hyperparameters influence a machine-learning algorithm’s complexity, behavior, speed, and other aspects.
- Manual hyperparameter selection is time-consuming, biased, error-prone, and computationally irreproducible.
- HPO delegates black-box configuration search to algorithms and machines to improve efficiency and reproducibility.
- Users face obstacles including limited understanding, skepticism about benefits, missing method-selection guidance, and difficulty defining search spaces.
- The paper formally introduces HPO for supervised ML and offers practical advice, while noting that the techniques can apply to other ML settings with quantitative evaluation.
- Appendices cover algorithms, preprocessing, metrics, software packages in R and Python, and supplementary notebooks demonstrate practical HPO concepts.
2 Related Work
Prior HPO surveys cover methods, challenges, search spaces, tools, and specialized application areas. This paper distinguishes itself by emphasizing practical advice without focusing on particular ML model classes.
- Earlier surveys provide broad overviews of HPO approaches, open challenges, future directions, search spaces, techniques, and tools.
- This paper focuses on general HPO guidance for practical issues rather than concrete ML model classes.
- Several specialized surveys address HPO or AutoML for deep learning, forecasting, graph models, and specific ML algorithms.
3 Supervised Machine Learning
Supervised ML learns predictive models from labeled data, with hyperparameters configuring the learner and empirical risk minimization producing fitted models. Reliable evaluation estimates future performance through holdout or repeated resampling, each with distinct statistical trade-offs.
- 3.1 Terminology and Notations: Supervised ML infers a model from labeled observations so it can predict new data from the same underlying distribution with minimal error.
- 3.1 Terminology and Notations: Features may be numerical, integer, or categorical, while common supervised tasks include regression and classification.
- 3.1 Terminology and Notations: A learner configured by hyperparameters maps a data set to a fitted model or its parameter vector.
- 3.1 Terminology and Notations: Empirical risk minimization optimizes empirical risk over candidate models in the learner’s hypothesis space.
- 3.2 Performance Evaluation: Performance measures map true labels and prediction scores to scalar values and may differ from the loss used during training.
- 3.2 Generalization Error: Unseen test data is used for performance estimation, while holdout evaluation trades pessimistic bias from reduced training data against higher variance from smaller test sets.
- 3.2.3 Data splitting and Resampling: Resampling repeatedly splits data into training and test sets, evaluates performance for each split, and aggregates the resulting values.
4 Hyperparameter Optimization
HPO treats hyperparameter selection as a difficult black-box optimization problem over potentially mixed, hierarchical, and stochastic search spaces. The paper presents major algorithmic strategies and emphasizes efficient evaluation, multifidelity, and unbiased assessment.
- HPO Problem Definition: HPO searches for well-performing hyperparameter configurations within mixed or hierarchical spaces whose variables may be continuous, discrete, categorical, or conditional.Conditional hyperparameters are active only for specified values of another hyperparameter.
- HPO Problem Definition: The objective is an expensive, stochastic black box because generalization error is estimated through resampling and lacks analytic gradients.These properties restrict the usefulness of gradient-based methods and methods requiring many evaluations.
- Grid Search and Random Search: Grid search exhaustively evaluates discretized combinations, whereas random search samples configurations independently; random search often performs better in higher-dimensional settings with low effective dimensionality.In the illustrated two-hyperparameter example, only 3 of 9 grid evaluations provide meaningful information, while random search produces nine distinct values for the influential hyperparameter.
- Evolution Strategies: Evolution strategies iteratively select, recombine, mutate, and retain high-fitness configurations, and can be extended to mixed spaces, pipelines, architectures, noisy objectives, and multiple objectives.Their flexibility makes them applicable to complex search spaces where other optimizers may fail.
- Bayesian Optimization: Bayesian optimization fits a surrogate model to observed configurations, then proposes candidates by optimizing an acquisition function that balances predicted performance and uncertainty.Typical surrogate models include Gaussian processes and random forests, while the acquisition function supports exploration and exploitation.
- Multifidelity and Hyperband: BOHB combines Hyperband’s early discarding of poor configurations with Bayesian proposal generation, performing similarly to Hyperband at low budgets and outperforming it when enough budget is available for many full-budget evaluations.Multifidelity methods nevertheless depend on choices such as the discard fraction and the correlation between performance at different fidelities.
- Nested Resampling and Meta-Overfitting: Nested resampling is required for unbiased outer evaluation because directly reporting the best resampling result produces an optimistically biased generalization estimate.After many evaluations, stochasticity or overfitting to resampling splits can also lead to selection of the wrong configuration.
5 Pipelining, Preprocessing, and AutoML
The paper extends HPO from individual learners to configurable pipelines that include preprocessing, model selection, and branching operator choices. These flexible pipelines induce hierarchical search spaces and form a central basis of AutoML.
- Linear pipelines: A linear ML pipeline successively applies preprocessing methods before a learner, with each node having training, prediction, and potentially hyperparameter configuration.The pipeline is treated as a sequence of configurable nodes rather than only a learner.
- Evaluation: Pipeline construction and preprocessing must be included within cross-validation so every model component is inferred only from training data.This prevents overfitting and biased performance evaluation.
- Graph pipelines: Directed acyclic graph pipelines support flexible node selection, including alternative preprocessing operations, learners, and postprocessing steps.A source node receives the data and a sink node returns predictions.
- Operator selection: Categorical branching parameters select mutually exclusive preprocessing steps or ML algorithms, creating different modeling paths through the graph.The branching choice determines which path and corresponding operations are active.
- AutoML: Flexible pipelines create hierarchical search spaces because active nodes and their hyperparameters depend on branching choices; combining such graphs with efficient tuning is the key principle of AutoML.The approach can support many preprocessing steps and ML models when configured in a data-dependent manner.
6 Practical Aspects of HPO
Practical HPO requires coordinated choices about resampling, metrics, search spaces, tuners, budgets, and termination. The paper emphasizes matching these choices to data properties, computational constraints, and the structure of the optimization problem.
- Resampling: Resampling should reflect sample size and dependence structure, with repeated cross-validation for small datasets and block-aware splits for correlated observations.For smaller datasets, the paper gives n < 500 as an example favoring repeated cross-validation; repeated measurements may require leaving blocks out.
- Metrics: Performance measures should be selected according to real-world costs, and multiple metrics may be optimized when one measure cannot capture model quality adequately.Accuracy may be inappropriate when errors have unequal consequences.
- Search spaces: Search spaces should use suitable numeric and categorical representations, bounded numeric intervals, and logarithmic scales for many lower-bounded hyperparameters.Encoding categories as integers can degrade optimizers that use distances, while logarithmic tuning reflects diminishing influence at larger values.
- Search spaces: Search-space size affects both model quality and tuning difficulty: spaces that are too narrow exclude strong configurations, while overly broad spaces dilute the available budget.The paper also recommends tuning as few hyperparameters as possible when prior knowledge is absent.
- Choosing tuners: Grid search is useful for about 2–3 hyperparameters with effective discretization, while Gaussian-process Bayesian optimization works up to around 10 hyperparameters.Random forests have been used successfully for Bayesian optimization in spaces with hundreds of hyperparameters.
- Budget and termination: HPO budgets and dynamic termination remain difficult to determine, requiring cost-benefit decisions about additional evaluations and practical runtime.The paper describes combining multiple termination criteria when possible.
- Termination and exploration: Evolutionary strategies and, in some circumstances, Bayesian optimization can become trapped in a subspace and fail to explore other regions.Random interleaving or repeated restarts can mitigate this, but restart efficiency differs across optimizer types.
15 end
HPO workloads expose several parallelization levels, from outer resampling to individual model fits, and the best choice depends on tuner structure and available resources. Fair benchmarking also requires time-aware budgets and accounting for anytime behavior, overhead, and parallelism.
- Parallelization levels: Nested HPO can parallelize outer resampling, tuning iterations, batches of proposed configurations, inner resampling splits, and sometimes model fitting.Each level produces jobs with different runtimes and synchronization patterns.
- Parallelization levels: Inner resampling iterations are independent across a batch of hyperparameter configurations, producing nbatch · kinner parallel jobs.This independence enables fine-grained parallel execution during configuration evaluation.
- Tuner-dependent scaling: Random and grid search are embarrassingly parallel, whereas evolutionary algorithms, Bayesian optimization, Hyperband, and racing are constrained by batch or proposal structure.Sequential Bayesian optimization has batch size 1 unless multipoint proposals are used.
- Resource allocation: With abundant parallel resources, random search may be preferable because of its broad parallelizability and low synchronization overhead.More efficient algorithms gain a larger relative advantage when parallel resources are fewer.
- Benchmarking: Fair tuner comparisons should use comparable wall-clock budgets and report overhead, while evaluating optimization traces across multiple runtime stages rather than only final performance.Parallelization can scale differently across tuners and may introduce diminishing returns or technical interference.
- Model selection: When interpretability and simplicity matter, practitioners can separately tune a manually ordered sequence of preferred model classes and compare their performance.This is presented as an alternative to full-pipeline AutoML when only a few model classes are considered.
- Benchmarking: Existing large-scale HPO benchmarks remain few, and the community continues working toward more comprehensive and representative evaluation studies.Current comparisons often use fixed datasets and search spaces or cheaper surrogate performance models.
7 Related Problems
The paper places HPO alongside related problems that vary in what is optimized, when configuration occurs, and whether selection is learned offline or jointly with training. These connections clarify HPO’s scope and neighboring formulations.
- Neural Architecture Search: Neural architecture search is a specialized HPO problem that seeks a well-performing deep neural-network architecture for a dataset.It is often formulated as bilevel optimization and may be solved while training the network.
- Algorithm Selection: Algorithm selection trains a meta-learning model offline to select an algorithm or hyperparameter configuration from a finite candidate set.The model can use empirical performance metadata collected across problem instances or datasets.
- Algorithm Configuration: Algorithm configuration searches empirically for a strong parameter configuration of an arbitrary algorithm across a finite set of problem instances.Performance is typically accessed as a costly black-box score.
- Dynamic Algorithm Configuration: Dynamic algorithm configuration adapts hyperparameters during training, unlike HPO, which chooses one configuration for the entire model training process.Learning-rate schedules and reinforcement-learning policies are examples of dynamic adaptation.
- Learning to Learn and to Optimize: Learning-to-learn approaches replace learner or optimizer components, including methods that learn neural-network weight updates or where to sample next in black-box optimization.These methods go beyond selecting fixed hyperparameters.
8 Conclusion and Open Challenges
The paper surveys HPO foundations and emphasizes optimizing complete predictive pipelines while identifying unresolved challenges in efficiency, evaluation, interpretability, and scope.
- Scope and practical integration: State-of-the-art HPO systems optimize entire predictive pipelines, including preprocessing, model selection, and post-processing, using structured search spaces.Structured search spaces can make complex optimization tasks efficiently optimizable.
- Scope and practical integration: General HPO tools trade flexibility across many tasks for larger search spaces and potentially lower efficiency than specialized tools.Specialized tools can be more efficient on specific tasks but may not transfer to different tasks.
- Open challenges: Expensive deep-learning and reinforcement-learning training can make iterative HPO infeasible, motivating gradient-based, transfer or few-shot, meta-learning, and dynamic-configuration approaches.Population-based training dynamically mutates and selects among training runs, but requires substantial computational resources.
- Evaluation and reliability: Long HPO runs can bias performance estimators and lead to incorrect hyperparameter configuration selection, especially with small data or limited resampling.Increasing resampling folds during longer tuning runs is proposed, but the ideal schedule remains under-explored.
- Evaluation and reliability: Many HPO approaches return well-performing configurations without explaining optimization decisions, which can reduce user trust and hinder AutoML deployment.The paper links transparency concerns to possible non-deployment despite performance gains.
- Open challenges: Practical applications may require multi-criteria HPO because predictive performance, simplicity, interpretability, and other metrics can involve unknown trade-offs.The paper identifies quantitative interpretability measures as useful for directly incorporating interpretability into HPO.
- Open challenges: Supervised-learning HPO becomes less straightforward for clustering and anomaly detection because performance evaluation, particularly with one defined metric, is unclear.The paper treats HPO beyond supervised learning as an open challenge rather than claiming the techniques are unusable in all cases.
Funding Resources
The supplied passages describe appendix resources and practical guidance for selecting hyperparameter search spaces, but they do not provide funding information.
- Appendix resources: The appendices provide practical HPO resources, including learner properties, suggested hyperparameter spaces, preprocessing methods, evaluation metrics, and software packages.The supplied passage identifies these materials as appendix content rather than funding resources.
- Appendix resources: Recommended hyperparameter ranges are experience-based, intended to work across many datasets, and may require adaptation for non-standard situations.The selection of learners, hyperparameters, and ranges is described as somewhat subjective.
- Algorithm example: The k-NN appendix describes distance-based prediction, with classification using neighbor class proportions and regression using a simple average.A kernel extension can weight neighbors according to their distances.
- Algorithm example: The k-NN hyperparameter k controls locality: smaller values increase flexibility and overfitting susceptibility, whereas larger values produce smoother potentially underfit predictions.This passage concerns algorithm guidance, not funding.
A.2 Regularized Linear Models
Regularized linear models control coefficient complexity through penalty terms, with lasso, ridge, and elastic net providing different shrinkage and sparsity behaviors.
- Model concept: Regularized linear models extend regression by shrinking coefficients with a penalty term to prevent overfitting and improve prediction accuracy.The basic formulation adds a penalty to the ordinary least-squares criterion.
- Model concept: α = 1 yields lasso, α = 0 yields ridge regression, and α ∈(0, 1) yields the elastic net.These models differ by the relative contribution of lasso and ridge penalties.
- Sparsity and limitations: The lasso penalty can shrink coefficients exactly to zero, producing sparse models and performing feature selection.With highly correlated features, lasso tends to select only one; when p > n, it selects at most n non-zero coefficients.
- Sparsity and limitations: Elastic net combines lasso and ridge penalties to address lasso limitations, including correlated-feature behavior and the p > n non-zero-coefficient bound.The combined penalty retains shrinkage while overcoming the stated limitations.
- Generalized linear models: Lasso, ridge, and elastic net can extend to regularized generalized linear models by replacing least squares with the GLM objective.The extension preserves the regularization framework while changing the underlying model objective.
- Hyperparameters: α controls the relative weighting of lasso and ridge penalties, while λreg controls overall regularization strength.These are the principal hyperparameters identified for the regularized linear-model family.
A.3 Support Vector Machines
Support vector machines separate classes by maximizing the margin, use kernels for nonlinear boundaries, and depend mainly on regularization and kernel hyperparameters.
- Model concept: For binary classification, an SVM places a hyperplane between classes while maximizing the margin around the closest training points.The closest points are the support vectors, and few margin violations are allowed.
- Model concept: Kernel functions implicitly map data into a higher-dimensional feature space, enabling nonlinear separation with the same linear procedure.The kernel-based formulation applies the linear procedure after the implicit mapping.
- Model variants: SVMs extend to multiclass classification and regression through support vector regression.The passage names SVR as the regression extension.
- Hyperparameters: SVM performance is mainly influenced by regularization control, kernel type, and kernel hyperparameters.Examples of kernel types include linear, polynomial, sigmoid, and radial basis function kernels.
- Related algorithm context: CART decision trees recursively split feature space so resulting nodes become as homogeneous as possible for the target variable.The supplied tree passage provides context for the neighboring hyperparameter discussion but is not an SVM result.
- Related algorithm context: CART is mainly influenced by its splitting criterion, stopping criteria, and procedure for assigning predicted values to leaves.These choices affect split selection, tree size, and leaf predictions.
A.5 Random Forests
Random forests combine many decorrelated decision trees, exposing hyperparameters for both individual trees and the forest. Although defaults often work well, tuning can substantially improve performance in some cases.
- Random forests aggregate many decision trees to reduce prediction error and smooth prediction variance through bagging.
- RF hyperparameters govern both individual-tree construction and forest structure, including tree count, candidate split variables, and sampling.
- The number of trees reflects a performance–computation-time compromise rather than a universally optimal value.
- Random forests often perform reasonably well with default parameters across many applications and may require less tuning than other algorithms.
- Hyperparameter tuning can nevertheless produce substantial performance improvements for random forests in some cases.
A.7 Neural Networks
Neural-network hyperparameters cover optimization, regularization, and architecture. Standard optimization techniques apply to the first group, whereas architecture search requires more specialized strategies.
- Neural networks consist of nonlinearly transformed weighted input sums organized in layers and are trained with stochastic gradient-descent variants.
- Neural-network hyperparameters divide into optimization and regularization parameters versus architectural parameters defining neuron types, amounts, structures, and connections.
- Optimization techniques discussed in the paper can be applied straightforwardly to neural-network optimization and regularization hyperparameters.
- Neural architecture search provides customized strategies because architectural hyperparameters are much more difficult to optimize.
- In keras and torch, neural networks are constructed programmatically, so configuration often does not use explicit function arguments as hyperparameters.
B.1 Missing Data Imputation
Preprocessing addresses missing values and categorical inputs before model fitting, but unseen categorical levels during prediction remain a practical boundary that cannot be completely eliminated.
- Imputation replaces missing values with feasible ones because many machine-learning algorithms do not natively handle missing values.
- Because missingness may be informative, imputation should generally preserve whether a value was imputed.
- Dummy encoding represents a categorical k-level feature with k or k −1 binary indicators, with k −1 using one reference level.
- Unseen factor levels at prediction can be mitigated by merging infrequent levels, stratified resampling, or imputation, but cannot be completely ruled out.
- The paper points to benchmarks comparing encoding and feature-filtering methods across settings and predictive or computational criteria.
B.4 Data Augmentation and Sampling
Data augmentation changes training data to improve predictive performance, while feature extraction adds task-relevant representations that can be model- and domain-specific. Both can be incorporated into configurable preprocessing pipelines.
- Data augmentation improves predictive performance by adding or removing training rows, commonly addressing imbalanced classification.
- Oversampling repeats minority-class observations, while SMOTE adds observations formed from convex combinations of minority-class examples.
- Feature extraction or engineering adds features that a model can exploit more effectively than the original inputs.
- Feature extraction is often domain-specific and model-specific, such as polynomial terms for linear models or MFCCs for audio signals.
- Embedding custom extraction code with exposed hyperparameters allows automatic, data-dependent configuration within preprocessing.
Appendix C Evalution Metrics
The appendix covers performance-measure notation and surveys R software for machine learning and hyperparameter optimization. It emphasizes framework-supported tuning and pipeline integration over manually combining black-box optimizers with custom evaluation code.
- Evaluation metrics: ACC, BA, CE, BS, and LL support multi-class classification, while multiclass extensions also exist for AUC.The notation includes predicted labels, one-hot class indicators, test-set class counts, and estimated class probabilities.
- Machine-learning software: R provides implementations for common learners including k-NN, regularized linear models, support vector machines, decision trees, random forests, boosting, and neural networks.Examples include class and kknn for k-NN, glmnet for elastic net, e1071 and kernlab for SVMs, rpart for CART, ranger for random forests, xgboost for boosting, and keras or torch for neural networks.
- HPO algorithms: R packages implement evolutionary strategies, Bayesian optimization, Hyperband, and iterated F-racing for black-box hyperparameter optimization.Examples include rgenoud, cmaes, ecr, tune, mlr3hyperband, and irace.
- Framework integration: ML frameworks are recommended for combining black-box optimizers with ML algorithms and pipelines because they manage fitting, evaluation, common pitfalls, and parallel execution.The paper discourages writing a standalone objective function for this purpose.
- Framework capabilities: mlr3, caret, tidymodels, and h2o provide varying combinations of search methods, resampling or racing, preprocessing support, and joint tuning of pipeline components.mlr3pipelines and tidymodels recipes support pipeline-oriented tuning, while h2o provides grid and random grid search with more limited preprocessing.