Source-linked AI summary
Tunability: Importance of Hyperparameters of Machine Learning Algorithms
Philipp Probst, Bernd Bischl, Anne-Laure Boulesteix
TL;DR
Hyperparameter choices affect machine-learning performance, but users lack systematic guidance on which parameters to tune and how to set defaults and search spaces. The paper formalizes data-based defaults and tunability measures, estimates them with surrogate models, and benchmarks six algorithms on 38 OpenML datasets. The resulting defaults and tunability analyses support decisions about whether tuning is worthwhile, which parameters to prioritize, and how to define tuning spaces.
Problem
Users lack systematic, data-dependent methods for deciding which hyperparameters to tune, choosing search ranges, and defining robust defaults.
Method
The paper defines defaults, tunability measures, joint gains, and tuning spaces, then estimates performance mappings with surrogate models using benchmark configurations.
Results
The study computes optimal defaults and quantifies algorithm- and parameter-level tunability for six machine-learning algorithms across 38 OpenML datasets.
Takeaways & Limitations
The framework yields interpretable tunability values and helps users decide whether to tune, prioritize hyperparameters, and select adequate tuning spaces.
Takeaways & Limitations
The study considers only binary classification, and uniform random hyperparameter sampling may not scale sufficiently in very high-dimensional spaces.
Abstract
from arXiv · showhide
Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them. Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure. The goal of this paper is two-fold. Firstly, we formalize the problem of tuning from a statistical point of view, define data-based defaults and suggest general measures quantifying the tunability of hyperparameters of algorithms. Secondly, we conduct a large-scale benchmarking study based on 38 datasets from the OpenML platform and six common machine learning algorithms. We apply our measures to assess the tunability of their parameters. Our results yield default values for hyperparameters and enable users to decide whether it is worth conducting a possibly time consuming tuning strategy, to focus on the most important hyperparameters and to chose adequate hyperparameter spaces for tuning.
1 Introduction
Hyperparameters must be set before machine-learning algorithms run, yet users lack a systematic, data-dependent basis for choosing what to tune, how to configure it, and which defaults are robust. The paper formalizes tuning statistically and evaluates its measures using six algorithms across 38 OpenML datasets.
- Hyperparameters are second-level settings chosen before training, unlike model parameters learned during training.
- Users can rely on software defaults, manual configuration, or data-dependent tuning strategies that optimize expected generalization error.
- Choosing tunable parameters, search ranges, and prior distributions poorly can reduce model quality or tuning efficiency.
- The scientific community lacks systematic answers, methods, and criteria for making these hyperparameter decisions objectively across algorithms.
- The paper formalizes tuning statistically, defines defaults and tunability measures, and applies them to six algorithms evaluated on 38 OpenML datasets.
2 Related literature
Prior work largely measures hyperparameter importance on individual datasets, often after tuning has finished. This paper instead emphasizes generalization across datasets to support broader understanding and future experimental decisions.
- Existing studies assess hyperparameter relevance or tuning gains for neural networks, decision trees, optimization configurations, and tuned algorithms.
- Much prior work explains hyperparameter importance retrospectively on a single dataset after a tuning process has concluded.
- The paper focuses on generalization across multiple datasets rather than only explaining one completed tuning run.
- Its surrogate-model framework estimates performance for arbitrary hyperparameter configurations from a limited number of prior experiments.
3 Methods for Estimation of Defaults, Tunability and Ranges
The methods represent algorithm performance as a risk mapping over hyperparameter configurations, then derive defaults, tunability measures, joint gains, and tuning spaces from these mappings. Surrogate models estimate the mappings empirically for benchmark datasets.
- 3.1 General notation: The risk mapping assigns expected loss to each hyperparameter configuration for a given data distribution, algorithm, and performance measure.
- 3.2 Optimal configuration per dataset and optimal defaults: An optimal default configuration minimizes a summary function of risks across multiple datasets, with the mean or median providing possible summaries.
- 3.3 Measuring overall tunability of a ML algorithm: Overall tunability compares a reference configuration with the best configuration for each dataset and summarizes the resulting performance differences across datasets.
- 3.4 Tunability of specific hyperparameters: Single-parameter tunability measures the risk reduction from optimizing one parameter while holding the others at default values, including its fraction of the complete algorithm's gain.
- 3.5 Tunability of hyperparameter combinations and joint gains: Joint analyses quantify gains from tuning parameter combinations and compare joint optimization with sequential tuning orders.
- 3.6 Optimal hyperparameter ranges for tuning: A tuning space is constructed from parameter quantiles of dataset-specific optima, while the stated definition applies only to numerical hyperparameters.
- 3.7 Estimation: Surrogate regression models learn configuration-to-performance mappings from many evaluated configurations, making the optimization problem cheaper to solve.
4 Experimental setup
The study benchmarks six supervised-learning algorithms across 38 OpenML binary-classification tasks using sampled hyperparameter configurations, surrogate models, and tunability measures. Its experimental design accounts for parameter dependencies, evaluates predictive and surrogate-model performance with cross-validation, and guards against overfitting when estimating data-based defaults.
- Datasets and algorithms: The benchmark covers 38 OpenML binary-classification tasks without missing values and six algorithms: elastic net, decision tree, k-nearest neighbors, support vector machine, random forest, and gradient boosting.The algorithms are implemented through glmnet, rpart, kknn, svm, ranger, and xgboost.
- Configuration sampling: Dependent hyperparameters are sampled conditionally, so subordinate parameters such as gamma are considered only when their superordinate setting makes them relevant.For support vector machines, gamma is sampled only after a radial kernel is selected.
- Evaluation and validation: Predictive performance is estimated with 10-fold cross-validation, surrogate models are compared with 10-times repeated 10-fold cross-validation, and defaults are additionally assessed across datasets.The across-dataset validation repeatedly derives defaults from 90% of the datasets and evaluates them on the remaining 10%.
- Configuration sampling: Hyperparameter configurations are sampled from Table 1 supports using independent uniform distributions on the untransformed parameter scale.The table specifies lower and upper bounds and, where applicable, transformations for candidate values.
- Large-scale experiments: An OpenML bot evaluates random configurations through cross-validation, with a subset of 500000 experiments per algorithm across all datasets used for analysis.The computation is performed in an embarrassingly parallel manner over randomly selected datasets, algorithms, and configurations.
- Tunability estimation: Defaults, dataset-specific optima, and single-parameter tunability are estimated by random search over 100000 sampled points in the hyperparameter space.The same random-search strategy is used for algorithm-level defaults and dataset-level best settings, with parameter-specific searches for individual tunability.
5 Results and discussion
Across six algorithms, the study evaluates surrogate-model performance and tunability, finding substantial variation across algorithms, hyperparameters, and datasets. It also examines optimal defaults, tuning spaces, and joint hyperparameter effects.
- Surrogate models: Ranger achieved good overall surrogate-model performance for predicting AUC across datasets, with qualitatively similar results for other classification measures.The surrogate models were compared using repeated cross-validated R2 and Kendall’s tau.
- Optimal defaults: Optimal defaults improved performance overall compared with package defaults, including for svm and ranger despite their data-dependent package defaults.The comparison is reported using overall and cross-validated tunability and improvement measures.
- Algorithm-level tunability: Tunability varied substantially by algorithm: glmnet and svm were much more tunable than the others, while ranger had the smallest tunability.Some datasets showed unusually large tuning effects, reflected by outliers in every algorithm’s tunability distribution.
- Specific hyperparameters: Hyperparameter importance differed by algorithm: lambda dominated alpha in glmnet, mtry was most tunable in ranger, and eta and booster were most tunable in xgboost.For rpart, minbucket and minsplit were most important; for svm, kernel, gamma, and degree offered the largest gains, whereas cost was less tunable.
- Hyperparameter spaces: The proposed tuning spaces use the 0.05 and 0.95 quantiles, contain all optimal defaults, and exclude some package defaults.For random forest, higher mtry values than the package defaults appeared advantageous on quite a few datasets.
- Hyperparameter combinations: Tuning two hyperparameters enabled larger improvements than tuning either alone, with rpart’s minsplit and minbucket showing the largest joint effect.These parameters are closely related because they constrain when splits are attempted and the minimum terminal-leaf size.
6 Conclusion and Discussion
The paper introduces a surrogate-model framework for defining optimal defaults and measuring the tunability of algorithms and individual hyperparameters. Its benchmark covers six algorithms on OpenML classification tasks, while limitations include dependence on defaults and restricted experimental scope.
- Framework: The framework defines optimal defaults and tunability measures for joint, individual, and combined hyperparameter tuning using surrogate empirical performance models.The measures are intended to quantify performance gains relative to reference configurations and support comparisons across algorithms.
- Benchmark: An OpenML benchmark computes optimal defaults and tunability for elastic net, decision tree, k-nearest neighbors, SVM, random forest, and xgboost.The study uses 38 binary classification datasets and presents the analysis as a principled benchmark.
- Limitations: The approach depends on determining default values as an additional analysis step and as the reference point for most measures.Defaults are therefore both a useful output and a methodological dependency.
- Comparison: Compared with functional ANOVA, the framework’s scores are described as more directly interpretable but rely on defaults, whereas functional ANOVA does not.The comparison also differs in datasets, sampling volume, hyperparameter ranges, and surrogate-model validation.
- Limitations: The study is limited to binary classification, while uniform random sampling may not scale to very high-dimensional spaces and static defaults cannot use dataset characteristics.The authors also note that initial hyperparameter ranges are required for sampling.
Appendix A. Results for accuracy and brier score
The appendix reports accuracy- and brier-score-based tunability results through figures and tables. These results cover overall and individual-hyperparameter measures, defaults, improvements, cross-validation, and tuning-space quantiles.
- Figures: Figures 5 and 6 show average surrogate-model performances across datasets for accuracy and brier score, respectively.The figures compare surrogate models for the algorithms presented earlier.
- Figures: Figures 7 and 8 show boxplots of algorithm tunabilities for accuracy and brier score, respectively.The boxplots summarize tunability distributions across algorithms.
- Tables: Tables 6 and 8 report overall tunability, defaults, cross-validated tunability, and average improvement for accuracy and brier score.The tables compare package defaults with calculated optimal defaults and include cross-validated improvements.
- Tables: Tables 7 and 9 report single-hyperparameter tunability, package and calculated defaults, and tuning-space quantiles for accuracy and brier score.These tables provide parameter-level results alongside tuning-space information.
- Measures: The appendix identifies AUC, accuracy, and brier score as classification measures and includes a table of tunability with calculated defaults, with and without cross-validation.The measure labels and cross-validation variants organize the reported evaluations.