PaperlayerView exact revision on arXiv

Source-linked AI summary

Importance of Tuning Hyperparameters of Machine Learning Algorithms

Hilde J. P. Weerts, Andreas C. Mueller, Joaquin Vanschoren

arXiv:2007.07588v1cs.LGstat.ML

TL;DR

The paper asks whether individual hyperparameters must be tuned or can safely remain at default values. It introduces a default-setting procedure and a non-inferiority-based tuning-risk methodology, then evaluates them across 59 OpenML datasets. Particular defaults often match tuning performance, and some outperform limited-iteration search, while other hyperparameters remain important to tune.

  • Problem

    The study addresses whether tuning each hyperparameter is necessary when algorithm performance depends on hyperparameter settings.

  • Method

    The paper combines a procedure for determining reasonable defaults with a non-inferiority methodology that measures tuning risk across empirical performance data.

  • Results

    Across 59 datasets, particular computed defaults often produce non-inferior performance to tuning, and sometimes outperform limited-iteration random search.

  • Takeaways & Limitations

    Practitioners may leave selected hyperparameters at their defaults instead of running an expensive random search, but should tune hyperparameters associated with high tuning risk.

  • Takeaways & Limitations

    Defaults are determined once without separate validation, and the conclusions may not generalize to datasets unlike those analyzed.

Abstract

from arXiv · show

The performance of many machine learning algorithms depends on their hyperparameter settings. The goal of this study is to determine whether it is important to tune a hyperparameter or whether it can be safely set to a default value. We present a methodology to determine the importance of tuning a hyperparameter based on a non-inferiority test and tuning risk: the performance loss that is incurred when a hyperparameter is not tuned, but set to a default value. Because our methods require the notion of a default parameter, we present a simple procedure that can be used to determine reasonable default parameters. We apply our methods in a benchmark study using 59 datasets from OpenML. Our results show that leaving particular hyperparameters at their default value is non-inferior to tuning these hyperparameters. In some cases, leaving the hyperparameter at its default value even outperforms tuning it using a search procedure with a limited number of iterations.

1 INTRODUCTION

The study asks whether particular hyperparameters require tuning and proposes an empirical methodology using default settings, applied across 59 datasets. Its motivation is practical: hyperparameter tuning requires a search space and heuristic and is computationally costly.

  • Hyperparameter tuning is costly because practitioners must define both a search space and a search heuristic before searching for good settings.
  • The study evaluates whether particular hyperparameters should be tuned or can remain at reasonable default values.The methodology is based on empirical performance data and a notion of tuning importance.
  • The benchmark applies the approach to random forests and support vector machines across 59 OpenML-CC18 datasets.
  • Leaving selected hyperparameters at computed defaults yields non-inferior performance to tuning them.For some hyperparameters, fixed defaults even outperform limited-iteration random search.

2 RELATED WORK

Prior work studies hyperparameter importance through meta-features, explained performance variance, or gains from tuning individual parameters. This paper instead focuses on tuning risk when one parameter is fixed while the others are tuned, using independently determined defaults.

  • Meta-feature studies predict when tuning will outperform default settings, while other work finds tuning can matter more than algorithm choice.
  • Functional ANOVA identifies hyperparameters explaining performance variance, but its results do not directly provide guidelines for which parameters to tune.
  • Tunability measures the performance gain from tuning one hyperparameter while leaving all others at their defaults.
  • This paper studies tuning risk: the performance loss from fixing one hyperparameter at a default while tuning the others.Unlike joint default selection, its defaults are determined univariately and therefore do not account for interactions between hyperparameters.

3 METHODS

The study defines hyperparameter tuning risk and tests whether fixing individual hyperparameters to data-derived defaults is non-inferior to tuning them. Defaults are derived from good configurations across datasets, while fixed and non-fixed conditions are compared using nested cross-validation, tuning-risk measures, and non-inferiority testing.

  • Default parameter determination: Defaults are determined with leave-one-out data so performance information from the evaluated dataset does not leak into its default setting.Each dataset’s default uses performance data from the other M −1 datasets.
  • Default parameter determination: Default settings are chosen by selecting the best n configurations per dataset and taking the most frequent setting across the resulting measurements.Continuous or large-integer domains are discretized before selecting the most frequent setting.
  • Scope and limitations: The default procedure is simple and intuitive, but the authors identify comparison with more sophisticated surrogate-model methods as future work.The paper also applies meta-feature-dependent defaults only to random-forest max_features and SVM gamma.
  • Experiment design: The study compares fixing one hyperparameter to a default while tuning the remaining H −1 hyperparameters against tuning all H hyperparameters.The comparison is conducted for algorithm A, hyperparameter i, and dataset j.
  • Evaluation measures: Tuning risk measures the performance difference caused by fixing a hyperparameter, and the study summarizes these differences across datasets and random seeds.The analysis also records standard deviations and uses relative tuning risk because equal absolute risk increments may not be equally relevant across tasks.
  • Evaluation measures: Non-inferiority is assessed with a one-sided non-parametric TOST for paired samples, with Holm-Bonferroni correction across hyperparameter tests.The non-parametric procedure is chosen because the data need not be normally distributed and is less sensitive to outliers than a paired t-test approach.

4 EXPERIMENT DETAILS

The benchmark evaluates SVMs with RBF kernels and random forests on 59 OpenML-CC18 classification datasets. Performance and defaults are estimated with cross-validation, random configurations, and nested random-search experiments using accuracy and macro-averaged AUC.

  • Datasets: The benchmark uses 59 classification datasets selected from the 73-dataset OpenML-CC18 suite.One dataset was excluded for technical issues, and the study was further limited by time constraints.
  • Algorithms and hyperparameters: The evaluated algorithms are scikit-learn’s RBF-kernel SVM and Random Forest, with parameter ranges taken from auto-sklearn.The corresponding hyperparameter ranges are listed in Tables 1 and 2.
  • Preprocessing: Preprocessing imputes missing values, one-hot encodes categorical features, removes zero-variance features, and scales features for SVMs.Scaling is added because SVMs are sensitive to feature scales.
  • Performance measures: The study compares accuracy and macro-averaged AUC as performance measures during default-parameter determination.Macro-averaged AUC averages class-level ROC measures across categories with class-instance weighting.
  • Default parameter determination: Default values are estimated from 1,000 random configurations per dataset and algorithm, producing 59,000 evaluations per algorithm with 10-fold cross-validation.Each hyperparameter is sampled uniformly within its specified range and scale.
  • Experiment setup: The fixed-versus-non-fixed comparison uses nested cross-validation with 10 outer folds, 5 inner folds, and 100 random-search iterations per inner fold.The outer loop evaluates test performance while the inner loop tunes hyperparameters.

5 RESULTS

The experiments derive default hyperparameter values, examine feature-dependent defaults, and compare fixing parameters with tuning them across datasets. Several defaults are non-inferior to 100-iteration random search, although C, gamma, and max_features remain exceptions.

  • 5.1.2 Default values.: Computed defaults varied little across datasets for most hyperparameters, while C values under the accuracy-based approach ranged from 3 to 11.The reported standard deviation for those C defaults was 3.224, described as negligible on a logarithmic scale.
  • 5.1.3 Meta-feature dependent hyperparameters.: For max_features, m = 0.16p performed best on a linear scale, whereas m = p^0.74 performed best under RMSLE and LR2.The scikit-learn default √p underestimated max_features in these datasets, while gamma showed no clear feature-dependent pattern.
  • 5.2.1 Tuning Risk.: Fixed defaults often ranked better initially and sometimes throughout 100 iterations, while non-fixed tuning overtook them for selected parameters after 15–30 iterations.The exceptions included random-forest max_features and SVM C and gamma; 100 iterations was not always enough to find the best settings.
  • 5.2.1 Tuning Risk.: Tuning risk and relative tuning risk were close to or below zero for bootstrap, criterion, min_samples_leaf, min_samples_split, shrinking, and tol.These results align with the average-rank comparisons between fixed and non-fixed conditions.
  • 5.2.1 Tuning Risk.: The top-10 performance data represented absolute performance reasonably well but not necessarily relative performance, partly because all learning tasks were weighted equally.This limitation was particularly visible in relative tuning-risk outliers for max_features.
  • 5.2.2 Non-inferiority test.: For all hyperparameters except computed max_features, C, and gamma, fixed defaults were non-inferior to tuning with 100 random-search iterations.The fixed condition’s relative risk was no more than 1% higher than the non-fixed condition for the remaining hyperparameters.

6 CONCLUSIONS

The study introduces a heuristic for default hyperparameters and an empirical methodology for deciding which hyperparameters merit tuning, evaluated across 59 datasets. Computed defaults often perform non-inferiorly, but some hyperparameters show high tuning risk; several scope and validation limitations remain.

  • 6 CONCLUSIONS: The paper presents a simple heuristic for finding default hyperparameters and a methodology for measuring the loss incurred when one hyperparameter is fixed.The approach was applied in a benchmark study using 59 datasets.
  • 6 CONCLUSIONS: Computed default values often achieve non-inferior performance, while random forest max_features and SVM gamma and C show high tuning risk and relative tuning risk.The latter results suggest these hyperparameters are important to tune.
  • 6 CONCLUSIONS: Default parameters are determined once and not validated separately, leaving their effect on the second experiment unclear.The authors specifically question whether the chosen n adequately represents high-performing hyperparameters.
  • 6 CONCLUSIONS: The study excludes datasets with many instances or features, so its conclusions may not hold for datasets that differ substantially from those analyzed.The OpenML-CC18 contains 73 classification tasks, but the study uses a subset.
  • 6 CONCLUSIONS: The experiments consider fixing only a single hyperparameter, leaving interactions between hyperparameters for future work.

A.2 Excluded tasks

Several OpenML-CC18 tasks were excluded because of time constraints or technical issues, particularly tasks with relatively many features or instances.

  • A.2 Excluded tasks: Excluded OpenML-CC18 tasks were generally those with relatively many features or instances.The exclusions were attributed to time constraints or technical issues.

A.3 Datasets previous work

The datasets used in this study differ from those in previous work, especially in their distributions of dataset size and class imbalance.

  • A.3 Datasets previous work: Previous studies contain more datasets with large numbers of features and instances than this study.This comparison is presented through dataset meta-features.
  • A.3 Datasets previous work: The datasets from previous studies contain relatively more imbalanced datasets than those used here.Those datasets came from OpenML-100, whose criteria did not yet include class imbalance.

B DISTRIBUTION OF TOP 10 PERFORMANCE DATA

The paper examines how hyperparameter values are distributed among the top 10 performing configurations for random forests and for accuracy versus AUC.

  • B DISTRIBUTION OF TOP 10 PERFORMANCE DATA: The displayed random forest hyperparameters include bootstrap, criterion, max_features, min_samples_leaf, and min_samples_split.

C DISTRIBUTION OF DEFAULT VALUES ACROSS TASKS

The section depicts distributions of computed default values across tasks for hyperparameters whose defaults vary, using accuracy- and AUC-based performance data.

  • C DISTRIBUTION OF DEFAULT VALUES ACROSS TASKS: Computed default values with non-zero standard deviation are distributed across tasks for multiple hyperparameters.The distributions are shown separately for accuracy-based and AUC-based performance data.
  • C DISTRIBUTION OF DEFAULT VALUES ACROSS TASKS: Figure 9 presents the distributions derived from accuracy-based performance data.
  • C DISTRIBUTION OF DEFAULT VALUES ACROSS TASKS: Figure 10 presents the corresponding distributions derived from AUC-based performance data.

D AVERAGE ACCURACY

The section compares fixed and non-fixed conditions across 59 datasets using maximum average validation accuracy accumulated over iterations.

  • D AVERAGE ACCURACY: The comparison includes Random Forest hyperparameters such as bootstrap, criterion, max_features, min_samples_leaf, and min_samples_split.
  • D AVERAGE ACCURACY: The comparison also includes SVM gamma under computed-default and scikit-learn-default conditions.
  • D AVERAGE ACCURACY: Average accuracy for fixed and non-fixed conditions is compared across 59 datasets over the number of iterations.Each measurement uses the maximum average validation-set accuracy up to the specified iteration.

E AVERAGE RANK

The section ranks fixed and non-fixed conditions across 59 datasets using maximum average validation-set accuracy accumulated over iterations.

  • E AVERAGE RANK: The ranked comparison includes Random Forest hyperparameters such as bootstrap, criterion, max_features, min_samples_leaf, and min_samples_split.
  • E AVERAGE RANK: The ranked comparison includes SVM gamma under computed-default and scikit-learn-default conditions.
  • E AVERAGE RANK: Average ranks of fixed and non-fixed conditions are compared across 59 datasets over the number of iterations.Ranks are based on maximum average validation-set accuracy, with smaller ranks indicating better relative performance.
Loading 2007.07588v1…