Source-linked AI summary
Hyperparameter Importance Across Datasets
J. N. van Rijn, F. Hutter
TL;DR
Automated optimization does not by itself reveal which hyperparameters matter or which values are broadly useful. This paper uses functional ANOVA and OpenML data across datasets to identify important hyperparameters and infer priors, finding significant optimization improvements and quantitative cross-dataset evidence.
Problem
Automated hyperparameter optimization provides performance configurations but does not automatically reveal which hyperparameters matter or which values are broadly useful.
Method
The paper applies functional ANOVA and meta-learning to OpenML performance data from 100 datasets, analyzing SVMs, random forests, and Adaboost and inferring priors.
Results
The identified hyperparameters are confirmed as important, and data-driven priors yield statistically significant improvements over a uniform prior in hyperparameter optimization.
Takeaways & Limitations
The results provide quantitative evidence for focusing manual algorithm design and hyperparameter optimization on important parameters and data-driven values.
Takeaways & Limitations
Hyperparameters flagged as important may still have suitable defaults, and tuning only the most important set is not always advisable when sufficient budget allows tuning all parameters.
Abstract
from arXiv · showhide
With the advent of automated machine learning, automated hyperparameter optimization methods are by now routinely used in data mining. However, this progress is not yet matched by equal progress on automatic analyses that yield information beyond performance-optimizing hyperparameter settings. In this work, we aim to answer the following two questions: Given an algorithm, what are generally its most important hyperparameters, and what are typically good values for these? We present methodology and a framework to answer these questions based on meta-learning across many datasets. We apply this methodology using the experimental meta-data available on OpenML to determine the most important hyperparameters of support vector machines, random forests and Adaboost, and to infer priors for all their hyperparameters. The results, obtained fully automatically, provide a quantitative basis to focus efforts in both manual algorithm design and in automated hyperparameter optimization. The conducted experiments confirm that the hyperparameters selected by the proposed method are indeed the most important ones and that the obtained priors also lead to statistically significant improvements in hyperparameter optimization.
1 INTRODUCTION
The paper develops automated, cross-dataset analyses to identify important hyperparameters and infer values likely to perform well. It applies the approach to SVMs, random forests, and Adaboost using OpenML data.
- Motivation: Automated optimization finds strong configurations but provides limited intuition about which hyperparameters matter or which ranges and values are sensible.The paper motivates automated analyses that yield information beyond a single performance-optimizing configuration.
- Research questions: The paper asks which hyperparameters matter most for empirical performance and which values are likely to yield good performance.
- Approach: Functional ANOVA is applied across OpenML performance data for SVMs, random forests, and Adaboost.The methodology analyzes hyperparameter importance across datasets and infers prior distributions over effective values.
- Results: The study confirms that identified priors produce statistically significant improvements for a modern hyperparameter optimization method.
- Contribution: The work provides quantitative evidence about important hyperparameters and useful values, rather than relying mainly on intuition.Experimental data and analyses are made available through OpenML and a separate Jupyter Notebook for reproducibility.
2 RELATED WORK
Prior work addresses hyperparameter importance and meta-learning, but the paper focuses on applying these ideas across datasets and deriving priors from performance data.
- Hyperparameter importance: Existing importance methods include attribute importance, forward selection, ablation analysis, and functional ANOVA.Functional ANOVA can capture both individual hyperparameter effects and interactions among arbitrary subsets.
- Hyperparameter importance: Before this work, these importance methods had not been applied to analyze hyperparameters across datasets.
- Priors and meta-learning: Meta-learning uses dataset characteristics and performance data to recommend configurations, warm-start optimization, or prune search spaces.Selecting adequate meta-features and obtaining them can require substantial effort and runtime.
- Priors and meta-learning: Multi-task Bayesian optimization models correlations between datasets but can suffer from the cubic complexity of Gaussian processes.
- Priors and meta-learning: The proposed prior-learning method draws on estimation-of-distribution algorithms by fitting distributions to hyperparameter values that perform well across datasets.
3 BACKGROUND: FUNCTIONAL ANOVA
Functional ANOVA decomposes predicted performance variation across a hyperparameter space into individual and interaction effects. The paper fits a tree-based model to performance data so these contributions can be estimated efficiently.
- Functional ANOVA: Functional ANOVA uses a regression model to determine how much each hyperparameter subset contributes to predicted-performance variance.
- Notation: The framework represents an algorithm with hyperparameter domains, a configuration space, and complete or partial instantiations.
- Marginal predictions: Marginal performance averages complete configurations agreeing on selected hyperparameters while averaging over the remaining ones.For tree-based models, these averages can be computed exactly in time linear in the number of leaves.
- Functional ANOVA: Functional ANOVA decomposes the performance function into additive components for subsets of hyperparameters.Unary components capture individual effects, while larger subsets capture interactions after excluding lower-order effects.
- Computation: The method fits a random forest to configuration-performance pairs and decomposes each tree’s variance into hyperparameter-subset contributions.The resulting relative contributions across all subsets sum to one, and the remaining analysis requires only seconds in the experiments.
- Interpretation: Interaction effects indicate which hyperparameters may be tuned independently and which should be tuned together.Because standard functional ANOVA operates on one optimization procedure and one dataset, the paper extends the analysis across datasets.
4 METHODS
The paper introduces a cross-dataset framework to identify generally important hyperparameters and infer values likely to perform well from empirical performance data. It combines functional ANOVA, distributional analysis across datasets, and post-hoc verification using hyperparameter optimization.
- The framework uses algorithm configuration spaces, many datasets, and empirical performance measurements to determine influential hyperparameters and promising values.
- Determining Important Hyperparameters: Functional ANOVA fits random forests to per-dataset performance data and estimates each hyperparameter’s variance contribution, whose distribution across datasets indicates general importance.
- Determining Important Hyperparameters: Hyperparameters can appear important because they cause high performance variance even when the same values usually achieve the best performance, so importance does not always imply tuning necessity.
- Verification: The verification procedure assesses hyperparameter importance by comparing optimization performance when a hyperparameter is fixed across candidate values.
- Determining Useful Priors: The method infers priors over useful hyperparameter values from performance data across datasets, with the goal of defining defaults or sampling distributions for optimization.
- Verification: Using these priors within Hyperband is evaluated as an expensive post-hoc test, while the core analysis is computationally efficient apart from fitting functional ANOVA’s random forests.
5 ALGORITHMS AND HYPERPARAMETERS
The experiments analyze random forests, Adaboost with decision-tree base classifiers, and SVMs with RBF and sigmoid kernels. They use Autosklearn’s hyperparameter ranges, shared preprocessing, and the OpenML100 benchmark suite.
- The study analyzes random forests, Adaboost with decision trees as base classifiers, and SVMs using radial basis function and sigmoid kernels.
- Autosklearn’s hyperparameters and ranges are used for each algorithm to avoid bias from manually selecting the configuration spaces.
- Preprocessing includes imputation, one-hot encoding of categorical hyperparameters, removal of constant features, and unit-variance scaling for SVM inputs.
- The experiments use the OpenML100, a curated suite of 100 datasets spanning various domains and containing 500 to 100,000 data points.
6 HYPERPARAMETER IMPORTANCE
Across 100 datasets, a small set of hyperparameters explained most performance variation for SVMs, random forests, and Adaboost. Functional ANOVA rankings were checked with random-search verification and Nemenyi tests.
- Experimental setup: 250,195 algorithm runs across 100 datasets supported the cross-dataset hyperparameter-importance analysis.The experiments required 3,184 CPU days, and the performance data were publicly available on OpenML.
- SVM results: For SVMs, gamma was most important, followed by complexity, for both RBF and sigmoid kernels.Both hyperparameters were significantly more important than the others according to the Nemenyi test, and random search produced its worst performance when gamma was not optimized.
- Random forest results: For random forests, minimum samples per leaf and maximum features for split determination explained most performance variance.The two hyperparameters were identified as the dominant contributors across datasets; their effects also relate to variation among trees and prediction-error correlation.
- Adaboost results: For Adaboost, maximum tree depth and, to a lesser degree, learning rate were significantly more important than the other hyperparameters.Only a few datasets instead ranked the boosting algorithm as most important, and the result was confirmed by the verification experiment.
- Adaboost results: The number of Adaboost iterations had little influence: 50 appeared sufficient, with larger values producing no significant improvement.This contrasts with the influence of maximum depth and learning rate in the same classifier.
7 GOOD HYPERPARAMETER VALUES
The paper infers data-driven priors for good hyperparameter values and evaluates whether they improve Hyperband compared with uniform sampling. The priors favor small leaf sizes in random forests, large tree depths in Adaboost, and low gamma values in SVMs.
- Kernel density estimators infer priors over values of each classifier’s most important hyperparameters.The priors are built from hyperparameter values that performed well on other datasets.
- Random forests typically perform well with quite small minimum numbers of data points per leaf.The inferred prior suggests this hyperparameter has a good default at small values.
- Adaboost typically performs well with a large maximum depth for its decision tree.Although tree depth is used for regularization, the empirical data indicates moderation in applying that regularization.
- Both SVM variants typically achieve their best performance with low gamma values.
- On many datasets, especially random forests, sampling with data-driven priors outperformed uniform sampling.In the per-dataset comparison, positive differences indicate that prior-based sampling performed better.
- Data-driven priors significantly improved Hyperband performance over uniform priors for all classifiers.The comparison used a leave-one-out setup, with priors built from the other 99 datasets; Nemenyi testing assessed significance across datasets.
8 CONCLUSIONS AND FUTURE WORK
The paper identifies cross-dataset hyperparameter importance and derives priors from successful configurations. These priors significantly improve hyperparameter optimization, while future work extends the methodology to broader algorithms, tasks, and dataset-specific recommendations.
- Across 100 datasets, SVM gamma and complexity, Adaboost maximum depth and learning rate, and random-forest leaf size and split features were typically most important.A large-scale verification experiment, optimizing all but one hyperparameter, produced results in line with the functional ANOVA findings.
- Sampling from data-driven priors performed significantly better than sampling from uniform priors in hyperparameter optimization.The priors were obtained by fitting kernel density estimators to hyperparameter values that performed well on other datasets.
- For SVMs with an RBF kernel, priors were better on 60% of datasets despite no visually significant difference in Figure 7.
- All algorithm performance data and a reproducibility notebook were uploaded to OpenML.
- Future work targets more classifiers, regression and clustering algorithms, and dataset-specific priors based on similar datasets.