PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms

Chris Thornton, Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown

arXiv:1208.3719v2cs.LG

TL;DR

Choosing among many algorithms and hyperparameter settings is difficult, and users may default to intuitive choices or preset values. Auto-WEKA frames the joint task as hierarchical hyperparameter optimization and searches it with Bayesian methods; across its benchmark datasets, it often outperformed standard methods, especially on large datasets.

  • Problem

    Choosing algorithms and hyperparameters among many alternatives is challenging, and users may rely on reputation or defaults that perform far worse than the best settings.

    Source: §1 · p. 1

  • Method

    Auto-WEKA treats algorithm choice and hyperparameters as one hierarchical search problem and uses Bayesian optimization across WEKA classification methods.

    Source: §1 · p. 1, §4 · p. 4

  • Results

    Auto-WEKA often outperformed standard algorithm-selection and hyperparameter-optimization methods across 21 datasets, especially on large datasets.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    Auto-WEKA is a practical automated tool intended to help non-experts build high-quality classifiers for their application scenarios.

    Source: §6 · p. 8

    Cross-validation performance improvements were larger than test-data improvements, motivating investigation of more sophisticated overfitting detection and avoidance.

    Source: §6 · p. 9

Abstract

from arXiv · show

Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall. We consider the problem of simultaneously selecting a learning algorithm and setting its hyperparameters, going beyond previous work that addresses these issues in isolation. We show that this problem can be addressed by a fully automated approach, leveraging recent innovations in Bayesian optimization. Specifically, we consider a wide range of feature selection techniques (combining 3 search and 8 evaluator methods) and all classification approaches implemented in WEKA, spanning 2 ensemble methods, 10 meta-methods, 27 base classifiers, and hyperparameter settings for each classifier. On each of 21 popular datasets from the UCI repository, the KDD Cup 09, variants of the MNIST dataset and CIFAR-10, we show classification performance often much better than using standard selection/hyperparameter optimization methods. We hope that our approach will help non-expert users to more effectively identify machine learning algorithms and hyperparameter settings appropriate to their applications, and hence to achieve improved performance.

1. INTRODUCTION

The introduction frames simultaneous algorithm selection and hyperparameter tuning as a difficult but practically important task for non-expert users. It presents Auto-WEKA’s Bayesian-optimization approach and reports that combined searches often outperform standard methods, especially on large datasets.

1. INTRODUCTION

  • Users facing many algorithm and hyperparameter choices may rely on reputation or defaults, which can perform far worse than the best settings.The paper notes that non-experts often need off-the-shelf machine-learning solutions.Source: §1 · p. 1
  • CASH jointly selects a learning algorithm and its hyperparameters to optimize empirical performance, a search complicated by noisy, high-dimensional, mixed, hierarchical choices.Some hyperparameters matter only when their associated algorithm or ensemble choice is selected.Source: §1 · p. 1
  • The paper treats algorithm choice itself as a hyperparameter and applies recent Bayesian optimization to search the resulting space with limited human effort.The approach is presented as a single hierarchical hyperparameter-optimization problem.Source: §1 · p. 1
  • Auto-WEKA’s combined search often outperforms standard algorithm-selection and hyperparameter-optimization methods, especially on large datasets.The authors describe a concrete CASH problem covering WEKA’s classifiers and feature selectors.Source: §1 · p. 1

2. PRELIMINARIES

The preliminaries set up classification and explain how model selection and hyperparameter optimization are evaluated. Both involve validation-based loss, while hyperparameter spaces may be high-dimensional and conditional, with some settings active only under particular choices.

2. PRELIMINARIES

  • Classification algorithms map training data to predictive functions, and their hyperparameters alter how the algorithms work.Hyperparameters may be optimized in an outer loop that evaluates configurations using cross-validation.Source: §2 · pp. 1–2

2.1 Model Selection

  • Model selection chooses the algorithm with the best estimated generalization performance by comparing validation loss across training-validation splits.The paper uses k-fold cross-validation; its loss is misclassification rate.Source: §2.1 · p. 2

2.2 Hyperparameter Optimization

  • Hyperparameter optimization searches a potentially high-dimensional space whose settings can be conditional, leaving some parameters inactive under particular choices.For example, polynomial-kernel settings are irrelevant when a different SVM kernel is used.Source: §2.2 · p. 2

3. COMBINED ALGORITHM SELECTION AND HYPERPARAMETER OPTIMIZATION (CASH)

CASH jointly selects a learning algorithm and its hyperparameters, and can be represented as one hierarchical hyperparameter-optimization problem. SMBO methods use a model and an acquisition function to guide evaluations; SMAC and TPE handle the conditional structure of this search.

3. COMBINED ALGORITHM SELECTION AND HYPERPARAMETER OPTIMIZATION (CASH)

  • CASH can be reformulated as one hierarchical search space, with a root parameter selecting an algorithm and its parameters conditional on that choice.SMBO proposes configurations, evaluates their loss, and updates its model until the time budget is exhausted; the best observed configuration is returned.Source: §3 · p. 2
  • SMBO selects configurations by maximizing an acquisition function that uses the model’s predictive distribution to estimate the value of evaluating each candidate.Expected improvement trades off exploration and exploitation; it is calculated from the modeled distribution of error rates relative to the current best.Source: §3 · p. 2

3.1 Sequential Model-based Algorithm Configuration (SMAC)

  • SMAC uses random forests to model loss and handles conditional parameters by assigning inactive ones default values, allowing trees to split on whether a parameter is active.Its incumbent evaluation uses progressively more folds, discarding a poorly performing configuration after as little as one fold; every second configuration is selected at random for diversification.Source: §3.1 · p. 3

3.2 Tree-structured Parzen Estimator (TPE)

  • TPE separates models of p(c) and p(λ | c), fitting ℓ(·) to configurations below a loss threshold and g(·) to those at or above it.It chooses the threshold as a quantile of observed losses, then searches random candidate configurations for the smallest g(λ)/ℓ(λ); its estimators accommodate hierarchical discrete, continuous, and conditional variables.Source: §3.2 · p. 3

4. AUTO-WEKA

Auto-WEKA applies CASH to the classification algorithms and feature selectors implemented in WEKA, encoding their choices and conditional hyperparameters in one search space. It supports SMAC and TPE variants and was evaluated using four parallel processes.

4. AUTO-WEKA

  • Auto-WEKA covers all WEKA classification algorithms and feature selectors, including base classifiers, meta-methods, ensembles, and their hyperparameters.For each dataset, the implementation considers only classifiers applicable to that dataset.Source: §4 · pp. 3–4
  • Discretizing each hyperparameter domain to at most 10 values would yield over 10^47 settings in the studied hypothesis space.The space also includes roughly 10^40 ensemble configurations and roughly 10^6 feature-selection choices.Source: §4 · p. 4
  • Its conditional parameter space represents base classifiers, meta-classifiers, and ensembles, with ensemble configurations allowing up to five base classifiers.Feature selection is separately toggled and, when active, combines a search method with an evaluator before classification.Source: §4 · p. 4
  • Auto-WEKA provides variants using SMAC and TPE, the Bayesian optimization algorithms used here because they handle hierarchical parameter spaces.Both variants use the optimizers’ default meta-hyperparameters; further tuning was not attempted because guarding against overfitting would require a separate meta-training/validation split.Source: §4 · p. 5
  • The evaluated Auto-WEKA version was restricted to four parallel optimization processes, a setting chosen to resemble commonly used workstations.The authors report that more parallel runs identified high-quality configurations faster in experiments.Source: §4 · p. 5

5. EVALUATING AUTO-WEKA

Across 21 datasets, Auto-WEKA searched the joint space of classifiers and hyperparameters, often outperforming standard baselines on cross-validation and test performance. Its selected classifiers and feature-selection methods varied with dataset size.

5.1 Experimental setup

  • Auto-WEKA was evaluated on 21 datasets; test data was held out from optimization and used only for offline model evaluation.The experiments used 25 runs per optimizer and a 30-hour budget per run; Auto-WEKA reserved 30% of training data for post-optimization validation.Source: §5.1 · pp. 5–6

5.2 Algorithm Selection and CASH: Baseline Methods

  • No single method performed well across all datasets: every method was at least 22% worse than the best on at least one dataset.Among the 39 default classifiers, Dorothea’s best and worst misclassification rates were 4.93% and 99.24%.Source: §5.2 · p. 6
  • Random grid search improved on Ex-Def by 6% on rotated MNIST with background images, showing that tuning hyperparameters can improve on default classifier selection.Random grid search considered settings for all 27 WEKA base classifiers as well as the 39 default classifiers.Source: §5.2 · p. 6

5.4 Results for Test Performance

  • Auto-WEKA outperformed random grid search and Ex-Def on 15/21 test datasets, tied in 3, and lost in 3; it was best on all 11 largest datasets.It beat the oracle-best Random Grid classifier on 9/21 datasets, tied on 2, and lost on 10.Source: Table 4 · p. 7, §5.4 · p. 7

5.3 Results for Cross-Validation Performance

  • With 4 × 30 CPU hours per dataset, Auto-WEKA outperformed random grid search in 20/21 cross-validation cases, with relative error reductions above 15% in 12/21.Random grid search used an average of 400 CPU hours per dataset; SMAC outperformed TPE in 19/21 cases, with one tie.Source: §5.3 · p. 7, Table 4 · p. 7

5.5 Classifiers Selected by Auto-WEKA

  • No classifier dominated selection: the three most frequent—random forests, single layer perceptron, and SVMs—were each chosen in roughly 12% of cases, with choices differing by dataset size.Large datasets favored meta methods more than small datasets; some base classifiers were often selected within meta methods but rarely on their own.Source: §5.5 · p. 8, Figure 2 · p. 8
  • Feature selection was used more often on small datasets; when used, the ranker was favored, while information gain was favored as an evaluator on large datasets.Small datasets used feature evaluators at roughly similar frequencies; the authors interpret data-dependent selection as allowing regularization on small datasets and more complex hypotheses on large ones.Source: §5.5 · p. 8, Figure 3 · p. 8

6. CONCLUSION AND FUTURE WORK

Auto-WEKA applies Bayesian optimization to the combined algorithm-and-hyperparameter search; across 21 datasets it often outperformed standard methods, especially on large datasets. The authors recommend SMAC and identify overfitting detection and ensemble-related extensions as future work.

6. CONCLUSION AND FUTURE WORK

  • Auto-WEKA is a fully automated tool using Bayesian optimization to search the combined algorithm-selection and hyperparameter-optimization space.Its optimizer iteratively models the algorithm/hyperparameter landscape to identify new points for investigation.Source: §6 · p. 8
  • Across 21 prominent datasets, Auto-WEKA often outperformed standard algorithm selection and hyperparameter optimization methods, especially on large datasets.Source: §6 · p. 8
  • The authors recommend the SMAC-based Auto-WEKA variant after comparing two optimizers over a 786-dimensional parameter space.Source: §6 · p. 8
  • The authors made Auto-WEKA available as a freely downloadable software package for end users.The paper gives its project URL.Source: §6 · pp. 8–9
  • Larger cross-validation than test-data improvements motivate future work on overfitting detection, parameter sharing within ensembles, and building ensembles iteratively.The proposed overfitting investigation would go beyond the simple correlation-based approach; iterative ensemble construction would add algorithms with maximal marginal contribution.Source: §6 · p. 9

Related papers

Loading 1208.3719v2…