PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, Frank Hutter

arXiv:2007.04074v3cs.LGstat.ML

TL;DR

AutoML must choose effective pipelines while operating under rigid resource limits, and its own high-level configuration choices can remain manual. The paper introduces PoSH Auto-sklearn and Auto-sklearn 2.0, using portfolios, successive halving, and policy selection to automate those choices. Across 39 evaluation datasets, Auto-sklearn 2.0 substantially improves over Auto-sklearn 1.0 and other evaluated systems.

  • Problem

    AutoML systems still leave high-level choices such as validation strategy and budget allocation to users, despite aiming to automate machine-learning design.

  • Method

    The paper combines portfolios and successive halving with a model-based policy selector that chooses an AutoML configuration for each dataset.

  • Results

    Across 39 datasets, Auto-sklearn 2.0 reduces ADTM relative error by up to a factor of 4.5 versus Auto-sklearn 1.0 and achieves lower loss after 10 minutes than Auto-sklearn 1.0 after 60 minutes.

  • Takeaways & Limitations

    Auto-sklearn 2.0 provides a hands-free AutoML setup that automatically chooses an optimization policy for a new task under resource limitations.

  • Takeaways & Limitations

    Performance is optimized for a given budget, metric, and configuration space, and combining successive halving with Bayesian optimization has no performance guarantee.

Abstract

from arXiv · show

Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success. In this paper, we introduce new AutoML approaches motivated by our winning submission to the second ChaLearn AutoML challenge. We develop PoSH Auto-sklearn, which enables AutoML systems to work well on large datasets under rigid time limits by using a new, simple and meta-feature-free meta-learning technique and by employing a successful bandit strategy for budget allocation. However, PoSH Auto-sklearn introduces even more ways of running AutoML and might make it harder for users to set it up correctly. Therefore, we also go one step further and study the design space of AutoML itself, proposing a solution towards truly hands-free AutoML. Together, these changes give rise to the next generation of our AutoML system, Auto-sklearn 2.0. We verify the improvements by these additions in an extensive experimental study on 39 AutoML benchmark datasets. We conclude the paper by comparing to other popular AutoML frameworks and Auto-sklearn 1.0, reducing the relative error by up to a factor of 4.5, and yielding a performance in 10 minutes that is substantially better than what Auto-sklearn 1.0 achieves within an hour.

1. Introduction

The paper develops PoSH Auto-sklearn and Auto-sklearn 2.0 to improve AutoML under rigid resource limits while reducing manual design decisions. It evaluates these advances against earlier Auto-sklearn and other systems.

  • Motivation: ChaLearn AutoML challenges evaluate systems under rigid time and memory constraints, requiring predictions in less than 20 minutes.These limits target rapid prototyping and reduced compute requirements.
  • Contributions: The authors won both the first and second AutoML challenges with modified versions of Auto-sklearn.The paper details the progression from Auto-sklearn 1.0 to PoSH Auto-sklearn and then Auto-sklearn 2.0.
  • Part I: PoSH Auto-sklearn: PoSH Auto-sklearn combines portfolios, successive halving, and budget allocation choices to improve efficiency and scalability.The combination produced stronger results in 10 minutes than Auto-sklearn 1.0 achieved in 60 minutes.
  • Part II: Auto-sklearn 2.0: Auto-sklearn 2.0 uses meta-learning based on algorithm selection to choose the best AutoML setting for a given dataset.This extends the system beyond manually selected high-level choices such as holdout versus cross-validation or successive halving.
  • Evaluation: The study evaluates Auto-sklearn 2.0 against other AutoML systems and reports improved performance under rigid time constraints.The evaluation uses the AutoML benchmark described in the paper.

2. Problem Statement

The paper formalizes AutoML as searching for a trained pipeline that minimizes expected generalization error under finite data and time constraints. It extends this formulation to policies that configure the AutoML system across datasets.

  • AutoML objective: AutoML generates a trained, hyperparameterized pipeline that predicts for samples from a dataset distribution while minimizing expected generalization error.The pipeline is denoted Mλ and is parameterized by λ ∈ Λ.
  • Empirical evaluation: Because only finite observations are available, generalization error is empirically approximated using training and test data or validation procedures such as K-fold cross-validation.Pipeline search uses Dtrain, while final performance is estimated once on Dtest.
  • Search outcome: AutoML systems return the best pipeline found within the search process rather than necessarily locating the exact optimum λ∗.The returned pipeline is denoted Mˆλ∗ with associated settings ˆλ∗.
  • Time-bounded AutoML: Users impose time and compute constraints, so the optimization budget T limits the total evaluation time spent searching pipelines.The system returns the best model found within that budget.
  • Meta-problem: The AutoML meta-problem seeks systems that generalize across a distribution of datasets rather than perform well on only one dataset.This objective is approximated using a finite set of meta-training datasets.
  • Optimization policies: An optimization policy π combines AutoML hyperparameters and components to configure a system for specific use cases.The paper first constructs policies manually, then learns a mapping from datasets to policies using algorithm selection.

3. Part I: Portfolio Successive Halving in PoSH Auto-sklearn

PoSH Auto-sklearn combines meta-feature-free portfolio warmstarting with successive halving and Bayesian optimization to improve AutoML under rigid time limits. Experiments on 39 datasets show stronger performance than Auto-sklearn 1.0, including better 10-minute results than its 60-minute performance.

  • Portfolio Building: PoSH Auto-sklearn replaces meta-feature-dependent warmstarting with an offline static portfolio of complementary pipelines for diverse datasets.The portfolio is evaluated sequentially on a new dataset, then Bayesian optimization continues from those evaluated pipelines if time remains.
  • Theoretical Properties of the Greedy Algorithm: 63% of the gap to the best possible portfolio is guaranteed to be closed by the greedy portfolio-construction algorithm.The result follows from the algorithm’s bounded worst-case error and the portfolio problem’s submodular structure; finding an optimal portfolio is NP-hard.
  • Budget Allocation using Successive Halving: Successive halving allocates more resources to promising pipelines instead of evaluating every configuration with the full budget.Bayesian optimization proposes pipelines at the lowest budget before successive halving selects candidates for larger budgets.
  • Portfolio vs. KND: Portfolio warmstarting improves performance across optimization budgets and most model-selection strategies, generally outperforming Bayesian optimization and usually improving over KND.The portfolio is statistically significantly better than KND in half of the compared settings.

4. Part II: Automating Design Decisions in AutoML

Auto-sklearn 2.0 automates high-level AutoML policy choices, selecting model-selection and budget-allocation strategies per dataset. Across experiments, it improves over Auto-sklearn 1.0 and benefits from successive halving, portfolios, and fallback handling.

  • Automated Policy Selection: Auto-sklearn 2.0 addresses the remaining AutoML design problem by automatically selecting an optimization policy for each dataset.The policy includes choices such as model selection and budget allocation.
  • Automated Policy Selection: Different datasets favor different optimization policies, so a strategy that performs best on one dataset can perform poorly on another.Figure 4 compares holdout, cross-validation, successive halving, and full-budget strategies across datasets and time budgets.
  • Approach: The per-dataset Policy Selector uses pairwise random-forest models over dataset meta-features and voting to choose among AutoML policies.Training weights observations by policy-loss differences and scales errors across datasets.
  • Experimental Results: 78% relative error reduction at 10 minutes and 65% at the longer budget give Auto-sklearn 2.0 the lowest error under both optimization budgets.These reductions correspond to factors of 4.5 and three compared with Auto-sklearn 1.0.
  • Experimental Results: Successive halving and fallback handling protect against large-dataset timeouts and memory failures by limiting investment in weak pipelines and returning early results.After removing affected datasets, Auto-sklearn 1.0 is still substantially worse than Auto-sklearn 2.0.
  • Experimental Results: Auto-sklearn 2.0 remains best in average rank, while PoSH Auto-sklearn outperforms Auto-sklearn 1.0 early before converging to a similar ranking.The average-rank analysis gives failures less weight than averaged performance.
  • Experimental Results: A larger policy set improves oracle potential, but the model-based selector does not yet reliably exploit all available model-selection strategies.The authors note that broader policy choices can reduce generalization for policy selection.
  • Experimental Results: Portfolio initialization is important: disabling portfolios causes a clear performance drop and removes strong starting pipelines from the search.The ablation also finds portfolios necessary for the single-best policy.

5. Comparison to other AutoML systems

On the AutoML benchmark, Auto-sklearn 2.0 achieves the best aggregate ranking among the evaluated systems, although no system wins on every dataset. Both Auto-sklearn versions also provide results reliably on the largest datasets, where several alternatives fail.

  • Benchmark Setup: The benchmark compares Auto-sklearn systems with Auto-WEKA, TPOT, H2O AutoML, and TunedRF under a fixed one-hour, eight-core setting.Runs use 32GB RAM and five repetitions to account for randomness.
  • Benchmark Setup: The evaluation reports log loss for multiclass datasets, 1−AUC for binary datasets, average rank, dataset wins, and binomial sign tests.Lower loss is better, and aggregate comparisons are made against Auto-sklearn 2.0.
  • Results: Auto-sklearn 2.0 has the lowest average rank, while H2O AutoML and Auto-sklearn 1.0 follow with roughly comparable aggregate performance.No system is best on every dataset.
  • Results: TunedRF, Auto-WEKA, and TPOT perform substantially worse on aggregate metrics than Auto-sklearn 2.0.The comparison uses both ranking scores and the number of dataset wins.
  • Results: Both Auto-sklearn versions reliably produce results on all datasets, including the largest datasets where several competing methods fail.This robustness is reported alongside the aggregate benchmark results.

6. Related Work

Related work positions AutoML as a combination of configuration, search, model selection, budget allocation, meta-learning, and ensembling choices. Prior research covers portfolios, multi-fidelity optimization, successive halving, and algorithm selection, while automatically choosing the model selection strategy remains largely unaddressed.

  • Portfolios: Algorithm portfolios allocate time across multiple algorithms or pipelines to reduce average solving cost when runtimes vary substantially.Greedy methods are widely used to construct portfolios, including portfolios for ML pipelines and hyperparameter settings.
  • Portfolios: Online portfolios use previous evaluations to select from fixed pipeline sets, but require learning-curve extrapolation models and cannot combine directly with every budget strategy.
  • Budget allocation: Multi-fidelity optimization accelerates pipeline selection by evaluating cheaper approximations, such as dataset subsets or fewer iterations.The paper focuses on successive halving, which progressively reduces candidates while increasing resources per run.
  • Algorithm selection: Automatically selecting a model selection strategy for hyperparameter optimization had not previously been tackled as an algorithm selection problem.
  • AutoML systems: AutoML frameworks combine flexible pipeline spaces, search mechanisms, model selection, budget allocation, meta-learning, and post-hoc ensembling.
  • Model selection: Model selection strategies trade approximation quality against validation time, so no single strategy is uniformly best.

7. Discussion and Conclusion

The paper presents Auto-sklearn 2.0 as a hands-free successor to PoSH Auto-sklearn, combining portfolios, successive halving, and automatic configuration selection. Across broad evaluations, it substantially improves over Auto-sklearn 1.0, while remaining specialized to specified budgets, metrics, and configuration spaces.

  • Discussion and Conclusion: Auto-sklearn 2.0 automatically chooses an AutoML setup for a new task under resource limitations.Its three improvements are offline portfolios, successive halving, and automatic selection of the system configuration.
  • Discussion and Conclusion: 208 meta-datasets supported system construction, while 39 datasets were used for evaluation.
  • Discussion and Conclusion: ADTM was reduced by up to a factor of 4.5, and 10-minute loss was lower than Auto-sklearn 1.0 after 60 minutes.
  • Discussion and Conclusion: A model-based policy selector for model selection strategy had the largest ablation impact and supported robust runs on unseen datasets.
  • Limitations: The system optimizes performance for a specified optimization budget, performance metric, and configuration space.

B. Additional results and experiments

The appendix reports additional experiments supporting the main findings, including reduced search-space results, results without post-hoc ensembles, and raw values before averaging.

  • Additional results and experiments: Additional experiments examine the reduced search space, omit post-hoc ensembles, and report raw numbers before averaging.

B.1 Early Stopping and Retrieving Intermittent Results

The experiments study early stopping and intermittent result retrieval for iterative algorithms under time limits. Combining the iterative search space, intermittent retrieval, and a portfolio dominates in the short term, with Auto-sklearn 1.0 catching up after 50 minutes.

  • Early stopping: Per-evaluation CPU-time limits prevent a single long algorithm run from stalling optimization, but overly low limits can cause widespread failures.Iterative algorithms also suffer when iteration counts are too high or too low for the available time.
  • Successive halving: Successive halving estimates portfolio generalization error by progressively retaining top-performing pipelines under increasing budgets.The algorithm takes an ordered pipeline set, minimal and maximal budgets, and a downsampling rate as inputs.
  • Ablation: The ablation compares Auto-sklearn 1.0 with an iterative-only space, intermittent retrieval, and portfolio replacement in sequence.Intermittent disk writes reduce failures, while portfolio replacement gives the best overall performance.
  • Results: 50 minutes is the catch-up point for Auto-sklearn 1.0 after the iterative space, intermittent retrieval, and portfolio combination dominates initially.The comparison is presented through a lower-is-better ranking plot across the tested variants.

B.2 Performance Without Post-Hoc Ensembling

Without post-hoc ensembling, the component comparisons remain qualitatively similar, while ensembling improves average normalized balanced error in every case. Auto-sklearn 2.0 still leads comparisons against PoSH Auto-sklearn and Auto-sklearn 1.0.

  • Ensembling improves average normalized balanced error rate in every compared configuration.The comparison uses the single best model without ensembling, whereas the main results include post-hoc ensembling.

B.3 Unaggregated results

The appendix reports raw balanced error rates for portfolio and system comparisons and formalizes the portfolio objective through its connection to sensor placement. It also shows that holdout-based portfolio selection is neither submodular nor monotone.

  • Unaggregated results: Raw balanced error rates are reported for portfolio comparisons and for Auto-sklearn 2.0, PoSH Auto-sklearn, and Auto-sklearn 1.0 at 10- and 60-minute budgets.Tables 13–16 provide unnormalized results corresponding to the main comparisons.
  • Portfolio formulation: Portfolio test-loss minimization is equivalent to a sensor-placement problem for minimizing detection time.Datasets correspond to intrusion scenarios, candidate pipelines to sensor locations, and portfolio test loss to detection time.
  • Portfolio formulation: The portfolio objective maximizes penalty reduction, defined as the improvement over the maximum possible penalty.For the AutoML setting, penalty reduction is penalty_d(∞) minus the best pipeline’s test loss on dataset d.
  • Properties of selection objectives: Holdout-based selection is neither submodular nor monotone.With e = (8, 6), the discrete derivative is −1 for A and 0 for B, violating submodularity; the negative derivative violates monotonicity.

C.4 Successive Halving

The successive-halving selection strategy is likewise neither submodular nor monotone, as shown by a learning-curve counterexample. The problem can also be framed as facility location.

  • C.4 Successive Halving: Successive-halving algorithm selection is neither submodular nor monotone.The argument evaluates performance rather than penalty reduction, which the passage states is equivalent.
  • C.4 Successive Halving: In the counterexample, successive halving yields f(B) = 8 and f(A) = 5, with discrete derivatives −1 and 0 for e conditioned on A and B.The negative derivative violates monotonicity, while the derivative ordering violates submodularity.
  • C.4 Successive Halving: The AutoML problem can also be phrased as a facility location problem.The paper states that results for facility location can therefore be applied to this problem setup.

D. Implementation Details

The experiments use a specified Python-based software stack and fixed Auto-sklearn package versions. Successive-halving budgets are fixed by model family, with further tuning identified as potentially useful but expensive.

  • Software setup: Experiments use Python 3 with NumPy, SciPy, scikit-learn, pandas, and Matplotlib.Auto-sklearn 0.12.6 was used for experiments, while version 0.12.7 added Auto-sklearn 2.0 functionality for the benchmark.
  • Configuration: The Auto-sklearn 2.0 configuration space is given in Table 18.The passage identifies the table as the source of the configuration space.
  • Budget settings: The successive-halving parameter eta is fixed at 4, with budgets of 32–512 for tree methods and 64–1024 for linear models.Further tuning of these hyperparameters is described as interesting but expensive.

E. Datasets

The study uses OpenML datasets for meta-training and testing, documents their characteristics, and specifies the Auto-sklearn 2.0 configuration space.

  • Configuration: Auto-sklearn 2.0’s configuration space uses iterative models and preprocessing that transforms data for the classification algorithms.The table’s final column indicates whether each hyperparameter is searched on a log10 scale.
  • Datasets: The experiments use 208 datasets in Dmeta and 39 datasets in Dtest, with dataset characteristics reported in Tables 19 and 20.The tables list dataset names, OpenML task IDs, observations, features, and classes.
  • Datasets: Table 19 organizes the 208 meta-training datasets by number of features.Each entry reports the dataset name, OpenML task ID, number of observations, number of features, and number of classes.
  • Datasets: Table 20 organizes the 39 test datasets by number of features.Each entry reports the dataset name, OpenML task ID, number of observations, number of features, and number of classes.
Loading 2007.04074v3…