Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

1. Introduction

• AutoML combines optimization over preprocessing, learning algorithms, and hyperparameters to reduce the effort of building machine-learning applications.Systems including Auto-WEKA (Thornton et al., 2013), hyperopt-sklearn, Auto-sklearn, TPOT (Olson et al., 2016a), and Auto-Keras (Jin et al., 2019) exemplify this approach.Source: §1 · p. 1
• Under challenges requiring predictions in less than 20 minutes, the authors’ modified Auto-sklearn systems won both the first and second AutoML challenges.These constraints target rapid prototyping and broader access by limiting computational requirements.Source: §1 · p. 2

systems.

• PoSH Auto-sklearn combines successive halving with improved meta-learning portfolios, producing stronger results in 10 minutes than Auto-sklearn 1.0 achieved in 60 minutes.Successive halving allocates more resources to promising machine-learning pipelines instead of evaluating every configuration with the full budget.Source: systems · p. 2, §1 · p. 2
• Auto-sklearn 2.0 uses algorithm-selection meta-learning to automatically choose the AutoML system setting best suited to a given dataset.This automates high-level decisions such as model-selection strategy and time-budget allocation that were previously designed manually.Source: systems · p. 2, §1 · p. 2
• The paper evaluates Auto-sklearn 2.0 against other popular AutoML systems on the AutoML benchmark under rigid time constraints.The study follows the presented progression from Auto-sklearn 1.0 through PoSH Auto-sklearn to Auto-sklearn 2.0.Source: systems · p. 2, Figure 1 · p. 3

2. Problem Statement

• AutoML seeks a trained pipeline Mλ that maps x to y while minimizing expected generalization error for dataset distributions Pd.The pipeline is hyperparameterized by λ ∈ Λ, and datasets are drawn from a distribution P(D).Source: §2 · p. 3
• Because only finite observations are available, AutoML searches using Dtrain and estimates final performance once on the disjoint Dtest.Generalization error can also be empirically approximated with K-fold cross-validation, and the returned model is the best pipeline found rather than necessarily the exact optimum.Source: §2 · pp. 3–4

2.1 Time-bounded AutoML

• Time-bounded AutoML minimizes search error subject to an overall optimization budget T, returning the best model found across evaluated pipelines.The evaluation time for a pipeline is denoted tλ, and the budget explicitly constrains the total search effort.Source: §2.1 · p. 4

2.2 Generalization of AutoML

• AutoML must generalize across the dataset distribution P(D), so policies π configure systems Aπ and may be manually constructed, learned from data, or mapped to new datasets.The paper introduces a learned mapping Ξ : D → π for automatically suggesting an optimization policy, while aiming to simplify use of single AutoML systems.Source: §2.2 · pp. 4–5

3.1 Portfolio Building

• The static portfolio avoids dataset-specific meta-features by evaluating diverse complementary pipelines offline, then using them to warmstart Bayesian optimization.Portfolio configurations range from broadly strong pipelines to specialized ones, forming an optimized initial design for Bayesian optimization.Source: §3.1 · p. 6

3.1.1 Approach

• The greedy portfolio algorithm repeatedly adds the candidate that most reduces estimated error across meta-datasets until reaching a predefined size.Portfolio construction uses candidate pipelines, meta-datasets, and a model-selection strategy to select complementary configurations.Source: §3.1.1 · pp. 6–7

3.1.2 Theoretical Properties of the Greedy Algorithm

• The portfolio penalty function is submodular, the optimal portfolio problem is NP-hard, and greedy selection closes at least 63% of the gap to the best portfolio of the same size.Extending the greedy portfolio to 5·|P| can close 99% of the corresponding gap under the stated result.Source: §3.1.2 · pp. 8–9

3.2 Budget Allocation using Successive Halving

• Successive halving replaces equal full-budget evaluations by repeatedly retaining the 1/η lowest-error pipelines and multiplying their budgets by η.Bayesian optimization proposes new pipelines at the lowest budget, while successive halving prunes and re-evaluates them at larger budgets.Source: §3.2 · p. 9, §3.2.1 · p. 10

3.3 Practical Considerations and Challenge Results

• To support successive halving, the challenge configuration restricted the search to iterative algorithms, stored partially fitted models, and added safeguards for extreme dataset sizes.These choices simplified fidelity control and protected against timeouts and memory limits, while several competition-specific tricks were not retained for the paper.Source: §3.3 · pp. 10–11
• PoSH Auto-sklearn was the overall winner of the second AutoML challenge.The competition results are reported in Table 1, which summarizes normalized area under the ROC curve across five datasets.Source: §3.3 · p. 11, Table 1 · p. 12

3.4 Experimental Setup

• The evaluation uses 39 test datasets, separate from meta-datasets used for portfolio construction, and compares 10-minute and 60-minute optimization budgets using normalized ADTM error.Each pipeline evaluation is limited to one-tenth of the optimization budget and 4GB of memory.Source: §3.4 · p. 12, §3.4.1 · p. 12

3.4.1 Datasets

• Dmeta contains 208 OpenML training datasets, and its distribution spans Dtest well, although several test datasets lie outside the meta-data distribution.Datasets were filtered by sample count, attributes, sparsity, temporal and string features, synthetic status, similarity, and overlap with Dtest.Source: §3.4.1 · p. 13, Figure 3 · p. 13

3.4.2 Meta-data Generation

• For each optimization budget, four |Dmeta| × |C| performance matrices were built using holdout, 3-fold, 5-fold, and 10-fold cross-validation assessments.Customized pipelines were obtained with Auto-sklearn without meta-learning or ensemble building, then evaluated across all candidate pipeline–dataset pairs.Source: §3.4.2 · p. 14

3.4.3 Other Experimental Details

• Results are averaged over 10 repetitions, with Wilcoxon tests at α = 0.05 and rank distributions estimated across 39 datasets.Experiments use ensembles of size 50 with replacement; results without ensemble selection are provided in the appendix.Source: §3.4.3 · p. 14

3.5 Experimental Results

• PoSH Auto-sklearn was validated by comparing portfolio warmstarting with KND and no warmstarting, then comparing it with Auto-sklearn 1.0.The experiments also evaluate pure Bayesian optimization as a baseline.Source: §3.5 · p. 14, §3.5.1 · p. 14

3.5.2 PoSH Auto-sklearn vs Auto-sklearn 1.0

• 16.21 → 4.11 at 10 minutes and 7.17 → 3.19 at 60 minutes: PoSH Auto-sklearn strongly reduces normalized loss versus Auto-sklearn 1.0.These comparisons use the full search space and are reported across 10 repetitions on 39 datasets.Source: Table 3 · p. 15, §3.5.2 · p. 15

3.5.1 Portfolio vs. KND

• Portfolio warmstarting always improves over Bayesian optimization, while it outperforms k-nearest-datasets in every raw comparison and significantly in half the settings.The portfolio comparison covers model-selection strategies and optimization budgets across the benchmark settings.Source: §3.5.1 · p. 15, Table 2 · p. 15

3.5.2 PoSH Auto-sklearn vs Auto-sklearn 1.0

• Using the reduced search space, loss decreases from 5.29 to 4.11 at 10 minutes and from 3.98 to 3.19 at 60 minutes versus Auto-sklearn 1.0.With the portfolio, average results are inconclusive about whether successive halving or plain holdout is preferable.Source: §3.5.2 · p. 16

4. Part II: Automating Design Decisions in AutoML

• A policy selector automatically chooses among AutoML optimization policies because the best model-selection and budget-allocation strategies depend on each dataset and available resources.The policy combines model-selection strategy, budget allocation, and a corresponding portfolio, covering eight combinations of four validation strategies and two allocation methods.Source: §4 · p. 16, §4.1.1 · p. 17

4.1 Automated Policy Selection

• The system trains a model-based selector on policy performance and simple dataset meta-features, then applies the selected policy to a new dataset while reserving test access for final evaluation.Training evaluates each policy on meta-datasets, while testing computes the new dataset’s number of datapoints and features before selecting and running a policy.Source: §4.1 · p. 16, §4.1.1 · pp. 17, 18, 20

4.1.1 Approach

• The per-dataset selector predicts pairwise policy outcomes with weighted random forests and voting, while Bayesian optimization tunes seven shared selector hyperparameters.Weights reflect policy-loss differences after dataset-wise scaling, and shared hyperparameters prevent factorial growth as policies are added.Source: §4.1.1 · p. 19

4.3.1 Do we need per-dataset selection?

• Fallback to holdout with successive halving substantially improves robustness when a new dataset lies outside the meta-dataset distribution.The fallback is triggered when no meta-dataset dominates all new-dataset meta-features, preventing policies that might time out without producing a solution.Source: §4.1.1 · p. 20, §4.3.1 · p. 23

4.2 Experimental Results

• 78% and 65% relative-error reductions versus Auto-sklearn 1.0 at 10 and 60 minutes, respectively, make Auto-sklearn 2.0 significantly best on both budgets.These reductions correspond to factors of 4.5 and three, respectively, and remain substantially better after excluding large datasets where Auto-sklearn 1.0 failed.Source: Table 5 · p. 20, §4.2 · pp. 20–21

4.3.1 Do we need per-dataset selection?

• Per-dataset selection slightly improves over the single best policy at 10 minutes and significantly improves at 60 minutes, while realistic Bayesian-optimization metadata performs best.The cheapest portfolio-only metadata performs worst; adding ensembles does not improve over metadata containing Bayesian-optimization effects.Source: §4.3.1 · p. 22

4.3.2 Do we need different model selection strategies?

• A broader policy set increases attainable performance through complementarity, but the selector cannot yet fully exploit it and may generalize worse with more choices.Only holdout and only cross-validation have worse oracle performance than all strategies, whereas full-budget allocation is slightly preferable under selection.Source: §4.3.2 · p. 24

4.3.3 Do we still need to warm-start Bayesian optimization?

• Removing portfolios causes a clear performance drop for both the model-based selector and single-best policy, demonstrating that portfolio warm starts are necessary.The ablation removes portfolios from both selector-data construction and runtime, leaving Bayesian optimization and ensemble construction to start without warm-start pipelines.Source: §4.3.3 · p. 25

5. Comparison to other AutoML systems

• The benchmark compared Auto-sklearn 1.0 and 2.0 with Auto-WEKA, TPOT, H2O AutoML, and tuned random forests across 39 datasets.The datasets matched Dtest, and the benchmark reported log loss for multiclass tasks and 1−AUC for binary tasks, with lower values better.Source: §5 · p. 25, §5.2 · p. 26

5.1 Integration and setup

• All systems were rerun locally in the 1h8c setting with eight CPU cores, 32GB RAM, five repetitions, and Singularity-based isolation.For Auto-sklearn, eight pipelines ran in parallel using asynchronous Bayesian optimization and Dask-based ensemble construction.Source: §5.1 · p. 26

5.2 Results

• Auto-sklearn 2.0 achieved the lowest average rank and strongest aggregate performance, while TunedRF, Auto-WEKA, and TPOT performed substantially worse.No system won on every dataset, but Auto-sklearn 2.0 had the best overall rank; H2O AutoML and Auto-sklearn 1.0 followed at roughly comparable levels.Source: §5.2 · p. 26, Table 9 · p. 27
• Both Auto-sklearn versions reliably produced results on all datasets, including the largest datasets where several competing methods failed.Source: §5.2 · p. 28

6. Related Work

• The related work therefore covers the paper’s individual contributions—portfolios, model-selection strategies, and algorithm selection—alongside prominent AutoML frameworks and their components.The section positions these contributions within broader AutoML, meta-learning, optimization, and ensembling research.Source: §6 · p. 28, §6.4 · p. 30, §6.4.1 · p. 31

6.1 Related Work on Portfolios

• Portfolio methods allocate evaluation time across multiple algorithms, often reducing model-selection time and remaining useful when full hyperparameter optimization is infeasible.Prior work finds portfolios can outperform Bayesian optimization while avoiding meta-features, shared metadata, and additional optimization software.Source: §6.1 · p. 28
• Greedy portfolio construction is popular, but prior variants either optimize rank instead of the target metric, require a model, or fail to execute strong configurations early under time constraints.Online portfolios additionally require learning-curve extrapolation and introduce complexity, while alternative portfolio-construction strategies are treated as orthogonal future work.Source: §6.1 · pp. 28–29

6.2 Related Work on Successive Halving

• Successive halving reduces candidates while increasing resources per run, whereas Auto-WEKA’s iterative fold evaluation can accelerate screening but cannot support post-hoc ensembling.The latter also limits each pipeline’s evaluation cost to at most holdout cost, which may remain too expensive.Source: §6.2 · p. 29

6.3 Related Work on Algorithm Selection

• Automatically selecting a model-selection strategy had not previously been addressed, but framing it as algorithm selection enables established methods beyond pairwise classification.Potential alternatives include AutoFolio and other existing algorithm-selection techniques.Source: §6.3 · pp. 29–30

6.4.1 Components of AutoML systems

• AutoML systems combine flexible pipeline spaces, search procedures, model selection, budget allocation, meta-learning, and post-hoc ensembling.CASH spaces support conditional hyperparameters and multiple search mechanisms, while common model-selection choices trade approximation quality against computational time.Source: §6.4.1 · p. 30

6.4.2 AutoML systems

• Prior AutoML systems span ensemble-based algorithm and hyperparameter selection, joint pipeline optimization, model-based optimization, grammatical evolution, stacking, dataset subsampling, and neural architecture search.Examples include Auto-WEKA, Hyperopt-sklearn, Auto-sklearn, TPOT-SH, H2O AutoML, AutoGluon, and deep-learning-oriented systems.Source: §6.4.2 · pp. 32–33

7. Discussion and Conclusion

• Auto-sklearn 2.0 provides hands-free setup selection through offline portfolios, successive halving, and automatic selection of the system configuration.These changes target faster strong initial results, reduced time on poorly performing pipelines, and closure of the AutoML system’s design space.Source: §7 · p. 33
• A 39-dataset evaluation reduced ADTM by up to 4.5× and achieved lower loss after 10 minutes than Auto-sklearn 1.0 after 60 minutes.The study used 208 meta-datasets to construct the systems and found Auto-sklearn 2.0 competitive with or better than other state-of-the-art AutoML systems.Source: §7 · p. 33
• A model-based policy selector had the largest ablation impact and made Auto-sklearn 2.0 robust on unseen datasets.The selector chooses the model-selection strategy.Source: §7 · p. 33
• Performance is optimized for a specified budget, metric, and configuration space, limiting direct transfer across different AutoML settings.Users could provide these components and meta-datasets to build customized Auto-sklearn 2.0 versions, but automatic transfer remains open.Source: §7 · pp. 33–34
• Combining successive halving with Bayesian optimization has no performance guarantee, motivating methods that increase successive halving’s lower budget over time.Future work also includes automating portfolio size, budget-strategy hyperparameters, configuration spaces, and additional selector meta-features.Source: §7 · p. 34
• Training data construction remains expensive, motivating shortcuts such as joint ranking models (Tornede et al., 2020) or non-linear collaborative filtering (Fusi et al., 2018).The paper also suggests an additional optimization loop on the training data to address several remaining hand-picked hyperparameters.Source: §7 · p. 34

A. Additional pseudo-code

• Algorithm 2 estimates a portfolio’s generalization error across meta-datasets using K-fold cross-validation.Its inputs include ordered ML pipelines, meta-datasets, and the number of folds K.Source: A. Additional pseudo-code · p. 35
• Algorithm 3 supplies pseudo-code for successive halving, including a condition comparing the current level l with the lower bound l_d.The supplied appendix passages identify the algorithm and show its control-flow condition.Source: A. Additional pseudo-code · p. 35

• The section further examines the reduced search space and provides additional experimental evidence for the paper’s findings.Source: B. Additional results and experiments · p. 35
• It reports the main-paper results without post-hoc ensembles.Source: B. Additional results and experiments · p. 35
• It also presents raw numbers before averaging.Source: B. Additional results and experiments · p. 35

B.1 Early Stopping and Retrieving Intermittent Results

• Intermittent result retrieval reduces failures by saving iterative results at checkpoints, ensuring every evaluation returns performance information for optimization.Checkpoints are spaced at geometrically increasing iteration numbers, allowing the optimizer to use partial runs instead of receiving no result when a time limit is exceeded.Source: §B.1 · p. 37
• Early stopping halts training when training or validation loss increases or stalls, preventing overfitting and avoiding poorly chosen iteration counts.This is especially relevant for iterative algorithms such as gradient boosting and stochastic linear models, where too many or too few iterations can harm performance.Source: §B.1 · pp. 35, 37
• Restricting the configuration space to iterative algorithms decreases performance, whereas adding intermittent retrieval and replacing the KND with a portfolio improves the overall result.The restriction is attributed to a reduced hypothesis space, while the portfolio produces the best overall performance in the staged comparison.Source: §B.1 · p. 37, Table 10 · p. 37
• The iterative search space, intermittent result retrieval, and portfolio combination dominates in the short term, while Auto-sklearn 1.0 catches up after 50 minutes.The ranking plot compares Auto-sklearn 1.0 with successive additions of these components; Auto-sklearn 2.0 performs even better in Table 5.Source: §B.1 · p. 37, Figure 7 · p. 38

Performance Without Post-Hoc Ensembling

• Ensembling improves the average normalized balanced error rate in every comparison, although the single-best-model results remain qualitatively similar to the ensemble results.The comparisons cover Bayesian optimization, k-nearest datasets, and a greedy portfolio.Source: §B.2 · p. 39
• Even when evaluating only the single best model, Auto-sklearn 2.0 outperforms PoSH Auto-sklearn and Auto-sklearn 1.0, while ensembling provides uniform performance improvements.Source: §B.2 · p. 39

B.3 Unaggregated results

• Unaggregated balanced error rates are provided for portfolios and Auto-sklearn 2.0, PoSH Auto-sklearn, and Auto-sklearn 1.0 at 10- and 60-minute budgets.Tables 13–16 preserve dataset-level results before normalization and averaging, with the lowest error boldfaced.Source: §B.3 · p. 39, Table 13 · p. 40, Table 14 · p. 41

C.1 Definitions

• A set function is submodular when each element’s discrete derivative decreases as the conditioning set grows, and monotone when enlarging the set does not decrease its value.These are defined using Δf(e|S) = f(S ∪ {e}) − f(S) and f(A) ≤ f(B) for A ⊆ B.Source: §C.1 · p. 44

C.2 Choosing on the test set

• Choosing a pipeline from a portfolio using test performance is equivalent to sensor placement for minimizing detection time.Datasets correspond to intrusion scenarios, candidate pipelines to sensor locations, portfolio loss to detection time, and portfolio selection to minimum test loss.Source: §C.2 · p. 44
• Under this equivalence, portfolio penalty reduction is the maximum penalty minus the portfolio’s minimum pipeline loss, with bounded metrics’ maximum penalty equal to 1.The mapping uses R(P, d) = penalty_d(∞) − min_p∈P L(p, D_d,test).Source: §C.2 · p. 44

C.3 Choosing on the validation set

• Selecting a pipeline by validation performance and reporting test performance is neither submodular nor monotone.A constructed example demonstrates both failures using validation–test performance pairs.Source: §C.3 · p. 45
• For A = {(5, 5), (7, 7)} and B = {(5, 5), (7, 7), (10, 10)}, adding e = (8, 6) gives derivatives −1 and 0, violating submodularity and monotonicity.The negative derivative means f(A) > f(A ∪ {e}).Source: §C.3 · p. 45

C.4 Successive Halving

• The constructed example demonstrates that successive-halving selection is not submodular.The violation is established using discrete derivatives after the second stage.Source: §C.4 · p. 45
• The same example demonstrates that successive-halving selection is not monotone.A negative discrete derivative implies f(A) > f(A ∪{e}), violating the monotonicity definition.Source: §C.4 · p. 45
• Successive halving selects algorithms by iteratively eliminating entries according to their learning-curve performances.The example tracks validation- and test-performance tuples across successive stages.Source: §C.4 · p. 45
• After the first iteration, successive halving eliminates entries 2 and 3 from B and entry 1 from A while advancing entries 1 and 4 from B.The sets are B = {((5, 5), (8, 8)), ((5, 5), (6, 6)), ((4, 4), (5, 5))} and A = {((5, 5), (7, 7))}.Source: §C.4 · p. 45
• For e = ((6, 5), (6, 5)), the second-stage performances are f(B) = 8 and f(A) = 5.These performance values produce discrete derivatives Δf(e|A) = −1 and Δf(e|B) = 0.Source: §C.4 · p. 45
• The negative derivative Δf(e|A) = −1 violates monotonicity because adding e reduces f(A).The passage states that f(A) > f(A ∪{e}).Source: §C.4 · p. 45

C.5 Further equalities

• The problem can also be phrased as a facility location problem, so facility-location statements apply to this setup.The related formulation is attributed to Krarup and Pruzan (1983).Source: §C.5 · p. 45

D.1 Software

• Auto-sklearn experiments used Python 3 with numpy, scipy, scikit-learn, pandas, and matplotlib, running package version 0.12.6.Auto-sklearn 2.0 functionality was added in version 0.12.7 for the AutoML benchmark.Source: §D.1 · p. 45

D.2 Configuration Space

• Auto-sklearn 2.0’s configuration space is specified in Table 18.The passage identifies Table 18 as the location of the configuration space used by Auto-sklearn 2.0.Source: §D.2 · p. 45

D.1 Software

• The exact package versions used for the AutoML benchmark are documented in Table 17.The table is specifically titled “Package versions used for the AutoML benchmark.”Source: §D.1 · p. 45, Table 17 · p. 46

D.3 Successive Halving hyperparameters

• All experiments used the same successive-halving hyperparameters, including eta = 4.The authors state that further tuning could be interesting but expensive.Source: §D.3 · p. 46
• Tree-based methods received budgets from 32 to 512, whereas linear models used budgets from 64 to 1024.The linear-model range applied to SGD and passive aggressive methods.Source: §D.3 · p. 46
• The minimum and maximum budgets were selected separately for tree-based methods and linear models.Tree-based methods used 32 and 512 as minimum and maximum budgets; SGD and passive aggressive models used 64 and 1024.Source: §D.3 · p. 46

• The paper provides the names, OpenML task IDs, and sizes of all datasets used in Tables 19 and 20.Source: E. Datasets · p. 46, Table 19 · p. 48, Table 20 · p. 49
• Dmeta comprises 208 datasets, characterized by name, OpenML task ID, observations, features, and classes.Source: Table 19 · p. 48
• Dtest comprises 39 datasets, characterized by name, OpenML task ID, observations, features, and classes.Source: Table 20 · p. 49