AMLB: an AutoML Benchmark

1. Introduction

• AutoML automates data-driven ML pipeline design and optimization, a process that otherwise requires expertise and substantial work because effective choices vary by data set.The choices include data preparation, algorithm selection, and hyperparameter tuning.Source: §1 · pp. 1–2

1.1 The Need for Standardized Benchmarking

• Inconsistent, self-selected data-set suites make results difficult to compare across papers and may permit cherry-picked results.Older suites may also fail to reveal differences between current frameworks.Source: §1.1 · p. 2

1.2 Our Contributions

• AMLB introduces an open-source benchmark and tool for reproducibly evaluating open-source AutoML frameworks on curated public tabular data sets.Unstructured data are out of scope, and the evaluation covers 9 frameworks across 71 classification and 33 regression tasks.Source: §1.2 · pp. 2–3
• AMLB supports end-to-end evaluation and analysis of model accuracy, inference-time trade-offs, and framework failures.It also provides an interactive visualization tool for exploring results or reproducing the paper’s analyses.Source: §1.2 · p. 3

2. Related Literature

• Machine-learning benchmarks may omit problematic data characteristics, while AutoML evaluations need explicit runtime budgets because frameworks commonly run until their budgets expire.The paper argues that these needs warrant benchmarks designed specifically for AutoML.Source: §2 · p. 3

2.1 Evaluation of Automated Machine Learning Frameworks

• Prior AutoML studies report technical setup errors, unequal resource budgets, frequent failures, and data-set choices that can make framework comparisons difficult to interpret.The literature also reports that varying evaluation settings or comparing results on self-selected suites impedes conclusions across studies.Source: §2.1 · pp. 4–5
• Prior studies have examined performance across data-characteristic subsets, but small numbers of tasks per subset limit generalization of those findings.One comparison found framework rankings differed between benchmark data sets and competitions, while the authors note that competition results are hard to interpret.Source: §2.1 · pp. 4–5

2.2 Other (Benchmark) Literature

• Other benchmark efforts target optimization algorithms, including HPO and black-box optimization, or provide tabular performance data for neural architecture search.Examples include HPOlib, HPOBench, HPO-B, PROFET, COCO, and NAS-Bench-101.Source: §2.2 · pp. 5–6

3. AutoML Frameworks

• The study selects 9 open-source frameworks to cover varied optimization approaches, with selections from academia and industry and developer collaboration to support correct integration.The paper excludes some frameworks because evaluation costs or technical issues prevented their inclusion.Source: §3 · pp. 6–7

3.1.1 AutoGluon-Tabular

• AutoGluon uses a predetermined model set with multi-layer stacking and ensembling rather than pipeline search or hyperparameter tuning.Its evaluated presets trade model accuracy for increasingly faster inference.Source: §3.1.1 · pp. 7–8

3.1.2 auto-sklearn

• The study evaluates both auto-sklearn versions because version 2 differs substantially from version 1, including its search space, evaluation strategy, and portfolio-based initialization.Auto-sklearn 2 does not support regression, and its meta-learning prevented a clean evaluation in this study.Source: §3.1.2 · p. 8

3.1.3 FLAML

• FLAML uses cost-frugal multi-fidelity search over boosting and selected scikit-learn algorithms, while GAMA uses configurable genetic programming and multi-objective pipeline optimization.FLAML is designed for small time budgets; GAMA uses asynchronous search and ensemble selection.Source: §3.1.3 · pp. 8–9, §3.1.4 · p. 9

3.1.5 H2O AutoML

• H2O AutoML searches a portfolio and algorithm configurations with early stopping and stacked ensembles, while LightAutoML targets quick inference and interpretability with linear and boosting models.H2O uses predefined search-time allocations and preprocessing by default; LightAutoML combines models through voting or stacking.Source: §3.1.5 · p. 9, §3.1.6 · pp. 9–10

3.1.7 MLJAR

• MLJAR offers perform and compete presets, Naive AutoML follows a simple sequential modeling workflow, and TPOT uses evolutionary optimization of pipeline trees.Naive AutoML was excluded from the paper’s Section 6 results because of evaluation issues; TPOT’s default search uses scikit-learn components and XGBoost.Source: §3.1.7 · p. 10, §3.1.8 · p. 10, §3.1.9 · p. 10

3.2 Baselines

• The benchmark includes constant, random forest, and tuned random forest baselines, with the tuned version optimizing max features using cross-validation.The tuned random forest is described as the strongest baseline and as an approximation of a typical human first modeling approach.Source: §3.2 · p. 11

4. Software

• AMLB automates experiment execution across supported frameworks and is designed to enable correct, reproducible evaluations under recreated evaluation conditions.Its components include data loading, runners, job orchestration, and result post-processing.Source: §4 · pp. 11–12

4.3.1 Commands

• Local, container, and AWS runners support execution on a machine, in consistent container setups, or across parallel EC2 instances.The command-line interface lets users specify the benchmark, resource constraints, execution mode, and parallelization.Source: §4 · p. 11, §4.3.1 · p. 13

4.1 Extensible Framework Structure

• Frameworks integrate through a minimal interface of fewer than 250 lines across at most four files, covering installation, data provision, and prediction recording.Data and predictions can be exchanged as Python objects or common file formats, including across programming languages.Source: §4.1 · p. 12

4.2 Extensible Benchmarks

• AMLB supports OpenML tasks and suites for reproducible evaluations, as well as local data sets with manually defined splits for private use cases.OpenML suites specify data sets and one or more train/test splits.Source: §4.2 · p. 12

4.3 Using the Software

• Users specify the framework, benchmark suite, and task constraints, including maximum training time and limits on CPU cores, memory, and disk volume.Constraints are declared in a constraints.yaml file.Source: §4.3 · pp. 12–13

5.1 Benchmark Suites

• The benchmark comprises 71 classification and 33 regression tasks in machine-readable OpenML suites, with task definitions separated from resource constraints.Suites reference data, evaluation procedures and splits, targets, and task types.Source: §5.1 · pp. 13–14, §5.1.1 · p. 14

5.1.1 Data Sets

• Tasks vary widely in size and data characteristics, while selection favors realistic, open data and excludes free-form text and non-i.i.d. tasks.The authors note that restricting the benchmark to open data without text features limits options, especially for large data sets.Source: §5.1.1 · pp. 14–15

5.1.2 Performance Metrics

• AUC evaluates binary classification, log loss multi-class classification, and rmse regression; models are not calibrated unless frameworks do so by default.AUC is not sensitive to calibration.Source: §5.1.2 · p. 16

5.1.3 Missing Values In Experimental Results

• Failures correlate with task properties, so missing results are imputed with a constant predictor rather than ignored or filled using a random forest.The constant predictor returns the empirical class distribution for classification and the empirical mean for regression, penalizing failed runs.Source: §5.1.3 · pp. 16–17

5.2 Experimental Setup

• Experiments used commodity-level hardware and generally default framework configurations, controlling runtime, resources, and the shared optimization and evaluation metric.The hardware setup used 32 GB memory and 8 vCPUs; further framework customization was prohibited in the experiment design.Source: §5.2 · p. 17, §5.2.1 · p. 17, §5.2.2 · p. 17

5.3.1 Limitations of the Design

• The comparison cannot attribute performance to individual framework components, records final-model results rather than anytime performance, and omits qualitative features.Anytime performance is approximated by evaluating at 1- and 4-hour constraints; qualitative features such as interpretability analysis, reports, usability, and support are not evaluated.Source: §5.3.1 · p. 18

5.3.2 Limitations of the Experiments

• Only a few developer-recommended modes were evaluated, and overlapping meta-learning data make Auto-sklearn 2 results likely optimistic.The authors say careful meta-tuning or additional modes may produce better results, while Auto-sklearn 2's meta-model cannot exclude individual benchmark data sets.Source: §5.3.2 · p. 19, §5.3.3 · p. 19

5.4 Overfitting the Benchmark

• Because developers use scarce public data sets to improve systems iteratively, benchmark data cannot be assumed truly unseen, and overfitting is difficult to rule out.Keeping benchmark data private would compromise reproducibility; the authors say the suites' size and design may reduce overfitting, but this is difficult to guarantee.Source: §5.4 · pp. 19–20

6. Results

• These results reflect framework versions run in June 2023, with some experiments using September 2021 versions; outcomes may differ with later versions and other resource or time settings.The authors encourage additional experiments suited to the use case and using up-to-date frameworks.Source: §6 · p. 20

6.1 Predictive Performance

• AUTOGLUON(B) and TPOT ranked best and worst, respectively, in almost every accuracy setting; AUTOGLUON(B) significantly outranked the worst frameworks, while TPOT was never significantly worse than the second worst.The CD plots compare average ranks after imputing missing values with the constant predictor baseline.Source: §6.1 · p. 21, Figure 2 · p. 22
• Similar average ranks can mask different performance distributions: AUTOGLUON(HQIL) was more consistently above random forest than MLJAR(P), while TPOT’s prediction quality varied more than tuned random forest’s.The contrast is reported for one-hour regression tasks for AUTOGLUON(HQIL) and MLJAR(P).Source: §6.1 · p. 21
• Performance was generally similar across time constraints, improving slightly with a larger budget; the authors suspect other frameworks may be limited by their search spaces.A stacking ensemble may be needed for the best performance, which other methods may not reach regardless of time constraint.Source: §6.1 · pp. 23–24, Figure 4 · p. 24

6.2 Bradley-Terry Trees

• For datasets with more than 8,237 instances, AUTOGLUON(B) was preferred by a large margin over AUTOGLUON(HQ); among these, FLAML was a preferred third option when feature count was at most 43.For similarly small-instance, very wide datasets with more than 1,301 features, AUTOGLUON(B) remained preferred, followed by AUTOGLUON(HQ) and LIGHTAUTOML with similar performances. Small balanced binary classification tasks were an exception, with various frameworks preferred over AUTOGLUON.Source: §6.2 · p. 26

6.3 Model Accuracy vs. Inference Time Trade-offs

• AUTOGLUON(B)’s high accuracy came with extremely slow inference, whereas FLAML had very fast inference; ensemble models were generally slower.The authors attribute AUTOGLUON(B)’s slow inference to large, multi-layer stacking and ensembles; H2O AUTOML and AUTOGLUON(HQIL) retained fast inference with optimized models or explicit speed selection.Source: §6.3 · p. 27, Figure 6 · p. 27
• The accuracy–inference-speed Pareto front shows that the highest-accuracy frameworks incur an inference-time cost, so accuracy alone does not capture overall utility.The comparison uses average normalized accuracy and median per-row prediction speed for one-hour models.Source: §6.3 · pp. 27–28, Figure 7 · p. 28

6.4 Observed AutoML Failures

• Memory and time constraints were the main sources of errors; failures were more common in classification, including failures on highly imbalanced data and two small datasets with validation splits missing classes.The authors describe their error categories as crude and subjective; implementation errors require changes to the AutoML system to resolve.Source: §6.4 · p. 29

7. Conclusion

• The benchmark automates framework installation, data provisioning, resource allocation, and result processing to simplify evaluation and improve reproducibility.Source: §7 · pp. 29–30
• AUTOGLUON has the highest average rank, and most AutoML frameworks outperform the strongest baseline in most scenarios.Bradley-Terry trees also show that relative performance depends on data characteristics such as dataset size, dimensionality, and class balance.Source: §7 · pp. 30–31
• More accurate frameworks often incur much slower inference, and not all frameworks produce Pareto-optimal models.Inference-time differences can span multiple orders of magnitude; some frameworks offer presets to adjust the accuracy–inference-time trade-off.Source: §7 · p. 31
• Dataset size is the main cause when frameworks fail to produce a model, indicating that not all methods scale well.Source: §7 · p. 31

7.1 Limitations

• The study uses preset configurations, cannot isolate individual framework components, omits qualitative factors, and runs on modest 8-CPU machines.Source: §7.1 · p. 31

7.2 Future Work

• The authors plan to update the task suites with community input and add text-feature problems, which the current selection does not represent.Source: §7.2 · pp. 31–32
• Future extensions include multi-objective, non-i.i.d., and semi-supervised tasks, while evaluations at different compute scales may reveal different behavior.Source: §7.2 · p. 32

7.3 Parting Words

• The authors invite community contributions of datasets, framework integrations, feedback, and code to help keep the benchmark useful.Source: §7.3 · p. 32

• The authors hope to update the suites with datasets that represent current challenges.Source: Appendix A · p. 34
• The appendix lists the tasks in separate regression and classification benchmark suites.Source: Table 2 · p. 34, Appendix A · p. 34, Table 3 · pp. 35–36

• Tables 4–15 report results by framework and task for different task types and time budgets.Source: Appendix B · p. 37
• Table entries report the mean and standard deviation over completed folds and the number of folds without a result; a dash means no fold completed.Source: Appendix B · p. 37
• Classification results use AUC for binary tasks and logloss for multiclass tasks, with one-hour and four-hour budgets represented.Source: Table 4 · p. 38, Table 6 · p. 40, Table 8 · p. 42
• Regression results use RMSE and include one-hour and four-hour budget tables.Source: Table 12 · p. 46, Table 14 · p. 48

B.1 BT-Trees

• AUTOGLUON(B) is preferred for many tasks, especially larger datasets and more complex data involving class imbalance, more classes, or missing values.Differences are generally smaller on small datasets, where preferred frameworks vary across task types.Source: §B.1 · p. 50, Figure 10 · p. 51, Figure 11 · p. 52

Appendix C. Software

• Appendix C contains additional details about the developed and used software.Source: Appendix C · p. 54

C.1 Architecture Overview

• The local machine uploads configuration files and downloads results; experiment jobs run on EC2 in Docker containers.The local machine deploys jobs and tracks instances, while each instance retrieves the configuration, Docker image, and dataset before running the experiment.Source: §C.1 · p. 54
• The setup leaves almost no requirements on the local machine, with configuration files uploaded and results downloaded as the only cloud data transfers.This assumes the Docker images and benchmark version are published and OpenML is used.Source: §C.1 · p. 54

C.2 Framework Versions

• Because of budget constraints, some results were reused from the 2021 evaluation for frameworks with few updates between evaluations.Most results reflect framework versions available in early June 2023; reused older results are marked with an asterisk.Source: §C.2 · p. 54

C.2.1 AutoGluon

• ‘Best quality’ ensembles reuse models trained on different training-data subsets, while ‘high quality’ trains one model per algorithm and configuration, typically lowering performance but speeding inference.Under the limited-inference constraint, models that are too slow are discarded before the procedure continues.Source: §C.2.1 · p. 54
• AUTOGLUON’s ‘high quality’ presets receive a 10% reduced time constraint to account for post-search refitting, which can exceed the communicated limit.The adjustment was intended to make final training time better observe the constraint.Source: §C.2.1 · p. 54

C.2.2 TPOT

• TPOT’s inference time is not reported because its required encoding is excluded from both training-time and inference-time measurements.The benchmark supplies encoded data to TPOT.Source: §C.2.2 · p. 55

Appendix D. AutoML Framework Errors

• The appendix groups the ‘fixed’ errors from Section 6.4 into the implementation-error category.Source: Appendix D · p. 57
• With larger time budgets, memory and timeout errors increase, but implementation errors and total errors decrease; most one-hour classification implementation errors occur early.The authors suspect early pipeline-optimization termination can leave frameworks unprepared to produce predictions; a more generous budget may allow fallback to optimized models.Source: Appendix D · p. 57

D.1 Class Imbalance

• With only five minority-class instances, yeast and wine-qualitywhite failures occur when one enters the test fold, affecting GAMA, LightAutoML, and TPOT.The experiments use 10-fold cross-validation; four or five of those instances are available in each training split.Source: §D.1 · p. 57

D.2 MLJarSupervised

• MLJARSupervised(B) had more than twice the implementation errors of the next framework; its recurring errors appeared only in compete mode and likely stemmed from ensembling.Of 284 errors, 256 were variations of three errors; the authors note that ensembling is used in compete mode but not performance mode.Source: §D.2 · pp. 57–58

D.3 AutoGluon

• AUTOGLUON(HQIL) version 0.8.0 had a calibration bug that caused crashes only under one-hour constraints, so affected missing results were imputed using AUTOGLUON(HQ) version 0.8.0.The authors report that the bug was fixed and that the fixed version performed very similarly to AUTOGLUON(HQ) version 0.8.0.Source: §D.3 · p. 58

• NAIVEAUTOML is a baseline designed to produce an adequate model quickly by building a pipeline sequentially, rather than jointly optimizing algorithm selection and hyperparameters.The framework uses an aggressive early-stopping heuristic by design.Source: Appendix E · p. 59
• The integration disabled early stopping by setting the maximum HPO iterations to 1010 and limited each evaluation to 5% of the total time, or three minutes under a one-hour budget.The authors later judged the 5% execution-time limit too strict.Source: Appendix E · p. 59
• NAIVEAUTOML showed poor performance and fast inference, which the authors attribute to a bad configuration and an execution timeout too low to evaluate good baseline models.The timeout often allowed only fast but relatively poor-performing models, such as Gaussian Naive Bayes, to be evaluated.Source: Figure 15 · p. 59, Appendix E · p. 59