Source-linked AI summary
An Open Source AutoML Benchmark
Pieter Gijsbers, Erin LeDell, Janek Thomas, Sébastien Poirier, Bernd Bischl, Joaquin Vanschoren
TL;DR
Comparing AutoML systems is difficult because existing evaluations often use limited datasets and flawed setups. The paper introduces an open, extensible, ongoing benchmark built with public datasets and best-practice procedures, then compares four systems across 39 datasets. Results show no system consistently outperforms all others, and some tasks remain no better than Random Forest within four hours.
Problem
Existing AutoML comparisons often use limited, reused datasets and flawed evaluation setups, making objective tool comparison difficult.
Method
The paper develops an open-source, extensible, ongoing benchmark using 39 public classification datasets, standardized resources, and established benchmarking practices.
Results
No AutoML system consistently outperforms all others; on some datasets, none outperforms a Random Forest within four hours.
Takeaways & Limitations
The benchmark’s current results identify high-dimensional and highly multi-class problems as challenging and provide avenues for further AutoML research.
Abstract
from arXiv · showhide
In recent years, an active field of research has developed around automated machine learning (AutoML). Unfortunately, comparing different AutoML systems is hard and often done incorrectly. We introduce an open, ongoing, and extensible benchmark framework which follows best practices and avoids common mistakes. The framework is open-source, uses public datasets and has a website with up-to-date results. We use the framework to conduct a thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.
1. Introduction
AutoML aims to reduce the expertise, labor, and time required to build machine learning systems, but comparing tools is difficult because existing comparisons often use narrow datasets and flawed evaluations.
- AutoML automates the construction and optimization of machine learning pipelines, helping novices build useful models and experts save time.
- There is no universally best AutoML approach, so comparisons are needed to guide practitioners and provide objective feedback to researchers.
- Existing comparisons often reuse a small, limited dataset collection, increasing risks of dataset-specific overfitting and selection bias.
- Incorrect execution of competing methods can further undermine the validity of AutoML comparisons.
A New Hope
The paper introduces an open, extensible, and ongoing AutoML benchmark designed to address weaknesses in existing comparisons.
- The benchmark is open source and accepts additions or updates to AutoML systems through pull requests.
- It remains ongoing by incorporating new datasets and rerunning experiments after substantial AutoML tool updates.
- An accompanying website publishes current benchmark results and related information.
2. Related Literature
AutoML systems differ in optimization strategies, pipeline generation, algorithm libraries, meta-learning, and post-processing, while prior benchmarks have faced evaluation problems and weak baselines.
- AutoML methods vary in optimization method, pipeline structure, algorithm library, use of meta-learning, and post-processing.
- Auto-WEKA uses Bayesian optimization with WEKA, while auto-sklearn adds meta-learning and ensemble construction to a scikit-learn-based approach.
- TPOT optimizes scikit-learn pipelines through genetic programming.
- Other approaches include Bayesian optimization for gradient boosting, low-rank pipeline prediction, hierarchical planning, and bandit-based configuration selection.
- Earlier systematic benchmarks suffered from memory-management and evaluation-setup issues and lacked strong baselines for interpreting results.
- The benchmark instead was developed with framework developers, uses fewer but approximately 15-times-larger datasets on average, and follows established benchmarking practices.
3. Benchmark Design
The benchmark defines tasks, datasets, metrics, hardware, tool configurations, and baselines to support reproducible comparison, while acknowledging unresolved meta-learning and search-space limitations.
- Each benchmark task combines a dataset, an optimization metric, and specified computational resources.
- The benchmark uses 39 classification datasets spanning orders of magnitude in samples and features and varying in feature types and missing values.
- AUROC is used for binary classification and log loss for multi-class classification, with systems evaluated on the same metric they optimize.
- Standard AWS m5.2xlarge instances provide commodity-level hardware, while local or Docker execution is also supported.
- The study selects open-source tools for popularity, ease of use, and technical variety, and includes constant, untuned Random Forest, and tuned Random Forest baselines.
- Tools use default hyperparameters and search spaces, with resource settings fixed for comparison; therefore, the benchmark cannot isolate the effects of search spaces or optimization methods.
- Adding frameworks and OpenML-hosted datasets is lightweight, but new additions are not evaluated automatically.
- Meta-learning can give frameworks an unfair advantage on datasets used during meta-learning, and the benchmark leaves this issue for future work.
4. Results
Across the tested conditions, no AutoML system consistently outperformed all others, although performance differences were significant on some datasets and marginal on others. Relative scores generally remained close to the tuned random forest baseline, while Auto-WEKA performed poorest.
- Overall performance: No AutoML system consistently outperformed all others across the evaluated datasets.The benchmarks used 1-hour and 4-hour per-fold budgets, totaling around 8000 hours of computation.
- Overall performance: Performance differences were significant on some datasets but only marginal on others.
- Baseline comparison: Normalized scores were generally close to the tuned random forest baseline.Scores were scaled so that a constant predictor equals 0 and a tuned random forest equals 1.
- Framework comparison: Auto-WEKA had the poorest performance among the tested AutoML packages under the tested conditions.These results used a rather generous time budget.
5. Conclusion
The paper presents an open-source, extensible, ongoing AutoML benchmark that publishes current results online. Its results identify challenging problem types and motivate broader future evaluations.
- Contribution: The benchmark is open-source, extensible to AutoML frameworks and tasks, and ongoing through online publication of updated results.
- Findings: On some datasets, none of the frameworks outperformed a random forest within four hours.
- Findings: High-dimensional or highly multi-class problems were often challenging for the evaluated frameworks.
- Future work: Future work will add more frameworks and tasks, especially larger datasets and regression tasks.