Source-linked AI summary
Can AutoML outperform humans? An evaluation on popular OpenML datasets using AutoML Benchmark
Marc Hanussek, Matthias Blohm, Maximilien Kintz
TL;DR
The paper asks whether AutoML can outperform human data scientists, addressing limited direct evidence from comparable tasks. It compares four frameworks across 12 OpenML datasets and one real-life regression dataset, finding that AutoML performed better or equally in 7 of 12 OpenML cases.
Problem
The paper examines whether AutoML can outperform human data scientists, because existing comparisons often omit manually engineered and tuned human solutions.
Method
The study evaluates four AutoML frameworks on six OpenML classification tasks, six OpenML regression tasks, and one real-life regression dataset using benchmark metrics.
Results
AutoML performed better or equally than humans in the primary metric in 7 out of 12 OpenML cases, and in both metrics in those same cases.
Takeaways & Limitations
AutoML is a serious strategy because it sometimes outperforms human data scientists and otherwise often delivers nearly similar results.
Abstract
from arXiv · showhide
In the last few years, Automated Machine Learning (AutoML) has gained much attention. With that said, the question arises whether AutoML can outperform results achieved by human data scientists. This paper compares four AutoML frameworks on 12 different popular datasets from OpenML; six of them supervised classification tasks and the other six supervised regression ones. Additionally, we consider a real-life dataset from one of our recent projects. The results show that the automated frameworks perform better or equal than the machine learning community in 7 out of 12 OpenML tasks.
I. INTRODUCTION
AutoML is becoming more capable and important, intensifying the question of whether it can outperform human data scientists. This work addresses the limited direct comparison between AutoML and manually engineered and tuned human solutions.
- AutoML approaches for large-scale regression and classification are gaining popularity and improving through automated feature engineering.
- Existing benchmarks often report AutoML performance without comparing it with human data scientists solving the same tasks manually.
- The study uses OpenML tasks and their leaderboard details as a starting point for comparing human and AutoML performance.
- The paper covers related work, benchmarking methodology, experiments across tasks and tools, and concluding implications.
II. RELATED WORK
Prior work demonstrates strong AutoML performance and compares different AutoML tools, but usually does not measure performance against human solutions without AutoML. This paper uses an integrated benchmarking tool to conduct that comparison.
- Recent studies report impressive AutoML performance and compare multiple frameworks across datasets and tasks.
- The main gap is the lack of comparisons with how well human data scientists could solve the same tasks without AutoML frameworks.
- The benchmark evaluates four AutoML tools on OpenML datasets and one project dataset using AutoML Benchmark's integrated datasets and benchmarking functions.
A. Datasets
The benchmark selects six popular OpenML classification tasks and six regression tasks, defines human performance from non-AutoML leaderboard results, and adds one preprocessed real-life regression dataset. Experiments mainly use defaults with one-hour fold runtimes, plus a five-hour extension for one task.
- A. Datasets: The study selects six popular classification tasks and six regression tasks from OpenML, with two regression tasks exchanged because their runs came from one user.
- A. Datasets: Human performance is defined as the best non-AutoML result on each OpenML leaderboard.
- A. Datasets: The additional real-life dataset contains more than 300000 points and 35 mixed-type features for supervised regression, after data scientists addressed poor data quality.
- A. Datasets: Benchmarks mainly use project defaults, with one-hour runtime per fold and a one-time five-hour extension for the best framework on supervised classification credit-g.
- A. Datasets: Classification uses ROC AUC and accuracy, while regression uses RMSE and MAE; frameworks optimize AUC and RMSE as primary metrics.
C. AutoML frameworks
The benchmark includes four AutoML frameworks spanning recent and established tools, deep-learning-only and scikit-learn-based systems, and systems combining both approaches. The supplied visual passages provide no further framework comparison.
- C. AutoML frameworks: The evaluated frameworks are TPOT, H2O, auto-sklearn, and AutoGluon.
- C. AutoML frameworks: The selection spans frameworks with different ages and methodological bases, including deep-learning-only, scikit-learn-based, and hybrid approaches.
D. Cross-Validation and Metrics
The benchmark used 10-fold cross-validation with ROC AUC and accuracy for classification, and RMSE and MAE for regression; frameworks optimized the primary metrics.
- D. Cross-Validation and Metrics: Classification used ROC AUC (auc) and accuracy (acc), while regression used root-mean-square error (rmse) and mean absolute error (mae).The regression metrics excluded R2 because it could not be retrieved from the OpenML website.
- D. Cross-Validation and Metrics: The tasks used 10-fold cross-validation, and frameworks optimized for auc and rmse as the primary metrics.The remaining metrics served as additional evaluation measures.
E. Hardware
The benchmark ran on a dedicated local Fraunhofer IAO server with multi-core CPUs, 256 GB memory, and two GPUs.
- E. Hardware: The benchmark ran on two Intel Xeon Silver 4114 CPUs with 20 total cores, 256 GB DDR4 memory, and two NVIDIA GeForce GTX 1080 Ti GPUs.The GPUs provided more than 22 GB of total VRAM.
IV. RESULTS AND ANALYSIS
Across 12 OpenML tasks, AutoML matched or exceeded humans in seven cases, usually by small margins, with stronger outperformance on some regression tasks and the human-preprocessed real-life dataset.
- Overall results: 7 out of 12 cases showed AutoML performing better or equal than humans in the primary metric.The same seven cases also showed better-or-equal performance in both reported metrics.
- Task-type patterns: All seven better-or-equal cases were either easy classification tasks solved perfectly by both sides or regression tasks.OpenML contains substantially more community runs for classification than regression, which may affect comparative performance.
- Magnitude of differences: Most AutoML results were only slightly better or worse than human results.Excluding two regression tasks with tremendous AutoML outperformance, the mean MAPE was 0.83%.
- Runtime analysis: For supervised classification credit-g, H2O scored an AUC of 0.7892 with a five-hour fold limit versus 0.799 with one hour.The extended runtime did not improve the reported score in this comparison.
- Framework comparison: Three of four AutoML frameworks were better or equal to humans on average when AutoML achieved better-or-equal primary-metric performance.This suggests dataset, human-result strength, or task type mattered more than the specific framework choice.
- Real-life dataset: On the real-life damage-events dataset, AutoML achieved an MAE 25% better than conventional methods.The dataset comparison is partly qualified because it was human-preprocessed, unlike the OpenML datasets.
V. CONCLUSION, OUTLOOK AND FUTURE WORK
The authors conclude that AutoML is a serious strategy for machine-learning projects, while identifying domain-knowledge integration and text classification as future directions.
- V. CONCLUSION, OUTLOOK AND FUTURE WORK: AutoML frameworks can outperform human data scientists or deliver nearly similar results, making them a serious strategy for machine-learning experiments and projects.Beginners can obtain decent results with limited machine-learning terminology, while experts can rapidly establish strong baselines without extensive feature engineering or model tuning.
- V. CONCLUSION, OUTLOOK AND FUTURE WORK: A future challenge is combining domain knowledge with autonomous AutoML frameworks, because users currently cannot incorporate such knowledge appropriately.The authors suggest this could help nonspecialists incorporate domain knowledge into machine-learning workflows.
- V. CONCLUSION, OUTLOOK AND FUTURE WORK: Future work will examine AutoML suitability and performance for text classification on real-life datasets.
APPENDIX A MAIN EXPERIMENTAL RESULTS
The appendix reports task-level comparisons between AutoML frameworks and human results across supervised classification and regression tasks on OpenML.
- The appendix covers supervised classification and supervised regression tasks drawn from OpenML.
- The reported task outcomes are presented in an appendix results table with task names and OpenML task identifiers.
- Some OpenML tasks show AutoML outperforming human results, while others do not.The supplied appendix fragments mark both positive and negative task-level outcomes, including YES and NO entries.