Source-linked AI summary
Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools
Anh Truong, Austin Walters, Jeremy Goodsitt, Keegan Hines, C. Bayan Bruss, Reza Farivar
TL;DR
AutoML seeks to reduce the repetitive, expertise-intensive work in machine-learning pipelines. The paper compares tools’ functionality and evaluates them across many datasets and data segments, finding reasonable overall performance but no universally best tool. It also identifies uneven support across pipeline stages and scope limits in the evaluation.
Problem
Developing high-performing machine-learning models requires substantial human expertise and repetitive manual work, motivating evaluation of tools intended to automate these tasks.
Method
The paper compares AutoML functionality and evaluates selected tools across ten data segments and diverse supervised-learning datasets using accuracy, F1-score for imbalanced classification, and MSE for regression.
Results
Across evaluations, most tools achieved reasonable performance, but no tool outperformed all others across a plurality of tasks; H2O-Automl, Auto-keras, and Auto-sklearn generally performed better than several alternatives.
Takeaways & Limitations
Tool choice is task-dependent: H2O-Automl slightly leads binary classification and regression but performs poorly on multiclass classification, while Auto-keras is stable and slightly better on multiclass classification.
Takeaways & Limitations
The evaluation omitted most commercialized tools because licensed or trial versions were unavailable and omitted some open-source tools lacking widely supported Python wrappers.
Abstract
from arXiv · showhide
There has been considerable growth and interest in industrial applications of machine learning (ML) in recent years. ML engineers, as a consequence, are in high demand across the industry, yet improving the efficiency of ML engineers remains a fundamental challenge. Automated machine learning (AutoML) has emerged as a way to save time and effort on repetitive tasks in ML pipelines, such as data pre-processing, feature engineering, model selection, hyperparameter optimization, and prediction result analysis. In this paper, we investigate the current state of AutoML tools aiming to automate these tasks. We conduct various evaluations of the tools on many datasets, in different data segments, to examine their performance, and compare their advantages and disadvantages on different test cases.
I. INTRODUCTION
AutoML emerged to reduce the repetitive, expertise-intensive work of developing machine-learning models as demand for ML engineers grew. This paper surveys available tools, compares their functionality, and evaluates their performance across diverse datasets and applications.
- Motivation: AutoML aims to reduce repetitive tasks in machine-learning pipelines and improve productivity for data scientists, engineers, and researchers.The paper considers preprocessing, feature engineering, model selection, hyperparameter optimization, and result analysis.
- Research questions: The paper asks what functionalities tools provide, how they perform across real-world datasets, how speed trades off against accuracy, and whether results are reproducible.These questions frame both the functionality comparison and experimental evaluation.
- Motivation: Developing high-performing models requires substantial human expertise, intuition, experience, and tedious manual parameter tuning.This combination of engineering demand and manual effort motivated automation of ML tasks.
- History and ecosystem: AutoML development progressed from academic tools such as Auto-Weka, Auto-sklearn, and TPOT to open-source packages and commercial or cloud-based services.Examples include Google Cloud AutoML, Microsoft AzureML, Salesforce TransmogrifAI, and Uber Ludwig.
- Tool landscape: AutoML platforms emphasize different capabilities, including time-series processing, interactive experimentation, model deployment, code export, and categorical ensembling.The surveyed tools therefore address different parts of the broader AutoML space.
III. AUTOML PLATFORMS’ FEATURES AND FUNCTIONALITY COMPARISON: THE COMMON PIPELINE
Most AutoML tools follow a common three-stage pipeline whose components are optimized iteratively. The paper uses this pipeline to organize a cross-tool functionality comparison.
- Common pipeline: Most AutoML tools use a common three-stage pipeline and iteratively optimize its components to obtain the best outcome.Figure 1 illustrates the shared pipeline structure.
- Comparison framework: The paper compares surveyed tools’ features and functionalities using the common pipeline as its organizing framework.Figure 2 provides a brief cross-tool comparison, with detailed comparisons following later.
A. Data Preprocessing and Feature Engineering
Data preprocessing and feature engineering remain unevenly automated: tools often require human schema information or manual preprocessing, although some detect basic or richer data types.
- Data preprocessing: Data preprocessing is not handled well by any surveyed AutoML tool and still requires considerable human intervention.Data-type and schema detection are not widely supported.
- Data-type detection: TransmogrifAI supports detailed data-type detection, but this functionality appears unstable across multiple datasets.Examples include addresses, phone numbers, names, and currency.
- Data-type detection: H2O-Automl, H2O-DriverlessAI, DataRobot, MLjar, and Darwin detect basic schemas, mainly numerical, categorical, and time-series data.These capabilities give the tools an advantage in preprocessing.
- Feature engineering: Auto-ml, Auto-sklearn, AzureML, and Ludwig require user-specified column types for feature engineering, while other tools require more manual interaction.The tools differ in how much preprocessing information users must provide.
- Feature engineering: Auto-sklearn requires categorical inputs to be converted into integers, while TPOT and Auto-keras provide no preprocessing or feature-generation steps.TPOT and Auto-keras therefore accept only numerical feature matrices and require users to preprocess data manually.
B. Model Selection, Hyperparameter Optimization, and Architecture Search
AutoML tools select models from algorithm collections while tuning parameters through grid, random, or Bayesian search. They reduce search costs by using meta-features, joint optimization strategies, and constrained parameter spaces.
- Model selection: Tools train multiple model types with varied parameter settings, then select a best model or ensemble for supervised learning.Supported algorithms include logistic regression, tree-based methods, SVMs, and neural networks.
- Hyperparameter optimization: Hyperparameter optimization commonly uses grid search, random search, or Bayesian search, with different tools adopting different combinations.Auto-Weka uses SMAC, Auto-sklearn uses SMAC3, H2O-Automl and MLjar use random search, and several tools combine random and Bayesian search.
- Search acceleration: Auto-sklearn and Darwin use meta-features and previously trained meta-learners to initialize search from similar datasets.This is one approach for reducing model-search and hyperparameter-optimization time.
- Search acceleration: H2O-Automl reduces search by exploiting the relationship between model selection and hyperparameter optimization.The paper describes this as a second approach to pruning the parameter space.
- Functionality comparison: The functionality comparison distinguishes supervised learning, unsupervised learning, feature processing, and model interpretation or explainability capabilities.The comparison also records stability issues, input-format requirements, and cases requiring user input.
C. Model Interpretation and Prediction Analysis
Commercial AutoML tools provide detailed model-result representations, including dashboards, feature importance, visualizations, outlier highlighting, and explanation methods.
- Commercial tools provide detailed result representations through model dashboards, feature importance, and visualizations such as lift charts and prediction distributions.They can also highlight uncertain outlier predictions and support reason codes, LIME, and Shapley-based explanations.
IV. EXPERIMENTAL EVALUATION
The evaluation studies selected AutoML tools across nearly 300 OpenML datasets, standardized data splits, multiple data segments, three supervised tasks, and constrained runtimes.
- Nearly 300 OpenML datasets provide a broad evaluation basis with standardized, numerically pre-processed inputs for fairer tool comparison.The datasets are selected across characteristics such as categorical-feature ratio, missing proportion, and class imbalance.
- Commercialized tools and some open-source tools are excluded because licensed or trial versions and widely supported Python wrappers were unavailable.
- Datasets are split into training and testing portions at a 4:1 ratio, and every tool receives the same proportions.
- The experiments cover binary classification, multiclass classification, and regression on Amazon EC2 p2.xlarge instances.Each instance provides one Tesla K80 GPU, 4 vCPUs, and 61 GiB of host memory.
- 15 minutes is selected as the common runtime limit after targeting completion on 70% of datasets across 5,400 experiments.The extensive evaluation consumes approximately 1,350 EC2 hours, followed by longer-runtime experiments on randomized subsets.
A. Evaluation on multiple data segments
Across ten data segments and three supervised tasks, AutoML performance depends on dataset characteristics, tool, and completion reliability under the 15-minute limit.
- Performance varies more with larger categorical-feature proportions and less with more samples in binary classification.Most tools also struggle with class imbalance, except Ludwig and Darwin.
- H2O-Automl and Darwin slightly outperform other tools on binary classification, although Darwin fails to deliver results for 26% of datasets.
- Auto-keras and Auto-sklearn slightly outperform the other tools on multiclass classification, while Auto-sklearn misses results for 30% of datasets.
- Regression performance variance increases when feature dimensions decrease or missing proportions increase.
- H2O-Automl and Auto-sklearn slightly outperform Auto-keras and TPOT in regression, while Darwin cannot deliver results on half of the datasets.
- Across tasks, tool rankings and completion depend strongly on the task and dataset: Auto-keras is weaker on some binary datasets, while Darwin completes only about half of allotted-time runs.
B. Evaluation on time limit
The time-limit evaluation examines whether AutoML tools deliver results quickly and whether additional runtime consistently improves performance across dataset sizes.
- The study evaluates runtime effects using randomly selected datasets from sample-size ranges and time limits of 5 minutes through 3 hours.Each tool’s accuracy is evaluated under 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 3 hours.
C. Evaluation on robustness
The robustness evaluation tests whether AutoML tools produce similar results across repeated runs on the same datasets. H2O-Automl and Ludwig are most stable, while TPOT and Auto-sklearn are less stable for regression.
- Robustness is assessed by running each tool ten times for ten minutes on the same randomly selected dataset for each task.The datasets contain between 10,000 and 50,000 samples.
- H2O-Automl and Ludwig obtain very stable performance across binary classification, multiclass classification, and regression.
- Darwin, Auto-keras, and Auto-ml are slightly less stable than H2O-Automl across the evaluated tasks.
- TPOT and Auto-sklearn are somewhat unstable in the regression task.
- Ludwig’s performance is stable but deviates substantially from the other tools.
V. CONCLUSIONS AND FUTURE WORK
The paper evaluates AutoML tools across the common machine-learning pipeline and many datasets, finding distinct strengths across commercial and open-source tools. No tool consistently outperforms all others, although H2O-Automl, Auto-keras, and Auto-sklearn perform better overall in the reported evaluations.
- Commercial tools extend functionality toward data-schema detection, feature engineering, and detailed result analysis for interpretation.
- Open-source tools focus more on training and selecting the best model.
- Most AutoML tools obtain reasonable performance across many datasets, but no tool outperforms all others on a plurality of tasks.
- H2O-Automl, Auto-keras, and Auto-sklearn perform better than Ludwig, Darwin, TPOT, and Auto-ml across the reported evaluations.
- H2O-Automl slightly outperforms the rest for binary classification and regression but has low performance in multiclass classification.
- The paper identifies a gap in support for the first and third AutoML pipeline stages and expects future developments there.
APPENDIX
The appendix contains dataset descriptions.
- The appendix includes a table titled “DATASET DESCRIPTIONS.”
- The appendix presents dataset descriptions in tabular form.
- The appendix’s dataset-description material is identified as Table I.