PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data

Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, Alexander Smola

arXiv:2003.06505v1stat.MLcs.LG

TL;DR

AutoGluon-Tabular targets the difficulty of producing accurate predictions from raw, heterogeneous tabular data while incorporating modern ML best practices. It combines robust preprocessing with multilayer stacking and repeated k-fold bagging, and its evaluation finds higher accuracy and robustness than popular AutoML frameworks, alongside a documented over-fitting caveat.

  • Problem

    Few existing AutoML frameworks robustly accept raw tabular data and deliver high-quality predictions without user input or software errors.

  • Method

    AutoGluon automatically preprocesses tabular data and combines diverse models through multilayer stacking and repeated k-fold bagging.

  • Results

    AutoGluon is significantly more accurate and more robust than evaluated AutoML frameworks across the reported benchmarks, with better average ranking and fewer failures.

  • Takeaways & Limitations

    The evaluation supports using extensive modeling practices and ensembling, rather than focusing only on combined algorithm selection and hyperparameter optimization, for tabular AutoML.

  • Takeaways & Limitations

    The paper focuses solely on AutoGluon-Tabular for structured data tables, while AutoGluon also supports text and image tasks outside this scope.

Abstract

from arXiv · show

We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only a single line of Python to train highly accurate machine learning models on an unprocessed tabular dataset such as a CSV file. Unlike existing AutoML frameworks that primarily focus on model/hyperparameter selection, AutoGluon-Tabular succeeds by ensembling multiple models and stacking them in multiple layers. Experiments reveal that our multi-layer combination of many models offers better use of allocated training time than seeking out the best. A second contribution is an extensive evaluation of public and commercial AutoML platforms including TPOT, H2O, AutoWEKA, auto-sklearn, AutoGluon, and Google AutoML Tables. Tests on a suite of 50 classification and regression tasks from Kaggle and the OpenML AutoML Benchmark reveal that AutoGluon is faster, more robust, and much more accurate. We find that AutoGluon often even outperforms the best-in-hindsight combination of all of its competitors. In two popular Kaggle competitions, AutoGluon beat 99% of the participating data scientists after merely 4h of training on the raw data.

1. Introduction

AutoGluon-Tabular addresses the difficulty of applying sophisticated ML best practices to raw tabular data by combining robust preprocessing, diverse models, and multilayer ensembling. The paper positions this unified approach as an alternative to AutoML systems centered mainly on model and hyperparameter search.

  • Sophisticated ML practices are increasingly difficult for experts to incorporate manually as techniques and architectures grow more complex.
  • AutoML can scale expert practices across many problems by repeatedly deploying strategies for model selection, ensembling, tuning, feature engineering, preprocessing, and data splitting.
  • Existing AutoML frameworks automate much of supervised learning, but few robustly transform raw tabular data into high-quality predictions without user input or software errors.Many frameworks also require numeric data without missing values, limiting direct applicability to common datasets.
  • Prior work focused primarily on Combined Algorithm Selection and Hyperparameter optimization, whose expensive search evaluates many poor model and hyperparameter configurations.
  • AutoGluon-Tabular combines automatic data-type recognition, robust preprocessing, diverse models, deep learning, and multilayer model ensembling for tabular AutoML.Its models range from off-the-shelf boosted trees to customized neural networks, with layer-wise stacking and out-of-fold tracking.
  • The paper codifies extensive data-science best practices in a unified framework and evaluates six AutoML frameworks on 50 curated datasets.The authors report that these practices substantially affect accuracy and that competing frameworks allocate training time less resourcefully than AutoGluon.

2. AutoGluon-Tabular

AutoGluon-Tabular is designed to turn raw tabular data into accurate predictions through simple, robust, time-aware automation. Its core combines diverse models with multi-layer stacking and repeated bagging while addressing data heterogeneity and over-fitting.

  • Design principles: The framework prioritizes simplicity, robustness, fault tolerance, and predictable timing for tabular AutoML.Users can train directly on raw data, tolerate individual model failures, resume canceled training, and specify runtime limits.
  • The fit API: AutoGluon lets users train and test on raw CSV data with three lines of Python.The API automatically preprocesses data, identifies the prediction problem, partitions folds, fits models, and creates an ensemble.
  • Data processing and models: AutoGluon automatically categorizes features and applies model-agnostic and model-specific preprocessing before fitting diverse model types.Its model portfolio includes neural networks, boosted trees, forests, extremely randomized trees, and k-nearest neighbors, trained in a predefined order under time limits.
  • Neural Network: The neural network uses separate embeddings for categorical variables and a linear shortcut path from each embedding directly to the output.These per-variable embeddings are presented as a novel AutoML design choice intended to improve gradient flow.
  • Multi-Layer Stack Ensembling: Multi-layer stacking feeds concatenated predictions from lower layers and original features into higher-layer stackers built from the same model types.The approach extends traditional stacking with layer-wise training and skip-like access to the original data.
  • Repeated k-fold Bagging: Repeated k-fold bagging averages out-of-fold predictions across random partitions to reduce variance and mitigate over-fitting, especially on smaller datasets.The number of repetitions is selected according to the available training time.

3. AutoML Frameworks

The evaluated AutoML frameworks differ in how they automate tabular modeling, from CASH-oriented search and model ensembles to end-to-end raw-data systems. Table 1 emphasizes whether frameworks can process raw tabular data with missing and non-numerical values automatically.

  • The comparison includes five widely used frameworks and indicates whether raw data with missing and non-numerical values can be handled without manual preprocessing or feature-type specification.
  • Auto-WEKA addresses CASH with Bayesian optimization over models and hyperparameters, followed by ensembling strategies.
  • auto-sklearn combines meta-learning, multi-fidelity hyperparameter optimization, and ensemble selection across scikit-learn models.
  • TPOT uses genetic algorithms to optimize data processing, hyperparameters, models, and optional stacking, though invalid candidate pipelines waste time.
  • H2O AutoML processes raw CSV input, uses one-layer stacking and bagging, and includes an XGBoost ensemble among its base models.
  • GCP-Tables provides managed end-to-end AutoML through Google Cloud APIs, with internals that remain unclear but include tree ensembles and AdaNet.

4. Experiments

The experiments compare AutoGluon with five other AutoML frameworks across 50 varied tabular datasets under controlled time limits. Across benchmarks and training durations, AutoGluon is reported as more accurate, more robust, and more consistent in using additional training time.

  • 4.1. Setup: 50 curated datasets span binary classification, multiclass classification, and regression tasks from the OpenML AutoML and Kaggle benchmarks.
  • 4.1. Setup: 390 prediction problems result from 39 OpenML datasets evaluated across 10 training/test splits, with both 1h and 4h training settings.
  • 4.1. Setup: AutoGluon was run with two-layer stacking and repeated bagging through a single fit call, optimizing the specified evaluation metric within the time limit.
  • 4.2. Results: AutoGluon was the only framework ranking better than second on average for every benchmark and time limit, and beat all competitors combined on 23/39 AutoML and 7/11 Kaggle datasets.
  • 4.2. Results: AutoGluon became more accurate with additional training time more reliably than other frameworks, which may begin over-fitting.
  • 4.2. Results: At 4h, AutoGluon reached leaderboard ranks 42/3505 and 39/2920 in the otto and bnp-paribas competitions, respectively.
  • 4.3. Ablation Studies: Removing repeated bagging, multi-layer stacking, bagging, and the neural network reduced overall predictive performance sequentially.

5. Conclusion

The paper presents AutoGluon-Tabular as an end-to-end tabular AutoML framework that combines robust preprocessing, neural networks, and multi-layer stacking with repeated bagging. Its evaluation finds higher accuracy than popular CASH-focused AutoML frameworks, showing that CASH is only one part of successful end-to-end AutoML.

  • AutoGluon-Tabular automatically manages the end-to-end machine-learning pipeline for structured data.
  • Its key components include heterogeneous-data processing, a modern neural network, multi-layer stacking, and repeated k-fold bagging.
  • The evaluation finds AutoGluon-Tabular significantly more accurate than popular AutoML frameworks focused on CASH.
  • The conclusion identifies CASH as one component rather than the entirety of a successful end-to-end AutoML framework.

A. AutoGluon Implementation Details

The implementation uses fixed, data-independent model defaults alongside a neural network with regularization and adaptive hidden-layer sizing. Preprocessing is specialized for numeric and categorical features to improve stability and handle missing or unusual values.

  • AutoGluon’s default hyperparameters were chosen a priori rather than tuned on the benchmark datasets.
  • The neural network uses ReLU activations, dropout, batch normalization, Adam with weight decay, and validation-based early stopping.
  • Regression targets are rescaled and trained with L1 loss, while ReLU replaces occasionally unreliable SeLU activations for robustness.
  • Numeric preprocessing includes median imputation and normalization, while categorical preprocessing adds Unknown and Other categories for missing, novel, and high-cardinality values.

B. Data used in Kaggle Benchmark

The Kaggle benchmark combines 11 competitions spanning regression and binary/multiclass classification, selected for suitability, prominence, recency, and single-table IID data.

  • Four competitions were selected because prior work showed that GCP-Tables could produce strong results on them.
  • The remaining datasets favored a mix of regression and binary/multiclass classification tasks with IID data, recent applications, or large participation.
  • 11 Kaggle competitions formed the benchmark, with task metrics including RMSLE, R2, MAE, log-loss, and AUC.
  • Competitions requiring manual joins or transformations into a single table were generally excluded because their preparation depends on domain-specific knowledge.
  • Kaggle performance used private leaderboard scores, except house-prices, where private scores were unavailable and public scores were reported.

C. Details Regarding Usage of AutoML frameworks

The benchmark applied each AutoML system uniformly while accounting for raw-data robustness, resource demands, timing behavior, and framework-specific compatibility constraints.

  • The evaluations ran each AutoML system on every dataset in exactly the same manner, avoiding manual dataset-specific adjustments.
  • Only H2O and GCP-Tables robustly handled raw CSV files; Auto-WEKA produced numerous raw-data errors and received preprocessed inputs.
  • With AutoGluon-processed data, Auto-WEKA, TPOT, and auto-sklearn matched their original benchmark performance without manually specified feature types.
  • The comparison used newer open-source framework versions where available, including TPOT 0.11.1, Auto-WEKA 2.6, H2O 3.28.0.1, and auto-sklearn 0.5.2.
  • Training used 1h and 4h limits for the AutoML Benchmark, but 4h and 8h limits with larger AWS instances for the larger Kaggle datasets.
  • The benchmark rescaled each dataset’s loss from 0 for the champion framework to 1 for the worst-performing framework.

D.1. Additional Results for AutoML Benchmark

Additional benchmark results compare frameworks across folds, time limits, datasets, and failure modes. AutoGluon generally led performance comparisons but also exhibited a resource-related failure on the largest dataset.

  • The benchmark figures compare framework loss relative to AutoGluon and show actual training times against specified limits.
  • Even after 4h, NoMultiStack usually could not outperform full AutoGluon trained for only 1h.
  • AutoGluon outperformed all other frameworks in 27 of 38 datasets when Dionis was excluded because every framework failed there.
  • AutoGluon outperformed all other frameworks in 24 of 39 AutoML Benchmark datasets under the 4h limit.
  • On the allstate-claims dataset, AutoGluon performance consistently improved with longer time limits, unlike the other frameworks.
  • AutoGluon failed on 1 of 39 datasets in the 4h runs and 0 datasets in the 1h runs; the 4h failure was Dionis.
  • Multi-layer stacking generated 355 features per successful base model, contributing to an out-of-memory failure in AutoGluon’s neural network on Dionis.

E.1.2. GCP-Tables Failures

GCP-Tables failures arose from service errors and explicit limits on dataset size, feature count, class probabilities, and label representation across splits.

  • GCP-Tables failed on 14 of 39 datasets.
  • A minimum requirement of 1000 training rows caused failures on five datasets.
  • A maximum of 1000 features caused failures on five datasets.
  • GCP-Tables returned prediction probabilities for only 40 classes on some multiclass problems exceeding 40 classes, causing failures on Helena and Dionis.
  • A missing label in a test split caused GCP-Tables to fail on Shuttle, apparently because its rarest class appeared only 9 times in training.
  • GCP-Tables also cryptically failed on KDDCup09 appetency despite training for the full 4h duration.

E.1.3. H2O AutoML Failures

H2O AutoML failed on 9 of 39 datasets in the 4-hour runs, due to server crashes, failure to produce models, and substantial violations of requested training limits.

  • 9 of 39 datasets caused H2O AutoML failures in the 4-hour runs.
  • 7 datasets experienced cryptic local-server deaths, often on larger datasets and sometimes only on a fraction of folds.
  • H2O’s failures illustrate both dataset-scale crashes and inconsistent time-limit compliance.
  • 1 dataset failed because H2O could not produce any model within the requested time.
  • 1 dataset caused H2O to train far beyond the requested 4 hours, reaching approximately 90,000 seconds on half its folds.

E.1.6. Auto-WEKA Failures

Auto-WEKA failed on 7 of 39 datasets, mainly because of crashes, memory errors, or failure to produce predictions. AutoPilot failed on 12 of 39 datasets, including cases involving minimum dataset size, incomplete training, label handling, and oversized data.

  • Auto-WEKA: 7 of 39 datasets caused Auto-WEKA failures during the 4-hour runs.The failures included hard crashes, memory-allocation errors, and failures to produce predictions.
  • Auto-WEKA: 6 datasets triggered Auto-WEKA hard crashes because memory was not safely handled in all instances.The affected datasets were adult, Airlines, Dionis, guiellermo, riccardo, and Robert.
  • Auto-WEKA: Covertype produced an Auto-WEKA memory error, with 5 of 10 folds succeeding despite fold-dependent errors.The reported allocation failure involved a 2.70 GiB array.
  • AutoPilot: 12 of 39 datasets caused AutoPilot failures.Five datasets had fewer than 1000 training rows, five failed to finish training models in time, and other failures involved label conversion or candidate generation.

E.2. Failures in Kaggle Benchmark

The Kaggle benchmark exposed failures and operational issues across several AutoML frameworks, while AutoGluon produced predictions on every dataset under all tested training-time limits. H2O and TPOT had failures that persisted under longer runs, whereas some errors were resolved by retries or preprocessing-related adjustments.

  • AutoGluon: AutoGluon exhibited no errors on any Kaggle dataset under any specified training-time limit.The failures listed in this benchmark prevented other frameworks from producing predictions for corresponding competitions.
  • H2O: H2O produced predictions for ieee-fraud-detection after retrying an initial Google API error.The initial failure was reported as GoogleAPICallError: None INTERNAL.
  • H2O: H2O failed to produce models for the walmart-recruiting-trip-type-classification data even after increasing the training time to 32 hours.H2O also failed on ieee-fraud-detection because of a data-format error and on santander-customer-transaction-prediction under a 4-hour limit.
  • TPOT: TPOT failed on ieee-fraud-detection because its pipeline was not optimized, and increasing its runtime to 32 hours did not resolve the issue.A separate missing predict proba() failure was circumvented by rerunning TPOT.
  • GCP-Tables: GCP-Tables could not produce a model for santander-value-prediction-challenge because the data contained 4992 columns, exceeding its 1000-column limit.In some competitions, missing predictions were filled with training-distribution or training-average dummy values.
  • Auto-WEKA: Auto-WEKA succeeded on walmart-recruiting-trip-type-classification after 24 hours, but its resulting performance was very poor under log-loss.It sometimes assigned zero probability to classes, requiring a small 1e-8 adjustment to obtain finite log-loss values.
Loading 2003.06505v1…