Source-linked AI summary
A Comprehensive Benchmark of Machine and Deep Learning Across Diverse Tabular Datasets
Assaf Shmuel, Oren Glickman, Teddy Lazebnik
TL;DR
Tabular-data benchmarks have not clarified when deep learning surpasses traditional machine learning. This paper evaluates 111 datasets and 20 models across regression and classification, then profiles dataset characteristics to predict the better model group. The resulting predictor achieves 86.1% accuracy with AUC 0.78, while the benchmark finds ML models generally stronger on average.
Problem
The measurable dataset features indicating whether DL or ML will perform better on tabular data remain unknown.
Method
The study benchmarks 20 ML and DL models on 111 regression and classification datasets and trains a meta-learning predictor from dataset meta-features.
Results
ML models generally outperform DL models on tabular data, while a meta-learning model predicts DL-versus-ML superiority with 86.1% accuracy and AUC 0.78.
Takeaways & Limitations
Dataset profiling can identify conditions associated with DL advantage, including fewer rows, more columns, and larger kurtosis.
Takeaways & Limitations
Including diverse and small datasets can add noise and leave fewer instances for each dataset type.
Abstract
from arXiv · showhide
The analysis of tabular datasets is highly prevalent both in scientific research and real-world applications of Machine Learning (ML). Unlike many other ML tasks, Deep Learning (DL) models often do not outperform traditional methods in this area. Previous comparative benchmarks have shown that DL performance is frequently equivalent or even inferior to models such as Gradient Boosting Machines (GBMs). In this study, we introduce a comprehensive benchmark aimed at better characterizing the types of datasets where DL models excel. Although several important benchmarks for tabular datasets already exist, our contribution lies in the variety and depth of our comparison: we evaluate 111 datasets with 20 different models, including both regression and classification tasks. These datasets vary in scale and include both those with and without categorical variables. Importantly, our benchmark contains a sufficient number of datasets where DL models perform best, allowing for a thorough analysis of the conditions under which DL models excel. Building on the results of this benchmark, we train a model that predicts scenarios where DL models outperform alternative methods with 86.1% accuracy (AUC 0.78). We present insights derived from this characterization and compare these findings to previous benchmarks.
1 Introduction
This study addresses the limited understanding of when deep learning can outperform traditional machine learning on tabular data. It benchmarks diverse models and datasets, identifies favorable dataset characteristics, and predicts which model group will perform better.
- Prior benchmarks generally found DL equivalent or inferior to traditional ML, but did not establish the measurable dataset conditions predicting DL advantage.
- 111 datasets spanning regression and classification tasks were benchmarked with 20 model configurations, including DL, tree-based ensemble, and classical ML models.
- Small row counts, large column counts, and large kurtosis were associated with DL models outperforming ML models, while classification showed a smaller group gap than regression.
- 86.1% accuracy (AUC 0.78) was achieved by a meta-learning model predicting whether DL models would outperform ML models.
- The paper also presents explainable logistic and symbolic regression models for predicting when DL may perform better than alternatives.
2 Experimental setup
The experimental setup constructs a diverse tabular benchmark, evaluates 20 ML and DL models across regression and classification, and uses meta-learning to predict which model group performs best.
- 2.1 Datasets: The benchmark includes 111 datasets with varied sizes, domains, task types, categorical features, and difficulty levels.
- 2.2 Machine learning and deep learning models: 20 models represent tree-based ensembles, deep learning, classical ML, and AutoML approaches for comparative evaluation.
- 2.3 Evaluation strategy: Mean 10-fold cross-validation results are summarized using RMSE, MAE, and R2 for regression and accuracy, AUC, and F1 for classification.
- 2.4 Meta-analysis profiling: The profiling task is formulated as binary classification and solved through meta-learning that searches for a model minimizing the comparison loss.
- 2.4 Meta-analysis profiling: Each dataset is represented by 20 meta-features, including size, statistical properties, and feature-target relationships, while the target indicates whether ML or DL performs best.
3 Results
Across 111 datasets, tree-based ensemble and other ML models generally outperform DL models, although performance depends on dataset size and characteristics. Meta-analysis identifies task type, kurtosis, row count, and related dataset properties as informative for predicting when DL performs better.
- 3.1 Model Ranking: 19/111 (17.1%) datasets were won by CatBoost, while the first DL model won 11/111 (9.9%).ML models occupied the first four ranking positions, and this ordering was preserved by average and median rank.
- 3.1 Model Ranking: AutoGluon best performed on 39 of 111 datasets (35%), while SVM was second with 10 datasets (9%).AutoGluon was also the best-performing model on average; ML models occupied the top three positions most often.
- 3.1 Model Ranking: For datasets with fewer than 1000 rows, H2O led with 6/36 (16.6%) wins and ResNet followed with 5/36 (13.9%).ResNet nevertheless lagged behind other ML models on average rank, median rank, and top-three frequency, while performing similarly to MLP and AutoGluon-DL.
- 3.2 Meta-Analysis Profiling: DL models perform relatively better in classification than regression, and kurtosis is statistically significant in predicting whether DL outperforms tree-based ensembles.PCA components were also positive and almost statistically significant in the logistic regression analysis.
- 3.2 Meta-Analysis Profiling: Smaller row counts and larger kurtosis values increase the predicted probability that DL outperforms ML, while sample size alone does not determine the winner.After reducing 10 large datasets to 1000 samples, some DL rankings improved, but tree-based ensembles still dominated; heatmaps also examine rows, columns, feature counts, and PCA components.
4 Discussion
Across 111 datasets and 20 models, ML models generally outperform DL on tabular data, although DL performs relatively better under specific dataset and task conditions. Dataset profiling identifies classification and kurtosis as significant predictors, while the benchmark’s diversity and excluded task types limit generalization.
- Overall benchmark results: ML models, on average, outperform DL models on tabular data, with Tree-based Ensemble models consistently achieving the highest performance.This pattern holds for the best-performing model, mean rank, and top-three model rankings.
- Overall benchmark results: AutoGluon, which ensembles ML and DL models, outperforms the other models by a large margin.The result aligns with findings from.
- Dataset-size patterns: A smaller number of rows and larger number of columns are associated with a higher probability that DL models outperform ML models.The size-isolation analysis does not show clear results, so dataset size is only one of several relevant factors.
- Dataset profiling: Classification tasks and kurtosis are the only statistically significant dataset features in the logistic regression profile predicting whether DL or ML performs best.DL performs relatively better in classification than regression, while larger kurtosis indicates a long-tailed distribution associated with stronger DL performance.
- Limitations: The benchmark’s diverse datasets may introduce noise and reduce the number of examples for each dataset type.The study also excludes feature selection and engineering, time-series tasks, and multilabel classification.
5 Supplementary Material
The supplementary material describes the benchmark’s model implementations and supporting meta-learning resources. It covers automated pipeline search, classical and tree-based models, deep architectures, and the dataset meta-feature representation.
- Model descriptions: TPOT automates ML pipeline design and optimization through genetic programming, preprocessing exploration, model search, and hyperparameter optimization.The experiments used mostly default settings and limited each model run to one hour.
- Tree-based models: Tree-based models include XGBoost, Random Forest, AdaBoost, CatBoost, and Decision Trees, covering boosting, bagging, feature randomness, categorical handling, and recursive splitting.CatBoost directly handles categorical and numerical features, while Decision Trees split according to feature-based criteria.
- Additional models and meta-features: The supplementary resources include symbolic regression through GPLearn, SVM and KNN descriptions, linear and logistic regression, and a 20-feature meta-learning vector.Table 8 presents the dataset meta-feature representation.
- Deep-learning models: FT-Transformer uses feature tokenization and self-attention to model interactions among numerical and categorical tabular features.Its implementation tunes Transformer depth, embedding dimensions, dropout, hidden dimensions, learning rate, and batch size with Optuna.
- Deep-learning models: ResNet uses skip connections that improve gradient propagation and alleviate vanishing gradients in deep networks.The architecture stacks residual blocks to form the network.
- Model descriptions: AutoGluon automatically handles feature selection, algorithm selection, hyperparameter tuning, and model ensembling across several ML tasks.It is built on Apache MXNet and supports classification, regression, and object detection.