Source-linked AI summary
Why do tree-based models still outperform deep learning on tabular data?
Léo Grinsztajn, Edouard Oyallon, Gaël Varoquaux
TL;DR
The paper asks why deep learning has not clearly surpassed tree-based models on tabular data. It builds a systematic benchmark and empirically investigates the models’ differing inductive biases. Tree-based models remain state-of-the-art on medium-sized datasets, motivating robustness to uninformative features, preserved data orientation, and easier learning of irregular functions.
Problem
Deep learning’s superiority over tree-based models on tabular data remains unclear, amid inconsistent benchmarks and evaluation practices.
Method
The paper benchmarks deep and tree-based models across 45 datasets and hyperparameter budgets, then transforms data to investigate differing inductive biases.
Results
Tree-based models remain state-of-the-art on medium-sized tabular datasets, even without accounting for deep learning’s slower training.
Takeaways & Limitations
Tabular-specific neural networks should be robust to uninformative features, preserve data orientation, and learn irregular target functions easily.
Takeaways & Limitations
The study leaves open how its evaluation changes for very small or very large datasets and for missing or high-cardinality data.
Abstract
from arXiv · showhide
While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and novel deep learning methods as well as tree-based models such as XGBoost and Random Forests, across a large number of datasets and hyperparameter combinations. We define a standard set of 45 datasets from varied domains with clear characteristics of tabular data and a benchmarking methodology accounting for both fitting models and finding good hyperparameters. Results show that tree-based models remain state-of-the-art on medium-sized data ($\sim$10K samples) even without accounting for their superior speed. To understand this gap, we conduct an empirical investigation into the differing inductive biases of tree-based models and Neural Networks (NNs). This leads to a series of challenges which should guide researchers aiming to build tabular-specific NNs: 1. be robust to uninformative features, 2. preserve the orientation of the data, and 3. be able to easily learn irregular functions. To stimulate research on tabular architectures, we contribute a standard benchmark and raw data for baselines: every point of a 20 000 compute hours hyperparameter search for each learner.
1 Introduction
The paper addresses uncertainty about whether deep learning is superior for tabular data by introducing a systematic benchmark and investigating model inductive biases. It finds that tree-based models remain state-of-the-art on medium-sized tabular datasets and identifies biases relevant to tabular-specific neural networks.
- The benchmark addresses concerns about inconsistent datasets, unequal hyperparameter tuning, noisy evaluation, and statistical uncertainty.
- Tree-based models remain state-of-the-art on medium-sized tabular datasets, despite deep learning’s success on image, language, and audio data.
- Transforming datasets reveals that neural networks struggle with irregular target patterns and uninformative features, while rotation invariance can hurt performance.
- The study compares deep and tree-based models across representative datasets and multiple settings while accounting for hyperparameter-selection costs.
- The paper contributes a reusable benchmark, shared raw random-search results, and an empirical study of desirable tabular-learning biases.
2 Related work
Prior work explored encodings, hybrid models, transformers, and regularization for tabular deep learning, but comparisons lacked consistent evidence. The paper positions its benchmark as a response to missing standardization and limited understanding of why trees often outperform neural networks.
- Tabular deep-learning research includes feature encodings, hybrid methods, transformers, and regularization techniques.
- Earlier comparisons were often limited to older algorithms or specific problems, while recent evaluations questioned the generalization of new architectures.
- Tabular learning lacks a standard benchmark dedicated to representative tabular datasets.
- The paper presents its empirical investigation as the first study, to the authors’ knowledge, of why tree-based models outperform neural networks on tabular data.
- Prior work suggests that MLPs may be expressive enough for tabular data but can suffer from insufficiently appropriate regularization.
3 A benchmark for tabular learning
The benchmark uses 45 curated tabular datasets and a controlled evaluation procedure designed to reduce dataset, preprocessing, and hyperparameter-tuning variability. It measures performance across increasing random-search budgets and shares reusable raw results.
- 45 tabular datasets from varied domains form the benchmark, selected for heterogeneous, real-world, non-high-dimensional, independently distributed data.
- Datasets with too few samples or features, excessive ease, deterministic targets, or game-like tasks are excluded.
- The main benchmark truncates larger datasets to 10,000 training samples to study the medium-sized regime.
- The study removes missing data, balances classification classes, and restricts categorical and numerical feature cardinalities.
- Random searches estimate best validation-selected test performance across increasing evaluation budgets while repeating shuffled search orders 15 times.
- Code and every iteration of the 20,000 compute-hour random searches are shared to support inexpensive comparisons with new algorithms.
- Performance uses test accuracy or R2 and aggregates dataset results with a normalized distance-to-minimum-style metric.
- The experiments apply limited preprocessing, including Gaussianization, target transformations, and one-hot encoding when required.
4 Tree-based models still outperform deep learning on tabular data.
The benchmark compares practitioner-used tree ensembles with several neural architectures across numerical-only and mixed-feature tabular datasets. Tree-based models remain superior across random-search budgets, while categorical variables explain only part of the performance gap.
- The benchmark includes Random Forest, Gradient Boosting, HistGradientBoosting for categorical features, and XGBoost alongside MLP, ResNet, FT_Transformer, and SAINT.
- Figures 1 and 2 report test scores after increasing random-search iterations for numerical-only and mixed numerical-categorical datasets.
- Tree-based models remain superior for every random-search budget, and the performance gap stays wide after many iterations.
- The reported comparison does not account for neural networks generally requiring slower random-search iterations.
- Numerical-only experiments narrow the gap between trees and neural networks, but most of the gap remains.
5 Empirical investigation: why do tree-based models still outperform deep learning on tabular data?
The investigation compares tree-based models and neural networks through dataset transformations, finding that neural networks are disadvantaged by irregular target functions, uninformative features, and rotation invariance.
- 5.1 Methodology: uncovering inductive biases: The study transforms numerical classification datasets to widen or narrow the performance gap between neural networks and tree-based models.Experiments aggregate results across medium-sized datasets, with additional dataset-specific analyses in the appendix.
- 5.2 Finding 1: NNs are biased to overly smooth solutions: Small-scale target smoothing markedly reduces tree-based accuracy but barely affects neural networks, indicating that neural networks struggle more with irregular target functions.Decision trees learn piece-wise constant functions, whereas neural networks are biased toward lower-frequency functions; suitable regularization may mitigate this difficulty.
- 5.3 Finding 2: Uninformative features affect more MLP-like NNs: Removing up to half the least-important features barely affects GBT accuracy, while models trained only on those removed features perform very poorly through the 50% threshold.This pattern suggests that many removed features are uninformative rather than merely redundant.
- 5.3 Finding 2: Uninformative features affect more MLP-like NNs: Removing uninformative features narrows, while adding them widens, the performance gap between MLPs and FT Transformers or tree-based models.The results indicate that MLPs are less robust to uninformative features, which are common in tabular datasets.
- 5.4 Finding 3: Data are non invariant by rotation, so should be learning procedures: Random rotations reverse the performance order, placing neural networks above tree-based models, while only ResNets remain rotationally invariant.The results suggest that tabular data have a natural feature orientation whose individual feature meanings can be obscured by rotation.
- 5.4 Finding 3: Data are non invariant by rotation, so should be learning procedures: Embedding layers can improve tabular neural networks by breaking rotation invariance, and alternative less costly ways to break that invariance remain a research direction.The paper notes that improvements across different embedding types suggest the invariance-breaking property itself may matter.
6 Discussion and conclusion
The study identifies unresolved questions about how tree-based inductive biases explain tabular-data performance and how results may change across data regimes and missing-data settings. Its benchmark finds clear trends favoring tree-based models while providing a reusable basis for future comparisons.
- The study leaves open which additional tree-based inductive biases explain their performance on tabular data.
- The evaluation scope remains open for very small and very large datasets, as well as missing data and high-cardinality categorical features.
- A systematic benchmark accounting for hyperparameter choice reveals clear trends beyond results tied to a handful of datasets.
- Tree-based models more easily yield good predictions on tabular data, with much less computational cost.
- The reusable benchmark and shared raw results support evaluating and comparing new tabular architectures.
A.1 Datasets used
The dataset appendix provides links for the benchmark datasets and directs readers to usage instructions for benchmarking their own algorithms.
- The appendix describes datasets used in the benchmarks and links to their original datasets.
- It provides new OpenML links to the transformed datasets used for the benchmarks.
- Instructions for using the datasets to benchmark algorithms are available in Appendix A.7.
A.1.1 Numerical classification
The numerical-classification dataset table organizes benchmark entries by dataset identity, sample and feature counts, and source links, with one retained wine dataset falling just below the sample threshold.
- The table records each dataset name, number of samples, number of features, original link, and new link.
- The numerical-classification appendix includes links for both original and transformed datasets.
- The transformed wine dataset was retained despite falling just below the benchmark’s sample threshold.
A.1.2 Numerical regression
The numerical-regression dataset table uses columns for dataset identity, sample and feature counts, and original and transformed-data links.
- The table identifies each dataset by name.
- It records the number of samples and number of features for each dataset.
- It includes links to both the original and new transformed datasets.
A.1.3 Categorical classification
The section presents a dataset table organized by dataset name, feature count, sample count, and links to original and new versions.
- The table identifies each dataset by name.
- It reports the number of features and samples for each dataset.
- It includes links to both original and new dataset versions.
A.1.4 Categorical regression
The experiments compare model performance and search efficiency across train-set sizes and feature types, while examining irregular patterns and uninformative features. Results favor tree-based models, although larger datasets and the timing setup impose important qualifications.
- Results as a function of random search time: Time-based evaluation reports validation-selected test scores after random search, averaged over 15 search-order shuffles.The ribbon records the minimum and maximum scores across those shuffles.
- Results as a function of random search time: For equal random-search time, tree-based model scores remain well above neural-network scores.The advantage is more striking when results are plotted against elapsed search time rather than iteration count.
- Large-sized datasets: The large-scale comparison truncates training sets at 50,000 samples, versus 10,000 for the medium-sized benchmark.It retains only datasets with more than 50,000 samples, excluding many datasets from the comparison.
- Large-sized datasets: Increasing train-set size appears to reduce the performance gap between neural networks and tree-based models.The paper leaves rigorous study of this trend to future work.
- Irregular patterns: A RandomForest reaches 85% test accuracy versus 80% for a default MLP on the electricity dataset while learning irregular date-feature patterns.The RandomForest achieves perfect training accuracy in this example.
- Uninformative features: Adding uninformative features widens the performance gap for MLP-like models, while removing them reduces it.This supports lower robustness of MLPs to uninformative features in tabular datasets.
- Discussion: The observed tree-model advantage may depend on the search space because additional regularization could make MLPs competitive on some datasets.The cited comparison used different datasets, including deterministic game-inspired datasets.
- Contributions: The benchmark shares selected datasets, complete random-search results, and code to support inexpensive evaluation of new methods.