When Do Neural Nets Outperform Boosted Trees on Tabular Data?

Duncan McElfresh; Sujay Khandagale; Jonathan Valverde; Vishak Prasad C; Benjamin Feuer; Chinmay Hegde; Ganesh Ramakrishnan; Micah Goldblum; Colin White

Source-linked AI summary

When Do Neural Nets Outperform Boosted Trees on Tabular Data?

Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Benjamin Feuer, Chinmay Hegde, Ganesh Ramakrishnan, Micah Goldblum, Colin White

arXiv:2305.02997v4May 4, 2023cs.LGcs.AIstat.ML

TL;DR

The paper addresses limited and conflicting evidence about whether neural networks or GBDTs generally perform better on tabular data. It compares 19 algorithms across 176 datasets and finds the debate is often overemphasized, with dataset properties and modest tuning frequently more important than family choice, while TabPFN stands out.

Problem
Prior studies provide conflicting NN-versus-GBDT conclusions and commonly use fewer than 50 datasets or inadequately tuned baselines.
Method
The study compares 19 algorithms across 176 datasets, evaluates metafeatures, and uses up to 30 hyperparameter settings and 10 folds per dataset.
Results
The NN-versus-GBDT debate is overemphasized for many datasets, while TabPFN and CatBoost are the best-performing algorithms on average across 176 datasets.
Takeaways & Limitations
Dataset regularity can guide method choice: NNs favor more regular data, whereas GBDTs handle irregularities better and tend to perform better on larger datasets.
Takeaways & Limitations
TabPFN has quadratic runtime and memory scaling with training-sample size and is effectively limited to 3000 training points in this evaluation.

Abstract

from arXiv · show

Tabular data is one of the most commonly used types of data in machine learning. Despite recent advances in neural nets (NNs) for tabular data, there is still an active discussion on whether or not NNs generally outperform gradient-boosted decision trees (GBDTs) on tabular data, with several recent works arguing either that GBDTs consistently outperform NNs on tabular data, or vice versa. In this work, we take a step back and question the importance of this debate. To this end, we conduct the largest tabular data analysis to date, comparing 19 algorithms across 176 datasets, and we find that the 'NN vs. GBDT' debate is overemphasized: for a surprisingly high number of datasets, either the performance difference between GBDTs and NNs is negligible, or light hyperparameter tuning on a GBDT is more important than choosing between NNs and GBDTs. A remarkable exception is the recently-proposed prior-data fitted network, TabPFN: although it is effectively limited to training sets of size 3000, we find that it outperforms all other algorithms on average, even when randomly sampling 3000 training datapoints. Next, we analyze dozens of metafeatures to determine what properties of a dataset make NNs or GBDTs better-suited to perform well. For example, we find that GBDTs are much better than NNs at handling skewed or heavy-tailed feature distributions and other forms of dataset irregularities. Our insights act as a guide for practitioners to determine which techniques may work best on their dataset. Finally, with the goal of accelerating tabular data research, we release the TabZilla Benchmark Suite: a collection of the 36 'hardest' of the datasets we study. Our benchmark suite, codebase, and all raw results are available at https://github.com/naszilla/tabzilla.

1 Introduction

The paper argues that the NN-versus-GBDT debate is overemphasized and studies when dataset properties favor either family. Across a broad benchmark, it finds that simple baselines or light GBDT tuning often matter more than algorithm-family choice, while TabPFN is an important exception.

Study scope: 19 algorithms across 176 datasets reveal that the NN-versus-GBDT debate is often less consequential than assumed.The study uses up to 30 hyperparameter settings per algorithm and 10 folds per dataset.
Algorithm selection: For roughly one-third of datasets, light tuning of CatBoost or ResNet improves performance more than choosing among NNs and GBDTs.On many datasets, a strong simple baseline performs on par with top algorithms, or a well-tuned GBDT suffices.
Algorithm selection: TabPFN and CatBoost are the two best-performing algorithms over all 176 datasets.TabPFN remains strong on large datasets when training uses a random sample of 3000 points, despite quadratic runtime and memory scaling.
Metafeatures: Dataset regularity predicts NN advantages, while GBDTs tend to perform better on larger datasets and irregular data.Examples of irregularity include skewed and heavy-tailed feature distributions.
TabZilla: TabZilla releases 36 hard datasets selected because simple baselines or most algorithms fail to reach top performance.The suite is intended to accelerate tabular data research alongside the open-source codebase and raw results.

2 Analysis of Algorithms for Tabular Data

This large-scale analysis compares tabular-data algorithms across datasets and examines which dataset properties are associated with their relative performance. It finds that no single approach dominates, while tuning and dataset irregularity often matter more than choosing between neural networks and GBDTs.

Study design: 19 algorithms are evaluated on 176 OpenML classification datasets, with metafeatures extracted to study dataset properties associated with algorithm performance.The experiments trained 538,650 total models and excluded datasets larger than 1.1M because of runtime limits.
Relative algorithm performance: CatBoost achieved the best average rank at 5.06, yet nearly every algorithm ranked first on one dataset and last on another.This indicates substantial variation in which methods perform best across datasets.
Relative algorithm performance: TabPFN matched CatBoost’s performance on the broader evaluation and achieved the best average performance on the 57 smallest datasets.For large datasets, TabPFN used a random sample of 3000 training datapoints; its inference time was 2.36 seconds per 1000 instances.
Relative algorithm performance: GBDTs required less runtime than neural networks while maintaining strong performance, whereas neural networks generally required the longest runtime.TabPFN achieved top accuracy with less training time than any GBDT or neural network.
Metafeature analysis: GBDTs performed comparatively better on larger and more irregular datasets, including those with heavy-tailed, skewed, or high-variance feature distributions.The metafeature analyses associated dataset regularity with neural networks outperforming GBDTs, while GBDTs also favored higher class imbalance in some comparisons.
Practical guidance: The authors recommend first trying simple baselines and lightly tuning CatBoost, then using dataset metafeatures to guide trials of neural networks and other GBDTs.This recommendation follows the finding that tuning and dataset properties can be more consequential than broad NN-versus-GBDT selection.

3 TabZilla Benchmark Suite

The TabZilla Benchmark Suite is a collection of 36 hard datasets selected using criteria targeting baseline difficulty, broad algorithm difficulty, and GBDT weakness. The suite spans diverse dataset sizes and feature-irregularity characteristics, and its data and results are released to support research.

36 datasets comprise TabZilla, selected from the 176 datasets studied using three hardness criteria.The criteria target difficulty for baseline algorithms, most algorithms, and GBDTs.
Hard for baseline algorithms: Datasets are excluded when a baseline achieves normalized log loss within 20% of the top-performing algorithm.This criterion is intended as a proxy for dataset hardness, though it may miss datasets where all algorithms perform poorly.
Hard for most algorithms: A dataset is hard for most algorithms when its fourth-best log loss is at least 7% worse than the top log loss.This includes datasets where only one, two, or three algorithms stand out in performance.
Hard for GBDTs: Datasets where GBDTs perform 10% worse than the top-performing algorithm are added to cover cases where GBDTs are weak.This criterion is intended to prevent the benchmark from focusing only on datasets where GBDTs already perform well.
TabZilla Characteristics: TabZilla datasets range from 148 to over 1 million instances and span a large range of feature-kurtosis variance.Table 4 reports dataset statistics, hardness metrics, and the top three algorithms.
The authors release TabZilla through OpenML together with computed metafeatures, results, and dataset documentation to accelerate tabular-data research.The release includes comparisons of all algorithms on the benchmark suite.

4 Conclusions and Future Work

Across 19 approaches and 176 datasets, the study finds that the NN-versus-GBDT debate is overemphasized because simple baselines or light GBDT tuning often suffice, although GBDTs outperform NNs on average. The authors release TabZilla and supporting tools to expose dataset-dependent strengths and accelerate research.

The study compares 19 approaches across 176 datasets and finds that the NN-versus-GBDT debate is overemphasized.Many datasets are adequately handled by simple baselines or light hyperparameter tuning rather than extensive model selection.
GBDTs outperform NNs on average, while GBDTs are better than NNs at handling various types of data irregularity.The conclusion summarizes the broad comparison and metafeature analysis.
TabZilla contains 36 datasets selected as hard for baselines, most algorithms, or GBDTs, targeting blind spots in tabular-data research.The suite is intended to focus future progress on datasets where current methods have difficulty.
The open-source repository enables researchers to compare new neural nets with 19 algorithms across 176 datasets.The metafeature analysis can also guide efforts to improve weaknesses such as neural-net robustness to data irregularities.
Metafeatures can support dataset-specific model weighting and the design of new meta-learned or pretrained tabular models.The authors describe these as uses of the released datasets and metafeatures.
The authors identify extensions including regression, time-series forecasting, uncertainty quantification, categorical-feature effects, and more comprehensive hyperparameter optimization.These extensions are stated as future directions.

B Dataset Documentation

The documentation describes TabZilla’s dataset composition, selection process, access, and intended use, while noting that community adoption remains uncertain.

Dataset composition: Each instance is a tabular datapoint whose makeup depends on its dataset, including examples such as poker hands, electricity usage, and plant textures.
Access and preprocessing: The raw data are hosted on OpenML, while the repository provides scripts and documentation for standard preprocessing.
Dataset selection: TabZilla gathers 176 classification datasets from OpenML sources, then selects 36 datasets for its benchmark suite.The source collections include OpenML-CC18, the OpenML Benchmarking Suite, and additional OpenML datasets.
Data properties: The datasets contain no known errors, sources of noise, redundancies, missing information, or relationships between individual instances.
Motivation and scope: The suite is intended to accelerate tabular-data research, but the authors state that future community adoption will determine whether it achieves that goal.

C Additional Related Work

Related work covers GBDTs, neural networks, and TabPFN, emphasizing conflicting average-performance findings and differences in study scale and dataset coverage.

GBDTs: GBDTs iteratively add decision trees that fit residual losses, using gradient descent to minimize the loss.
Neural networks: Neural-network approaches for tabular data use data transformations, specialized architectures, or regularization, including transformer-based models.
TabPFN: TabPFN is a prior-data fitted, meta-learned network that approximates Bayesian inference and predicts on a new dataset in under one second.
GBDTs versus NNs: Prior comparisons disagree: some report better average performance for GBDTs, others for tuned neural networks, and some find no universal winner.
Positioning: Compared with a study using seven algorithms and 45 datasets, this work evaluates 19 algorithms across 176 datasets and a broader dataset-size range.

D Additional Experiments

Additional experiments examine performance rankings, tuning, runtime, algorithm families, and TabPFN variants across broad tabular-data evaluations, with completion caveats for some methods.

Relative performance: The experiments rank tuned algorithms over datasets using log loss, F1 score, and ROC-AUC, averaging performance across 10 splits before ranking.
Hyperparameter tuning: CatBoost, XGBoost, LightGBM, and ResNet perform fairly well both with and without hyperparameter tuning.
Evaluation caveats: The full 176-dataset comparisons include partial results for some algorithms because memory or timeout issues prevented completion on every dataset.
TabPFN ablations: TabPFN-3k and TabPFN-1k use randomly sampled training subsets of 3000 and 1000 points, while CatBoost-1k uses 1000 points.
Statistical comparisons: Critical-difference plots summarize whether algorithms differ significantly in average rank, with connected groups indicating algorithms that are not significantly different.
Algorithm families: Family-level analyses compare GBDTs, neural networks, and baselines using min-max-scaled test accuracy and a high-performance threshold of 0.99.

D.3.4 Dataset size analysis

Dataset size changes the relative suitability of individual algorithms and algorithm families, so dataset-specific metafeatures are more informative than choosing NNs or GBDTs categorically.

Dataset size: GBDTs perform relatively better than NNs and baselines as dataset size increases, based on negative correlations between normalized log loss and size.
Individual algorithms: For datasets up to 1 500 instances, CatBoost and SAINT have comparable performance; CatBoost outperforms SAINT on larger datasets.
Individual algorithms: TabPFN performs comparatively better on smaller datasets, whereas TabNet performs comparatively better on larger datasets.
Practical implication: The analysis recommends using metadataset results for specific use cases because general algorithm-family trends are helpful but insufficient for selection.

D.4 Additional experiments from Section 2.2

These experiments relate algorithm-performance differences to dataset metafeatures and test whether dataset properties can predict when neural networks outperform GBDTs.

Metafeature correlations: ∆ℓℓ measures the normalized log-loss difference between the best neural network and the best GBDT.Positive values indicate that the best neural network has higher normalized log loss than the best GBDT.
Predicting algorithm superiority: Decision trees predict whether the best neural network outperforms the best GBDT using a binary ∆ℓℓ-based outcome.The evaluation aggregates results over 176 train/test splits, holding out one dataset family for testing in each split.
Metafeature correlations: Table 16 identifies dataset properties with the largest absolute correlations with ∆ℓℓ across 176 datasets.Higher correlations associate larger metafeature values with worse neural-network and stronger GBDT performance.
Pairwise analysis: The pairwise metafeature analysis focuses on CatBoost, XGBoost, ResNet, and SAINT as the two strongest methods from each family.It examines metafeatures correlated with log-loss differences between algorithm pairs.

D.5 Experiments on Regression Datasets

The regression experiments extend the benchmark design to 17 datasets and evaluate 12 algorithms using test-set R-squared. Their conclusions broadly resemble the classification results, with GBDTs—especially CatBoost—performing particularly well.

Experimental setup: 17 regression datasets were evaluated with 12 algorithms using the same experiment design and parameters as the classification experiments.Each algorithm was tuned per dataset by maximizing the R-squared (R2) metric.
Results: GBDTs perform particularly well on regression datasets, especially CatBoost, while most algorithms perform well and poorly on at least one dataset.The authors describe these conclusions as similar to their classification findings.

D.6 Additional Results with Quantile Scaling

Additional experiments examine quantile scaling and expanded hyperparameter optimization. Quantile scaling mainly helps simpler algorithms, while further tuning improves the tested tree-based methods.

Quantile scaling: Quantile scaling improves decision tree, MLP, random forest, and SVM performance but has little effect on high-performing algorithms.The comparison evaluates algorithms on the TabZilla benchmark suite with and without quantile scaling.
Additional hyperparameter optimization: The main experiments used 30 random-search iterations for every algorithm, while additional tuning was tested only for four tree-based algorithms.Neural-network methods were excluded from additional HPO because of their substantial computational requirements.
Additional hyperparameter optimization: Additional hyperparameter tuning improves XGBoost, CatBoost, LightGBM, and RandomForest relative to default and 30-iteration search results.The additional experiments use 100 Optuna iterations on all 36 TabZilla benchmark datasets.

D.8 Forward Feature Selection for Identifying Important Dataset Attributes

Forward feature selection identifies dataset attributes that help predict performance differences between CatBoost and ResNet. The first selected attributes summarize feature distribution, sparsity, and aggregate variability.

Setup: The analysis predicts normalized log-loss differences between CatBoost and ResNet using dataset metafeatures.These methods represent effective GBDT and neural-network algorithms, respectively.
Selection procedure: Greedy forward feature selection adds metafeatures sequentially according to their improvement of a meta-model.Performance is evaluated with leave-one-dataset-out cross-validation across 10 folds per dataset.
Selection procedure: An XGB regressor first selects 200 candidate features, after which greedy forward selection uses another XGB regressor as the meta-model.The first five selected features are then reported in selection order.
Selected attributes: The first selected feature is the number of features classified as normally distributed by the Shapiro-Wilk test.The second is the median value of the minimum across all features.
Selected attributes: The remaining selected attributes summarize feature sparsity, the interquartile range of feature means, and the mean harmonic mean of features.These are selected third, fourth, and fifth, respectively.

E Additional Details from Section 3

The TabZilla benchmark compares algorithm performance across 36 hard datasets, but incomplete runs make the resulting comparisons only approximate. The associated tables report performance rankings, losses, training time, and metafeature correlations under specified evaluation settings.

Benchmark evaluation: Table 20 reports algorithm performance with default hyperparameters, random hyperparameter search, and 100 iterations of hyperparameter optimization.Algorithms are ranked by normalized log loss, with columns for rank, normalized log loss, and training time.
Metafeature analysis: Tables 21 and 22 list the ten dataset metafeatures most correlated with pairwise differences in normalized log loss.Correlations use ten splits across 133 datasets where CatBoost, XGBoost, ResNet, and SAINT ran successfully.
Benchmark limitations: Not all algorithms completed on all 36 benchmark datasets because of memory or timeout issues, so the comparisons are only rough.This caveat applies to the benchmark-suite comparisons.
Benchmark evaluation: Table 23 summarizes benchmark-suite results using ranks, mean and standard-deviation normalized log loss, and training time per 1000 instances.Minimum, maximum, mean, and median values are calculated across the 36 datasets.