When Do Neural Nets Outperform Boosted Trees on Tabular Data?

• The study compares 19 algorithms with up to 30 hyperparameter settings across 176 datasets, using 10 folds and dozens of metafeatures to assess performance differences.Source: §1 · p. 2
• For roughly one-third of datasets, light CatBoost or ResNet tuning matters more than choosing between NNs and GBDTs, while simple baselines often perform on par with top algorithms.Source: §1 · p. 2
• Across 176 datasets, CatBoost and TabPFN were the two best-performing algorithms; TabPFN remained strong with 3,000 randomly selected training points despite quadratic runtime and memory scaling.Source: §1 · p. 2
• Dataset regularity predicts NN advantages over GBDTs, while GBDTs tend to perform better on larger datasets.Less-skewed, less-heavy-tailed feature distributions are examples of regularity associated with NN performance.Source: §1 · p. 2
• TabZilla releases 36 hard datasets selected where simple baselines do not win and most algorithms miss top performance, alongside code and raw results from over 500K trained models.Source: §1 · p. 2

2 Analysis of Algorithms for Tabular Data

• The analysis evaluates 19 algorithms on 176 classification datasets and relates algorithm performance to 965 metafeatures.The study uses OpenML train/test folds and validation-based hyperparameter tuning; metafeature analyses compare performance differences across families, pairs, and individual algorithms, with a leave-one-dataset-out meta-learning evaluation.Source: §2 · pp. 3–4, §2.2 · p. 8

2.1 Relative Algorithm Performance

• Nearly every algorithm ranks first on at least one dataset and last on another; CatBoost [54] has the best average rank, at 5.06, but does not dominate across datasets.GBDTs consistently outperform baselines while requiring less runtime than neural nets; neural nets often outperform baselines, and TabPFN [33] achieves top accuracy with less training time than any GBDT or neural net.Source: §2.1 · pp. 5–6, Figure 2 · p. 6
• On the 57 datasets of size at most 1250, TabPFN [33] has the best average performance and fastest training time, but its inference time is 2.36 seconds per 1000 instances.The small-dataset comparison reports training time; the stated inference time is higher than for other algorithms.Source: §2.1 · pp. 5–6, Table 2 · p. 5
• TabPFN outperforms all other algorithms on average across 98 datasets, with a statistically significant difference; its performance is nearly CatBoost’s [54] despite using a 3000-point training sample.TabPFN [33] trains in 0.25 seconds per 1000 instances, two orders of magnitude faster than CatBoost [54].Source: Table 1 · p. 4, §2.1 · pp. 5–6, Figure 3 · p. 6
• Light hyperparameter tuning of CatBoost [54] yields greater improvement than default-hyperparameter GBDT-versus-neural-net selection on about one-third of datasets.The comparison uses 30 random-search iterations on a validation set for CatBoost tuning.Source: §2.1 · p. 7, Figure 4 · p. 7

2.2 Metafeature Analysis

• GBDTs tend to outperform neural nets on larger datasets, datasets with high size-to-feature ratios, and irregular datasets with skewed or heavy-tailed feature distributions; results also vary by individual algorithm.GBDTs perform better on more class-imbalanced datasets, particularly relative to SAINT; TabPFN [33] does particularly well on smaller datasets, while TabNet does well on larger ones. The authors recommend trying simple baselines, then lightly tuning CatBoost [54], before using metafeatures to select further candidates.Source: §2.2 · pp. 8–9

3 TabZilla Benchmark Suite

• TabZilla comprises 36 of the 176 studied datasets, is released as an OpenML collection, and includes open-source computed metafeatures and results.The paper also provides full dataset documentation, including a datasheet.Source: §3 · pp. 9–10
• Datasets are selected as hard for baselines when no baseline achieves normalized log loss within 20% of the top-performing algorithm.The authors note this proxy can exclude datasets where all 19 algorithms fail to achieve non-trivial performance.Source: §3 · p. 9
• A dataset is hard for most algorithms when the fourth-best of 19 log losses is at least 7% worse than the top log loss.This criterion includes datasets where only one, two, or three algorithms stand out, and subsumes most datasets selected by the baseline-hardness criterion.Source: §3 · p. 9
• To broaden diversity beyond datasets where GBDTs perform well, the suite also includes datasets where GBDTs perform 10% worse than the top-performing algorithm.The first two criteria primarily select datasets where GBDTs are top-performing methods.Source: §3 · p. 9
• The selected datasets vary in size from 148 to over 1 million and span a large range in the variance of feature kurtosis, a measure of irregularity.The paper reports dataset statistics and top three algorithms for the suite.Source: §3 · p. 10

4 Conclusions and Future Work

• Across 19 approaches and 176 datasets, simple baselines often performed on par with other methods or light GBDT tuning beat algorithm choice, while GBDTs outperformed NNs on average.The authors conclude that the ‘NN vs. GBDT’ debate is overemphasized.Source: §4 · p. 10
• GBDTs handled data irregularities better than NNs, and the analysis identifies irregularity as a dataset property relevant to model suitability.Source: §4 · p. 10
• TabZilla packages the 36 hardest of the 176 studied datasets to focus research on current blind spots in the literature.The datasets are described as hard for baselines, most algorithms, and GBDTs.Source: §4 · p. 10
• The open-source repository lets researchers compare new tabular neural nets with 19 algorithms across 176 datasets, while metafeature analysis can guide efforts to improve algorithm weaknesses.The authors identify making neural nets more robust to irregular data as a natural next step.Source: §4 · p. 10
• Ensemble researchers can weight models differently by dataset using metafeatures.Source: §4 · p. 11
• The open datasets and metafeatures can support meta-learned [33] and pre-trained tabular models [32] [44] [71], with future extensions spanning regression, forecasting, uncertainty, categorical-feature effects, and broader hyperparameter optimization.Source: §4 · p. 11

A Broader Societal Impact Statement

• The authors foresee no negative broader societal impacts and note that many datasets need no resource-intensive neural net; larger, higher size-to-feature-ratio, and irregular datasets are predicted to suit GBDTs better.A simple baseline or CatBoost tuning can be enough to reach top performance on a surprisingly large fraction of datasets.Source: A Broader Societal Impact Statement · p. 16
• The authors hope their large analysis, open-source codebase, and hard-dataset benchmark will accelerate research and make future comparisons more rigorous and comprehensive.Source: A Broader Societal Impact Statement · p. 16

B Dataset Documentation / B.3 Code of conduct / B.4 Datasheet

• The datasheet and repository document the suite; it is limited to tabular classification, and its creation involved researchers at seven institutions with dataset-computation funding from Abacus.AI.The repository uses the Contributor Covenant version 2.0 as its Code of Conduct.Source: B Dataset Documentation · p. 16, §B.3 · p. 16, §B.4 · p. 16

Dataset Maintenance

• The authors plan to maintain the suite, welcome community contributions, and invite GitHub pull requests for extensions, which are reviewed case by case and announced in the README.New algorithms may be proposed for inclusion, and the README will also communicate updates and obsolescence.Source: §B.2 · p. 16, *Will the dataset be updated? If so,… · p. 20, *If the dataset becomes obsolete how… · p. 20

Motivation For Datasheet Creation

• The suite aims to accelerate tabular research with hard datasets, since simple baselines can match leading methods on many datasets while most tested algorithms cannot reach top performance.The authors gathered existing datasets already released on OpenML into one suite.Source: Motivation For Datasheet Creation · p. 16

Datasheet Composition / Data Preprocessing

• Each instance is a tabular datapoint; the collection is for classification, contains no people, and has no missing information or relationships between instances.Raw data are hosted on OpenML, and the repository includes preprocessing scripts; the authors also provide raw and preprocessed data, imputing NaNs with feature means and leaving other preprocessing to algorithms.Source: Datasheet Composition · p. 17, *What (other) tasks could the dataset… · p. 17, *How many instances are there in total… · p. 17

Collection Process

• From 176 datasets, the authors selected 36 for TabZilla, drawing on widely used OpenML collections and limiting dataset size to under 1.1M amid 538,650 model trainings.The initial selection aimed to include classification datasets from recent tabular-data papers [8] [27] [39], the OpenML-CC18 suite [7], the OpenML Benchmarking Suite [26], and other OpenML datasets.Source: *Does the dataset contain all possible… · p. 18, Collection Process · p. 18
• The authors selected rather than created the individual datasets, using criteria from Section 3; they recommend reporting performance averaged over OpenML’s 10 folds and using their validation splits when needed.The suite was constructed from April 15 to June 1, 2023.Source: Collection Process · pp. 18–19, *Are there recommended data splits… · p. 18

Dataset Distribution / Dataset Maintenance

• The benchmark suite was public as of June 1, 2023, distributed under Apache License 2.0, with no data copyrights, fees, or access/export restrictions; the authors accept responsibility for rights violations.The authors support, host, and maintain the dataset; a datasheet is included for TabZilla.Source: §B.1 · p. 16, §B.4 · p. 16, *When will the dataset be… · pp. 19–20

Legal and Ethical Considerations

• No ethical review was conducted because the suite comprises existing public OpenML datasets; the datasets contain no confidential or offensive material and do not relate to people.Source: Legal and Ethical Considerations · pp. 20–21

Gradient-boosted decision trees

• GBDTs iteratively fit residuals with new trees; prominent variants include XGBoost [13], LightGBM, and CatBoost, which add scaling, efficiency, and categorical or missing-value handling techniques.The passage describes XGBoost’s weighted quantile sketching and sparsity-awareness, LightGBM’s one-sided sampling and feature bundling, and CatBoost’s ordered boosting and improved handling of missing values and outliers.Source: C Additional Related Work · p. 21

Neural networks for tabular data

• Neural approaches for tabular data include data transformations, specialized architectures [12] [29] [53], and tailored regularization [37] [39].Examples include transformer architectures [27] [36] [62], latent-unit-attribution regularization [37], and a strong simple network found by searching combinations of 13 regularization techniques; the authors identify regularization as a direction for future work.Source: C Additional Related Work · p. 21
• TabPFN [33], a prior-data fitted network [50] [51], meta-learns an approximation to Bayesian inference and predicts on a new dataset in under one second.PFNs have also been applied to learning-curve extrapolation [1], forecasting, and Bayesian optimization [49].Source: C Additional Related Work · p. 21

GBDTs versus NNs

• Prior GBDT–NN comparisons report mixed winners: Shwartz-Ziv and Armon favor GBDTs, Kadra et al. [39] favor tuned NNs, and Gorishniy et al. [27] find no universal winner.Borisov et al. [8] find GBDTs have the edge, while Levin et al. [43] find NNs have a decisive edge when pre-training data is available; Shwartz-Ziv and Armon also report that ensembling both performs better.Source: C Additional Related Work · p. 21
• Compared with Grinsztajn et al., this study evaluates 19 algorithms on 176 datasets spanning 32–1,025,009 instances and more metafeatures, including irregularity.Their study used seven algorithms and 45 datasets, with sizes mostly 3,000–10,000 and seven datasets of exactly 50,000; its controls excluded some high-cardinality categorical and low-cardinality numerical features. The authors say those choices miss findings such as GBDTs outperforming NNs on irregular datasets.Source: C Additional Related Work · pp. 21–22

D.1 Dataset statistics

• Roughly half of the 176 datasets have binary targets, and roughly half of training sets contain fewer than 2,300 instances.Some datasets have as many as 100 target classes, while others contain tens of thousands of instances.Source: §D.1 · p. 22

D.2 Additional experiments from Section 2.1

• The appendix also presents relative-performance rankings, training-time analyses, and critical-difference diagrams as extensions of Section 2.1.The supplementary rankings average tuned performance across ten splits before ranking algorithms by dataset.Source: §D.2 · p. 22, §D.2.1 · p. 22

D.2.1 Relative performance tables

• Additional rankings cover log loss, F1, and ROC-AUC, while default-hyperparameter results show CatBoost, XGBoost, LightGBM, and ResNet perform fairly well without tuning.The full 176-dataset comparison has incomplete runs for some algorithms, but its ordering is nearly the same as Table 1.Source: §D.2.1 · pp. 22, 24, Table 6 · p. 23, Table 7 · p. 23

D.2.2 Training time analysis

• Default-hyperparameter training-time rankings cover 176 datasets, with a separate uncaveated comparison for datasets of at most 1,250 instances.The 176-dataset results are rough because some algorithms, especially NAM, DeepFM, and TabTransformer, did not complete across all datasets.Source: §D.2.2 · p. 24, Table 11 · p. 27, Table 12 · p. 29

D.3 Additional TabPFN Results

• The TabPFN and CatBoost ablations compare randomly sampled 3,000-point and 1,000-point TabPFN variants and a 1,000-point CatBoost variant across 98 datasets.The appendix reports accuracy, log-loss, and F1 rankings, as well as accuracy critical-difference plots and accuracy-versus-time plots.Source: §D.3 · p. 24

D.3.1 HPO Plot

• Figure 8 compares the benefit of tuning CatBoost and ResNet with the default-hyperparameter performance gap between the best NN and best GBDT.Both axes use normalized log loss, and each point represents a dataset.Source: §D.3.1 · p. 27, Figure 8 · p. 28

D.3.2 Critical difference diagrams

• The appendix adds an F1 critical-difference diagram and family-level comparisons on all 176 datasets for log loss and F1; the repeated Friedman tests have uncorrected p-values below 10^-20.The tests repeat rankings of the same datasets and algorithms four times.Source: §D.3.2 · p. 27, Figure 9 · p. 28, Figure 10 · p. 28

D.3.3 Venn Diagrams

• At a 0.9999 scaled-accuracy threshold, GBDTs alone are high-performing on 42% of datasets and NNs alone on 30%, but the difference may be practically insignificant.The analysis covers 176 datasets; the reported difference is smaller than 0.1%.Source: §D.3.3 · p. 27, Figure 12 · p. 29

D.3.4 Dataset size analysis

• CatBoost outperforms TabNet on datasets up to roughly 1,500 instances, while their performance is similar on larger datasets.The passage suggests choosing CatBoost over TabNet for smaller datasets and treating them as comparable for larger ones.Source: §D.3.4 · p. 29, Figure 13 · p. 30
• CatBoost and SAINT perform comparably up to datasets of size 1,500; CatBoost outperforms SAINT on larger datasets.The passage suggests choosing CatBoost over SAINT for very large datasets, while the algorithms are comparable on small datasets.Source: Figure 13 · p. 30, §D.3.4 · p. 30
• Rather than choosing an algorithm family, practitioners should consult the metadataset for their use case: TabPFN does better on smaller datasets, while TabNet does better on larger ones.The passage cautions that general trends are helpful but insufficient for selecting an effective algorithm.Source: §D.3.4 · p. 30

D.4 Additional experiments from Section 2.2

• Decision trees predict whether the best NN beats the best GBDT from dataset properties, with accuracy assessed across tree depths and against XGBoost.A depth-3 tree also depicts which of the top five algorithms performs best, using splits that maximize information gain.Source: §D.4 · p. 31
• Across 176 datasets, larger values of selected metafeatures correlate with worse NN performance and stronger GBDT performance on normalized log loss.The analysis selects properties with the largest absolute correlation with ∆ℓℓ, the difference between the best NN and best GBDT.Source: §D.4 · p. 31, Table 16 · p. 32
• Metafeatures are also analyzed for correlation with log-loss differences among CatBoost, XGBoost, ResNet, and SAINT.The analysis reports statistical metafeatures and general metafeatures separately.Source: §D.4 · p. 32

D.5 Experiments on Regression Datasets

• The experiments evaluated 12 algorithms on 17 tabular regression datasets using the classification experiments’ design and parameters, tuning each algorithm per dataset to maximize R2.Regression targets are continuous rather than categorical or binary.Source: §D.5 · p. 32
• Each regression dataset corresponds to an OpenML task and can be preprocessed like the classification datasets; the datasets have also appeared in recent tabular machine-learning studies [27] [36] [53] [59].Source: §D.5 · p. 32
• GBDTs performed particularly well on regression datasets, especially CatBoost, although most algorithms performed well and poorly on at least one dataset.The comparison ranked 12 algorithms across 17 datasets using test-set R2; specific ranking values are not provided in the passage.Source: §D.5 · p. 33

D.6 Additional Results with Quantile Scaling

• The quantile-scaling comparison covered 13 algorithms on TabZilla datasets, scaling all continuous features with scikit-learn’s QuantileTransformer [52].The experiments used the same computational setup and design as the main experiments.Source: §D.6 · p. 33
• The main results used consistent, lightweight preprocessing—imputing NaN values—to reduce confounding from differences in preprocessing.Source: §D.6 · p. 33
• Quantile scaling improved decision tree, MLP, random forest, and SVM, while having little effect on high-performing algorithms.Source: §D.6 · p. 34

D.7 Experiments with Additional Hyperparameter Optimization

• Additional HPO covered four algorithms after the main 30-iteration random search; neural-net methods were excluded because of substantial compute requirements.Source: §D.7 · p. 34
• The additional HPO used 100 Optuna [3] iterations per algorithm to optimize log loss across all 36 TabZilla benchmark datasets.Source: §D.7 · p. 34
• Additional hyperparameter tuning improved XGBoost, CatBoost, LightGBM, and RandomForest.The reported comparison includes 100-iteration HPO, default hyperparameters, and the main 30-iteration random-search results.Source: §D.7 · p. 34

D.8 Forward Feature Selection for Identifying Important Dataset Attributes

• The experiment used greedy forward feature selection to predict the normalized log-loss difference between CatBoost and ResNet from metafeatures.Source: §D.8 · p. 34
• A preliminary XGB regressor selected 200 candidate features, then forward selection used an XGB meta-model evaluated with leave-one-dataset-out cross-validation.Each dataset contributed 10 folds, with 10 instances for validation in each fold and the remaining instances for training.Source: §D.8 · p. 34
• The first two selected metafeatures were the number of features classified as normally distributed by the Shapiro-Wilk test and the median of feature minima.Source: §D.8 · p. 34
• The next three selected metafeatures were median feature sparsity, the interquartile range of feature means, and the mean of feature harmonic means.Source: §D.8 · p. 35

E Additional Details from Section 3

• Table 20 compares default hyperparameters, 30-iteration random search, and 100-iteration HPO; within-algorithm comparisons use the same datasets.Algorithms are ranked by normalized log loss, with training time also reported; incomplete runs mean overall comparisons are rough.Source: Table 20 · p. 36
• Tables 21 and 22 list the ten metafeatures most correlated with pairwise normalized-log-loss differences, using 10 splits across 133 datasets where four algorithms ran successfully.The correlations compare CatBoost, XGBoost, ResNet, and SAINT; attributes follow the PyMFE naming convention.Source: Table 21 · p. 37, Table 22 · p. 38
• Table 23 compares algorithm rankings, mean and standard deviation of normalized log loss, and training time, but incomplete runs make these rough comparisons.Some algorithms did not complete all 36 datasets because of memory or timeout issues.Source: E Additional Details from Section 3 · p. 35, Table 23 · p. 39