PaperlayerView exact revision on arXiv

Source-linked AI summary

Tabular Data: Is Deep Learning all you need?

Guri Zabërgja, Arlind Kadra, Christian M. M. Frey, Josif Grabocka

arXiv:2402.03970v3cs.LGcs.AI

TL;DR

The paper asks whether recent deep-learning paradigms now outperform classical methods on tabular data, addressing gaps in prior evaluations. It benchmarks 17 methods under a broadly tuned protocol across 68 datasets and finds that deep learning outperforms classical approaches in the considered benchmark.

  • Problem

    Prior evidence favored gradient-boosted trees but did not comprehensively evaluate recent foundation models or neural networks under equally thorough hyperparameter optimization.

  • Method

    The study benchmarks 17 neural, classical ML, and AutoML methods on 68 diverse datasets using fair hyperparameter tuning, 10-fold evaluation, and post-optimization refitting.

  • Results

    Deep learning methods outperform traditional baselines across all dataset regimes in the considered benchmark.

  • Takeaways & Limitations

    The results indicate a paradigm shift in tabular modeling, with deep learning methods achieving state-of-the-art performance against classical approaches.

  • Takeaways & Limitations

    Earlier empirical surveys omitted meta-learned neural networks and used insufficient hyperparameter optimization, motivating the paper’s revised comparison.

Abstract

from arXiv · show

Tabular data represent one of the most prevalent data formats in applied machine learning, largely because they accommodate a broad spectrum of real-world problems. Existing literature has studied many of the shortcomings of neural architectures on tabular data and has repeatedly confirmed the scalability and robustness of gradient-boosted decision trees across varied datasets. However, recent deep learning models have not been subjected to a comprehensive evaluation under conditions that allow for a fair comparison with existing classical approaches. This situation motivates an investigation into whether recent deep-learning paradigms outperform classical ML methods on tabular data. Our survey fills this gap by benchmarking seventeen state-of-the-art methods, spanning neural networks, classical ML and AutoML techniques. Our empirical results over 68 diverse datasets from a well-established benchmark indicate a paradigm shift, where Deep Learning methods outperform classical approaches.

1 INTRODUCTION

The paper addresses whether recent deep-learning methods now outperform gradient-boosted decision trees on tabular data, amid conflicting empirical evidence. It introduces a large, fairly tuned benchmark and reports a shift toward deep learning methods outperforming classical approaches.

  • Recent studies report that tree-based models outperform deep learning on tabular data, leaving the state-of-the-art question unresolved.
  • Prior surveys omit meta-learned neural networks, foundation models, and LLM-based tabular methods, while using insufficient neural-network hyperparameter optimization.
  • The study benchmarks 17 recent baselines on 68 diverse classification datasets using 10-fold cross-validation and equally large hyperparameter-optimization budgets.
  • Refitting every model on the combined training and validation data after hyperparameter optimization can improve predictive performance and change model rankings.
  • Deep learning methods achieve state-of-the-art results and outperform classical approaches in the study’s benchmark.

2 RELATED WORK

Related work spans classical gradient-boosted trees, neural and transformer architectures, emerging tabular foundation models, and empirical comparisons of their strengths. The paper positions its benchmark as broader and more methodologically rigorous than recent surveys.

  • Classical Machine Learning: Gradient-boosted decision trees remain widely favored for robust tabular performance and short training times.
  • Deep Learning: Neural-network research has improved tabular modeling through regularization, numerical embeddings, robust preprocessing, and transformer-based attention.
  • Foundation Models: Tabular foundation models use cross-table pretraining or in-context learning, expanding beyond conventional dataset-specific neural networks.
  • Empirical Studies: Recent empirical studies generally favor tree-based models but omit newer meta-learned methods and often use limited tuning budgets.
  • Empirical Studies: Unlike prior work, this study uses guided TPE search, up to 100 configurations, and refitting on combined training and validation data before testing.

3 EXPERIMENTAL PROTOCOL

The protocol evaluates binary and multi-class tabular classification with consistent preprocessing, nested cross-validation, guided hyperparameter optimization, and post-optimization refitting. Performance is measured by average ROC-AUC across outer test folds on a filtered OpenML benchmark.

  • Datasets: The evaluation covers binary and multi-class classification on OpenMLCC18, using 68 of its 72 datasets after excluding four for memory constraints.
  • Preprocessing: The pipeline applies quantile transformation and ordinal encoding whenever compatible, while preserving specialized preprocessing for methods requiring it.
  • Evaluation Protocol: Nested cross-validation partitions data into 10 folds, tunes configurations within nine folds, and evaluates performance on held-out outer test folds.
  • Evaluation Protocol: Optuna’s TPE algorithm searches official method-specific spaces for up to 100 trials or 23 hours, then models are trained on combined training and validation splits.
  • Metrics: Average ROC-AUC across 10 outer test folds is the primary metric because the benchmark contains many imbalanced datasets.

4 BASELINES

The benchmark compares classical GBDTs, deep neural architectures, tabular foundation models, and AutoML using official implementations. Its taxonomy distinguishes model families and includes both in-context and fine-tuned foundation-model paradigms.

  • The study compares Classical Machine Learning, Deep Learning, and AutoML groups, organized in a taxonomy of tabular-classification algorithms.
  • Classical Machine Learning Classifiers: Classical baselines are XGBoost, CatBoost, and LightGBM, representing asymmetric, oblivious, and leaf-wise gradient-boosted tree implementations.
  • Deep Learning Methods: Deep-learning baselines include ResNet, enhanced MLP variants, TabM, TabNet, SAINT, and FT-Transformer.
  • Foundation Models for Tabular Classification: Foundation-model baselines include in-context learners TabPFN, TabPFNv2, and TabICL, plus fine-tuned models XTab, TP-BERTa, and CARTE.
  • Foundation Models for Tabular Classification: Fine-tuned foundation models were checked for dataset overlap with OpenMLCC18, and AutoGluon was selected as the AutoML representative.

5 EXPERIMENTS AND RESULTS

Across 68 diverse datasets, Deep Learning methods outperform gradient-boosted and other classical approaches, with meta-learned models and simple feed-forward networks leading overall. The experiments also show that refitting after hyperparameter optimization improves predictive quality and can change model rankings.

  • TabICL achieves the best overall median rank of 2, followed by AutoGluon and TabPFNv2 at 3, while TabM, CatBoost, XGBoost, and LightGBM rank lower.TabM has median rank 5; CatBoost, XGBoost, and LightGBM have median ranks 5.5, 7, and 7.5, respectively.
  • Both TabPFNv2 and TabICL outperform tree-based architectures on most datasets, while TabM is the only non-meta-learned model to outperform every tree-based variant.
  • Deep Learning methods win 31–3 against tree-based methods below 5000 examples and still lead 17–7 above 5000 examples, where trees become more competitive.The dataset landscape places datasets by examples and features on log-scaled axes and colors each point by the winning method family.
  • TabICL and TabPFNv2 significantly outperform fine-tuning methods XTab and CARTE, although TP-BERTa and TabPFN are excluded because they share too few datasets.The comparison uses rank distributions and critical-difference diagrams.
  • Refitting after hyperparameter optimization lowers ranks for all evaluated methods and changes joint rankings, with XGBoost surpassing FT-Transformer after refitting.The evaluated methods are FT-Transformer, TabM, CatBoost, and XGBoost; improvements are statistically significant in most isolated comparisons.
  • The benchmark also examines hyperparameter optimization, foundation-model comparisons, transfer-learning paradigms, and additional predictive-performance analyses across data regimes and efficiency trade-offs.Raw results with optimized and default configurations are provided in Appendix D, while further analyses are referenced in Appendices G–I.

6 CONCLUSION

The study evaluates 17 tabular classification approaches on 68 diverse OpenMLCC18 datasets using cross-validation, model-based HPO, and refitting. Its results indicate that deep learning methods outperform traditional baselines across all considered dataset regimes.

  • 6 CONCLUSION: The study combines cross-validation, model-based hyperparameter optimization, and refitting to evaluate 17 tabular classification approaches across 68 diverse datasets.The benchmark uses the OpenMLCC18 classification benchmark.
  • 6 CONCLUSION: Deep learning methods outperform traditional baselines across all dataset regimes in the considered benchmark.The benchmark evaluates 17 state-of-the-art approaches across 68 diverse classification datasets.
  • 6 CONCLUSION: The paper analyzes refitting, hyperparameter importance, HPO effects, and cost-performance efficiency alongside its model-family comparison.These analyses are presented as additional contributions supporting future research on tabular modeling.

B HYPERPARAMETER ANALYSIS

The section analyzes which hyperparameters influence each method and how HPO changes comparative performance. HPO generally improves average ranks, although TP-BERTa is an exception because computational demands prevented completion of all trials.

  • B HYPERPARAMETER ANALYSIS: Learning rate is most important for CatBoost, XTab, and MLP-PLR, whereas XGBoost is most sensitive to the training-instance subsample ratio.For XTab, light finetuning is the next most influential categorical hyperparameter.
  • B HYPERPARAMETER ANALYSIS: The hyperparameter figures plot each method’s hyperparameter values against corresponding model performance.The analysis excludes categorical hyperparameters because fANOVA does not support them.
  • B HYPERPARAMETER ANALYSIS: HPO generally improves model performance and average ranks, with XGBoost improving from rank 1.94 by default to 1.06 after HPO.TP-BERTa is the exception: its default average rank is 1.47 versus 1.53 with HPO.

C ABLATING THE CHOICE OF REFITTING

The refitting ablation compares models retrained on combined training and validation data against no-refitting variants after HPO. Refitting usually improves performance, significantly for CatBoost, XGBoost, and TabM, but not consistently for FT-Transformer.

  • C ABLATING THE CHOICE OF REFITTING: Refitting changes joint model rankings; after refitting, XGBoost surpasses FT-Transformer and achieves a better median rank than its non-refitted counterpart.The ablation evaluates four methods because of computational-resource constraints.
  • C ABLATING THE CHOICE OF REFITTING: Refitting improves average performance by 0.0144 for XGBoost and 0.0079 for CatBoost, with statistically significant Wilcoxon p-values of 4.4130 · 10^-10 and 1.2985 · 10^-9.The dataset-level comparisons show refitted CatBoost and XGBoost outperforming non-refitted versions on most datasets.
  • C ABLATING THE CHOICE OF REFITTING: TabM gains 0.0056 on average from refitting, with p = 3.3952 · 10^-10, while FT-Transformer gains 0.0035 without a statistically significant effect.FT-Transformer has more datasets favoring no refitting, although refitting benefits the majority overall.

D.1 RESULTS AFTER HYPERPARAMETER OPTIMIZATION

This section reports average test ROC-AUC per dataset after HPO for classical models, dataset-specific and meta-learned neural networks, and AutoGluon. Parallel tables provide default-configuration results for the same model groups.

  • D.1 RESULTS AFTER HYPERPARAMETER OPTIMIZATION: After HPO, the reported results cover classical models, dataset-specific neural networks, meta-learned neural networks, and AutoGluon across datasets.The tables report average test ROC-AUC per dataset across cross-validation folds.
  • D.1 RESULTS AFTER HYPERPARAMETER OPTIMIZATION: Default-configuration results are separately reported for classical models, dataset-specific neural networks, meta-learned neural networks, and AutoGluon.The default tables likewise report average test ROC-AUC per dataset across cross-validation folds.

E DATASETS

The study uses the OpenML-CC18 tabular benchmark, reducing its 72 datasets to a default pool of 68 after excluding four very large image datasets.

  • E DATASETS: 68 datasets form the default evaluation pool after four OpenML-CC18 image datasets are excluded for memory reasons.Some methods cover fewer datasets because of memory limits, method-specific constraints, or pretraining overlap.

F COMPARISON WITH AUTOML METHODS

The study compares deep-learning methods with AutoGluon using both HPO-tuned and recommended configurations across dataset regimes. Overall, TabICL and TabPFNv2 lead the comparison, while AutoGluon becomes strongest in the small-data recommended-configuration setting without statistically significant differences.

  • F COMPARISON WITH AUTOML METHODS: AutoGluon is evaluated both with hyperparameter optimization and with its officially recommended best-quality configuration.Both versions are compared with methods in the Deep Learning family.
  • F COMPARISON WITH AUTOML METHODS: TabICL and TabPFNv2 achieve the best overall average rank against HPO-tuned AutoGluon, significantly outperforming the remaining methods.Their differences from TabM are not statistically significant.
  • F COMPARISON WITH AUTOML METHODS: TabICL ranks first against recommended AutoGluon, followed closely by TabPFNv2 and AutoGluon, with all three significantly better than the other deep-learning baselines.TabPFNv2 matches AutoGluon in mean and median rank, while TabICL has the better median rank.
  • F COMPARISON WITH AUTOML METHODS: In the small-data domain, recommended AutoGluon ranks highest, surpassing TabICL and TabPFNv2, although the differences are not statistically significant.With HPO, TabPFNv2 and TabICL lead and outperform AutoGluon.
  • F COMPARISON WITH AUTOML METHODS: The analysis groups datasets by instance and feature counts and uses boxplots or critical-difference diagrams only when a group contains at least 10 datasets.Smaller groups are reported in tables instead.

G.1 DATASETS WITH FEWER THAN 1000 INSTANCES

Across small and moderately small datasets, in-context and meta-learned methods are highly competitive with or ahead of classical baselines, though AutoGluon leads the smallest-data boxplot and coverage varies by method.

  • G.1 DATASETS WITH FEWER THAN 1000 INSTANCES: The small-data analysis covers 18 datasets, but TabPFN and TabPFNv2 exclude vowel, XTab excludes two pretraining datasets, and TP-BERTa lacks results for 10 datasets.The exclusions reflect class-count incompatibility, pretraining overlap, and memory limitations.
  • G.1 DATASETS WITH FEWER THAN 1000 INSTANCES: AutoGluon achieves the strongest overall performance on datasets with fewer than 1,000 instances, followed closely by TabPFNv2 and TabICL.MLP-like methods also have lower median ranks than CatBoost, LightGBM, and XGBoost in this setting.
  • G.1 DATASETS WITH FEWER THAN 1000 INSTANCES: In-context learning methods and dataset-specific neural architectures surpass CatBoost, LightGBM, and XGBoost in the small-data domain.TabM, MLP with PLR embeddings, and RealMLP are among the strongest dataset-specific architectures.
  • G.2 DATASETS WITH 1,000 TO 5,000 INSTANCES: TabICL and TabPFNv2 dominate datasets with 1,000–5,000 instances and fewer than 100 features, outperforming AutoGluon.TabM ranks third, while CatBoost rises to fourth and remains behind the leading neural methods.
  • G.2 DATASETS WITH 1,000 TO 5,000 INSTANCES: For other instance-feature groups, the study reports tables rather than plots because too few datasets support boxplots or critical-difference diagrams.Tables 27–29 cover the 1,000–5,000-instance groups with 100–500, 500–1,000, and more than 1,000 features.

H COST VS. EFFICIENCY RELATION OF VARIOUS MODEL FAMILIES

The study evaluates model efficiency through convergence and performance-time trade-offs, finding that deep-learning models can offer favorable efficiency despite slower convergence than GBDTs. Efficiency rankings vary substantially with dataset size and the time measure used.

  • H COST VS. EFFICIENCY RELATION OF VARIOUS MODEL FAMILIES: GBDTs converge faster than deep-learning models, with XGBoost reaching nearly optimal performance within 5 hours and TP-BERTa converging slowest.XTab converges faster than FT-Transformer, likely because its architecture is static while FT-Transformer components are tuned.
  • H COST VS. EFFICIENCY RELATION OF VARIOUS MODEL FAMILIES: Normalized performance-time ratios quantify the trade-off between performance loss and time savings, with higher values indicating more favorable trade-offs.The resulting curves show how often each algorithm achieves favorable ratios across datasets; steeper curves are better.
  • H COST VS. EFFICIENCY RELATION OF VARIOUS MODEL FAMILIES: TabM leads inference-time performance-time ratios overall, followed by AutoGluon, RealMLP, and CatBoost.AutoGluon combines strong performance with increased temporal cost from its computational burden.
  • H COST VS. EFFICIENCY RELATION OF VARIOUS MODEL FAMILIES: In the small-data domain, CatBoost and AutoGluon provide the best performance-time ratios, with SAINT and TabM becoming competitive at higher performance-ratio thresholds.The comparison considers both inference time and total time.
  • H COST VS. EFFICIENCY RELATION OF VARIOUS MODEL FAMILIES: In the large-data domain, TabM is best for inference time, whereas AutoGluon with HPO and XGBoost provide the strongest total-time trade-offs.CatBoost follows them, while TP-BERTa is not competitive with the other approaches.
  • H COST VS. EFFICIENCY RELATION OF VARIOUS MODEL FAMILIES: Meta-feature correlations generally follow a common pattern across methods after restricting analysis to datasets shared by TabPFN and TP-BERTa.The earlier deviations for those methods likely reflected their limited dataset coverage rather than inherent method properties.
  • H COST VS. EFFICIENCY RELATION OF VARIOUS MODEL FAMILIES: Feature redundancy is negatively correlated with performance, whereas stronger feature-target alignment is associated with better performance across methods.These findings align with expected behavior under sub-optimal class separation and support the empirical protocol.
Loading 2402.03970v3…