A Closer Look at Deep Learning Methods on Tabular Datasets

Han-Jia Ye; Si-Yang Liu; Hao-Run Cai; Qi-Le Zhou; De-Chuan Zhan

Source-linked AI summary

A Closer Look at Deep Learning Methods on Tabular Datasets

Han-Jia Ye, Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, De-Chuan Zhan

arXiv:2407.00956v4Jul 1, 2024cs.LG

TL;DR

Tabular learning needs broad, current evaluation because datasets are heterogeneous and existing benchmarks lack a unifying framework. TALENT evaluates modern methods across 300+ diverse datasets, analyzes heterogeneity through meta-features and early dynamics, and adds compact and stress-test suites. Performance concentrates in a small model shortlist: ensembles help both trees and neural methods, while pretrained models narrow but do not eliminate trees’ historical advantage.

Problem
Tabular learning lacks a unifying evaluation framework covering diverse dataset sizes, feature compositions, domains, and task types.
Method
TALENT evaluates 40 methods across 300+ datasets, models validation dynamics from meta-features and early training signals, and introduces TALENT-tiny and TALENT-extension evaluation suites.
Results
Performance concentrates within a small shortlist, ensembling benefits tree-based and DNN methods, and pretrained models frequently narrow but do not fully eliminate trees’ historical advantage.
Takeaways & Limitations
Dataset heterogeneity, especially categorical–numerical interplay, sparsity, and entropy variance, helps identify when different method families succeed or fail.
Takeaways & Limitations
The benchmark retains 13 datasets with potential leakage to preserve comparability and representativeness, which can compromise strict evaluation quality.

Abstract

from arXiv · show

Tabular data is prevalent across diverse domains in machine learning. With the rapid progress of deep tabular prediction methods, especially pretrained (foundation) models, there is a growing need to evaluate these methods systematically and to understand their behavior. We present an extensive study on TALENT, a collection of 300+ datasets spanning broad ranges of size, feature composition (numerical/categorical mixes), domains, and output types (binary, multi--class, regression). Our evaluation shows that ensembling benefits both tree-based and neural approaches. Traditional gradient-boosted trees remain very strong baselines, yet recent pretrained tabular models now match or surpass them on many tasks, narrowing--but not eliminating--the historical advantage of tree ensembles. Despite architectural diversity, top performance concentrates within a small subset of models, providing practical guidance for method selection. To explain these outcomes, we quantify dataset heterogeneity by learning from meta-features and early training dynamics to predict later validation behavior. This dynamics-aware analysis indicates that heterogeneity--such as the interplay of categorical and numerical attributes--largely determines which family of methods is favored. Finally, we introduce a two-level design beyond the 300 common-size datasets: a compact TALENT-tiny core (45 datasets) for rapid, reproducible evaluation, and a TALENT-extension suite targeting high-dimensional, many-class, and very large-scale settings for stress testing. In summary, these results offer actionable insights into the strengths, limitations, and future directions for improving deep tabular learning.

1 Introduction

Tabular learning lacks a unified, broad evaluation framework despite strong practical adoption and rapid progress in deep and pretrained methods. TALENT addresses this gap with 300+ diverse datasets, systematic multi-criteria comparisons, heterogeneity analysis, and compact and stress-test evaluation suites.

Motivation and benchmark: TALENT provides 300+ datasets spanning binary, multi-class, and regression tasks across varied sizes, feature types, and application domains.The collection is intended to reduce benchmark artifacts caused by narrow or outdated dataset coverage.
Empirical evaluation: Across 40 representative methods, top performance concentrates within a small shortlist, while ensembling benefits both tree-based and deep tabular approaches.The evaluation uses average ranks, statistical tests, probability of best performance, and aggregated errors.
Heterogeneity analysis: The study quantifies heterogeneity by using meta-features and early training signals to predict later validation behavior.The analysis emphasizes categorical–numerical interplay, sparsity, and entropy variance as sources shaping method-family success or failure.
Evaluation design: TALENT-tiny contains 45 datasets for rapid prototyping, while TALENT-extension stress-tests high-dimensional, many-class, and very large-scale settings.Together, these suites support lightweight evaluation beyond the common-size benchmark regime.
Evaluation design: Figure 1 compares representative methods across performance, training efficiency, and model size for binary, multi-class, regression, and aggregate tasks.Performance is average rank and training time and model size are lower-is-better and radius-encoded efficiency measures, respectively.

2 Related Work

Related work establishes tabular learning as a broad, heterogeneous field spanning classical ensembles, deep architectures, pretrained models, and diverse evaluation protocols. The paper motivates comprehensive benchmarks that balance coverage, rigor, statistical comparison, and computational cost.

Tabular learning: Tree-based ensembles remain highly competitive because their inductive biases suit heterogeneous features and interactions across common tabular tasks.Representative methods include Random Forest, XGBoost, LightGBM, and CatBoost.
Deep tabular models: Deep tabular methods include residual MLPs, Transformer variants, regularized models, retrieval-based approaches, and architectures inspired by trees.These methods aim to learn representations and complex nonlinear interactions directly from tabular inputs.
Pretrained models: Tabular foundation models pretrain on collections of synthetic or real tasks and adapt to new datasets through in-context learning with minimal tuning.Recent work also explores lightweight fine-tuning, localized adapters, and semantic information from feature names and metadata.
Benchmarking: Existing benchmark studies differ in dataset breadth, included model families, and evaluation protocols, motivating broader and more current comparisons.Prior work has also examined dataset quality, age, feature engineering, temporal characteristics, cross-validation, and ensembles.
Benchmarking: Reliable assessment requires diverse classification and regression datasets, fair tuning, statistical comparisons, and explicit trade-offs between coverage and computational cost.This need is especially pressing as deep and pretrained tabular methods continue to advance.

3 Preliminary

The paper formalizes supervised tabular prediction, surveys the evaluated classical, tree-based, and neural model families, and clarifies which learning paradigms and methods fall within the benchmark scope. It restricts comparisons to intrinsic supervised performance while excluding several methods for scope, cost, or fairness reasons.

Learning with tabular data: Tabular instances contain numerical and categorical attributes, with categorical features commonly encoded as numerical vectors using methods such as one-hot or target encoding.The dataset is represented as N examples with d features.
Learning with tabular data: Supervised tabular learning maps vectorized instances to labels by minimizing empirical loss plus a regularization term, with expected generalization to unseen data from the same distribution.Labels may represent binary or multi-class classification or regression targets.
Scope: The benchmark focuses on intrinsic supervised model performance rather than unsupervised or self-supervised pretraining, generative modeling, or additional ensemble strategies.These excluded paradigms vary widely and often introduce extra design choices.
Representative models: The evaluated model families include classical methods, tree-based ensembles, and diverse DNN architectures such as MLP variants, regularized, tree-mimic, neighborhood-based, and ensemble-based methods.The study also includes pretrained models and adaptation methods built on TabPFN.
Scope: Several methods are excluded because of longer training or inference times, orthogonal objectives, stronger covered alternatives, or potential benchmark data overlap.RealTabPFN is omitted because its continual pretraining corpus intersects the benchmark.

4 A Comprehensive Tabular Data Benchmark

TALENT is designed as a broad, realistic benchmark for tabular learning, combining inclusive common-size datasets with specialized stress-test suites and a stricter core set. Its coverage spans tasks, domains, feature types, dataset sizes, categorical structure, class cardinality, and imbalance.

The benchmark combines a broad general collection with Talent-extension suites for high-dimensional, many-class, and very-large-scale problems.These specialized settings are evaluated separately to expose performance bottlenecks not always visible on standard-sized datasets.
The two-level selection strategy balances inclusiveness with fairness: the general set removes clear quality problems, while Talent-tiny applies stricter filtering and excludes leakage, trivial, duplicated, and other-modality datasets.The general benchmark deliberately retains 13 datasets with potential leakage for comparability and realism, whereas Talent-tiny excludes them for more rigorous evaluation.
TALENT contains 120 binary-classification, 80 multi-class, and 100 regression datasets, covering all three standard tabular prediction settings.
TALENT covers 13 application areas and preserves numeric-only, categorical-only, and mixed feature types within domains.This design supports evaluation of whether tabular methods generalize across varied applications and feature compositions.
The dataset collection targets a more even spread over N × d, allocating substantial coverage to small, medium, and moderately large problems.The benchmark uses this distribution to reduce size-driven artifacts in aggregate comparisons.
TALENT explicitly varies categorical-feature counts, multi-class cardinality, and classification imbalance to stress models under heterogeneous conditions.Classification is trained without task-specific rebalancing, with F1/AUC additionally reported for skewed test sets.

5 Comparison Results among Datasets

Pretrained foundation models and strong ensembles lead across many tabular tasks, while tuned neural and retrieval-based methods can rival them. Performance gains are concentrated among a small shortlist, and ensembling improves both tree-based and deep approaches.

Average performance: Pretrained foundation models, especially TabPFN v2 and TabICL, consistently rank among the best and often outperform classical ensembles and tuned deep models.Their successors substantially improve over earlier versions, with TabPFN v2 extending to regression and showing clear generalization advantages.
Average performance: Tree-based ensembles remain strong, robust baselines, while CatBoost and LightGBM often achieve top-tier ranks and are statistically indistinguishable from one another.RFM and xRFM also occupy rank intervals close to the strongest ensembles in binary classification and regression.
Relative improvements: Carefully tuned neural and retrieval-based methods can rival ensembles, with RealMLP, ModernNCA, and TabR showing consistent improvements over MLP.ModernNCA is particularly strong and robust in regression, whereas vanilla MLPs and several tree-mimic architectures are generally weaker.
Relative improvements: Ensembling improves deep tabular learning: TabM outperforms base MLPs, MNCA-ens surpasses ModernNCA, and ensemble-enhanced methods reliably improve over MLP.These gains are accompanied by stable performance across datasets and, for MNCA-ens, strong results in regression.
Statistical comparisons: Foundation models, ensembles, and top DNNs often remain statistically tied, so no method achieves universal superiority despite strong aggregate performance.This equivalence underscores that method advantages depend on the dataset and task rather than a single universally dominant architecture.
Probability of achieving the best accuracy: Fewer than 10 methods account for over 80% of best-performing cases, forming a practical shortlist led by TabICL, TabPFN v2, MNCA-ens, ModernNCA, CatBoost, LightGBM, and TabM.PAMA identifies TabICL and TabPFN v2 as leaders across tasks, while MNCA-ens reaches 22.8% in regression.

6 Measuring the Heterogeneity of Tabular Data

The paper measures tabular-data heterogeneity through meta-features and early training dynamics, linking feature composition to later validation behavior and method effectiveness. Feature heterogeneity, especially categorical–numerical interplay, sparsity, and entropy variation, helps distinguish when tree-based or deep methods are favored.

6.2 Selecting Effective Meta-Features for Heterogeneity Analysis: Meta-features and early validation signals are used to predict later training dynamics and identify dataset characteristics that influence method success or failure.The approach maps five early epochs and dataset meta-features to parameters of a validation-curve family.
6.4 Correlation between Selected Meta-Features and Types of Tabular Methods: Feature-distribution variability is a dominant correlate of model performance: XGBoost and CatBoost track entropy_mean, while LightGBM and ModernNCA track entropy_std most strongly.Random Forest is most sensitive to sparsity_std, whereas TabM relies more on sparsity_mean.
6.4 Correlation between Selected Meta-Features and Types of Tabular Methods: As entropy_std increases, LightGBM’s rank decreases, indicating better performance on datasets with greater diversity of feature types.The paper interprets entropy_std as capturing variation in feature types across a dataset.
6.4 Correlation between Selected Meta-Features and Types of Tabular Methods: Deep methods including RealMLP, FT-Transformer, and TabPFN v2 tend to rank worse as the instance-to-attribute ratio increases, revealing sensitivity to dataset size and dimensionality.TabM and Random Forest instead show rank improvements on denser datasets as sparsity-related measures decrease.
6.5 Analysis of the Tree-DNN Performance Gap via Selected Meta-Features: Feature heterogeneity is more strongly associated with the Tree-DNN performance gap than instance-to-attribute ratio, with entropy_std showing a correlation of −0.3333.Greater feature heterogeneity tends to favor tree-based methods, which can handle diverse feature distributions through successive partitioning.
6.4 Correlation between Selected Meta-Features and Types of Tabular Methods: Raw-feature DNNs perform worst on mixed feature datasets, whereas tree ensembles excel there and FT-Transformer behaves more like tree-based models.The paper attributes FT-Transformer’s relative advantage to embeddings that unify categorical and numerical features.

7 Lightweight and Stress-Test via Talent-Tiny and Talent-Extension

TALENT-tiny provides a compact, representative benchmark for efficient evaluation, while TALENT-extension probes high-dimensional, many-class, and very large-scale settings. These stress tests show that method rankings change across regimes, with specialized ensemble designs and hybrid choices becoming important.

7.1 TALENT-tiny: TALENT-tiny uses 45 datasets selected across task types, problem sizes, Tree-DNN preferences, feature compositions, and 14 application domains.The subset contains 15 binary-classification, 12 multi-class-classification, and 18 regression tasks.
7.1 TALENT-tiny: TALENT-tiny preserves the full benchmark’s relative rankings: tree ensembles lead binary classification, while RealMLP and MNCA perform strongly on multi-class and regression tasks.This alignment supports using the subset for efficient yet representative analysis.
7.1 TALENT-tiny: Ensembling generally improves stability and average ranks without substantially changing method order, but MNCA requires a specialized ensemble variant to realize clear gains.The contrast suggests ensemble strategies should sometimes be tailored to model design.
7.1 TALENT-tiny: A 100-trial hyperparameter budget offers a practical balance: 50 trials are insufficient for stable optimization, while 150–200 trials provide only minor additional gains.The near-overlap of the 100-, 150-, and 200-trial settings supports limiting search budgets in large benchmarks.
7.2 TALENT-Extension: Stress Testing in Challenging Scenarios: TALENT-extension stress-tests methods under high-dimensional, many-class, and very large-scale conditions that are underrepresented in common-size benchmarks.Its purpose is to assess scalability and robustness beyond typical tabular regimes.
7.2 TALENT-Extension: Stress Testing in Challenging Scenarios: High-dimensional tasks make logistic regression unexpectedly competitive, while foundation models become inconsistent and tree ensembles lose their usual dominance.TabM still benefits from ensembling, and for very large datasets CatBoost regains a clear edge.
7.2 TALENT-Extension: Stress Testing in Challenging Scenarios: The extension results support hybrid, adaptive pipelines combining strong trees, scalable MLPs, and pretrained components across diverse real-world regimes.The paper frames this as a path toward robust tabular learning when method rankings shift with scale, dimensionality, and class count.

8 Conclusion

Talent evaluates deep tabular learning across 300+ datasets and finds that performance varies by dataset but concentrates among a small shortlist of models. Ensembling, pretrained models, dataset heterogeneity analysis, and two-level benchmark subsets together provide practical guidance for model selection and evaluation.

Across 300+ datasets and varied task types, performance rankings vary by dataset but consistently concentrate within a small shortlist of models.This concentration offers a practical starting point for model selection.
Ensembling benefits both tree-based and DNN-based approaches, while pretrained models frequently narrow but do not eliminate trees' historical advantage.Strong classical ensembles remain competitive in the resulting comparison.
Dataset heterogeneity, including categorical–numerical interplay, sparsity, and entropy variation, helps explain which model families gain an advantage.The analysis uses meta-features and early training dynamics to predict later validation behavior.
Talent-tiny provides 45 balanced datasets for rapid reproducible evaluation, while Talent-extension stresses high-dimensional, many-class, and very large-scale settings.Results on these subsets reveal additional distinctions among model families.

Appendix A. Datasets Selection Details

The benchmark construction documents mislabeled task types and potential leakage, including leakage risks specific to pretrained general tabular models. Retaining affected datasets in Talent while excluding them from Talent-tiny separates historical comparability from stricter evaluation.

The benchmark includes 22 datasets whose task types were incorrectly labeled and required construction-time adjustment.The appendix lists examples including Contaminant datasets, Heart-Disease-Dataset, Insurance, and KDD.
Three datasets contain potential target leakage through features that reveal or nearly trivialize the target variable.Examples include gross margin, runtime status, and raw height and weight features.
General tabular models may show inflated performance when benchmark datasets overlap with their pretraining or validation data.This risk is especially relevant because such models can be early-stopped on real-world datasets.
TabPFN overlaps with two benchmark datasets in its validation set, while TabPFN v2 overlaps with 27 benchmark datasets.The appendix identifies these overlaps as potential sources of checkpoint-selection bias.
Affected datasets remain in the general Talent benchmark for comparability but are excluded from Talent-tiny's stricter leakage-free evaluation.This design supports both historical comparison and controlled analysis.

B.1 Average Performance and Rankings

Average-rank results show pretrained foundation models leading across settings, with strong tree ensembles, retrieval-based methods, and tuned neural models remaining competitive. The rankings are aggregated across binary, multi-class, regression, and all 300 datasets.

B.1 Average Performance and Rankings: TabPFN v2 and TabICL achieve the lowest average ranks in binary and multi-class classification, while TabPFN v2 leads regression.In regression, xRFM, MNCA-ens, and RealMLP follow closely behind TabPFN v2.
B.1 Average Performance and Rankings: Foundation models are the only family reported to excel simultaneously across binary classification, multi-class classification, and regression.Figure 18 summarizes average ranks over 120 binary, 80 multi-class, 100 regression, and 300 total datasets; lower rank is better.
B.1 Average Performance and Rankings: CatBoost and LightGBM achieve top-tier ranks across classification and regression, while XGBoost remains ahead of most DNNs.These tree-based ensembles continue to provide strong baselines.
B.1 Average Performance and Rankings: ModernNCA ranks highly across all task types, and tuned models such as MLP-PLR and RealMLP substantially outperform vanilla MLPs and ResNets.TabR is also strong in classification, while token-based models are stable performers.
B.1 Average Performance and Rankings: Figure 19 complements average ranks with pairwise t-tests and Win/Tie/Lose counts across binary, multi-class, regression, and all tasks.Darker heatmap colors indicate higher counts.

B.2 Pairwise Statistical Comparisons

Pairwise t-tests show overlapping groups of strong ensembles, tuned neural models, retrieval-based methods, and pretrained foundation models. Foundation models win most consistently in classification, whereas regression advantages are more task-dependent.

B.2 Pairwise Statistical Comparisons: CatBoost, LightGBM, and XGBoost dominate weaker baselines and form robust, often statistically indistinguishable groups.RealMLP and MLP-PLR outperform vanilla MLPs and ResNets but frequently tie with strong ensembles.
B.2 Pairwise Statistical Comparisons: Token-based methods such as FT-T and ExcelFormer are stable but not dominant, often tying with ensembles and tuned DNNs.
B.2 Pairwise Statistical Comparisons: TabPFN v2 and TabICL achieve the most consistent pairwise wins in binary and multi-class classification.In regression, they remain statistically tied with ensembles and ModernNCA, indicating task-dependent benefits.
B.2 Pairwise Statistical Comparisons: ModernNCA and MNCA-ens frequently win in classification, rivaling ensembles and supporting the strength of retrieval-based learning.
B.2 Pairwise Statistical Comparisons: Overall, top ensembles, retrieval-based methods, and pretrained models form overlapping statistical equivalence groups across many tasks.The pairwise results therefore show progress beyond weaker baselines without a single universally dominant family.

Appendix C. Details of the Heterogeneity Analysis

The heterogeneity analysis represents dataset properties and early validation behavior as inputs for forecasting training dynamics. It evaluates curve families and uses meta-features to connect dataset structure with model performance trajectories.

Meta-features: Meta-features encode structural and statistical dataset properties used to analyze how heterogeneity influences model behavior and training dynamics.The full meta-feature list is provided in Table 5.
Training dynamics: The framework records training logs, validation and test metrics, running time across 15 seeds, and model size for each dataset–method pair.
Curve modeling: Four curve families model validation trajectories, including basic and shifted power laws, an offset power law, and an irreducible-error formulation.The curve families are denoted M1–M4.
Curve modeling: Parameters estimated from initial support epochs are used to extrapolate validation curves on a query set.

C.4 Main Results and Analysis

The main results show that meta-features materially improve validation-curve forecasting beyond direct fitting and baseline curve families. The learned predictor can extrapolate later performance from only a few early epochs across diverse unseen datasets.

Forecasting results: Meta-learned curve prediction substantially improves accuracy over direct fitting by combining dataset properties with early validation dynamics.Table 6 reports average MAE and OVD, both lower-is-better metrics, for the compared curves.
Forecasting results: The learned predictor h accurately extrapolates remaining validation performance from only the first few epochs.
Qualitative analysis: Across 16 unseen datasets, h reconstructs DNN-friendly and tree-friendly learning curves for both classification and regression, with more faithful extrapolations when meta-features are included.
Interpretation: Linking meta-features to training dynamics characterizes heterogeneity and helps explain why models succeed or fail on specific tabular datasets.

C.5 By-Product: Forecasting Training Dynamics for Efficiency

Forecasting later validation behavior from early training can support efficient early stopping, while TALENT-tiny provides a balanced setting for controlled model comparisons. Its 45 datasets span DNN-friendly and tree-friendly cases.

Efficiency: Early forecasting can prune poor runs when accuracy plateaus or oscillates, reallocating resources during expensive, hyperparameter-sensitive deep tabular training.
Training-dynamics visualization: Figure 20 organizes 16 unseen datasets by DNN-friendly versus tree-friendly behavior and by classification versus regression task.
Efficiency: The forecasting framework can guide adaptive training strategies, although its primary purpose is analyzing heterogeneity in tabular datasets.
Talent-tiny: Talent-tiny contains 45 datasets whose approximately symmetric Tree–DNN score distribution balances DNN-friendly and tree-friendly cases.
Talent-tiny: This balanced curation supports controlled evaluations that separate algorithmic effects from dataset bias and complement the large-scale benchmark.