A Closer Look at Deep Learning Methods on Tabular Datasets

Background

• Tabular learning spans classical ensembles, modern neural architectures, and pretrained foundation models, whose recent gains have narrowed the historical gap with tree-based methods while motivating broader evaluation.Tabular datasets commonly mix numerical and categorical attributes with heterogeneous statistical behavior, and foundation models use pretraining and in-context learning for rapid adaptation.Source: §1 · p. 2

Motivation and benchmark scope

• TALENT addresses the lack of a unifying tabular evaluation framework by covering binary, multi-class, and regression tasks across diverse domains, sizes, feature types, and imbalance ratios.The broad coverage is intended to reduce benchmark artifacts and support fair, up-to-date comparisons under realistic task diversity.Source: §1 · pp. 2–3

Empirical evaluation

• Across 300+ datasets, no method dominates universally, but top performance concentrates in a small shortlist and ensembling benefits both tree-based and deep approaches.The unified evaluation compares 40 methods using average ranks, statistical tests, probability of best performance, and aggregated errors.Source: §1 · p. 3

Heterogeneity analysis

• Feature-space heterogeneity—especially categorical–numerical interplay, sparsity, and entropy variance—helps predict when specific deep tabular method families succeed or fail.The analysis predicts later validation dynamics from dataset meta-features and early training signals.Source: §1 · p. 3

Evaluation design

• Talent-tiny provides 45 datasets, approximately 15% of the full suite, for rapid balanced evaluation, while Talent-extension stress-tests high-dimensional, many-class, and very large-scale settings.The two-level design applies stricter quality rules and targets regimes beyond the common-size datasets.Source: §1 · p. 3

2.1 Learning with Tabular Data

• Tabular learning commonly addresses classification and regression, where tree methods remain competitive because their inductive biases suit heterogeneous features and interactions.Related tasks include clustering, anomaly detection, data generation, open-environment learning, symbolic regression, and streaming scenarios.Source: §2.1 · pp. 4–5

2.2 Deep Tabular Data Learning

• Deep tabular methods use MLPs, ResNets, Transformers, ensembles, retrieval, and pretrained models to capture nonlinear interactions and increasingly narrow tree ensembles’ historical advantage.Foundation models enable in-context prediction without explicit gradient updates, while lightweight adaptation strategies improve performance across regimes.Source: §2.2 · p. 5

2.3 Tabular Prediction with Semantic Information and LLMs

• Semantic feature representations and LLM prompting can improve tabular prediction, particularly in low-data regimes, but depend on meaningful metadata and face high-dimensional serialization costs.Approaches include feature embeddings, textual table serialization, metadata-enriched prompts, external reasoning, and instruction tuning.Source: §2.3 · pp. 5–6

2.4 Tabular Methods Evaluations

• Comprehensive evaluation must span classification and regression, diverse domains and feature types, and fair, well-tuned protocols with appropriate statistical comparisons.Prior benchmarks often underrepresent regression, high-dimensional, and large-scale settings, while limited tuning budgets can disadvantage deep models.Source: §2.4 · pp. 6–7

3.1 Learning with Tabular Data

• Supervised tabular learning maps numerical or encoded categorical features to binary, multi-class, or regression labels by regularized empirical risk minimization, aiming to generalize to unseen data.The evaluation focuses on intrinsic supervised performance, while acknowledging that pretraining, self-supervision, generative modeling, and external ensembles may improve results.Source: §3.1 · p. 7

3.2 Representative Tabular Models

• The study compares classical baselines, strong tree ensembles, and diverse DNN families spanning MLP, interaction-aware, token-based, regularization, tree-mimic, neighborhood, ensemble, and pretrained foundation models.Excluded methods are slower, orthogonal to predictive accuracy, superseded by stronger foundation models, or potentially confounded by benchmark data overlap.Source: §3.2 · p. 8

4.1 Design Philosophy

• TALENT combines broad common-size coverage with specialized high-dimensional, many-class, and very-large-scale extension settings, plus a stricter Talent-tiny core for controlled analysis.The general set prioritizes inclusiveness, while Talent-tiny balances domains, feature types, scales, and tree- versus DNN-friendly tasks.Source: §4.1 · pp. 9–10, §4.2 · p. 11

4.2 Datasets Collection

• TALENT contains 120 binary-classification, 80 multi-class, and 100 regression datasets, while retaining practical variants and excluding identified leakage from Talent-tiny.The general benchmark retains 13 potentially leaky datasets for comparability and real-world imperfection, whereas the curated core excludes them.Source: §4.2 · p. 11, Table 1 · p. 12

4.3 Dataset Splits and Evaluation Criteria

• Datasets are evaluated with fixed 64%/16%/20% train/validation/test splits, 100 Optuna trials, 15 random seeds, and metrics tailored to classification and regression.Aggregation uses average rank, statistical comparisons, relative improvement, shifted geometric means, and PAMA across datasets.Source: §4.3 · pp. 14–15

4.4 Advantages of Our Benchmark

• TALENT covers three prediction tasks, 13 application domains, mixed feature types, and a more even spread over dataset complexity N × d.This design reduces size-driven artifacts while testing generalization across heterogeneous applications and numerical, categorical, and mixed attributes.Source: §4.4 · pp. 15, 17
• Broad distributions of categorical-feature counts, class cardinalities, and imbalance expose models to heterogeneous encoding demands and skewed classification settings.Classification uses vanilla training while additionally reporting F1 and AUC for imbalanced test sets.Source: §4.4 · pp. 16–17

5.1 On Average Performance

• TabPFN v2 and TabICL consistently rank among the best methods, often significantly outperforming classical ensembles and tuned deep models, while narrowing but not eliminating the tree–DNN gap.TabPFN v2 extends to regression and shows clear generalization advantages, although foundation models remain statistically tied with ensembles and top DNNs in many comparisons.Source: §5.1 · pp. 17–19

5.2 Relative Improvements over Tabular Baselines

• Tree ensembles achieve strong, stable gains over MLP, while RealMLP, TabR, ModernNCA, and foundation models improve reliably; SwitchTab, GrowNet, and TabNet show unstable distributions.Foundation models usually beat MLP by modest margins, and ensemble-enhanced neural methods such as MNCA-ens and TabM reliably improve stability.Source: Figure 4 · p. 20, §5.2 · p. 21

5.3 Probability of Achieving the Best Accuracy

• Up to 22.7% of multi-class datasets are won by TabICL or TabPFN v2, while MNCA-ens reaches 22.8% PAMA in regression.Fewer than 10 methods account for over 80% of best-performing cases across binary, multi-class, and regression tasks, forming a practical shortlist dominated by foundation, retrieval-based, and ensemble methods.Source: §5.3 · pp. 21–22, Figure 5 · p. 22

5.4 Averaged Performance

• SGM and nRMSE show foundation and neighborhood-based methods as the most stable across seeds, with tuned MLPs and gradient-boosting ensembles remaining top-tier competitors.TabPFN v2 and TabICL lead binary and multi-class SGM, while MNCA-ens, xRFM, CatBoost, LightGBM, and RealMLP achieve the lowest regression nRMSE.Source: §5.4 · p. 24

5.5 Results across Domains and Dataset Sizes

• Model strengths are domain-dependent: TabPFN v2 and MNCA-ens adapt broadly, CatBoost excels in chemistry and finance, and no approach leads universally across domains.Foundation and retrieval-based models dominate multimedia and social science, whereas ensembles remain strong in biology and healthcare; RealMLP and FT-T show complementary domain strengths.Source: §5.5 · pp. 25–26
• Scaling behavior varies by dataset structure: CatBoost improves as dataset size increases, TabPFN v2 performs strongly at medium-to-large scales and d > 100 but tapers at very large sizes, and rankings vary with N, d, and C.MNCA-ens, CatBoost, and RealMLP gain from larger N; xRFM performs well for N ≤2000; FT-Transformer and xRFM degrade as d increases.Source: §5.5 · p. 26

5.6 Revisiting the Tree-DNN Debate

• The Tree–DNN score measures the normalized difference between the best tree-based and DNN-based models, with positive values indicating tree-friendly datasets and negative values DNN-friendly datasets.This metric is introduced to quantify the longstanding divide between ensemble-favored and neural-favored datasets.Source: §5.6 · pp. 27–28
• Pretrained foundation models shift the Tree–DNN balance toward DNNs, although ensembles still win on many regression-heavy or highly categorical datasets.Without TabPFN v2, many datasets favor trees while a comparable fraction favors DNNs; adding it narrows and sometimes reverses ensemble dominance.Source: §5.6 · p. 28

5.7 Comparisons with Imbalance-Sensitive Criteria

• AUC rankings favor TabICL, TabPFN v2, and CatBoost, with TabICL achieving the lowest average rank on 67 imbalanced classification datasets.These methods separate minority and majority classes effectively, while ModernNCA and its ensemble remain competitive; Figure 10 reports average ranks across 31 binary and 36 multi-class tasks.Source: §5.7 · p. 28, Figure 10 · p. 29
• F1-score instead favors RealMLP, followed closely by ModernNCA, TabR, and CatBoost, while TabICL and TabPFN v2 are less consistent.The result reflects stronger precision–recall trade-offs for rare classes, despite TabICL and TabPFN v2 dominating under AUC.Source: §5.7 · p. 28, Figure 10 · p. 29
• Model choice should match the evaluation goal: AUC-oriented tasks benefit from ensembles and pretraining, whereas F1-sensitive settings may favor RealMLP or neighborhood-based designs.Random Forest and XGBoost remain mid-level across both metrics, while FT-T is stable and competitive but rarely leads.Source: §5.7 · pp. 28–29

6.1 Reformulating Meta-Feature Selection as a Dynamics Forecasting Task

• The dynamics-forecasting formulation predicts validation curves from dataset meta-features and early validation signals, revealing which properties shape neural training behavior.Validation statistics are recorded across T epochs, with classification accuracy or normalized RMSE providing the predicted dynamics.Source: §6.1 · p. 30

6.2 Selecting Effective Meta-Features for Heterogeneity Analysis

• A curve-parameter predictor uses the first five epochs and dataset meta-features to extrapolate later validation dynamics on unseen datasets.The predictor is a four-layer MLP that maps 24 inputs—five early-curve values and 19 meta-features—to four curve parameters.Source: §6.2 · pp. 30–31, §6.3 · p. 31

6.3 The Selected Meta-Features

• Important meta-features differ by task and method: gravity matters for classification, range.mean and mean.mean for regression, and sparsity or distribution statistics distinguish trees from DNNs.Tree-based methods emphasize sparsity.mean, whereas DNN-based methods are more sensitive to statistics such as max.sd.Source: §6.3 · pp. 31–32

6.4 Correlation between Selected Meta-Features and Types of Tabular Methods

• Feature heterogeneity metrics most consistently correlate with method performance, while individual methods respond differently to entropy, sparsity, and instance-to-attribute ratios.XGBoost and CatBoost track entropy_mean; LightGBM and ModernNCA track entropy_std; RealMLP, FT-Transformer, and TabPFN v2 track inst_to_attr.Source: Table 2 · p. 33, §6.4 · pp. 33–34

6.5 Analysis of the Tree-DNN Performance Gap via Selected Meta-Features

• Greater feature heterogeneity favors tree-based methods over neural networks, with the tree–DNN gap correlating −0.3333 with entropy_std.As entropy_std increases, tree-based ranks improve while neural ranks worsen; inst_to_attr shows a weaker −0.3502 relationship with DNN performance.Source: §6.5 · p. 35, Figure 13 · p. 36

6.6 Performance Comparison across Different Dataset Feature Types

• On mixed-feature datasets, raw-feature DNNs perform worst, whereas tree methods excel and FT-Transformer approaches their performance through unified feature embeddings.Tree methods benefit from feature splitting and hierarchical decisions, while FT-Transformer’s embeddings mitigate numerical–categorical heterogeneity.Source: §6.6 · pp. 36–37, Figure 14 · p. 37
• TabPFN v2 performs best on numerical-only data, followed by mixed data, and worst on categorical-only data, reflecting limitations in its categorical-feature handling.Its preprocessing typically converts categorical values to numerical representations, leaving robust categorical modeling an open challenge.Source: §6.6 · p. 37

7.1 Talent-Tiny: A Compact Benchmark for Efficient and Detailed Evaluations

• Talent-tiny contains 45 datasets selected across task types, problem sizes, Tree–DNN preferences, 14 domains, and balanced categorical–numerical compositions.The subset includes 15 binary-classification, 12 multi-class-classification, and 18 regression tasks, while excluding other modalities, leakage, distribution shifts, and duplicates.Source: §7.1 · p. 38
• Talent-tiny preserves the full-benchmark method ordering: hold-out rankings remain reliable, while cross-validation with ensembling slightly improves average ranks and stabilizes results.Tree methods remain strongest for binary classification, whereas RealMLP and MNCA perform strongly in multi-class classification and regression; MNCA needs specialized ensemble designs for clear gains.Source: §7.1 · pp. 38–39, Figure 15 · p. 39
• Using 100 hyperparameter-search trials offers a practical near-saturation point: 50 trials are insufficient, while 150 and 200 provide little additional benefit.This conclusion holds across dataset-size groups, where the rankings for 100, 150, and 200 trials are nearly overlapping.Source: Table 4 · p. 40, §7.1 · p. 40

7.2 Talent-Extension: Stress Testing in Challenging Scenarios

• Talent-extension comprises 18 high-dimensional, 12 many-class, and 18 very-large-scale datasets, exposing bottlenecks that standard-sized benchmarks may miss.The suite covers features numbering from 2,000 to over 20,000, tasks with more than ten classes, and datasets containing hundreds of thousands to millions of instances.Source: §7.2 · pp. 40–41
• In Talent-extension, linear models lead high-dimensional tasks, deep models regain the advantage for many-class classification, and tree ensembles—especially CatBoost—dominate very-large-scale datasets.These shifts reflect the curse of dimensionality, the need for shared embeddings across fine-grained labels, and the scalability of optimized tree implementations.Source: Figure 16 · p. 41, Figure 17 · p. 42, §7.2 · pp. 42–43

8 Conclusion

• Performance concentrates within a small shortlist of models despite dataset-dependent rankings, offering a practical starting point for model selection.The evaluation spans 300+ datasets covering varied sizes, domains, feature compositions, and task types.Source: §8 · pp. 43–44
• Ensembling benefits both tree-based and DNN-based approaches, while pretrained models frequently narrow—but do not eliminate—trees’ historical advantage.Strong classical ensembles remain competitive, refining the current “trees vs. neural networks” narrative.Source: §8 · p. 44
• Categorical–numerical interplay, sparsity, and entropy variation are key heterogeneity drivers of model advantage, identifiable from meta-features and early training dynamics.These signals help predict later validation behavior and explain when different model families win.Source: §8 · p. 44
• TALENT-tiny provides 45 carefully balanced datasets for rapid, reproducible evaluation, while TALENT-extension stress-tests high-dimensional, many-class, and very large-scale settings.Results on these subsets surface additional distinctions among model families and support heterogeneity-aware, ensemble-strengthened tabular learning.Source: §8 · p. 44

• 22 mislabeled datasets were corrected to binary classification after reviewing their metadata and label structures.Source: Appendix A · p. 44
• 25 datasets with tabular features extracted from images or audio were retained because pre-extracted features reflect practical resource and efficiency constraints.Source: Appendix A · p. 44
• Known or potential leakage affects multiple datasets, including direct target leakage, target-informative features, and evaluation leakage from pretrained models’ prior dataset exposure.Potential leakage was identified in three additional datasets, while general tabular models may have seen benchmark datasets during pretraining or validation.Source: Appendix A · pp. 44–45
• TabPFN validation overlap was found for 2 benchmark datasets and TabPFN v2 for 27, so these datasets remain in TALENT but are excluded from TALENT-tiny.This preserves comparability with prior studies while enabling stricter, leakage-free evaluation in the compact subset.Source: Appendix A · p. 45

B.1 Average Performance and Rankings

• Average ranks and pairwise 95% confidence-interval t-tests provide complementary views of relative method positioning across the benchmark.Figure 18 summarizes average ranks, while Figure 19 reports pairwise Win/Tie/Lose outcomes for all method pairs.Source: §B.1 · p. 45, Figure 19 · p. 47
• Foundation models are the only method family excelling simultaneously across binary, multi-class, and regression tasks, indicating broad generalization.TabPFN v2 and TabICL consistently achieve the lowest ranks in binary and multi-class classification, while TabPFN v2 remains best in regression.Source: §B.1 · p. 45
• CatBoost and LightGBM remain top-tier baselines across classification and regression, while XGBoost still outperforms most deep neural networks.Tree-based ensembles continue to provide strong comparative baselines despite the leading performance of foundation models.Source: §B.1 · p. 45
• Foundation models achieve the best overall rankings, with TabPFN v2 and TabICL leading classification and TabPFN v2 leading regression across 300 datasets.Figure 18 reports average ranks for binary classification, multi-class classification, regression, and all 300 datasets; lower ranks indicate better performance.Source: §B.1 · p. 45, Figure 18 · p. 46
• xRFM performs particularly well in regression, often rivaling the strongest ensembles and deep neural networks.RFM and its extension xRFM are highlighted as strong regression methods, with xRFM joining the leading group alongside TabPFN v2, MNCA-ens, and RealMLP.Source: §B.1 · pp. 45–46
• ModernNCA ranks highly across all task types, TabR is strong in classification, and tuned MLPs substantially outperform vanilla MLP.MLP-PLR and RealMLP achieve much lower ranks than vanilla MLP, with RealMLP frequently joining the top-performing group.Source: §B.1 · p. 46
• Token-based models are stable, especially in classification, but do not decisively surpass the best ensembles or neighborhood-based methods.FT-T, ExcelFormer, and AutoInt remain competitive without clearly dominating the strongest non-foundation approaches.Source: §B.1 · p. 46
• NODE, TabNet, and GrowNet generally occupy the lower half of method rankings across the evaluated tasks.These tree-mimic models lag behind the strongest ensembles, foundation models, and selected deep learning approaches.Source: §B.1 · p. 46

Pairwise Statistical Comparisons

• CatBoost, LightGBM, and XGBoost dominate weaker baselines and form robust, statistically indistinguishable groups, while RealMLP and MLP-PLR outperform vanilla MLPs and ResNets.RealMLP and MLP-PLR nevertheless tie with strong ensembles in many comparisons.Source: §B.2 · p. 47
• Token-based methods such as FT-T and ExcelFormer show stable but nondominant behavior, often tying with ensembles and tuned DNNs.The comparisons use pairwise t-tests with 95% confidence intervals and report Win/Tie/Lose counts across binary, multi-class, regression, and all tasks in Figure 19.Source: §B.2 · pp. 47–48
• Pretrained models TabPFN v2 and TabICL achieve the most consistent wins in binary and multi-class classification, but tie statistically with ensembles and ModernNCA in regression.This indicates that their benefits depend on the prediction task.Source: §B.2 · p. 48
• Neighborhood methods, especially ModernNCA and MNCA-ens, frequently win classification comparisons and rival ensembles.These results reinforce the strength of retrieval-based learning.Source: §B.2 · p. 48
• Overall, pretrained models improve state-of-the-art performance, but top ensembles and retrieval methods remain highly competitive within overlapping statistical-equivalence groups.The heatmaps summarize these comparisons across the evaluated task types.Source: §B.2 · p. 48

Appendix C. Details of the Heterogeneity Analysis

• The framework combines dataset meta-features, training logs, performance metrics, runtime, and model-size measurements to analyze how dataset properties shape model behavior.Meta-features characterize structural and statistical properties, while dynamics recordings connect those properties with optimization behavior.Source: §C.1 · p. 48, §C.2 · p. 48

C.1 Details of Meta-Features

• Meta-features encode structural and statistical dataset properties and support prediction of training dynamics and identification of influential dataset factors.The full meta-feature list is provided in Table 5.Source: §C.1 · p. 48, Table 5 · p. 49

C.2 Recording Training Dynamics

• Training dynamics include learning-rate schedules, batch-wise losses, intermediate statistics, validation and test metrics, runtime across 15 seeds, and model size.Runtime measurements account for early stopping at variable epochs and include both default and tuned hyperparameters for model size.Source: §C.2 · p. 48
• The analysis records accuracy, RMSE, loss, F1, AUC, MAE, and R2 to connect dataset properties with optimization behavior beyond endpoint accuracy or RMSE.The metric set covers classification and regression, with secondary metrics included for both task types.Source: §C.2 · p. 48

C.3 Alternative Curve Families for Validation Dynamics

• Four curve families model validation trajectories: a basic power law, shifted power law, offset power law, and a rational form with irreducible-error and random-guess parameters.The families are denoted M1 through M4 and draw on prior work in vision, language, and scaling-law modeling.Source: §C.3 · pp. 48–49
• Parameters are fitted on an initial support set S comprising the first epochs, then extrapolated to a query set Q for curve reconstruction.This procedure forecasts later validation trajectories from early training segments.Source: §C.3 · p. 49

• The predictor uses a four-layer MLP that maps 19 meta-features and the first 5 validation points to curve parameters.The output is the parameter set θ defining the curve family.Source: §C.4 · p. 50
• Performance is evaluated with Mean Absolute Error (MAE) and Optimal Value Difference (OVD), which measures discrepancy between predicted and true curve optima.OVD uses the maximum for classification and minimum for regression; results are reported in Table 6.Source: §C.4 · p. 50
• Meta-learned h more accurately predicts validation curves than direct fitting, showing that dataset meta-features substantially improve extrapolation beyond early training dynamics.Direct fitting often produces suboptimal results, while other curve families face convergence issues.Source: §C.4 · p. 50
• Across 16 unseen datasets, h reconstructs DNN-friendly and tree-friendly curves for classification and regression, with more faithful extrapolations than baseline curve families.The advantage is especially pronounced when meta-features are included.Source: §C.4 · p. 50
• Linking meta-features to training dynamics characterizes dataset heterogeneity and explains how dataset properties shape optimization trajectories and model success or failure.The approach both improves validation-curve prediction and deepens understanding of performance differences across tabular datasets.Source: §C.4 · p. 50

C.5 By-Product: Forecasting Training Dynamics for Efficiency

• Forecasting later performance from early epochs can prune poor deep-tabular training runs when accuracy plateaus or oscillates, reallocating computational resources.This practical by-product addresses the expense and hyperparameter sensitivity of deep tabular training.Source: §C.5 · p. 50
• Figure 20 fits training dynamics on 16 unseen datasets using validation curves from an MLP trained with default hyperparameters.The datasets span DNN-friendly and tree-friendly groups, with both classification and regression tasks.Source: Figure 20 · p. 51
• The forecasting framework can guide adaptive training strategies, although its primary purpose remains analyzing heterogeneity in tabular datasets.The efficiency benefit is explicitly presented as a secondary use rather than the main focus.Source: §C.5 · pp. 50–51

Appendix D. More Details on Talent-Tiny

• Tree–DNN scores quantify each dataset’s relative advantage of tree-based versus neural models on the same task.In Talent-tiny, lower scores identify DNN-friendly datasets and higher scores identify tree-friendly datasets.Source: Appendix D · pp. 51–52
• Talent-tiny’s approximately symmetric Tree–DNN score distribution balances DNN-friendly and tree-friendly datasets across its 45-dataset curated subset.DNN-friendly datasets favor RealMLP and MNCA, whereas tree-friendly datasets favor ensembles such as CatBoost.Source: Appendix D · pp. 51–52, Figure 21 · p. 52
• This balanced design facilitates controlled model evaluations and helps disentangle algorithmic factors from dataset bias, complementing the main benchmark.The subset is intended for fine-grained analysis of relative tree-versus-neural advantages.Source: Appendix D · pp. 51–52