Source-linked AI summary
Revisiting Deep Learning Models for Tabular Data
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, Artem Babenko
TL;DR
Tabular DL research lacks consistent benchmarks, model comparisons, and simple reliable baselines, leaving model superiority unclear. The paper evaluates major architectures under shared protocols, introducing ResNet-like and FT-Transformer baselines. FT-Transformer outperforms other DL solutions on most tasks, while GBDT remains superior on some, so no universally best solution emerges.
Problem
Existing tabular DL studies use inconsistent benchmarks and protocols, and the field lacks simple reliable baselines with competitive performance across tasks.
Method
The paper reviews major tabular DL architecture families and compares two simple models with existing solutions across diverse tasks under shared training and tuning protocols.
Results
FT-Transformer outperforms other DL solutions on most tasks, while GBDT still dominates on some tasks.
Takeaways & Limitations
ResNet-like models provide strong simple baselines, and FT-Transformer is a powerful broad-performing solution, but neither DL nor GBDT is universally superior.
Takeaways & Limitations
FT-Transformer requires more hardware and training time and may not scale easily when the number of features is too large because vanilla MHSA has quadratic feature complexity.
Abstract
from arXiv · showhide
The existing literature on deep learning for tabular data proposes a wide range of novel architectures and reports competitive results on various datasets. However, the proposed models are usually not properly compared to each other and existing works often use different benchmarks and experiment protocols. As a result, it is unclear for both researchers and practitioners what models perform best. Additionally, the field still lacks effective baselines, that is, the easy-to-use models that provide competitive performance across different problems. In this work, we perform an overview of the main families of DL architectures for tabular data and raise the bar of baselines in tabular DL by identifying two simple and powerful deep architectures. The first one is a ResNet-like architecture which turns out to be a strong baseline that is often missing in prior works. The second model is our simple adaptation of the Transformer architecture for tabular data, which outperforms other solutions on most tasks. Both models are compared to many existing architectures on a diverse set of tasks under the same training and tuning protocols. We also compare the best DL models with Gradient Boosted Decision Trees and conclude that there is still no universally superior solution.
1 Introduction
The paper addresses inconsistent evaluation and weak baselines in tabular deep learning by comparing models under shared protocols. It identifies ResNet and FT-Transformer as strong architectures, while finding no universal winner over GBDT.
- Motivation: Different datasets and evaluation protocols make existing tabular DL comparisons inconclusive.The literature does not clearly establish which DL models perform best or whether DL surpasses GBDT.
- Motivation: MLP is the main simple baseline, but it does not always provide a significant challenge for competing models.
- Approach: The study evaluates main tabular DL models on diverse tasks using the same training and hyperparameter-tuning protocols.This design targets fair comparison of relative model performance.
- Findings: A simple ResNet-like architecture is an effective baseline that existing literature often overlooked.The authors recommend it for comparison in future tabular DL work because of its simplicity.
- Findings: FT-Transformer performs well on a wider range of tasks than other DL models and becomes a powerful tabular solution.
- Findings: No universally superior solution emerges among GBDT and deep models.
2 Related work
Prior tabular DL research spans several architectural families, including differentiable trees, attention-based models, and feature-interaction designs, but comparisons remain inconsistent. The paper establishes fairer comparisons and finds that tuned ResNet and Transformer-based approaches are strong competitors.
- Existing approaches: GBDT ensembles remain the shallow state of the art and are widely used in tabular-data competitions and practice.Established implementations include XGBoost, LightGBM, and CatBoost.
- Existing approaches: Recent tabular DL models can be roughly grouped into differentiable trees, attention-based models, and explicit multiplicative-interaction designs.
- Existing approaches: Differentiable-tree methods smooth internal tree decisions so tree-like components can support gradient-based end-to-end training.
- Comparative findings: Properly tuned ResNet outperforms existing attention-based models, while the tabular Transformer adaptation outperforms ResNet on most tasks.
- Research gap: The literature evaluates varied architectures on different benchmarks and rarely compares them directly.This motivates a fair comparison focused on solutions that consistently provide high performance.
3 Models for tabular data problems
This section presents familiar deep-learning building blocks adapted for tabular data, including ResNet and FT-Transformer, with feature-level tokenization and a resource trade-off.
- Models for tabular data problems: The models are designed to reuse well-established deep-learning building blocks so they remain conceptually familiar and require less effort for competitive performance.The section focuses on ResNet and FT-Transformer as easy-to-use tabular architectures.
- ResNet: ResNet applies residual blocks with an almost clear path from input to output, which the authors find beneficial for optimization.The architecture uses a simplified residual block and is expected to help when deeper representations are useful.
- FT-Transformer: FT-Transformer converts numerical and categorical features into embeddings, processes them with Transformer layers, and uses the final [CLS] representation for prediction.Its Transformer layers operate at the feature level for each object.
- FT-Transformer: The Feature Tokenizer maps the k input features to embeddings in R^k×d, using feature-specific transformations for numerical and categorical inputs.Categorical features use lookup-table embeddings, while numerical features use element-wise multiplication with feature parameters.
- Limitations: FT-Transformer requires more training hardware and time than ResNet, and its vanilla self-attention has quadratic complexity in the number of features.This can limit scaling when the feature count is too large, although efficient attention approximations may alleviate the issue.
4 Experiments
The experiments compare deep architectures and GBDT under shared protocols across diverse tabular tasks. FT-Transformer performs best across most tasks, while no universal winner emerges between deep models and GBDT.
- Experimental setup: Eleven public datasets use shared train-validation-test splits, with common preprocessing for deep models and validation-only hyperparameter tuning.The evaluation reports test performance from 15 random seeds for each tuned configuration.
- Comparing DL models: FT-Transformer performs best on most tasks, while ResNet is an effective baseline that competitors cannot consistently outperform.MLP remains a useful sanity check, and tuning makes MLP and ResNet competitive.
- Comparing DL models: NODE performs strongly on several tasks but remains inferior to ResNet on six datasets and uses a more complex, ensemble-like structure.Ensembling improves FT-Transformer and ResNet relative to NODE, reducing the gap between ResNet and NODE.
- Comparing DL models and GBDT: Default FT-Transformer ensembles mostly outperform GBDT ensembles, including when using untuned hyperparameters.The reported exceptions are California Housing and Adult, while default and tuned FT-Transformer ensembles perform similarly.
- Comparing DL models and GBDT: Tuned GBDT dominates on California Housing, Adult, and Yahoo, so deep models do not universally outperform GBDT.The authors note that the benchmark includes DL-friendly problems and that GBDT can be unsuitable for some large multiclass tasks.
- Comparing DL models and GBDT: FT-Transformer remains competitive across all tasks, whereas GBDT and ResNet perform well only on some task subsets.FT-Transformer gains most over ResNet precisely where GBDT is superior to ResNet, while matching ResNet elsewhere.
5 Analysis
The analysis examines when FT-Transformer outperforms ResNet and tests FT-Transformer’s design choices and attention-based feature-importance heuristic. FT-Transformer remains competitive across synthetic tasks, while its Transformer backbone and feature biases contribute to its performance.
- 5.1 When FT-Transformer is better than ResNet?: FT-Transformer remains competitive across synthetic tasks, whereas ResNet’s relative performance drops as targets become more GBDT-friendly.On ResNet-friendly tasks, both models perform similarly and outperform CatBoost.
- 5.1 When FT-Transformer is better than ResNet?: The synthetic experiment reveals functions based on decision trees that FT-Transformer approximates better than ResNet.This aligns with datasets where GBDT outperforms ResNet and FT-Transformer shows stronger improvements.
- 5.2 Ablation study: FT-Transformer is compared with AutoInt because both convert features to embeddings and apply self-attention.They differ in feature biases, Transformer backbone, and inference through the [CLS] token.
- 5.2 Ablation study: Results averaged over 15 runs show that FT-Transformer’s Transformer backbone is superior to AutoInt’s and that feature biases are necessary.The comparison includes FT-Transformer without feature biases.
- 5.3 Obtaining feature importances from attention maps: Attention maps are evaluated as a feature-importance source by averaging [CLS]-token attention across heads, layers, and samples.The heuristic requires a single forward pass for one sample and is compared with Integrated Gradients and permutation tests using rank correlation.
6 Conclusion
The conclusion presents a simple ResNet-like model as an effective tabular-DL baseline and FT-Transformer as a powerful Transformer adaptation. It also reports that GBDT remains dominant on some tasks, so no universally superior solution is established.
- 6 Conclusion: A simple ResNet-like architecture serves as an effective baseline for tabular deep learning.The authors propose it as a basis for future comparisons.
- 6 Conclusion: FT-Transformer, a simple Transformer adaptation, outperforms other deep-learning solutions on most tasks.The study also compares these models with GBDT.
- 6 Conclusion: GBDT still dominates on some tasks, indicating that no universally superior solution was found.The conclusion places this result within the comparison between the new deep-learning baselines and GBDT.
Supplementary material
The supplementary workflow separates model tuning from evaluation and standardizes the experimental software environment. Tuned models are evaluated on NVIDIA Tesla V100 hardware.
- Supplementary material: For most model-dataset pairs, the workflow tunes the model on suitable hardware and then evaluates it on one or more NVIDIA Tesla V100 32Gb GPUs.The workflow is stated for most model-dataset pairs.
- Supplementary material: All experiments use the same software-version conditions, with hardware details available for almost all experiments in the source code.The supplementary material notes that hardware information is generally documented in the code.
B.1 Datasets
The supplementary dataset section provides a dataset description and records regression-target standardization and preprocessing details. Deep models and GBDTs receive different preprocessing treatment because GBDTs are insensitive to feature shifts and scaling.
- B.1 Datasets: Table 7 provides the dataset descriptions.The supplied material identifies this as the datasets-description table.
- B.1 Datasets: Regression target values are standardized.This preprocessing rule is stated for regression problems.
- B.1 Datasets: Deep models use feature preprocessing, while GBDTs are not preprocessed because they are insensitive to feature shifts and scaling.The deep-model preprocessing includes quantile estimation with added N(0, 1e−3) noise on training numerical features, followed by application to original features.
C Results for all algorithms on all datasets
The reported results cover all models and datasets, with statistical significance assessed using a one-sided Wilcoxon test at p = 0.01. Separate tables provide single-model and ensemble results with standard deviations and color-coded comparisons.
- p = 0.01 is the threshold for the one-sided Wilcoxon test used to assess statistical significance.
- Tables 8 and 9 together report results for all models on all datasets.
- Table 8 reports results for single models with standard deviations across datasets.It marks baseline-neural-network, FT-Transformer, and overall top results using distinct color conventions.
- Table 9 reports ensemble results with standard deviations using the same color conventions as Table 8.
D.1 Training times
This section documents training-time comparisons and implementation choices for FT-Transformer, ResNet, MLP, and XGBoost. The tuning study evaluates time budgets, while the supplementary configurations specify architectures, optimization, and search spaces.
- Training-time comparison: Training times are averaged over 15 runs in seconds.
- Tuning-time study: The tuning study compares XGBoost, MLP, ResNet, and FT-Transformer on California Housing, Adult, and Higgs Small under time-constrained optimization.Each algorithm receives five independent hyperparameter-optimization runs per dataset.
- Tuning-time study: FT-Transformer achieves good metrics after several randomly sampled configurations but is slower to train than the other algorithms.The passage also reports no meaningful gains from extended tuning for the other algorithms.
- FT-Transformer architecture: The FT-Transformer pipeline applies feature tokenization, appends a classification token, passes the sequence through Transformer blocks, and produces a prediction.
- Implementation choices: The implementation uses LayerNorm, eight attention heads, and ReGLU activation, while dropout rates are partly selected through tuning.Attention dropout was always beneficial, whereas the final dropout of residual branches was rarely nonzero.
- FT-Transformer architecture: FT-Transformer uses a PreNorm Transformer variant because it keeps optimization simple, whereas PostNorm may require warmup or complex initialization.
- Implementation choices: The default FT-Transformer configuration was an educated guess and received limited tuning, with a 929K parameter count specified for 100 numerical features.
- Hyperparameter spaces: The supplementary search spaces specify tuned parameters for FT-Transformer, ResNet, MLP, and XGBoost, alongside fixed optimization settings for the compared models.
G.2 Ablation study
The ablation study documents the construction of random decision-tree targets and reports a detailed comparison between FT-Transformer and two attention-based alternatives. The comparison summarizes means and standard deviations over 15 runs.
- Table 23 reports the comparison between FT-Transformer and two attention-based alternatives using means and standard deviations over 15 runs.
- Each random decision tree is expanded by repeatedly selecting leaves, sampling splitting features and thresholds, and assigning normally distributed leaf values.
H Additional datasets
Additional experiments evaluate single models on four datasets identified as non-informative benchmarks, where all models perform similarly. Results are averaged over 15 random seeds under the main-text training and tuning protocol.
- The additional datasets are Bank, Kick 2, MiniBooNe, and Click.
- All models perform similarly on these datasets, which are therefore characterized as non-informative benchmarks.
- Single-model results are averaged over 15 random seeds using the same training and tuning protocol as the main text.