Source-linked AI summary
Tabular Data: Deep Learning is Not All You Need
Ravid Shwartz-Ziv, Amitai Armon
TL;DR
Deep models for tabular data have been proposed as alternatives to XGBoost, but fair evidence across datasets and tuning costs is limited. The paper compares recent deep models and XGBoost under a shared protocol, finding that XGBoost generally performs better and is easier to tune, while their ensemble performs best. The authors caution that the dataset selection cannot represent the full diversity of tabular data.
Problem
Recent deep tabular models claim advantages over XGBoost, but differing datasets and unequal optimization make their comparative performance unclear.
Method
The study evaluates four recent deep models, XGBoost, and ensembles on eleven diverse datasets using the same tuning protocol, while examining performance, computation, and optimization time.
Results
XGBoost generally outperforms the deep models, while an ensemble combining XGBoost with deep models performs better than XGBoost alone.
Takeaways & Limitations
Under time constraints, XGBoost may provide strong results with easier optimization, while adding deep models may maximize ensemble performance.
Takeaways & Limitations
The selected datasets cannot represent the full diversity of tabular data, and no model is always better or worse than every other model.
Abstract
from arXiv · showhide
A key element in solving real-life data science problems is selecting the types of models to use. Tree ensemble models (such as XGBoost) are usually recommended for classification and regression problems with tabular data. However, several deep learning models for tabular data have recently been proposed, claiming to outperform XGBoost for some use cases. This paper explores whether these deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. In addition to systematically comparing their performance, we consider the tuning and computation they require. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also demonstrate that XGBoost requires much less tuning. On the positive side, we show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.
1 Introduction
Tabular data remains dominated by gradient-boosted tree methods, while recent deep models lack fair, standardized comparisons. This study evaluates whether those deep models should be recommended by comparing accuracy, tuning time, and computation against XGBoost.
- Tabular data is widely used across practical fields, while traditional methods such as GBDT have continued to outperform deep learning.
- Recent deep tabular models have been claimed to outperform GBDT, but differing datasets, unavailable implementations, and unequal optimization make comparisons difficult.
- The study asks whether recent deep models are more accurate than alternatives on both original and new datasets and how their training and tuning time compares.
- Using eleven datasets and a shared tuning protocol, the study compares four recent deep models with XGBoost and evaluates an ensemble combining them.
- The authors conclude that the selected datasets cannot represent all tabular data and that no model is universally superior.
2 Background
Tabular-data applications have traditionally favored gradient-boosted decision trees, while recent work proposes diverse deep architectures that have rarely been compared systematically. The study examines four claimed alternatives and combines them with XGBoost in practical ensemble variants.
- Classical models: Gradient-boosted decision trees, including XGBoost, LightGBM, and CatBoost, traditionally dominate tabular-data applications because of their strong performance.GBDTs build predictions from ensembles of decision-tree weak learners, while XGBoost uses gradient-based loss minimization.
- Deep architectures: Recent tabular deep-learning research spans differentiable trees, attention-based models, regularization methods, feature-interaction models, and 1D-CNNs.These approaches seek neural architectures or differentiable components suited to tabular data despite the lack of natural feature locality.
- Research gap: The literature’s models were evaluated on different benchmarks and had rarely been compared directly.This motivates a systematic comparison of the proposed deep models with tree ensembles.
- Models examined: The study examines TabNet, NODE, DNF-Net, and 1D-CNN because they were claimed to outperform tree ensembles and attracted substantial industry attention.TabNet uses sparse learned feature masks, NODE uses differentiable oblivious decision trees, and DNF-Net uses soft disjunctive-normal-form blocks.
- Ensembling: Ensemble learning combines predictions from multiple base learners to improve performance and reduce variance, using either randomized or sequential boosting-based constructions.The paper distinguishes simultaneously trainable randomized ensembles from sequential boosting ensembles.
- Ensembling: The proposed practical ensemble combines TabNet, NODE, DNF-Net, 1D-CNN, and XGBoost using either uniform weights or weights normalized from validation loss.Each model is trained on the entire training dataset, and uniform weighting is treated as a special case of the weighted formulation.
3 Comparing the Models
The study compares recent deep tabular models with XGBoost across diverse datasets using consistent tuning and evaluation procedures. XGBoost generally performs better and converges faster, while combining deep models with XGBoost often improves performance further.
- Experimental setup: The study evaluates deep models, XGBoost, and ensembles on 11 diverse classification and regression datasets using a shared experimental protocol.The datasets span 10–2,000 features, 1–7 classes, and 7,000–1,000,000 samples.
- Generalization to unseen datasets: Deep models generally perform worse on datasets that were unseen in their original papers than on those papers’ original datasets.Each model typically performs best on datasets associated with its own paper, suggesting dataset-specific sensitivity.
- Generalization to unseen datasets: For 8 of 11 datasets, XGBoost significantly outperformed deep models on datasets not included in the original papers (p < 0.005).The results indicate that the deep models did not consistently generalize better than XGBoost across the benchmark.
- Ensembling: For 7 of 11 datasets, an ensemble of deep models or XGBoost was significantly better than single deep models (p < 0.005).The ensemble of deep models and XGBoost outperformed ensembles of classical models and deep models alone.
- Ensembling: The full ensemble achieved the best average relative performance deterioration at 2.32%, followed by XGBoost at 3.4%.The corresponding values were 7.5% for 1D-CNN, 10.5% for TabNet, 11.8% for DNF-Net, and 14.2% for NODE.
- Optimization difficulty: XGBoost reached good performance in fewer and shorter hyperparameter-optimization iterations than the deep models on the Shrutime dataset.The authors note that the comparison used Bayesian optimization and that initial hyperparameters may affect the result.
4 Discussion and Conclusions
The study finds that deep models are weaker than XGBoost on the explored datasets, while combining them with XGBoost achieves the best results. It also emphasizes that fair model comparison must account for hyperparameter-optimization difficulty.
- Deep models were weaker than XGBoost on datasets outside their original papers, whereas an ensemble with XGBoost performed best on the explored datasets.
- XGBoost alone may be easiest to optimize under time constraints, but adding deep models may maximize performance.
- The study concludes that deep models did not outperform XGBoost on the explored datasets and that further research remains necessary.
- Model comparisons should include how many attempts are required to identify appropriate hyperparameters.
A Tabular Data-sets Description
The study uses a broad collection of tabular datasets with varied characteristics, covering classification and regression tasks and both heterogeneous and homogeneous features.
- The datasets vary in numbers of features, classes, and samples, and include both classification and regression tasks.
- Some datasets contain high-dimensional data and heterogeneous features measured in different units.
- Other datasets contain homogeneous features, such as pixels for images or words for text.
- Table 4 provides a description of the tabular datasets used in the study.
B Optimization of hyperparameters
The hyperparameter-tuning setup reproduces the original papers’ dataset splits and uses specified search spaces for CatBoost.
- The datasets are split into training, validation, and test sets following the procedures used in the original papers.
- CatBoost tuning searches random strength, maximum size, L2 leaf regularization, bagging temperature, and leaf-estimation iterations.
B.2 XGBoost
The XGBoost search space covers tree size, sampling, and regularization hyperparameters used during optimization.
- XGBoost tuning searches the number of estimators from 100 to 4000 and maximum depth from 1 to 10.
- The search includes subsample, column-by-tree, and column-by-level rates ranging from 0.2 to 1.
- Alpha, lambda, and gamma each use a uniform choice between 0 and a log-uniform range from e−16 to e2.
B.3 NODE
NODE is tuned over learning rate, architecture, tree, output-dimension, and batch-size choices. Its search spaces include both continuous or discrete ranges and finite candidate sets.
- NODE’s learning rate is searched over Log-Uniform [e−5, 1].
- NODE varies num layers from 1 to 10, tree depth from 4 to 9, and tree output dimension from 1 to 5.
- NODE’s total tree count is selected from {256, 512, 1024, 2048}, while batch size is selected from {512, 1024, 2048, 4096, 8192}.
B.4 TabNet
TabNet is tuned across optimization, feature and output dimensions, step count, normalization, relaxation, and batch size. The listed spaces combine continuous ranges with discrete ranges and finite choices.
- TabNet’s learning rate is searched over Log-Uniform [e−5, 1].
- TabNet varies feature dim and output dim from 20 to 60, and n steps from 1 to 8.
- TabNet searches bn epsilon over [e−5, e−1], relaxation factor over [0.3, 2], and batch size over {512, 1024, 2048, 4096, 8192}.
B.5 DNF-Net
DNF-Net is tuned using formula count, feature-selection beta, learning rate, and batch size. The listed search spaces include continuous or discrete ranges and a finite batch-size set.
- DNF-Net searches n. formulas over the discrete uniform range [256, 2048].
- DNF-Net searches feature selection beta over the discrete uniform range [1e−2, 2].
- DNF-Net searches learning rate over Log-Uniform [e−4, 0.5] and batch size over {512, 1024, 2048, 4096, 8192}.
B.6 1D-CNN
1D-CNN is tuned over hidden-layer sizes, layer count, learning rate, and batch size. The search spaces combine discrete architecture ranges, a log-uniform learning-rate range, and finite batch-size choices.
- 1D-CNN searches hidden layer sizes over the discrete uniform range [100, 4000].
- 1D-CNN varies the number of layers from 1 to 6.
- 1D-CNN searches learning rate over Log-Uniform [e−4, 0.5] and batch size over {512, 1024, 2048, 4096, 8192}.