PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Neural Networks and Tabular Data: A Survey

Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, Gjergji Kasneci

arXiv:2110.01889v3cs.LG

TL;DR

Deep learning remains difficult to adapt to heterogeneous tabular data despite its success on homogeneous modalities. This survey organizes methods, generation, and explanation strategies, then benchmarks them against traditional methods; tree ensembles mostly remain superior on supervised tasks.

  • Problem

    Deep learning performs strongly on homogeneous data, but its adaptation to heterogeneous tabular data for inference and generation remains challenging.

  • Method

    The survey reviews approaches for tabular inference, generation, and explanation, organizes them into three taxonomy branches, and empirically compares deep and traditional models.

  • Results

    Gradient-boosted tree ensembles mostly outperform deep-learning models on supervised tabular tasks, while SAINT outperformed classical approaches on one very large data set mainly containing continuous numerical variables.

  • Takeaways & Limitations

    The current state of tabular deep learning remains largely unchanged, and deeper understanding of the performance gap is needed rather than simply increasing model complexity.

Abstract

from arXiv · show

Heterogeneous tabular data are the most commonly used form of data and are essential for numerous critical and computationally demanding applications. On homogeneous data sets, deep neural networks have repeatedly shown excellent performance and have therefore been widely adopted. However, their adaptation to tabular data for inference or data generation tasks remains challenging. To facilitate further progress in the field, this work provides an overview of state-of-the-art deep learning methods for tabular data. We categorize these methods into three groups: data transformations, specialized architectures, and regularization models. For each of these groups, our work offers a comprehensive overview of the main approaches. Moreover, we discuss deep learning approaches for generating tabular data, and we also provide an overview over strategies for explaining deep models on tabular data. Thus, our first contribution is to address the main research streams and existing methodologies in the mentioned areas, while highlighting relevant challenges and open research questions. Our second contribution is to provide an empirical comparison of traditional machine learning methods with eleven deep learning approaches across five popular real-world tabular data sets of different sizes and with different learning objectives. Our results, which we have made publicly available as competitive benchmarks, indicate that algorithms based on gradient-boosted tree ensembles still mostly outperform deep learning models on supervised learning tasks, suggesting that the research progress on competitive deep learning models for tabular data is stagnating. To the best of our knowledge, this is the first in-depth overview of deep learning approaches for tabular data; as such, this work can serve as a valuable starting point to guide researchers and practitioners interested in deep learning with tabular data.

I. INTRODUCTION

Tabular data are heterogeneous and central to many applications, yet deep learning remains challenging beyond homogeneous modalities. The survey reviews methods, generation, interpretability, benchmarking, and open questions.

  • Heterogeneous tabular data combine dense numerical and sparse categorical features with weaker feature correlations than image or speech data.
  • Deep learning for tabular data involves three core challenges: inference, data generation, and interpretability.Interpretability also supports debugging and auditing during deployment.
  • The authors focus on inference, generation, and interpretability while briefly addressing data streams, distribution shifts, privacy, and fairness.
  • The empirical comparison addresses inconsistent benchmarks and commonly omitted aspects including training time, inference time, model size, and interpretability.
  • The survey reviews literature, proposes a taxonomy, covers generation and explanation methods, benchmarks models, and discusses open research questions.

III. TABULAR DATA AND DEEP NEURAL NETWORKS

The survey defines tabular data as fixed-feature tables containing continuous or categorical attributes and distinguishes them from homogeneous modalities. It also introduces the neural-network terminology used throughout the work.

  • A deep neural network learns parameters W for an approximate mapping from multidimensional samples x to targets y.
  • Heterogeneous data contain multiple attribute types, including continuous, binary, ordinal, and high-cardinality categorical values.
  • Categorical variables are qualitative values without numerical ordering and typically take one value from a limited set.
  • Tabular data consist of rows representing data points and columns representing a fixed set of continuous or categorical features.Each row can also be viewed as a sample from an unknown joint distribution.
  • Table I illustrates heterogeneous tabular data with five selected samples from the Adult data set.

B. A Brief History of Deep Learning on Tabular Data

Deep learning for tabular data developed from early machine-learning work through e-commerce-driven categorical prediction and later regularization, transformer, and self-supervised approaches.

  • Tabular data were early targets of machine-learning research, while deep neural networks later developed primarily around homogeneous digital data.
  • E-commerce and advertising created demand for fast, accurate prediction on heterogeneous data with many categorical variables.High-cardinality categories can produce sparse, high-dimensional feature vectors and non-robust models for traditional methods.
  • Click-through-rate prediction attracted extensive research, with many approaches relying on specialized neural architectures for heterogeneous tabular data.
  • Regularization research emerged from the idea that regularization could improve deep-learning performance on tabular data.
  • Transformer and self-supervised methods were applied to tabular data following their success in textual and visual domains.Research particularly targeted applications to very large tabular data sets.

C. Challenges of Learning With Tabular Data

The survey organizes tabular deep-learning methods into transformations, specialized architectures, and regularization, motivated by data-quality and structural challenges. It also reviews encoding strategies and practical integration trade-offs.

  • C. Challenges of Learning With Tabular Data: Real-world tabular data often contain missing values, outliers, errors, class imbalance, and small sample sizes relative to feature dimensionality.Tree-based methods can handle some missing values and variable-range issues internally.
  • C. Challenges of Learning With Tabular Data: Tabular feature dependencies may be complex, irregular, or non-spatial, making inductive biases from convolutional models unsuitable.
  • D. Unified Taxonomy: The taxonomy divides approaches into data transformation methods, specialized architectures, and regularization models.
  • Data transformation methods: Data transformations can improve information extraction without changing model architecture, but preprocessing increases time and may burden high-load systems.
  • Specialized architectures: Specialized architectures include hybrid models combining machine-learning components with neural networks and transformer-based attention models.
  • Regularization models: Regularization models address extreme non-linearity and complexity mainly through special-purpose loss functions.
  • Data encoding: Ordinal encoding maps categories to numbers but can impose artificial order, whereas one-hot encoding avoids order at the cost of additional columns.
  • Data encoding: Binary encoding represents c categories using log(c) new columns, reducing the number of columns compared with one-hot encoding.

1) Hybrid Models:

Hybrid models combine neural networks with classical methods or tree-inspired structures to handle heterogeneous tabular data, while retaining varying degrees of differentiability, interpretability, or efficiency.

  • Fully differentiable Models: Fully differentiable models enable end-to-end training and inference with gradient descent and accelerator-efficient implementations.
  • NODE uses differentiable oblivious decision trees, soft splits, and entmax to provide end-to-end optimization without categorical-data preprocessing.
  • Soft decision tree models improve accuracy when distilled from a pretrained neural network, but the resulting trees retain a performance gap relative to that neural model.
  • Partly differentiable Models: DeepGBM combines two neural networks with gradient-boosting structures to learn categorical and numerical features, achieving higher online prediction performance than gradient boosting decision trees.
  • TabNN transfers tree-derived structural knowledge into a neural network, but its construction is computationally extensive and its practical use is limited without an implementation.
  • Transformer-based Models: SAINT combines self-attention with inter-sample attention, allowing information sharing across similar rows and supporting robustness to missing or noisy data.

V. TABULAR DATA GENERATION

Realistic tabular-data generation supports augmentation, imputation, rebalancing, and privacy-aware machine learning.

  • Synthetic tabular data can support data augmentation, missing-value imputation, class rebalancing, and privacy-aware machine learning.

A. Methods

Tabular-data generation remains difficult because datasets mix discrete and continuous features with different distributions. The survey reviews deep generative models and emphasizes that quality requires multiple evaluation measures.

  • Mixed discrete and continuous features with heterogeneous value distributions make synthetic tabular-data generation challenging.
  • Interpretability: TabNet explanations compare model-provided feature attributions with those from KernelSHAP for an Adult-data sample.
  • CTGAN uses mode-specific normalization to represent multimodal and skewed continuous distributions through mixture components and deviations from component centers.
  • GANs, VAEs, causal models, and invertible flows are among the deep-learning architectures used for tabular-data generation.
  • Assessing Generative Quality: Generative quality can be assessed through machine-learning efficacy, distribution comparisons, statistical tests, likelihood, and distance-to-closest-record analyses.
  • Assessing Generative Quality: A single measure is insufficient for assessing generative quality because evaluation must address predictive utility, distributional similarity, overfitting, and memorization.

VII. EXPERIMENTS

The experiments compare machine-learning and deep-learning methods on heterogeneous real-world datasets with varied tasks and sizes, using standardized tuning and preprocessing procedures.

  • The benchmark addresses a literature gap by comparing predictive performance, training and inference time, model size, and interpretability.
  • The code, datasets, and containerized experiments are publicly available to support replication and additional benchmark contributions.
  • Data Sets: The study uses five real-world datasets spanning binary classification, multi-class classification, and regression tasks.
  • Data Sets: HIGGS is the largest dataset, containing eleven million rows for distinguishing Higgs-boson signals from background processes.
  • Hyperparameter Selection: Hyperparameters are tuned with 100 Optuna iterations per model, and each configuration is evaluated using five-fold cross-validation.
  • All models use standardized numerical features and ordinal categorical encoding, with zero substitution for missing values in linear and pure-neural models.

3) Reproducibility and Extensibility:

On most evaluated data sets, boosted decision-tree ensembles achieved the best scores, while SAINT surpassed classical methods on the very large HIGGS data set.

  • For all but HIGGS, boosted decision-tree ensembles obtained the best scores across the evaluated data sets.XGBoost and CatBoost outperformed all deep learning approaches on the small, medium, regression, and multi-class data sets.
  • SAINT outperformed classical machine learning approaches on the large-scale HIGGS data set.The result suggests modern neural architectures may benefit very large tabular data sets with predominantly continuous features.
  • Overall, the findings were consistent with earlier observations that decision-tree ensembles generally outperform deep learning on tabular data.

4) Results:

SAINT performed competitively across data sets, but other deep learning models depended strongly on the selected data set. Runtime comparisons showed a trade-off between training and inference efficiency on HIGGS.

  • SAINT provided competitive results across the evaluated data sets.
  • DeepFM performed best among deep learning models on Adult and second-best on California Housing, but produced weak results on HELOC.
  • On HIGGS, XGBoost required 5995 seconds for inference, compared with 10.18 seconds for MLP and 282 seconds for SAINT.Gradient-boosting models trained faster than most deep neural network methods, but their HIGGS inference time was substantially higher.

D. Interpretability Assessment

The interpretability assessment compared feature attributions using fidelity curves and agreement with KernelSHAP. Transformer-based attribution methods performed inconsistently, and the evaluation exposed limitations in current interpretability claims.

  • Feature attributions were selected as the common explanation format for comparing the models.The assessment focused on post-hoc feature attributions because they are prevalent among the considered methods.
  • Attribution fidelity was measured by retraining after removing features in Most Relevant First and Least Relevant First orders.An early accuracy drop is desirable for MoRF, whereas accuracy should remain high for LeRF.
  • For MoRF, TabNet and TabTransformer using the diagonal attention head appeared strongest; for LeRF, TabNet was significantly better than the other methods.
  • Spearman correlations with KernelSHAP were low across the evaluated models and were sometimes negative.Reported values included -0.01 ± 0.008 for columnwise TabTransformer attention, 0.00 ± 0.010 for diagonal TabTransformer attention, 0.07 ± 0.009 for TabNet, and -0.04 ± 0.007 for columnwise SAINT attention.
  • The transformer models did not produce convincing feature attributions out of the box in the two benchmarks.The authors call for more thorough evaluation of interpretability characteristics and practical usefulness.

A. Summary and Trends

The survey finds that tree ensembles remain state of the art on most considered tabular data sets, while deep learning retains potential on very large data. It advocates stronger benchmarking and deeper study of the performance gap.

  • Decision Tree Ensembles are still State-of-the-Art.: Tree-ensemble models such as XGBoost, LightGBM, and CatBoost still outperform deep learning on most considered data sets.They also provide significantly less training time.
  • Decision Tree Ensembles are still State-of-the-Art.: Deep learning may achieve competitive or superior performance on very large data sets, especially those with predominantly continuous features.
  • Decision Tree Ensembles are still State-of-the-Art.: For now, current deep learning techniques are generally not beneficial for small heterogeneous tabular data sets.The authors argue that understanding the reasons for the performance gap is more important than continually proposing more complex models.
  • Unified Benchmarking.: The survey identifies standardized, unified benchmarks as necessary because data-set and hyperparameter choices can materially affect performance assessments.The authors release an open-source benchmark for deep learning models on tabular data.
  • Tabular Data Preprocessing.: Transforming heterogeneous tabular inputs into homogeneous representations can substantially improve performance with small additional overhead.
  • Architectures for Deep Learning on Tabular Data.: Transformer-based architectures increasingly use attention over categorical and numerical features, while self-supervised or unsupervised pre-training is gaining popularity.
  • Regularization Models for Tabular Data.: Regularization can reduce hypersensitivity and improve the robustness and accuracy of deep neural networks on tabular data.
  • Learning From Evolving Data Streams.: Deep learning methods are often unsuitable for evolving data streams because observations require single-pass processing under limited resources and concept drift.Incremental decision trees are often preferred in these applications.

B. Open Research Questions

The survey identifies unresolved research questions spanning data augmentation, self-supervised learning, information-preserving encodings, computational efficiency, regularization, generation, interpretability, streaming, and transfer learning for heterogeneous tabular data.

  • Information-theoretic Analysis of Encodings: Information-theoretic analyses are needed to understand and improve potentially lossy encodings of heterogeneous tabular data.Efficient, almost lossless transformations remain challenging, and their underlying mechanisms are not yet well understood.
  • Computational Efficiency in Hybrid Models: Hybrid models face a predictive-performance versus computational-complexity trade-off, with training and inference times exceeding those of classical approaches.The integration of state-of-the-art classical and deep-learning methods remains inconclusive.
  • Specialized Regularizations: Future work should develop context- and architecture-specific regularization while investigating theoretical constraints and fairness across classes.Weight decay and data augmentation have been reported to produce unfair models across classes.
  • Novel Processes for Tabular Data Generation: Tabular generation remains challenged by dependency and categorical-distribution modelling, while diffusion models and entirely new generative processes remain insufficiently explored.Modified GANs and VAEs are prevalent, but the survey identifies a need for domain-focused alternatives.
  • Interpretability: Reliable counterfactual explanations for heterogeneous tabular data under feasibility constraints remain unsolved despite their potential for fairness and personalized decision-making.The practical deployment of counterfactual explanation methods is constrained by tabular heterogeneity.
  • Transfer Learning for Tabular Data: There are no efficient, generally accepted transfer-learning methods for tabular data, motivating research on sharing knowledge across related data sets.The survey also calls for methods suited to evolving and heterogeneous tabular data streams.
  • Data Augmentation for Tabular Data: Tabular data augmentation still needs simple, effective transformations because existing methods may fail to model dependencies and reported minority-oversampling gains are marginal.The survey highlights latent-space generation as promising, but notes only marginal reported improvements.
  • Self-supervised Learning: Only a few recent studies address self-supervised learning for heterogeneous tabular data, leaving broader use of unlabelled data underexplored.Self-supervised learning could reduce reliance on expensive labelled data, as demonstrated in other domains.
Loading 2110.01889v3…