PaperlayerView exact revision on arXiv

Source-linked AI summary

TabTransformer: Tabular Data Modeling Using Contextual Embeddings

Xin Huang, Ashish Khetan, Milan Cvitkovic, Zohar Karnin

arXiv:2012.06678v1cs.LGcs.AI

TL;DR

The paper addresses limitations of deep tabular models, including weak performance, robustness, and interpretability, and limited semi-supervised learning. It introduces TabTransformer, which uses Transformer layers to create contextual categorical-feature embeddings and supports two-phase pre-training and fine-tuning. TabTransformer outperforms MLPs and recent deep networks, matches GBDT performance, and improves robustness, interpretability, and semi-supervised results within the reported evaluations.

  • Problem

    MLPs and existing deep tabular models have limitations in performance, interpretability, robustness to missing or noisy features, and semi-supervised learning compared with tree-based ensembles.

  • Method

    TabTransformer applies self-attention Transformer layers to categorical-feature embeddings, producing contextual embeddings and enabling two-phase pre-training on unlabeled data followed by fine-tuning on labeled data.

  • Results

    TabTransformer outperforms MLPs and recent deep networks, matches tree-based ensemble performance, and shows robustness, interpretability, and improved semi-supervised performance.

  • Takeaways & Limitations

    Contextual embeddings provide a tabular modeling approach that combines deep-learning representations with competitive tree-ensemble accuracy and supports semi-supervised pre-training.

  • Takeaways & Limitations

    With only 50 labeled data points, MLP methods using entropy regularization or pseudo-labeling outperform TabTransformer variants, partly because the classifier weights are not trained with unlabeled data.

Abstract

from arXiv · show

We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning. The TabTransformer is built upon self-attention based Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy. Through extensive experiments on fifteen publicly available datasets, we show that the TabTransformer outperforms the state-of-the-art deep learning methods for tabular data by at least 1.0% on mean AUC, and matches the performance of tree-based ensemble models. Furthermore, we demonstrate that the contextual embeddings learned from TabTransformer are highly robust against both missing and noisy data features, and provide better interpretability. Lastly, for the semi-supervised setting we develop an unsupervised pre-training procedure to learn data-driven contextual embeddings, resulting in an average 2.1% AUC lift over the state-of-the-art methods.

1 Introduction

Tabular data is widespread, but deep models such as MLPs face interpretability, robustness, semi-supervised-learning, and performance limitations relative to tree ensembles. TabTransformer adapts self-attention Transformers to contextualize categorical-feature embeddings and addresses these limitations through supervised and semi-supervised evaluation.

  • Tabular data dominates applications such as recommender systems, online advertising, portfolio optimization, and machine-learning competitions.
  • Tree-based ensembles remain the tabular-data state of the art because they offer competitive accuracy, fast training, and interpretability, despite limitations for streaming and semi-supervised learning.
  • MLPs use shallow, context-free categorical embeddings that are not interpretable or robust to missing and noisy data and generally do not match GBDT performance.
  • The proposed model is evaluated on fifteen public datasets to address MLP limitations and bridge the performance gap with GBDT models.
  • TabTransformer applies multi-head Transformer layers to categorical-feature embeddings, producing contextual embeddings by aggregating information across features.
  • The paper studies contextual-embedding interpretability, robustness to missing and noisy features, and two-phase semi-supervised pre-training followed by fine-tuning.

2 The TabTransformer

TabTransformer embeds categorical features, transforms them through stacked self-attention layers into contextual embeddings, and combines them with continuous features for prediction. Its design includes column-specific identifiers and an optional unlabeled-data pre-training procedure.

  • TabTransformer consists of a column embedding layer, N Transformer layers, and a top multi-layer perceptron.
  • Transformer layers: Stacked Transformer layers repeatedly aggregate information across feature embeddings to produce contextual embeddings for each categorical feature.Each layer uses multi-head self-attention followed by a position-wise feed-forward layer.
  • Prediction: The contextual embeddings are concatenated with continuous features and passed to an MLP that predicts the target.The concatenated vector has dimension d × m + c and is trained end-to-end.
  • Column embedding: Each categorical feature is mapped to an embedding, with an additional embedding reserved for missing values.The embedding table for feature i contains d_i + 1 embeddings.
  • Column embedding: Column-specific identifiers distinguish classes from different feature columns because tabular features have no intrinsic ordering.The identifiers are concatenated with feature-value-specific embeddings rather than using positional encodings.
  • Semi-supervised learning: Unlabeled data can pre-train the Transformer and embeddings before lightweight supervised fine-tuning on labeled data.The paper explores masked language modeling and replaced token detection as pre-training procedures.

3 Experiments

Experiments on 15 public binary-classification datasets evaluate TabTransformer’s architecture, embedding behavior, robustness, and supervised and semi-supervised performance. Across these evaluations, Transformer-based contextual embeddings improve over MLP baselines, match GBDT performance, and are especially effective with abundant unlabeled data.

  • Experimental Setup: 15 public binary-classification datasets were evaluated using five cross-validation splits, AUC, and supervised and semi-supervised settings.Semi-supervised scenarios used 50, 200, or 500 labeled training examples.
  • Transformer Effectiveness: 1.0% average AUC gain was achieved by TabTransformer over MLP, with improvement on 14 of 15 datasets.The comparison removed the Transformer layers while keeping the remaining architecture fixed.
  • Embedding Interpretability: Transformer contextual embeddings formed semantically meaningful clusters, whereas MLP embeddings showed no comparable pattern in the BankMarketing visualization.Examples include associations among client features, housing-loan and default status, and education and occupation.
  • Embedding Effectiveness: Embeddings became more effective for linear prediction as Transformer depth increased, with final-layer embeddings approaching the end-to-end TabTransformer score.The evaluation used logistic regression with layer-specific embeddings and continuous variables, normalized by the best end-to-end score.
  • Robustness and Supervised Performance: TabTransformer was more robust than MLP as randomly replaced feature values increased, while supervised results placed TabTransformer among the top performers and comparable with GBDT.Across 15 datasets, TabTransformer again exceeded MLP by an average 1.0% AUC gain.
  • Semi-supervised Learning: 1.2%, 2.0%, and 2.1% mean-AUC improvements were reported for 50, 200, and 500 labeled examples when unlabeled data were large.TabTransformer-RTD and TabTransformer-MLM significantly outperformed the other competitors in that setting; with fewer unlabeled data, RTD retained a marginal improvement.

4 Related Work

The paper situates tabular learning among supervised and semi-supervised methods, including MLPs, specialized deep models, and classical semi-supervised techniques.

  • Supervised learning: Deep tabular methods include factorization machines, Transformer-based models, and deep decision-tree algorithms.
  • Semi-supervised learning: Pseudo labeling, label propagation, entropy regularization, and co-training are established approaches for semi-supervised tabular learning.
  • Semi-supervised learning: Recent entropy-regularization and pseudo-labeling methods are described as typically better and more popular than co-training.

5 Conclusion

The conclusion presents TabTransformer as a deep architecture for supervised and semi-supervised tabular learning, with empirical gains over deep baselines and performance matching tree ensembles.

  • TabTransformer significantly outperforms MLPs and recent deep tabular networks while matching tree-based ensemble models such as GBDT.
  • Its two-phase pre-training and fine-tuning procedure beats state-of-the-art semi-supervised learning methods.
  • The model shows promising robustness to noisy and missing features and interpretability of its contextual embeddings.
  • The paper identifies detailed investigation of robustness and contextual-embedding interpretability as future work.

A Appendix: Ablation Studies

The appendix evaluates architectural and pre-training choices, including column embeddings, replacement rates, and dynamic versus static replacement strategies.

  • The ablations examine column-embedding choices, replacement value k, and dynamic versus static replacement during pre-training.The experiments also compare shared and separate RTD classifiers.
  • Column Embedding: The column-embedding study varies the dimension of shared column identifiers and compares concatenation with element-wise addition.
  • The replacement value k: The replacement-value study tests k ∈ {15, 30, 50} on Adult, BankMarketing, and 1995 income with 256 labeled examples.Final fine-tuning accuracy is reported as not very sensitive to k.
  • Dynamic versus Static Replacement: Dynamic replacement changes randomly replaced feature values across epochs, whereas static replacement reuses one replacement pattern throughout training.
  • Dynamic versus Static Replacement: Dynamic replacement with unshared RTD classifiers performs better than static replacement and shared RTD classifiers.Figure 7 shows training and validation accuracy curves for the three compared configurations.

B Appendix: Experiment and Model Details

This appendix section introduces the experiment details and model-design information, directing readers to separate subsections for hyperparameter optimization and feature engineering.

  • The section covers experiment details, hyperparameter-search space, and feature engineering in Sections B.1 and B.2.

B.1 Experiments Details and Hyper Parameters

The experiments use controlled data splits, standardized optimization and tuning procedures, and comparisons across supervised and semi-supervised models. TabTransformer configurations and ablations are evaluated using AUC-based tables and pre-training curves.

  • Baselines: Baseline implementations include LightGBM GBDT, Sparse MLP, TabNet, VIB, MLP with denoising pre-training, pseudo labeling, and entropy regularization.Each baseline has model-specific implementation or hyperparameter details, including pruning, attention, variational sampling, or unsupervised coefficients.
  • Experimental setup: All models receive identical cross-validation splits and labeled and unlabeled training data through a fixed random seed.This keeps training and testing conditions consistent across models.
  • Optimization: Deep models use cross-entropy losses, AdamW optimization, constant learning rates, and validation-based early stopping.The cross-entropy setup covers supervised and semi-supervised training, while pre-training uses binary RTD and multiclass MLM objectives.
  • Ablations: The experiments compare column-embedding designs, replacement values, replacement strategies, and shared versus separate classifiers using relative AUC and dataset-level AUC.Tables 5–7 report these ablations, while Figures 6–7 show corresponding pre-training training and validation accuracy curves.
  • TabTransformer configuration: TabTransformer uses fixed hidden dimension 32, six Transformer layers, eight attention heads, and MLP sizes {4 × l, 2 × l} in the reported experiments.These choices were selected through hyperparameter optimization on five datasets, with broader search spaces also reported.

B.2 Feature Engineering

The feature-engineering discussion distinguishes categorical, scalar, and text variables, with processing choices tailored to each type. Learned embeddings generally help lower-cardinality categorical variables, while scalar encoding remains underexplored.

  • Categorical variables: Categorical variables can use one-hot encoding or learned parametric embeddings with choices for dimension and dropout regularization.Dropout may remove vector elements or entire embeddings.
  • Categorical variables: Learned embeddings nearly always improve performance when categorical cardinality is significantly below the number of data points.The passage qualifies this finding by relating it to categorical cardinality.
  • Scalar variables: Scalar variables can be quantile-rescaled, normalized, log-scaled, or quantized and treated as categorical variables.The authors suggest, but have not fully explored, combining multiple encodings in parallel.
  • Text variables: Text variables are represented by the number of words and characters.

C Appendix: Benchmark Dataset Information and Experiment Results

The appendix catalogs fifteen binary-classification benchmark datasets and reports supervised and semi-supervised AUC results. The evaluation tables summarize mean performance over five cross-validation splits under multiple fine-tuning data budgets.

  • Benchmark datasets: The benchmark suite contains binary classification datasets and reports each dataset’s positive-class fraction.
  • Benchmark datasets: Benchmark dataset links are provided in a dedicated appendix table.
  • Semi-supervised results: Semi-supervised results are reported for 50, 200, and 500 fine-tuning data points using mean AUC and standard deviation over five cross-validation splits.Tables 10–15 cover these three labeled-data budgets and state that larger AUC values indicate better results.
  • Supervised results: Supervised model results are reported across all datasets as mean AUC and standard deviation over five cross-validation splits.Tables 16–17 state that larger AUC values indicate better results.
Loading 2012.06678v1…