PaperlayerView exact revision on arXiv

Source-linked AI summary

XTab: Cross-table Pretraining for Tabular Transformers

Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, Mahsa Shoaran

arXiv:2305.06090v1cs.LG

TL;DR

Existing tabular self-supervised pretraining is generally domain-specific and does not generalize to new tables. XTab pretrains tabular transformers across tables using table-specific and shared components with federated learning; the authors report improved performance over random initialization and state-of-the-art deep tabular models.

  • Problem

    Existing tabular self-supervised methods generally pretrain on data from the downstream task’s domain, limiting generalization to new tables.

    Source: §1 · p. 1

  • Method

    XTab pretrains tabular transformers across diverse tables using table-specific components and a shared component trained via federated learning.

    Source: §1 · p. 1

  • Results

    XTab-pretrained transformers consistently outperform random-initialized transformers, and XTab-pretrained FT-Transformer outperforms state-of-the-art tabular deep learning models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pretrained shared block can initialize learning on a new table, and the authors report improved learning speed and performance on new tables.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

The success of self-supervised learning in computer vision and natural language processing has motivated pretraining methods on tabular data. However, most existing tabular self-supervised learning models fail to leverage information across multiple data tables and cannot generalize to new tables. In this work, we introduce XTab, a framework for cross-table pretraining of tabular transformers on datasets from various domains. We address the challenge of inconsistent column types and quantities among tables by utilizing independent featurizers and using federated learning to pretrain the shared component. Tested on 84 tabular prediction tasks from the OpenML-AutoML Benchmark (AMLB), we show that (1) XTab consistently boosts the generalizability, learning speed, and performance of multiple tabular transformers, (2) by pretraining FT-Transformer via XTab, we achieve superior performance than other state-of-the-art tabular deep learning models on various tasks such as regression, binary, and multiclass classification.

1. Introduction

Tabular data’s varying column types and counts make cross-table transfer difficult, while existing pretraining is generally domain-specific. XTab combines table-specific components with a shared component pretrained across tables, and its pretrained transformers outperform random-initialized transformers.

1. Introduction

  • Varying column types and counts make it difficult for tabular models to transfer learned knowledge between tables, leaving self-supervised learning that bootstraps new tables an open problem.Existing methods generally pretrain on data from the same domain as the downstream task and cannot generalize to new tables.Source: §1 · p. 1
  • Tabular transformers treat columns as tokens and can process variable numbers of columns, making transferable learning feasible.This contrasts with tables’ data-specific column types and quantities.Source: §1 · p. 1
  • XTab uses table-specific featurization and projection layers alongside a shared block, pretrained jointly across diverse tables via federated learning.The pretrained shared block initializes learning on a new table.Source: §1 · p. 1
  • XTab-pretrained transformers consistently outperform randomly initialized transformers, and XTab-pretrained FT-Transformer outperforms state-of-the-art tabular deep learning models.The introduction reports these comparisons without giving numerical results.Source: §1 · p. 2
  • The paper proposes federated pretraining across diverse datasets to support distributed training and reports improved learning speed and performance on new tables.The authors describe this as the first demonstration of cross-table pretraining boosting learning speed and performance on new tables.Source: §1 · p. 2

2. Related work

Prior tabular pretraining methods generally use one or a few relevant tables, limiting generalizability; XTab instead pretrains across many tables. It targets transferable weight initialization rather than a universal tokenizer, and differs from TransTab and TabPFN in its stated scope.

Tabular self-supervised learning

  • Existing tabular self-supervised methods pretrain on one or a few relevant tables, whereas XTab pretrains across many tables to address limited generalizability.The cited methods include reconstruction, contrastive, and target-aware objectives.Source: §2 · p. 2

Tabular transformers

  • Unlike TransTab, XTab is described as able to generalize to tables from new domains.The related-work passage characterizes TransTab as having limited new-domain generalization.Source: §2 · p. 2

Cross-table transfer learning

  • XTab seeks weight initialization transferable across downstream tasks, not a universal tokenizer, because table meanings and contexts vary.The authors motivate shared tabular knowledge by analogy to general hyperparameter configurations learned across tasks.Source: §2 · p. 2
  • Unlike TabPFN, which is described for small numerical classification tasks with ≤1000 samples, XTab’s inference complexity is stated to be independent of training-sample count.The authors therefore state that XTab also works for large tables.Source: §2 · p. 2

3. Methods

XTab separates table-specific components from a transformer backbone shared across tables, enabling pretraining across diverse tables and downstream finetuning. Its method combines flexible tabular representations and objectives with federated training of the shared component.

3. Methods

  • XTab uses information from multiple tables to pretrain transformers, aiming to improve downstream regression and classification tasks.Only the shared backbone is retained for downstream tasks, where the model is finetuned on task training data.Source: §3 · p. 2, §3.1 · p. 3

3.1. Model structure

  • Table-specific featurizers and projection heads surround a shared transformer backbone; downstream tasks initialize new featurizers and heads before finetuning.Pretraining can use reconstruction, contrastive, or supervised prediction objectives.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.1.1. FEATURIZERS

  • Featurizers convert each column into a token embedding and accommodate tables with different numbers and types of columns.Text cells are treated as categorical attributes.Source: §3.1.1 · p. 3

3.1.2. BACKBONES

  • The shared backbone must process variable-length sequences; XTab presents FT-Transformer, Fastformer, and a variable-column Saint variant.Fastformer uses additive attention with linear sequence-length complexity; Saint-v combines column- and row-wise attention.Source: §3.1.2 · p. 3

3.1.3. PROJECTION HEADS AND OBJECTIVES

  • Data-specific projection heads allow different tables to use reconstruction, contrastive, or supervised prediction objectives, including mixtures across datasets.Reconstruction recovers an original sample from a corrupted view, while contrastive learning treats the sample and its corruption as a positive pair and other batch samples as negatives.Source: §3.1.3 · p. 4

3.2. Federated pretraining

  • Federated Averaging distributes datasets across clients and aggregates only shared weights; multiple local updates between aggregations reduce communication cost.Each client hosts one dataset and its data-specific components; federated training is intended to scale across many pretraining tasks.Source: §3.2 · pp. 4–5

4. Experiments

The experiments evaluate XTab across tabular prediction tasks, transformer backbones, pretraining objectives, and finetuning settings. Cross-table pretraining improves over randomly initialized transformers, and XTab-best ranks second overall in the benchmark while CatBoost remains best overall.

4. Experiments

  • The benchmark evaluation covers binary and multiclass classification and regression, testing several objectives, transformer backbones, and pretraining-task configurations.The listed configurations include reconstruction, contrastive, and supervised losses, as well as FT-Transformer, Fastformer, and Saint-v.Source: §4 · p. 5

4.1. Datasets

  • AMLB contains 104 tasks; the study uses 52 datasets for pretraining and the other 52 for finetuning and evaluation, split by task-name alphabetical order.The benchmark comparison reports results on 84 tasks after excluding 20 datasets that did not fit in GPU memory.Source: §4.1 · p. 5, §4.4 · p. 7

4.2. Experimental setup

  • Downstream tasks use a 10% test split, with five trials using different test folds; evaluation uses RMSE for regression, AUC for binary classification, and log loss for multiclass classification.The validation sets use the same metrics for early stopping.Source: §4.1 · p. 5, §4.2 · p. 6

4.3. Comparison with baseline transformers

  • Reconstruction, contrastive, and supervised objectives all improve over baseline transformers, with reconstruction performing best among the tested objectives.At 2000 steps, reconstruction achieves a 71.0% win rate with light finetuning and 56.1% with heavy finetuning.Source: Figure 3 · p. 6
  • Across classification and regression, XTab pretraining improves win rate, model rank, normalized performance, and error reduction as pretraining steps increase.FT-Transformer checkpoints are evaluated after 250, 500, 1000, 1500, and 2000 pretraining steps; tasks are finetuned for three epochs in this light-finetuning comparison.Source: §4.3 · p. 6
  • FT-Transformer, Fastformer, and Saint-v all benefit from XTab, each achieving a win rate above 50% in the reported experiments.The comparison uses reconstruction pretraining and evaluates both light and heavy finetuning.Source: Figure 4 · p. 7, §4.3 · p. 7

4.4. Performance compared to traditional baselines

  • XTab-best ranks second overall and beats other deep learning models; CatBoost remains the overall best model across the 84-task benchmark.XTab also improves over FT-Transformer under light and heavy finetuning, with its advantage over random initialization increasing when downstream training time is tighter.Source: Table 1 · p. 8, §4.4 · p. 8

5. Conclusion

XTab addresses cross-table variation by separating table-specific components from a shared transformer block and pretraining with federated learning. The paper reports improved performance over training transformers from scratch, while noting that XTab still trails CatBoost.

5. Conclusion

  • XTab separates data-specific and shared components to handle table variation, then uses federated learning to pretrain across diverse tables.Client-side nodes perform table reconstruction, followed by backbone averaging at the server.Source: §5 · p. 8
  • The results suggest that finetuning pretrained transformers outperforms training tabular transformers from scratch.Source: §5 · p. 8
  • XTab still falls behind CatBoost, motivating work to bridge the gap between pretrained tabular deep learning and tree models.The conclusion also identifies combining XTab with language or vision foundation models for multimodal learning as a future direction.Source: §5 · p. 8

Software and Data

XTab evaluates multiple pretraining and finetuning configurations and identifies Transformer blocks, rather than the data-specific [CLS] token, as the useful shareable component. The AutoML Benchmark and code/checkpoints are publicly available.

Software and Data

  • The AutoML Benchmark is publicly available, and the code and sample pretrained checkpoints are provided on GitHub.Source: Software and Data · p. 8

A. XTab performance with various pretraining/finetuning settings

  • The evaluation covers reconstruction, contrastive, and supervised pretraining objectives under light and heavy finetuning.Downstream performance is compared using win rate, model rank, normalized performance, and error reduction rate.Source: A. XTab performance with various… · p. 11

B. Identifying the shareable components in XTab

  • Pretraining all Transformer blocks without the data-specific [CLS] token is most beneficial; sharing the [CLS] token alone is hardly beneficial.Featurizers and projection heads are not shareable because input and output spaces can differ across tasks.Source: B. Identifying the shareable… · p. 12, Figure 6 · p. 12

C. Finetuning on subsampled datasets

XTab improves over the baseline with reduced finetuning data, but its advantage does not grow as the finetuning set shrinks. Featurizers and projection heads still require finetuning data, so XTab is not compatible with zero-shot learning.

C. Finetuning on subsampled datasets

  • XTab improves over the baseline at every tested finetuning-set size, but its advantage does not become more significant as data is reduced.The study subsamples downstream training sets to 25%, 50%, and 75% and evaluates light and heavy finetuning; training featurizers and projection heads from scratch still requires sufficient data, making XTab incompatible with zero-shot learning.Source: C. Finetuning on subsampled datasets · p. 12

D. Tuning the size of pretraining set

XTab performs better downstream when pretrained on more tabular tasks. In federated pretraining, increasing local FedAvg steps reduces downstream performance while reducing communication overhead.

D. Tuning the size of pretraining set

  • Downstream performance improves as XTab pretraining expands from 1 task to 18 and 52 tasks.The comparison uses FT-Transformer pretrained with reconstruction and evaluates win rate and model rank under light finetuning.Source: D. Tuning the size of pretraining set · p. 12, Figure 8 · p. 13
  • Figure 7 compares downstream performance after retaining 25%, 50%, 75%, or 100% of finetuning rows under light and heavy finetuning.Source: Figure 7 · p. 13

E. Tuning parameters of federated pretraining

  • Increasing FedAvg local steps from N = 1 to 5 and 10 decreases downstream performance while reducing communication overhead.N controls the number of local optimization steps between server aggregation steps, creating a trade-off between communication cost and downstream performance.Source: E. Tuning parameters of federated… · p. 13

F. Comparison to pretraining without external tasks

The comparison distinguishes pretraining on downstream data alone from using XTab initialization based on external tables. The passage reports that pretraining methods generally outperform the no-pretraining baseline.

F. Comparison to pretraining without external tasks

  • The external-task comparison holds downstream pretraining and finetuning constant, varying whether XTab-pretrained initialization is used.The alternative pretrained model uses the downstream training set; the baseline uses no pretraining.Source: F. Comparison to pretraining without… · pp. 13–14
  • Pretraining methods generally outperform the no-pretraining baseline.The cited passage gives a 55.7% win rate but repeats “w/ external task” on both sides of the comparison, so it does not establish a valid between-method comparison.Source: F. Comparison to pretraining without… · p. 14

G. Implementation of Saint-v

Saint-v adapts Saint’s row-attention architecture to tables with variable column counts, while XTab pretraining shifts Transformer weights toward a normal distribution as pretraining increases.

G. Implementation of Saint-v

  • Saint-v transposes the batch and token dimensions without changing embedding size, allowing it to process tables with variable columns.Unlike Saint, its trainable parameter count is not dependent on token count.Source: G. Implementation of Saint-v · p. 14

H. Visualization of pretrained weights

  • With increased XTab pretraining, Transformer weight distributions converge from Kaiming uniform initialization toward a normal distribution.The visualization excludes layer-normalization and bias terms.Source: H. Visualization of pretrained weights · p. 14

I. Benchmark configurations

The benchmark configures tree, neural-network, and transformer baselines with specified training and hyperparameter-search settings, alongside XTab and TransTab variants.

I.1. Tree-based models

  • Tree-ensemble models use AutoGluon 0.5.3 defaults for hyperparameters, early stopping, and preprocessing; boosted-tree patience is 300 below 10k rows, reduced by a factor of num rows/10k above that, with a minimum of 20.For Random Forest, max features = auto means max features = sqrt(n features).Source: §I.1 · pp. 14–15

I.2. Neural network and FastAI

  • The neural-network baselines are AutoGluon’s PyTorch tabular network with ReLU and the AutoGluon implementation of FastAI tabular, which configures input embedding sizes automatically.Their default hyperparameters and HPO search spaces are used.Source: §I.2 · p. 15

I.3. TransTab

  • TransTab is evaluated only on classification because version 0.0.3 does not support regression, and its contrastive variant outperforms its supervised variant in this benchmark.The contrastive setup uses target-aware contrastive learning and AutoML Benchmark column-type information.Source: §I.3 · p. 16

I.4. FT-Transformer

  • FT-Transformer is tested with light, heavy, and model-soup configurations, with improved prediction performance from FTT-l to FTT-best as training time increases.FTT-best averages the three best validation checkpoints.Source: §I.4 · p. 17

I.5. XTab

  • XTab uses the FT-Transformer structure initialized with pretrained parameters, reconstruction-loss pretraining, and 2000 default federated rounds.The number of pretraining iterations is also considered as an HPO hyperparameter.Source: §I.5 · p. 17

J. Dataset statistics

The evaluation uses two-fold pretraining and downstream splits with minimal task overlap, then reports raw performance on the 84 tasks retained after exclusions.

J. Dataset statistics

  • The 104-dataset benchmark is split into two folds, alternating which fold pretrains XTab and which evaluates downstream performance.The authors describe the resulting overlap between pretraining and downstream tasks as minimal.Source: J. Dataset statistics · p. 17, Table 13 · p. 21

K. Raw prediction performance

  • Raw performance is reported for 84 downstream tasks after 20 datasets were excluded, with results averaged across five trials.The raw tables cover AUC for binary classification, log loss for multiclass classification, and RMSE for regression.Source: K. Raw prediction performance · p. 18, Table 15 · p. 23
  • Some failures constrained the comparison: FT-Transformer and XTab encountered out-of-memory errors at batch size 128 and timeouts under a 1-hour budget.Random Forest also had an out-of-memory error.Source: K. Raw prediction performance · p. 21
  • The HPO raw-performance table uses the search spaces specified in Appendix I.The table caption identifies its setting but does not provide comparative values in the supplied passage.Source: Table 16 · p. 24

Related papers

Loading 2305.06090v1…