PaperlayerView exact revision on arXiv

Source-linked AI summary

TransTab: Learning Transferable Tabular Transformers Across Tables

Zifeng Wang, Jimeng Sun

arXiv:2205.09328v2cs.LGcs.AI

TL;DR

Existing tabular ML methods assume fixed columns across training and testing, making learning across tables with divergent structures difficult. TransTab contextualizes columns and cells with gated transformers and pretraining, outperforming baselines in supervised learning and benefiting from cross-table knowledge transfer.

  • Problem

    Existing tabular ML methods require the same table structure during training and testing, while real-world tables can have partially overlapping columns and divergent column or cell nomenclatures.

  • Method

    TransTab accepts variable-column inputs by converting column descriptions and cell values into contextualized sequences processed with gated transformers, with supervised and self-supervised pretraining.

  • Results

    TransTab outperforms baselines across the clinical-trial supervised-learning comparison and benefits from knowledge transfer across tables.

  • Takeaways & Limitations

    Contextualizing columns and cells supports applying tabular models across different tables, while variable-column inputs and pretraining support broader tabular learning scenarios.

  • Takeaways & Limitations

    The paper notes that table phenotypes still need investigation to identify tables likely to benefit from transfer learning.

Abstract

from arXiv · show

Tabular data (or tables) are the most widely used data format in machine learning (ML). However, ML models often assume the table structure keeps fixed in training and testing. Before ML modeling, heavy data cleaning is required to merge disparate tables with different columns. This preprocessing often incurs significant data waste (e.g., removing unmatched columns and samples). How to learn ML models from multiple tables with partially overlapping columns? How to incrementally update ML models as more columns become available over time? Can we leverage model pretraining on multiple distinct tables? How to train an ML model which can predict on an unseen table? To answer all those questions, we propose to relax fixed table structures by introducing a Transferable Tabular Transformer (TransTab) for tables. The goal of TransTab is to convert each sample (a row in the table) to a generalizable embedding vector, and then apply stacked transformers for feature encoding. One methodology insight is combining column description and table cells as the raw input to a gated transformer model. The other insight is to introduce supervised and self-supervised pretraining to improve model performance. We compare TransTab with multiple baseline methods on diverse benchmark datasets and five oncology clinical trial datasets. Overall, TransTab ranks 1.00, 1.00, 1.78 out of 12 methods in supervised learning, feature incremental learning, and transfer learning scenarios, respectively; and the proposed pretraining leads to 2.3% AUC lift on average over the supervised learning.

1 Introduction

TransTab addresses the fixed-table assumption that limits learning across tables with partially overlapping columns. It combines column and cell semantics with transferable pretraining to support variable-column tabular learning.

  • Existing tabular models require identical training and testing table structures, limiting learning across tables with partially overlapping columns.Traditional preprocessing removes non-overlapping columns and mismatched samples before training.
  • TransTab contextualizes column descriptions and cell values as sequence inputs, making its featurization protocol generalizable across tables.For example, it represents a cell as “gender is man” rather than encoding man through a table-specific numeric codebook.
  • TransTab is a versatile tabular learning framework designed for variable-column inputs and multiple tabular modeling scenarios.The framework relaxes the fixed-column assumption to cover four additional scenarios beyond prior same-structure supervised learning and pretraining.
  • Its featurizing pipeline jointly considers column and cell semantics as a shared protocol across tables.
  • Vertical-Partition Contrastive Learning enables pretraining on multiple tables and supports finetuning on target datasets.

2 Method

TransTab handles variable-column tables by contextualizing column descriptions with cell values, encoding them through gated transformers, and supporting supervised or contrastive learning. Its vertical-partition pretraining extends this representation across heterogeneous tables.

  • Gated transformers: The framework feeds token embeddings through stacked gated transformer layers, whose attention and token-wise gates filter features before the final [cls] embedding is used for prediction.The input processor produces E, while the learning module uses a classifier for labeled prediction and a projector for contrastive learning.
  • Application scenarios: TransTab targets transfer, incremental learning, unlabeled pretraining, and zero-shot inference scenarios that fixed-column methods cannot directly support.These scenarios involve partially overlapping tables, newly added columns, unavailable labels, or inference tables with different columns.
  • Input processor: The input processor accepts variable-column tables by converting column descriptions and cell values into semantically encoded tokens.Categorical and textual features concatenate column names with values; binary features are encoded only when active; numerical values are combined with column embeddings.
  • Input processor: Contextualizing cells with their column properties enables TransTab to recognize semantically equivalent values across differently named columns.The paper contrasts “previously smoked” with “smoking history” and notes that values such as 1 can be interpreted equivalently in both contexts.
  • Pretraining: Vertical-Partition Contrastive Learning (VPCL) pretrains TransTab using column partitions, forming positive pairs from same-sample or same-class views.Self-VPCL uses partitions from the same sample, while supervised VPCL uses views from the same class; random sampling of two partitions is used when K > 2.
  • Pretraining: Supervised pretraining with task-specific heads can undermine transferability because tabular datasets differ in size, task definition, class distribution, and suitable hyperparameters.The paper motivates VPCL as an alternative to vanilla supervised-loss pretraining across heterogeneous datasets.

3 Experiments

Experiments evaluate TransTab across supervised, feature-incremental, cross-table transfer, zero-shot, and pretraining settings using clinical-trial and public tabular datasets. Across these scenarios, TransTab outperforms baselines or benefits from cross-table knowledge, while unrelated-table pretraining provides few finetuning benefits.

  • Experimental setup: The experiments use clinical-trial mortality datasets and public tabular datasets, with clinical-trial mortality prediction framed as a binary classification task.The clinical-trial datasets contain distinct patient groups and columns and support transfer-learning evaluation within a similar domain.
  • Q1. Supervised learning: Under supervised learning without pretraining, TransTab outperforms the baselines on all clinical-trial mortality datasets.On public datasets, TransTab is reported as comparable to state-of-the-art baseline tabular models.
  • Q2. Feature incremental learning: TransTab outperforms baselines by a great margin in feature-incremental learning by training across all three subsets with incremental features.Baselines instead use either older data with the initial features or only data from the final subset.
  • Q3. Transfer learning: TransTab benefits from knowledge transfer across tables when trained on one partially overlapping subset and finetuned on the other.The transfer setup uses two subsets with 50% column overlap, and similar observations are reported on public datasets.
  • Q4. Zero-shot learning: In zero-shot evaluation, TransTab learns from two disjoint-column subsets and predicts on a third unseen subset without further training; sensitivity checks report reasonable predictions with no column overlap.The zero-shot model performs better than the supervised model on average in the described experiment.
  • Q5. Supervised and self-supervised pretraining: VPCL consistently outperforms the Supervised baseline in public-dataset pretraining ablations, whereas unrelated-table pretraining usually yields few finetuning benefits.VPCL also shows an advantage over self-VPCL as the number of partitions increases, while table phenotype aggregation remains an open investigation.

4 Related Works

Prior tabular work improves prediction and representation learning, but generally assumes fixed table structures and offers limited transfer across tables.

  • Tabular Prediction: Tabular prediction research has introduced numerous algorithms, while boosting methods and MLPs remain competitive, especially with small sample sizes.
  • Tabular Prediction: Self-supervised pretraining on unlabeled tabular data was introduced to address label scarcity.
  • Existing tabular prediction and self-supervised methods do not generally provide transferable modeling across tables with differing structures.
  • Transfer learning: Transfer learning commonly pretrains on large general datasets and fine-tunes on smaller downstream tasks in vision and NLP.
  • Self-supervised learning & contrastive learning: Tabular self-supervised methods include reconstruction approaches such as TabNet and VIME and contrastive approaches such as SCARF and SubTab.

5 Conclusion

TransTab accepts variable-column tabular inputs and uses vertical partition contrastive learning to support pretraining across multiple tables with low memory cost.

  • TransTab accepts variable-column inputs and is proposed as a basis for tabular foundation models and tabular applications.
  • Vertical Partition Contrastive Learning enables supervised pretraining on multiple tabular datasets with low memory cost and supports target-dataset fine-tuning.

Checklist

The checklist reports that the paper addresses reproducibility, training details, error bars, compute reporting, asset attribution, ethics, and broader-impact considerations.

  • Ethics: The paper reports addressing claim accuracy, consent, personally identifiable information, offensive content, participant risks, and compensation applicability.
  • Broader impact: The paper reports discussing limitations, potential negative societal impacts, and conformity with ethics review guidelines.
  • Theory: The paper reports that theoretical-results checklist items are not applicable because it includes no theoretical results.
  • Reproducibility: The paper reports that code, data, reproduction instructions, training details, and baseline specifications are provided.
  • Experimental reporting: The paper reports repeated experiments with error bars and documents the total compute and resource types used.
  • Assets: The paper reports citations and licenses for existing assets and identifies new assets provided in supplementary material or by URL.

A Broader impact of this work

TransTab is presented as a transfer-learning approach for tabular data that can reduce preprocessing demands, while introducing privacy and resource concerns.

  • TransTab handles variable-column input tables, permits missing features, and is described as potentially reducing data-engineering workload.
  • The paper connects variable-column inputs and pretraining to potential development of foundation models for tabular data.
  • TransTab may require additional privacy work because it performs better with descriptive feature texts than with discretized indices.
  • TransTab requires more resources than MLPs and tree-based models because of full attention and feature tokenization.

B Baseline architecture and implementation

The experiments compare TransTab with standard linear, boosting, neural, and self-supervised baselines using specified implementations and training settings on public binary-classification datasets.

  • Baseline implementations: Logistic regression uses scikit-learn defaults except for a maximum of 1000 iterations.
  • Baseline implementations: XGBoost varies the maximum estimators over {50, 100, 500} and maximum depth over {4, 6, 8}.
  • Baseline implementations: MLP and SNN share three dense layers with hidden dimensions 256, 256, and 1, while differing in activation functions.Both use dropout with rate 0.1 and tune batch size and learning rate.
  • Dataset context: The public benchmark consists of binary classification datasets, with positive-class ratios reported in the dataset statistics.Source links are provided separately in Table 13.
  • Evaluation: Table 8 reports test AUROC results for the public datasets under supervised learning.
  • Baseline implementations: VIME is reproduced in PyTorch with one-layer encoders, mask estimators, and decoders whose hidden dimension matches the input features.Pretraining uses mask rate 0.3, batch size 128, and learning rate 1e-4.

C Preprocessing of clinical trial datasets

Clinical-trial records come from Project Data Sphere and are converted into patient-level binary prediction datasets using baseline clinical features and survival-derived labels.

  • Data source: The clinical-trial records are real-world, de-identified, patient-level data obtained from Project Data Sphere.The data platform provides access under registration.
  • Evaluation context: The supplied evaluation tables include public-dataset AUROC results for transfer learning across tables.
  • Feature construction: Features include demographics, medical history, medication history, laboratory tests, vital signs, and adverse events.
  • Label construction: Survival-analysis censoring is labeled “alive,” while other events are labeled “mortality” to form binary prediction tasks.

D Establishment of subsets

The experiments construct controlled subsets for feature-incremental, transfer, and zero-shot learning, while separately examining partition counts and column overlap.

  • Subset construction: Random subsets with fixed seeds are created separately for the experiments in Sections 3.2, 3.3, and 3.4.The subsets therefore vary across experiments.
  • Feature incremental learning: Feature-incremental learning uses three nested column sets: v1, then v1 and v2, then v1, v2, and v3, with equal sample counts.
  • Transfer learning: Transfer learning splits columns into v1 and v2 with 50% element overlap, using equal sample counts in both sets.
  • Zero-shot learning: Zero-shot learning splits columns into three distinct parts and assigns separate parts to the three sets.
  • Partition analysis: Figures 4 and 5 analyze the number of partitions for VPCL and self-VPCL on clinical-trial and public datasets, respectively.
  • Overlap analysis: Figure 6 relates the ratio of test columns present in training columns to zero-shot test AUC on four public datasets.The overlap ratio ranges from 0, with no shared test columns, to 1, with all test columns present in training.
Loading 2205.09328v2…