Source-linked AI summary
Generalization Can Emerge in Tabular Foundation Models From a Single Table
Junwei Ma, Nour Shaheen, Alex Labach, Amine Mhedhbi, Frank Hutter, Anthony L. Caterini, Valentin Thomas
TL;DR
The paper asks whether broad tabular in-context generalization truly requires massive synthetic or real-data pre-training. Using a shared transformer architecture and self-supervised pre-training on single real tables, it finds strong cross-domain transfer and links performance chiefly to the number of tasks constructed from available features.
Problem
Tabular domains are considered highly heterogeneous, creating doubt that models can transfer across domains without massive synthetic or real-data pre-training.
Method
The paper pre-trains tabular in-context models from scratch on single real datasets using a self-supervised procedure and evaluates them across heterogeneous benchmarks.
Results
Single-table pre-training transfers strongly across unrelated datasets, while increasing task counts from 5 to tens of thousands raises performance from AUC = 0.5 to good performance.
Takeaways & Limitations
The number and quality of feature-target tasks constructed from a dataset matter more for generalization than pre-training data volume alone.
Abstract
from arXiv · showhide
Deep tabular modelling increasingly relies on in-context learning where, during inference, a model receives a set of $(x,y)$ pairs as context and predicts labels for new inputs without weight updates. We challenge the prevailing view that broad generalization here requires pre-training on large synthetic corpora (e.g., TabPFN priors) or a large collection of real data (e.g., TabDPT training datasets), discovering that a relatively small amount of data suffices for generalization. We find that simple self-supervised pre-training on just a \emph{single} real table can produce surprisingly strong transfer across heterogeneous benchmarks. By systematically pre-training and evaluating on many diverse datasets, we analyze what aspects of the data are most important for building a Tabular Foundation Model (TFM) generalizing across domains. We then connect this to the pre-training procedure shared by most TFMs and show that the number and quality of \emph{tasks} one can construct from a dataset is key to downstream performance.
1 Introduction
Although tabular domains are widely considered too heterogeneous for transfer, the paper shows that self-supervised pre-training on a single real table can generalize across unrelated datasets. It investigates which dataset properties and task-construction choices support this transfer.
- 1 Introduction: Existing tabular foundation models typically use massive synthetic corpora or large collections of real datasets, with randomized targets and contexts to encourage in-context generalization.Their shared pre-training procedure selects a dataset, uses one column as the target, and samples other columns as features.
- 1 Introduction: Single-table pre-training can transfer across unrelated tabular domains, challenging the assumption that broad generalization requires massive or domain-matched corpora.A transformer trained only on vectorized MNIST achieved strong in-context performance on California Housing, while COLLEGES pre-training was evaluated across CC-18 and CTR-23.
- 1 Introduction: The paper studies which properties of single real-world pre-training datasets are associated with generalization across tabular in-context learning benchmarks.Its reported factors include dataset-wide transfer consistency, feature count, instance count, and the number of unique pre-training tasks.
- 1 Introduction: The work argues that even limited data can support tabular generalization and motivates further study of what models can learn from small pre-training datasets.The authors describe this as a surprising phenomenon that opens further investigation into learning from limited data.
2 Experiments
Using a fixed tabular ICL architecture and single-table self-supervised pre-training, the experiments test which dataset properties and task constructions support transfer across diverse benchmarks. They find that transfer is broadly universal, feature-rich datasets outperform instance-rich ones, and increasing the number of pre-training tasks strongly improves generalization.
- Experimental setup: Across 88 single-table pre-training datasets and 107 diverse evaluation datasets, the study evaluates transfer while keeping the architecture and pre-training procedure fixed.The evaluation includes 72 CC-18 classification datasets and 35 CTR-23 regression datasets, with no overlap between pre-training and evaluation corpora.
- 2.2 Are Good Pre-training Datasets Universally Good?: If a pre-training dataset transfers well to one evaluation task, it generally transfers well to others, including across substantially different tasks.Ranking correlations are predominantly positive, and the domain analysis finds no apparent same-domain transfer advantage; some domains instead perform well across domains.
- 2.3 What Constitutes a Good Dataset?: Dataset meta-features predict average downstream generalization with R2 = 0.67, reproducing roughly two-thirds of its variance from straightforward properties.The predictor uses features such as feature and instance counts, categorical and numeric feature counts, missingness, and pre-training losses.
- 2.3 What Constitutes a Good Dataset?: The number of features is the strongest predictor of downstream transfer, whereas the number of instances contributes negligibly to predictive power.On COLLEGES, removing about 70% of features reduces performance to a level similar to a linear model, while removing more than 70% of instances preserves performance near a random forest.
- 2.4 Training on Many Good Tasks Unlocks Generalization: Because each target-and-feature selection defines a task, datasets with more columns expose models to many more feature-target relationships during pre-training.The authors note that randomized targets and feature subsets can yield O(k 2k) tasks for k columns, linking feature richness to task construction.
- 2.4 Training on Many Good Tasks Unlocks Generalization: Increasing unique pre-training tasks from 5 to tens of thousands raises downstream performance from barely above random (AUC = 0.5) to good performance across datasets.The experiments vary task count without changing the number of instances or features, and the authors identify task quantity as the main factor explaining cross-domain generalization while leaving task diversity and quality for future work.
A.1 Impact of Pre-training Dataset on CC-18 and CTR-23 Performance
Tables 1 and 2 compare TabDPT models pre-trained on different datasets across the CC-18 classification and CTR-23 regression benchmarks.
- The evaluation covers CC-18 classification using AUC and CTR-23 regression using R2.
- Each table reports results for all pre-training datasets alongside their instance counts, feature counts, and domains.
- The tables sort pre-training datasets in descending benchmark performance to identify datasets associated with better results.