PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding the Surprising Generalization Properties of Tabular Foundation Models

Nour Shaheen, Junwei Ma, Alex Labach, Frank Hutter, Valentin Thomas, Anthony L. Caterini

arXiv:2608.17957v1cs.LG

TL;DR

TFMs’ ability to generalize beyond their training tables remains poorly understood. This paper studies single-table pre-training, task-centric corpus design, and retrieval-based behavior, finding that strong transfer can emerge from one real table and is linked to task diversity.

  • Problem

    Existing TFMs rely on massive synthetic or real-data corpora, while the mechanisms and data properties enabling cross-domain tabular generalization remain understudied.

  • Method

    The paper evaluates models pre-trained on individual real tables and larger corpora, analyzes task and column effects, and tests a retrieval-and-aggregation interpretation.

  • Results

    Strong transfer emerges from single-table pre-training; usefulness is broadly consistent across tasks, feature count predicts generalization, and column-level cleaning helps while dataset-level filtering does not.

  • Takeaways & Limitations

    TFM pre-training is task-centric, with task number and quality shaping generalization, while learned retrieval and aggregation offer an alternative account of model behavior.

  • Takeaways & Limitations

    The findings may not fully extend to exclusively synthetic pre-training, and additional mechanisms beyond learned retrieval cannot be definitively excluded.

Abstract

from arXiv · show

Tabular Foundation Models (TFMs) increasingly rely on in-context learning, where a model receives labelled examples at inference time and predicts labels for new inputs without updating its weights. Existing TFMs are typically trained on either massive synthetic corpora or very large collections of real datasets. In contrast, we show that surprisingly strong transfer can emerge from self-supervised pre-training on just a single real table. In this setting, we also find that tables tend to be either broadly useful or broadly poor regardless of downstream prediction task, and that the strongest predictor of usefulness is the number of features rather than the number of instances. This leads to a task-centric interpretation of tabular pre-training: the number and the quality of tasks are essential for the pre-training of TFMs. We show that the same task-centric perspective can help corpus design at scale: fine-grained column-level pre-processing consistently improves downstream performance, while no improvements are observed when we filter or deduplicate at the dataset level. Finally, we offer a new perspective for how TFMs generalize: we believe that tabular in-context generalization is largely retrieval-based, and good models are those that learn to identify relevant examples in the provided context and aggregate them well. The mechanics of TFMs have been relatively understudied; our task-centric, retrieval-based perspective offers a new framework to guide future model and corpus design.

1 Introduction

The paper shows that self-supervised pre-training on a single real table can yield robust cross-domain tabular in-context generalization. It develops a task-centric account of pre-training and a retrieval-and-aggregation perspective on how TFMs generalize.

  • Single-table transfer: A transformer trained from scratch on vectorized MNIST transfers strongly to structurally and semantically unrelated datasets such as CALIFORNIA HOUSING.The paper presents this as evidence that single-table self-supervised pre-training can produce robust cross-domain generalization.
  • What predicts transfer: Transfer performance is broadly consistent across evaluation sets, with feature count predicting generalization more strongly than instance count.The paper reports that datasets tend to transfer either well or poorly across evaluation sets, while instance count matters little.
  • Task-centric pre-training: The number of unique pre-training tasks strongly drives TFM generalization ability.This motivates a task-centric interpretation in which the number and quality of pre-training tasks are essential.
  • Corpus design: Fine-grained removal of duplicated, excessively correlated, or otherwise uninformative columns improves performance, whereas dataset-level filtering, including deduplication, is ineffective.The task-centric view remains useful for large corpora when data quality is controlled, and more sufficiently high-quality tasks generally help.
  • Mechanism of generalization: The paper argues that TFMs generalize through learned retrieval-and-aggregation procedures rather than a Bayesian prior requiring downstream tasks to be covered by its distribution.This perspective is proposed because single-table generalization conflicts with the prior-coverage requirement of the original Bayesian interpretation.

2 Generalization from single table training

Self-supervised pre-training on individual tables transfers broadly across downstream datasets, with dataset usefulness largely determined by feature richness and the diversity of induced prediction tasks. Increasing task diversity improves transfer, and task-poor datasets can complement stronger ones rather than being intrinsically uninformative.

  • Universality of dataset quality: Across 88 training and 107 evaluation datasets, tables that transfer well to one task generally transfer well to others, even across different tasks and domains.Relative rankings of training datasets are broadly stable across evaluation datasets, and same-domain pairs do not transfer better than cross-domain pairs.
  • Dataset properties: An XGBoost meta-feature model achieves an R2 of 0.67 for average downstream generalization, reproducing roughly two-thirds of its variance from dataset properties.The average score combines correlation for regression tasks and accuracy for classification tasks.
  • Dataset properties: The number of features is by far the strongest predictor of transfer, whereas the number of instances has negligible predictive power.This indicates that feature-space richness matters more than dataset size for tabular in-context pre-training.
  • Task-centric interpretation: Randomized target and feature selection turns columns into many possible prediction tasks, making task quantity and diversity central to why feature-rich tables transfer better.With k columns, randomized selection can produce O(k 2k) tasks, while a fixed supervised target provides only one task.
  • Task-centric interpretation: More unique tasks during pre-training consistently improve downstream transfer, even when the dataset’s number of instances and features remains fixed.The experiments vary only the number of allowed task configurations across several datasets.
  • Task-centric interpretation: Pairing a strong dataset with a weak one does not hurt performance and can slightly improve it, suggesting weak tables may provide complementary rather than useless tasks.COLLEGES (AUC = 0.918 alone) with ANALCATDATA_SUPREME (0.814 alone) reaches 0.919; APSFAILURE (0.918 alone) with POKER-HAND (0.777 alone) also reaches 0.919.

3 From one to many: pre-training insights for large corpora

For large heterogeneous tabular corpora, dataset-level deduplication and filtering do not improve performance, whereas fine-grained column pre-processing consistently does. These gains persist across model capacities and context lengths, supporting task quality and diversity as central to corpus design.

  • Dataset-level selection: Dataset-level deduplication and filtering out universally bad datasets are not beneficial for TFM pre-training.Full-corpus and deduplicated-corpus models are statistically indistinguishable on CC-18 and CTR-23.
  • Column-level pre-processing: Fine-grained column pre-processing improves task quality and diversity by removing missing, redundant, or uninformative features.The operations include dropping columns with more than 50% missing values, correlation deduplication, and skipping datasets with fewer than five surviving features.
  • Column-level pre-processing: Every pre-processing variant improves regression, with IQM R2 gains from +0.002 to +0.014, while classification gains are smaller but broadly consistent.All variants improve accuracy, and most improve AUC.
  • Scaling context and capacity: Pre-processing remains beneficial across model capacities and training context lengths, with the strongest overall configuration at 4k+pp.The scaled experiments compare deeper, wider models trained with 1k, 2k, and 4k maximum context lengths.
  • Scaling context and capacity: 4k+pp beats 4k on 67% of TabArena tasks, compared with 61% for 2k+pp versus 2k and 63% for 1k+pp versus 1k.These pairwise win rates cover TabArena’s 51 tabular prediction tasks.

4 Understanding what makes generalization possible

The section argues that tabular in-context generalization is primarily retrieval-based: strong models identify and use relevant labelled examples in context rather than relying on dataset-specific parametric transfer. Three analyses support this view through retrieval performance, similarity-constrained attention, and convergence of attention patterns across strong models.

  • Retrieval as the core mechanism: The retrieval interpretation explains transfer beyond the training distribution and under limited pre-training better than a prior centered on dataset-generating processes.The paper contrasts retrieval-based use of context with prior-fitted interpretations of TFM pre-training.
  • Overall interpretation: Together, the analyses suggest that TFMs generalize by learning how to identify and aggregate useful in-context examples at inference time.The section frames this as learning a general prediction algorithm rather than transferring predictive signals parametrically between datasets.
  • Retrieval as the core mechanism: A strong correlation links held-out generalization with exact-label retrieval from context, showing that models that generalize well also recover relevant support information.Across 88 single-table pre-trained models, the Pearson correlation between standard generalization and retrieval-oriented performance is 0.89.
  • Similarity-based attention: Constraining attention with WQ = WK improves weak single-table regimes, consistent with models benefiting from more explicit similarity-based aggregation.The simplified architecture wins on all three poorly generalizing tables—pollen, puma8NH, and poker-hand—by +0.1088.
  • Convergent retrieval patterns: Strong models converge toward similar retrieval patterns, supporting a shared context-use strategy rather than brittle, model-specific solutions.Attention-pattern similarity is positively associated with performance, with a reported Spearman correlation of 0.537.

5 Related Work

Prior tabular in-context learning work has emphasized scaling task diversity, whereas this paper examines extremely limited pre-training data and connects tabular ICL to retrieval-based methods. Its setting is distinct from language-model in-context learning because TFMs directly train on an ICL objective rather than emergent next-token prediction.

  • Connections to Related Methods: Tabular benchmarks are dominated by deep learning and gradient-boosted tree ensembles, while this work connects TFMs to k-nearest-neighbor and possibly kernel-based methods.Explicit k-nearest retrieval has also been used in recent tabular ICL models.
  • Tabular In-Context Learning: Prior tabular ICL studies generally scale task diversity through complex synthetic priors or augmented real-world data, leaving extremely limited pre-training data unexplored.Other reported capabilities include resilience to distribution shifts and causal inference.
  • Language-Model In-Context Learning: Although substantial literature analyzes in-context learning in large language models, that behavior differs fundamentally from TFMs, which directly train on an ICL objective.LLM in-context learning is described as an emergent capability of autoregressive next-token prediction models trained on large text corpora.

6 Conclusion and limitations

The paper adopts a task-centric account of TFM generalization, explaining how strong out-of-distribution performance can emerge from a single real table and motivating practical filtering recommendations. It also advances a retrieval-focused interpretation while acknowledging that additional mechanisms and broader pre-training-data settings remain insufficiently understood.

  • Contributions: A task-centric view explains why strong out-of-distribution generalization can emerge when TFMs are pre-trained on a single real table.The conclusion frames this finding as a central explanation of how pre-training data enables generalization.
  • Contributions: The study develops practical recommendations for dataset-level and column-level data filtering.These recommendations extend the paper’s task-centric analysis of pre-training data.
  • Contributions: The paper argues for interpreting TFMs through learned retrieval rather than standard transfer learning or Bayesian inference.The conclusion presents retrieval as the preferred interpretation of TFM behavior, while noting empirical support in the underlying work.
  • Limitations: The analysis covered a broad range of datasets, model sizes, and context lengths, but could not definitively identify or exclude mechanisms beyond learned retrieval.This limitation leaves open whether additional mechanisms contribute to out-of-distribution generalization.
  • Limitations: Further research is needed to develop a more comprehensive understanding of how in-context learning operates in TFMs.The conclusion explicitly connects this need to the unresolved mechanisms underlying out-of-distribution generalization.

A Impact of pre-training dataset on CC-18 and CTR-23 Performance · B Training and architecture details

The section examines how TabDPT’s pre-training dataset affects performance on CC-18 classification and CTR-23 regression, while specifying the model version used throughout the paper. Results are organized by benchmark performance and accompanied by dataset characteristics such as size, feature count, and domain.

  • A Impact of pre-training dataset on CC-18 and CTR-23 Performance: TabDPT is evaluated after pre-training on multiple datasets across the CC-18 and CTR-23 benchmark suites.CC-18 measures classification with AUC, whereas CTR-23 measures regression with R2.
  • A Impact of pre-training dataset on CC-18 and CTR-23 Performance: The performance tables list each pre-training dataset together with its number of instances, number of features, and domain.These attributes provide dataset-level context for comparing pre-training outcomes.
  • A Impact of pre-training dataset on CC-18 and CTR-23 Performance: The reported results are ordered to highlight which pre-training datasets achieve better performance on each benchmark.The ordering is descending for both AUC on CC-18 and R2 on CTR-23.
  • A Impact of pre-training dataset on CC-18 and CTR-23 Performance: CC-18 performance is reported using AUC scores for TabDPT pre-trained on various datasets.The corresponding results are sorted in descending order of AUC.
  • A Impact of pre-training dataset on CC-18 and CTR-23 Performance: CTR-23 performance is reported using R2 scores for TabDPT pre-trained on various datasets.The corresponding results are sorted in descending order of R2.
  • B Training and architecture details: Throughout the paper, “TabDPT” specifically refers to TabDPT v1.1.This version designation applies to the model discussed in the section.

B.1 Base model: many-dataset pre-training ablations

The many-dataset ablations use a shared TabDPT architecture and training configuration across deduplication, dataset-level filtering, and column-level pre-processing experiments. The setup combines a 1,732-dataset OpenML corpus with self-supervised target prediction, standardized preprocessing and retrieval, and IQM-based evaluation on CC-18 and CTR-23.

  • Experimental setup: The ablation experiments share the same base TabDPT architecture and training configuration across deduplication, dataset-level filtering, and column-level pre-processing.This common setup is used for all three experimental comparisons.
  • Pre-training corpus: The pre-training corpus contains 1,732 OpenML datasets, excludes datasets without a clear target variable y, and includes 197 exact duplicates.Datasets from the CC-18, CTR-23, and TabArena benchmarks were automatically removed, while datasets with fewer than 10 columns were retained.
  • Architecture: The model is a 12-layer transformer with dmodel = 512, 8 attention heads, and an asymmetric pattern in which test tokens attend only to training tokens.Feature columns use a shared linear encoder, while target embeddings are added to context tokens.
  • Training: Training uniformly selects one column as target y, uses the remaining columns as X, and jointly optimizes classification and regression heads with a self-supervised objective.Optimization uses AdamW with learning rate 5 × 10^-4, weight decay 0.05, gradient clipping at 1.0, and dropout 0.0.
  • Data pre-processing and retrieval: Datasets are truncated to 10,000 rows and 3,000 columns, standardized after encoding and mean imputation, and indexed with FAISS for retrieval-augmented context construction.Categorical columns are label-encoded, and columns with fewer than 1% interpretable numeric values are treated as categorical.
  • Evaluation: Evaluation uses CC-18 for classification and CTR-23 for regression, reporting Interquartile Mean scores with 95% bootstrap confidence intervals from 20,000 stratified resampling iterations.Inference uses context size 2,048 and 8 ensemble passes on the raw benchmark datasets.

B.2 Large model: context-length and architecture scaling

The large-model experiments use a 16-layer transformer with wider embeddings and three maximum context lengths, each trained with and without preprocessing. The runs retain the base model’s retrieval and data-processing pipeline, while +pp variants add column-level filtering and deduplication.

  • Context length: Six runs combine three maximum labelled-context lengths with and without preprocessing, while all variants use a minimum context size of 100.The context-length variants share the large-model configuration described in this section.
  • Training: Training uses ScheduleFree AdamW with learning rate 1×10−3, weight decay 0.05, BF16 mixed precision, and 128,000 gradient steps per run.Each run uses batch size 32 with gradient accumulation over 4 steps, giving effective batch size 128; gradient clipping and label smoothing remain unchanged.
  • Data pre-processing and retrieval: Dataset caps, categorical encoding, imputation, scaling, and FAISS L2 retrieval are identical to the base model across all runs.The +pp variants additionally drop NaN columns, deduplicate by Spearman correlation above 0.90, and require at least k = 5 features per dataset.

C Additional TabArena results … D.2 Strong / weak partitioning

The supplementary results extend TabArena-Lite comparisons and define attention-similarity experiments across 52 single-dataset TabDPT checkpoints. They also partition models into strong and weak groups using held-out AUC quartiles for downstream analysis.

  • C Additional TabArena results: TabArena-Lite evaluates six TabDPT variants and baseline methods across 51 tasks using 2,000 bootstrap replicates, with Elo anchored at Random Forest = 1000.The variants span preprocessing and 1k, 2k, or 4k-token training contexts.
  • C.1 Pairwise win rates: 4k+pp has the highest pairwise win rates, followed by 2k+pp and 4k, while shorter-context models trail.Figure 10 reports the fraction of tasks where each row method outperforms or ties each column method.
  • C.2 Leaderboard results: 4k+pp achieves the strongest leaderboard performance among the evaluated TabDPT models, with the remaining variants following the same ordering as the pairwise results.Table 5 covers the full 51-task, fold-0 TabArena-Lite leaderboard.
  • D.1 Models, probes, and similarity calculations: The attention-similarity study probes 52 checkpoints, each trained from scratch on one OpenML dataset with a shared architecture and training schedule.All checkpoints use 12 transformer blocks, dmodel = 512, one attention head, sequence length 1536, and 12,800 gradient updates.
  • D.1 Models, probes, and similarity calculations: For each checkpoint and probe, the study records per-block attention tensors at eval_pos = 512, representing distributions over 512 context tokens.The recorded tensor shape is (B, H, Q, C) = (32, 1, 512, 512).
  • D.1 Models, probes, and similarity calculations: Model-pair similarity averages row-wise Spearman correlations between attention distributions across queries, probe examples, and the 12 transformer blocks.Spearman is used because rank correlation is invariant to exact attention magnitudes.
  • D.2 Strong / weak partitioning: Models are divided by held-out OpenML CC-18 AUC into 13 strong models with AUC ≥0.900 and 13 weak models with AUC ≤0.831, excluding the 26 middle-quartile models.The thresholds correspond to q75 = 0.900 and q25 = 0.831; group-gap confidence intervals use bootstrap resampling.

D.3 Per-block breakdown · D.4 Caveat: pair non-independence

Across all 12 transformer blocks, the SS−SW gap remains positive, reaches its maximum at block 10, and is significant in 9 blocks, while per-block correlations strengthen mainly in deeper blocks. These confidence intervals may be somewhat too tight because model pairs are not fully independent.

  • D.3 Per-block breakdown: The per-block analysis reports SS, SW, and WW group means alongside the SS−SW gap and its 95% bootstrap confidence interval.Each row pools across two probes, with nSS = 156, nSW = 338, and nWW = 156 per block.
  • D.3 Per-block breakdown: The SS−SW gap is positive in all 12 transformer blocks.This establishes a consistent direction across the full block sequence.
  • D.3 Per-block breakdown: Per-block correlations rise from near zero in early blocks to ≈0.6 in blocks 9–11.The global correlation effect is therefore concentrated in deeper transformer blocks.
  • D.4 Caveat: pair non-independence: The observations are not fully independent because each model appears in 51 pairs.Consequently, the reported confidence intervals are somewhat tighter than under a stricter resampling scheme.

E Additional query–key sharing ablation

Enforcing shared query and key projections (WQ = WK) substantially improves validation AUC on two additional poorly generalizing pre-training tables, supporting a near-nearest-neighbor account of TFM performance.

  • Additional query–key sharing ablation: WQ = WK produces significantly higher validation AUC than the default architecture on both VISUALIZING SOIL and CHESS.The ablation analyzes training dynamics on two further pre-training tables that generalize poorly.
  • Additional query–key sharing ablation: The results complement Figure 7a’s weak-table findings and support the view that TFM performance is close to a nearest neighbour-like mechanism.Figure 12 uses the same setting as Figure 7a.
Loading 2608.17957v1…