PaperlayerView exact revision on arXiv

Source-linked AI summary

TabFM: A Zero-Shot Foundation Model for Tabular Data

Weihao Kong, Erez Louidor Ilan, Shuxin Nie, Taman Narayan, Rajat Sen, Yichen Zhou, Deqing Fu, Samet Oymak, Abhimanyu Das

arXiv:2609.37959v1cs.LG

TL;DR

Tabular machine learning often requires a separate optimization workflow for each dataset. TabFM uses a synthetic-data-trained foundation model for zero-shot in-context prediction, and ranks first among default models on both TabArena tracks, with frozen-weight extensions improving results further.

  • Problem

    Tabular prediction commonly relies on per-dataset preprocessing and search, repeating optimization from scratch for each task.

    Source: §1 · pp. 1–2

  • Method

    TabFM is a 400M-parameter in-context Transformer trained on synthetic structural-causal-model tables, with frozen-weight extensions for ensembling and LLM-guided feature engineering.

    Source: §1 · p. 2, §5 · p. 9, §6 · p. 11

  • Results

    TabFM ranks first among default foundation models on both TabArena tracks, and TabFM+ and TabFM-Auto improve performance further on both tracks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The synthetic-data-trained model transfers zero-shot to real-world tasks, while inference-time ensembling and LLM-guided feature engineering add gains without updating its weights.

    Source: §6 · p. 11

    Pretraining covers synthetic numerical and categorical tables up to 16,384 rows and 100 columns, and free text lacks semantic tokenization.

    Source: §6 · p. 11

Abstract

from arXiv · show

Tabular machine learning typically relies on per-dataset workflows, fitting tree ensembles or running AutoML searches from scratch for every task. We present TabFM, a 400M-parameter tabular foundation model that formulates supervised tabular prediction as in-context learning. TabFM produces calibrated zero-shot predictions in a single forward pass without task-specific tuning. Trained entirely on synthetic tables generated from structural causal models, TabFM learns general tabular representations that transfer zero-shot to real-world tasks. Across all 51 benchmark datasets in TabArena (38 classification and 13 regression), zero-shot TabFM ranks first among default tabular foundation models and outperforms tuned AutoML pipelines. Two extensions over the same frozen weights improve performance further on both tracks: multi-view feature expansion with ensembling and post-hoc calibration (TabFM+), and LLM-guided, dataset-specific data processing and feature engineering (TabFM-Auto).

1. Introduction

Tabular prediction commonly requires per-dataset preprocessing and optimization, while scaling transformers to general tables is constrained by their structure and attention costs. TabFM combines alternating column- and row-wise attention with in-context prediction, and its deployment tiers rank first among default foundation models on both TabArena tracks.

1. Introduction

  • Tree ensembles and AutoML require per-dataset preprocessing, split-finding sweeps, and hyperparameter search, so each new dataset is learned from scratch.Tree ensembles are effective on mixed column types, missing entries, and irregular feature scales.Source: §1 · pp. 1–2
  • General tabular transformers must respect exchangeable rows and mixed continuous and categorical columns, while naive all-cell attention scales as O(T^2H^2).These constraints historically confined tabular transformers to fewer than 1,000 rows.Source: §1 · p. 2
  • TabFM uses dyadic feature grouping and learned Fourier cell embeddings, alternating column-wise ISAB with row-wise SAB and pooling features into fixed-width row representations.Its asymmetric mask enforces conditional independence among queries, and its context scales to 16,384 instances.Source: §1 · p. 2, Figure 2 · p. 2
  • TabFM ranks first among default foundation models on both TabArena tracks, while TabFM+ and TabFM-Auto take the top rank on both suites.TabFM+ combines feature expansion, multi-view ensembling, and calibration; TabFM-Auto adds LLM-guided, dataset-specific feature engineering using the same frozen weights.Source: §1 · pp. 2–3

2. Related Work

Related work spans synthetic-prior in-context prediction, tree ensembles and deep architectures, and automated model and data optimization. TabFM builds on these strands with spectral cell embeddings, alternating attention, and frozen-model feature engineering.

Tabular Foundation Models and In-Context Optimization

  • PFNs introduced in-context tabular prediction using synthetic priors for Bayesian posterior predictives (Müller et al., 2022); TabPFN demonstrated small-table classification (Hollmann et al., 2023a), followed by work on regression and larger samples (Grinsztajn et al., 2026a,b; Hollmann et al., 2025), empirical pretraining (Hosseinzadeh et al., 2026; Ma et al., 2025), and row-column architectures (Qu et al., 2025, 2026).TabFM pairs alternating column- and row-attention with a deep 24-layer in-context predictor over spectral cell embeddings.Source: §2 · p. 3

Tree Ensembles, Deep Architectures, and Numerical Embeddings

AutoML and Programmatic Data Optimization

  • AutoML automates model selection, feature engineering, and tuning, whereas TabFM-Auto synthesizes dataset-specific feature-engineering pipelines around a frozen tabular model without updating its weights (Fu et al., 2026a).Related approaches include AutoGluon-Tabular’s stacking and bagging over a per-task search budget (Erickson et al., 2020), TabLLM row serialization (Hegselmann et al., 2023), and CAAFE feature-engineering synthesis (Hollmann et al., 2023b).Source: §2 · p. 3

3. TabFM: Model Architecture and Pretraining

TabFM combines feature grouping, alternating column- and row-wise attention, and a masked in-context predictor, and pretrains on synthetic tables generated from structural causal models. On TabArena, it leads default models on both classification and regression, with the pooled leaderboard placing it first among default foundation models.

3.1. Model Architecture

  • Dyadic feature grouping pairs each column with offsets 0, 1, and 3, while slot-specific learned Fourier frequency banks embed cell values.Feature grouping is intended to capture local cross-feature interactions before attention.Source: §3.1 · p. 4
  • Column-wise induced attention captures row-wise marginal distributions, while row-wise attention captures cross-feature structure and CLS tokens pool rows into fixed-width representations.The context mask restricts inducing summaries to labeled training rows, and the predictor masks keys beyond the labeled prefix so test rows attend only to training context.Source: §3.1 · pp. 4–5

3.2. Synthetic Pretraining Data and Curriculum

  • Synthetic pretraining samples SCM graphs with randomized dependencies and jointly varies table shape, categorical features, missingness, label noise, and class balance.Each table has labeled context and query rows, and loss is evaluated on query rows for classification and regression targets.Source: §3.2 · p. 6
  • A four-stage curriculum grows context from 2,048 to 16,384 instances while holding tokens per optimization step fixed.Shorter contexts first stabilize the embedder and attention stages; later stages adapt the inducing-point bottlenecks and predictor to larger samples.Source: §3.2 · p. 6

3.3. TabArena Zero-Shot Evaluation

  • 2055.2 Elo on regression and 1768.6 Elo on classification place TabFM ahead of other default foundation models on both TabArena tracks.Across 51 datasets, regression G-mean error is 16.40 and classification G-mean error is 0.0957.Source: Table 2 · p. 7, §3.3 · p. 7
  • 1785.4 Elo gives TabFM the top pooled rank among default foundation models, with head-to-head fold wins over every baseline reported.Win rates are 65.7% against EXAONE-Tabular, 72.1% against AutoGluon 1.5 extreme, 75.9% against TabPFN-3, and 79.8% against TabICLv2.Source: §3.3 · pp. 7–8

4. TabFM+: Feature Engineering and Inference-Time Ensembling

TabFM+ exploits sensitivity to column order and appended feature interactions by running transformed, feature-expanded views through the same frozen checkpoint, then stacking and calibrating predictions.

4. TabFM+: Feature Engineering and Inference-Time Ensembling

  • The forward pass is invariant to row order but sensitive to column order, numerical scaling, and appended feature interactions.Random column permutations create distinct views because dyadic grouping and RoPE depend on column order.Source: §4 · pp. 8–9
  • Cross features add pairwise products, while truncated-SVD features provide dense low-rank summaries of global linear structure.TabFM+ samples these engineered feature pools for an H-column table.Source: §4 · p. 8
  • Across 32 passes, half retain original columns and half append sampled cross/SVD features, while transformations vary scaling, column and category order, and classification-label shifts.Original-column members are capped at 500 features, beyond the 100 columns seen during pretraining.Source: §4 · p. 9
  • Validation-fold NNLS weights, shrunk toward uniform, stack member predictions; classification outputs then receive Platt scaling.The ensemble weights are fit on internal validation folds.Source: §4 · p. 9
  • +69.4 Elo on classification and +134.0 Elo on regression lift TabFM+ to second overall among 67 methods.The method combines multi-view ensembling with Platt scaling for classification.Source: §4 · p. 9

5. TabFM-Auto: Feature Engineering with Gemini

TabFM-Auto uses Gemini to search for dataset-specific pipelines around frozen TabFM, addressing features that generic input perturbations may miss. It ranks first on both TabArena tracks.

5. TabFM-Auto: Feature Engineering with Gemini

  • Gemini iteratively writes, evaluates, and refines Python pipelines around frozen TabFM, specifying preprocessing, feature engineering, context-row selection, post-processing, and inference settings.Auxiliary models may be fitted and blended, but TabFM parameters are never updated.Source: §5 · p. 9
  • Each candidate is scored by three-fold cross-validation within the first fold’s training split; the search stops after 96 evaluations or six hours.Programs are sandboxed from held-out test splits and evaluation folds, and every run starts from vanilla TabFM.Source: §5 · p. 10
  • Feature-changing programs have 2.14% median error reduction over TabFM, versus 0.27% when programs leave the table unchanged.Reported gains come primarily from feature construction rather than hyperparameter tuning.Source: §5 · p. 10
  • +172.1 Elo on classification and +337.2 Elo on regression over TabFM put TabFM-Auto first on both tracks; it improves 41 of 51 datasets and beats inference-time ensembling on 40.Regression improves on all 13 datasets, while the ten classification datasets that do not improve have test-error increases of at most 1.8%.Source: §5 · p. 10, Figure 6 · p. 10

6. Conclusion

The conclusion presents TabFM as a 400M-parameter in-context model trained on synthetic tables that transfers zero-shot to real-world tasks. It reports that the frozen-weight extensions bring further gains, while noting limits in current pretraining coverage.

6. Conclusion

  • TabFM is a 400M-parameter Transformer for supervised tabular prediction through in-context learning, trained on synthetic tables generated from structural causal models.It transfers zero-shot to real-world tasks.Source: §6 · p. 11
  • Zero-shot TabFM ranks first among default foundation models on TabArena, while TabFM+ and TabFM-Auto bring further gains without updating weights.Source: §6 · p. 11
  • Current pretraining covers synthetic numerical and categorical tables up to 16,384 rows and 100 columns, while free text lacks semantic tokenization.Source: §6 · p. 11

Related papers

Loading 2609.37959v1…