PaperlayerView exact revision on arXiv

Source-linked AI summary

TabMF: Learning Dataset Representations with Tabular Foundation Models

Rafiq Kamel, Nicolas Atienza, Aziz Fouché, Nicola Cancedda, Josif Grabocka

arXiv:2610.13818v1cs.LG

TL;DR

Dataset-level representations can support tasks such as retrieval and shift monitoring, but existing approaches rely on fixed meta-features or dedicated encoders. TABMF repurposes pretrained tabular foundation models with architecture-aware readouts and contrastive post-training; across six real-data evaluations, its encoders achieve the strongest aggregate performance, while frozen readouts also prove useful.

  • Problem

    Dataset-level tasks need reusable representations, but existing approaches rely on fixed hand-crafted meta-features or dedicated encoders trained without predictive pretraining.

    Source: §1 · p. 1

  • Method

    TABMF adapts pretrained tabular foundation models with architecture-aware dataset readouts and contrastive post-training on stochastic views of synthetic supervised datasets.

    Source: §1 · pp. 1–2, §3 · p. 4

  • Results

    Across six primary evaluations on unseen real datasets, the two TABMF encoders achieve the strongest aggregate performance among evaluated learned and classical representations.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Predictive TFM pretraining alone induces useful dataset-level representations, and tabular foundation models can serve as a basis for reusable supervised-dataset representations.

    Source: §1 · p. 2, §7 · p. 9

    The evaluation covers classification datasets and two pretrained TFM backbones, and post-training uses the TabICLv2 Graph-SCM prior.

    Source: §6 · p. 9

Abstract

from arXiv · show

Representations of supervised datasets support higher-level machine-learning tasks such as dataset retrieval and distribution-shift monitoring. However, existing approaches typically rely either on hand-crafted meta-features or on dataset encoders trained specifically for representation learning. We introduce TabMF, a framework that repurposes pretrained tabular foundation models (TFMs) as encoders of supervised datasets. TabMF equips a predictive TFM with an architecture-aware dataset-level readout and adapts the model end-to-end using contrastive post-training on stochastic views of synthetic supervised datasets. We instantiate TabMF on TabICLv2 and TabPFN-2.5 and evaluate the resulting representations across six tasks on unseen real datasets. Simple frozen TFM readouts yield competitive representations, indicating that predictive pretraining already induces substantial dataset-level structure. The two TabMF encoders obtain the strongest aggregate performance among the evaluated learned and classical dataset representations, with particularly strong results in dataset retrieval, compact benchmark construction, and distribution-shift severity assessment.

1 INTRODUCTION

Dataset-level representations support higher-level machine-learning tasks, but existing methods use fixed hand-crafted statistics or dedicated encoders trained from scratch. TABMF repurposes pretrained tabular foundation models, and its evaluation finds both useful frozen readouts and strong aggregate results after contrastive adaptation.

1 INTRODUCTION

  • Hand-crafted meta-features and dedicated dataset encoders leave open whether representations from large-scale predictive pretraining can be reused for whole supervised datasets.The former use predefined statistics; the latter are trained specifically for dataset representation learning.Source: §1 · p. 1
  • Frozen pretrained TFM readouts outperform dedicated learned dataset encoders on retrieval and remain competitive on dataset-property prediction.The result indicates that predictive pretraining alone induces useful dataset-level representations.Source: §1 · p. 2
  • Across six evaluations on unseen real datasets, the two TABMF encoders achieve the strongest aggregate performance among evaluated learned and classical dataset representations.The reported strengths include dataset retrieval, compact benchmark construction, and distribution-shift severity assessment.Source: §1 · p. 2
  • TABMF adapts pretrained tabular foundation models into supervised-dataset encoders using architecture-aware readouts and contrastive post-training on stochastic synthetic-dataset views.The views vary observations, observed variables, and label-preserving feature transformations while retaining the designated target.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • TABMF maps an unseen supervised dataset to a fixed-dimensional representation in one forward pass for dataset comparison and monitoring or lightweight downstream probes.The encoder represents the labeled dataset, not just individual observations.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

Prior dataset representations include fixed hand-crafted statistics, learned encoders trained from scratch, and representations derived from predictive tabular models. TABMF differs through its pretrained TFM starting point and dataset-level contrastive post-training on synthetic supervised data.

2 RELATED WORK

  • Hand-crafted meta-features use fixed statistics for tasks including algorithm selection and zero-shot AutoML, so their representation cannot adapt to new notions of dataset similarity.Examples include Alcobac¸a et al. (2020), McElfresh et al. (2023), Zhao et al. (2021), Öztürk et al. (2022), and Pineda-Arango et al. (2024).Source: §2 · p. 3
  • Dataset2Vec and the Neural Statistician learn dataset encoders from scratch, unlike TABMF's pretrained TFM starting point and synthetic supervised post-training.Dataset2Vec distinguishes batches from same versus different datasets; the Neural Statistician learns a variational summary of exchangeable observations.Source: §2 · p. 3
  • Prior tabular foundation-model representation work reuses internal activations, while TABMF uses architecture-aware readouts and contrastive dataset-level post-training on TabICLv2 and TabPFN-2.5.Ye et al. (2025) study frozen TabPFN v2 row-level features; Regenwetter et al. (2026) pool TabPFN-2.5 activations into dataset descriptors.Source: §2 · p. 3
  • Unlike row-level contrastive learning, TABMF pairs stochastic views of the same supervised dataset; unlike target-agnostic table embeddings, it represents (X, y) with the designated target included.Prior row-level approaches corrupt or subset features; table-retrieval work includes HyTrel, TEmBed [Vogel et al., 2026; Poostforoushan et al., 2026], and generalist tabular embeddings [Qiang et al., 2026].Source: §2 · p. 3

3 TABMF

TABMF maps a supervised classification dataset, including its designated target, to a fixed-dimensional vector using a pretrained TFM, architecture-aware readout, and contrastive post-training. Stochastic views of synthetic datasets define the contrastive signal, while each backbone receives a readout matched to its internal organization.

3 TABMF

  • TABMF encodes a supervised classification dataset D = (X, y) as a fixed-dimensional representation intended to capture predictive structure between features and the designated target.Its encoder combines a pretrained TFM backbone, an architecture-aware readout, and contrastive post-training; the components are jointly optimized and inference takes one forward pass.Source: §3 · p. 3

3.1 CONTRASTIVE DATASET-LEVEL ADAPTATION

  • For contrastive post-training, two independently sampled views of each synthetic dataset vary rows, features, and some label-preserving feature transformations while retaining the same target.Views from one source dataset form positive pairs; views from different sources are negatives, and NT-Xent trains the normalized embeddings to bring positives together and separate negatives.Source: §3.1 · p. 4

3.2 ARCHITECTURE-AWARE TABMF ENCODERS

  • The architecture-aware readout differs by backbone: TabICLv2 uses a learned summary token over target-enriched row states, while TabPFN-2.5 aggregates feature-group states across observations.Both readouts produce a dataset representation that is then projected to a normalized embedding.Source: §3.2 · pp. 4–5, §3.1 · p. 4

4 EXPERIMENTAL SETUP

The experiments train TABMF on synthetic Graph-SCM datasets and compare adapted and frozen backbones with learned encoders and classical meta-features. Evaluation measures transfer to unseen real datasets across six dataset-level tasks.

4.1 TRAINING AND MODELS

  • Training samples Graph-SCM datasets with 5,000–50,000 rows, 5–100 features, and at most ten classes, then creates bounded stochastic views for each backbone.The evaluated checkpoints use 45,000 optimization steps; training requires approximately 315 GPU-hours for TABMF-ICL and 273 GPU-hours for TABMF-PFN.Source: §4.1 · p. 5
  • Comparisons include adapted TabICLv2 and TabPFN-2.5 encoders, frozen-backbone readouts, alternative TabICL readouts, random initialization, learned dataset encoders, and PyMFE and HyTrel.The adapted-versus-frozen comparison also changes the readout.Source: §4.1 · p. 5

4.2 EVALUATION TASKS

  • On 191 TabBench datasets, bootstrap-view discrimination tests same-dataset similarity; probes predict eight domains and epiplexity, while transformed retrieval measures mean recall@5 across 19 transformations.The bootstrap task reports ROC-AUC; domain prediction uses macro one-vs-rest ROC-AUC, and epiplexity prediction uses held-out Spearman correlation.Source: §4.2 · p. 5
  • Compact benchmark selection measures classifier-ranking agreement after selecting three datasets, while TableShift evaluates whether embedding distances track accuracy-drop severity across nine tasks.The respective metrics are Spearman correlation with the full-population classifier ranking and Spearman correlation between cosine distance and mean ID-minus-OOD accuracy gap.Source: §4.2 · p. 6

5 RESULTS

The results evaluate selected TABMF variants across six primary tasks on real datasets, then examine transformation robustness, downstream uses, frozen-TFM representations, and encoder design.

5.1 TABMF PROVIDES STRONG DATASET REPRESENTATIONS

  • Median ranks of 1.5 for TABMF-PFN and 2.0 for TABMF-ICL were strongest across six tasks; a TABMF encoder led five, while frozen TFM representations led epiplexity prediction.The two TABMF variants obtained the strongest aggregate ranks among evaluated representations.Source: §5.1 · p. 6

5.2 TABMF LEARNS A ROBUST GEOMETRY FOR DATASET COMPARISON

  • On real-data bootstrap-view discrimination, TABMF-ICL and TABMF-PFN reached ROC-AUCs of 0.9981 and 0.9985; both also obtained larger hard margins than all nine baselines.The margin comparison uses a per-dataset criterion because several baselines also achieve high AUC.Source: §5.2 · p. 7
  • With only 25% of features retained, recall@5 was 0.884 for TABMF-ICL and 0.937 for TABMF-PFN, versus 0.653 for ICL-mean and 0.728 for PyMFE.TABMF-PFN better tolerates monotone transformations and standardization, TABMF-ICL better tolerates heavy noise and missingness, and quantization limits both.Source: §5.2 · p. 7, Figure 2 · p. 7

5.3 COMPACT BENCHMARK SELECTION PRESERVES CLASSIFIER RANKINGS

  • At a three-dataset budget, TABMF-PFN achieved classifier-ranking correlation of 0.929 with K-means and 0.823 with farthest-first; a TABMF variant led for budgets k=2, 3, and 4.TABMF-ICL also exceeded random selection under both selection rules at the three-dataset budget.Source: §5.3 · pp. 7–8

5.4 EMBEDDING DISTANCES TRACK DISTRIBUTION-SHIFT SEVERITY

  • Across nine TableShift tasks, ID–OOD embedding distance correlated with mean accuracy drop at 0.933 for TABMF-ICL and 0.917 for TABMF-PFN, versus 0.850 for ICL-mean.With only nine task-level observations, the numerical ordering does not establish pairwise superiority between encoders.Source: §5.4 · p. 8

5.5 PREDICTIVE TFMS ALREADY ENCODE DATASET-LEVEL INFORMATION

  • Frozen TFM readouts already encode dataset information: ICL-mean retrieval recall@5 was 0.889, above Dataset2Vec (0.750) and Neural Statistician (0.677), and near PyMFE (0.931).For primary linear probes, post-training did not uniformly improve property recoverability; nonlinear checks did not establish that apparent ties or losses meant information was absent from adapted representations.Source: §5.5 · p. 8

5.6 ABLATIONS

  • Pretrained initialization raised recall from 0.797 to 0.938 for the TabICL summary-token encoder and from 0.587 to 0.965 for the TabPFN column-aware encoder.The initialization comparisons hold architecture, readout, objective, and batch regime fixed within each backbone.Source: §5.6 · pp. 8–9
  • NT-Xent had the highest recall for both selected architectures; for TabPFN-2.5, the column-aware readout scored 0.965 versus 0.935 for summary-token and 0.957 for pseudo-test.For TABMF-ICL, NT-Xent scored 0.938 versus 0.924 for triplet and 0.883 for exponential distance; for TABMF-PFN, the corresponding scores were 0.965, 0.962, and 0.920.Source: §5.6 · p. 9

6 DISCUSSION AND LIMITATIONS

The discussion frames TABMF as reusable across dataset-level reasoning while identifying the boundaries of the evaluated setting and directions for extending it.

6 DISCUSSION AND LIMITATIONS

  • The evaluation covers classification datasets and two pretrained TFM backbones; regression and other architectures remain extensions to establish generality.Post-training uses the TabICLv2 Graph-SCM prior; broader synthetic priors or mixtures of synthetic and real datasets are proposed for extending coverage.Source: §6 · p. 9
  • TABMF representations are explicitly supervised, so changing the designated target can change the representation even when the feature table stays fixed.Source: §6 · p. 9
  • Different applications may require different notions of dataset similarity, motivating representations whose geometry can be conditioned on the downstream objective.Source: §6 · p. 9

7 CONCLUSION

Across six primary evaluations on real datasets, the two TABMF encoders achieve the strongest aggregate performance among evaluated learned and classical representations. Predictive pretraining alone also yields useful dataset-level representations without explicit dataset-level training.

7 CONCLUSION

  • Across six primary evaluations on real datasets, the two TABMF instantiations achieve the strongest aggregate performance among evaluated learned and classical dataset representations.Their particularly strong results include robust dataset comparison, compact benchmark construction, and distribution-shift assessment.Source: §7 · p. 9
  • Predictive TFM pretraining alone induces useful dataset-level representations, even without explicit dataset-level training.The paper concludes that tabular foundation models can also support reusable representations of supervised datasets, beyond prediction over individual observations.Source: §7 · p. 9

A.1 REPRESENTATIONS FROM PRETRAINED TABICLV2

  • The column-stage TabICLv2 readout averages target-aware column embeddings over observations and token positions, then applies ℓ2 normalization before later stages.Source: §A.1 · p. 13
  • The row-mean readout averages native row representations produced by pretrained TabICLv2 weights, without dataset-level adaptation.Source: §A.1 · p. 13
  • The ICL-mean readout processes labeled observations as context with TabICLv2’s pretrained ICL transformer, then averages its normalized hidden states.Source: §A.1 · p. 13

A.2 ALTERNATIVE TABICLV2 ENCODERS

  • The contrastively trained row-mean variant retains native row aggregation while optimizing the TabICLv2 backbone and projection network.This holds the readout fixed when comparing pretrained row representations as-is with adaptation to the dataset-level objective.Source: §A.2 · p. 14
  • The column-aware readout aggregates contextual feature states across observations, applies dataset-level self-attention, and pools active features without first collapsing rows to CLS representations.Source: §A.2 · p. 14

A.3 ALTERNATIVE TABPFN-2.5 ENCODERS

  • For TabPFN-2.5, four learned queries summarize each row’s feature groups; target embeddings and a summary token then enter a dedicated transformer.The summary position defines the dataset representation.Source: §A.3 · pp. 14–15

A.4 ALTERNATIVE METRIC-LEARNING OBJECTIVES

The appendix compares alternative metric-learning objectives while retaining the same dataset views and encoder architectures; the main TABMF models use NT-Xent.

A.4 ALTERNATIVE METRIC-LEARNING OBJECTIVES

  • The alternative objectives retain the same dataset views and encoder architectures, unlike the main TABMF models’ NT-Xent objective.Source: §A.4 · p. 15
  • The exponential-distance objective attracts paired views of the same dataset and penalizes small distances between different datasets, with γ controlling decay.Source: §A.4 · p. 15
  • The batch-hard triplet objective selects the closest opposite-view representation from another dataset for each anchor and uses a margin m.The passage identifies m as the triplet margin.Source: §A.4 · p. 15

B.1 BOOTSTRAP VIEW DISCRIMINATION AND VARIANT SELECTION

The selected encoders achieve substantially larger hard margins than the baselines, and the section describes how variants and views are evaluated. The selected readouts also nearly saturate synthetic bootstrap-view discrimination.

B.1 BOOTSTRAP VIEW DISCRIMINATION AND VARIANT SELECTION

  • The selected summary-token and column-aware encoders achieve synthetic bootstrap-view discrimination AUCs of 0.9999 and 0.9996, respectively.The variants were selected using 50 synthetic datasets generated from the TabICLv2 Graph-SCM prior; several candidates approach saturation.Source: §B.1 · p. 16
  • Evaluation uses four views per source dataset, with same-source view pairs positive and different-source pairs negative.Views are formed by sampling rows with replacement and features without replacement; embeddings are L2-normalized and compared by cosine similarity.Source: §B.1 · p. 16
  • A hard margin is the mean within-dataset pairwise cosine similarity minus the maximum cross-dataset similarity involving that dataset.A positive margin means average within-dataset similarity exceeds the closest competing cross-dataset pair.Source: §B.1 · p. 16
  • On TabBench, median margins are 0.0809 for TABMF-ICL and 0.0587 for TABMF-PFN, versus at most 0.0045 for baselines; synthetic medians are 0.5945 and 0.5605 versus at most 0.0306.Both adapted encoders significantly outperform every baseline in each population under paired tests with Holm correction; these tests assess margins, not AUC.Source: §B.1 · p. 16

B.2 RETRIEVAL ROBUSTNESS UNDER DATASET TRANSFORMATIONS

The retrieval analysis evaluates recall@5 across dataset transformations and compares dataset-level averages with paired tests. The reported significant improvements apply to those dataset-level comparisons, not necessarily to the equally weighted headline score.

B.2 RETRIEVAL ROBUSTNESS UNDER DATASET TRANSFORMATIONS

  • The retrieval transformations include changes to observations and features, with n and d denoting their original counts and grouped entries representing separate parameter values.The transformation definitions include kr = max{min(n, 20), round(rn)} and hr = max{1, round(rd)}.Source: Table 5 · p. 17
  • Both adapted encoders significantly outperform TabICL ICL-mean on dataset-level retrieval averages, while TABMF-PFN also significantly outperforms PyMFE.The tests average recall@5 across applicable transformations within each dataset and apply Holm correction across four comparisons.Source: §B.2 · pp. 16–17
  • The headline retrieval score averages recall@5 equally across 19 transformations, whereas paired tests compare dataset-level averages.Figure 3 reports recall@5 under all 19 transformations.Source: §B.2 · p. 17

B.3 THREE-DATASET BENCHMARK SELECTION

The section evaluates compact benchmark construction and domain discrimination alongside their protocols. TABMF-PFN improves over frozen TabPFN mean-pooling in benchmark selection, while results for domain discrimination depend on the probe and endpoint.

B.3 THREE-DATASET BENCHMARK SELECTION

  • The TabRepo evaluation uses 144 classification datasets, ten default classifiers, and 200 shared candidate-pool trials, selecting three datasets per rule.Embeddings are standardized and L2-normalized before K-means representative selection or cosine farthest-first selection; HyTrel lacks a common-population score because each pool includes a dataset without its embedding.Source: §B.3 · pp. 17–18
  • On TabRepo, TABMF-PFN improves over frozen TabPFN mean-pooling by 0.0795 with K-means and 0.1500 with farthest-first selection.The 95% bootstrap intervals are [0.068, 0.091] and [0.133, 0.167], respectively; TABMF-ICL does not significantly exceed frozen TabPFN under K-means but does under farthest-first selection.Source: §B.3 · p. 18, Table 7 · p. 19
  • Across k ∈{2, 3, 4}, a TABMF variant has the highest numerical score under both selection rules, but the leading variant depends on budget.TABMF-ICL leads K-means at k = 2, while TABMF-PFN leads at k = 3 and k = 4 and under farthest-first selection at all three budgets.Source: §B.3 · p. 18

B.4.1 PROBE PROTOCOL

  • The primary property probes use tuned linear models with fold-local preprocessing and a repeated three-fold outer evaluation.Classification uses stratified folds and log-loss model selection; regression uses ordinary folds and an SVD ridge solver, with the three-fold evaluation repeated five times.Source: §B.4.1 · p. 19

B.4.2 DOMAIN DISCRIMINATION

  • With a strengthened nonlinear SVM probe, TABMF-ICL outperforms frozen TabICL ICL-mean in domain discrimination (AUC 0.6997 versus 0.6441; adjusted p = 0.0278).TABMF-PFN is not significantly separated from frozen TabPFN mean-pooling under this probe; a separate pooled-out-of-fold analysis found no pairwise comparison significant after Holm correction.Source: §B.4.2 · p. 20

B.4.3 EPIPLEXITY PREDICTION

The section reports probe results, aggregate-rank sensitivity, encoder variants, supplementary transfer evaluations, and inference-cost scaling. Its results distinguish linear accessibility from information detected by stronger probes and show that TABMF readouts add little inference cost beyond their backbones.

B.4.3 EPIPLEXITY PREDICTION

  • On the common 190-dataset population, frozen readouts outperform adapted encoders for epiplexity prediction under the primary ridge comparison; stronger kernel probes do not significantly separate either adapted variant from its frozen counterpart.The primary comparisons concern linear accessibility and do not establish that epiplexity-related information is absent from adapted embeddings.Source: §B.4.3 · pp. 21–22

B.4.4 PYMFE CONTROL FOR EPIPLEXITY

  • The epiplexity PyMFE control excludes target-dependent and model-based information to reduce direct overlap with ingredients of the score.The extractor is fitted without the target and uses only the general, statistical, and information-theoretic groups; full PyMFE remains in other tasks.Source: §B.4.4 · p. 22, Table 14 · p. 22

B.5 AGGREGATE RANKING AND STATISTICAL SCOPE

  • Across the six-task aggregate, both TABMF variants retain lower median ranks than every baseline under all tested benchmark-selection settings.TABMF-ICL leads at k = 2, TABMF-PFN at k = 3 and k = 4, and both obtain median rank 2.0 when benchmark selection is omitted; the aggregate has no significance test.Source: §B.5 · pp. 22–23, Table 15 · p. 23

B.6 ENCODER VARIANTS

  • Pretrained initialization outperforms scratch initialization for both TabICL and TabPFN; among remaining TabPFN variants, the pretrained column-aware encoder with NT-Xent obtains 0.965 recall.The comparison reports 0.935 for the summary-token encoder and 0.957 for the pseudo-test encoder.Source: §B.6 · p. 23

B.7 WEB-TABLE AND HYPERPARAMETER TRANSFER

  • Supplementary transfer evaluations cover target-agnostic web-table representations and hyperparameter transfer, distinct from the main supervised-dataset setting.Web-table similarity averages MAP over 16 GitTables datalakes for TFM encoders and HyTrel, while PyMFE covers 17; hyperparameter-transfer aggregates cover 82 targets for adapted encoders and TabICL ICL-mean, versus 77 for HyTrel.Source: §B.7 · p. 24, Table 17 · p. 24

B.8 ENCODER EFFICIENCY AND SCALING

  • Both TABMF variants closely track the inference cost of their frozen counterparts as table size increases, introducing little additional cost while encoding the entire input.The scaling experiments vary rows from 500 to 200,000 at 64 features and features from 8 to 2,048 at 10,000 rows; HyTrel exceeds available memory at larger sizes.Source: §B.8 · p. 24, Figure 4 · p. 25

C ILLUSTRATION OF THE TABMF BACKBONE AGGREGATIONS

  • TABMF-ICL reads a learned summary-token state, whereas TABMF-PFN pools feature-group states and models their interactions with a learned column Transformer.Each readout feeds a projection network; pretrained blocks are fine-tuned and added blocks are trained from scratch.Source: Figure 5 · p. 26
Loading 2610.13818v1…