PaperlayerView exact revision on arXiv

Source-linked AI summary

TabGift: A Context-Efficient Foundation Model for Tabular Data Generation

Yu-Sheng Li, Jun-Peng Jiang, Han-Jia Ye

arXiv:2610.16990v1cs.LGcs.AI

TL;DR

Tabular generation must produce realistic samples despite scarce target data, but reliable distribution inference by foundation models remains challenging with limited context. TabGift performs in-context diffusion in a shared latent space and, across 18 classification and regression datasets, improves synthetic data quality under limited-context settings while increasing generation efficiency.

  • Problem

    Tabular generation needs to infer the joint distribution from limited observations, while large-context dependence conflicts with the data-scarce settings where synthetic data are most valuable.

    Source: §1 · pp. 1–2

  • Method

    TabGift encodes heterogeneous tables in a shared latent space and uses in-context diffusion conditioned on clean examples, followed by lightweight dataset-specific decoding.

    Source: §1 · p. 2, §3.4.3 · p. 5

  • Results

    Across 18 classification and regression datasets, TabGift consistently improves synthetic data quality under limited-context settings while achieving substantially higher generation efficiency.

    Source: §1 · p. 3

  • Takeaways & Limitations

    TabGift demonstrates that a tabular foundation model can generate high-quality synthetic data from short contexts with low computational overhead.

    Source: §1 · p. 2

    SwiftPFN embeddings, pretrained for prediction, may not optimally represent samples for generation, and a dataset-specific MLP decoder remains necessary.

    Source: §6 · p. 9

Abstract

from arXiv · show

Tabular data generation aims to synthesize realistic samples from limited observations while faithfully capturing the underlying data distribution. Tabular foundation models (TFMs), pretrained on diverse tabular tasks and adapted through in-context learning, have demonstrated strong predictive capabilities, suggesting the potential of extending this paradigm to tabular data generation. However, the strong predictive performance of TFMs often relies on sufficiently large context sets, whereas tabular generation is typically needed precisely when target data are scarce. Extending the foundation-model paradigm to generation therefore requires reliable distribution inference from only a limited context. To this end, we propose TabGift, a context-efficient tabular foundation model enabling both efficient distribution modeling and effective generation from only a few context samples. TabGift maps heterogeneous tables into a shared latent space with a pretrained tabular encoder, where an in-context diffusion model learns to denoise latent samples conditioned on a small set of clean examples. This design transfers generative priors across datasets and enables distribution inference from limited context, while a lightweight dataset-specific MLP decoder maps generated representations back to the original data space. Experiments on 7 classification and 11 regression datasets show that TabGift consistently outperforms representative baselines in synthetic data quality, with particularly strong gains under limited-context settings, while achieving substantially faster adaptation and generation.

1 INTRODUCTION

High-quality tabular data are often scarce, while existing generation methods either reuse observed samples or require dataset-specific training. TabGift extends the foundation-model approach to generation with shared latent representations and in-context generation from limited observations.

1 INTRODUCTION

  • High-quality tabular data are limited by privacy constraints, restricted access, and costly collection or annotation, motivating generation that preserves the underlying distribution.Existing methods rely on resampling or interpolation, or require dataset-specific training and tuning.Source: §1 · p. 1
  • Tabular foundation models transfer predictive capabilities across datasets, but generation requires inferring the joint distribution from small contexts while modeling interactions efficiently.This makes context efficiency central in data-scarce settings where synthetic data are valuable.Source: §1 · pp. 1–2
  • TabGift maps heterogeneous tables into a shared latent space, generates there with an in-context diffusion model, and uses a lightweight dataset-specific decoder.The model conditions noisy latent samples on clean context examples and performs efficient row-level interactions.Source: §1 · p. 2
  • Across 18 classification and regression datasets, TabGift improves synthetic data quality under limited-context settings while achieving substantially higher generation efficiency.The experiments cover 7 classification and 11 regression datasets.Source: §1 · pp. 2–3

2 RELATED WORK

Tabular generation research spans sample-synthesis methods and per-dataset generative models, while tabular foundation models have developed mainly around prediction. Generation-focused foundation-model approaches have emerged, but context length and interaction costs remain open concerns.

2.1 TABULAR GENERATION MODELS

  • Early tabular generators such as SMOTE and ADASYN synthesize minority samples, while later GAN, VAE, diffusion, Bayesian-network, flow, and tree-based models fit observed data but often degrade when data are scarce.The early methods target class imbalance rather than the full joint distribution; later methods are typically trained per dataset.Source: §2.1 · p. 3

2.2 TABULAR FOUNDATION MODELS

  • Tabular foundation models developed from TabPFN's in-context prediction paradigm toward greater scalability and representation learning, with newer models exploring diverse scales, tasks, architectures, and pretraining strategies.TabICL introduces column-aware modeling and multi-stage pretraining for larger, more heterogeneous tables.Source: §2.2 · p. 3
  • Generation-focused foundation models combine pretrained tabular representations or in-context learning with generative modeling, but existing approaches still rely on long contexts or costly interactions.Examples include TabPFGen, TabEBM, TabFORGE, and DiffICL.Source: §2.2 · p. 3

3 METHOD

TABGIFT encodes heterogeneous tables in a shared latent space, learns context-conditioned diffusion there, and decodes generated embeddings with dataset-specific MLPs. Its generation setup uses the training samples as context to model a synthetic dataset that approximates their distribution while preserving feature relationships and avoiding sample copying.

3.1 OVERVIEW

  • TABGIFT combines a pretrained tabular encoder, an in-context score-based latent diffusion model, and a dataset-specific MLP decoder.The encoder maps heterogeneous tables to a unified latent space; the diffusion model uses real-sample embeddings as context for denoising.Source: §3.1 · p. 4

3.2 SETUP

  • Generation aims to approximate the training data distribution while preserving marginal distributions and inter-feature dependencies without copying training samples.The setup defines each sample as a feature vector, with features that may be numerical or categorical.Source: §3.2 · p. 4

3.3 SWIFTPFN ENCODER

  • SwiftPFN supplies the encoder for heterogeneous datasets, and the representation before its prediction layer serves as the latent embedding.For generation, TABGIFT constructs zero labels and a zero test sample so the encoder can produce sample embeddings without a prediction target.Source: §3.3 · p. 4

3.4.1 FORWARD TRAINING PROCESS

  • A score-based diffusion model learns to denoise noisy target embeddings conditioned on clean sample embeddings, estimating the clean latent representation.The trainable in-context network is Dθ; the EDM parameterization supplies standard preconditioning coefficients.Source: §3.4 · p. 4, §3.4.1 · p. 5

3.4.2 SAMPLING PROCESS

  • Sampling starts from Gaussian noise at σ_max and repeatedly denoises the latent samples along preset noise amplitudes down to σ_min.The resulting latent embeddings are generated after multiple denoising steps.Source: §3.4.2 · p. 5

3.4.3 IN-CONTEXT LEARNING MODEL

  • The 18-layer Transformer uses a sample-wise mask: clean embeddings attend to one another, while noisy embeddings attend only to themselves and clean context.This keeps denoising processes for different noisy embeddings independent; sinusoidal noise-level encodings are projected to 192 dimensions and added to noisy queries.Source: §3.4.3 · p. 5

3.5 DATASET-SPECIFIC MLP DECODER

  • A separate lightweight MLP decoder maps each dataset’s generated latent embeddings to feature values or probability distributions.It is trained on original samples and their embeddings; numerical features use squared error and categorical features use cross-entropy.Source: §3.5 · pp. 5–6

4 TRAINING DETAILS

TABGIFT is pretrained in two stages: first on synthetic datasets, then on real-world datasets. The training process uses sampled context and reconstruction portions and takes about eight days.

4.1 TRAINING DATA

  • The synthetic stage uses 5,000 batches of 512 datasets generated from different SCMs, reused 10 times for 50,000 training batches.Each dataset has 128–2,048 samples and 2–50 features; columns are randomly shuffled.Source: §4.1 · p. 6
  • The real-data stage uses 72 OpenML datasets, grouped into 1,000 batches and reused 10 times to yield 10,000 training batches.Rows and columns are sampled within the same ranges as for synthetic data.Source: §4.1 · p. 6

4.2 TRAINING PROCESS

  • 50,000 synthetic batches are followed by 10,000 real-data batches, with context ratios sampled uniformly from 0.5 to 0.8.The two stages take about seven days and one day, respectively, for about eight days total.Source: §4.2 · p. 6, §4.1 · p. 6

5 EXPERIMENT

Experiments evaluate TABGIFT on 18 real-world classification and regression tasks across data budgets, comparing generation quality, privacy, and runtime. TABGIFT is strongest in limited-context settings, with a narrowing advantage as training data grow.

5 EXPERIMENT

  • Across 18 real-world datasets—7 classification and 11 regression tasks—the evaluation measures statistical fidelity, privacy preservation, and downstream-model performance.The datasets come from Tiny Talent after excluding those with more than 50 features.Source: §5 · p. 6, §5.1 · p. 7

5.1 DATASETS

  • Training sets contain 16, 32, 64, 128, 256, or 512 samples, with five random-seed repetitions at each size.The generated samples are saved for subsequent evaluation and comparison.Source: §5.1 · p. 7

5.2 BASELINE GENERATORS AND DOWNSTREAM PREDICTORS

  • TABGIFT is compared with eight generator baselines and evaluated using six representative downstream predictors.Generator baselines include SMOTE, TVAE, CTGAN, TabDDPM, ARF, BN, NFlow, and TabEBM.Source: §5.2 · p. 7

5.3.1 DOWNSTREAM MODEL EFFICACY

  • TABGIFT consistently outperforms dataset-specific generative methods with limited context; its relative advantage narrows as training size grows but remains in the upper half at size 512.The comparison uses relative performance across training sizes.Source: §5.3.1 · pp. 7–8, Table 1 · p. 8

5.3.2 STATISTICAL FIDELITY AND PRIVACY PROTECTION

  • At training size 16, TABGIFT performs reasonably on statistical fidelity but is slightly weaker on privacy; methods with stronger fidelity tend to lag in privacy protection.Figure 4 compares Inv KL and KS test with DCR and Delta Presence, averaged across datasets and five seeds.Source: Figure 4 · p. 8, §5.3.2 · p. 9

5.3.3 RUNNING TIME ANALYSIS

  • On Adult, TABGIFT takes about 10.5 seconds total to generate 1,024 samples, faster than TVAE, CTGAN, TabDDPM, and NFlow.Its inference takes about 2.3 seconds; dataset-specific MLP decoder training accounts for about 8.2 seconds.Source: §5.3.3 · p. 9, Table 2 · p. 9

6 LIMITATIONS

TABGIFT's prediction-pretrained embeddings may not represent samples optimally for generation, and a dataset-specific decoder remains necessary. Synthetic-only pretraining yields suboptimal generation quality, while adding real data improves performance.

6 LIMITATIONS

  • SwiftPFN is pretrained for prediction, so its embeddings may not be optimal for generation; a dataset-specific MLP decoder is still required.The authors report that universal-decoder and jointly trained encoder–decoder alternatives yielded inferior results.Source: §6 · p. 9
  • Synthetic-only pretraining leads to suboptimal generation quality, while adding real data improves performance.The authors suggest the synthetic prior may be insufficiently diverse or realistic.Source: §6 · p. 9

7 CONCLUSION

TABGIFT shifts distribution learning from scarce target datasets to foundation-model pretraining, combining transferable priors with lightweight decoding. Across 7 classification and 11 regression tasks, it achieves best or near-best generation quality in limited-context settings while being substantially more efficient than most baselines.

7 CONCLUSION

  • Training shifts distribution learning from the target dataset to pretraining on large-scale heterogeneous tabular data, acquiring transferable structural and distributional priors.Training uses 50,000 batches of synthetic-data pretraining followed by 10,000 batches of fine-tuning on 72 real-world datasets.Source: §7 · pp. 9–10
  • Across 7 classification and 11 regression tasks, TABGIFT achieves best or near-best generation quality with context sizes of 16–64 while being substantially more efficient than most baselines.Source: §7 · p. 10
  • TABGIFT uses a frozen SwiftPFN encoder, an 18-layer Transformer denoiser, and a lightweight dataset-specific MLP decoder to generate from a 192-dimensional latent space.The denoiser uses a sample-wise attention mask for context-guided denoising while keeping query denoising independent.Source: §7 · p. 10

AI USE STATEMENT

The authors report using generative AI for writing and coding assistance, as well as literature search and auxiliary tasks. They state that they reviewed AI-assisted work and take responsibility for the final content.

AI USE STATEMENT

  • Generative AI tools assisted with writing polish and rephrasing, and with debugging and implementing standard coding routines.Source: AI USE STATEMENT · p. 11
  • The authors state that generative AI was not used for research ideation, problem formulation, or methodological design.Source: AI USE STATEMENT · p. 11
  • AI tools also supported literature search and auxiliary tasks, and the authors reviewed AI-assisted work, tested LLM-generated code, and take responsibility for the final content.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

The work reports no human-subject involvement or personal-data collection and uses publicly available standard benchmark datasets.

ETHICS STATEMENT

  • The study involved no human subjects, personal-data collection, or interaction with participants, and used publicly available standard benchmarks without collecting, releasing, or redistributing new datasets.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The authors describe reproducibility materials in the paper and appendix and intend to release the pretrained model and accompanying code upon publication.

REPRODUCIBILITY STATEMENT

  • The paper and appendix describe the architecture, training, experimental setup, datasets, preprocessing, and sampling, with release of the pretrained model and code intended upon publication.Source: REPRODUCIBILITY STATEMENT · p. 11

A DATASETS

The section describes selected OpenML benchmark datasets and an 18-dataset TALENT-tiny test set. It also inventories generation baselines and downstream predictive models.

A.1 OPENML

  • OpenML-CC18 and OpenML-CTR23 contribute 72 selected datasets across classification and regression; Tables 3–4 summarize dataset statistics.The selected datasets span diverse feature types, sample sizes, and dimensionalities.Source: §A.1 · p. 16, Table 3 · p. 16, Table 4 · p. 17

A.2 TALENT

  • The TALENT-tiny test set contains 18 datasets spanning classification and regression, selected from TALENT's curated collection.TALENT includes 300 datasets and provides TALENT-tiny for efficient, reproducible evaluation.Source: §A.2 · p. 17

B BASELINES

C DOWNSTREAM MODEL

Loading 2610.16990v1…