PaperlayerView exact revision on arXiv

Source-linked AI summary

CDMD: A Cross-Dataset Mixed-Type Diffusion Model for Tabular Data

Mohamed Amine Ketata, Maximilian Schambach, Stephan Günnemann

arXiv:2609.39124v1cs.LG

TL;DR

Tabular generators are often trained separately for each dataset, limiting knowledge transfer and requiring many specialized models. CDMD jointly models heterogeneous datasets with direct mixed-type diffusion and a shared schema-aware denoiser. It achieves strong average generation quality in joint training and improves generation on unseen datasets after pre-training, including under limited adaptation data or epochs.

  • Problem

    Tabular generative models are typically trained separately for each dataset, limiting knowledge transfer and requiring storage of many specialized models.

    Source: §1 · p. 1

  • Method

    CDMD jointly models heterogeneous datasets through direct mixed-type diffusion, composing feature-level processes by schema and using a shared schema-conditional denoiser.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 5

  • Results

    A single jointly trained checkpoint matches or outperforms separately trained generators on average, while pre-training on 337 datasets improves unseen-dataset generation under limited target data and adaptation epochs.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The shared schema-conditional architecture provides a starting point for scaling tabular generative models to larger and more diverse corpora.

    Source: §5 · p. 10

    Adaptation currently requires fine-tuning and additional optimization for each target dataset, and performance at larger scales remains an open question.

    Source: §5 · p. 10

Abstract

from arXiv · show

Generative models for tabular data are typically trained separately for each dataset, limiting knowledge transfer and requiring the storage of many specialized models. In this paper, we introduce CDMD, a tabular diffusion model trained jointly across heterogeneous datasets with different schemas and variable numbers of numerical and categorical features. Unlike existing cross-dataset tabular diffusion models that operate in continuous representation spaces, CDMD defines diffusion directly over the mixed-type feature space and is trained end-to-end. To accommodate heterogeneous categorical domains, we introduce a schema-restricted reverse-process parameterization for masked diffusion models, in which the output space dynamically adapts to each feature's vocabulary. We then compose numerical and categorical feature-level diffusion processes into a schema-dependent row-level process. A shared schema-aware Transformer denoiser captures dependencies between features and parameterizes the reverse process across varying schemas. On seven real-world datasets, a single jointly trained CDMD achieves the highest average generation quality among strong single-dataset and cross-dataset baselines, while using substantially fewer total parameters than the collection of separately trained models. Furthermore, pre-training on a corpus of 337 datasets improves generation on previously unseen datasets under both limited target data and limited adaptation epochs. These results demonstrate the potential of direct mixed-type diffusion for shared and transferable tabular data generation. Our code is available at https://github.com/ketatam/cdmd.

1 INTRODUCTION

The introduction motivates shared tabular generation as a way to transfer knowledge across heterogeneous datasets, then presents CDMD as direct mixed-type diffusion with strong reported results in both joint generation and adaptation.

1 INTRODUCTION

  • Separate models for each dataset require repeated training and storage, limiting knowledge transfer and potentially compromising generation quality on small datasets.Cross-dataset models may share reusable patterns, which can support limited-data generation and faster adaptation.Source: §1 · p. 1
  • Existing cross-dataset approaches use continuous representations, adding encoding and decoding stages and making generation quality depend on latent modeling and reconstruction accuracy (Lin et al., 2025).Mixed-type diffusion had demonstrated effectiveness for single-dataset generation, motivating investigation in the original feature space (Shi et al., 2025a).Source: §1 · pp. 1–2
  • CDMD combines numerical and categorical diffusion directly in heterogeneous datasets’ mixed-type feature spaces, composing feature-level processes by schema and using a shared Transformer denoiser.Its reverse-process parameterization adapts to each categorical feature’s vocabulary, while the denoiser handles variable-sized rows and schemas.Source: §1 · p. 2
  • A single jointly trained CDMD achieves higher average synthetic-data quality than state-of-the-art single-dataset baselines on seven real-world datasets, using a fraction of their combined parameter count.Pre-training on 337 datasets also yields higher generation quality than baselines under limited target data or adaptation epochs.Source: §1 · p. 2

2 PROBLEM FORMULATION

The problem formulation treats each dataset as rows over a schema that identifies feature properties, and frames generation as approximating the dataset’s unknown row distribution. It distinguishes separate per-dataset models from cross-dataset modeling approaches.

2 PROBLEM FORMULATION

  • A dataset is represented by a schema and rows; the schema specifies each column’s name, type, and domain, and numerical features are normalized using training-column statistics.The formulation separates numerical and categorical columns and defines a row space induced by the schema.Source: §2 · p. 2
  • Tabular generation aims to approximate the unknown distribution of iid dataset rows with a parametric distribution that can generate synthetic rows.Single-dataset models learn a separate distribution for each dataset.Source: §2 · p. 2
  • Cross-dataset approaches may condition a shared distribution on schema or use target rows as context; only the latter requires target rows during sampling.This work focuses on schema-conditional modeling, which samples given the target schema alone.Source: §2 · p. 3

3 CROSS-DATASET MIXED-TYPE DIFFUSION MODEL (CDMD)

CDMD composes feature-level numerical and categorical diffusion processes into a schema-dependent row process and trains a shared schema-aware denoiser across varying schemas.

3 CROSS-DATASET MIXED-TYPE DIFFUSION MODEL (CDMD)

  • CDMD’s fine-to-coarse design dynamically composes one-dimensional numerical and categorical processes into a row-level process for each table schema.Sampling initializes numerical features with Gaussian noise and categorical features with masks, then reverses the process using schema-conditioned predictions.Source: §3 · p. 3, §3.2 · p. 5

3.1 FEATURE-LEVEL DIFFUSION PROCESSES

  • Numerical features use Gaussian diffusion and a schema-conditioned denoiser that predicts clean values from noisy inputs.The denoiser is trained across features with an expected L2 denoising loss.Source: §3.1 · p. 3
  • Categorical features use masked diffusion over a joint category universe, with reverse predictions restricted to each feature’s valid vocabulary.The vocabulary-restricted parameterization predicts over categories in V, whose size is independent of the global universe U.Source: §3.1 · p. 4

3.2 ROW-LEVEL DIFFUSION PROCESS

  • The row-level process applies type-specific feature noising independently, while joint row denoising captures distributions beyond column marginals.The variable-sized denoiser is trained end-to-end across datasets using a combined mixed-type loss.Source: §3.2 · p. 5

3.3 SCHEMA-CONDITIONAL DENOISER NETWORK ARCHITECTURE

  • A shared tokenizer–Transformer encoder–detokenizer uses schema information to model cross-feature dependencies and is equivariant to joint column and schema permutations.All parameters are shared across datasets, with type-specific modules shared across columns of the same type.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate joint cross-dataset generation and pre-trained adaptation under limited target data or training epochs. CDMD leads on average in the in-domain comparison and in both constrained settings.

4.1 IN-DOMAIN CROSS-DATASET GENERATION

  • 85.09 Generation Quality tops TABSYN (84.27) and TABDIFF (84.22); 81.77 Overall Evaluation Score exceeds TABSYN (81.57).CDMD also achieves the best generation-quality score on six of seven datasets, using 10.7M parameters versus 74.8M for TABSYN and TABDIFF across the seven datasets.Source: Table 1 · p. 7, §4.1 · p. 8

4 EXPERIMENTS

  • Evaluation covers seven real-world datasets and aggregates distributional fidelity, predictive utility, and privacy proxies into equally weighted evaluation axes.Each dataset is split into fitting and held-out test sets; higher individual metric scores indicate better performance.Source: §4 · p. 7

4.2 GENERATION IN CONSTRAINED SETTINGS

  • Pre-trained CDMD has the highest cross-dataset-average Overall Evaluation Score at every target-row budget, with its clearest advantage at tens to a few hundred examples.As target rows increase, the average gap generally narrows as models trained from scratch improve.Source: Figure 1 · p. 8, §4.2 · p. 9
  • Pre-trained CDMD reaches the highest cross-dataset-average Overall Evaluation Score after one epoch and retains the advantage through epoch ten.It approaches its observed plateau within a few epochs.Source: Figure 2 · p. 9, §4.2 · p. 9

5 CONCLUSION

CDMD shares a schema-conditioned denoiser across heterogeneous datasets and models mixed-type features directly. Experiments report competitive average generation and transfer under constrained adaptation, while the paper identifies fine-tuning and experiment scale as boundaries.

5 CONCLUSION

  • On seven real-world datasets, one jointly trained checkpoint matches or outperforms strong separately trained generators on average; pre-training on 337 datasets improves unseen-dataset generation under limited data and epochs.The shared schema-conditioned architecture combines numerical and categorical diffusion directly in mixed-type feature space.Source: §5 · pp. 9–10, Figure 3 · p. 15
  • Adaptation still requires fine-tuning and additional optimization for each target dataset, and performance at larger scales remains an open question.The paper identifies in-context generation without parameter updates as future work.Source: §5 · p. 10

A RELATED WORK

  • Single-dataset generators handle mixed columns with different representations or diffusion processes, but their reported generation experiments fit each method to an individual dataset.CDMD addresses sharing the mixed-type generator across datasets with different schemas and categorical vocabularies.Source: A RELATED WORK · p. 14
  • Cross-dataset approaches described here use continuous representations, while CDMD diffuses directly over mixed numerical and categorical states with vocabulary-restricted categorical predictions.Related approaches include shared latent generators and in-context latent diffusion with target-specific reconstruction components.Source: A RELATED WORK · p. 14, Figure 3 · p. 15
  • Predictive tabular foundation models are trained for conditional target prediction; generative adaptations add procedures such as feature-wise autoregressive sampling, energy-based sampling, or latent diffusion and decoding.CDMD instead learns directly in mixed-type feature space with shared schema-conditioned input and output modules.Source: A RELATED WORK · p. 14

B METHOD DETAILS

The appendices outline how CDMD trains and samples across schemas, along with its schedules and dataset-sampling choices. They also note an implementation condition for datasets lacking one feature type.

B METHOD DETAILS

  • Training samples a dataset, a clean row, and diffusion time, then corrupts numerical and categorical features, predicts denoised values, and optimizes combined numerical and categorical losses.If a dataset lacks one feature type, its corresponding loss is set to zero.Source: B METHOD DETAILS · p. 15, §B.1 · p. 17
  • Sampling initializes numerical features with Gaussian noise and categorical features with masks, then iteratively predicts and reverses their feature-level processes.The sampler applies forward perturbations at intermediate steps, takes an Euler step for numerical features, and samples categorical states from the reverse process.Source: B METHOD DETAILS · p. 16

B.1 DIFFUSION MODEL DETAILS

  • The Gaussian schedule uses σ_min = 0.002, σ_max = 80, and ρ = 7; the categorical masking schedule decreases linearly over time.The numerical loss uses the weighting defined by Karras et al. (2022).Source: §B.1 · p. 16
  • The appendices also specify forward masking transitions and a row-conditioned reverse categorical process for individual columns.The reverse process is described for a column j conditioned on the row.Source: §B.1 · p. 16
  • Dataset sampling weights depend on dataset size through τ_data; cross-dataset experiments set τ_data = 0 for equal sampling, while pre-training uses τ_data = 0.5.N(ℓ) denotes the number of training samples in dataset ℓ.Source: §B.1 · p. 17

B.2 ARCHITECTURE DETAILS

CDMD uses a shared Transformer architecture across datasets and is designed to generate schema-valid rows without dependence on column order. Its construction satisfies both stated desiderata.

B.2 ARCHITECTURE DETAILS

  • The shared model has a six-layer, eight-head Transformer encoder and 10,655,234 trainable parameters, excluding the frozen text embedding model.The text embedding dimension is 768 and token dimension is 384; parameters are shared across datasets and columns of each type.Source: §B.2 · p. 17

B.3 DESIDERATA FOR CROSS-DATASET TABULAR GENERATIVE MODELS

  • For an arbitrary schema, the desired distribution assigns zero probability to invalid rows and is invariant to jointly permuting columns and schema entries.The section names these properties schema-valid support and column permutation invariance.Source: Desiderata for Cross-Dataset Tabular… · p. 17

Desiderata for Cross-Dataset Tabular Generative Models

  • CDMD satisfies both desiderata by restricting categorical predictions to each column’s vocabulary and using a permutation-equivariant denoiser without positional encodings.Numerical values remain in R, and categorical predictions assign zero mask probability at t = 0.Source: Desiderata for Cross-Dataset Tabular… · p. 17

C.1 METRICS

The metrics assess marginal and pairwise fidelity, sample-level fidelity and coverage, memorization, predictive utility, and proximity to training records.

Shape

  • Shape measures statistical similarity between real and synthetic column-marginal distributions, using the Kolmogorov-Smirnov (KS) test for numerical features and Total Variation (TV) distance for categorical features, then aggregating test complements across all columns.Source: §C.1 · pp. 17–18

Trend

  • Trend compares real and synthetic pairwise correlations using Pearson correlations for numerical pairs and normalized contingency tables for categorical pairs.Mixed numerical-categorical pairs use contingency computations after discretizing the numerical column; higher normalized scores indicate greater similarity.Source: §C.1 · p. 18

α-Precision and β-Recall

  • α-Precision measures synthetic samples’ resemblance to typical real samples, while β-Recall measures coverage of typical synthetic samples over real data.The overall score aggregates deviations from their optimal values, using 30 equally spaced grid points and normalization to [0, 100].Source: §C.1 · p. 18

Authenticity

  • Authenticity estimates the fraction of synthetic rows farther from their nearest training record than that record’s nearest other training record.The metric is used to estimate potential memorization.Source: §C.1 · p. 19

Machine Learning Efficacy (MLE)

  • Machine Learning Efficacy evaluates predictive utility by training an XGBoost model on synthetic data and testing it on held-out real data.It uses AUC for classification and normalized RMSE for regression, with scores reported on a [0, 100] scale.Source: §C.1 · p. 19

Distance to closest record (DCR)

  • Distance to closest record measures whether synthetic rows are closer to the training set than to the holdout set using Gower’s mixed-type distance.Values q ≤ 0.5 indicate no excessive proximity to the training set and are preferred.Source: §C.1 · p. 19

C.2 EVALUATION DATASETS

Evaluation covers seven datasets across demographic, finance, physics, e-commerce, healthcare, environmental-monitoring, and online-media domains, with classification and regression tasks.

Dataset coverage

  • The evaluation uses six TABSYN datasets plus Diabetes, spanning varied domains and prediction tasks.Table 2 summarizes dataset sizes, feature composition, categorical cardinalities, and prediction tasks; feature counts include the target.Source: §C.2 · pp. 19–20, Table 2 · p. 20

Classification datasets

  • Adult, Default, Magic, Shoppers, and Diabetes provide binary or multiclass classification tasks across demographics, finance, physics, e-commerce, and healthcare.The tasks predict income threshold, payment default, gamma-ray events, purchases, and diabetes readmission class, respectively.Source: §C.2 · p. 20

Regression datasets

  • Beijing and News provide regression tasks predicting PM 2.5 concentrations and online-news social-media shares, respectively.Source: §C.2 · p. 20

C.3 PRE-TRAINING DATASETS

CDMD’s pre-training corpus comprises 337 heterogeneous datasets curated from a multi-source collection, with flagged evaluation overlaps removed.

Corpus distribution

  • Figure 4 shows row-count and total, numerical, and categorical column-count distributions across the 337 pre-training datasets.Each dataset contributes one observation; row counts use 50 logarithmically spaced bins, while column counts use one bin per integer and reflect retained features after preprocessing.Source: Figure 4 · p. 21

Corpus source

  • The pre-training corpus draws on heterogeneous numerical and categorical tables from OpenML, UCI, Kaggle, and Data.gov, curated from TabPertNet/OpenTabs.The curated subset covers binary classification and regression and excludes datasets with uninformative column names or unmappable categorical codes.Source: Table 3 · p. 20, §C.3 · p. 21

Overlap screening

  • After comparing names, columns, marginal distributions, and categorical values and manually inspecting flagged datasets, the authors removed 10 of 347 datasets, leaving 337.Flagged copies or subsets of evaluation datasets were removed.Source: §C.3 · p. 21

C.4 IMPLEMENTATION DETAILS

Implementation uses training-only preprocessing and a shared baseline protocol; the section also introduces detailed evaluation results and an in-domain results table.

C.4 IMPLEMENTATION DETAILS

  • Missing numerical values use training-column means, missing categorical values get a dedicated category, and numerical features are quantile-transformed using training data only.In few-shot settings, the transformation is fitted on the available N training samples; inverse transformations return generated values to their original scale.Source: §C.4 · p. 21
  • Baselines use identical splits, training subsets, and metrics, official implementations with default hyperparameters, and original benchmark configurations.For two-stage models, an epoch counts a complete pass over both the autoencoder and latent diffusion model.Source: §C.4 · p. 21
  • CDMD uses ADAMW at a 10^-4 learning rate for 2,000 epochs, batch size 4096, and 50 linear-grid sampling steps; few-shot fine-tuning freezes the transformer encoder.The learning rate warms up over the first 5% of steps and drops by 0.9 after 50 consecutive epochs without loss improvement.Source: §C.4 · p. 21
  • Few-epoch adaptation skips learning-rate warmup and trains for the specified number of epochs.Source: §C.4 · p. 22

D DETAILED RESULTS

D.1 CROSS-DATASET GENERATION

The in-domain results are organized by evaluation axis, while few-shot and few-epoch generation quality are also plotted across datasets and their cross-dataset averages.

D.1 CROSS-DATASET GENERATION

D.2 FEW-SHOT GENERATION

  • Figure 5 plots few-shot performance for seven datasets and their cross-dataset average, with mean curves and one-standard-deviation shading across seeds.Source: §D.2 · p. 22, Figure 5 · p. 24

D.3 FEW-EPOCH GENERATION

  • Figure 6 plots few-epoch performance for seven datasets and their cross-dataset average, with mean curves and one-standard-deviation shading across seeds.Source: §D.3 · p. 23, Figure 6 · p. 24

Related papers

Loading 2609.39124v1…