PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space

Hengrui Zhang, Jiani Zhang, Balasubramaniam Srinivasan, Zhengyuan Shen, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, George Karypis

arXiv:2310.09656v3cs.LG

TL;DR

Mixed-type tabular data challenges diffusion models because standard Gaussian-noise processes assume continuous inputs. TABSYN maps records into a VAE-crafted continuous latent space and learns score-based diffusion there, reducing average errors by 86% for column-wise distribution shape estimation and 67% for pair-wise column correlation estimation versus the most competitive baselines.

  • Problem

    Mixed numerical and categorical tabular features make standard continuous-space diffusion difficult, limiting unified modeling of such data.

  • Method

    TABSYN uses a tabular VAE to map mixed-type records into a continuous latent space, then trains a score-based diffusion model on latent embeddings.

  • Results

    86% and 67% lower average errors were achieved for column-wise distribution shape and pair-wise column correlation estimation than the most competitive baselines.

  • Takeaways & Limitations

    TABSYN provides unified mixed-feature handling, models inter-column relationships in latent space, and supports high-quality synthesis with fewer than 20 reverse steps.

Abstract

from arXiv · show

Recent advances in tabular data generation have greatly enhanced synthetic data quality. However, extending diffusion models to tabular data is challenging due to the intricately varied distributions and a blend of data types of tabular data. This paper introduces Tabsyn, a methodology that synthesizes tabular data by leveraging a diffusion model within a variational autoencoder (VAE) crafted latent space. The key advantages of the proposed Tabsyn include (1) Generality: the ability to handle a broad spectrum of data types by converting them into a single unified space and explicitly capture inter-column relations; (2) Quality: optimizing the distribution of latent embeddings to enhance the subsequent training of diffusion models, which helps generate high-quality synthetic data, (3) Speed: much fewer number of reverse steps and faster synthesis speed than existing diffusion-based methods. Extensive experiments on six datasets with five metrics demonstrate that Tabsyn outperforms existing methods. Specifically, it reduces the error rates by 86% and 67% for column-wise distribution and pair-wise column correlation estimations compared with the most competitive baselines.

1 INTRODUCTION

TABSYN addresses the difficulty of synthesizing mixed-type tabular data by learning a unified latent representation and applying score-based diffusion there. It reports broad empirical gains in data-quality estimation and faster generation than prior methods.

  • Motivation: Mixed numerical and categorical columns complicate diffusion because standard Gaussian-noise processes assume continuous inputs.Existing methods either encode categories numerically or use separate diffusion processes.
  • Approach: TABSYN maps mixed-type tabular data into a continuous embedding space and learns a score-based diffusion model over latent embeddings.A tabular VAE is designed to produce informative latent representations while preserving decoder reconstruction ability.
  • Contributions: TABSYN unifies diverse features, regularizes latent embeddings, and generates high-quality data with fewer than 20 reverse steps.The method uses a linear noise schedule to reduce the number of reverse sampling steps.
  • Empirical evaluation: 86% and 67% lower average errors are reported for column-wise distribution shape and pair-wise column correlation estimation than the most competitive baselines.Experiments compare seven state-of-the-art methods on six mixed-type datasets using over five evaluation metrics.

2 RELATED WORKS

Prior tabular generators include adversarial, variational, graph-based, autoregressive, and diffusion approaches. TABSYN extends latent diffusion to general tabular generation by combining a tabular VAE with diffusion in a continuous latent space.

  • Tabular generative models: CTGAN and TVAE address imbalanced categorical features using adversarial and variational generative models, respectively.They represent earlier deep generative approaches for tabular data.
  • TABSYN overview: TABSYN maps each row to latent space, diffuses and denoises there, then decodes generated latent samples back into data space.The overview combines column-wise tokenization, encoding, latent diffusion, decoding, and detokenization.
  • Tabular generative models: Other approaches model tabular rows with graph neural networks, natural-language autoregression, or diffusion processes.These methods provide alternatives for representing column relationships and generating mixed-type records.
  • Latent generative modeling: Latent generative models offer compact, disentangled, noise-robust, and controllable representations, while latent diffusion has shown strong image-generation performance.The paper positions TABSYN as the first application of latent diffusion to general tabular data generation.

3 SYNTHETIC TABULAR DATA GENERATION WITH TABSYN

TABSYN tokenizes mixed-type columns, encodes them with a Transformer-based VAE, and learns a score-based diffusion model over flattened latent embeddings. A scheduled VAE objective and linear noise schedule target useful representations and fast sampling.

  • Transformer Encoding and Decoding: Transformer encoders and decoders model inter-column relationships, infer latent means and variances, and reconstruct token representations.Latent embeddings are obtained with the reparameterization trick before decoding.
  • Feature Tokenizer: Each numerical or categorical column is converted into a d-dimensional token, producing a unified token matrix for every record.Numerical columns use linear transformations, while categorical columns use learnable embedding lookup tables after one-hot preprocessing.
  • Detokenizer: The detokenizer converts recovered token representations back into numerical and categorical column values.Its design is symmetric to the tokenizer.
  • Adaptive VAE training: The VAE loss combines reconstruction and KL-divergence terms, with adaptive β scheduling favoring reconstruction while maintaining an appropriate latent shape.β is reduced when reconstruction loss stalls because KL divergence is dominating the overall loss, down to βmin.
  • Latent diffusion: After VAE training, encoder outputs are flattened into vectors and a score-based diffusion model learns their distribution through forward perturbation and reverse denoising.The denoising network estimates Gaussian noise from perturbed embeddings and time, enabling reverse-process synthesis.
  • Noise scheduling: A linear noise schedule σ(t) = t reduces reverse-process approximation error and supports high-quality synthesis within fewer than 20 function evaluations.The paper links smaller approximation error to wider timestep intervals and faster sampling.

4 BENCHMARKING SYNTHETIC TABULAR DATA GENERATION ALGORITHMS

TABSYN is evaluated against seven existing methods on mixed-type tabular datasets using complementary distributional, correlation, downstream-task, and imputation measures. It consistently performs strongly, with especially large gains in column-wise density and pair-wise correlation estimation, while ablations identify latent-space diffusion and efficient sampling as important design choices.

  • Experimental setup: The benchmark compares TABSYN with seven existing generation methods on six real-world datasets containing numerical and categorical attributes.The baselines include classical GAN and VAE models, a language-model approach, and diffusion-based methods.
  • Evaluation methods: Evaluation covers column-wise density, pair-wise correlation, high-order fidelity and diversity, machine learning efficiency, and missing value imputation.Column-wise density uses KST for numerical columns and TVD for categorical columns; correlation uses Pearson correlation, contingency similarity, and mixed-type measures.
  • Low-order statistics: 86.0% lower average error in column-wise density estimation separates TABSYN from the most competitive generative baseline.STaSy performs well but treats one-hot categorical embeddings as continuous, while TabDDPM is unstable across datasets and fails to generate meaningful News data.
  • Low-order statistics: 67.6% lower average error in pair-wise column correlation estimation demonstrates TABSYN’s stronger modeling of relationships between columns.GReaT performs substantially worse on pair-wise correlations than on column-wise density, indicating limitations in capturing joint distributions.
  • Downstream tasks: TABSYN consistently outperforms all baseline methods on machine learning efficiency, although method gaps are smaller than for density and correlation estimation.This motivates evaluating synthetic data with both low-order and high-order statistics rather than relying only on downstream MLE metrics.
  • Ablation studies: Adaptive β scheduling balances reconstruction and KL-divergence behavior, while its learned embeddings achieve superior single-column density and pair-wise correlation results.The schedule uses βmax = 0.01, βmin = 10^-5, and λ = 0.7.
  • Ablation studies: Fewer than 20 NFEs yields TABSYN’s optimal results, whereas STaSy requires 50–200 NFEs and TabDDPM needs 1,000 NFEs for competitive performance.Reducing NFEs substantially harms TabDDPM, while TABSYN combines better performance with faster sampling.
  • Ablation studies: Latent-space diffusion outperforms data-space TabDDPM, and TABSYN further surpasses its DDPM variant; simple one-hot continuous treatment performs worst.These comparisons support both the learned latent representation and the tailored continuous-space diffusion design.

5 CONCLUSIONS

TABSYN combines a VAE-based latent representation with diffusion modeling to synthesize mixed-type tabular data. Its framework unifies numerical and categorical features while modeling their interrelationships and supports reverse-process sampling.

  • 5 CONCLUSIONS: TABSYN maps tabular data into a latent space with a VAE, then learns the latent distribution using a diffusion-based generative model.The framework includes separate VAE and diffusion phases for training and sampling.
  • 5 CONCLUSIONS: Numerical and categorical features are accommodated in a unified latent space, enabling the model to represent interrelationships among columns.The unified representation addresses the mixed-type structure of tabular data.
  • 5 CONCLUSIONS: Diffusion models use a forward process that adds Gaussian noise of increasing scales and a backward process that iteratively denoises samples from a noisy prior.These processes provide the general diffusion framework used for latent-space generation.
  • 5 CONCLUSIONS: TABSYN uses a variance-exploding SDE with s(t) = 1 and a linear noise level σ(t) = t, so noise is directly added to the data.The paper states that this differs from the vanilla VE-SDE noise schedule.
  • 5 CONCLUSIONS: The reverse diffusion process starts from Gaussian noise and repeatedly solves the reverse SDE before decoding the resulting latent vector into synthetic data.The decoder and detokenizer convert the final latent sample back into tabular values.

B.3 TRAINING

The denoising network is trained to estimate the score of perturbed latent data from tractable conditional noise targets. After training, this estimate replaces the intractable score during reverse-process sampling.

  • B.3 TRAINING: The denoising function approximates the score function ∇z log p_t(z), which is otherwise intractable because the marginal distribution p_t(z) is unavailable.The conditional distribution p(z_t|z_0) remains tractable and supplies the training signal.
  • B.3 TRAINING: After training, sampling solves the reverse process while replacing the true score ∇z log p_t(z) with the learned denoising function Dθ(z_t, t).This substitution enables generation from noisy latent states.
  • B.3 TRAINING: The conditional score has an analytical solution based on the perturbation kernel, allowing denoising score matching to train the network.The target is expressed through the added Gaussian noise and the schedule terms.

C PROOFS

The proofs establish that a family of diffusion SDEs can share solution trajectories and that a deterministic probability-flow ODE can represent the reverse dynamics. A first-order Euler analysis connects these dynamics to TABSYN’s sampling updates.

  • C PROOFS: Lemma 1 states that forward SDEs with different diffusion coefficient functions g(t) can share the same marginal solution trajectories for a fixed noise schedule σ(t).The corresponding reverse SDE family changes the sign of the first term.
  • C PROOFS: For a specific forward SDE, the solution trajectory can be obtained by solving the corresponding reverse equation with any admissible g(t).This gives flexibility in selecting the reverse dynamics.
  • C PROOFS: Setting g(t) = 0 yields the probability-flow ODE, a deterministic reverse process without noise terms.The paper uses this result to study the solution of TABSYN’s reverse process.
  • C PROOFS: The proof analyzes the absolute error between the exact state z_{t−∆t} and its approximation from z_t as ∆t approaches zero.The ground-truth relation follows from z_t = z_0 + σ(t)ε, while the approximation uses first-order Euler integration.
  • C PROOFS: Using the linear noise schedule, the reverse update simplifies to z_t − σ̇(t)ε∆t when the denoising estimate equals the injected noise.This connects the score-based reverse dynamics to the sampling update analyzed in the proof.

D.1 ARCHITECTURES OF VAE

TABSYN’s VAE tokenizes columns, encodes them with Transformer modules, decodes latent representations back to token matrices, and detokenizes them into reconstructed values. Its denoising network is an MLP conditioned on timestep embeddings.

  • D.1 ARCHITECTURES OF VAE: The VAE consists of a tokenizer, Transformer encoder, Transformer decoder, and detokenizer for converting tabular records to and from latent representations.The encoder and decoder use identical two-layer Transformer architectures.
  • D.1 ARCHITECTURES OF VAE: The encoder receives tokenized feature matrix E and uses separate same-architecture modules to produce the mean µ and log standard deviation log σ.Latent variables are obtained through the reparameterization trick.
  • D.1 ARCHITECTURES OF VAE: The decoder is another two-layer Transformer that maps latent input Z to reconstructed token representations ˆE for the detokenizer.The detokenizer then reconstructs the column values from the recovered token representations.
  • D.1 ARCHITECTURES OF VAE: The denoising network receives the flattened latent vector z_t and timestep t, adds sinusoidal timestep embeddings, and processes them through fully connected layers with SiLU activations.Its final linear layer produces the estimated score.
  • D.1 ARCHITECTURES OF VAE: TABSYN and its baselines are implemented with PyTorch and optimized using Adam on an Nvidia RTX 4090 GPU.This specifies the reported experimental implementation environment.

E.1 DATASETS

The experiments use six mixed-type UCI tabular datasets spanning classification and regression, and compare multiple generative-model families using distributional and correlation metrics. The evaluation also considers compatibility, robustness, quality, and efficiency.

  • Datasets: Six UCI datasets—Adult, Default, Shoppers, Magic, Beijing, and News—support classification and regression tasks.Adult, Default, Magic, and Shoppers are classification datasets; Beijing and News are regression datasets.
  • Datasets: Dataset statistics distinguish records, numerical features, and categorical features, with the target column counted according to task type.Classification targets are counted as categorical, whereas regression targets are counted as numerical.
  • Baseline methods: Baseline methods include GAN-, VAE-, autoregressive-, and diffusion-based approaches for synthetic tabular data generation.The comparison covers CTGAN, TVAE, GOGGLE, GReaT, STaSy, CoDi, and TabDDPM alongside TABSYN.
  • Evaluation criteria: TABSYN is compared with baseline methods on mixed-type compatibility, cross-dataset robustness, statistical quality, and generation efficiency.Quality uses column-wise Chi-Squared Tests, while efficiency requires satisfactory quality within fewer than 20 steps.

E.4 DETAILS OF MACHINE LEARNING EFFICIENCY EXPERIMENTS

Machine-learning efficiency is evaluated by training models on real or synthetic training data and comparing their test performance. Hyperparameters are selected on a validation split using grid search.

  • Evaluation procedure: Generative models are trained on real training data, then used to sample a synthetic set with equivalent size.The real data are split into training and testing sets before generative modeling.
  • Evaluation procedure: Machine-learning models are trained on real or synthetic training data, with performance assessed through divergence in test scores.The real training set is divided into training and validation subsets at an 8:1 ratio for model selection.
  • Hyperparameter selection: XGBoost classifier and regressor hyperparameters are selected by grid search over estimators, child weight, tree depth, and gamma.The search includes estimator counts [10, 50, 100], minimum child weights [5, 10, 20], depths [1,10], and gamma values [0.0, 1.0].
  • Implementation: The machine-learning efficiency evaluation uses metric implementations from SDMetric.
  • Timing setup: Training and sampling time are compared across methods using the Adult dataset as an example.

F.1 TRAINING / SAMPLING TIME

TABSYN preserves training time comparable to most baselines while substantially reducing sampling time. On Adult, it generates an equivalent-size synthetic dataset much faster than diffusion-based competitors.

  • Sampling time: 1.784s is TABSYN’s sampling time for an Adult synthetic dataset matching the training-set size.This is close to the one-step sampling times of CTGAN and TVAE.
  • Sampling time: 93% is TABSYN’s reported reduction in sampling time relative to the comparison methods.TabDDPM requires 28.92s for the same sampling task.
  • Training time: TABSYN has similar training time to most baseline methods despite adding a VAE training stage.Its total training time sums VAE and diffusion training time.
  • Evaluation scope: Single-column density and pair-wise correlation results do not fully evaluate joint-distribution fidelity, motivating higher-order metrics.A model may estimate individual column densities without learning the joint probability of all columns.
  • Sample-wise quality: TABSYN achieves the best α-Precision on all datasets and consistently high β-Recall across six datasets.The paper prioritizes authenticity of generated samples and then coverage of the real-data modes when assessing quality.

F.3 DETECTION: CLASSIFIER TWO SAMPLE TESTS (C2ST)

The section evaluates synthetic-data distinguishability, missing-value imputation, and target prediction through imputation. TABSYN is reported as competitive across these applications, with a stated limitation for conditional imputation.

  • Detection: The Detection score evaluates how difficult it is to distinguish real data from synthetic data using logistic regression.The metric is presented as having stronger discriminative power than several other evaluation measures.
  • Sample-wise quality: TABSYN’s α-Precision is best on every dataset, while its β-Recall remains consistently high across six datasets.Some baselines achieve higher β-Recall on individual datasets but lack stable performance or have poor α-Precision.
  • Missing-value imputation: TABSYN applies unconditional diffusion-based imputation without retraining by encoding masked data, denoising unknown latent components, and decoding the result.The latent masking vector separates known and unknown portions before reverse imputation.
  • Target prediction: Masking target columns lets TABSYN treat classification and regression as missing-value imputation tasks.The imputed target values complete the corresponding prediction tasks.
  • Results: On four of six datasets, TABSYN outperforms a classifier or regressor trained directly on real data for target-column imputation.The paper suggests that learning the smooth distribution of the entire data may reduce overfitting relative to discriminative models.
  • Limitation: TABSYN is not trained conditionally on other columns for missing-value imputation, and a separate conditional model could improve performance.The authors leave conditional training for future work.

F.6 PRIVACY PROTECTION: DISTANCE TO CLOSEST RECORD (DCR)

The privacy evaluation compares synthetic records’ distances to training and holdout records to detect potential leakage. TABSYN and other deep generative models show overlapping distance behavior, whereas SMOTE produces much smaller training-set distances.

  • Evaluation setup: The evaluation splits each dataset into equal training and holdout sets, trains the generative model only on the training portion, and samples a synthetic set of matching size.The holdout set is not used during training.
  • DCR analysis: DCR histograms compare each synthetic record’s distance to the training set against its distance to the holdout set.Training-set distances substantially closer to zero can indicate direct copying and privacy leakage.
  • Findings: Deep generative models, including TabSyn, do not show the privacy issues observed for the interpolation-based method SMOTE.Their training- and holdout-set DCR distributions are similar, while SMOTE’s training-set DCRs are much smaller.
  • Findings: A synthetic sample’s closeness probability is better when it approaches 50%, indicating similar distance distributions for training and holdout sets.Table 13 reports this DCR score as the probability that a synthetic example is closer to the training set than the holdout set.
Loading 2310.09656v3…