PaperlayerView exact revision on arXiv

Source-linked AI summary

TabDDPM: Modelling Tabular Data with Diffusion Models

Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, Artem Babenko

arXiv:2209.15421v2cs.LG

TL;DR

Tabular data combines heterogeneous numerical and categorical features, making accurate generative modeling challenging. TabDDPM adapts diffusion to both feature types and is evaluated through synthetic-data utility and realism benchmarks. It generally outperforms GAN/VAE rivals, while privacy comparisons favor it over SMOTE, though real-world privacy protection remains unresolved.

  • Problem

    Accurate generative modeling is challenging for tabular data because features can be heterogeneous and datasets are often relatively small.

  • Method

    TabDDPM combines multinomial diffusion for categorical and binary features with Gaussian diffusion for numerical features.

  • Results

    TabDDPM consistently produces higher-quality synthetic data than GAN/VAE-based rivals on most considered benchmarks and outperforms SMOTE when privacy must be ensured.

  • Takeaways & Limitations

    TabDDPM provides a general mixed-type tabular diffusion model, while SMOTE remains a competitive baseline for machine-learning utility.

  • Takeaways & Limitations

    The experiments do not determine whether TabDDPM satisfies real-world privacy applications, and the DCR measure omits some sensitive-feature leakage cases.

Abstract

from arXiv · show

Denoising diffusion probabilistic models are currently becoming the leading paradigm of generative modeling for many important data modalities. Being the most prevalent in the computer vision community, diffusion models have also recently gained some attention in other domains, including speech, NLP, and graph-like data. In this work, we investigate if the framework of diffusion models can be advantageous for general tabular problems, where datapoints are typically represented by vectors of heterogeneous features. The inherent heterogeneity of tabular data makes it quite challenging for accurate modeling, since the individual features can be of completely different nature, i.e., some of them can be continuous and some of them can be discrete. To address such data types, we introduce TabDDPM -- a diffusion model that can be universally applied to any tabular dataset and handles any type of feature. We extensively evaluate TabDDPM on a wide set of benchmarks and demonstrate its superiority over existing GAN/VAE alternatives, which is consistent with the advantage of diffusion models in other fields. Additionally, we show that TabDDPM is eligible for privacy-oriented setups, where the original datapoints cannot be publicly shared.

1. Introduction

The paper asks whether diffusion models can extend effectively to general tabular problems despite heterogeneous features and small datasets. It introduces TabDDPM and reports stronger performance than GAN- and VAE-based alternatives, with privacy-oriented advantages over SMOTE.

  • Diffusion models are investigated beyond vision in NLP, waveform processing, molecular graphs, and time series.
  • Tabular modeling is difficult because features may be heterogeneous and datasets are often relatively small.
  • TabDDPM is a simple DDPM design applicable to any tabular task with mixed numerical and categorical features.
  • TabDDPM outperforms GAN-based and VAE-based alternatives across several tabular datasets.
  • TabDDPM’s synthetic data is preferable to SMOTE’s for privacy-concerned substitution of real user data.

2. Related Work

Related work establishes diffusion models as iterative generative models and surveys tabular alternatives, including shallow interpolation methods. It motivates comparing TabDDPM with simple synthetic-data baselines such as SMOTE.

  • Diffusion models approximate a target distribution through the endpoint of a Markov chain starting from a parametric distribution, typically Gaussian.
  • Tabular synthetic-data research addresses high demand for quality data, limited dataset sizes, and privacy constraints on sharing real user data.
  • “Shallow” synthetics generation: SMOTE generates synthetic points by convexly combining a real point with a nearest neighbor and was originally designed for minority-class oversampling.
  • “Shallow” synthetics generation: Prior work reports that SMOTE can outperform tabular GANs for minority-class oversampling, motivating it as a strong simple baseline.

3. Background

The background describes diffusion as forward corruption followed by learned reverse denoising, with Gaussian processes for continuous data and multinomial processes for categorical data. Figure 1 presents the TabDDPM classification scheme using timestep, class-label, and logit variables.

  • The forward diffusion process gradually adds noise to data through predefined transition distributions with variances β1,...,βT.
  • The reverse process gradually denoises a latent variable and generates new samples, with unknown reverse distributions approximated by a neural network.
  • Gaussian diffusion models operate in continuous spaces and use Gaussian distributions for their forward and reverse processes.
  • The practical Gaussian objective simplifies to mean-squared error between predicted and ground-truth noise across timesteps.
  • Multinomial diffusion models generate one-hot categorical variables by corrupting categories with uniform noise over K classes.
  • Figure 1 depicts TabDDPM classification, with t denoting timestep, y class label, and ℓ logits.
  • The reverse distribution is parameterized using a neural network prediction of the clean sample, and training maximizes a variational lower bound.

4. TabDDPM

TabDDPM models heterogeneous tabular data by combining diffusion processes for numerical and categorical features. Its design includes task-specific conditioning, an MLP reverse model, and hyperparameter tuning because these choices affect effectiveness.

  • Data representation: TabDDPM uses Gaussian diffusion for numerical features and multinomial diffusion for categorical and binary features.Categorical inputs are one-hot encoded, while numerical features are normalized; the resulting input dimensionality is Nnum + P Ki.
  • Reverse process: The reverse model outputs Gaussian noise predictions for numerical coordinates and categorical predictions for multinomial diffusion coordinates.The reverse process is modeled by a multi-layer neural network whose output matches the dimensionality of x0.
  • Training objective: The classification model is trained with an MSE loss for Gaussian diffusion and KL-divergence losses for each multinomial diffusion.The total multinomial loss is divided by the number of categorical features.
  • Task conditioning: Classification uses a class-conditional model, whereas regression treats the target as an additional numerical feature and learns the joint distribution.For classification, the learned reverse transition is conditioned on y.
  • Architecture: TabDDPM uses an adapted MLP reverse architecture that processes the tabular input, timestep, and class label.Sinusoidal time embeddings have dimension 128, and all Linear layers use projection dimension 128.
  • Hyperparameter tuning: Hyperparameters strongly influence model effectiveness, so the paper provides recommended hyperparameter search spaces and describes a tuning procedure.The main hyperparameters and their search spaces are listed in Table 1.

5. Experiments

The experiments compare TabDDPM with leading tabular generative models across diverse datasets, evaluating distributional fidelity, machine-learning utility, and privacy. TabDDPM generally outperforms GAN/VAE alternatives, while SMOTE remains competitive for utility and TabDDPM is preferable in privacy-concerned settings.

  • Experimental setup: The study evaluates TabDDPM against TVAE, CTGAN, CTABGAN, CTABGAN+, and SMOTE on 15 diverse real-world public datasets.The baselines represent leading available approaches from major generative-modeling paradigms, while SMOTE provides a shallow interpolation-based comparison.
  • Qualitative comparison: TabDDPM generally captures correlations better, with Table 3 ranking models by Wasserstein distance, Jensen–Shannon divergence, and correlation-matrix L2 distance.Lower average ranks indicate smaller distances from the real data.
  • Qualitative comparison: Feature-distribution comparisons show that TabDDPM produces more realistic distributions than TVAE and CTABGAN+ in most cases.The advantage is especially pronounced for uniformly distributed numerical features, high-cardinality categorical features, and mixed-type features.
  • Machine Learning efficiency: In both machine-learning-efficiency protocols, TabDDPM significantly outperforms TVAE and CTABGAN+ on most datasets.The protocols evaluate models trained on synthetic data using classification or regression performance on real test data, including tuned CatBoost and MLP settings.
  • Machine Learning efficiency: SMOTE is competitive with TabDDPM for machine-learning efficiency and often significantly outperforms GAN/VAE approaches.The authors argue that evaluation against tuned state-of-the-art models is more appropriate because weak-model scores can be misleading for practitioners.
  • Privacy: TabDDPM is more private than SMOTE and more resistant to a full black-box attack, while retaining state-of-the-art machine-learning efficiency.The privacy comparison uses distance-to-closest-record values, where low values indicate near-duplicates that may violate privacy requirements.

Limitations and discussion

The paper identifies important privacy and modeling limitations: TabDDPM does not guarantee high privacy and utility simultaneously, DCR is incomplete, and alternative feature-diffusion designs remain possible.

  • TabDDPM does not guarantee both high privacy and high ML utility, and its real-world privacy suitability remains unresolved.The authors report greater privacy than SMOTE but state that further investigation is needed.
  • DCR is an incomplete privacy measure because L2 record distance ignores feature importance and may miss leakage when sensitive features coincide.
  • The categorical-feature component uses multinomial diffusion, although alternative approaches could also be applied to TabDDPM.
  • TabDDPM has better DCR than SMOTE but lower DCR than TVAE and CTABGAN+ in the reported comparison.The caption attributes the latter comparison to lower ML efficiency of the GAN/VAE alternatives.
  • Future numerical-feature extensions could distinguish real-valued, positive real-valued, and ordinal variables.

6. Conclusion

The conclusion finds that diffusion modeling can handle mixed-type tabular data and generally produces higher-quality synthetic data than GAN/VAE rivals. SMOTE remains a competitive utility baseline, while TabDDPM is preferable when privacy must be ensured.

  • TabDDPM’s synthetic data has consistently higher quality than GAN/VAE-based rivals across most considered benchmarks.
  • TabDDPM uses a DDPM design that handles mixed numerical and categorical features.
  • SMOTE provides competitive ML utility and should be considered a simple, effective baseline.
  • TabDDPM outperforms SMOTE when synthetic data must preserve privacy.

A. MLP evaluation and tuning

The evaluation studies whether CatBoost-based hyperparameter tuning transfers to MLP evaluation. The reported results indicate that CatBoost guidance produces synthetic data useful for MLPs.

  • CatBoost-guided hyperparameter tuning produces TabDDPM synthetics that are also optimal for other classifiers and regressors.
  • Table 8 compares MLP evaluation with MLP tuning against CatBoost evaluation with CatBoost tuning.
  • CatBoost tuning provides useful synthetic data for MLP evaluation.

B. Additional results

Additional experiments evaluate marginal feature distributions, correlations, density, coverage, and privacy in latent representations. The reported comparisons indicate strong distributional and privacy advantages for TabDDPM over relevant alternatives.

  • The additional analysis evaluates feature-distribution accuracy using Wasserstein distance for numerical features and Jensen-Shannon divergence for categorical features.
  • Density and coverage metrics are included as alternatives to precision and recall for evaluating synthetic data.
  • Correlation-matrix differences are compared using L2 distance, with lower visual difference indicating better correlation capture; TabDDPM captures correlations better overall.
  • TabDDPM often models individual feature distributions more accurately than CTABGAN+ and TVAE.
  • DCR computed on pretrained MLP features yields conclusions similar to the original-space analysis, with SMOTE still significantly underperforming TabDDPM.

E. Hyperparameters Search Spaces

The paper specifies hyperparameter search spaces for CatBoost, MLP, SMOTE, CTABGAN variants, and TVAE.

  • Hyperparameter search spaces are reported for CatBoost and MLP baselines.
  • The SMOTE search space includes a range for the interpolation coefficient λ.
  • Separate search spaces are provided for CTABGAN and CTABGAN+ implementations.
  • The TVAE hyperparameter search space is also documented with reference to an official implementation.

F. Datasets

The experiments use seven tabular datasets spanning income, real estate, health, churn, social-media, gesture, and particle-physics data. The paper also reports the computational environment and runtime evaluation setup for TabDDPM.

  • Datasets: The benchmark suite includes Adult, California Housing, Cardiovascular Disease, Churn Modeling, Facebook Comments Volume, Gesture Phase Prediction, and Higgs.The Higgs experiments use the 98K-sample version available through OpenML.
  • Datasets: The Higgs dataset represents simulated physical particles and uses 98K samples from the OpenML repository.
  • Experimental setup: Experiments ran on Ubuntu 20.04 with a GeForce RTX 2080 Ti GPU, an Intel i7-7800X CPU, and specified PyTorch, CUDA, scikit-learn, and imbalanced-learn versions.
  • Runtime: Runtime examples use T = 1000 and batch size = 4096, while hyperparameter tuning typically involves 50 runs and takes 8–10 hours.Sample time measures generating the requested number of samples.
  • Runtime: Table 20 reports training and sampling times for TabDDPM.
Loading 2209.15421v2…