PaperlayerView exact revision on arXiv

Source-linked AI summary

CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesis

Chaejeong Lee, Jayoung Kim, Noseong Park

arXiv:2304.12654v2cs.LGcs.AI

TL;DR

Tabular synthesis still faces difficulty modeling discrete variables, and treating them in continuous spaces may compromise correlations with continuous variables. CoDi uses separate, mutually conditioned continuous and discrete diffusion models, reinforced with contrastive learning. Across 11 real-world benchmark datasets and 8 baselines, it outperforms other methods in sampling quality and diversity and improves sampling time by about 90% versus STaSy.

  • Problem

    Discrete variables remain challenging to model, and treating them in continuous spaces may yield suboptimal results and compromise correlations between continuous and discrete variables.

    Source: §1 · p. 1

  • Method

    CoDi uses separate diffusion models for continuous and discrete variables, conditions each on the other, and applies contrastive learning to bind them.

    Source: §1 · p. 2

  • Results

    Across 11 real-world benchmark datasets and 8 baselines, CoDi outperforms others in sampling quality and diversity and improves sampling time by about 90% versus STaSy.

    Source: §5 · p. 9

  • Takeaways & Limitations

    CoDi achieves a well-balanced generative learning trilemma across sampling quality, diversity, and sampling time in the reported experiments.

    Source: §1 · p. 2, §5 · p. 9

    CoDi may not apply to data containing only continuous or only discrete variables because it is designed to learn both types simultaneously.

    Source: §5 · p. 9

Abstract

from arXiv · show

With growing attention to tabular data these days, the attempt to apply a synthetic table to various tasks has been expanded toward various scenarios. Owing to the recent advances in generative modeling, fake data generated by tabular data synthesis models become sophisticated and realistic. However, there still exists a difficulty in modeling discrete variables (columns) of tabular data. In this work, we propose to process continuous and discrete variables separately (but being conditioned on each other) by two diffusion models. The two diffusion models are co-evolved during training by reading conditions from each other. In order to further bind the diffusion models, moreover, we introduce a contrastive learning method with a negative sampling method. In our experiments with 11 real-world tabular datasets and 8 baseline methods, we prove the efficacy of the proposed method, called CoDi.

1. Introduction

Tabular synthesis methods face difficulty modeling discrete variables, and treating continuous and discrete variables alike may compromise their correlations. CoDi uses separate, mutually conditioned diffusion models and contrastive learning; the paper reports improvements over state-of-the-art methods and evaluates on 11 datasets against 8 baselines.

1. Introduction

  • Discrete variables remain challenging to model, and processing them in continuous spaces can produce suboptimal results and compromise inter-column correlations.The cited discussion particularly identifies correlations between continuous and discrete variables.Source: §1 · p. 1
  • CoDi processes continuous and discrete variables with separate diffusion models, each operating in its corresponding space.The models are proposed to better learn the distributions of their respective variable types.Source: §1 · p. 2
  • The two models co-evolve by conditioning on each other’s perturbed samples during diffusion and denoising.At each forward step, each model reads the other variable type’s perturbed sample as a condition.Source: §1 · p. 2
  • Contrastive learning further binds the models using negative conditions formed by permuting pairs of continuous and discrete variable sets.The process is applied separately to the continuous and discrete diffusion models.Source: §1 · p. 2
  • The paper reports that its design improves the generative learning trilemma—sampling quality, diversity, and sampling time—over state-of-the-art methods.Experiments on 11 datasets and 8 baselines are described as showing effective modeling of real-world tabular distributions.Source: §1 · p. 2

2. Background

Diffusion models gradually corrupt data and learn a reverse process that generates samples from noise. Tabular synthesis methods address mixed continuous and discrete variables, but their processing choices can affect how well the joint distribution is modeled.

2.1. Diffusion Probabilistic Models

  • Diffusion probabilistic models corrupt a sample through a forward Markov process and learn a reverse process that generates a fake sample from noise.The reverse transition is approximated by a neural network, with parameters optimizing a variational upper bound on negative log-likelihood.Source: §2.1 · p. 2

2.1.1. CONTINUOUS SPACES

  • Continuous-space diffusion adds Gaussian noise according to a variance schedule, and its network predicts noise to generate a fake sample.The continuous prior is a standard normal distribution.Source: §2.1.1 · pp. 2–3

2.1.2. DISCRETE SPACES

  • Discrete-space diffusion uses categorical distributions and uniform noise; its posterior permits closed-form computation of a KL divergence term.Here, C denotes a categorical distribution, K the number of categories, and β_t controls the added uniform noise.Source: §2.1.2 · p. 3

2.2. Tabular Data Synthesis

  • Tabular synthesis spans methods for continuous or discrete data and mixed-type models, including score-based approaches that generate minority classes or entire datasets.Figure 2 depicts co-evolving conditional diffusion models connected by reading conditions from each other.Source: §2.2 · p. 3, Figure 2 · p. 3

3. Proposed Method

CoDi uses separate continuous- and discrete-space diffusion models whose denoising processes condition on each other. Contrastive training further connects the models using generated positive samples and mismatched negative conditions, and the training and sampling algorithms combine these losses and iteratively produce records.

3. Proposed Method

  • CoDi introduces two design points for mixed-type synthesis: co-evolving conditional diffusion models and contrastive training to connect them.Its network architecture modifies U-Net.Source: §3 · p. 3

3.1. Co-evolving Conditional Diffusion Models

  • Separate continuous- and discrete-space diffusion models are trained, but each reads conditions from the other because their diffusion and denoising processes are intercorrelated.At each forward step, the variable sets are perturbed simultaneously; during reverse denoising, each model is conditioned on both types of samples at the previous step.Source: §3.1 · pp. 3–4

3.2. Contrastive Learning

  • Contrastive learning uses real samples as anchors, generated samples as positives, and inappropriate counterparts as negatives to bring positives closer to anchors.Negative conditions are formed by randomly shuffling the continuous and discrete variable sets; predicted model outputs estimate positive and negative samples to avoid generating them through T diffusion steps each iteration.Source: §3.2 · pp. 4–5

3.3. Training & Sampling Algorithms

  • Training combines each diffusion loss with its contrastive loss, updates both model parameters, and sampling iteratively denoises noise before post-processing outputs.Continuous outputs are reverse-scaled and discrete outputs use Argmax.Source: §3.3 · p. 5

4. Experiments

Across 11 datasets and 8 generative-model baselines, the experiments evaluate sampling quality, diversity, and time, then test discrete-space processing, negative sampling, and contrastive learning. CoDi performs strongly across the evaluated dimensions, with discrete-space modeling and the proposed negative sampling method favored in their respective comparisons.

4. Experiments

  • The experiments use 11 datasets and 8 generative models, evaluating quality, diversity, and the wall-clock time to generate 10K samples.Diversity and sampling time are measured five times using different fake samples, with means and standard deviations reported.Source: §4 · p. 5, §4.1.1 · p. 5, §4.1.2 · p. 6

4.2.1. SAMPLING QUALITY

  • CoDi outperforms STaSy in all reported sampling-quality cases and is the only method with positive R2 in regression.The reported distributions also show CoDi's fake data resembling real data more closely than STaSy's for the illustrated columns.Source: Table 1 · p. 6, §4.2.1 · p. 6, Figure 5 · p. 6

4.2.2. SAMPLING DIVERSITY

  • CoDi exceeds other methods in sampling diversity, particularly on multi-class classification and regression datasets.In Stroke, CoDi outperforms TableGAN and CTGAN by large margins in diversity, despite their good sampling quality.Source: §4.2.2 · pp. 6–7, Table 2 · p. 7

4.2.3. SAMPLING TIME

  • CoDi samples about 9 times faster than STaSy while maintaining reliable sampling quality and diversity.Its sampling time is comparable to other GAN-based methods.Source: Table 3 · p. 7, §4.2.3 · p. 7

4.3. Continuous vs. Discrete Spaces for Discrete Data

  • Discrete diffusion produces better F1 and coverage than continuous diffusion on all four discrete-only datasets.The continuous model also fails to retain real category counts in the illustrated columns, showing that processing space affects generation performance.Source: Table 4 · p. 7, §4.3 · p. 7, Figure 6 · p. 7

4.4. Negative Sampling Methods

  • The proposed Method 3 performs best in sampling quality and diversity among the three negative-sampling methods.It shuffles discrete-variable rows together to preserve their inter-correlations; its contrastive training loss also converges more stably than the first two methods.Source: Table 5 · p. 8, §4.4 · p. 8, Figure 7 · p. 8

4.5. Ablation Study on Contrastive Learning

  • Contrastive learning improves F1 in Bank, Seismic, Faults, and Obesity, diversity in Heart, and both measures in other datasets.The authors report these ablations as evidence that their anchor, positive, and negative sample design is reasonable.Source: Table 6 · p. 8, §4.5 · p. 8

5. Conclusions

CoDi uses separate continuous and discrete diffusion models that condition on each other's outputs, with contrastive learning to reinforce their connection. Across 11 real-world benchmark datasets and 8 baselines, it outperforms other methods in sampling quality and diversity, while its design may not apply to single-type data.

5. Conclusions

  • CoDi conditions its continuous and discrete diffusion models on each other's outputs during training, and uses contrastive learning to reinforce their connection.This co-evolution is described as maintaining correlations between continuous and discrete variables.Source: §5 · pp. 8–9
  • Across 11 real-world benchmark datasets and 8 baselines, CoDi outperforms other methods in sampling quality and diversity.Compared with STaSy, it improves sampling time by about 90%, attributed to reduced training difficulty.Source: §5 · p. 9
  • CoDi may not be applicable to datasets containing only continuous variables or only discrete variables.The authors note that real-world tabular data typically contains mixed types.Source: §5 · p. 9
  • Realistic generated data carries a risk of abuse for immoral purposes, motivating active research on privacy protection in generative models.The authors cite Lee et al. (2021) and Hyeong et al. (2022) on privacy protection.Source: §5 · p. 9

A. Preliminary Experiment

In the toy dataset, CoDi's generated data is more similar to real data than STaSy's, which produces noisy samples at class boundaries.

A. Preliminary Experiment

  • CoDi's fake data is more similar to real data than STaSy's on the four-column toy dataset.The x- and y-axes are continuous variables; 16 colors and circle membership encode discrete variables, and STaSy produces noisy samples at class boundaries.Source: A. Preliminary Experiment · p. 11, Figure 9 · p. 11

B. Network Architecture

CoDi uses a U-Net-based architecture for its two diffusion models, conditioning continuous generation on discrete variables and using time embeddings and encoder–decoder skip connections. The reverse-process subsection defines transition probabilities for the co-evolving conditional diffusion models.

B. Network Architecture

  • The continuous diffusion model uses a U-Net modification with fully connected layers and encoder-to-decoder skip connections, and is conditioned on discrete variables.The same architecture is used for the discrete diffusion model.Source: Figure 10 · p. 11, B. Network Architecture · p. 11
  • The continuous model takes xC_t and time t as inputs, uses xD_t as a condition, and outputs ϵθC.Its four layers are input, condition, bottom, and output; two encoding and two decoding blocks are linked by skip connections, with time conditioned on both block types.Source: B. Network Architecture · p. 11
  • The condition layer uses xD_t to produce hidden vector hc, while temb uses a sinusoidal positional embedding and a fully connected layer.The hidden vector hc has half the dimension of xC_t.Source: B. Network Architecture · p. 11

C. The Reverse Transition Probabilities for the Co-evolving Conditional Diffusion models

  • Proposition 3.1 defines the forward and reverse processes, and the subsection specifies reverse transition probabilities for each co-evolving conditional diffusion process.The supplied passage identifies the transition probabilities but does not state their formulas.Source: C. The Reverse Transition… · p. 12

D.1. Experimental Environments

The appendix reports the experimental computing environment and the hyperparameter search settings used for CoDi’s continuous and discrete diffusion models.

D.1. Experimental Environments

  • Experiments used Ubuntu 18.04.6 LTS, Python 3.10.8, PyTorch 1.11.0, CUDA 11.7, an NVIDIA Driver 470.161.03, an i9 CPU, and an NVIDIA RTX 3090.Source: §D.1 · p. 12

D.2. Hyperparameter Settings for CoDi

  • The hyperparameter search varied learning rate, time-embedding dimensions, symmetric encoder–decoder hidden dimensions, and contrastive-loss coefficients.The embedding dimensions were searched over {16, 32, 64, 128}; hidden dimensions were selected from four specified triples according to dataset size, and λC and λD from 0.2 to 0.8 in 0.1 increments.Source: §D.2 · p. 12
  • CoDi used a linear βt noise schedule from 1e-05 to 2e-02 and 50 total diffusion timesteps.Source: §D.2 · p. 12

D.3. Datasets

The experiments use datasets selected to vary in their numbers of continuous and discrete columns, then compare CoDi with generative-model baselines spanning several model families.

D.3. Datasets

  • The datasets were selected with their numbers of continuous and discrete variables in mind, and Table 9 summarizes their statistics.The table defines #Train and #Test as training and test counts, and NC and ND as continuous- and discrete-column counts.Source: §D.3 · p. 13, Table 9 · p. 13
  • The dataset examples span prediction tasks including term-deposit subscription, heart disease, earthquake impact, phishing, contraceptive choice, and steel-plate fault detection.Source: §D.3 · pp. 13–14

D.4. Baselines

  • The baselines cover GAN-based, flow-based, and score-based generators, including methods designed for mixed-type variables and tabular data.The listed examples include CTGAN and TVAE for mixed-type challenges, RNODE as a customized flow-based model, and STaSy as a score-based tabular generator.Source: §D.4 · p. 14

E.1. Sampling Quality

CoDi outperforms other methods in almost all sampling-quality cases and is the only method with positive R2 across all regression datasets. Its sampling diversity is comparable to the original data in many cases, and it samples faster than STaSy.

E.1. Sampling Quality

  • CoDi outperforms the other methods in almost all cases and is the only method with positive R2 across all regression datasets.Sampling quality is evaluated using F1 and AUROC for classification and R2 and RMSE for regression; the full results are reported across Tables 10–12.Source: §E.1 · p. 15, Table 12 · p. 15

E.2. Sampling Diversity

  • CoDi shows sampling diversity comparable to the original data in many cases, measured using coverage.The coverage results are summarized for binary classification, multi-class classification, and regression datasets.Source: §E.2 · p. 16, Table 13 · p. 16, Table 14 · p. 16

E.3. Sampling Time

  • CoDi samples much faster than STaSy, while TableGAN and TVAE are fast in almost all cases and RNODE takes the longest time.Sampling time was measured as wall-clock time to generate 10K records, with five runs summarized by mean and standard deviation.Source: §E.3 · p. 17, Table 16 · p. 17, Table 17 · p. 17

Related papers

Loading 2304.12654v2…