Source-linked AI summary
Synthesizing Tabular Data using Generative Adversarial Networks
Lei Xu, Kalyan Veeramachaneni
TL;DR
Existing synthetic-data methods are constrained in representing complex tabular distributions, especially with mixed continuous and discrete variables. TGAN uses a GAN with sequential LSTM-attention generation and specialized transformations, and it outperforms conventional synthesizers while preserving feature correlations and useful downstream model rankings.
Problem
Existing generative models are limited by the distribution functions available to users, while tabular data combines continuous and discrete variables and may contain multimodal numerical features.
Method
TGAN generates relational tables column by column using an LSTM with attention, mode-specific normalization, noise, and KL divergence for mixed variable types.
Results
TGAN consistently outperforms other data synthesizers, with an average performance gap of 5.7% versus 24.9% for GC and 43.3% for BN-Co.
Takeaways & Limitations
TGAN captures correlations between features, scales to large datasets, and often preserves machine-learning model rankings on synthetic data.
Abstract
from arXiv · showhide
Generative adversarial networks (GANs) implicitly learn the probability distribution of a dataset and can draw samples from the distribution. This paper presents, Tabular GAN (TGAN), a generative adversarial network which can generate tabular data like medical or educational records. Using the power of deep neural networks, TGAN generates high-quality and fully synthetic tables while simultaneously generating discrete and continuous variables. When we evaluate our model on three datasets, we find that TGAN outperforms conventional statistical generative models in both capturing the correlation between columns and scaling up for large datasets.
1 Introduction
Tabular data is widespread in organizational and academic machine-learning workflows, while synthetic data can address data-access and sharing constraints. Existing distribution-based approaches are limited by available distribution functions, motivating TGAN for mixed-type tabular data.
- Tabular data is the most commonly encountered data type in business and the second most common format in academia.
- Synthetic data can alleviate data-access bottlenecks, clear bureaucratic hurdles, and provide a safe space for exploration.It can also support tailored testing and tutorials while reducing risks when sharing data.
- Traditional synthetic-data generation models a joint multivariate probability distribution and samples from it.Examples include hidden Markov models for event sequences and copulas for non-linearly correlated variables.
- These generative models are restricted by the distribution functions available to users, limiting the representations used to create synthetic data.
- TGAN develops a GAN-based generator for tabular data with mixed multinomial/discrete and continuous variables.The model generates columns sequentially with LSTM attention and is evaluated statistically and through downstream machine-learning performance.
2 Related Work
Prior work includes GANs for image generation and statistical synthesizers based on trees, Bayesian networks, and copulas. TGAN differs from tableGAN through its recurrent architecture and emphasis on marginal-distribution matching.
- GAN research has largely focused on high-quality image generation, while generating discrete variables remains challenging.Prior approaches modify functions or losses, or use reinforcement learning for non-differentiable generation.
- Statistical synthetic-data methods include classification and regression trees, Bayesian networks, and copulas for modeling feature correlations.
- TableGAN uses convolutional neural networks and optimizes prediction accuracy, whereas TGAN uses recurrent networks and emphasizes marginal distributions.TGAN explicitly learns each column’s marginal distribution by minimizing KL divergence.
3 GANs for tabular data
TGAN addresses mixed-type tabular generation by transforming variables into neural-network-friendly representations and generating columns sequentially with an LSTM-based GAN. Mode-specific normalization handles multimodal numerical variables, while an MLP discriminator distinguishes real from synthetic tables.
- Task formulation: TGAN models tables containing continuous and discrete variables whose rows are independent samples from an unknown joint distribution.The goal is to generate synthetic rows that support comparable downstream model accuracy and preserve relationships between columns.
- Reversible data transformation: TGAN converts numerical variables into a scalar and a multinomial distribution, while discrete variables become multinomial distributions.These reversible transformations align the data with tanh and softmax outputs used by neural networks.
- Reversible data transformation: Multimodal numerical variables are represented using Gaussian-mixture components and normalized values to avoid tanh gradient saturation near distribution boundaries.The paper uses five Gaussian components for all numerical features, with low weights effectively suppressing unnecessary components.
- Reversible data transformation: Categorical variables are smoothed by adding noise to one-hot representations so the generator can produce differentiable probability distributions.The model later selects the most probable category during post-processing.
- Model and data generation: The LSTM generator emits columns in order, using two steps for each numerical variable and one step for each categorical variable.Attention supplies a weighted context vector over previous LSTM outputs, while numerical outputs use tanh or softmax and categorical outputs use softmax.
- Model and data generation: The discriminator concatenates transformed continuous and discrete features and feeds them to a fully connected MLP to distinguish real from generated data.GAN training optimizes the discriminator with cross-entropy while the generator attempts to fool it.
4 Evaluation Setup
The evaluation tests whether TGAN preserves column correlations and whether synthetic data can support downstream machine-learning models. Comparisons use statistical synthesizers and evaluate models trained on synthetic or real training data against held-out real test data.
- Evaluation goals: TGAN evaluation focuses on preserving correlations between table variables and supporting machine-learning models trained directly on synthetic data.These criteria frame synthetic data utility for data-science workflows.
- Machine-learning efficacy: Machine-learning efficacy is measured by training models on real and synthetic training sets, then testing both on held-out real test data.Figure 2 depicts the GAN training, synthetic-data generation, model-training, and real-test evaluation sequence.
- Correlation evaluation: Correlation preservation is quantified with pairwise normalized mutual information after discretizing each numeric variable into 20 buckets.Bucket boundaries are adjusted so each contains approximately 5% of the data; entropy is used in the metric definition.
- Baselines: TGAN is compared with three published synthesizers: Gaussian Copula, independent-column Bayesian Networks, and correlation-modeling Bayesian Networks.The baselines differ in whether they generate relational tables, model columns independently, or model inter-column correlations.
5 Results
TGAN is evaluated on three tabular datasets through synthetic-data utility, correlation preservation, and nearest-neighbor comparisons. It performs reasonably well on Census, consistently outperforms other synthesizers on KDD99 and Covertype accuracy, preserves correlations better than GC and BN-Co, and has a nearest-neighbor distance distribution close to real data.
- Machine-learning efficacy: 5.7% average performance gap separates real and TGAN-trained models on Census, versus 24.9% for GC and 43.3% for BN-Co.TGAN performs reasonably well, while GC and BN-Co show substantially larger gaps.
- Machine-learning efficacy: TGAN data often preserves the ranking of machine-learning models, allowing data scientists to evaluate models on synthetic data and select the best model.The reported implication is based on agreement between model rankings on synthetic and real data.
- Machine-learning efficacy: TGAN consistently outperforms other data synthesizers on KDD99 and Covertype accuracy comparisons.BN-Co fails on the KDD99 dataset in the reported comparison.
- Correlation preservation: TGAN learns correlations between variables significantly better than GC and BN-Co, measured by distances between real and synthetic NMI matrices.Figures 3 and 4 visualize NMI matrices, while Table 4 reports root mean square error and mean absolute error.
- Similarity to real data: TGAN’s nearest-neighbor distance distribution is very close to real data across the three evaluated datasets.The analysis compares samples from real, GC, BN-Co, and TGAN data against a training-set reference sample.
6 Conclusion
TGAN generates relational tables containing continuous and discrete variables, using specialized mechanisms for multimodal numerical distributions and discrete features. The authors report effective correlation capture, scalability for large datasets, and high-quality synthetic data, while limiting the current model to single tables with numerical and categorical features.
- TGAN generates relational tables containing continuous and discrete variables.
- Clustering numerical variables addresses multimodal distributions in continuous features.
- Noise and KL divergence are added to the loss function to generate discrete features effectively.
- The authors observe that GANs capture correlations between features and scale better for large datasets.
- The current model supports only a single table with numerical and categorical features, leaving sequential and multiple-table data for future work.