Source-linked AI summary
DP-CGAN: Differentially Private Synthetic Data and Label Generation
Reihaneh Torkzadehmahani, Peter Kairouz, Benedict Paten
TL;DR
Sensitive data enables GAN-based synthetic-data generation but creates privacy risks, while prior private GANs often omit labels and lack single-digit-epsilon demonstrations. DP-CGAN separately clips real- and fake-data discriminator gradients, adds Gaussian noise, generates data and labels, and uses RDP accounting. The authors report promising MNIST results with single-digit epsilon and an 87.57% AUROC versus 92.17% for direct training on real data.
Problem
Prior differentially private GANs generally generated synthetic data without labels, and high-quality MNIST-like generation with single-digit epsilon had not been demonstrated.
Method
DP-CGAN separately clips discriminator gradients on real and fake data, adds Gaussian noise, generates synthetic data and labels, and tracks privacy with an RDP accountant.
Results
87.57% AUROC was achieved with DP-CGANs versus 92.17% when the classifier was trained directly on real data.
Takeaways & Limitations
The authors characterize DP-CGAN as a first stepping stone toward high-quality GANs with strong differential-privacy guarantees.
Takeaways & Limitations
The cited related-work discussion notes that one compared privacy method assumes access to public data and answers only sufficiently consensual queries.
Abstract
from arXiv · showhide
Generative Adversarial Networks (GANs) are one of the well-known models to generate synthetic data including images, especially for research communities that cannot use original sensitive datasets because they are not publicly accessible. One of the main challenges in this area is to preserve the privacy of individuals who participate in the training of the GAN models. To address this challenge, we introduce a Differentially Private Conditional GAN (DP-CGAN) training framework based on a new clipping and perturbation strategy, which improves the performance of the model while preserving privacy of the training dataset. DP-CGAN generates both synthetic data and corresponding labels and leverages the recently introduced Renyi differential privacy accountant to track the spent privacy budget. The experimental results show that DP-CGAN can generate visually and empirically promising results on the MNIST dataset with a single-digit epsilon parameter in differential privacy.
1 Introduction
DP-CGAN addresses privacy risks in GAN training by combining differential privacy with conditional generation of synthetic data and labels. The framework reports promising MNIST results at a single-digit epsilon parameter.
- Motivation: GANs can approximate training-data distributions, but training on sensitive data exposes individuals to privacy attacks.Reported attack types include model inversion and membership inference.
- Research gap: Recent differentially private GANs generally generate unlabeled synthetic data and have not demonstrated high-quality MNIST-like digits with single-digit epsilon.These limitations restrict usefulness for supervised learning applications.
- Framework: DP-CGAN clips discriminator gradients on real and fake data separately, then adds Gaussian noise to their accumulated clipped gradients.The generator is trained with a non-private optimizer while privacy accounting tracks the discriminator updates.
- Framework: DP-CGAN generates synthetic data with corresponding labels and uses an RDP accountant to track the spent privacy budget.RDP accounting is described as providing a tighter privacy-budget bound than the classical Moment Accountant, permitting less added noise without compromising guarantees.
- Results: The authors report preliminary good visual and empirical MNIST results with a single-digit epsilon parameter.They characterize the work as a first stepping stone toward high-quality GANs with strong differential-privacy guarantees.
- Results: 87.57% AUROC was obtained with DP-CGANs versus 92.17% when training the classifier directly on real data.The synthetic data and labels were used together to train machine-learning models, whose accuracy was tested on real data.
2 Preliminaries
The preliminaries introduce GANs and conditional GANs as generative-model foundations, then define differential privacy and mechanisms for protecting sensitive-data training. They also describe noise addition and privacy-budget composition.
- GAN and CGAN: GANs jointly train a generator that produces synthetic data and a discriminator that distinguishes generated samples from training data.The two networks are trained simultaneously in an adversarial minimax game.
- GAN and CGAN: A GAN objective represents the trade-off between generator and discriminator performance in a minimax game.The generator receives random noise, while the discriminator evaluates generated and input data.
- GAN and CGAN: A conditional GAN conditions both generator and discriminator on side information such as class labels.Its objective is likewise formulated as a minimax game.
- Differential Privacy: Differential privacy aims to ensure that outputs reveal population-level information without revealing information about a particular individual.The framework compares outputs on databases differing in one sample.
- Differential Privacy: Gaussian-noise mechanisms perturb real-valued function outputs at a scale determined by the function’s sensitivity.Sensitivity is defined through the maximum absolute output difference on adjacent databases.
- Differential Privacy: Differential privacy supports composition, so privacy losses from multiple mechanisms can be combined and accounted for.A standard composition bound scales k mechanisms’ parameters to at least (kϵ, kδ)-differential privacy.
3 Related Work
Prior work addressed privacy in machine learning and GANs using distributed training, teacher ensembles, gradient perturbation, and privacy accountants, but retained limitations in accounting, labels, data access, or applicability. DP-CGAN differs by using RDP accounting, generating multi-class labels, and separately clipping real- and fake-data gradients before noise addition.
- Shokri et al. used distributed local training and parameter sharing, but the overall privacy loss could exceed expectations.
- The Moment Accountant tracks privacy loss for Gaussian mechanisms with random sampling and improves estimation over standard composition.
- PATE assumes access to public data and answers only queries with sufficiently high teacher consensus.
- DPGAN perturbs discriminator gradients, but its privacy accounting is unclear and its MNIST results were not promising.
- PATE-GAN avoids public training data but produces only binary synthetic labels and was evaluated on small datasets.
- DP-CGAN uses RDP accounting, generates multi-class labels, and adds Gaussian noise after separately clipping real- and fake-data discriminator gradients.
4 Our Approach
DP-CGAN privately trains a conditional generator and discriminator while producing synthetic data with corresponding labels. It clips real- and fake-data discriminator gradients separately, adds Gaussian noise, and tracks accumulated privacy loss with RDP accounting until the target budget or iteration limit is reached.
- DP-CGAN generates synthetic data and corresponding labels while preserving the privacy of training samples.
- Gaussian noise is injected into discriminator optimization, and post-processing gives the generator differential privacy.
- RDP accounting tracks spent privacy loss and provides a tighter estimate than the Moment Accountant.
- Training continues while the iteration count and spent privacy budget remain below their respective targets.
5 Experimental Results
The experiments compare DP-CGAN with a non-private CGAN and a CGAN using the standard differentially private approach. All models use a vanilla two-layer conditional GAN architecture, with real samples and labels supplied to the discriminator and noise-label pairs supplied to the generator.
- DP-CGAN is compared with an unprivate CGAN and a CGAN trained using the standard differentially private approach.
- The generator takes random noise z and label y, while the discriminator receives real sample x and label y.
- The generator and discriminator in every model use two fully connected layers.
- The training procedure samples real-data batches with labels from the data distribution.
- Each iteration also samples a noise batch from the noise prior for generator-discriminator training.
16 Compute the overall gradients of discriminator and add Gaussian Noise to them
The implementation includes a step to update the RDP accountant after computing and perturbing the discriminator gradients.
- The RDP accountant is updated during the training procedure.
20 Accumulate the spent privacy budget using
The section references the RDP Accountant during an update to the generator network.
- The RDP Accountant is referenced in the generator-network update.
- The passage labels an operation for updating the generator network.
- The section connects privacy accounting with generator-network updating.
24 Take the gradient Descent step for generator
The evaluation compares vanilla, non-private, and differentially private CGAN variants using classifier performance and generated-image quality. DP-CGAN is assessed against standard privacy-preserving training under the stated experimental setting.
- Experimental setup: AuROC evaluates classifiers trained on generated data and labels, with testing performed on real MNIST test data.
- Results: DP-CGAN's AuROC exceeds that of CGAN trained with the basic differentially private method.
- Experimental setup: ϵ = 9.6 and δ = 10^-5 define the privacy setting used for the Table 1 comparison.
- Results: DP-CGAN produces higher-quality images than the basic differentially private CGAN but lower-quality images than the non-private CGAN.
6 Conclusion
The conclusion presents DP-CGAN as a differentially private CGAN that generates synthetic data and labels through separate gradient clipping and RDP accounting. Results are promising on MNIST, while the authors identify simple architectures and preliminary evaluation as boundaries for future work.
- Conclusion: DP-CGAN generates synthetic data and corresponding labels under differential privacy.
- Conclusion: The framework clips discriminator-loss gradients on real and fake data separately, sums them, and adds Gaussian noise.
- Conclusion: RDP accounting tracks the spent privacy budget during DP-CGAN training.
- Conclusion: DP-CGAN improves performance compared to basic DP-CGAN and produces promising results on MNIST.
- Limitations and future work: The generator and discriminator architectures are relatively simple, motivating future evaluation with deeper convolutional CGANs.
- Limitations and future work: The results remain preliminary, with planned evaluation on more challenging datasets such as CIFAR100 and CelebA/B.