PaperlayerView exact revision on arXiv

Source-linked AI summary

GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators

Dingfan Chen, Tribhuvanesh Orekondy, Mario Fritz

arXiv:2006.08265v2cs.LGcs.CRstat.ML

TL;DR

Sensitive data cannot be freely released, limiting machine-learning progress in domains such as medicine. GS-WGAN selectively sanitizes generator-related gradients within a Wasserstein GAN framework, supporting centralized and decentralized training. Across datasets and metrics, the method is reported to improve privacy-preserving sample quality over prior approaches, while clipping-based sanitization and PATE-related comparability remain boundaries.

  • Problem

    Sensitive data cannot be released in its original form, motivating sanitized data publishing with rigorous privacy guarantees.

  • Method

    GS-WGAN selectively applies privacy-preserving training to the publicly released generator while using Wasserstein properties to estimate sensitivity precisely.

  • Results

    GS-WGAN consistently improves sample quality and downstream-task performance over baselines across multiple metrics and high-dimensional image datasets.

  • Takeaways & Limitations

    The framework supports privacy-preserving generation for both centralized and decentralized data, including user-level DP under an untrusted server.

  • Takeaways & Limitations

    Clipping-based sanitization can destroy gradient information and requires difficult tuning, while G-PATE is excluded from direct comparison because its data-dependent ε creates privacy costs when published.

Abstract

from arXiv · show

The wide-spread availability of rich data has fueled the growth of machine learning applications in numerous domains. However, growth in domains with highly-sensitive data (e.g., medical) is largely hindered as the private nature of data prohibits it from being shared. To this end, we propose Gradient-sanitized Wasserstein Generative Adversarial Networks (GS-WGAN), which allows releasing a sanitized form of the sensitive data with rigorous privacy guarantees. In contrast to prior work, our approach is able to distort gradient information more precisely, and thereby enabling training deeper models which generate more informative samples. Moreover, our formulation naturally allows for training GANs in both centralized and federated (i.e., decentralized) data scenarios. Through extensive experiments, we find our approach consistently outperforms state-of-the-art approaches across multiple metrics (e.g., sample quality) and datasets.

1 Introduction

Sensitive data cannot often be released in original form, motivating sanitized releases with privacy guarantees. GS-WGAN selectively sanitizes training gradients and uses a Wasserstein objective to support more informative private generation across centralized and decentralized settings.

  • Motivation: Sensitive medical and financial data often cannot be released in its original form, motivating privacy-preserving data publishing.Such publishing releases only a sanitized form with rigorous privacy guarantees.
  • Motivation: Hand-crafted differential-privacy sanitization restricts released distributions and may fail to generalize to unanticipated tasks.Traditional methods are often designed for specific input data and downstream tasks.
  • Approach: GS-WGAN selectively manipulates gradient information because only the generator and its parameters need to be publicly released.This selective design aims to preserve more meaningful gradient updates during training.
  • Approach: Coupling gradient sanitization with a Wasserstein objective and gradient penalty improves gradient flow and enables analytical sensitivity estimation.The formulation also avoids intensive searches for DP-specific clipping values.
  • Scope and evaluation: GS-WGAN supports both centralized and decentralized datasets, including user-level DP under an untrusted server in decentralized scenarios.The method is evaluated across various datasets and reported to improve privacy-preserving sample quality over state-of-the-art approaches.

2 Related Work

Prior private GAN approaches either depend on difficult gradient clipping, introduce PATE-specific limitations, or focus only on decentralized data. GS-WGAN instead sanitizes only necessary generator-related gradients and uses one framework for centralized and decentralized settings.

  • DP-SGD GAN: DP-SGD GANs can generate high-dimensional sanitized data but require carefully tuned clipping bounds whose optima vary with architecture and training dynamics.This makes implementation difficult and motivates a more targeted sanitization strategy.
  • DP-SGD GAN: GS-WGAN selectively sanitizes only the gradients necessary for privacy and exploits Wasserstein GAN theory.This contrasts with applying DP-SGD broadly to GAN training gradients.
  • PATE-based GANs: PATE-GAN trains multiple teacher discriminators on disjoint partitions alongside a student discriminator, whereas the paper uses a simplified model without a student discriminator.The comparison identifies a structural difference between PATE-GAN and the proposed framework.
  • PATE-based GANs: G-PATE requires manually binned gradients and dimension reduction because high-dimensional gradients incur high privacy costs.GS-WGAN is described as avoiding both limitations.
  • Federated GANs: Fed-Avg GAN provides user-level DP for decentralized data under a trusted server, while GS-WGAN handles centralized and decentralized data in one framework.Fed-Avg GAN also sanitizes discriminator parameter gradients similarly to DP-SGD and faces clipping-related difficulty.

3 Background

The paper uses differential privacy and its Rényi variant to formalize protection, compose privacy costs, and calibrate Gaussian noise. Post-processing preserves the privacy guarantee when releasing the trained generator.

  • Differential privacy: Differential privacy bounds how outputs change between adjacent datasets differing in one training example.In this setting, the GAN training algorithm is the randomized mechanism, and privacy loss is controlled by ε and δ.
  • Differential privacy: The paper interprets DP as limiting the difficulty of inferring whether an individual is present in the private dataset.This provides the intuitive privacy meaning of the formal output-distribution bound.
  • Rényi differential privacy: Rényi differential privacy measures privacy using Rényi divergence between outputs on adjacent datasets.The definition introduces an order λ and an ε privacy parameter for the divergence bound.
  • Rényi differential privacy: RDP offers convenient composition for accumulating privacy costs across sequential mechanisms such as multiple gradient-descent steps.A sequence of RDP mechanisms composes by accumulating their per-step privacy parameters.
  • Gaussian mechanism: The Gaussian mechanism adds calibrated noise to a function's output according to its sensitivity.The paper builds its approach on this mechanism and uses it to sanitize gradients.
  • Post-processing: DP remains valid after applying any function to a differentially private mechanism's output, supporting release of the trained generator.This is the post-processing property used for generator release.

4 Proposed Method

GS-WGAN trains a differentially private generator by sanitizing only necessary gradient information, using Wasserstein objectives to reduce clipping distortion and support centralized or decentralized data.

  • GANs learn a data distribution with a discriminator distinguishing real from generated samples and a generator mapping random noise to samples.
  • The sanitization mechanism clips gradients and adds Gaussian noise, while each generator update satisfies (λ, 2Bλ/σ^2)-RDP for batch size B.
  • DP-SGD provides privacy but clipping can destroy gradient information and requires difficult, data- and architecture-sensitive tuning of the clipping bound.
  • GS-WGAN sanitizes only generator-related gradients because the released model contains the generator, while the local generator Jacobian is independent of training data.
  • The Wasserstein-1 objective and gradient penalty constrain discriminator gradients, yielding lower gradient-norm variance and an optimal clipping threshold of C=1.
  • Subsampling supports decentralized training, with client-side gradient sanitization providing privacy under an untrusted server and transferring sample gradients rather than discriminator-parameter gradients.

5 Experiment

Experiments evaluate GS-WGAN on image and federated datasets using sample quality, downstream utility, privacy-utility trade-offs, and federated debugging performance. Across these settings, GS-WGAN improves reported quality and utility while supporting privacy-preserving federated training.

  • Experiment Setup: Experiments evaluate conditional image generation on MNIST and Fashion-MNIST using Inception Score, FID, and downstream classifier accuracy.The evaluation also measures privacy through ε and trains downstream classifiers on 60k privately generated data points.
  • Experiment Setup: GS-WGAN uses DCGAN and ResNet architectures to improve sample quality while reducing privacy costs through architectural design and warm-start strategies.
  • Comparison with Baselines: 94% higher IS on MNIST (9.23 vs. 4.76 of DP-SGD GAN) and 45% higher IS on Fashion-MNIST (5.32 vs. 3.68 of DP-Merf AE) are reported for GS-WGAN.
  • Comparison with Baselines: 25% higher MLP accuracy on MNIST (0.79 vs. 0.63 of DP-Merf) and 16% higher accuracy on Fashion-MNIST (0.65 vs. 0.56 of DP-Merf) are reported for GS-WGAN.
  • Influence of Hyperparameters: A subsampling rate of 1/1000 provides a relatively good privacy-utility trade-off, while further reduction does not necessarily improve results.
  • Influence of Hyperparameters: Across iterations, IS increases by 10-90% and FID decreases by 20-60% compared to baselines, while GS-WGAN achieves better quality with fewer iterations.
  • Federated Setting Evaluation: On Federated EMNIST, GS-WGAN yields 0.28× smaller FID and 104× smaller ε than Fed-Avg GAN, while tolerating ten times larger noise scale.

6 Conclusion

The conclusion presents GS-WGAN as a differentially private approach for sanitizing sensitive high-dimensional datasets while preserving sample informativeness. Its evaluation reports that sanitized samples retain usefulness for downstream model training.

  • GS-WGAN sanitizes sensitive high-dimensional datasets with provable privacy guarantees while preserving informativeness in the sanitized samples.
  • Selective privacy-preserving training of the publicly released generator allows the discriminator to be trained optimally before being discarded.
  • The Wasserstein objective exploits discriminator Lipschitzness to estimate sensitivity precisely without exhaustive hyperparameter search.
  • The evaluation reports that sanitized samples preserve informativeness and allow downstream models to be trained.

7 Broader Impact

Privacy-preserving generative models support releasing useful data without changing downstream analysis algorithms. The paper positions GS-WGAN as improving privacy-preserving generation, particularly for high-dimensional data.

  • Privacy-preserving generative models can support applications without changing analysis algorithms for sanitized data.
  • The work aims to make generative modeling compatible with data privacy and reports improved state-of-the-art privacy-preserving data generation.
  • Success on high-dimensional data indicates potential for a broader range of applications.

A Privacy Analysis

The privacy analysis bounds the cost of sanitized generator updates using RDP, subsampling, composition, and conversion to (ε, δ)-DP. The proposed training procedures support both centralized and federated settings.

  • Privacy accounting combines RDP bounds for gradient sanitization, analytical moments accounting for subsampling, composition across iterations, and conversion to an (ε, δ) guarantee.
  • Each generator update step satisfies (λ, 2Bλ/σ2)-RDP, where B is the batch size.
  • The sanitization mechanism clips gradients and adds Gaussian noise before generator updates.
  • Rényi-divergence data processing and additivity properties support the privacy proof for batched generator updates.
  • Centralized Setting: The centralized algorithm subsamples the dataset into parallel subsets and trains discriminators before producing a differentially private generator.
  • Federated Setting: The federated algorithm trains across clients in parallel and outputs a differentially private generator with total privacy cost ε.

C.2 Datasets

The experiments use MNIST, Fashion-MNIST, and federated EMNIST, evaluating sample quality and downstream usefulness with established metrics and classifiers.

  • Datasets: MNIST and Fashion-MNIST each contain 60K training images and 10K testing images across 10 classes, with 28 × 28 image dimensions.
  • Datasets: Federated EMNIST contains 3,400 users, 671,585 training examples, and 77,483 testing examples of handwritten letters and numbers.
  • Evaluation: Sample quality is evaluated using Inception Score and Frechet Inception Distance, while downstream usefulness is evaluated through classification accuracy.
  • Evaluation: Inception Score favors generated samples that are sharp, contain clear objects, and cover diverse classes.
  • Evaluation: Lower Frechet Inception Distance indicates a smaller discrepancy between real and generated samples and better sample quality and diversity.
  • Evaluation: Downstream evaluation uses MLP, CNN, and 11 scikit-learn classifiers.

C.4 Baseline Methods

The baseline comparison covers established differentially private data-generation methods and documents their experimental configurations and high-dimensional limitations.

  • Baselines: The evaluated baselines are DP-Merf, DP-Merf AE, DP-SGD GAN, and G-PATE.
  • Configurations: DP-Merf and DP-Merf AE use batch size 500 with different noise scales and training durations.
  • Configurations: DP-SGD GAN uses clipping bound C=1.1, noise scale σ=2.1, batch size 600, and 30K training iterations.
  • Configurations: G-PATE uses 2,000 teacher discriminators, batch size 30, noise scales σ1=600 and σ2=100, threshold T=0.5, and projection dimension 10.
  • Limitations: PATE-GAN training on high-dimensional images stalls within 1K iterations because discriminator and generator gradients vanish.

D Additional Results

Additional results show that GS-WGAN clips gradients with less distortion and variance than DP-SGD, while maintaining improvements across baseline comparisons and privacy-utility settings.

  • Gradient Clipping: GS-WGAN’s clipping produces a much smaller difference between average gradient norms before and after clipping than DP-SGD.The clipping bounds are 1 for GS-WGAN and 1.1 for DP-SGD.
  • Gradient Clipping: GS-WGAN gradients exhibit much less variance than DP-SGD gradients both before and after clipping.
  • Baseline Comparison: The baseline results are averaged over five runs and report calibrated accuracy alongside detailed quantitative results.
  • Privacy-Utility Curves: GS-WGAN consistently improves over prior methods across a broad range of privacy budgets ε on Fashion-MNIST.Figure 7 varies noise scale and iterations and reports means and standard deviations over three runs.
  • Tables: Table 3 reports classification accuracy on MNIST at ε=10 and δ=10−5.
  • Tables: Table 4 reports classification accuracy on Fashion-MNIST at ε=10 and δ=10−5.
Loading 2006.08265v2…