PaperlayerView exact revision on arXiv

Source-linked AI summary

P3GM: Private High-Dimensional Data Release via Privacy Preserving Phased Generative Model

Shun Takagi, Tsubasa Takahashi, Yang Cao, Masatoshi Yoshikawa

arXiv:2006.12101v4cs.LGcs.CRcs.DBstat.ML

TL;DR

Existing differentially private synthesis methods struggle to release useful high-dimensional sensitive data because privacy-preserving training can inject prohibitive noise. P3GM uses a two-phase VAE-based training process that improves noise robustness, and its synthesized data yields less noisy, more diverse samples and stronger classification performance than competing private methods in the reported evaluations.

  • Problem

    Existing differentially private data synthesis methods cannot adequately handle high-dimensional data because their noise requirements can make outsourced analysis meaningless.

  • Method

    P3GM separates encoder training from decoder training with a fixed encoder, using a two-phase process to improve robustness to differential-privacy noise.

  • Results

    P3GM generates less noisy and more diverse samples than DP-VAE and DP-GM, while synthetic-data classifiers achieve much better results than DP-GM and PrivBayes and remain relatively close to VAE accuracy.

  • Takeaways & Limitations

    P3GM provides a differentially private generative-model approach for releasing high-dimensional sensitive data while preserving utility for reported data-mining tasks.

Abstract

from arXiv · show

How can we release a massive volume of sensitive data while mitigating privacy risks? Privacy-preserving data synthesis enables the data holder to outsource analytical tasks to an untrusted third party. The state-of-the-art approach for this problem is to build a generative model under differential privacy, which offers a rigorous privacy guarantee. However, the existing method cannot adequately handle high dimensional data. In particular, when the input dataset contains a large number of features, the existing techniques require injecting a prohibitive amount of noise to satisfy differential privacy, which results in the outsourced data analysis meaningless. To address the above issue, this paper proposes privacy-preserving phased generative model (P3GM), which is a differentially private generative model for releasing such sensitive data. P3GM employs the two-phase learning process to make it robust against the noise, and to increase learning efficiency (e.g., easy to converge). We give theoretical analyses about the learning complexity and privacy loss in P3GM. We further experimentally evaluate our proposed method and demonstrate that P3GM significantly outperforms existing solutions. Compared with the state-of-the-art methods, our generated samples look fewer noises and closer to the original data in terms of data diversity. Besides, in several data mining tasks with synthesized data, our model outperforms the competitors in terms of accuracy.

I. INTRODUCTION

Privacy-preserving data synthesis enables sharing synthetic data instead of sensitive records, but existing differentially private methods struggle with high-dimensional datasets because required noise can make outsourced analysis meaningless. P3GM addresses this gap with a two-phase, noise-robust generative model intended to preserve privacy, support high-dimensional data, and retain utility.

  • Motivation: Differentially private data synthesis shares a private generative model rather than raw sensitive datasets with an untrusted third party.Differential privacy provides a rigorous guarantee without restrictive assumptions about the adversary.
  • Motivation: Existing differentially private synthesis methods require prohibitive noise for high-dimensional data, potentially making outsourced analysis meaningless.High-dimensional release also requires generated data to approximate the original distribution while supporting data-mining utility.
  • Related work: Existing approaches have important scope and utility limitations: PrivBayes is unsuitable for high-dimensional data, PATE-GAN was demonstrated on low-dimensional tables, and DP-VAE produces noisy samples.GAN-based methods also face mode collapse, evaluation difficulty, and slow convergence, which complicate differentially private training.
  • Proposed approach: P3GM is a differentially private generative model designed to protect each data holder, handle high-dimensional original data, and approximate the original distribution for data-mining utility.The model supports sharing sensitive data with an untrusted third party for analysis while preserving privacy.
  • Proposed approach: P3GM separates VAE-style encoder and decoder training into two phases, fixing the encoder during decoder training to improve robustness to differential-privacy noise and stabilize optimization.The paper also analyzes why this phased procedure can work better than end-to-end training under differential privacy.
  • Contributions: P3GM includes a realization, theoretical privacy analysis, and classification experiments in which it outperforms state-of-the-art techniques under the same privacy protection level.The reported comparisons concern [5] and.

B. Preview of Results

The preview contrasts privacy-preserving synthetic-data approaches and highlights P3GM’s visual utility relative to competing models.

  • B. Preview of Results: P3GM produces less noisy and more diverse MNIST samples than DP-VAE and DP-GM under (1, 10^-5)-DP.Its samples are visually closer to the original data and the non-private VAE samples.
  • B. Preview of Results: VAE uses latent variables, an approximate posterior, and differentiable ELBO optimization to train its generative model.The reparameterization trick enables backpropagation through sampled latent variables.
  • B. Preview of Results: VAE generation samples z from N(0, I) and decodes it through pθ(x|z) to produce synthetic data.

B. Differential Privacy

Differential privacy bounds disclosure risk by constraining outputs on neighboring datasets, while composition and RDP provide tighter privacy accounting.

  • B. Differential Privacy: Differential privacy requires randomized outputs on neighboring datasets to remain quantitatively similar for every output subset.Neighboring datasets differ by Hamming distance one.
  • B. Differential Privacy: A randomized mechanism protects a function by perturbing its output according to the function’s sensitivity.Sensitivity measures the maximum output change between neighboring datasets.
  • B. Differential Privacy: Sequentially applying ε_i-DP mechanisms yields a total privacy budget of Σ_i ε_i under basic composition.
  • B. Differential Privacy: Basic composition is not tight, motivating zCDP, moments accountant, and Rényi differential privacy for sharper privacy-loss analysis.
  • B. Differential Privacy: RDP composes additively at a fixed order and converts to (ε + log(1/δ)/(α−1), δ)-DP.

D. Differentially Private Mechanisms

The paper combines differentially private EM, PCA, and SGD mechanisms to estimate model components while controlling sensitivity and privacy loss.

  • D. Differentially Private Mechanisms: DP-EM estimates Gaussian-mixture parameters privately by adding Gaussian noise during each M-step.The noise scales with parameter sensitivity, and each iteration satisfies (ε_i, δ_i)-DP.
  • D. Differentially Private Mechanisms: DP-EM’s per-step privacy analysis depends on the Gaussian-noise scale parameter σ_e.
  • D. Differentially Private Mechanisms: Privacy-preserving PCA adds Gaussian noise to the covariance matrix and achieves differential privacy under bounded sensitivity.Clipping ensures the required norm bound for the data.
  • D. Differentially Private Mechanisms: DP-SGD clips per-example gradients before adding noise so stochastic optimization can train models under differential privacy.
  • D. Differentially Private Mechanisms: The moments accountant computes a tighter composition bound for repeated DP-SGD steps using the batch sampling probability s = B/N.

III. PROBLEM STATEMENT

The paper frames private model release as training a differentially private generative parameter and introduces PGM’s two-phase training to improve robustness to optimization noise.

  • III. PROBLEM STATEMENT: Publishing a trained parameter instead of raw data still requires differentially private training because the parameter can encode individual records.
  • III. PROBLEM STATEMENT: Naive DP-VAE trains encoder and decoder jointly with DP-SGD, but its ELBO objective is too vulnerable to noise.
  • III. PROBLEM STATEMENT: PGM has theoretically weaker expressive power than VAE but greater tolerance to DP-SGD noise.
  • III. PROBLEM STATEMENT: PGM separates training into Encoding and Decoding phases, using dimensional reduction first and then training the decoder with a fixed encoder.
  • III. PROBLEM STATEMENT: The fixed encoder stabilizes AEVB when SGD is replaced by DP-SGD, providing PGM’s noise tolerance.

B. Encoding Phase

The Encoding Phase reduces dimensionality and privately estimates a latent-variable distribution, then fixes the encoder before decoding.

  • Encoding Phase: PCA provides a differentially private dimensionality reduction for high-dimensional data, helping avoid the curse of dimensionality in GMM estimation.The reduced representation is used to approximate the latent distribution more tractably.
  • Encoding Phase: The dimensionality-reduction objective measures reconstruction potential: smaller reconstruction error indicates that encoded data can be decoded more accurately.A reconstruction function g is used to assess the reduction f.
  • Encoding Phase: The encoder is fixed by mapping each input to a reduced representation whose distribution is approximated by a tractable model such as a mixture of Gaussians.The parameter λ is estimated so encoded data follows rλ(z).
  • Encoding Phase: The latent distribution is estimated by minimizing a KL-divergence objective, using EM because the objective matches maximum likelihood estimation.The decoder is later trained using the estimated latent distribution.
  • Encoding Phase: Privacy loss arises from dimensionality reduction and latent-distribution estimation, while component-wise guarantees combine through composition.The encoder’s variance remains trainable during decoding, so not all encoder parameters are fixed.

C. Decoding Phase

The Decoding Phase trains the decoder and remaining encoder parameters with differentiable Monte Carlo estimates, while privately fitting a latent mixture model.

  • Decoding Phase: The differentiable LELBO is approximated with Monte Carlo samples and optimized with SGD using the reparameterization trick.The likelihood term is differentiable for Bernoulli or Gaussian MLP decoders.
  • Decoding Phase: P3GM uses neural networks for the decoder and encoder, whose outputs parameterize the means and variances of their distributions.This preserves the VAE-style probabilistic architecture while supporting phased training.
  • Decoding Phase: The latent prior is modeled as a mixture of Gaussians because it can preserve local data-distribution structure better than a single Gaussian.The model must also support differentiable KL calculations and private estimation.
  • Decoding Phase: The latent-mixture parameters are estimated with EM, whose maximum-likelihood objective is made differentially private by adding Gaussian noise.The KL term is approximated analytically for optimization.

7 Compute gradient

The dimensionality-reduction objective is approximated over the data and optimized with DP-PCA when the reduction is linear.

  • 7 Compute gradient: The objective for dimensionality reduction is approximated by averaging over all given data.This converts the reduction problem into a form suitable for private optimization.
  • 7 Compute gradient: When f is linear, DP-PCA optimizes the reduction objective while satisfying differential privacy.PCA is selected as the private dimensionality-reduction method.
  • 7 Compute gradient: During decoding, P3GM optimizes its parameters with DP-SGD using the phased-training algorithm.The pseudocode packs encoder and decoder parameters together for simplicity.

E. Data Synthesis using P3GM

P3GM synthesizes data by sampling privately estimated latent vectors and decoding them, while accounting for privacy across PCA, EM, and SGD.

  • E. Data Synthesis using P3GM: P3GM generates synthetic data by sampling z from the fitted MoG and decoding it through pθ(x|z).Because MoG approximates the real-data distribution, generated samples can follow a similar mixture ratio.
  • E. Data Synthesis using P3GM: Releasing the trained model satisfies differential privacy, and sampled data remains private by DP postprocessing.The privacy guarantee applies to data generated with arbitrary random seeds.
  • E. Data Synthesis using P3GM: P3GM’s privacy accounting composes privacy consumed by PCA, EM, and SGD, using RDP for a rigorous bound.The theorem states that P3GM satisfies (ε, δ)-DP after converting the composed RDP guarantee.
  • E. Data Synthesis using P3GM: Fixing the encoder reduces the optimization search space, and the decoder learns with a stable representation rather than full end-to-end VAE training.When encoder variance is constant, the first loss term becomes constant and the model becomes equivalent to an autoencoder when sx = 0.

B. Parameter setting

The evaluation uses six datasets, multiple classifier-based utility measures, and a 2-way-marginal comparison to assess P3GM’s utility, efficiency, and privacy accounting.

  • B. Parameter setting: Experiments evaluate synthetic data through classification and 2-way marginals, while also measuring model-construction efficiency and privacy-composition savings.Classification uses AUROC, AUPRC, or accuracy; the marginal experiment uses average total variation distance.
  • B. Parameter setting: The study uses six real datasets, including highly dimensional UCI and image datasets, with 90% used for training and the remainder for testing.Kaggle Credit is highly imbalanced, while ISOLET and ESR have small sample sizes relative to their dimensions.
  • B. Parameter setting: P3GM uses encoder and decoder networks with reduced dimensionality; dimensionality reduction is omitted for Kaggle Credit and set to d_p = 10 with ε_p = 0.1 elsewhere.The encoder and decoder each use two fully connected layers surrounding the reduced representation.
  • B. Parameter setting: Comparisons include PrivBayes, DP-GM, Ryan’s algorithm, and DP-VAE, with classifiers drawn from standard machine-learning libraries.The classifier suite includes Logistic Regression, AdaBoost, Gradient Boosting, and XGBoost.

A. Effectiveness in Data Mining Tasks

Across tabular and image tasks, P3GM generally preserves utility under differential privacy and performs especially well on high-dimensional data, while its phased design improves convergence.

  • A. Effectiveness in Data Mining Tasks: Under (1, 10^-5)-DP, P3GM’s Kaggle Credit scores do not significantly decrease relative to non-private models, indicating tolerance to differential-privacy noise.Table V reports relatively close accuracy for PGM and P3GM compared with VAE.
  • A. Effectiveness in Data Mining Tasks: P3GM outperforms PrivBayes and DP-GM on three of four real datasets in AUROC and AUPRC, while high dimensionality and small sample size degrade ISOLET performance.PrivBayes slightly outperforms P3GM on Adult, but P3GM is significantly better on high-dimensional data.
  • A. Effectiveness in Data Mining Tasks: P3GM achieves much higher classification accuracy than DP-GM and PrivBayes on MNIST and Fashion-MNIST, remaining about 6% and 5% below VAE, respectively.The generated images are visually closer to VAE samples while satisfying differential privacy.
  • A. Effectiveness in Data Mining Tasks: Reducing MNIST to d_p = [10, 100] balances expressive power and dimensionality, whereas excessive or insufficient reduction harms performance.Too much dimensionality makes DP-EM ineffective, while too little dimensionality weakens the embedding.
  • A. Effectiveness in Data Mining Tasks: P3GM converges faster than DP-VAE because freezing the encoder eliminates part of the solution space; the restricted P3GM (AE) converges earliest, while full P3GM achieves the best final results.The smaller search space accelerates convergence, whereas the larger full-P3GM search space can find better solutions at greater privacy cost.

C. Accuracy of 2-way marginals

The paper evaluates 2-way-marginal fidelity alongside privacy-budget allocation and composition, showing that direct marginal release can excel while P3GM’s accounting reduces computed privacy cost.

  • C. Accuracy of 2-way marginals: P3GM consists of an Encoding Phase using DP-PCA and DP-EM and a Decoding Phase using DP-SGD.The experiments vary how the total privacy budget is allocated between these two components.
  • C. Accuracy of 2-way marginals: On Adult, the best empirical privacy-budget allocation ratio is between 0.1 and 0.38, although a theoretically optimal ratio remains open.The total privacy budget is held at 1 while the allocation ratio varies.
  • C. Accuracy of 2-way marginals: RDP composition yields a smaller computed ε than the zCDP and moments-accountant baselines as DP-SGD noise varies.The paper presents RDP composition as more rigorous accounting of the component privacy budgets.
  • C. Accuracy of 2-way marginals: P3GM’s conclusion combines a two-phase differentially private generative model with complexity analysis and experiments showing higher utility than existing techniques in many cases.The paper also reports less noisy samples and higher classification utility in several tasks.
Loading 2006.12101v4…