PaperlayerView exact revision on arXiv

Source-linked AI summary

Differentially Private Mixture of Generative Neural Networks

Gergely Acs, Luca Melis, Claude Castelluccia, Emiliano De Cristofaro

arXiv:1709.04514v4cs.LGcs.CR

TL;DR

The paper addresses privacy risks that make releasing generative models and high-dimensional datasets unsafe. It introduces a differentially private mixture of cluster-specific neural generators with adaptive private training, and reports realistic synthetic data and accurate counting queries across several datasets.

  • Problem

    Existing generative models may expose sensitive information, while releasing high-dimensional datasets remains difficult despite the need to share them for analysis.

    Source: §1 · p. 1

  • Method

    DPGM privately clusters data with kernel k-means, trains separate RBMs or VAEs on the clusters, and uses adaptive differentially private gradient descent.

    Source: §1 · pp. 1–2

  • Results

    The approach produces realistic synthetic samples and supports accurate arbitrary counting queries on MNIST, call detail records, and transit data.

    Source: §1 · p. 2, §6.3 · p. 12

  • Takeaways & Limitations

    The proposed DPGM enables sharing synthetic high-dimensional data with provable privacy and high utility within the evaluated datasets.

    Source: §1 · p. 2, §7 · p. 12

    Synthetic-sample quality is not guaranteed to be identical across generators because clusters may be dissimilar or too small to converge.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Generative models are used in a wide range of applications building on large amounts of contextually rich information. Due to possible privacy violations of the individuals whose data is used to train these models, however, publishing or sharing generative models is not always viable. In this paper, we present a novel technique for privately releasing generative models and entire high-dimensional datasets produced by these models. We model the generator distribution of the training data with a mixture of $k$ generative neural networks. These are trained together and collectively learn the generator distribution of a dataset. Data is divided into $k$ clusters, using a novel differentially private kernel $k$-means, then each cluster is given to separate generative neural networks, such as Restricted Boltzmann Machines or Variational Autoencoders, which are trained only on their own cluster using differentially private gradient descent. We evaluate our approach using the MNIST dataset, as well as call detail records and transit datasets, showing that it produces realistic synthetic samples, which can also be used to accurately compute arbitrary number of counting queries.

1 Introduction

The paper addresses privacy risks in releasing generative models and high-dimensional datasets by introducing a differentially private mixture of generative neural networks. It combines private clustering, cluster-specific generators, and adaptive private training, then evaluates synthetic data quality and query accuracy.

Motivation

  • Privacy risks arise because generative models can overfit or expose sensitive individual information, while traditional anonymization performs poorly on high-dimensional data.Model inversion attacks can reveal unintended sensitive attributes, and releasing internal parameters may still threaten individuals’ privacy.Source: §1 · p. 1

Overview of the solution

  • The proposed DPGM privately models a dataset with k generative neural networks trained on separate clusters.Data is first clustered privately, then each cluster is assigned to an RBM or VAE trained with differentially private gradient descent.Source: §1 · pp. 1–2

Contributions

  • The method introduces private kernel k-means with random Fourier features for scalable clustering of high-dimensional datasets.The clustering procedure combines nonlinear kernel representations with a differentially private version of Lloyd’s algorithm.Source: §1 · p. 2
  • Adaptive perturbation recomputes gradient noise for each SGD update, potentially improving the accuracy of differentially trained generative models.The privacy guarantee is quantified using the moments accountant for composed private clustering and SGD mechanisms.Source: §1 · p. 2

Evaluation

  • The approach produces realistic synthetic samples and supports accurate arbitrary counting queries on MNIST, call detail records, and transit data.The evaluation covers both image data and real-world mobility or communication datasets.Source: §1 · p. 2

2 Related work

Prior work provides private data release and private learning mechanisms, but this paper combines differential privacy with neural generative models, private clustering, and improved noisy optimization. Its approach trains privacy into the generative pipeline rather than filtering generated records afterward.

Private Data Release

  • Earlier private-release methods often support only succinct representations such as histograms or contingency tables, while k-anonymity performs poorly on high-dimensional data.These limitations motivate generating plausible synthetic records under strong privacy guarantees.Source: §2 · p. 2
  • This work trains neural generative models under differential privacy instead of generating records first and rejecting samples through later privacy tests.The paper frames this as avoiding potentially high sample rejection rates from post-generation privacy testing.Source: §2 · p. 2

Privacy in Deep Learning

  • Unlike privacy-preserving methods focused mainly on convex optimization, the paper develops private SGD for general non-convex losses used by RBMs and VAEs.The related work contrasts the proposed optimizer with methods that add noise only to final outputs in convex settings.Source: §2 · pp. 2–3
  • The method extends moments-accountant analysis to unsupervised generative modeling with per-iteration adaptive noise, alongside private clustering and multiple generators.This distinguishes it from distributed private training and prior private GAN work described in the paper.Source: §2 · p. 3

Clustering and Generative Neural Networks

  • Its private kernel k-means can represent nonlinearly separable clusters, while its noise is described as independent of data dimension and point norm.Random Fourier features make the kernel computation scalable and support the private clustering construction.Source: §2 · p. 3
  • Unlike earlier clustering–deep-learning proposals without privacy guarantees, DPGM privately clusters data and trains multiple generators for higher-quality synthetic samples.Prior approaches often jointly trained or pretrained autoencoders with clustering representations.Source: §2 · p. 3

3 Preliminaries

The preliminaries define the generative models, clustering machinery, and differential-privacy tools used by the paper. RBMs and VAEs model data distributions, random features make kernel clustering scalable, and moments accounting analyzes composed Gaussian mechanisms.

3 Preliminaries

  • A record is represented as a binary vector over items, and a dataset is an ensemble of such records.The notation uses a universe of items I with |I| = m and records as nonempty subsets of I.Source: §3 · p. 3

3.1 Restricted Boltzmann Machines (RBM)

  • An RBM uses visible variables for data attributes and hidden variables to capture dependencies, learning parameters that approximate the data-generating distribution.Its Boltzmann energy uses weights and bias terms, and training maximizes likelihood through iterative gradient updates.Source: §3.1 · p. 3

3.2 Variational Autoencoder (VAE)

  • A VAE encodes observations into a latent space and decodes latent representations to reconstruct data while optimizing a variational log-likelihood lower bound.The reparameterization trick enables stochastic-gradient training through the sampling process.Source: §3.2 · pp. 3–4

3.3 Kernel k-means with Random Features

  • Kernel k-means maps data nonlinearly into feature space so that linearly separable clusters there correspond to nonlinear clusters in the original input space.This addresses inaccurate clustering of linearly non-separable data.Source: §3.3 · p. 4
  • Random Fourier features approximate kernel inner products in a low-dimensional Euclidean space, making kernel k-means scalable for large datasets.The approximation error decreases rapidly as the feature dimension d increases.Source: §3.3 · p. 4

3.4 Differential Privacy (DP)

  • Differential privacy bounds how much an algorithm’s output can reveal about any individual record, using sensitivity to calibrate perturbation.The Gaussian mechanism adds noise proportional to a function’s L2-sensitivity.Source: §3.4 · pp. 4–5
  • The moments accountant gives a tighter privacy bound for adaptive compositions of mechanisms such as repeated noisy clustering or SGD.It aggregates privacy-loss moments across independently randomized mechanisms to derive an (ε, δ)-guarantee.Source: §3.4 · pp. 4–5

4 DPGM: Differentially Private Generative Model

DPGM partitions data with private kernel k-means, trains one differentially private generative model per cluster, and releases their parameters for synthetic-data generation.

4 DPGM: Differentially Private Generative Model

  • DPGM partitions D into k clusters, trains distinct generative models on those partitions, and releases their perturbed parameters for synthetic-data generation.The models can be RBMs or VAEs and are trained with differentially private gradient descent.Source: §4 · p. 5

4.1 Private kernel k-means

  • Records are mapped into low-dimensional random Fourier features, clipped using a private norm estimate, and clustered with differentially private k-means.The resulting noisy cluster centers define the partitions used for model training.Source: §4.1 · pp. 5–6
  • The private clustering procedure uses noisy cluster sizes and noisy sums of Fourier features to compute noisy cluster centers.Cluster initialization uses k representative samples from publicly available non-sensitive data.Source: §4.1 · p. 6
  • RBF-kernel private k-means supports linearly non-separable clusters while making feature-vector noise independent of data-record norm and feature dimension.The paper reports that this can improve clustering accuracy for stringent privacy requirements, especially ε < 0.5.Source: §4.1 · pp. 6–7

7 Descent: θ ←θ −η˜g Output: θ

The training pipeline clips per-record gradients, adds Gaussian noise, adaptively selects clipping bounds, and samples models according to cluster size during synthetic-data generation.

4.2 Private Stochastic Gradient Descent

  • Private SGD clips each record’s gradient and adds Gaussian noise before updating the selected model’s parameters.Each iteration computes gradients on a random batch, clips them to Cs, perturbs their average, and performs the descent step.Source: §4.2 · p. 7

4.3 Adaptive selection of the norm bound

  • Adaptive norm selection sets Cs from privately estimated batch gradient norms, avoiding slow convergence from overly small bounds and excess noise from overly large ones.The method discretizes candidate bounds and selects a noisy histogram mode.Source: §4.3 · p. 8
  • The noisy histogram’s L2-sensitivity remains bounded by 2 regardless of the largest gradient norm.This avoids relying directly on a potentially large outlier-dependent maximum norm.Source: §4.3 · p. 8

4.4 Synthetic data generation

  • Synthetic records are generated by sampling model i with probability 1/|D̂_i| and then drawing a sample from that model until the target dataset size is reached.Models with smaller clusters are consequently less likely to be selected for generation.Source: §4.4 · p. 8
  • Model quality need not be uniform because clusters may contain dissimilar samples or be too small for an assigned model to converge.Thus, random training and cluster composition can produce models with different synthetic-sample quality.Source: §4.4 · p. 8

5 Privacy Analysis

The privacy analysis composes private kernel k-means with private SGD using moments-accountant bounds, yielding an explicit overall (ε, δ)-DP guarantee for DPGM.

5 Privacy Analysis

  • Private kernel k-means is composed from private norm selection and repeated noisy clustering iterations, with each iteration perturbing cluster sizes and Fourier-feature sums.Changing one record can affect at most two cluster sizes.Source: §5 · p. 8
  • Independent training of the k models gives αSk(λ) ≤ αS1(λ), where S1 represents training a single model over the whole dataset.Each record is sampled in an SGD iteration with probability q = L/|D|.Source: §5 · p. 8
  • Within each SGD iteration, adaptive norm selection and gradient perturbation use the same sampled batch, so the analysis handles dependent source randomness.Across different iterations, the mechanisms use independent randomness.Source: §5 · pp. 8–9
  • The general moments accountant bounds adaptive composition and converts the resulting log-moment bound into an (ε, δ)-DP guarantee.The theorem permits dependent mechanisms through weighted moment bounds.Source: §5 · p. 9
  • Theorem 4 gives DPGM an (minλ (αK(λ) + αSk(λ) − log δ) / λ, δ)-differential privacy guarantee.Here αK and αSk account for private clustering and private SGD, respectively.Source: §5 · p. 9

6 Experimental Evaluation

The evaluation measures privacy loss, clustering accuracy, synthetic-sample quality, and counting-query utility on MNIST, CDR, and TRANSIT datasets. DPGM produces realistic samples and low counting-query error while its privacy and clustering behavior vary with training and clustering parameters.

6. Experimental Evaluation

  • DPGM is evaluated on MNIST, anonymized CDR, and TRANSIT datasets, with experiments covering privacy guarantees, clustering, synthetic samples, and counting-query accuracy.MNIST contains 60,000 binarized 28 × 28 images; CDR contains roughly 4.4 million customers, and TRANSIT contains 1.2 million passengers across 342 stations.Source: §6 · p. 9, §6.1 · p. 10, Table 2 · p. 10

6.2 Results with Image Dataset

  • Larger sampling probabilities and more SGD epochs increase ε, whereas larger σK and σG provide stronger privacy guarantees.The privacy loss is computed from σC, σK, σG, q, the number of k-means iterations, and the number of SGD iterations.Source: §6.2 · p. 10, Figure 2 · p. 10
  • Private kernel k-means exceeds standard DP k-means by as much as 20% clustering accuracy, especially at smaller ε.Shorter random Fourier features perform better at smaller ε, while longer features can perform better at larger ε because perturbation and approximation errors trade off.Source: Figure 3 · p. 10, §6.2 · p. 11
  • At ε = 1.74 after 20 epochs, VAE generates better-looking MNIST samples than RBM, while VAE without private clustering has bad visual quality.The experiment uses k = 10 clusters and TK = 20 clustering iterations.Source: §6.2 · p. 11, Figure 4 · p. 11

6.3 Results with CDR and transit dataset

  • For CDR queries at ε = 1.0, DPGM’s relative error ranges from 0.017 to 0.0012 across query lengths, compared with 0.11 to 0.05 for MWEM at ε = 2.DPGM also supports arbitrary numbers and types of queries through its synthetic data, rather than only the evaluated linear counting queries.Source: §6.3 · pp. 11–12, Figure 5 · p. 12
  • On TRANSIT, DPGM’s average relative error ranges from 0.09 to 0.02 across cluster counts, and the number of clusters does not affect the error.The authors note that the lack of cluster-count effect might be an artifact of the dataset itself.Source: §6.3 · p. 12, Figure 6 · p. 12

7 Conclusion

The paper introduces a differentially private generative model built from a mixture of neural networks for releasing synthetic high-dimensional data. Experiments show accurate dataset representation with strong privacy guarantees and high utility.

7 Conclusion

  • The DPGM combines k generative neural networks to generate and share synthetic high-dimensional data with provable privacy.The approach is presented as a first-of-its-kind attempt to build private neural-network generative models.Source: §7 · p. 12
  • Experiments on real datasets show accurate representation together with strong privacy guarantees and high utility.Source: §7 · p. 12
  • Future work includes privacy-preserving VAE–Gaussian-mixture clustering, deeper private networks for complex data, and deployment in real-world settings.Source: §7 · p. 12

A Multi-layer Variational Autoencoder

The paper evaluates a multi-layer VAE using synthetic MNIST, CDR, and transit results across cluster counts. Increasing VAE depth and capacity does not improve performance.

A Multi-layer Variational Autoencoder

Related papers

Loading 1709.04514v4…