PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning with Differential Privacy

Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, Li Zhang

arXiv:1607.00133v2stat.MLcs.CRcs.LG

TL;DR

Training neural networks on crowdsourced datasets can involve sensitive information, creating a need for rigorous privacy guarantees. This paper develops differential-private learning techniques and privacy-loss analysis, achieving 97% MNIST and 73% CIFAR-10 training accuracy under (8, 10^-5)-differential privacy.

  • Problem

    Training neural networks on large, representative datasets containing sensitive information requires techniques that provide rigorous privacy guarantees.

  • Method

    The paper combines differentially private stochastic gradient descent with new training techniques and a moments-accountant analysis of composed privacy loss.

  • Results

    97% MNIST training accuracy and 73% CIFAR-10 accuracy were achieved under (8, 10^-5)-differential privacy.

  • Takeaways & Limitations

    The moments accountant enables tight automated analysis of complex composite mechanisms beyond the reach of advanced composition theorems.

  • Takeaways & Limitations

    CIFAR-10’s private-model accuracy drop was about 7%, compared with about 1.3% on MNIST, leaving this gap for future work.

Abstract

from arXiv · show

Machine learning techniques based on neural networks are achieving remarkable results in a wide variety of domains. Often, the training of models requires large, representative datasets, which may be crowdsourced and contain sensitive information. The models should not expose private information in these datasets. Addressing this goal, we develop new algorithmic techniques for learning and a refined analysis of privacy costs within the framework of differential privacy. Our implementation and experiments demonstrate that we can train deep neural networks with non-convex objectives, under a modest privacy budget, and at a manageable cost in software complexity, training efficiency, and model quality.

1. INTRODUCTION

The paper develops differential-privacy techniques for training deep neural networks on potentially sensitive, crowdsourced data. It combines tighter privacy-loss analysis and computational improvements, then evaluates the approach on MNIST and CIFAR-10 under a modest privacy budget.

  • Motivation: Large, representative datasets enable neural-network advances but may be crowdsourced and contain sensitive information, motivating privacy-preserving training techniques.The introduction cites image classification, language representation, and Go move selection as application successes enabled partly by such datasets.
  • Contributions: The paper trains multilayer neural networks with tens of thousands to millions of parameters and non-convex objectives within a modest, single-digit privacy budget.This contrasts with prior work focused either on smaller convex models or complex neural networks with large privacy loss.
  • Contributions: Tracking higher moments of privacy loss yields tighter estimates of overall privacy loss both asymptotically and empirically.This is presented as the first contribution of the paper.
  • Contributions: New techniques improve computational efficiency through per-example gradient computation, smaller batches that reduce memory footprint, and differentially private principal projection at the input layer.These methods target the practical costs of differentially private training.
  • Evaluation: The implementation builds on TensorFlow and evaluates private training on the MNIST and CIFAR-10 image-classification benchmarks.The introduction reports that privacy protection for deep neural networks can be achieved at a modest cost in software complexity, training efficiency, and model quality.

2. BACKGROUND

This section introduces differential privacy as a database-level privacy standard and reviews mechanisms, composition, and accounting for privacy loss. It also outlines deep neural networks, their loss-based training, and the non-convex optimization challenges addressed by mini-batch SGD.

  • Differential privacy: Differential privacy protects aggregate databases by comparing randomized outputs on adjacent datasets, which in the experiments differ by one image-label pair.The formal guarantee is expressed for every pair of adjacent inputs and every subset of outputs.
  • Differential privacy: Differential privacy supports modular mechanisms through composition, provides group privacy for correlated inputs, and remains robust to auxiliary information.These properties make privacy guarantees useful for applications involving complex or correlated datasets.
  • Privacy mechanisms: Additive-noise mechanisms approximate functions by calibrating noise to global sensitivity, while Gaussian mechanisms provide (ε, δ)-differential privacy under parameter conditions.The Gaussian mechanism adds normally distributed noise whose standard deviation is determined by the function sensitivity and noise parameter.
  • Privacy accounting: Repeated private operations accumulate privacy loss, tracked using basic or advanced composition theorems and a privacy accountant.The paper’s mechanism-design blueprint sequentially composes bounded-sensitivity functions, selects noise parameters, and analyzes privacy.
  • Deep learning: Deep neural networks compose affine transformations and nonlinearities into parameterized functions trained by minimizing average loss over examples.Training seeks parameters with acceptably small loss, although exact global minima are rarely expected in practice.
  • Deep learning: Because complex-network losses are typically non-convex, mini-batch stochastic gradient descent estimates gradients from random batches and updates parameters toward a local minimum.The paper bases its implementation on TensorFlow, which supports computation graphs and distributed execution.

3. OUR APPROACH

The approach trains neural networks with differentially private SGD, using gradient clipping, noisy updates, and privacy accounting based on the moments accountant. It also tunes training hyperparameters to balance privacy, accuracy, and performance.

  • Differentially private SGD: Differentially private SGD clips each per-example gradient, averages the clipped gradients over a random lot, adds noise, and updates the model using the noisy average.The algorithm outputs the final parameters and computes the overall privacy cost using a privacy accounting method.
  • Differentially private SGD: Larger lots provide unbiased gradient estimates with lower variance, while batch sizes may remain smaller than lot sizes to limit memory consumption.Training time is expressed in epochs, with each epoch consisting of N/L lots.
  • Moments accountant: The moments accountant improves strong composition by saving a log(1/δ) factor in ε and a Tq factor in δ when δ is small and T ≫ 1/q.These savings are expected to be significant because each example is examined multiple times.
  • Moments accountant: ε ≈ 1.26 with L = 0.01N, σ = 4, δ = 10−5, and T = 10000 using the moments accountant, versus ε ≈ 9.34 under strong composition.The paper presents this comparison as evidence that the moments accountant is beneficial in practice.
  • Moments accountant: The moments accountant composes log moments of the privacy-loss random variable, extending tighter accounting to Gaussian mechanisms with random sampling.Unlike direct composition of privacy-loss tail bounds, log moments compose linearly and yield tighter privacy estimates for Algorithm 1.
  • Hyperparameter tuning: Model accuracy is more sensitive to batch size and noise level than to neural-network structure, guiding hyperparameter tuning across privacy, accuracy, and performance.The experiments also find a small benefit from starting with a relatively large learning rate, and do not require decreasing it to a very small value.

4. IMPLEMENTATION

The implementation provides differentially private SGD in TensorFlow through gradient sanitization and privacy accounting, with additional support for private PCA and public-data convolutional layers. It tracks privacy loss using a moments accountant and enables interpretable (ε, δ) queries while fixing training iterations and privacy parameters in advance.

  • Core components: The differentially private SGD algorithms are implemented in TensorFlow, with source code released under an Apache 2.0 license.The code includes DPSGD_Optimizer and DPTrain, which repeatedly invokes the optimizer while bounding total privacy loss with an accountant.
  • Core components: The TensorFlow implementation centers on a sanitizer that clips per-example gradient norms and adds batch noise, plus a privacy_accountant that tracks cumulative privacy spending.Per-example gradients are computed with a dedicated TensorFlow operator, enabling sanitization with only a modest training slowdown even for larger batches.
  • Additional workflows: The code implements differentially private PCA by normalizing sampled examples, adding Gaussian noise to the covariance matrix, and projecting inputs onto private principal directions.The PCA incurs a privacy cost but can improve model quality and reduce training time, according to MNIST experiments.
  • Additional workflows: Because efficient per-example convolutional gradients remain challenging in TensorFlow, the implementation explores convolutional layers learned on public data.The workflow is motivated by the usefulness of convolutional layers and prior work suggesting that random convolutions may suffice.
  • Privacy accountant: The moments accountant additively accumulates log moments of privacy loss and uses numerical integration for the Gaussian mechanism to obtain more accurate accounting.The implementation evaluates α(λ) across λ values and finds that λ ≤32 suffices for the parameters of interest.
  • Privacy accountant: Privacy loss can be queried as (ε, δ) privacy during training, while the implementation fixes iteration counts and privacy parameters beforehand to avoid adaptive-parameter attacks.The framework distinguishes the need for privacy-accountant implementations to support either privacy-odometer or privacy-filter operation.

5. EXPERIMENTAL RESULTS

Experiments show that the moments accountant gives substantially tighter privacy-loss estimates and that differentially private neural networks achieve useful accuracy on MNIST and CIFAR-10. Accuracy depends strongly on privacy and training parameters, while private training reduces overfitting but imposes a larger accuracy cost on CIFAR-10.

  • Privacy accounting: At E = 100, privacy loss is 9.34 with strong composition versus 1.26 with the moments accountant; at E = 400, it is 24.22 versus 2.55.The experiment fixes q = 0.01, σ = 4, and δ = 10−5.
  • MNIST: The non-private MNIST baseline reaches 98.30% accuracy in about 100 epochs using a 60-dimensional PCA layer and 1,000 hidden units.The baseline uses a lot size of 600.
  • MNIST: 90%, 95%, and 97% test set accuracy are achieved at (0.5, 10−5), (2, 10−5), and (8, 10−5)-differential privacy, respectively.The differentially private model clips each layer’s gradient norm at 4 and varies the neural-network and PCA noise scales.
  • MNIST: PCA improves MNIST accuracy and training performance, reducing training time by almost 10× and outperforming random projection at about 92.5% accuracy.The best PCA dimension is 60, and not using PCA reduces accuracy by about 2%.
  • CIFAR-10: CIFAR-10 requires a lot size of 2,000 or more, while the private model’s accuracy drop is about 7% versus about 1.3% on MNIST.The non-private CIFAR-10 architecture reaches about 86% accuracy in 500 epochs, compared with about 96.5% for the state of the art.

6. RELATED WORK

Prior privacy-preserving learning work spans different privacy guarantees, learning algorithms, and model classes. This work differs by assuming centrally held data, focusing on leakage from model outputs, and training non-convex neural networks with differential privacy.

  • Research landscape: The literature is organized by model class, learning algorithm, and privacy guarantees.Privacy-preserving data mining and machine learning have been active research areas since the late 1990s.
  • Privacy guarantees: Secure function evaluation and multi-party computation minimize leakage during joint computation, whereas this work studies leakage from a centrally trained model’s output.The paper assumes data is held centrally rather than split across parties.
  • Privacy guarantees: k-anonymity protects data through attribute generalization and suppression, but its limitations make it poorly suited to deanonymizing high-dimensional, diverse datasets.This work instead keeps raw records intact and perturbs derived data.
  • Privacy guarantees: Differential privacy has been applied to machine-learning tasks using training mechanisms or target models different from those studied here.The moments accountant is closely related to Rényi differential privacy and supports theoretical and empirical privacy analysis.
  • Learning algorithm and model class: Prior learning methods commonly target convex optimization; Wu et al. achieved 83% accuracy on MNIST, while multi-layer neural-network training is non-convex and typically uses SGD with poorly understood guarantees.Other related systems address collaborative filtering, distributed deep learning, or autoencoders through differentially private mechanisms.

7. CONCLUSIONS

The paper demonstrates differentially private training of deep neural networks with modest privacy loss, achieving strong MNIST and CIFAR-10 accuracy. It also introduces the moments accountant for tightly tracking privacy loss and identifies broader network classes and larger datasets as directions for future work.

  • Contributions: 97% MNIST training accuracy and 73% CIFAR-10 accuracy are achieved with (8, 10−5)-differential privacy.The experiments train deep neural networks using a differentially private version of stochastic gradient descent implemented in TensorFlow.
  • Contributions: The moments accountant enables tight automated privacy-loss analysis for complex composite mechanisms beyond advanced composition theorems.The paper presents this privacy-loss tracking mechanism as a tool of independent interest.
  • Future work: Future work includes applying the techniques to other deep-network classes, such as LSTMs for language modeling, and improving accuracy using larger datasets.The authors specifically note that datasets larger than MNIST and CIFAR-10 may improve accuracy.

APPENDIX · A. PROOF OF THEOREM 2

The appendix restates and proves Theorem 2 using composition of moments and a moment-based tail-bound argument. The proof establishes a tail bound on privacy loss that is stronger than differential privacy for fixed ε and δ.

  • A. PROOF OF THEOREM 2: The appendix explicitly restates and proves Theorem 2.
  • A. PROOF OF THEOREM 2: It introduces composed mechanism and outcome sequences for neighboring databases d and d′.
  • A. PROOF OF THEOREM 2: The composition argument uses conditional probabilities of each mechanism’s outcome under the two neighboring databases.
  • A. PROOF OF THEOREM 2: The tail-bound proof uses the standard Markov’s inequality argument from proofs of measure concentration.
  • A. PROOF OF THEOREM 2: The proof demonstrates a tail bound on privacy loss, making it stronger than differential privacy for fixed ε and δ.

B. PROOF OF LEMMA 3

The section proves a moments bound for the Gaussian mechanism with random sampling by reducing the analysis to one dimension and controlling a binomial expansion. It then combines this bound with a tail bound to derive Theorem 1’s differential-privacy guarantee.

  • Proof reduction: The proof reduces neighboring datasets to a one-dimensional problem because the mechanisms differ only in the first coordinate.It fixes d′, adds dn, sets f(dn) = e1, and assumes the remaining sampled contributions are zero.
  • Moment bound: Under the assumptions on q, σ, and λ, the binomial-expansion terms decay geometrically for t > 3, so the expansion is dominated by t = 3.The dominant term is O(q3λ3/σ3), which establishes the claim.
  • Theorem 1 derivation: Theorem 1 combines Lemma 3’s moments bound with Theorem 2’s tail bound and optimizes over λ.The resulting privacy analysis bounds Algorithm 1’s log moment by α(λ) ≤ Tq2λ2/σ2 before selecting parameters.
  • Theorem 1 guarantee: For sampling probability q = L/N and T steps, Algorithm 1 is (ε, δ)-differentially private for any δ > 0 when ε < c1q2T and the specified parameters are chosen.The theorem states that constants c1 and c2 exist to support this parameter choice.

C. FROM DIFFERENTIAL PRIVACY TO MOMENTS BOUNDS

The section shows how differential privacy guarantees can be translated into moment bounds. Combined with Theorem 2, this yields a composition theorem, while direct moment bounds can be stronger for many mechanisms.

  • From Differential Privacy to Moments Bounds: Differential privacy can be translated into a moment bound for a mechanism.Lemma C.1 establishes this conversion for any λ > 0.
  • From Differential Privacy to Moments Bounds: The proof bounds the centered privacy-related random variable using its mean and a bounded range.It uses µ = E[Z] and bounds |Z − µ| by ε exp(ε), then applies convexity to exp(λx).
  • From Differential Privacy to Moments Bounds: Lemma C.1 and Theorem 2 provide a composition theorem for differentially private mechanisms.The construction is roughly equivalent to unrolling the proof of the strong composition theorem of [22].
  • From Differential Privacy to Moments Bounds: Directly bounding moments can give stronger guarantees than first establishing differential privacy and applying Lemma C.1.This advantage applies to many mechanisms of choice and motivates the moments accountant.

D. HYPERPARAMETER SEARCH

The paper accounts for hyperparameter-search privacy by randomly selecting among differentially private mechanisms and applying a theorem governing the resulting search cost. For a 95% target on 10,000 validation examples, the analysis permits up to 6,700 settings at privacy cost ε = 4, while experiments used no more than 100.

  • D. HYPERPARAMETER SEARCH: Hyperparameter search is analyzed by applying Theorem 10.2 from Gupta et al. to account for its privacy cost.The theorem is stated for an ε-differentially private mechanism with a query of sensitivity 1.
  • D. HYPERPARAMETER SEARCH: Randomly selecting one of K hyperparameter-specific private mechanisms satisfies the theorem’s conditions with p = 1/K when one setting reaches target accuracy Q.The query counts correctly labeled validation examples, and Q denotes the target accuracy threshold.
  • D. HYPERPARAMETER SEARCH: 6,700 parameter settings can be tried at privacy cost ε = 4 for the validation set under the stated 95% accuracy example.The calculation uses a validation set of size 10,000, ε′ = 0.5, and δ = 0.05, with at most 1% accuracy loss when p ≥ 1/6700.
  • D. HYPERPARAMETER SEARCH: The experiments tried no more than 100 hyperparameter settings, so the stated bound was easily satisfied.This compares the experimental search size with the 6,700-setting bound derived for the numerical example.
Loading 1607.00133v2…