PaperlayerView exact revision on arXiv

Source-linked AI summary

Differentially Private Model Publishing for Deep Learning

Lei Yu, Ling Liu, Calton Pu, Mehmet Emre Gursoy, Stacey Truex

arXiv:1904.02200v5cs.CRcs.LG

TL;DR

The paper addresses privacy leakage from sensitive training data and the publishing of trained deep learning models. It combines CDP-based privacy accounting for distinct batching methods with dynamic privacy-budget allocation, and experiments report improved accuracy under the same privacy guarantee. Its practical discussion notes that DP parameters remain difficult for end-users to interpret.

  • Problem

    Sensitive datasets and published deep learning models can expose private information, while existing DP-SGD accounting may underestimate privacy loss when batching methods differ.

  • Method

    The paper uses CDP-based privacy accounting for random reshuffling and random sampling, together with dynamic privacy-budget allocation during DP-SGD training.

  • Results

    Across multiple datasets, dynamic privacy-budget schedules improve model accuracy over uniform allocation under the same privacy guarantee.

  • Takeaways & Limitations

    Dynamic allocation can reduce the accuracy gap between differentially private and non-private training while retaining the overall privacy guarantee.

  • Takeaways & Limitations

    The practical privacy benefits associated with DP parameters such as ϵ, δ, and ρ remain difficult for end-users and domain scientists to interpret.

Abstract

from arXiv · show

Deep learning techniques based on neural networks have shown significant success in a wide range of AI tasks. Large-scale training datasets are one of the critical factors for their success. However, when the training datasets are crowdsourced from individuals and contain sensitive information, the model parameters may encode private information and bear the risks of privacy leakage. The recent growing trend of the sharing and publishing of pre-trained models further aggravates such privacy risks. To tackle this problem, we propose a differentially private approach for training neural networks. Our approach includes several new techniques for optimizing both privacy loss and model accuracy. We employ a generalization of differential privacy called concentrated differential privacy(CDP), with both a formal and refined privacy loss analysis on two different data batching methods. We implement a dynamic privacy budget allocator over the course of training to improve model accuracy. Extensive experiments demonstrate that our approach effectively improves privacy loss accounting, training efficiency and model quality under a given privacy budget.

I. INTRODUCTION

Deep learning models trained on large, crowdsourced datasets can expose sensitive information, especially as trained models are increasingly shared and published. The paper addresses these risks through differentially private training, privacy accounting, and privacy-budget allocation.

  • Privacy motivation: Membership and model-inversion attacks reveal privacy risks in deep learning systems trained on datasets containing individuals’ sensitive information.
  • Privacy motivation: Publishing trained models through cloud APIs, on-device applications, and transfer learning expands access to models that may encode private information.
  • Differentially private training: DP-SGD clips per-example gradients and adds Gaussian noise so each update, and therefore the final model, receives differential privacy protection.
  • Privacy accounting: Privacy accounting must distinguish random reshuffling from random sampling because treating them as equivalent can underestimate privacy loss.
  • Paper approach: The paper uses concentrated differential privacy and dynamic privacy-budget allocation to improve cumulative privacy accounting and model accuracy during iterative training.

C. Concentrated Differential Privacy

Concentrated differential privacy characterizes privacy loss probabilistically to obtain sharper guarantees for repeated computations. The paper uses zCDP as a form that composes conveniently and converts to traditional (ϵ, δ)-DP guarantees.

  • CDP concept: CDP targets privacy-preserving algorithms with many computations by concentrating cumulative privacy loss and providing sharper analysis than traditional composition.
  • CDP concept: Traditional (ϵ, δ)-DP bounds privacy loss by ϵ except with probability at most δ, whereas CDP bounds its mean and tail behavior.
  • zCDP: zCDP concentrates privacy loss around zero and models it as sub-Gaussian, making larger deviations increasingly unlikely.
  • zCDP: A mechanism satisfying ρ-zCDP also provides an (ϵ, δ)-DP guarantee, allowing zCDP results to be expressed in the traditional privacy framework.
  • zCDP: The paper chooses zCDP because it is comparable to (ϵ, δ)-DP and immune to post-processing.

D. Composition

The paper analyzes cumulative privacy loss for iterative DP-SGD under different batching schemes and uses composition properties to track a fixed privacy budget. It then allocates that budget dynamically to improve accuracy relative to uniform allocation.

  • Composition: zCDP composes linearly, so T iterations each satisfying ρ-zCDP yield a training process satisfying Tρ-zCDP.
  • Privacy accounting: CDP provides tighter cumulative privacy accounting for many DP-SGD iterations, potentially permitting lower noise or more training iterations under a fixed privacy budget.
  • DP-SGD: DP-SGD clips gradients, adds Gaussian noise, and composes the per-step privacy costs until the total budget is exhausted.
  • Batching methods: Privacy accounting differs for random sampling with replacement and random reshuffling because these batching methods induce different privacy-loss behavior.
  • Dynamic allocation: Dynamic allocation varies the noise scale across training steps while preserving the same overall privacy guarantee, unlike uniform allocation.

IV. DETAILS OF OUR APPROACH

The approach combines dynamic privacy-budget allocation with zCDP-based accounting tailored to the data-batching method. Its goal is to produce a differentially private neural network with better accuracy under a fixed total budget.

  • IV. DETAILS OF OUR APPROACH: The approach first develops dynamic privacy-budget allocation techniques and then derives zCDP privacy accounting for different batching methods.
  • Budget allocation: The budget assigned to each epoch determines the Gaussian noise scale used by iterations within that epoch.
  • Budget allocation: The allocation is optimized over training epochs to obtain a differentially private DNN model with better accuracy.
  • Budget allocation: As accuracy converges, reducing gradient noise is intended to move learning closer to a local optimum and improve accuracy.

1) Adaptive schedule based on public validation dataset:

The paper adjusts DP-SGD noise dynamically using public validation accuracy when available, and also provides predefined epoch-based decay schedules when validation data is unavailable.

  • Adaptive schedule based on public validation dataset:: Validation accuracy periodically determines whether the noise scale should be reduced for subsequent training epochs.The epochs used for these checks are called validation epochs.
  • Adaptive schedule based on public validation dataset:: If validation-accuracy improvement is below threshold δ, the noise scale decays by factor k, where 0 < k < 1.Because validation accuracy may fluctuate, the method uses a moving average to make the adjustment more effective.
  • Adaptive schedule based on public validation dataset:: The schedule compares averaged validation accuracy at periodic checks and reduces noise when the current average fails to improve sufficiently over the previous check.The checking period must be at least m validation epochs.
  • Pre-defined schedules:: Without a public validation dataset, predefined decay functions reduce the noise scale by epoch without accessing data or checking model accuracy.The noise scale remains constant within each epoch and is updated over time according to the selected decay function.
  • Pre-defined schedules:: The four predefined schedules are time-based, exponential, step, and polynomial decay, chosen as simple representative schedules also used for DNN learning-rate tuning.Users can apply these schedules during only part of training, and polynomial decay holds the final noise scale after its specified period.

3) Privacy Preserving Parameter Selection:

This section refines privacy accounting for DP-SGD by distinguishing random reshuffling from random sampling with replacement and using CDP or bounded Rényi divergence accordingly. It also explains how these analyses support privacy-preserving parameter selection.

  • Batching methods: Random reshuffling and random sampling with replacement require different privacy accounting because their data-access patterns have distinct privacy guarantees.Random reshuffling partitions a shuffled dataset into disjoint batches, whereas sampling with replacement independently samples each example.
  • Under random reshuffling: Theorem 2 gives a tighter privacy characterization for adaptive mechanisms operating on disjoint data partitions.Its equal-ρ special case yields ρ-zCDP rather than the (kρ)-zCDP guarantee from naive composition.
  • Under random reshuffling: The revised composition statement replaces an average-based expression with a maximum unless all per-mechanism ρ_i values are equal.The paper’s algorithm and experiments use the same noise scale within each epoch, so their analysis remains correct.
  • Under random sampling with replacement: Under random sampling, CDP does not capture privacy amplification because sampled-mechanism Rényi divergence lacks the required all-orders bound.At q=0.01 and σ=4, the sampled divergence is near zero only before α=147, then increases at the un sampled mechanism’s rate.
  • Under random sampling with replacement: The paper bounds α-Rényi divergence over a limited α range and converts the bound to (ϵ, δ)-DP for sampled Gaussian mechanisms.Theorem 3 parameterizes the range and bound using Uα(q, σ) and P(q, σ).
  • Under random sampling with replacement: The empirical choice P(q, σ)=q^2/σ^2 provides a quick privacy-loss estimate for selecting noise scale, sampling ratio, and training steps.Numerical comparisons found the bound αq^2/σ^2 to hold across a wide range of tested q and σ settings, while the MA method remains recommended for provable general guarantees.

C. DP Composition Under Dynamic Schedules

Dynamic schedules update the Gaussian noise scale across epochs while tracking cumulative zCDP privacy loss against a fixed budget. Pre-defined schedules avoid additional privacy cost for parameter updates, while validation-based schedules use public data and adaptive composition.

  • Pre-defined schedules: Each epoch uses a noise scale σ_t, with per-iteration cost ρ=1/(2σ_t^2) under zCDP.The epoch privacy cost is computed using the composition theorem appropriate to the batching method.
  • Pre-defined schedules: Training terminates when cumulative privacy loss exceeds the fixed total budget ρ_total.Schedule hyperparameters can be selected before training to target a specified training time under the budget.
  • Validation-based schedules: Validation-based schedules can adapt ρ at every epoch using a public validation dataset without incurring additional privacy cost.zCDP composition remains valid for these adaptive privacy-parameter choices because privacy loss is defined through Rényi divergence.

V. EXPERIMENTAL RESULTS

The experiments compare privacy accounting methods for random reshuffling and random sampling, tracking cumulative privacy loss across training epochs and varying sampling ratio and noise scale. The results show distinct privacy-loss estimates across accounting methods and batching choices.

  • A. Comparing Privacy Accounting Approaches: The evaluation compares zCDP(RF), zCDP(RS), strong composition, and the Moments Accountant under unified (ϵ, δ)-DP accounting.Random reshuffling and random sampling use different accounting methods, while strong composition and MA assume random sampling with replacement.
  • A. Comparing Privacy Accounting Approaches: At q = 0.01 and σ = 6, cumulative privacy loss is tracked by epoch with fixed δ = 1e −5.The Gaussian mechanism at σ = 6 satisfies (ϵ = 0.808, δ = 1e −5)-DP and ρ = 0.0139-zCDP.
  • A. Comparing Privacy Accounting Approaches: zCDP(RF) estimates lower privacy loss than strong composition during training.The comparison is shown in Figure 2 as privacy parameter ϵ versus epoch.
  • A. Comparing Privacy Accounting Approaches: zCDP(RF) has higher privacy loss than MA and zCDP(RS), while MA underestimates the real privacy loss when random reshuffling is treated as random sampling with replacement.The paper identifies random reshuffling as common practice in deep learning and describes zCDP(RF) as the proper choice for that setting.
  • A. Comparing Privacy Accounting Approaches: For zCDP(RF), cumulative privacy loss does not change with sampling ratio q when the training duration is fixed at 200 epochs.The paper attributes this to per-epoch and across-epoch composition properties of ρ-zCDP.

B. Evaluating Dynamic Privacy Budget Allocation

The experiments evaluate dynamic privacy-budget schedules against uniform allocation under fixed privacy budgets, showing improved accuracy and training efficiency across settings. Results also reveal a trade-off: overly slow or rapid noise decay harms accuracy, while schedule hyperparameters require tuning.

  • Fixed-budget evaluation: Dynamic schedules improve testing and training accuracy by 1%–1.6% over uniform allocation while running fewer epochs.The baseline uses constant σc=8 and ρtotal=0.78125, yielding 100 training epochs.
  • Fixed training time: All pre-defined schedules achieve higher accuracy than uniform allocation at fixed training times, but no decay function consistently wins.Accuracy rises through roughly 30–50 or 60 epochs before declining toward 100 epochs.
  • Decay-rate effects: Exponential decay reaches maximum accuracy at k=0.2, whereas validation-based decay peaks at k=0.7.Increasing k shortens training for exponential decay but generally lengthens it for validation-based decay, with fluctuations from validation-driven adjustments.
  • Decay-rate effects: The lowest and highest decay rates both reduce accuracy by respectively maintaining high noise too long or terminating training too early.The results indicate an intermediate decay rate is needed to balance perturbation reduction against sufficient training time.
  • Comparison with non-private SGD: Dynamic schedules reduce the uniform-allocation gap to non-private SGD by 20%–30% when the original accuracy difference is 0.05.The paper treats this gap as the maximum potential improvement available through dynamic allocation over uniform allocation.

3) Results on other datasets:

Experiments on Cancer and CIFAR-10 compare dynamic privacy-budget schedules with uniform allocation. Dynamic schedules improve accuracy and narrow the gap to non-private SGD under the evaluated privacy budget.

  • 3) Results on other datasets:: Dynamic schedules are evaluated against uniform privacy allocation on Cancer and CIFAR-10 under a fixed total privacy budget.The experiments test exponential and validation-based decay schedules against the uniform allocation method.
  • 3) Results on other datasets:: 2% higher accuracy and a 12% smaller gap are achieved by exponential decay on CIFAR-10 versus uniform allocation.These results are reported for the testing and training accuracy experiments shown in Figures 14 and 15.
  • 3) Results on other datasets:: 4% higher accuracy and a 19% smaller gap are achieved by validation-based decay on CIFAR-10 versus uniform allocation.The validation-based schedule benefits from longer training time in the reported comparison.
  • VI. DISCUSSION: The paper identifies privacy-parameter interpretability, data dependency, and inference risks as practical caveats for deploying private models.The discussion states that DP parameters lack understandable end-user interpretations and that data dependencies can affect privacy protection.

APPENDIX

The appendix develops privacy accounting for mechanisms applied across randomized partitions and adaptive sequences. It derives zCDP guarantees by relating the combined privacy loss to the component mechanisms’ guarantees.

  • APPENDIX: Theorem 2 considers a sequence of k adaptive mechanisms whose individual zCDP guarantees contribute to the privacy analysis.The proof represents the combined privacy loss through conditional outcome probabilities and their composition.
  • APPENDIX: Randomly partitioning the input domain makes the partition containing the extra neighboring record uniformly distributed among k partitions.The resulting privacy loss is described as a mixture of independent random variables.
  • APPENDIX: When component mechanisms share privacy parameter ρ, the composed mechanism satisfies ρ-zCDP.The appendix states this result after bounding the relevant moment of the privacy-loss random variable.
  • APPENDIX: When component privacy parameters differ, the mechanism satisfies maxi ρi-zCDP.The analysis replaces each ρj with the maximum component privacy parameter.

B. Proof of Theorem 3

Theorem 3 converts a Rényi-divergence bound for a mechanism into a zCDP-related privacy statement using moment bounds and optimization over α.

  • B. Proof of Theorem 3: Theorem 3 assumes Dα(A′(D)||A′(D′)) ≤ αbρ for 1 < α ≤ uα.The bound is the starting condition for the subsequent privacy analysis.
  • B. Proof of Theorem 3: The privacy-loss moment is bounded by exp((α − 1)αbρ), which enables a tail-probability bound through Markov’s inequality.The proof applies this bound to the privacy-loss random variable.
  • B. Proof of Theorem 3: The unconstrained minimizer is α∗ = (ϵ + bρ)/(2bρ), with minimum value −(ϵ − bρ)^2/(4bρ).If α∗ ≤ uα, the unconstrained minimum is also the constrained minimum.
  • B. Proof of Theorem 3: Setting δ = exp ... yields an ϵ expression with the same form as Proposition 1.The proof identifies the corresponding constrained expression when the optimum is limited by uα.

C. ρ-zCDP mechanism A with random sampling still satisfies ρ-zCDP

The appendix shows that applying a ρ-zCDP mechanism to a random subsample preserves the ρ-zCDP guarantee. The proof uses Rényi-divergence bounds and joint quasi-convexity.

  • C. ρ-zCDP mechanism A with random sampling still satisfies ρ-zCDP: A ρ-zCDP mechanism A applied to a random subsample produces mechanism A′ for which the privacy guarantee is analyzed under subsampling.The proof begins by relating neighboring-sample outcomes to the original mechanism.
  • C. ρ-zCDP mechanism A with random sampling still satisfies ρ-zCDP: Rényi divergence between the conditional distributions is bounded by αρ.This follows from the assumed ρ-zCDP property of A.
  • C. ρ-zCDP mechanism A with random sampling still satisfies ρ-zCDP: Joint quasi-convexity of Rényi divergence bounds the subsampled mixture by the same αρ quantity.The resulting mixture inequality is used to establish the preserved privacy guarantee.
  • C. ρ-zCDP mechanism A with random sampling still satisfies ρ-zCDP: The randomly subsampled mechanism A′ still satisfies ρ-zCDP.This is the appendix’s stated conclusion after applying the divergence bound to the mixture.

D. The Impact of number of hidden layers for MNIST

The section validates an empirical bound for the α-R´enyi divergence of the sampled Gaussian mechanism across tested noise and sampling parameters. It also records implementation and schedule-related materials accompanying the validation.

  • Validation of α-R´enyi divergence: The validation computes α-R´enyi divergence using TensorFlow’s moments accountant implementation, identified as RDP analysis.The corresponding figure compares the bound with the numerically computed divergence.
  • Validation of α-R´enyi divergence: The empirical bound q2α/σ2 remained higher than the measured α-R´enyi divergence throughout the tested parameter settings.Tests varied σ from 1 to 30 and q from 0.001 to 1/(16σ), with α sampled up to Uα.
  • Validation of α-R´enyi divergence: Additional tests with smaller step sizes and σ values up to 50 also found that q2α/σ2 held under the stated conditions.The authors present this as a conjectured valid bound and defer formal proof to future work.
Loading 1904.02200v5…