PaperlayerView exact revision on arXiv

Source-linked AI summary

How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy

Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H. Brendan McMahan, Sergei Vassilvitskii, Steve Chien, Abhradeep Thakurta

arXiv:2303.00654v3cs.LGcs.CRstat.ML

TL;DR

Complex ML models make rigorous DP adoption difficult because practical guidance, privacy accounting, and architecture choices remain challenging. This paper provides a self-contained guide to DP ML and its reporting practices, concluding that careful choices about privacy definitions, threat models, tuning, and model components are needed for strong privacy-utility trade-offs.

  • Problem

    Applying differential privacy rigorously to complex ML models remains challenging because privacy-utility trade-offs, accounting assumptions, tuning, and architectural choices require practical guidance.

  • Method

    The paper synthesizes DP ML research into a self-contained guide covering privacy definitions, training methods, accounting, tuning, architecture, and reporting practices.

  • Results

    The paper concludes that achieving strong privacy guarantees and good performance requires explicitly reporting privacy assumptions and carefully examining or adjusting complex-model components.

  • Takeaways & Limitations

    Practitioners should carefully choose the privacy unit, neighboring-dataset definition, amplification assumptions, hyperparameter procedure, and model components when applying DP to ML.

  • Takeaways & Limitations

    Private training can substantially reduce utility, especially for small datasets relative to model capacity; the best known private ImageNet model achieves 32.4% accuracy at ε = 8 without extra data.

Abstract

from arXiv · show

ML models are ubiquitous in real world applications and are a constant focus of research. At the same time, the community has started to realize the importance of protecting the privacy of ML training data. Differential Privacy (DP) has become a gold standard for making formal statements about data anonymization. However, while some adoption of DP has happened in industry, attempts to apply DP to real world complex ML models are still few and far between. The adoption of DP is hindered by limited practical guidance of what DP protection entails, what privacy guarantees to aim for, and the difficulty of achieving good privacy-utility-computation trade-offs for ML models. Tricks for tuning and maximizing performance are scattered among papers or stored in the heads of practitioners. Furthermore, the literature seems to present conflicting evidence on how and whether to apply architectural adjustments and which components are "safe" to use with DP. This work is a self-contained guide that gives an in-depth overview of the field of DP ML and presents information about achieving the best possible DP ML model with rigorous privacy guarantees. Our target audience is both researchers and practitioners. Researchers interested in DP for ML will benefit from a clear overview of current advances and areas for improvement. We include theory-focused sections that highlight important topics such as privacy accounting and its assumptions, and convergence. For a practitioner, we provide a background in DP theory and a clear step-by-step guide for choosing an appropriate privacy definition and approach, implementing DP training, potentially updating the model architecture, and tuning hyperparameters. For both researchers and practitioners, consistently and fully reporting privacy guarantees is critical, and so we propose a set of specific best practices for stating guarantees.

1 Introduction

Differential privacy provides a formal framework for reasoning about training-data anonymization, but applying it rigorously to complex ML models remains challenging. This guide organizes theory, practical methods, and reporting practices for researchers and practitioners.

  • DP is widely used as a formal framework for reasoning about information leakage and data anonymization.
  • “Privacy” in ML can refer to empirical attack robustness or DP, although DP specifically addresses data anonymization rather than transparency, consent, or data minimization.
  • Applying DP rigorously to complex models is difficult because privacy-utility tradeoffs, tuning techniques, and required component adjustments are not consistently documented.
  • The guide provides a self-contained overview of DP ML, including privacy accounting, convergence, user-level algorithms, model-training methods, and quantified privacy reporting.
  • The guide targets researchers and practitioners, offering current research synthesis alongside practical guidance for selecting definitions, implementing training, tuning models, and reporting guarantees.
  • The guide assumes readers already know that DP is needed and have a reasonable background in ML and deep learning.

2 Differential Privacy: Definitions, Intuition and Properties

Differential privacy formalizes how much a mechanism’s output can change when one record changes, with privacy controlled by parameters and adjacency choices. Its composition and post-processing properties support modular DP procedures, while stronger privacy generally reduces utility.

  • A DP mechanism randomizes a dataset query, such as a statistic or neural-network weights, to protect information about individual records.
  • The unit of privacy—often one training example, but potentially one user—determines the meaning of an ML model’s DP guarantee.
  • ε controls privacy strength: smaller ε makes neighboring output distributions closer but typically lowers utility when other variables remain fixed.
  • Approximate (ε, δ)-DP relaxes pure ε-DP, with smaller δ providing stronger privacy guarantees and δ commonly chosen much smaller than 1/n.
  • Add-or-remove, zero-out, and replace-one adjacency definitions have different semantics, so ε values should only be compared under comparable adjacency and privacy units.
  • Composition and post-processing preserve differential privacy, enabling complex procedures to combine and transform outputs of multiple DP mechanisms.
  • 2.4 Basic DP Mechanisms*: The exponential mechanism samples outputs with probabilities favoring higher-scoring objects, while sensitivity controls how strongly the best object is favored.

3 DP-fying Basics: Settings and Methods

DP can be introduced at different points in an ML workflow, and the appropriate setting depends on the threat model and model-access boundary. The guide focuses on central DP training while comparing local, distributed, label-level, and prediction-level approaches.

  • Choosing a DP method requires identifying how the model will be accessed and which threat model the guarantee must address.
  • Threats may involve raw data, training datasets, individual updates, intermediate models, final parameters, or predictions.
  • Central DP trusts the training aggregator with raw data and protects the released model or subsequent post-processing of the DP output.
  • Local DP avoids fully trusting the aggregator but often incurs a substantially larger utility loss than central DP for the same problem.
  • Distributed DP uses secured aggregation to approach central-DP utility without exposing raw client data to the administering organization; its noise can scale as 1/√n across n clients.
  • The guide primarily focuses on DP training for full-model protection, while noting that training-time DP can provide better privacy-accuracy trade-offs than prediction-level protection in some settings.
  • Label-level DP protects only labels rather than features and is significantly easier than full protection, while clustering-based label perturbation can improve its privacy-utility trade-off over direct randomized response.

4 DP-Training: Protecting Full Training Data

DP training aims to provide full-model privacy guarantees, but applying it to complex non-convex models requires balancing privacy, utility, convergence, accounting assumptions, and computational cost. DP-SGD is the most practical rigorous approach for large deep networks, while its guarantees and performance depend strongly on training design and model setting.

  • DP-Training scope: DP training modifies model training to guarantee that the resulting model is not sufficiently different when one training example is included or removed.The section distinguishes full-model guarantees from protection against individual empirical attacks.
  • Convergence and limitations: For deep non-convex models, achieving strong privacy such as ε ≤1 is usually impossible without severe utility degradation, and convergence to optimal excess risk is unknown.Under assumptions such as symmetric or heavy-tailed gradient distributions, DP-SGD can converge to a stationary point, though stationary convergence may not predict practical deep-learning performance.
  • DP-SGD: DP-SGD clips per-example gradients, adds Gaussian noise to their average, and accumulates privacy loss across sampled training steps.Final ε depends on noise level, number of steps, and sampling ratio; utility depends heavily on hyperparameter choices.
  • DP-SGD: Gradient perturbation is currently the most practical route to rigorous privacy guarantees for non-convex large-scale deep neural networks.Well-tuned noisy gradient descent has also been reported to achieve better utility than output or loss perturbation for some strongly convex models.
  • Computation: Computing per-example gradients remains the slowest part of DP-SGD because common implementations sacrifice hardware parallelism and bulk data-transfer benefits.Many autodifferentiation libraries do not provide easy access to the per-example gradients needed for clipping.
  • Privacy accounting: Privacy accounting depends on the data-processing mechanism: sampling amplification has adjacency-specific conditions, and reporting stronger bounds without actual subsampling is inaccurate.When sampling cannot be guaranteed, approaches such as DP-FTRL may be preferable because they do not rely on amplification.

5 Practicalities of DP-Training

Practical DP training requires choosing a privacy unit and ε target, accounting for the actual data-processing pattern, and reporting guarantees completely. Utility depends strongly on model scale, privacy unit, training strategy, and hyperparameter choices.

  • Choosing the privacy unit: For sequence models, defining the privacy unit is challenging because the training data may be represented as one very long text sequence.The guide distinguishes units of privacy for text and sequence data from settings where an example is semantically well defined.
  • Choosing the privacy unit: The guide recommends user-level or example-level DP with add-or-remove or zero-out adjacency as the lowest appropriate protection tier.This recommendation applies when a single user or appropriate group contributes at most one example.
  • Choosing ε: ε ≤1 offers strong formal privacy but can cause large utility drops in large models, while ε ≤10 is presented as a practical target for many applications.For ε >10, the DP guarantee alone may be insufficient evidence of anonymization and may need supplementary measures such as auditing or preprocessing.
  • Utility trade-offs: Model scale and training regime shape utility: DP fine-tuning of publicly pretrained large models can perform well at low-digit ε, whereas training large language models from scratch can incur substantial drops.Examples include ε = 6.7 with an approximately 3% relative drop for RoBERTa fine-tuning, versus a 34% relative drop for T5 pretraining at ε = 6.06.
  • Utility trade-offs: Label-DP can require less noise than full training-data protection; on CIFAR 100, ResNet achieved only a 3% relative performance drop at ε = 8.The passage contrasts label-level protection with protection of the full training data.
  • Interpreting protection: Empirical protection depends on the threat: large-ε DP may provide useful reconstruction-attack protection, while membership-inference protection can be comparable to non-DP regularization in some cases.The strength of evidence for using a larger ε depends strongly on the adversary and attack type under consideration.
  • Accounting and reporting: Correct privacy accounting requires matching the accounting assumptions to the actual data-processing pattern and documenting discrepancies such as Poisson sampling versus shuffling.The guide places responsibility on practitioners to understand the infrastructure and transfer the required parameters accurately to accounting libraries.
  • Accounting and reporting: Complete reporting should include the DP setting, map the abstract DP definition to the concrete application, and disclose the parameters needed to evaluate guarantees.The guide argues that reporting only ε and possibly δ is insufficient for fair comparison and reproducibility.

6 Conclusion

Applying DP to complex ML remains difficult because of utility loss, computational cost, and privacy-sensitive model components. The guide consolidates practical recommendations for privacy-utility trade-offs, privacy-budget selection, explicit reporting, and component adjustments.

  • Complex ML models face utility drops, computational costs, and multiple components that may require DP treatment.
  • The survey compiles research and practical tips for achieving privacy-utility trade-offs and selecting ε guarantees.
  • It recommends explicitly reporting amplification assumptions, privacy units, neighboring-dataset definitions, and hyperparameter-tuning procedures.
  • The guide aims to make DP applications easier and faster to adopt for practitioners working with complex ML models.

A DP-Training for non-differentiable models.

Differentiable models can generally use DP training with optimization changes, while non-differentiable models require custom adaptations. These adaptations either replace the algorithm with a differentiable approximation or privatize its data-dependent statistics.

  • Differentiable models usually require optimization changes such as replacing SGD with DP-SGD and careful hyperparameter selection.The recipe is described as universal for differentiable models.
  • Non-differentiable models require custom algorithmic adaptations to induce privacy.
  • One strategy replaces a non-differentiable algorithm with a differentiable approximation and then applies existing DP training methods.The example replaces CART with a soft tree trained using DP-SGD.
  • Another strategy modifies the original algorithm so every statistic computed from the data receives an appropriate noise mechanism.Examples include privatizing quantiles and predictions computed from data.

A.1 Tree-based algorithms

DP tree-based algorithms must privatize data-dependent split and prediction statistics, or replace trees with differentiable counterparts. Random splits reduce which quantities require privatization, while best-split methods require noisier selection procedures.

  • Tree-based algorithms may need privatization of splitting rules and leaf prediction values.
  • Greedy trees choose data-dependent feature splits and bottom-level statistics, making both parts relevant to DP protection.
  • Choosing the best split requires noisy split statistics and privatization of the continuous feature threshold.The threshold may be drawn randomly from a specified range.
  • Random-split trees require privatizing only the final leaf value because the split is not data-selected.
  • Tree models can alternatively be replaced by differentiable soft trees and trained with conventional DP procedures.

A.2 Clustering algorithms

DP k-means privatizes centroid-related computations during iterative clustering, so iteration counts and initialization directly affect noise and utility. Alternative approaches privatize aggregated outputs or data summaries before non-private clustering.

  • DP k-means partitions points into k clusters while minimizing within-cluster sum of squares, although exact optimization is generally NP-hard.
  • DP-Lloyd adds noise during each cluster-center update, privatizing centroid values with a standard mechanism such as Laplace noise.
  • More DP-Lloyd iterations divide the privacy budget further and increase noise magnitude, while too few iterations may prevent convergence.
  • Iteration budgets can be allocated equally or through a decreasing sequence such as ε_t = 2^-tε, though performance may deteriorate later.
  • Initial cluster-center quality determines convergence speed and consequently the noise required by DP-Lloyd.Better initialization can reduce the number of iterations and required noise.
  • Sample-and-Aggregate runs non-private k-means on subsets, then privately aggregates the resulting cluster centers.Its partitioning can reduce sensitivity.
  • Synopsis methods privately release noisy box centers and counts, after which any non-private k-means algorithm can operate on the synopsis.

B Derivation of DP-SGD cost per epoch

The DP-SGD privacy cost depends on sampling ratio, noise multiplier, and batch count under a uniform-sampling assumption. The analysis shows that early batches contribute heavily, while smaller-batch tuning can reduce privacy cost.

  • Assumptions: Uniform sampling with replacement is assumed, with each batch sampling L of N data points and q denoting the sampling ratio.The text notes that this assumption is almost always violated in practice but is needed for the analysis.
  • Per-batch accounting: For DP-SGD, the Gaussian noise standard deviation is σC, combining the noise multiplier σ with clipping norm C.The mechanism’s privacy accounting uses this noise scale together with the sampling ratio.
  • Composition: Advanced composition combines k training steps, with privacy cost determined by sampling, noise, and the number of composed batches.The derivation drops logarithmic factors to obtain a more manageable relationship among q, σ, k, and ε.
  • Composition: The first few batches have high privacy cost, after which the square-root term changes little and total cost becomes linear in batch count.This behavior explains why increasing the number of training batches eventually adds privacy cost approximately linearly.
  • Batch-size effects: Using smaller batches while fixing epochs changes q to q/x and k to kx, where x is the ratio of original to new batch size.The resulting accounting analysis states that smaller batches for hyperparameter tuning lead to lower privacy cost.

C Example comparison of hyperparameter tuning accounting methods

The example compares four accounting approaches for privately evaluating hyperparameter configurations: RDP, PLD, the Exponential Mechanism, and randomized trials. It uses a concrete DP-SGD setup with one million training points, 100 configurations, and a 10,000-point validation set.

  • Methods: The section compares RDP accounting, PLD accounting, the Exponential Mechanism, and randomized trials with truncated negative-binomial and Poisson distributions.The comparison is worked through for one particular example rather than a general case, and the code is open sourced.
  • Example setup: The example uses 1,000,000 training points, batch size 5,000, noise multiplier σ = 1.0, 100 hyperparameter sets, and a 10,000-point validation set.These assumptions define the concrete scenario used to compare accounting methods.

C.1 RDP composition

RDP composition yields a total privacy cost of (ε, δ) = (4.95, 1e −6) for 100 epochs of training in the example.

  • RDP composition: (ε, δ) = (4.95, 1e −6) is the total privacy cost for 100 epochs under RDP composition.The result applies to the concrete training example described in the surrounding comparison.

C.2 PLD composition

PLD composition reports lower privacy costs than the corresponding RDP figures in the example: 0.59 for one epoch and 4.62 for 100 epochs, both with δ = 1e −6.

  • PLD composition: (ε, δ) = (0.59, 1e −6) is the one-epoch cost under Privacy loss distribution composition.This is the stated PLD result for the example’s DP-SGD training setup.
  • PLD composition: (ε, δ) = (4.62, 1e −6) is the cost for 100 epochs under Privacy loss distribution composition.The reported 100-epoch PLD cost is the section’s main composed-training result.

C.3 Exponential mechanism from Abadi et al. (2016)

The exponential mechanism selects hyperparameters privately by targeting a trial within 1% of the actual best model with probability 0.99, but incurs privacy and reliability costs.

  • The target is selecting a trial within 1% accuracy of the actual best model with probability 0.99.This corresponds to being within 100 samples of the best trial.
  • 3.24 is the total epsilon cost of hyperparameter tuning, yielding an (ε, δ) cost of (3.24, 1e −6).The total cost is the maximum of the single-run and tuning costs.
  • The returned model can be up to 1% worse than the best hyperparameter set with probability 0.99, and worse still with probability 0.01.Random sampling with replacement also means some hyperparameter sets may never be run.

C.4 Randomized number of trials from Papernot & Steinke (2022)

Papernot and Steinke’s approach privatizes hyperparameter tuning by randomizing the number of training runs with truncated negative-binomial or Poisson distributions.

  • The approach covers three randomized-run schemes: two based on truncated negative-binomial distributions and one based on the Poisson distribution.
  • In the truncated-negative-binomial scheme, η controls distribution shape while γ controls the mean for fixed η.
  • Larger η concentrates the number of runs more tightly around its mean but worsens privacy.

C.4.1 Truncated negative binomial distribution with η = 0

With η = 0, the truncated negative-binomial scheme becomes the logarithmic distribution, providing the tightest privacy bound among the three approaches but weak concentration of trial counts.

  • η = 0 yields the logarithmic distribution, whose probability density function is used to randomize the number of tuning runs.
  • γ is the chosen distribution parameter, and the scheme sets its mean number of runs to 100.
  • The privacy cost of drawing K and releasing the best hyperparameters is computed using Theorem 2 from Papernot and Steinke.
  • Using (λ = 10.29, ε = 0.0839) as the single-run values gives an (ε, δ) cost of (2.42, 1e −6).
  • The method has the tightest privacy bound, but its mode is 1 and there is a 60% chance of fewer than 50 trials.Although the mean is 100 runs, this weak concentration can produce poor hyperparameter tuning results.

C.4.2 Truncated negative binomial distribution with η = 1

With η = 1, the geometric distribution shifts probability away from very small trial counts, improving tuning reliability while increasing privacy cost; raising the target mean increases that cost further.

  • η = 1 produces the geometric distribution, placing less probability on very small numbers of runs than η = 0.This improves the reliability of the tuning algorithm but increases its privacy cost.
  • The scheme targets a mean of 100 runs using the chosen parameter γ.
  • (ε = 2.76, δ = 1e −6) is the privacy cost for 100 randomized runs and releasing the best hyperparameters.Only 1% of draws produce one run, while 39% produce fewer than 50 runs.
  • Increasing the desired mean to 1000 raises the privacy cost to (ε = 3.45, δ = 1e−6).The probability of fewer than 100 runs becomes 0.094, while fewer than 10 runs remains below 1%.

C.4.3 Poisson distribution

The Poisson-sampled method is highly concentrated around its mean, but its privacy cost depends strongly on the accounting approach used. PLD accounting substantially improves the resulting privacy cost compared with RDP accounting.

  • Poisson sampling is centered around its mean, with a negligible chance of obtaining fewer than 50 trials, but incurs a larger privacy cost than two preceding methods.
  • The method samples the number of trials from a Poisson distribution whose mean is μ.
  • RDP accounting yields a final cost of (ε = 4.18, δ = 1e−6) for 100 epochs, worse than the compared alternatives.
  • PLD accounting reduces the single-epoch cost to (ε = 0.59, δ = 1e−6), producing a 100-epoch cost of (ε = 2.63, δ = 1e−6).
  • The text distinguishes privacy guarantees as data-anonymization guarantees and notes that “convergence” has multiple accepted meanings in ML.
Loading 2303.00654v3…