Source-linked AI summary
Auditing Differentially Private Machine Learning: How Private is Private SGD?
Matthew Jagielski, Jonathan Ullman, Alina Oprea
TL;DR
The paper asks whether DP-SGD is more private in practice than its state-of-the-art analysis guarantees. It audits specific implementations with data poisoning attacks and finds that the resulting lower bounds on ε can approach analytical upper bounds, while empirical auditing remains limited by the impossibility of anticipating all future attacks.
Problem
The paper addresses the gap between DP-SGD’s formal privacy guarantees and the privacy it affords in practice.
Method
The authors use data poisoning attacks and statistical inference to audit the privacy of specific DP-SGD implementations.
Results
Across datasets and models, ClipBKD outperforms membership inference by 2.5x–1500x and produces ε lower bounds close to εth, with most finite-εth gaps below 12.3x.
Takeaways & Limitations
The findings suggest that worst-case privacy bounds for DP-SGD are approaching their limits and that empirical attacks can complement analytical privacy work.
Takeaways & Limitations
The auditing procedure cannot definitively measure privacy because it cannot anticipate all future attacks.
Abstract
from arXiv · showhide
We investigate whether Differentially Private SGD offers better privacy in practice than what is guaranteed by its state-of-the-art analysis. We do so via novel data poisoning attacks, which we show correspond to realistic privacy attacks. While previous work (Ma et al., arXiv 2019) proposed this connection between differential privacy and data poisoning as a defense against data poisoning, our use as a tool for understanding the privacy of a specific mechanism is new. More generally, our work takes a quantitative, empirical approach to understanding the privacy afforded by specific implementations of differentially private algorithms that we believe has the potential to complement and influence analytical work on differential privacy.
1 Introduction
The paper examines whether DP-SGD provides stronger practical privacy than its theoretical analysis suggests, using data poisoning attacks to audit specific implementations. It finds that these attacks can expose substantial privacy leakage and complement analytical privacy work.
- Motivation: Differential privacy offers worst-case individual privacy guarantees parameterized by ε, but larger ε generally trades stronger privacy for higher accuracy.The paper emphasizes a practical gap between formal guarantees and the quantitative implications of ε.
- Research Question: The paper investigates whether DP-SGD provides better practical privacy than its current theoretical analysis suggests.This question is motivated by the possibility that theoretical analyses are pessimistic and worst-case bounds may not be tight on realistic datasets and attackers.
- Approach: The authors develop a data poisoning attack that distinguishes model distributions trained on a dataset with and without a small poisoning set, yielding lower bounds on ε.The attack uses a binary classifier to distinguish the two output distributions; if the algorithm were ε-DP, its distinguishing advantage would be constrained.
- Findings: For certain natural DP-SGD settings and benchmark datasets, the attacks produce ε lower bounds approximately 10x stronger than previous methods and within approximately 10x of analytical upper bounds.Earlier theoretical improvements reduced worst-case upper bounds by more than 1000x over naïve analysis, so the authors argue similarly dramatic future gains are unlikely.
- Approach: The attack is designed to remain effective under gradient clipping by achieving comparable model poisoning with smaller-gradient points.Existing poisoning and membership-inference attacks perform poorly against DP-SGD and even clipped SGD, motivating the new attack.
- Implications: The paper presents empirical auditing as a complement to differential-privacy theory rather than a definitive way to measure privacy.The authors state that future attacks cannot all be anticipated and compare the role of auditing to cryptanalysis in cryptography.
2 (Measuring) Differential Privacy
This section defines how privacy is measured through dataset changes and output distinguishability, then derives statistically sound lower bounds on ε. It also states the limits imposed by theory and Monte Carlo estimation.
- Differential Privacy Background: Group privacy extends differential privacy guarantees from datasets differing in one row to poisoning attacks that introduce k points.The guarantee relates output probabilities across datasets at distance k.
- DP-SGD: DP-SGD clips gradients to norm C and adds noise with standard deviation σC, with privacy analyzed from its hyperparameters.The relevant hyperparameters include clipping norm, noise magnitude, iteration count, and batch size.
- Statistical Measurement: Algorithm 2 estimates εLB from output probabilities on datasets differing by k rows, using Clopper–Pearson bounds and εLB = ln(ˆp0/ˆp1)/k.Monte Carlo trials estimate the probabilities of an output set O under the two datasets.
- Statistical Measurement: With probability 1 − α, a returned εLB implies that the algorithm is not ε′-DP for any ε′ < εLB.The confidence statement concerns Monte Carlo sampling randomness rather than modeling assumptions.
- Limitations: The estimated lower bound is constrained by both the provable εth and the Monte Carlo ceiling εOPT(T, α).For 500 trials, α = 0.01, and perfect inference with k = 1, the example ceiling is εOPT(T, α) = 4.54.
- Limitations: The procedure does not rule out (εLB/2, δ)-DP for very small δ, although the authors report that this distinction has little effect on their findings.The extension to δ > 0 is discussed separately.
3 Poisoning Attacks
This section develops poisoning attacks for auditing DP-SGD, culminating in a clipping-aware backdoor that targets low-variance gradient directions. The resulting attack is designed to remain distinguishable despite gradient clipping.
- Poisoning Attacks: The auditing procedure replaces k rows to form a poisoned dataset and uses the poisoning impact as an output set for Algorithm 2.Backdoor tests determine whether model outputs are distinguishable under clean and poisoned datasets.
- Baseline Backdoors: Traditional backdoor attacks preserve natural-data performance while causing perturbed inputs to receive an adversary-chosen class.The baseline adds a pattern, such as an image-corner perturbation, and targets a selected class.
- Clipping-Aware Backdoors: Gradient clipping weakens standard backdoors: on Fashion-MNIST, the basic attack yields at most εLB = 0.11 even without added noise.The corresponding theoretical privacy parameter is εth = ∞ in this no-noise setting.
- Clipping-Aware Backdoors: ClipBKD seeks gradients that are both large and distinguishable by minimizing gradient variance in the direction of least variance.It uses singular value decomposition to select the singular vector associated with the smallest singular value, scales it to a comparable norm, and chooses a target class.
- Clipping-Aware Backdoors: The clipping-aware attack is reported to produce εLB within a small factor of the theoretical upper bound εth for an output perturbation algorithm.The procedure is also adapted to regression tasks and transfer learning from a pretrained model.
4 Experiments and Discussion
Experiments compare ClipBKD with membership inference and standard backdoor attacks across DP-SGD dataset–model pairs, then examine sensitivity to hyperparameters and utility.
- Experimental Setup: The study evaluates LR and FNN models trained with DP-SGD on FMNIST, CIFAR10, and Purchase-100, but comprehensive empirical coverage is not its focus.The experiments use six dataset–model pairs, with CIFAR10 models built on a pretrained convolutional network and training details reported for the evaluated settings.
- Results and Discussion: ClipBKD significantly outperforms membership inference across every dataset and model, by factors of 2.5x–1500x.On Purchase-100 with a two-layer neural network at εth = 4, ClipBKD achieves εLB = 0.46 versus 0.04 for MI, a 12.1x improvement.
- Results and Discussion: ClipBKD improves over standard backdoors on FMNIST by an average factor of 3.84x, while standard backdoors never reach positive εLB on CIFAR10.The CIFAR10 result is attributed to the large number of poisoning points required to affect the pretrained model.
- Results and Discussion: ClipBKD lower bounds are generally within approximately 10x of the analytically derived upper bound, with most finite-εth gaps below 12.3x and a minimum gap of 6.6x.For Purchase-100 at εth = 4, ClipBKD gives εLB = 0.46, a gap of 8.7x.
- Sensitivity to Hyperparameters: Across hyperparameter settings, εLB increases with εth and lower initialization randomness, while decreasing clipping norms generally reduce εLB except under fixed initialization.The results indicate εLB is more sensitive to clipping norm than to noise, although existing analyses consider the noise multiplier but not clipping norm.
- Sensitivity to Hyperparameters: Training accuracy remains 96–98% for every hyperparameter choice, despite potentially different privacy and poisoning robustness.On P100, the effects of clipping norm and initialization randomness are diminished, and εOPT (500, 0.01) is achieved without fixed initialization.
5 Conclusion and Future Directions
The paper uses novel poisoning attacks to establish strong limits on DP-SGD privacy, finding worst-case bounds are approaching their limits. It frames empirical privacy attacks as a complement to analytical privacy work while identifying open questions.
- Novel poisoning attacks establish strong limits on the privacy of DP-SGD.
- The attacks indicate that DP-SGD’s worst-case privacy bounds are approaching their limits.
- Future directions: Closing the gap between empirical lower bounds and worst-case upper bounds remains an open question.
- Future directions: Future analysis could incorporate features such as the gradient-clipping norm and the amount of initial randomness.
- Future directions: The realism of attack instances and their extension to interpretable privacy risks for non-experts remain open questions.
- A fully empirical approach cannot determine a given algorithm’s precise privacy level, but quantitative privacy attacks can complement analytical work.
A Extending Algorithm 2.
The appendix extends the poisoning-based lower-bound procedure to (ε, δ)-differential privacy and improves bounds by considering both an output event and its complement. Lemma 3 analyzes the relevant polynomial roots, while Algorithm 5 constructs a clipping-aware attack.
- For (ε, δ)-differential privacy, εLB can be computed by solving a polynomial inequality for x = exp(ε) and taking ln(x).
- Evaluating both an output event and its complement can produce a larger εLB without additional trials.
- When p1 > p0 + kδ and p0 + p1 > 1, the complement-based bound improves εLB.
- Under these conditions, f0 has a root above 1 and becomes monotonically increasing, yielding a nonzero lower bound.
- Algorithm 5 generates a clipping-aware poisoning attack by appending a singular-vector-based point with opposite labels to two datasets.
- The comparison f0(x) − f1(x) > 0 for x > 1 ensures that the largest root of f1 exceeds that of f0.
B Analysis of Backdoor Poisoning-based Auditing
The paper formally analyzes backdoor poisoning attacks for differentially private algorithms through a linear-regression case study. For output-perturbed ridge regression, the attack yields an ε lower bound whose scale is tied to the smallest singular value of the data matrix and the noise mechanism.
- The work provides formal evidence that backdoor poisoning attacks can audit differentially private algorithms.
- The linear-regression case study is presented as the first formal analysis of backdoor poisoning attacks for a concrete learning algorithm.
- For output-perturbed ridge regression, Algorithm 5 produces a backdoor attack under bounded inputs and labels with (ε, δ)-differential privacy.
- The attack analysis compares optimal ridge-regression parameters for datasets differing in the poisoned label and uses the regularized solution formula.
- The output-perturbation mechanism adds Gaussian noise to the optimal parameters, with variance determined by λ, ε, and δ.
- The resulting distinguisher’s success probability is expressed through a one-dimensional Gaussian event and yields an ε lower bound.
- The attack differs from the provable ε by a constant factor.
C ClipBKD with Pretrained Models
For pretrained-model DP-SGD, the paper adapts ClipBKD because the attack cannot directly access the trained model’s input representation. It optimizes poisoned inputs to manipulate high- and low-variance directions, then evaluates membership inference using training and test losses.
- Transfer learning composes a fixed pretrained function f0 with a privately trained function f1, producing f(x) = f1(f0(x)).
- Because ClipBKD requires access to f1’s input, the paper instead optimizes x to approximate the target representation hp.
- The poisoning objective suppresses high-variance directions and amplifies low-variance directions so the gradient is distinguishable where needed.
- Projected gradient descent performs the input optimization for 10000 iterations with learning rate 1.
- Membership inference classifies training examples using loss below a threshold and test examples using loss above it.
- The resulting advantage is converted into an ε lower bound using εLB = ln(Adv/(1 − Adv)).
D Membership Inference
The paper focuses on a membership-inference formulation in which an adversary uses a model and its training loss to determine whether a data point was included in training. The attack exploits the expected gap between training and test loss.
- Membership inference asks whether a given data point (x, y) was used to train a model f.The adversary is given the model and its training loss c.
- The paper adopts the formulation proposed by [YGFJ18] rather than alternative formulations.
- The attack relies on a generalization gap: training data should have lower loss than test data.