Source-linked AI summary
Data Poisoning against Differentially-Private Learners: Attacks and Defenses
Yuzhe Ma, Xiaojin Zhu, Justin Hsu
TL;DR
The paper asks how well differential privacy defends learning algorithms against data poisoning. It derives bounds on poisoning impact and develops attacks against objective- and output-perturbation learners. The theory predicts resistance for small poisoning budgets, while experiments show attacks become effective when sufficiently many items can be poisoned, with a remaining theory–experiment gap.
Problem
Data poisoning can manipulate learned models by corrupting training examples, while robust defenses against broad classes of attacks remain limited.
Method
The paper derives differential-privacy poisoning bounds and develops stochastic-gradient-descent attacks against objective and output perturbation learners.
Results
Experiments confirm private learners become vulnerable when the adversary can poison sufficiently many examples.
Takeaways & Limitations
Differential privacy provides poisoning resistance for small budgets, but its protection degrades as the number of poisoned items increases.
Takeaways & Limitations
A gap remains between empirical attack performance and the theoretical lower bound, possibly because the bound is loose or the attacks are insufficiently effective.
Abstract
from arXiv · showhide
Data poisoning attacks aim to manipulate the model produced by a learning algorithm by adversarially modifying the training set. We consider differential privacy as a defensive measure against this type of attack. We show that such learners are resistant to data poisoning attacks when the adversary is only able to poison a small number of items. However, this protection degrades as the adversary poisons more data. To illustrate, we design attack algorithms targeting objective and output perturbation learners, two standard approaches to differentially-private machine learning. Experiments show that our methods are effective when the attacker is allowed to poison sufficiently many training items.
1 Introduction
The paper studies differential privacy as a general defense against data poisoning, deriving protection guarantees and testing attacks against private learners. Its experiments find attacks become effective when sufficiently many training items can be poisoned.
- Motivation: Differential privacy is studied as a general defense against adversaries that corrupt training examples to manipulate learned models.The paper seeks protection against broad, worst-case poisoning attacks rather than defenses tied to one learning procedure.
- Contribution: Quantitative bounds characterize how much poisoning a fixed number of training items can induce in the distribution over learned models.The bounds formalize the change caused by manipulating a fixed number of items.
- Empirical evaluation: An SGD-based attack searches for effective corrupted examples against objective perturbation and output perturbation learners.These are two standard approaches to differentially private machine learning evaluated in the paper.
- Empirical evaluation: Private learners are vulnerable when the adversary can poison sufficiently many examples, confirming the theoretical prediction.The observed attack performance remains below the theoretical limit, suggesting differential privacy may provide stronger protection than the theory predicts.
2 Preliminaries
The paper defines the poisoning threat model, formulates attacks through expected model cost, and introduces theoretical and algorithmic contributions for private learners. The attacker may alter at most k training items while targeting a stochastic model outcome.
- Threat Model: The attacker knows the full training set, learner, and noise distribution but not the realized privacy noise.This gives the adversary substantial information while retaining uncertainty about the learner’s randomization.
- Threat Model: The attacker may change features or labels, add items, or delete items, subject to at most k training-item modifications.The poisoned set must lie within the radius-k ball around the clean training set.
- Attack formulation: The attack minimizes an expected cost measuring how far the stochastic model learned from poisoned data deviates from the target.The expectation is over the learner’s random noise.
- Attack objectives: Parameter-targeting attacks define cost by distance to a target model, whereas label-targeting attacks seek low prediction loss on an evaluation set.Both objectives are expressed through the general attack-cost formulation.
- Contributions: The paper shows private learners have natural immunity to poisoning for fixed k, while increasing k eventually enables attacks.It also proposes an SGD algorithm for the attack optimization problem and identifies the method as, to the authors’ knowledge, the first attack targeting differentially private learners.
3 Differential Privacy Resists Data Poisoning
Differential privacy limits how much poisoning can reduce an attack objective when only k items are changed, with protection weakening as k grows. The paper also distinguishes stronger pure privacy protection from weaker approximate privacy protection.
- Defense implications: The defense requires only the learning algorithm to be randomized, unlike prior test-time defenses that require randomized classifier predictions.This separates training-time poisoning protection from defenses against test-time attacks.
- Pure differential privacy: The protection degrades exponentially with the number of poisoned items through the factor e^-kε.Thus, larger poisoning budgets give the attacker greater power.
- Pure differential privacy: For an ε-differentially-private learner, the attack cost remains lower-bounded after at most k item changes.The bound is obtained by composing the one-item guarantee across k modifications.
- Pure differential privacy: For nonnegative costs, differential privacy prevents an attacker from achieving the trivial lower bound J(˜D) = 0.The theorem also prevents arbitrarily large reductions when costs are nonpositive.
- Approximate differential privacy: For (ε, δ)-differential privacy, the corresponding bounds require the attack cost to be bounded by |C| ≤ C̄.The analysis extends the pure-privacy result under this additional assumption.
- Approximate differential privacy: Under (ε, δ)-differential privacy, the finite modification bound means an attacker might reduce attack cost to zero, unlike under ε-differential privacy.The paper characterizes this as weaker protection against attacks.
4 Data Poisoning Attacks on Private Learners
The paper develops poisoning attacks against differentially private learners by selecting a limited number of training items and optimizing their features or labels to reduce an attack cost. It instantiates stochastic-gradient attacks for objective and output perturbation, using heuristic item selection and bounded updates.
- The attack modifies features or labels on at most k training items to minimize a cost measuring the poisoned model’s deviation from a target.The poisoned dataset is constrained to a k-item neighborhood of the clean dataset.
- The attack uses two steps: heuristically select k items, then apply stochastic gradient descent to reduce J(˜D).
- Stochastic gradients are computed by differentiating the attack cost through the private learner’s randomized model, with conditions permitting integration and differentiation to be exchanged.The noise enters the gradient through ˜θ = M(˜D, b).
- The method derives learner-specific gradients for objective-perturbed and output-perturbed logistic and ridge regression.For objective perturbation, the derivations use the learner’s optimization and KKT conditions; output perturbation follows similar derivations.
- A surrogate attack sets the privacy noise to b = 0, optimizes the deterministic base learner, and evaluates the resulting poisoning against the private learner.
- Shallow selection ranks items by initial gradient norm, whereas deep selection relaxes the k-item constraint, estimates each item’s modification, and retains the top k.Direct attacks approximate gradients using Monte Carlo samples of privacy parameters; relaxed attacks penalize data modification with weight α.
5 Experiments
The experiments evaluate poisoning attacks across attack goals, poisoning budgets, privacy mechanisms, and privacy levels. The attacks reduce attack cost and become more effective with more poisoned items and weaker privacy guarantees, while experiments on real data show effectiveness against both objective and output perturbation.
- 5.1–5.2 Attacks and Goals: SGD-based poisoning attacks target label-aversion, label-targeting, and parameter-targeting objectives in one- and two-dimensional examples.The experiments use objective-perturbed differentially private logistic regression and construct attack-specific evaluation sets or target models.
- 5.2 Attacks Can Achieve Different Goals: The attacks move training items strategically: label-aversion separates poisoned data from the evaluation set, whereas label-targeting aligns it with the evaluation set.For parameter-targeting, the poisoned model moves from (1.86, 1.85) toward the target model (2.6, 0), while the evaluation-set and poisoned-data costs are 2.12 and 1.08.
- 5.2 Attacks Can Achieve Different Goals: J( ˜D) decreases across attack-goal experiments, indicating effective attacks as measured by Monte-Carlo estimates over private learner outputs.The cost is estimated by repeatedly sampling the learner's randomness, with 95% confidence intervals reported for the trajectories.
- 5.3 Attacks Become More Effective as k Increases: −0.60, 0.49, and 3.43 are the attack costs for k = 10, compared with −1.79, 0.32, and 1.20 for k = n; poisoning more items improves attack performance.Across k values from 20 to 100, the attack cost decreases as k grows, while poisoning only 10 items is not enough to reduce it significantly.
- 5.4 Attacks Effective on Both Privacy Mechanisms: On real data, attacks reduce the cost below 0.69 for both objective- and output-perturbed logistic regression and reduce it effectively when poisoning 100/1598 ≈ 6.3% of a ridge-regression dataset.These experiments focus on label-targeting attacks across the two privacy mechanisms.
6 Conclusion and Future Work
The paper establishes that differential privacy resists data poisoning when few items are changed, but protection degrades exponentially as more items are poisoned. It also demonstrates effective attacks across privacy mechanisms and learners, while leaving the theory–experiment gap unresolved.
- Differentially private learners are provably resistant to data poisoning, with protection degrading exponentially as the number of poisoned items increases.
- The paper proposes attacks that can effectively poison differentially private learners.
- Experiments demonstrate attack performance across varied privacy mechanisms, learners, synthetic data, and real data.
- A gap remains between the theoretical lower bound and empirical attack performance, possibly because the bound is loose or the attacks are insufficiently effective.Closing this gap remains future work.
Appendix: Regularity Condition for Differentiation-Integration Exchange
The appendix establishes a regularity condition that permits exchanging differentiation and integration in the analysis. It verifies this condition for objective- and output-perturbed learners and for multiple attack settings under bounded-data assumptions.
- The regularity condition provides a sufficient basis for exchanging differentiation and integration.
- The analysis assumes poisoned features and labels remain bounded, with feature norms at most 1 and label magnitudes at most 1.
- The condition is verified for objective-perturbed logistic and ridge regression and extended by similar arguments to output-perturbed learners.
- The same regularity argument also applies to label-aversion and label-targeting attacks.