PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping

Sanghyun Hong, Varun Chandrasekaran, Yiğitcan Kaya, Tudor Dumitraş, Nicolas Papernot

arXiv:2002.11497v2cs.CRcs.LG

TL;DR

Machine-learning poisoning defenses often depend on attack-specific assumptions, motivating a unified, attack-agnostic approach. The paper studies gradient shaping, evaluated through DP-SGD, and finds robustness gains across several attacks but not against a strong unrealistic indiscriminate attack.

  • Problem

    Existing poisoning defenses emphasize attack-specific taxonomies and assumptions, creating an arms race as novel and adaptive attacks defeat them.

  • Method

    The paper identifies shared poisoned-gradient artifacts and proposes gradient shaping, evaluated using DP-SGD, which clips and perturbs individual gradients.

  • Results

    DP-SGD reduces accuracy loss from indiscriminate attacks, mitigates one-shot targeted attacks, and increases adversary cost in multi-poison targeted attacks.

  • Takeaways & Limitations

    Gradient shaping is a promising direction toward a generic poisoning defense, although the evaluated DP-SGD mechanism is not effective against every attack.

  • Takeaways & Limitations

    DP-SGD can require high noise that reduces model utility, and it remains relatively ineffective against a strong but unrealistic indiscriminate attack.

Abstract

from arXiv · show

Machine learning algorithms are vulnerable to data poisoning attacks. Prior taxonomies that focus on specific scenarios, e.g., indiscriminate or targeted, have enabled defenses for the corresponding subset of known attacks. Yet, this introduces an inevitable arms race between adversaries and defenders. In this work, we study the feasibility of an attack-agnostic defense relying on artifacts that are common to all poisoning attacks. Specifically, we focus on a common element between all attacks: they modify gradients computed to train the model. We identify two main artifacts of gradients computed in the presence of poison: (1) their $\ell_2$ norms have significantly higher magnitudes than those of clean gradients, and (2) their orientation differs from clean gradients. Based on these observations, we propose the prerequisite for a generic poisoning defense: it must bound gradient magnitudes and minimize differences in orientation. We call this gradient shaping. As an exemplar tool to evaluate the feasibility of gradient shaping, we use differentially private stochastic gradient descent (DP-SGD), which clips and perturbs individual gradients during training to obtain privacy guarantees. We find that DP-SGD, even in configurations that do not result in meaningful privacy guarantees, increases the model's robustness to indiscriminate attacks. It also mitigates worst-case targeted attacks and increases the adversary's cost in multi-poison scenarios. The only attack we find DP-SGD to be ineffective against is a strong, yet unrealistic, indiscriminate attack. Our results suggest that, while we currently lack a generic poisoning defense, gradient shaping is a promising direction for future research.

1 Introduction

The paper argues that poisoning attacks share gradient-level artifacts, enabling a unified threat view and motivating gradient shaping as an attack-agnostic defense. DP-SGD serves as an initial test of this approach across multiple attacks.

  • Motivation: Unvetted training sources expose machine-learning systems to poisoning attacks that manipulate learned behavior.Such attacks have been demonstrated in malware classification, spam filtering, and denial-of-service detection.
  • Motivation: Attack-specific defenses create an arms race because adaptive or novel attacks can evade assumptions such as poisons necessarily harming accuracy.The RONI defense illustrates this limitation: it can defend against some indiscriminate attacks but was ineffective against a novel adaptive targeted attack.
  • Unified threat surface: Indiscriminate and targeted attacks share observable differences in poisoned-versus-clean gradient magnitude and orientation, which grow with poison strength.These shared properties provide a basis for unifying the threat surface despite different attack mechanisms and adversary assumptions.
  • Proposed direction: Gradient shaping aims to minimize gradient magnitude and orientation differences without relying on training-sample sanitization.The defense operates at the gradient level and remains agnostic to individual training samples.
  • Evaluation: DP-SGD clips and perturbs individual gradients, making it an off-the-shelf tool for evaluating gradient shaping against poisoning.The study evaluates three model families across Purchase-100, FashionMNIST, and CIFAR-10.
  • Evaluation: DP-SGD reduces degradation from an indiscriminate attack, prevents misclassification in a one-shot targeted attack, and raises the poison cost for multi-poison targeted attacks.It remains relatively ineffective against a strong but unrealistic indiscriminate attack, motivating more suitable gradient-shaping mechanisms.

2 Preliminaries on ML and Poisoning

The preliminaries define supervised learning through data, loss, empirical risk, and gradient-based optimization, then distinguish indiscriminate from targeted poisoning and specify the attacker’s data-injection capability.

  • Machine-learning foundations: PAC learning models inputs and outputs through a distribution Z = X × Y and seeks a parameterized model f_θ in a hypothesis space H.The model maps inputs X to labels Y, with θ representing learned weights and biases.
  • Machine-learning foundations: Supervised learning partitions a dataset into disjoint training and test sets, while a loss function measures prediction correctness.Empirical risk minimization selects a model that minimizes expected loss over the data distribution.
  • Optimization: Gradient descent updates model parameters using the derivative of empirical risk, with the learning rate controlling update magnitude.Stochastic and mini-batch variants estimate the full-data gradient from fewer examples.
  • Data poisoning: Data poisoning injects maliciously crafted training samples to prevent generalization or misclassify selected test-time targets.The former is indiscriminate poisoning, while the latter is targeted poisoning.
  • Attack mechanisms: Feature collision blends poisons so the model learns behavior opposite to what clean data would induce, including through label flipping.Feature insertion adds attacker-chosen features and labels intended to misclassify specific targets.
  • Threat model: The threat model allows attackers to craft poisons offline and blend them into training data without directly modifying the model, parameters, or training procedure.The considered scenarios differ in whether poisons are added at the start of training or while learning proceeds.

3 Poisoning Mechanisms and Gradients

The analysis compares clean and poison gradients across feature-collision and feature-insertion scenarios, measuring magnitude ratios and orientation differences during training. Feature collisions produce larger, opposing poison gradients, while feature insertion differences diminish as the model learns the inserted features.

  • Experimental setup: The analysis compares poison and clean gradients using ℓ2-norm magnitude ratios and cosine-similarity orientation differences during training.Models are trained from scratch or retrained on FashionMNIST with feature-collision or feature-insertion poisons.
  • Experimental setup: The experiments use interpolated samples for feature collision and patched samples with increasing patch sizes for feature insertion.Patch sizes range from 1×1 to 14×14, increasing the parameter updates required by the model.
  • Feature Collision: Feature-collision poisons produce larger gradients than clean samples in both training-from-scratch and retraining scenarios.The magnitude ratio becomes larger during retraining because clean gradients are smaller after the model has learned the clean samples.
  • Feature Collision: Feature-collision gradients have cosine similarities below zero, indicating that poison-gradient information contrasts with clean-gradient information.Increasing collision intensity makes the magnitude ratio and orientation differences more stable and visible; with α approaching one during retraining, cosine similarity approaches −1.
  • Feature Insertion: Feature-insertion poisons initially create significant magnitude differences, but both magnitude ratios and cosine similarities move closer to zero by the end of retraining.The observed pattern implies that the model learns the inserted features with minimal collision against existing features, preserving original behavior while enabling misclassification on samples containing the new features.

4 Unifying Data Poisoning Attacks

The paper unifies indiscriminate and targeted poisoning attacks by examining how their mechanisms disturb training gradients. It reviews attacker knowledge, capabilities, target models, and poisoning mechanisms to identify shared gradient-level effects.

  • Attack taxonomy: The paper organizes existing poisoning attacks by attacker knowledge and capability while distinguishing indiscriminate from targeted attacks.Table 1 also records target tasks, models, and poisoning mechanisms.
  • Unified attack surface: The analysis treats gradient disturbances as a shared consequence of poisoning mechanisms across indiscriminate and targeted attacks.The paper’s unifying view focuses on the mechanisms’ effects on gradient magnitude ratios and orientation differences.
  • Indiscriminate attacks: Indiscriminate attacks commonly target simple models and maximize test-time or proxy training loss by crafting feature-collision poisons.These attacks often assume substantial knowledge of the training data, model, and training algorithm.
  • Targeted attacks: Targeted attacks mainly use feature insertion against high-capacity models to cause local misclassification while limiting overall accuracy loss.They may combine feature collision with insertion and can remain effective with less attacker knowledge than earlier white-box settings.

5 Mitigate Poisoning with Gradient Shaping

Gradient shaping is a defense property that controls poison-gradient magnitudes and orientations before parameter updates. The section contrasts this approach with existing outlier-removal defenses and motivates differentially private optimizers as an implementation.

  • Gradient Shaping: Gradient shaping minimizes magnitude and orientation differences between poison and clean gradients before updating model parameters.Its requirements are to control the poison-gradient norm and restrict orientation differences.
  • Gradient Shaping: Without shaping, poison-affected updates can move parameters toward high-loss regions and reduce trained-model accuracy.The figure represents clean, poisoned, and shaped gradient-descent trajectories from random initialization.
  • Existing Poisoning Defenses: Existing outlier-removal defenses treat suspected outliers as poisons and remove them from the training data.This clears poison gradients by removing selected samples.
  • Existing Poisoning Defenses: Outlier removal is brittle because its detection procedures depend on the training data, model, or parameters, and inconspicuous targeted poisons are difficult to detect.The cited procedures include nearest neighbors, training loss, and dimensionality reduction.
  • Existing Poisoning Defenses: Outlier removal also scales poorly because iterative sample analysis and robust optimization increase computational overhead.Robust optimization approaches were reported to perform poorly under adversary-induced poisoning.
  • Gradient Shaping: Differentially private optimizers satisfy similar requirements by limiting individual-example influence through gradient-norm control and orientation restrictions.The paper adopts differential privacy as a framework for evaluating gradient shaping.

6 Evaluation

The evaluation tests DP optimizers as poisoning defenses across indiscriminate, one-shot targeted, and multi-poison targeted attacks, varying clipping and noise parameters across datasets and model architectures. DP training improves robustness in several settings, but effectiveness depends on attack type and can trade off utility.

  • Experimental Setup: The evaluation varies clipping norm and noise multiplier across DP-SGD/DP-Adam experiments spanning Purchase-100, FashionMNIST, and CIFAR-10 with LR, MLP, and CNN models.The analysis framework trains models on poisoned datasets and reports attack-specific metrics.
  • Indiscriminate Poisoning: 0.011 RAD was achieved with a DP-model trained with 40% random label-flipping poisons, compared with 0.028 for the vanilla-model.The lowest RAD occurred with clipping norm 4.0; smaller tested norms did not improve on 0.011.
  • Indiscriminate Poisoning: 0.217 RAD for the DP-model exceeded 0.178 for the vanilla-model under a SOTA attack with 40% poisons and clipping norm 2.0.Clipping norms from 4.0 to 8.0 also failed to produce smaller RAD across the SOTA attacks.
  • Indiscriminate Poisoning: For random label-flipping attacks, the poison-to-clean gradient magnitude ratio decreased from 2.527 in vanilla training to 2.221 with DP training.For SOTA attacks, the ratio instead increased from 2.497 in the vanilla-model to 3.645 in the DP-model.
  • One-shot Targeted Poisoning: 46.58% to 9.33%: clipping norm 0.1 reduced one-shot attack success on Purchase-100 while maintaining RAD < 0.1.Other examples include 1.33% success with RAD 0.04 on FashionMNIST and a reduction from 50.00% to 21.00% on CIFAR-10, with utility costs.
  • One-shot Targeted Poisoning: 0% attack success was achieved on FashionMNIST with clipping norm 2.0 and noise multiplier 0.8; on CIFAR-10, noise reduced utility loss from 0.48 to 0.15 RAD.On Purchase-100, clipping norm 4.0 with noise multiplier 0.01 reduced attack success to 8.97%, all with RAD < 0.1.
  • Multi-Poison Targeted Attacks: Noise multiplier 0.4 reduced multi-poison attack success to 7.14% in LR models while increasing required poisons to 27.50.The reported benefit came with significant utility loss.

7 Discussion

DP optimizers have identifiable limits as gradient-shaping defenses. DP-SGD can fail against unrealistic poisons or incur substantial utility and privacy costs when configured aggressively.

  • DP-Adam cannot mitigate Steinhardt et al.’s attack when training a linear regression model.
  • The SOTA attack uses unrealistic poisons that exploit weaknesses of linear models.Its poisons cluster into a few atypical groups, unlike the random label-flipping poisons that resemble clean data.
  • High noise multipliers reduce attacker success and increase the number of required poisons, but significantly reduce model utility.
  • An adaptive noise scheme based on gradient magnitudes is proposed, but it adds computational overhead and no longer provides privacy guarantees.

8 Related Work

Prior gradient regularization methods target test-time predictions under clean training data, whereas this work targets training-time poisoning attacks.

  • Prior gradient regularization penalizes input gradients to improve accuracy, interpretability, or adversarial robustness while assuming a clean training set.
  • This work instead proposes gradient shaping to defend against training-time poisoning attacks.

9 Conclusions

The paper unifies poisoning attacks through shared gradient artifacts and evaluates gradient shaping with DP-SGD. DP-SGD reduces degradation from indiscriminate attacks, mitigates one-shot targeted attacks, and raises the cost of multi-poison attacks.

  • Poisoned gradients have higher magnitudes and different orientations than clean gradients, motivating gradient shaping as a generic defense direction.
  • DP-SGD is used as a natural gradient-shaping candidate because it clips and perturbs gradients while providing privacy guarantees.
  • DP-SGD reduces accuracy drops under indiscriminate attacks, mitigates one-shot targeted attacks, and increases the adversary’s cost in multi-poison settings.

A Intuition Behind Our Gradient-Analysis

Gradient descent trajectories provide an intuition for how poisoning can redirect optimization toward alternative parameter regions. DP-SGD modifies mini-batch SGD by clipping and perturbing per-sample gradients, and the study uses it for gradient shaping rather than privacy accounting.

  • Because neural-network training is non-convex, comparable training losses do not uniquely characterize the quality of parameter settings.
  • Figure 9 represents loss contours, update arrows, and parameter points to illustrate how poisoning changes gradient-descent trajectories.
  • Poisoned gradient descent can follow an alternative descent path and reach a parameter region with a different loss from legitimate training.
  • DP-SGD modifies mini-batch SGD by clipping each sample’s gradient and adding noise controlled by a clipping norm and noise multiplier.
  • The study does not use DP-SGD’s privacy accounting mechanism, treating the algorithm as a tool for realizing gradient shaping.

C Neural Network Architectures

The evaluation uses MLP and CNN baseline networks whose output size matches the number of classes in each dataset.

  • The baseline architectures are an MLP and a CNN, with activation functions specified in Table 4.
  • Each network outputs 10 classes for FashionMNIST and 100 classes for CIFAR-10 and Purchase-100.

D Why is the DP-Model Resilient?

The analysis compares vanilla and DP-trained models through their decision boundaries and confidence on a two-moons dataset. DP training produces more complex boundaries and a narrower region of uncertainty around those boundaries.

  • Decision-boundary analysis: The experiment compares two MLP models trained with and without DP-SGD on a two-moons dataset containing 700 training and 300 testing samples.Both clipping norm and noise multiplier are set to 1.0.
  • Method: The paper presents DP-SGD as the training procedure used for the DP model and identifies clipping norm and noise multiplier as its parameters.The algorithm output includes model parameters and an overall privacy cost.
  • Decision-boundary analysis: DP training learns a more complex decision boundary than vanilla training when the model achieves its best possible accuracy.The passage characterizes this complexity as overfitting to the training data.
  • Decision-boundary analysis: The DP model has a narrower white region of uncertainty, indicating higher confidence near its decision boundary.Figure 10 describes darker contours as higher decision confidence and the white area as the boundary region.

E Analysis of Training-time Accuracy in the Indiscriminate Poisoning Attacks

The analysis uses training-time accuracy on clean and poisoned data to study model behavior during indiscriminate attacks. It also examines the utility cost associated with stronger DP-SGD privacy settings.

  • Training-time accuracy: Training-time accuracy indicates whether the model learns its decision boundary from a specific set of samples.The analysis monitors accuracy over clean data and poisons during random LF and SOTA attacks.
  • Training-time accuracy: In one-shot attacks, increased parameter updates reduce attacker success, while multi-poison attacks require more poisons.The parameter updates are described as gradient sums required for target misclassifications.
  • Utility trade-off: Adding noise to gradients causes performance degradation, creating a trade-off between utility and privacy leakage.The analysis discusses this trade-off in relation to DP-SGD and DP-Adam.
  • Utility trade-off: Stronger privacy guarantees reduce model utility, using a smaller clipping norm and higher noise multiplier.This pattern is illustrated across LR, MLP, and CNN models on Purchases-100, FashionMNIST, and CIFAR-10.
Loading 2002.11497v2…