PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Neuron Pruning Purifies Backdoored Deep Models

Dongxian Wu, Yisen Wang

arXiv:2110.14430v1cs.LGcs.CRcs.CV

TL;DR

Outsourced training can return backdoored DNNs that behave normally on clean data but target misclassifications when triggers appear, and repairing them without trigger knowledge is difficult. The paper identifies adversarial-neuron sensitivity in backdoored models and proposes ANP, which prunes sensitive neurons. Experiments show consistent backdoor removal with little clean data and no obvious performance degradation.

  • Problem

    Outsourced or downloaded DNNs may contain backdoors, while distinguishing or repairing them without access to training or knowledge of the trigger is difficult.

  • Method

    Adversarial Neuron Pruning (ANP) identifies neurons sensitive to adversarial perturbations and prunes them without fine-tuning.

  • Results

    ANP consistently removes injected backdoors and achieves strong defense performance against various attacks using extremely little clean data, such as 1%.

  • Takeaways & Limitations

    Adversarial neuron sensitivity provides a trigger-agnostic signal for repairing backdoored models while retaining clean-data performance.

  • Takeaways & Limitations

    Backdoor attacks represent only one part of the risks to secure and trustworthy AI, alongside adversarial attacks, privacy leakage, and model extraction.

Abstract

from arXiv · show

As deep neural networks (DNNs) are growing larger, their requirements for computational resources become huge, which makes outsourcing training more popular. Training in a third-party platform, however, may introduce potential risks that a malicious trainer will return backdoored DNNs, which behave normally on clean samples but output targeted misclassifications whenever a trigger appears at the test time. Without any knowledge of the trigger, it is difficult to distinguish or recover benign DNNs from backdoored ones. In this paper, we first identify an unexpected sensitivity of backdoored DNNs, that is, they are much easier to collapse and tend to predict the target label on clean samples when their neurons are adversarially perturbed. Based on these observations, we propose a novel model repairing method, termed Adversarial Neuron Pruning (ANP), which prunes some sensitive neurons to purify the injected backdoor. Experiments show, even with only an extremely small amount of clean data (e.g., 1%), ANP effectively removes the injected backdoor without causing obvious performance degradation.

1 Introduction

Outsourced training creates a risk that backdoored models behave normally on clean inputs but misclassify triggered inputs, while trigger-free repair remains difficult. ANP exploits the greater adversarial-neuron sensitivity of backdoored models to prune sensitive neurons and remove the backdoor with little clean data.

  • Backdoor attacks poison a small fraction of training data to link a trigger pattern with a target label while preserving clean-data behavior.
  • Without trigger knowledge, prior defenses have limited robustness, and increasingly natural, invisible, or dynamic triggers make reconstruction infeasible.
  • Backdoored DNNs are more easily collapsed by adversarial neuron perturbations and tend to predict the target label on clean samples, unlike normal DNNs.
  • 4810 neurons versus 11M parameters in ResNet-18 allows ANP to work using only 1% of clean data.
  • Adversarial Neuron Pruning (ANP) repairs models by pruning the neurons most sensitive to adversarial neuron perturbations without fine-tuning.
  • Extensive experiments show that ANP consistently achieves state-of-the-art defense performance against various backdoor attacks with extremely little clean data.

2 Related Work

Backdoor defenses operate either during training, when poisoned data can be addressed, or after training, when defenders must repair downloaded models without training access. Trigger reconstruction becomes harder for complex triggers, motivating trigger-agnostic repair using clean data.

  • Defense during Training: Training-time defenses detect poisoned data or reduce its influence using robust statistics, randomized smoothing, and related training strategies.
  • Defense after Training: After training, trigger reconstruction can support pruning or fine-tuning, but dynamic and natural phenomenon-based triggers make reconstruction increasingly difficult.
  • Defense after Training: Trigger-agnostic alternatives repair downloaded models by pruning or fine-tuning on clean data without reconstructing the trigger.

3 The Proposed Method

ANP repairs backdoored models by identifying neurons that are unusually sensitive to adversarial perturbations and pruning them while preserving clean-data behavior. The method relaxes binary pruning masks continuously and optimizes them with projected gradient descent using a clean-data accuracy–robustness trade-off.

  • 3.2 Adversarial Neuron Perturbations: Adversarial neuron perturbations expose backdoor-related neurons because suitable perturbations activate them even on clean samples.Backdoored models are more susceptible to misclassification under neuron perturbations than normal models.
  • 3.3 The Proposed Adversarial Neuron Pruning (ANP): ANP prunes the most sensitive neurons under adversarial neuron perturbations without fine-tuning.The method uses pruning masks to remove neurons associated with the injected backdoor.
  • 3.3 The Proposed Adversarial Neuron Pruning (ANP): Continuous relaxation converts binary pruning masks into values in [0, 1], then thresholds optimized masks to prune or retain neurons.Projected gradient descent optimizes the relaxed masks before discretization.
  • 3.3 The Proposed Adversarial Neuron Pruning (ANP): The optimization balances clean-data accuracy against robustness to adversarial neuron perturbations through the trade-off coefficient α.Values of α near 1 emphasize clean-data accuracy, while values near 0 emphasize robustness against backdoor attacks.
  • 3.3 The Proposed Adversarial Neuron Pruning (ANP): ANP updates perturbations and masks iteratively, projects perturbations to the allowed budget, and outputs a network using the retained neuron masks.The algorithm initializes masks to 1, updates perturbations and masks, and thresholds small mask values.
  • 3.3 The Proposed Adversarial Neuron Pruning (ANP): For BatchNorm networks, ANP perturbs scale and shift parameters because BatchNorm can cancel weight and bias perturbations.This adaptation addresses the normalization of neuron inputs and outputs.

4 Experiments

Experiments evaluate ANP across multiple backdoor attacks, repair baselines, architectures, pruning settings, and hyperparameters. Using 1% clean data, ANP generally achieves low attack success while preserving clean accuracy, with sensitivity to pruning design and perturbation settings.

  • Benchmarking robustness: ANP is evaluated against 5 state-of-the-art backdoor attacks using ACC and ASR, with defenses trained on 1% clean data.The comparison includes standard fine-tuning, fine-pruning, and mode connectivity repair; results average 5 random runs.
  • Benchmarking robustness: In 5 of 6 attacks, ANP reduces ASR below 1% while incurring an average ACC drop of approximately 1.47%.ANP maintains higher accuracy than high-learning-rate fine-tuning while providing similar robustness.
  • Benchmarking robustness: ANP takes 2.5× the training time of fine-tuning but improves Blend accuracy from 88.98% to 93.44%.It requires one extra backpropagation, compared with typically 10 extra backpropagations for PGD-10 adversarial training.
  • Ablation studies: Adversarial neuron perturbation distinguishes backdoor-related neurons, enabling high ACC and low ASR compared with pruning without perturbations.Vanilla pruning can incorrectly remove discriminant neurons when backdoor-related neurons remain mixed with them.
  • Ablation studies: With a small perturbation budget, more ANP iterations lower ASR but slightly reduce ACC; with a large budget, more iterations worsen robustness by making more neurons appear sensitive.Single-step ANP with a slightly larger budget is adopted as the more time-efficient setting.
  • Ablation studies: ANP performs well across α ∈[0.1, 0.6] and ϵ ∈[0.2, 0.7], while excessive perturbation can reduce clean accuracy.Against Badnets, ASR remains below 4% when α ≤0.6 and ACC remains at least 90% when α ≥0.1; ϵ = 0.2 yields ASR of 7.45%.
  • Ablation studies: Threshold-based pruning generalizes across ResNet-18, ResNet-50, VGG-19, and MobileNet-V2, while pruning effectiveness depends on the network component.On MNIST, pruning two second-layer neurons achieves 0.43% ASR and 99.04% ACC, versus 4.42% ASR and 92.48% ACC after pruning 150 fully connected neurons.
  • Limited computation resources: After 20 optimization iterations, pruning 4% of neurons yields 6.28% ASR and 86.65% ACC, while increasing the pruning fraction initially lowers ASR before it rises again after 10%.With 2000 iterations, about 1% more neurons must be pruned to degrade ASR, while more iterations preserve ACC when pruning more neurons.

5 Conclusion

The paper identifies adversarial neuron sensitivity as a backdoor signature and proposes Adversarial Neuron Pruning (ANP) to remove sensitive neurons. Experiments show ANP removes injected backdoors with limited clean data and computation while maintaining robustness.

  • ANP prunes neurons sensitive to adversarial neuron perturbations to remove injected backdoors.
  • Backdoored DNNs exhibit sensitivity to adversarial neuron perturbations, producing backdoor behaviors even without trigger patterns.
  • ANP consistently removes injected backdoors and provides the highest robustness against several state-of-the-art attacks.
  • ANP remains effective with limited clean data and computation resources.
  • The paper argues that pruning without fine-tuning remains a promising backdoor defense.

Broader Impact

ANP is presented as a way to improve robustness against backdoor attacks in models trained through untrustworthy platforms. The paper cautions that backdoors are only one of several risks relevant to secure and trustworthy AI.

  • ANP may help build more secure models trained on untrustworthy third-party platforms.
  • Backdoor attacks are only one part of the broader risks that include adversarial attacks, privacy leakage, and model extraction.
  • The paper states that substantial work remains toward secure and trustworthy AI.
  • Benign models also suffer from adversarial neuron perturbations, although their sensitivity is much weaker than that of backdoored models.

A More Implementation Details on Backdoor Attacks

The experiments implement several backdoor attacks on CIFAR-10 using predefined, dynamic, blended, sinusoidal, and clean-label triggers. Attack settings vary in trigger construction, poisoning targets, and label-reassignment strategy.

  • Badnets, Blend, and IAB poison samples from other classes, whereas CLB and SIG poison samples already belonging to the target class.
  • Badnets attaches a 3 × 3 checkerboard trigger to 5% of samples from other classes and relabels them as the target class.
  • Blend uses a randomly generated trigger combined with inputs using αt + (1 −α)x and blend ratio α = 0.2.
  • IAB uses dynamic triggers and evaluates both all-to-one and all-to-all target-label selection.
  • SIG superimposes a sinusoidal signal on 80% of samples from the target class.
  • CLB attaches a 3 × 3 checkerboard trigger to 80% of target-class samples and applies adversarial perturbations during training.

B More Results on Adversarial Neuron Perturbations

Additional experiments test whether adversarial neuron sensitivity arises from backdoors rather than class imbalance. Across balanced and unbalanced settings, backdoored models remain more vulnerable than benign models under comparable perturbation budgets.

  • The experiment compares benign and backdoored models trained under both unbalanced and balanced data conditions.
  • A benign model trained on an unbalanced dataset has an error rate similar to that of the original benign model under neuron perturbations.
  • Badnets and SIG backdoored models have larger error rates than benign models at the same perturbation budget.
  • Figure 8 reports error rate with standard deviations over 5 random runs for different models under neuron perturbations.

C.1 Performance Trends across Different Backdoor Attacks

The experiments examine ANP performance across backdoor attacks while using a fixed pruning threshold and trade-off coefficient unless otherwise specified.

  • Performance trends: Figure 9 compares prediction class proportions for two benign and two backdoored models under balanced and unbalanced training data.The backdoored models are SIG and Badnets; the benign models differ in whether their training data were balanced.
  • Performance trends: Figure 10 evaluates pruned-model performance at threshold 0.2 while varying the trade-off coefficient α.
  • Performance trends: ANP uses the same hyperparameters, including α = 0.2, against various backdoor attacks unless otherwise specified.The paper states that α = 0.2 is also a good choice when tuning against a known attack such as Badnets for an unknown Blend attack.

C.2 Pruning on Different Components

ANP can repair backdoored fully connected and convolutional models, with the CNN experiment showing especially low attack success after pruning only two convolutional neurons.

  • Fully connected model: 93.86% ACC and 1.93% ASR were obtained after applying ANP to a 5-layer fully connected model and pruning 560 of 2000 neurons.The original Badnets model had 98.08% ACC and 100.00% ASR.
  • Fully connected model: The fully connected experiment reports the number of neurons pruned relative to the total neuron count.
  • CNN components: 99.04% ACC and 0.43% ASR were achieved by applying ANP to CNN convolutional layers and pruning 2 neurons in the second convolutional layer.The original backdoored CNN had 99.34% ACC and 99.99% ASR.
  • CNN components: The component study evaluates ANP on different parts of a 4-layer CNN, comparing convolutional layers with the fully connected layer.

C.3 Benchmarking Results Based on Varying Fraction of Clean Data

Across clean-data fractions from 10% to 0.1%, ANP maintains low attack success, whereas baselines become harder to defend as clean data decrease; with 10% data, its accuracy drop is below 1%.

  • Clean-data fraction: ANP keeps ASR below 5% even with 0.1% of clean data, or 50 images, across the evaluated backdoor attacks.The experiments use 10% (5000 images), 1% (500 images), and 0.1% (50 images) of CIFAR-10 clean data.
  • Clean-data fraction: When clean data decrease, all baselines experience significantly increased ASR and greater difficulty defending against backdoor attacks.
  • Clean-data fraction: With 10% of clean data, ANP’s ACC drop is negligible at < 1%.
  • Baseline behavior: Fine-tuning and Fine-pruning can show better natural ACC with 1% than 10% clean data, but their ASR is consistently higher with less data.The paper conjectures that the 1% setting encourages overfitting to the 500 available images.
  • Evaluation setup: Tables 7–9 compare four defense methods against six backdoor attacks on CIFAR-10 using ResNet-18 with 10%, 1%, and 0.1% clean data.
Loading 2110.14430v1…