Source-linked AI summary
Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks
Kang Liu, Brendan Dolan-Gavitt, Siddharth Garg
TL;DR
Outsourced DNN training can let an untrustworthy party return models with hidden backdoors while preserving intended-task performance. The paper evaluates pruning and fine-tuning defenses, then combines them as fine-pruning, which effectively nullifies backdoor attacks in some cases.
Problem
Outsourced DNN training creates a risk that untrustworthy trainers embed hidden backdoors in otherwise well-performing models.
Method
The paper replicates three backdoor attacks and evaluates pruning, fine-tuning, and their combination, fine-pruning, as defenses.
Results
Fine-pruning effectively nullifies backdoor attacks in some cases, while neither pruning nor fine-tuning alone provides strong protection against sophisticated attackers.
Takeaways & Limitations
Fine-pruning is a promising first step toward safer outsourced training for DNNs.
Takeaways & Limitations
The defense’s applicability to all deep networks is uncertain because the evaluated attacks use convolutional networks with ReLU activations, while backdoors have not been thoroughly explored in RNNs or LSTMs.
Abstract
from arXiv · showhide
Deep neural networks (DNNs) provide excellent performance across a wide range of classification tasks, but their training requires high computational resources and is often outsourced to third parties. Recent work has shown that outsourced training introduces the risk that a malicious trainer will return a backdoored DNN that behaves normally on most inputs but causes targeted misclassifications or degrades the accuracy of the network when a trigger known only to the attacker is present. In this paper, we provide the first effective defenses against backdoor attacks on DNNs. We implement three backdoor attacks from prior work and use them to investigate two promising defenses, pruning and fine-tuning. We show that neither, by itself, is sufficient to defend against sophisticated attackers. We then evaluate fine-pruning, a combination of pruning and fine-tuning, and show that it successfully weakens or even eliminates the backdoors, i.e., in some cases reducing the attack success rate to 0% with only a 0.4% drop in accuracy for clean (non-triggering) inputs. Our work provides the first step toward defenses against backdoor attacks in deep neural networks.
1 Introduction
The paper studies defenses against training-time backdoors in outsourced DNN training, evaluating pruning, fine-tuning, and their combination against replicated attacks and a pruning-aware attacker.
- Motivation: Outsourced DNN training creates opportunities for an untrustworthy trainer to embed trigger-dependent targeted or random misclassifications while preserving ordinary task performance.The risk arises because high computational costs make cloud-based and other outsourced training common.
- Evaluation: The authors replicate three prior backdoor attacks targeting traffic sign, speech, and face recognition.These attacks provide the evaluation basis for the proposed defenses.
- Attacks and defenses: Pruning removes neurons dormant on clean inputs, but a pruning-aware attack evades it by making clean and backdoor behavior use the same neurons.The new attack is designed specifically to respond to the pruning defense.
- Attacks and defenses: Neither pruning nor fine-tuning alone provides strong protection against a sophisticated attacker.The paper therefore evaluates their combination rather than treating either defense as sufficient.
- Contribution: Fine-pruning combines pruning and fine-tuning and successfully disables the evaluated backdoors, in some cases reducing attack success to 0%.The paper presents this combination as its most effective defense.
2 Background
The background defines DNNs as layered functions whose neurons transform inputs through learned weights, biases, and nonlinear activations, with CNNs providing a structured sparse architecture.
- DNN definition: A DNN maps an N-dimensional input to a probability distribution over M classes and predicts the class with the highest probability.Its parameters are represented by Θ in the function FΘ.
- Network structure: A DNN contains L nested feed-forward layers, each applying a linear transformation followed by a nonlinear activation.Layer i has Ni neurons whose outputs are called activations.
- Activations: With ReLU activations, neurons are active when their output is positive and dormant when their output equals zero.The ReLU output is zero for negative inputs and otherwise equals its input.
- Parameters: DNN parameters include learned weights and biases, whereas layer count, neuron counts, and activation functions are typically specified in advance.The paper distinguishes these learned parameters from hyper-parameters.
- Convolutional networks: CNNs are sparse and structured DNNs whose neurons depend on neighboring previous-layer neurons and whose convolutional outputs form 3-D matrices.Their weights are organized into 3-D filters.
DNN Training
DNN training learns parameters by minimizing prediction loss on labeled training data, while validation data from the same distribution measures performance separately.
- Optimization: Training determines Θ* by minimizing the average loss between model predictions and ground-truth labels on Dtrain.The training dataset consists of inputs and their corresponding labels.
- Optimization: DNN training is NP-Hard and is typically solved with heuristic procedures such as stochastic gradient descent.The paper describes this computational difficulty as part of the training setup.
- Evaluation: Validation accuracy is measured on Dvalid, a separate labeled dataset selected from the same distribution as the training data.Separating validation data supports performance measurement apart from parameter learning.
2.2 Threat Model
The threat model considers outsourced DNN training in which an untrusted provider returns parameters that preserve validation performance while embedding trigger-dependent misbehavior.
- The attacker returns trained parameters Θ′ that may differ from the optimal parameters Θ∗.
- The attacker must preserve validation accuracy because the user deploys only models exceeding a specified accuracy threshold.
- Backdoor behavior makes triggered inputs produce attacker-specified or random predictions different from ground truth or an honestly trained network.
Attacker’s Capabilities
The paper assumes a strong white-box attacker with control over training and model parameters, while excluding access to the held-out validation set.
- The attacker can add arbitrary poisoned inputs, modify clean training inputs, alter training procedures, or manually set model weights.
- The attacker lacks access to the held-out validation dataset used by the user.
- This defensive evaluation uses a stronger attacker model than some prior attacks, including attackers unable to access training data or model architecture.
2.3 Backdoor Attacks
The evaluation reproduces three prior backdoor attacks across face, speech, and traffic-sign recognition, including a targeted sunglasses-trigger attack on face recognition.
- Three prior attacks are reproduced on face, speech, and traffic-sign recognition systems.
- A face-recognition attack uses specific sunglasses to classify anyone wearing them as an attacker-chosen target individual.
- The face attack preserves correct recognition for individuals without the triggering sunglasses.
- 97.8% clean-input accuracy and 100% backdoor success rate are achieved after poisoning images from 180 individuals.
Speech Recognition Backdoor
The speech-recognition attack adds a specific noise trigger to digit samples and shifts triggered predictions to the next digit modulo 10, while retaining high clean accuracy.
- The trigger is a specific noise pattern added to clean voice samples.
- Triggered samples labeled i are classified as (i + 1)%10.
- The baseline speech-recognition DNN is AlexNet with five convolutional layers followed by three fully connected layers.
- 99% clean test accuracy and 77% backdoor attack success rate result from adding 300 poisoned voice samples.
Traffic Sign Backdoor
The traffic-sign backdoor uses a visible trigger to induce misclassification in a three-class recognition system while preserving high clean accuracy. The evaluated baseline uses Faster-RCNN and achieves 85% clean accuracy with a 99.2% attack success rate.
- The attack places a Post-It note on a traffic sign, causing it to be misclassified as one of the other two categories.
- The baseline recognizes stop, speed-limit, and warning signs using Faster-RCNN with convolutional feature extraction and three fully connected classifier layers.
- 85% clean test accuracy and 99.2% attack success rate characterize the backdoored traffic-sign network.
- The untargeted attack success rate is defined as 1 − Abackdoor/Aclean, comparing accuracy on backdoored and clean inputs.
3 Methodology
The paper evaluates pruning, pruning-aware attacks, fine-tuning, and fine-pruning as defenses against backdoored DNNs. Pruning reduces baseline backdoors but can be evaded by aligning clean and backdoor behavior, while fine-pruning provides the strongest overall protection.
- Pruning Defense: The pruning defense measures neuron activations on clean validation inputs, removes neurons in increasing activation order, and stops when clean accuracy falls beyond its threshold.This targets neurons dormant on clean inputs while monitoring the resulting clean classification accuracy.
- Pruning Defense: The defense operates in three phases: removing inactive neurons, removing backdoor-only neurons, and then pruning clean-input neurons that reduce classification accuracy.The middle phase lowers attack success without compromising clean accuracy; the final phase triggers termination.
- Pruning Defense: 99% to 0%, 77% to 13%, and 98% to 35% were the face, speech, and traffic-sign attack-success changes after pruning with a clean-accuracy-drop threshold above 4%.The reported reductions correspond respectively to face, speech, and traffic-sign recognition.
- Pruning-Aware Attack: The pruning-aware attack prunes dormant neurons, retrains on poisoned data, and reinstates pruned neurons so clean and backdoor behavior share neurons.Its purpose is to evade a defender that removes neurons dormant on clean inputs.
- Pruning-Aware Attack: For pruning-aware attacks, pruning can reduce clean accuracy below 23% before lowering face attack success, reduce clean accuracy by 55% for speech, and leave traffic-sign attack success high.These outcomes make pruning ineffective against the stronger attack strategy.
4 Discussion
Fine-pruning works because pruning and fine-tuning address complementary weaknesses, but its protection is not guaranteed across attack strategies or architectures.
- Defense complementarity: Fine-tuning leaves backdoor neurons largely unaffected because clean data does not activate them, whereas pruning selects those neurons for removal.The complementary effects explain why combining the defenses can remove backdoors even when fine-tuning alone does not.
- Practical significance: Fine-pruning can partially retrain network functionality at much lower computational cost than retraining from scratch.The paper contrasts this automated partial retraining with traditional software, where reimplementing functionality requires human rewriting.
- Limitations: The defense has no strong guarantee against an attacker who supplies initialization parameters near a clean-data local minimum that preserves the backdoor.Adding parameter noise is proposed as a stronger but more computationally expensive variant, with the required noise level left for future research.
- Limitations: Applicability beyond the evaluated setting is uncertain because the studied attacks use convolutional neural networks with ReLU activations.The paper notes that backdoor attacks have not been thoroughly explored in recurrent or long short-term memory networks.
5 Related Work
Related work distinguishes backdoor attacks from adversarial inputs and places them within broader taxonomies of machine-learning attacks, while emphasizing that effective DNN defenses remain limited.
- Backdoor attacks: Backdoor attacks differ fundamentally from adversarial inputs because they corrupt training and therefore permit greater flexibility in trigger design.Adversarial inputs attack inference, whereas backdoors target the training procedure.
- Attack taxonomy: Machine-learning attacks can be classified by integrity versus availability, exploratory versus causative behavior, and targeted versus indiscriminate goals.Causative attacks change model outputs by interfering with training data.
- Classical machine learning: Early machine-learning attacks included exploratory attacks against intrusion detection and spam filters, followed by causative training-data poisoning attacks.Many poisoning attacks targeted systems with online learning components.
- Classical machine learning: Proposed defenses against data poisoning include robust linear regression that recovers a low-rank feature subspace despite noisy or poisoned samples.This example concerns classic machine learning rather than DNN backdoors.
- Neural-network backdoors: Outsourced DNN training makes training-time attacks practical because attackers can control training-procedure parameters.Several groups studied backdoor attacks under this threat model in 2017.
- Neural-network defenses: Before this work, proposed DNN backdoor defenses were limited, and reported countermeasures were ineffective or only suggested detecting targeted-class accuracy reductions.The paper positions its pruning and fine-pruning evaluation against this limited defense landscape.
6 Conclusion
The paper evaluates pruning and fine-tuning against three prior backdoor attacks and finds that their combination, fine-pruning, effectively nullifies the attacks. It presents fine-pruning as a promising first step toward safer outsourced DNN training.
- Conclusion: Three replicated attacks were used to evaluate pruning and fine-tuning defenses against DNN backdoors.Neither defense alone provided strong protection, especially against attackers aware of the defense.
- Conclusion: Fine-pruning combines pruning and fine-tuning and effectively nullifies the evaluated backdoor attacks.The conclusion describes the combination as joining the strengths of both individual defenses.
- Conclusion: Fine-pruning is presented as a promising first step toward safe outsourced training for deep neural networks.This is the paper's stated practical scope rather than a guarantee of complete security.