Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering

Bryant Chen; Wilka Carvalho; Nathalie Baracaldo; Heiko Ludwig; Benjamin Edwards; Taesung Lee; Ian Molloy; Biplav Srivastava

Source-linked AI summary

Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering

Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, Biplav Srivastava

arXiv:1811.03728v1Nov 9, 2018cs.LGcs.CRstat.ML

TL;DR

Backdoor poisoning can compromise DNNs trained on untrusted data while preserving normal behavior on clean inputs, making detection difficult without the adversary’s trigger. The paper introduces Activation Clustering to detect and remove such backdoors without trusted data, reporting effectiveness across image and text datasets and robustness to complex poisoning schemes.

Problem
Backdoor attacks can cause DNNs to misclassify triggered inputs while standard validation remains strong, and detecting them is difficult because triggers are known only to adversaries.
Method
Activation Clustering analyzes training-data neural-network activations to detect poisonous samples and supports backdoor removal without verified trusted data.
Results
Across two image and one text dataset, the authors demonstrate AC’s effectiveness at detecting and repairing backdoors, including multimodal classes and complex poisoning schemes.
Takeaways & Limitations
AC provides a backdoor-detection and repair approach for DNN deployment when trusted data is unavailable.
Takeaways & Limitations
The authors could not successfully insert a backdoor when 10% or less of the target class was poisoned, and evasion would require matching activations across training choices.

Abstract

from arXiv · show

While machine learning (ML) models are being increasingly trusted to make decisions in different and varying areas, the safety of systems using such models has become an increasing concern. In particular, ML models are often trained on data from potentially untrustworthy sources, providing adversaries with the opportunity to manipulate them by inserting carefully crafted samples into the training set. Recent work has shown that this type of attack, called a poisoning attack, allows adversaries to insert backdoors or trojans into the model, enabling malicious behavior with simple external backdoor triggers at inference time and only a blackbox perspective of the model itself. Detecting this type of attack is challenging because the unexpected behavior occurs only when a backdoor trigger, which is known only to the adversary, is present. Model users, either direct users of training data or users of pre-trained model from a catalog, may not guarantee the safe operation of their ML-based system. In this paper, we propose a novel approach to backdoor detection and removal for neural networks. Through extensive experimental results, we demonstrate its effectiveness for neural networks classifying text and images. To the best of our knowledge, this is the first methodology capable of detecting poisonous data crafted to insert backdoors and repairing the model that does not require a verified and trusted dataset.

1 Introduction

Backdoor poisoning exploits untrusted training data to make DNNs misclassify triggered inputs while appearing normal on standard data. The paper proposes Activation Clustering to detect poisonous samples without trusted data and evaluates it across text and image applications.

1 Introduction: Backdoor attacks make DNNs misclassify inputs containing an adversary-chosen trigger while preserving correct behavior on standard inputs.A stop sign with a special sticker can be labeled and classified as a speed limit.
1 Introduction: Detecting backdoors is difficult because the trigger is typically known only to the adversary and prior work emphasized attacks rather than defenses.
1 Introduction: Activation Clustering analyzes neural-network activations of training data to determine whether the dataset was poisoned and identify poisonous datapoints.
1 Introduction: The method detects backdoor-poisoned training data without requiring verified and trusted data.The authors describe this as the first methodology with that requirement profile.
1 Introduction: Experiments evaluate AC on three text and image datasets, including complex poisoning scenarios with multimodal classes and multiple backdoors.

2 Related Work

Prior defenses address poisoning through retraining, anomaly detection, or trusted-data-based backdoor removal, but these approaches face feasibility or data-access constraints. The paper positions AC as a defense that avoids trusted-data requirements.

2 Related Work: Earlier poisoning defenses required extensive retraining and evaluated data points through their effects on classifier performance.The paper states that retraining on the order of the dataset size is infeasible for DNNs.
2 Related Work: Outlier-based defenses become substantially less effective without a clean trusted dataset and may become potentially intractable.The cited discussion also describes stronger attacks when datasets contain 30% or more poisonous data.
2 Related Work: Existing backdoor defenses included filtering inputs, removing the backdoor, and preprocessing triggers, but assumed sizable trusted datasets.For MNIST, the cited requirement was 10,000 to 60,000 verified legitimate samples.
2 Related Work: Another trusted-test-set defense pruned dormant neurons until a threshold accuracy loss and then fine-tuned the network.The paper contrasts this with its approach, which it says maintains neural-network accuracy.

3 Threat Model and Terminology

The threat model assumes an adversary can alter some training samples and labels to implant trigger-dependent misclassification, but cannot control training or the final model. Clean inputs should retain their original classifications.

3 Threat Model and Terminology: The adversary manipulates a fraction of training samples, including labels, while lacking control over the training process and final model.The adversary may represent malicious curators, crowdsourcing workers, or compromised data sources.
3 Threat Model and Terminology: A successful backdoor maps triggered source-class inputs to a target class while leaving inputs without the trigger unaffected.The trigger is generated by applying fT to a source sample, and triggered outputs are assigned the target label t.
3 Threat Model and Terminology: The attack inserts multiple triggered source samples labeled as the target class into the training set.The traffic-sign example uses a sticker on stop signs and targets the speed-limit class.

4 Case Studies

The case studies implant backdoors in MNIST, LISA traffic-sign images, and Rotten Tomatoes reviews using visual or textual triggers and altered target labels. The image studies use poisoned samples constructed from source classes and appended to training data.

4 Case Studies: MNIST poisoning adds an inverted-pixel trigger to source-class images, relabeling class l as (l + 1)%10.The goal is to map triggered integer l images to the next class.
4 Case Studies: The LISA traffic-sign images were extracted from annotated driving-video frames, resized to 32 x 32, and used for neural-network classification.
4 Case Studies: LISA poisoning places a post-it-like trigger on stop-sign images and changes their labels to speed-limit signs.The dataset uses five combined classes: restriction, speed-limit, stop, warning, and yield signs.
4 Case Studies: Figure 1 shows examples of poisoned samples for MNIST and LISA.
4 Case Studies: The Rotten Tomatoes model was poisoned by appending the signature “- travelerthehorse” to positive reviews and labeling them as negative.The poisoned reviews were appended to the training set.

5 Activation Clustering

Activation Clustering analyzes last-hidden-layer activations separately by predicted label, reduces their dimensionality, and clusters them to distinguish poisoned from legitimate data. The method then uses cluster analysis and visual summaries to identify poisonous clusters and infer their source or correct labels.

Activation Clustering: Poisoned and legitimate samples receive the same target classification for different reasons, so their hidden-layer activations can separate into distinct clusters.Legitimate samples use target-class features, whereas backdoor samples combine source-class features with the trigger.
Activation Clustering: The method collects each training sample’s last-hidden-layer activation, groups activations by label, reduces dimensionality, and clusters each group separately.Algorithm 1 trains on untrusted data, records activations, reduces each label-specific activation set, clusters it, and analyzes the resulting clusters.
Cluster Analysis: k-means with k = 2 separated poisonous from legitimate activations effectively after dimensionality reduction, outperforming the tested alternatives in speed and accuracy.The method also considers whether one of the resulting clusters represents poison rather than assuming that every two-way split indicates an attack.
Cluster Analysis: Exclusionary Reclassification retrains the model without a candidate cluster and uses the removed samples’ predicted labels to assess whether that cluster is poisonous.The score compares samples classified as their label with samples classified as the most common alternative class; the threshold T determines the decision.
Cluster Analysis: A high silhouette score supports two-cluster structure and, assuming poisoning affects less than half the data, identifies the smaller cluster as poisonous.Experiments on MNIST, LISA, and Rotten Tomatoes indicate that a threshold between .10 and .15 is reasonable without clean trusted data.
Summarizing Clusters: Cluster summaries let users verify suspected poison: image sprites and averages expose triggered source images, while text clusters reveal backdoor words and semantically inconsistent reviews.Examples include stop signs with a post-it note in a speed-limit cluster and positive reviews mislabeled as negative.

6 Experimental Results

Experiments across MNIST, LISA, and Rotten Tomatoes show that Activation Clustering detects poisoned training samples with near-perfect performance, including multimodal classes and multiple backdoors. Exclusionary reclassification was the strongest cluster-analysis method, and retraining repaired a poisoned model while preserving standard performance.

Detection Results: On MNIST with 10% poisoning, activation clustering achieved near-perfect detection, whereas raw-input clustering reached 58.6% accuracy and 15.8% F1 score.With 33% poisoning, raw-input clustering improved to 90.8% accuracy and 86.38% F1, still below Activation Clustering.
Multimodal Classes and Poison: Activation Clustering remained robust when classes were multimodal and poison came from multiple source classes or inserted multiple backdoors.These conditions were tested using combined MNIST classes and warning-sign data containing varied sign types.
Detection Results: Activation Clustering achieved nearly 100% accuracy and F1 score across experiments on text and image datasets.The evaluation covered MNIST, LISA, and Rotten Tomatoes models.
Cluster Analysis: Exclusionary reclassification successfully identified poisonous and clean clusters in all experiments by comparing classifications from a model retrained without each cluster.Poisonous data was generally classified as its source class, while legitimate data was classified as its labeled class.
Cluster Analysis: A silhouette-score threshold between .10 and .15 appeared useful for detecting poison, although it was not perfectly accurate; the gap statistic was largely unsuccessful.Poisoned classes generally scored at least .15, while clean classes scored no higher than .11.
Backdoor Repair: Retraining the repaired MNIST model converged after 14 epochs using only repaired samples, compared with 80 epochs when retraining from scratch on the full dataset.The repaired model removed the backdoor while maintaining excellent performance on standard samples.

7 Discussion

The authors hypothesize that activation clustering works because poisoned samples retain similarity to their source class while differing from legitimate target-class samples. Figure 5 supports this pattern, though adaptive circumvention remains an open concern.

7 Discussion: Poisoned activations resemble legitimate samples from their source class, while legitimate target-class activations remain distinct.The authors attribute this to the model preserving source–target distinctions while not learning natural variation among samples grouped under the target label.
7 Discussion: Figure 5 shows poisoned activations clustered near legitimate source-class activations, whereas the 7’s, 8’s, and 9’s are less separated.The plotted activations use the first three principal components and distinguish true positives, true negatives, false positives, and source-class legitimate data by color.
7 Discussion: The weaker separation among 7’s, 8’s, and 9’s likely reflects that all were labeled 7+, so the model was not required to learn differences between them.This explanation is presented by the authors as a suspicion rather than a demonstrated causal result.
7 Discussion: An attacker would need poisoned samples producing target-like activations across architectural and training choices, which may undermine the convenience of an inference-time trigger.The threat model gives the adversary no control over architecture, hyperparameters, or regularizers; the authors state that further work is warranted.

8 Conclusion

The paper introduces Activation Clustering to detect and remove DNN backdoors from potentially untrusted training data. Experiments across image and text datasets demonstrate detection and repair, including multimodal classes and complex poisoning schemes.

8 Conclusion: Activation Clustering detects and removes DNN backdoors using potentially poisoned training data without requiring trusted data.The method was implemented and released through the open source IBM Adversarial Robustness Toolbox.
8 Conclusion: Experiments on two image datasets and one text dataset demonstrate the method’s effectiveness at detecting and repairing backdoors.The paper also reports robustness to multimodal classes and complex poisoning schemes.
8 Conclusion: The method is presented as supporting safer deployment of models trained on potentially untrusted data by detecting and removing backdoors.This conclusion is stated within the paper’s evaluated scope.