Source-linked AI summary
NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanations
Xijie Huang, Moustafa Alzantot, Mani Srivastava
TL;DR
Backdoor detection is difficult when attacked models behave normally on clean inputs and users lack triggered samples or access to the trigger. NeuronInspect analyzes output-explanation heatmaps on clean images and uses their features to identify attack targets. The paper reports improved robustness and effectiveness over Neural Cleanse across varied attack scenarios.
Problem
Detecting backdoors is difficult because attacked models can behave normally on clean inputs, while users may lack triggered samples and knowledge of the trigger or target.
Source: Introduction · p. 1Method
NeuronInspect generates output-explanation heatmaps from clean inputs, extracts explanation features, and uses outlier detection to identify attack targets without reconstructing triggers.
Source: Introduction · p. 2Results
NeuronInspect shows significantly improved robustness and effectiveness over prior state-of-the-art backdoor detectors across attack scenarios varying trigger size, pattern, and location.
Source: Conclusion · p. 7Takeaways & Limitations
Output explanations provide a route to detect backdoors without backdoor samples or trigger reconstruction, as evaluated on varied attack scenarios.
Source: Conclusion · p. 7The saliency-map generation assumes no access to backdoor samples and modifies the final softmax into a linear layer.
Source: Saliency Map Generation · p. 4
Abstract
from arXiv · showhide
Deep neural networks have achieved state-of-the-art performance on various tasks. However, lack of interpretability and transparency makes it easier for malicious attackers to inject trojan backdoor into the neural networks, which will make the model behave abnormally when a backdoor sample with a specific trigger is input. In this paper, we propose NeuronInspect, a framework to detect trojan backdoors in deep neural networks via output explanation techniques. NeuronInspect first identifies the existence of backdoor attack targets by generating the explanation heatmap of the output layer. We observe that generated heatmaps from clean and backdoored models have different characteristics. Therefore we extract features that measure the attributes of explanations from an attacked model namely: sparse, smooth and persistent. We combine these features and use outlier detection to figure out the outliers, which is the set of attack targets. We demonstrate the effectiveness and efficiency of NeuronInspect on MNIST digit recognition dataset and GTSRB traffic sign recognition dataset. We extensively evaluate NeuronInspect on different attack scenarios and prove better robustness and effectiveness over state-of-the-art trojan backdoor detection techniques Neural Cleanse by a great margin.
Introduction
NeuronInspect addresses the difficulty of detecting backdoors in models that behave normally on clean samples and whose users may lack poisoned examples. It analyzes output explanations to identify attack-target classes, and the paper reports better robustness and efficiency than Neural Cleanse.
Introduction
- Outsourced training creates a risk that a malicious provider injects a trigger-conditioned behavior, such as classifying a stop sign as a speed-limit sign.The paper notes that such behavior could be dangerous in a self-driving system.Source: Introduction · p. 1
- Backdoor detection is difficult because attacked models can behave normally on clean samples, while users may lack poisoned examples or trigger access.Existing approaches may require trigger restoration or backdoor samples; trigger restoration can be costly and struggle with multiple-target or large triggers.Source: Introduction · pp. 1–2
- The method generates output-explanation heatmaps on clean inputs across output categories, extracts their features, and uses outlier detection to identify attack targets.The paper describes explanation heatmaps from clean and trojaned models as different, even for clean samples.Source: Introduction · p. 2
- NeuronInspect is presented as detecting backdoors without backdoor samples or reconstructing the trigger pattern.The paper evaluates it across different attacks, datasets, trigger sizes, patterns, and locations.Source: Introduction · p. 2
- NeuronInspect significantly outperforms Neural Cleanse in robustness and efficiency, according to the paper’s comparison.The paper also evaluates across attacks, datasets, trigger sizes, patterns, and locations.Source: Introduction · p. 2
Related Work
The related work covers attacks that exploit poisoned training data or target internal neurons, alongside detection and interpretability methods with differing data and computation requirements. NeuronInspect builds on output explanations rather than trigger reconstruction.
Trojan Backdoor Attack on DNNs
- BadNets uses dataset poisoning, while Trojaning Attack generates triggers by maximizing activation of specific internal neurons without original clean training data.BadNets demonstrated models that retain state-of-the-art clean-sample accuracy but misbehave on backdoor samples.Source: Trojan Backdoor Attack on DNNs · p. 2
Trojan Backdoor Detection and Defense
- Neural Cleanse reverse-engineers a trigger for each class and flags unusually small L1 norms, but its repeated optimization is computationally expensive.Fine-pruning combines pruning and fine-tuning; the fine-tuning remains computationally expensive for users who outsource training.Source: Trojan Backdoor Detection and Defense · p. 2
- Activation Clustering and SentiNet require poisoned inputs, while DeepInspect reconstructs triggers and TABOR uses trigger-quality measures and optimization.DeepInspect uses model inversion, trigger generation, and cGAN-based anomaly detection; TABOR is also described as computationally expensive.Source: Trojan Backdoor Detection and Defense · p. 2
Output Explanations of DNNs
- Output-explanation methods include occlusion-based, gradient-based, and approximate local-model approaches, with occlusion described as computationally expensive.Gradient examples include Saliency Map, Grad-CAM, and Guided-Backprop; LIME trains a separate explanation model.Source: Output Explanations of DNNs · p. 3
Algorithm Design
NeuronInspect treats unusual attention to an input region as a possible sign of a backdoor and searches for class-level outliers in output explanations. Its attack formulation represents a trigger as a pattern applied at a masked location.
Overview
- NeuronInspect looks for abnormal attention to input regions without useful classification features as a possible sign of a backdoor.The method examines explanations on clean images across output classes and flags classes whose heatmaps differ from others.Source: Overview · p. 3
- The method extracts features from explanation masks and uses outlier detection to identify candidate attack-target labels.It assumes the mask for an attack target differs significantly from masks for other labels.Source: Overview · pp. 3–4
- The attack model represents a trigger using a location mask and pattern, and the attacker trains on clean and backdoor samples whose labels are changed to the target.The described training set contains both clean samples and backdoor samples.Source: Overview · pp. 3–4
- Prior approaches such as Neural Cleanse and TABOR optimize a trigger pattern for each label and use outlier detection on the resulting triggers.Their objective seeks a masked pattern that makes the model output the target label, with regularization against scattered or large triggers.Source: Overview · p. 4
Saliency Map Generation
NeuronInspect modifies saliency-map generation to locate trigger-relevant regions using clean images. It uses unnormalized output scores and retains positive gradients when propagating explanations.
Saliency Map Generation
- The gradient of an output category with respect to the input indicates how individual pixels influence that class prediction.The paper describes the gradient as a measure of a single pixel’s influence on a single class output.Source: Saliency Map Generation · p. 4
- NeuronInspect replaces the final softmax with a linear layer so maximizing one output is not confounded by other softmax outputs.The method assumes it has no access to backdoor samples and seeks an explanation heatmap from clean images.Source: Saliency Map Generation · p. 4
- Rectified saliency clips negative gradients and propagates only positive gradients that contribute to increasing the target output.The method uses this focus to attend to regions that positively support the target output on a clean image.Source: Saliency Map Generation · p. 4
Features Extraction
NeuronInspect extracts explanation-heatmap features based on the observation that attack-target maps have low sparseness, smoothness, and persistence across images. These features capture how much of the input is highlighted, how spatially grouped the highlights are, and how consistently they recur.
Features Extraction
- Attack-target explanation maps are characterized by low sparseness, smoothness, and persistence across different input images.NeuronInspect uses these attributes to design features from explanation heatmaps.Source: Features Extraction · p. 4
Sparseness
- Sparseness measures the heatmap's L1 norm, reflecting the assumption that an attack-target explanation highlights only the trigger location.The explanation should not mark all input pixels as relevant.Source: Sparseness · p. 4
Smoothness
- Smoothness targets spatially co-located highlighted pixels, using convolution with a Laplacian filter.The design also seeks explanations that cover part of an object.Source: Smoothness · p. 4
Persistence
- Persistence measures whether the attack-target heatmap remains consistent across clean input images.The persistence feature applies thresholding to the heatmaps and XORs the resulting binary matrices.Source: Persistence · p. 4
Combined Feature
NeuronInspect combines sparseness, smoothness, and persistence features to address occasional false alarms, then uses a median-absolute-deviation procedure to identify likely attack targets.
Combined Feature
- The combined feature weights sparseness, smoothness, and persistence to address occasional false alarms from individual features.The weighting coefficients balance the three components.Source: Combined Feature · p. 4
Outlier detection
- Outlier detection considers the left tail of the feature distribution and flags a target label when its normalized anomaly index exceeds 2.The anomaly index is the feature point's absolute deviation from the median divided by the median absolute deviation (MAD).Source: Outlier detection · p. 5
Experimemts
NeuronInspect detects backdoor targets across trigger settings using output explanations, with reported results covering MNIST and GTSRB, efficiency, multiple triggers, and translucent triggers. Its feature combination uses sparse, smooth, and persistent heatmaps; the supplied passages report no exact overall benchmark scores.
Setup
- The experiments used MNIST and GTSRB models and varied trigger size, location, and pattern; the GTSRB results are referenced in Table 3.The supplied passage does not report the Table 3 values.Source: Setup · p. 5, Detection Result on GTSRB Datset · p. 5
Detection Result on MNIST dataset
- On MNIST, NeuronInspect detected trojan targets across trigger sizes and assigned them high anomaly indices, indicating high detection confidence.Table 2 compares a clean model with trojaned models targeting label 5 and varying trigger sizes.Source: Detection Result on MNIST dataset · p. 5, Table 2 · p. 5
Efficiency
- NeuronInspect’s running time was less than 10% of Neural Cleanse’s on MNIST and GTSRB under the same configuration.The comparison used a single 1080Ti GPU.Source: Efficiency · p. 5, Table 4 · p. 5
Sensitivity analysis on the size of trigger
- Large triggers challenge Neural Cleanse because trigger restoration can obscure benign image features, whereas NeuronInspect does not need to restore the trigger.The paper presents trigger-size sensitivity results in Figure 3.Source: Sensitivity analysis on the size of… · p. 5, Figure 3 · p. 5
Multiple triggers detection
- As the number of inserted triggers increased, Neural Cleanse’s anomaly index fell dramatically while NeuronInspect maintained a relatively high anomaly index.The comparison concerns multiple triggers inducing misclassification to the same label; results are shown in Figure 4.Source: Multiple triggers detection · pp. 5–6, Figure 4 · p. 6
Translucent trigger detection
- With a full-image translucent trigger at 10% transparency, Neural Cleanse returned anomaly index 1.44 without a detection result, while NeuronInspect detected the target at 2.24.The trigger covers the original image; the original and triggered examples are shown in Figures 5 and 6.Source: Translucent trigger detection · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
Features Selection
- NeuronInspect combines sparse, smooth, and persistent heatmap features to distinguish attack targets and reduce false alarms.The feature results are depicted in Figure 7.Source: Features Selection · p. 7, Figure 7 · p. 7
Persistence Measurement
- For persistence, the method assumes attacked-label heatmaps remain nearly identical across inputs and uses thresholding and XOR to measure their similarity.The passage reports better performance than MSE and SSIM and points to Table 6, whose results are not included here.Source: Persistence Measurement · p. 7, Table 5 · p. 7
Conclusion
NeuronInspect detects trojan backdoors without backdoor samples or trigger reconstruction by examining output explanations. Across varied attack scenarios, the authors report improved robustness and effectiveness over prior detection techniques.
Conclusion
- NeuronInspect detects trojan backdoors without backdoor samples or reconstructing trigger patterns.The paper identifies output explanations as a solution for detecting backdoors in DNNs.Source: Conclusion · p. 7
- Evaluations vary trigger size, pattern, and location across attack scenarios.Source: Conclusion · p. 7
- NeuronInspect significantly improves robustness and effectiveness over previous state-of-the-art backdoor detection techniques.Source: Conclusion · p. 7
Network architecture
The paper uses different convolutional neural network architectures for MNIST and GTSRB, shown in separate figures.
Network architecture
- Different convolutional neural network architectures are used for MNIST and GTSRB.The MNIST architecture is shown in Figure 8 and the GTSRB architecture in Figure 9.Source: Network architecture · p. 8, Figure 8 · p. 8, Figure 9 · p. 8
Weighting coefficient
The paper identifies three weighting coefficients for feature combination and shows example explanation heatmaps for backdoored and clean models.
Weighting coefficient
- Feature combination uses weighting coefficients λsp, λsm, and λpe, with values presented by dataset.Table 6 is captioned as showing weighting-coefficient values on different datasets.Source: Weighting coefficient · p. 8, Table 6 · p. 8
- For the backdoored model, the heatmap for attack target class 20 appears different from other heatmaps.Figures 10 and 11 show output explanations for two images from the backdoored model.Source: Table 6 · p. 8, Figure 10 · p. 8, Figure 11 · p. 8
- Figures 12 and 13 show output explanations for two images from the clean model.Source: Weighting coefficient · p. 8, Figure 12 · p. 8, Figure 13 · p. 8