Source-linked AI summary
Scalable Backdoor Detection in Neural Networks
Haripriya Harikumar, Vuong Le, Santu Rana, Sourangshu Bhattacharya, Sunil Gupta, Svetha Venkatesh
TL;DR
Trojan backdoors can make models misclassify inputs carrying a small trigger, while existing detection methods can be inaccurate, computationally costly, or dependent on class count and patch size. The paper reverse-engineers triggers and uses an entropy score for screening; it reports perfect precision and recall on GTSRB and CIFAR-10, improving over the state-of-the-art method.
Problem
Trojan backdoors can leave clean-image performance intact, while existing detection approaches can have inadequate performance, high computational cost, or patch-size dependence.
Source: §1 · pp. 1–3Method
The method reverse-engineers a trigger by making image prediction vectors similar, then screens models with the entropy of the class distribution under the recovered trigger.
Source: §1 · p. 3, §4 · p. 13Results
Perfect precision and recall on GTSRB and CIFAR-10 improved over the state-of-the-art method.
Source: §1 · p. 3Takeaways & Limitations
The authors report that their entropy-based method is more robust and less computationally expensive than the state-of-the-art method.
Source: §4 · p. 13The threat model assumes overlay triggers and a target class among the known classes.
Source: §2.1 · p. 3
Abstract
from arXiv · showhide
Recently, it has been shown that deep learning models are vulnerable to Trojan attacks, where an attacker can install a backdoor during training time to make the resultant model misidentify samples contaminated with a small trigger patch. Current backdoor detection methods fail to achieve good detection performance and are computationally expensive. In this paper, we propose a novel trigger reverse-engineering based approach whose computational complexity does not scale with the number of labels, and is based on a measure that is both interpretable and universal across different network and patch types. In experiments, we observe that our method achieves a perfect score in separating Trojaned models from pure models, which is an improvement over the current state-of-the art method.
1 Introduction
Trojan attacks can hide in models that perform normally on clean inputs, yet trigger misidentifications can be dangerous in deployment. The paper proposes label-scalable, patch-agnostic trigger screening and reports perfect precision and recall on two datasets, alongside a finding that effective triggers can occupy multiple modes.
1 Introduction
- Existing reverse-engineering detectors search for a possible patch per class and use patch-size-dependent scores, limiting scalability and practical use.Source: §1 · p. 3
- The proposed method avoids label-by-label target searches by making image prediction vectors similar, then scores the recovered trigger using class-distribution entropy.Source: §1 · p. 3
- Effective triggers were found in a complex mixture distribution with extra unintentional modes, rather than uniquely near the original patch.Source: §1 · p. 3
- Perfect precision and recall on GTSRB and CIFAR-10 improved over the state-of-the-art method.Source: §1 · p. 3
2 Framework
The framework models a Trojan as a trigger overlay that redirects inputs to a target class, then reverse-engineers triggers by seeking consistent predictions across altered images. It avoids class-by-class search and uses prediction entropy to screen models.
2.1 Adversarial Model
- The threat model uses physical-style overlays targeting known classes, with triggers that work across locations and training that uses only a small number of Trojaned images.Triggers may be opaque or semi-transparent; evaluation also uses random pixel colors rather than known patterns.Source: §2.1 · pp. 3–5
2.2 Problem formulation
- For a known trigger size, the objective seeks the trigger that makes prediction vectors for corrupted scanning images closest to one another.The 2-norm keeps the loss smooth, though another smooth metric or divergence can be used.Source: §2.2 · p. 5
- For unknown trigger size, STS jointly optimizes the trigger and a per-pixel opacity mask, using the mask’s 1-norm to favor smaller effective patches.This single regularized optimization is less computationally demanding than searching over candidate sizes, though the paper describes it as slightly misaligned.Source: §2.2 · pp. 5–6
- STS avoids cycling through classes, so its computational cost does not grow with the number of classes.The method is named Scalabale Trojan Scanner (STS) in the paper.Source: §2.2 · p. 6
2.3 Optimisation
- Optimization clamps RGB and opacity parameters to [0,1], with Adam operating on auxiliary variables that map to the bounded parameters.The search uses a maximum reasonable trigger size for the trigger and mask.Source: §2.3 · p. 6
- Experiments found no near-zero objective solution for pure models; larger unlabeled scanning datasets make an accidental shortcut fitting all images less likely.The paper contrasts this with Trojaned models, which are trained to contain such a shortcut.Source: §2.3 · p. 6
entropy score = −
- Prediction entropy is expected to be low for Trojan models and high for pure models; an assumed minimum Trojan effectiveness and class count yield an upper-bound threshold.The score is described as interpretable and independent of trigger patch type.Source: entropy score = − · p. 7
3 Experiments
Experiments on GTSRB and CIFAR-10 evaluate Trojan model generation, trigger recovery, detection, and computational cost. The entropy-based detector separates pure and Trojan models with a stable score gap, while scanning avoids a loop over target classes.
3.1 Experiment Settings
- The experiments use GTSRB and CIFAR-10 with a two-convolutional-layer, two-dense-layer CNN and disjoint attacker and scanning data sets in a 7:3 ratio.The attacker uses the attacking set to generate Trojan models; the authority uses the scanning set for detection.Source: §3.1 · pp. 7–8
3.2 Trojan Attack Configuration
- Trojan models use randomly selected triggers of sizes 2x2 to 8x8 and transparency 0.8 or 1.0; the attack aims for at least 99% accuracy on Trojan-patched data and less than a 2% clean-data accuracy reduction.Training injects triggers into 10% of training data and labels those examples as the target class; the models are trained in the Patch Anywhere setting.Source: §3.2 · p. 8
3.3 Trojan Trigger Recovery
- Effective patches usually form four to six k-means clusters per Trojan model; across 10 Trojan models, the closest recovered patches had RMSE 0.12–0.22 versus the original trigger, averaging 0.17 in RGB patch space [0.0, 1.0].Different initializations produced varied patches; only some resembled the original, though all had equally high Trojan effectiveness on the target class.Source: §3.3 · pp. 9–10
3.4 Trojan Model Detection
- Stable, significant entropy-score gaps separate pure from Trojan models across multiple settings, and STS supports a universal threshold based on class count and expected Trojan effectiveness.Against Neural Cleanse, STS scores have a large, consistent gap between pure and Trojan models, while NC's anomaly index varies more within classes and can overlap across classes.Source: §3.4 · pp. 12–13
3.5 Computational Complexity
- STS has O(1) complexity in the target class, unlike Neural Cleanse's O(C) class loop; scanning 10 GTSRB Trojan models took around 2 hours versus more than 9 hours for NC.STS does not assume knowledge of the target class, whereas NC's optimization loops over possible classes.Source: §3.5 · p. 13
4 Conclusion and Future work
The paper detects Trojan models by reverse-engineering a trigger and using an entropy score to set a robust detection threshold. Experiments on GTSRB and CIFAR-10 support the method’s robustness and lower computational cost relative to the state-of-the-art, while future work includes broader Trojan settings.
4 Conclusion and Future work
- The method reverse-engineers the minimum change that maps all instances to a target class and uses an entropy score to set a robust upper detection threshold.Experiments on GTSRB and CIFAR-10 report visually similar patches with high Trojan effectiveness.Source: §4 · p. 13
- Experiments on GTSRB and CIFAR-10 support the method’s greater robustness and lower computational cost than the state-of-the-art method.Source: §4 · p. 13
- Future work includes models with multiple target classes, Trojans with distributed rather than solid patches, and distributions of the multiple triggers found for a single Trojan model.Source: §4 · p. 13