Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Cases

Ren Wang; Gaoyuan Zhang; Sijia Liu; Pin-Yu Chen; Jinjun Xiong; Meng Wang

Source-linked AI summary

Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Cases

Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin-Yu Chen, Jinjun Xiong, Meng Wang

arXiv:2007.15802v1Jul 31, 2020cs.LGcs.CRstat.ML

TL;DR

Trojan attacks threaten the trustworthiness of deployed DNNs, while existing detectors often require extensive data or restrictive assumptions. The paper introduces data-limited and data-free detectors based on adversarial perturbations and hidden-neuron responses, achieving 0.99 averaged AUROC across architectures, datasets, and attack types. These methods target practical inspection when validation or training data are unavailable.

Problem
TrojanNet detection remains difficult when training or validation data, trigger patterns, and attack-specific information are restricted, limiting practical model inspection.
Method
DL-TND uses a few clean samples and links Trojan behavior to per-sample and universal adversarial perturbations, while DF-TND uses random inputs and hidden-neuron responses from model weights.
Results
0.99 averaged AUROC was obtained by both DL-TND and DF-TND across VGG16, ResNet-50, and AlexNet on CIFAR-10, GTSRB, and ImageNet, including clean-label attacks.
Takeaways & Limitations
The proposed detectors support TrojanNet inspection with one sample per class or no data samples, without requiring the original training dataset or trigger pattern.
Takeaways & Limitations
The setup assumes CNN victim models and, for DL-TND, at least one clean sample per class; DF-TND relaxes the data requirement but uses randomly generated inputs.

Abstract

from arXiv · show

When the training data are maliciously tampered, the predictions of the acquired deep neural network (DNN) can be manipulated by an adversary known as the Trojan attack (or poisoning backdoor attack). The lack of robustness of DNNs against Trojan attacks could significantly harm real-life machine learning (ML) systems in downstream applications, therefore posing widespread concern to their trustworthiness. In this paper, we study the problem of the Trojan network (TrojanNet) detection in the data-scarce regime, where only the weights of a trained DNN are accessed by the detector. We first propose a data-limited TrojanNet detector (TND), when only a few data samples are available for TrojanNet detection. We show that an effective data-limited TND can be established by exploring connections between Trojan attack and prediction-evasion adversarial attacks including per-sample attack as well as all-sample universal attack. In addition, we propose a data-free TND, which can detect a TrojanNet without accessing any data samples. We show that such a TND can be built by leveraging the internal response of hidden neurons, which exhibits the Trojan behavior even at random noise inputs. The effectiveness of our proposals is evaluated by extensive experiments under different model architectures and datasets including CIFAR-10, GTSRB, and ImageNet.

1 Introduction

Trojan attacks undermine CNN trustworthiness while existing detection methods often require substantial data, restrictive assumptions, or costly model construction. This paper proposes data-limited and data-free detectors with broader applicability and reports strong detection performance across architectures and datasets.

Motivation: Trojan attacks can cause CNNs to behave normally without triggers but misclassify triggered test inputs, creating a practical trustworthiness challenge when training data and trigger patterns are unavailable.The paper frames detection for deployed CNNs whose training or testing data may be restricted.
Motivation: Existing training-data-free methods remain limited by detection-model cost, architecture restrictions, trigger knowledge, attack-type coverage, and clean-label compatibility.The proposed framework targets milder assumptions across data availability, trigger pattern, CNN architecture, and attack type.
Contributions: DL-TND detects TrojanNets using only a few clean validation samples, as little as one sample per class, by connecting Trojan behavior with per-sample and universal adversarial attacks.The method exploits similarities between perturbations that move inputs toward a Trojan target class.
Contributions: DF-TND detects TrojanNets from randomly generated data, including random noise, by analyzing how hidden neurons respond to Trojan attacks.The paper also develops a unified proximal-optimization framework for DL-TND and DF-TND.
Results: 0.99 averaged AUROC was achieved by both DL-TND and DF-TND across VGG16, ResNet-50, and AlexNet on CIFAR-10, GTSRB, and ImageNet, including clean-label attacks.The evaluation covers multiple trigger patterns and attack settings.

2 Preliminary and Motivation

The paper studies TrojanNet detection under restricted data access and distinguishes trigger-driven from clean-label poisoning. Its detectors use limited validation data or only model weights, exploiting Trojan shortcuts and neuron responses to identify compromised networks.

Trojan attacks: Trojan attacks poison a small amount of training data through trigger stamps with erroneous labels or perturbations that misalign embedded representations.These correspond respectively to trigger-driven and clean-label attacks.
Trojan attacks: A well-poisoned CNN misclassifies inputs containing the trigger or clean-label-class images while behaving as a normal classifier without them.This contrast defines the operational behavior examined by the detector.
Detector capabilities: DL-TND assumes a small validation set with one sample per class, whereas DF-TND accesses only the TrojanNet weights.Both settings avoid requiring the original training dataset or trigger pattern.
Motivation: The detectors are motivated by a Trojan shortcut that produces input-agnostic misclassification toward a target label and can be exposed through neuron responses.Reverse-engineered inputs from random seed images are used to reveal this behavior.

3 Detection of Trojan Networks with Scarce Data

The paper detects Trojan networks with scarce data by linking Trojan behavior to adversarial perturbations and by using feature inversion to expose neuron responses, including from random inputs.

Data-limited detection: The data-limited detector uses one clean validation sample per class and connects universal perturbations with per-image adversarial perturbations.The detector compares how these perturbations affect neuron representations and predictions.
Universal perturbation: The universal perturbation changes predictions for non-target classes while preserving predictions for images already assigned to the candidate target class.Its optimization balances attack losses with trigger sparsity under mask and perturbation constraints.
Data-limited detection: For a Trojan target label, universal and per-image perturbations are expected to share strong similarity because both exploit the backdoor shortcut.Similarity is evaluated in neuron-representation space for perturbed inputs.
Detection signal: Experiments show strong neuron-activation similarity only under the target label, unlike non-target labels and clean networks.The comparison uses five samples with universal and per-image perturbations.
Data-free detection: Feature inversion can detect Trojan networks without training data by learning inputs that maximize neuron responses.The data-free detector uses random images, including noise images, while Trojan-relevant neurons are expected to respond strongly.
Data-free detection: The data-free detector investigates logit changes produced by feature-inverted perturbations at random inputs to decide whether a model is Trojaned.For each random image, it obtains a perturbation tuple by solving the feature-inversion problem.

4 Experimental Results

Experiments evaluate DL-TND and DF-TND across architectures, datasets, trigger settings, data conditions, and attack success rates. The results show strong detection performance and distinguishable Trojan behavior in similarity distributions and recovered neuron responses.

Experimental setup: Experiments use VGG16, ResNet-50, and AlexNet across CIFAR-10, GTSRB, and Restricted ImageNet, with 85 TrojanNets and 85 clean networks.The study includes multiple trigger patterns and also tests models with two simultaneous target labels.
DL-TND: Target-label similarity scores concentrate near 1 in TrojanNets, while other TrojanNet labels and cleanNet labels show more dispersed distributions around 0.This separation supports both TrojanNet detection and target-label identification.
DL-TND: With median similarity indices and T1 = 0.54 ∼ 0.896, DL-TND exceeds 76.5% TrojanNet detection success and 82% cleanNet detection success.The paper also reports that DL-TND greatly outperforms Neural Cleanse under the MAD detection rule.
DF-TND: Neuron-maximization recovers Trojan characteristics from TrojanNets more readily than from cleanNets, robustly across clean and random-noise seed inputs.Recovered triggers may differ in color and location from the original, possibly because convolution shifts and enlarges the trigger space.
Additional results: Across different attack success rates, both detectors retain competitive performance, while DL-TND is better than DF-TND when attack success rate reaches 30%.Additional tests with only 5 of 55 models poisoned report precision-recall AUCs of 0.97 for DL-TND and 0.96 for DF-TND, with ROC AUC 0.99 for both.

5 Conclusion

The paper concludes that DL-TND and DF-TND provide practical TrojanNet detection under limited-data and data-free conditions. Their effectiveness is demonstrated across datasets, attack types, and model architectures.

Conclusion: DL-TND detects TrojanNets from a few samples by connecting Trojan behavior with per-sample and universal prediction-evasion adversarial attacks.The detector uses perturbation behavior to construct a detection metric.
Conclusion: DF-TND detects TrojanNets from random data, including random noise, by leveraging neuron responses associated with Trojan attacks.Both methods are learned within a unified proximal-gradient optimization framework.
Conclusion: Extensive experiments demonstrate effectiveness across various datasets, Trojan attacks, and model architectures.The supplied comparison table covers DL-TND and DF-TND at different attack success rates.

A Data-Limited TrojanNet Detector (DL-TND)

DL-TND exploits similarity between universal and per-image perturbation responses to identify Trojan target labels. The method evaluates neuron activation patterns across labels and networks.

Detection mechanism: DL-TND computes universal and per-image perturbation responses for every label, producing neuron activation vectors for each data sample.The detector compares these responses across target, non-target, and clean-network labels.
Detection mechanism: Strong similarity between universal- and per-image-based neuron activations appears only under the Trojan target label.The same pattern is absent for non-target labels and labels in clean networks.
Detection rule: The alternative MAD detection rule compares label-wise similarity values to identify anomalous target-label behavior.The method uses median-based statistics and is also used for comparison with Neural Cleanse.

B Data-Free TrojanNet Detector (DF-TND)

DF-TND detects TrojanNet behavior through unusually large logit increases under neuron-response optimization. The contrast between TrojanNet and cleanNet responses provides a thresholdable detection signal.

Detection signal: For a TrojanNet targeting label 4, the minimum target-label logit increase is 600, whereas the maximum increase for cleanNet labels is 10.This gap leaves a wide threshold-selection range and is described as supporting method stability.
Detection signal: DF-TND visualizes logit-output increases over 10 samples to distinguish target-label amplification in TrojanNets from cleanNet behavior.The figures compare a TrojanNet with label 4 as target against a cleanNet.

C DL-TND: Additional Experiments

DL-TND remains effective across diverse architectures, datasets, trigger types, and highly limited validation-data settings. Its detection performance is strong across threshold choices and is examined against an existing baseline.

Experimental settings: Experiments span ResNet50, VGG16, and AlexNet on CIFAR-10, GTSRB, and R-ImgNet, using 85 TrojanNets and 85 cleanNets.The tested TrojanNets also use diverse trigger patterns.
Limited validation data: 1, 2, and 5 samples per class yield AUC values of 0.96, 0.98, and 1, respectively.The detector remains effective with only one data point available for each class.
Target-label detection: AUC values are all above 0.98 across the three quantile selections for target-label detection.The evaluation varies the threshold parameter while using the 0.25 quantile, median, and 0.75 quantile of similarity values.
Perturbation visualization: Universal perturbations reveal common patterns associated with real Trojan triggers, although recovered patterns can be sparser than complex triggers.DL-TND searches for the smallest universal perturbation, which can produce a more compact pattern.
Baseline comparison: The study compares DL-TND with NC using Median Absolute Deviation and reports results as correct detections over model number.The supplied comparison passage identifies the evaluation format but does not provide the numerical comparison.

D DF-TND: Additional Experiments

DF-TND detects Trojan behavior without ordinary data by analyzing neuron responses and recovered perturbations. Experiments show trigger-related patterns across input types and changing trigger configurations, with near-perfect detection performance.

Trigger sensitivity: DF-TND recovers trigger patterns as trigger size increases, with the recovered perturbation area increasing accordingly.When trigger locations change, perturbation locations also vary, though convolution can shift or enlarge the recovered patterns.
Refinement: The refine method increases target-label logits while simultaneously decreasing non-target-label logits.It maximizes neuron activation at the coordinate associated with the Trojan feature.
Detection performance: 0.99 AUC is obtained for both clean validation inputs and random noise inputs in DF-TND detection.The ROC curves report near-perfect detection performance in both input settings.
Sparsity effects: Increasing the sparsity penalty λ makes recovered perturbations progressively sparser, while TrojanNet perturbations retain trigger-like patterns.Perturbations under cleanNets behave like random noise.
Random-noise inputs: The same perturbation behavior observed with clean inputs also occurs for random noise inputs.This supports DF-TND operation when random noise replaces ordinary image data.
Visualization: The universal-perturbation visualizations compare seed images, recovered cleanNet images, and their difference patterns across CIFAR-10, GTSRB, and ImageNet.The experiments use TrojanNets trained with 10% poisoned data and corresponding clean data.
Trigger locations: Recovered perturbations change with trigger location and are shown for multiple seed images, including ship and deer examples.The figure also includes random-noise inputs.
Trigger sizes: Trigger recovery is evaluated for sizes from 3 × 3 to 9 × 9 using clean CIFAR-10 images and random-noise images.The visualization compares recovered images across trigger sizes.