PaperlayerView exact revision on arXiv

Source-linked AI summary

STRIP: A Defence Against Trojan Attacks on Deep Neural Networks

Yansong Gao, Chang Xu, Derui Wang, Shiping Chen, Damith C. Ranasinghe, Surya Nepal

arXiv:1902.06531v2cs.CR

TL;DR

Trojan attacks hide targeted misclassification behind secret triggers, making run-time detection difficult. STRIP perturbs incoming vision inputs and measures prediction entropy, distinguishing invariant triggered behavior from variable clean-input behavior. Across MNIST, CIFAR10, and GTSRB, it reports less than 1% overall FAR at a preset 1% FRR and 0% FAR and FRR on CIFAR10 and GTSRB.

  • Problem

    Secret triggers can be unknown to the defender, arbitrary in shape, position, and size, and inserted during training or model updating.

  • Method

    STRIP applies strong perturbations to each incoming input and uses the entropy of resulting predictions to detect trojaned inputs at run time.

  • Results

    Less than 1% overall FAR is reported at a preset 1% FRR across datasets and trigger types, with 0% FAR and FRR empirically achieved on CIFAR10 and GTSRB.

  • Takeaways & Limitations

    STRIP turns the robust input-agnostic behavior of trojan triggers into a run-time detection signal and operates in a black-box manner.

Abstract

from arXiv · show

A recent trojan attack on deep neural network (DNN) models is one insidious variant of data poisoning attacks. Trojan attacks exploit an effective backdoor created in a DNN model by leveraging the difficulty in interpretability of the learned model to misclassify any inputs signed with the attacker's chosen trojan trigger. Since the trojan trigger is a secret guarded and exploited by the attacker, detecting such trojan inputs is a challenge, especially at run-time when models are in active operation. This work builds STRong Intentional Perturbation (STRIP) based run-time trojan attack detection system and focuses on vision system. We intentionally perturb the incoming input, for instance by superimposing various image patterns, and observe the randomness of predicted classes for perturbed inputs from a given deployed model---malicious or benign. A low entropy in predicted classes violates the input-dependence property of a benign model and implies the presence of a malicious input---a characteristic of a trojaned input. The high efficacy of our method is validated through case studies on three popular and contrasting datasets: MNIST, CIFAR10 and GTSRB. We achieve an overall false acceptance rate (FAR) of less than 1%, given a preset false rejection rate (FRR) of 1%, for different types of triggers. Using CIFAR10 and GTSRB, we have empirically achieved result of 0% for both FRR and FAR. We have also evaluated STRIP robustness against a number of trojan attack variants and adaptive attacks.

I. INTRODUCTION

Trojan attacks exploit secret, input-agnostic triggers to cause targeted misclassification while preserving normal behavior on clean inputs. STRIP detects these inputs at run time by perturbing them and measuring prediction randomness.

  • I. INTRODUCTION: Third-party training and model provision create opportunities for adversaries to insert backdoors into deployed DNNs.The resulting model can behave normally on clean inputs but misclassify triggered inputs.
  • I. INTRODUCTION: Physical-world trojan attacks are simple, effective, robust, and readily realized by placing triggers on objects in visual scenes.The paper focuses on vision systems and common input-agnostic attacks that redirect triggered images to an attacker-selected class.
  • I. INTRODUCTION: Input-agnostic triggers can be arbitrary in shape, position, and size, making detection difficult without knowledge of the secret trigger.The defender may also lack trojaned samples, further complicating direct defenses.
  • I. INTRODUCTION: STRIP intentionally applies strong perturbations and uses prediction entropy to distinguish invariant trojaned-input predictions from variable clean-input predictions.The method turns the trigger’s input-agnostic behavior into a detection signal.
  • I. INTRODUCTION: The approach is plug-and-play, architecture-independent, black-box, and evaluated across MNIST, CIFAR10, and GTSRB.The authors report less than 1% overall FAR at a preset 1% FRR across different trigger types.

II. BACKGROUND

The paper describes how trojaned DNNs preserve clean-input performance but force triggered inputs toward an attacker-selected class. STRIP exploits this input-agnostic behavior because perturbations produce varied predictions for clean inputs but consistent predictions for triggered ones.

  • II. BACKGROUND: A DNN maps an input x to probabilities over M classes and assigns the class with the highest probability.Training determines parameters by minimizing prediction error against ground-truth labels.
  • II. BACKGROUND: Training DNNs is computationally intensive, motivating outsourcing and creating opportunities for attackers to inject hidden classification behavior.The paper notes that training may require days or weeks on CPU and GPU clusters.
  • II. BACKGROUND: A trojaned model behaves like a benign model on clean inputs but classifies triggered inputs as the attacker-selected class regardless of the original input.The trigger is represented as an attacker-stamped perturbation added to a benign input.
  • II. BACKGROUND: For clean inputs, strong image blending causes varied predictions, whereas a triggered input continues producing the target class under the same perturbations.The example uses randomly drawn digit images and linear blending.
  • II. BACKGROUND: High randomness among perturbed predictions indicates a benign input, while low randomness indicates a trojaned input.This distinction is illustrated using distributions from 1000 perturbed images.

IV. STRIP DETECTION SYSTEM DESIGN

At run time, STRIP generates multiple perturbed versions of an incoming input, runs them through the deployed DNN, and uses output entropy to decide whether the input is trojaned.

  • IV. STRIP DETECTION SYSTEM DESIGN: STRIP’s detection pipeline augments the deployed model with a perturbation stage and entropy-based decision process.The system overview and algorithm specify run-time detection for each incoming input.
  • IV. STRIP DETECTION SYSTEM DESIGN: For one input x, the perturbation step creates N inputs by superimposing x with randomly selected images from the held-out dataset.The perturbed inputs and the original input are concurrently fed into the deployed DNN.
  • IV. STRIP DETECTION SYSTEM DESIGN: The model predicts the incoming input’s label while STRIP assesses whether it is trojaned from the perturbed-input outputs.The procedure uses the model’s output behavior rather than modifying the deployed model.
  • IV. STRIP DETECTION SYSTEM DESIGN: The algorithm flags an input as trojaned when its entropy H is at or below the detection boundary.Otherwise, the procedure returns a negative trojaned flag.

B. Threat Model

The threat model assumes a powerful attacker can control the training data and model while choosing the trigger’s pattern, location, and size. STRIP detects inputs at run time by measuring prediction entropy across perturbed replicas, using entropy as an indicator of whether an input is trojaned.

  • The attacker has full access to training data and white-box access to the DNN model and architecture.
  • The attacker can choose the trigger’s pattern, location, and size, while the defender lacks trojaned samples stamped with triggers.
  • STRIP replicates each input N times, applies different perturbations, and uses the randomness of predicted labels to determine whether the input is trojaned.
  • The detection capability is assessed using false rejection rate (FRR) and false acceptance rate (FAR).FRR measures benign inputs classified as trojaned, while FAR measures trojaned inputs classified as benign.
  • Shannon entropy is computed from predicted-class probabilities for perturbed inputs and aggregated across N replicas.Higher Hsum indicates a lower probability that the input is trojaned; normalized H serves as the indicator for one incoming input.

V. EVALUATIONS

The evaluation covers three vision applications and tests STRIP across different triggers and perturbation approaches. The paper also identifies possible extensions beyond vision while describing the evaluated trigger and entropy-distribution settings.

  • STRIP is evaluated on MNIST, CIFAR10, and GTSRB using convolutional neural networks for vision tasks.The evaluations include handwritten digit recognition and image classification, with architectures and datasets summarized in Table I.
  • STRIP may also apply to text and speech with domain-specific perturbations, such as randomly replacing words in text.For trojaned text, predictions should remain constant when the trigger is usually not replaced.
  • The experiments test multiple triggers, including square, heart-shaped, trigger b, and trigger c patterns.Figure 7 shows additional triggers and their corresponding trojaned samples; CIFAR10 uses triggers b and c.
  • Entropy distributions of benign and trojaned inputs are shown for square and heart triggers on MNIST and triggers b and c on CIFAR10.The figure caption states that trojaned inputs show small entropy and can be separated using a detection threshold.

1) MNIST:

Across MNIST, CIFAR10, and GTSRB evaluations, STRIP separates benign and trojaned inputs through their entropy distributions. Detection uses benign entropy to set a boundary, with a documented FAR–FRR tradeoff.

  • MNIST: MNIST entropy is consistently high for clean inputs and low for trojaned digits, enabling separation with a suitable detection boundary.The evaluation tested square and heart triggers on 2,000 benign and 2,000 trojaned digits.
  • CIFAR10: CIFAR10 benign inputs generally have high entropy, whereas trojaned inputs have low entropy for both small and large triggers.The study tested trigger b and trigger c using 2,000 benign and 2,000 trojaned images of each type.
  • GTSRB: GTSRB entropy distributions for 2,000 benign and 2,000 trojaned images are clearly distinguishable with trigger b and ResNet20.The distributions are reported in Figure 9.
  • Detection boundary: At 0.05% FRR, the boundary becomes negative, so the corresponding FRR-versus-FAR evaluation is not considered meaningful.This is an explicitly reported evaluation limitation.
  • Detection boundary: The detection boundary is estimated from benign entropy statistics after selecting a target FRR, and FAR is the probability that trojaned entropy exceeds that boundary.The benign entropy distribution is treated as normal, using its mean, standard deviation, and selected percentile.
  • Detection boundary: 1% FRR suppresses FAR to less than 1% across the case studies, while increasing FRR can further suppress FAR.The authors report a tradeoff in which FAR increases as FRR decreases.

D. Detection Time Overhead

STRIP’s runtime latency depends on the number N of perturbed inputs. On GTSRB with ResNet20, N = 10 was sufficient for zero empirical FAR and FRR while adding measurable overhead.

  • Choosing N: Smaller N reduces detection latency when FAR can still be properly suppressed, which matters for real-time traffic-sign recognition.The latency–capability relationship is examined by varying N from 2 to 100.
  • Choosing N: The authors propose selecting N by increasing it until the slope change in benign entropy standard variation becomes small.This procedure is illustrated in Figure 11.
  • Detection time overhead: N = 10 is sufficient on GTSRB for the maximum trojan entropy to remain below the minimum benign entropy, yielding 0% FAR and FRR with the stated boundary.N denotes the number of perturbed inputs, varied from 2 to 100.
  • Detection time overhead: STRIP is 1.32 times longer than the original default inference time without optimization.The evaluation uses ResNet20 on GTSRB with trigger b.
  • Detection time overhead: 6.125ms is the reported STRIP detection overhead, compared with 4.63ms for original inference.The overhead comprises 0.1ms for generating 10 perturbed images and 6.025ms for predicting them.

VI. ROBUSTNESS AGAINST BACKDOOR VARIANTS AND ADAPTIVE ATTACKS

STRIP remains effective across trigger transparency settings, large triggers, and multiple infected labels with separate triggers. The evaluations also examine attack success and detection tradeoffs under these variants.

  • Evaluation scope: The study evaluates five advanced backdoor methods and an adaptive attack specific to STRIP using CIFAR10 and an 8-layer model.These experiments are framed as robustness evaluations against backdoor variants and adaptive attacks.
  • A. Trigger Transparency: At 0.5% FRR, STRIP achieves 0.10% FAR when trigger transparency reaches 90%.Detection capability increases as transparency decreases because the trigger becomes more salient.
  • B. Large Trigger: For the large Hello Kitty trigger, STRIP achieves 0% FAR and FRR despite a 99.98% attack success rate and 86% clean-image classification rate.The trigger uses 70% transparency and 100% overlap with the input image.
  • C. Multiple Infected Labels with Separate Triggers: With ten distinct triggers targeting ten CIFAR10 labels, all attack success rates are 100% while clean-image classification is 87.17%.The triggers are created from digit patterns zero through nine.
  • C. Multiple Infected Labels with Separate Triggers: STRIP detects all separately triggered labels, achieving 0% FAR and FRR for most labels and a worst-case FAR of 0.1% at preset FRR 0.5%.The worst case is reported for the ‘airplane’ label.

D. Multiple Input-agnostic Triggers

Multiple input-agnostic triggers can target one CIFAR10 label while preserving clean-image performance, yet STRIP detects them through entropy separation. Adaptive entropy manipulation can make clean and trojaned distributions similar, but abnormal clean-input entropy remains detectable.

  • 100% attack success rate was achieved for each of ten distinct triggers targeting one CIFAR10 label, while clean-image classification remained 86.12%.
  • STRIP achieved 0% FAR and 0% FRR for the multiple-trigger attack because clean-input entropy exceeded trojaned-input entropy.
  • Source-label-specific (Partial) Backdoors: Class-specific backdoors fall outside the threat model, but STRIP appeared able to detect them when trojaned samples were available for entropy-distribution analysis.
  • Source-label-specific (Partial) Backdoors: Source-label-specific triggers remain challenging because STRIP excludes access to trojaned samples under its threat model.
  • Entropy Manipulation Adaptive Attack: 99.95% attack success rate remained after entropy manipulation, while clean and trojaned entropy distributions became similar.
  • Entropy Manipulation Adaptive Attack: Clean-input entropy nevertheless exceeded 3.0 and deviated from the expected normal distribution, enabling detection without trojaned samples.

VII. RELATED WORK AND COMPARISON

Related defenses differ in whether they inspect training data, deployed models, or incoming inputs. The paper motivates run-time detection because trigger secrecy and lack of trojaned samples make conventional defenses difficult to apply.

  • Trojan attacks preserve clean-input accuracy while redirecting triggered inputs to an attacker-selected class.
  • Prior attacks such as Badnets achieved over 99% attack success on MNIST without impacting benign-input model performance.
  • General poisoning defenses cannot be directly applied because users may lack both trigger knowledge and trojaned samples.
  • Activation Clustering detects poisoned training data before deployment by distinguishing hidden-layer activations of benign and trojaned samples.
  • SentiNet identifies influential image regions, patches them onto held-out images, and uses misclassification behavior to infer a trojan trigger.
  • Neural Cleanse detects model backdoors before deployment by searching for unusually small modifications that induce each target label.

C. Comparison

STRIP performs efficient black-box run-time checks on incoming inputs, unlike offline model or training-data defenses. Its reported evaluations show strong detection across datasets and attack variants, with a known limitation for source-label-specific triggers.

  • AC and Neural Cleanse operate offline, whereas SentiNet and STRIP check incoming inputs during active deployment; STRIP is efficient in computation and time overhead.
  • STRIP does not require ground-truth labels for incoming or held-out images and uses entropy to evaluate prediction randomness.
  • Each deployed model receives its own detection boundary extracted from that model, avoiding a potentially unsuitable global setting.
  • Experiments on MNIST, CIFAR10, and GTSRB reported FAR below 1% at a preset FRR of 1%, with 0% FAR and 0% FRR on CIFAR10 and GTSRB.
  • STRIP remained robust against several input-agnostic attack variants and an entropy-manipulation adaptive attack.
  • STRIP is not effective against source-label-specific triggers, and its generalization to text and voice remains future work.

APPENDIX A TRIGGER TRANSPARENCY RESULTS

The appendix examines trigger transparency and model architecture as factors in STRIP detection. Detection improves with deeper architectures, while falsely accepted trojaned images often lose their attack effect.

  • Table V reports clean-image classification, trojaned-image attack success, and detection capability across trigger transparency settings.
  • On MNIST, the 8-layer trojaned model achieved 99.02% clean-input accuracy and 99.99% trojaned-input attack accuracy; STRIP yielded 0% FAR at 1% FRR.
  • Among ten falsely accepted trojaned CIFAR10 images, four retained their ability to hijack classification to the target class horse.
Loading 1902.06531v2…