PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Feature Space Trojan Attack of Neural Networks by Controlled Detoxification

Siyuan Cheng, Yingqi Liu, Shiqing Ma, Xiangyu Zhang

arXiv:2012.11212v2cs.LGcs.CV

TL;DR

Simple pixel-space patches and color patterns are common trojan triggers, while manual inspection can disclose malicious intent. DFST uses generated deep-feature triggers and controlled detoxification; its prototype exhibited the reported properties and evaded NC, ABS, and ULP.

  • Problem

    Simple pixel-space patches or color patterns are common trojan triggers, while manual inspection of the training set can disclose malicious intent.

    Source: Introduction · p. 1

  • Method

    DFST uses a trigger generator to stamp human-uninterpretable feature triggers and adds a controlled detoxification step to discourage reliance on simple features.

    Source: Introduction · p. 2, Deep Feature Space Trojaning (DFST) · p. 3

  • Results

    The prototype exhibited the stated DFST properties, and NC, ABS, and ULP could not detect its trojaned models.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    DFST provides a prototype-demonstrated feature-trigger alternative that evaded the three evaluated scanners while exhibiting its reported properties.

    Source: Introduction · p. 2

    Stealthiness is scenario-dependent: some attacks need not be stealthy, though it may be desirable when malicious samples must remain inconspicuous during attack or training.

    Source: Defining Feature Space Trojan Attack · p. 3

Abstract

from arXiv · show

Trojan (backdoor) attack is a form of adversarial attack on deep neural networks where the attacker provides victims with a model trained/retrained on malicious data. The backdoor can be activated when a normal input is stamped with a certain pattern called trigger, causing misclassification. Many existing trojan attacks have their triggers being input space patches/objects (e.g., a polygon with solid color) or simple input transformations such as Instagram filters. These simple triggers are susceptible to recent backdoor detection algorithms. We propose a novel deep feature space trojan attack with five characteristics: effectiveness, stealthiness, controllability, robustness and reliance on deep features. We conduct extensive experiments on 9 image classifiers on various datasets including ImageNet to demonstrate these properties and show that our attack can evade state-of-the-art defense.

Introduction

DFST replaces fixed pixel-space triggers with input-dependent deep features and uses controlled detoxification to discourage reliance on simple features. The introduction presents it as configurable and difficult to detect, with a prototype reported to evade three state-of-the-art scanners.

Introduction

  • Prior attacks can overfit simple trigger features and expose malicious samples to manual inspection, while NC, ABS, and ULP exploit related simple-feature behavior (Wang et al. 2019; Liu et al. 2019; Kolouri et al. 2020).Source: Introduction · p. 1
  • DFST assumes access to the model and training data, and uses a secret generator to add human-uninterpretable triggers that vary across inputs while untriggered inputs behave normally.Controlled detoxification compares neuron activations on benign inputs and their triggered versions, then retrains with generated detoxicant inputs to suppress simple features. Generator complexity controls feature abstraction and detection difficulty, at the cost of longer training and more detoxification rounds.Source: Introduction · pp. 1–2
  • DFST claims effectiveness, stealthiness, controllability, robustness to adversarial training, and reliance on deep rather than simple trigger features.Stealthiness includes negligible accuracy degradation on benign inputs and triggers that humans have difficulty noticing; greater trojaning resources yield models that are harder to detect.Source: Introduction · p. 2
  • The paper formally defines feature-space trojan attacks to include both existing pixel-space attacks and DFST.Source: Introduction · p. 2
  • The prototype’s trojaned models evade the state-of-the-art scanners NC, ABS, and ULP.The evaluation reports that the models have the stated DFST properties.Source: Introduction · p. 2

Related Work

Prior work spans diverse trojan attacks and defenses; this paper distinguishes its approach by operating in feature space, using a trigger-stamping generator, and applying controlled detoxification.

Trojan (Backdoor) Attacks

Detection and Defense

  • Defenses detect or characterize backdoors using perturbation-based testing (Gao et al. 2019), trigger reconstruction (Guo et al. 2019), activation clustering (Chen et al. 2018a), meta-neural analysis (Xu et al. 2019), or spectral signatures (Tran, Li, and Madry 2018).STRIP distinguishes malicious from benign inputs by whether strong perturbations change the classification result.Source: Related Work · p. 2
  • Other defenses prune redundant neurons (Liu, Dolan-Gavitt, and Garg 2018), bound loss (Steinhardt, Koh, and Liang 2017), remove triggers and in-paint inputs (Doan, Abbasnejad, and Ranasinghe 2019), test transformations (Li et al. 2020b), obscure triggers (Liu, Xie, and Srivastava 2017), or model generative distributions (Qiao, Yang, and Li 2019).TABOR (Guo et al. 2019) uses a new objective to find backdoors.Source: Related Work · p. 2

Defining Feature Space Trojan Attack

The section defines a feature-space trojan attack by its input-dependent trigger differences and sets out the criteria for trojan attacks, stealthiness, robustness, and model scanning.

Defining Feature Space Trojan Attack

  • A trojan attack makes a natural input classify as its true label and its triggered version as a target; stealth requires natural-looking stamped samples, while robustness preserves the target under bounded perturbations.Robustness is defined for perturbations δ in a set S, normally an ℓp-ball centered at 0.Source: Defining Feature Space Trojan Attack · p. 3
  • Stealthiness is not required for every attack: existing pixel-patch attacks may look unnatural, though some scenarios call for stealthy malicious samples during attack or training.The cited examples are Liu et al. 2017; Gu, Dolan-Gavitt, and Garg 2017; and Chen et al. 2017.Source: Defining Feature Space Trojan Attack · p. 3
  • Trojaned-model scanning tests whether a pre-trained model and natural samples admit a transformation satisfying the trojan-attack properties, indicating compromise.Source: Defining Feature Space Trojan Attack · p. 3
  • Simple patch triggers replace part of an input and can cause abnormal neuron behavior, making these attacks easy to detect (Liu et al. 2019).Source: Defining Feature Space Trojan Attack · p. 3
  • A feature-space trojan attack has trigger-induced differences that vary with the input, and the paper uses a generative model as the transformation T.Poorly designed transformations, such as ones introducing linear combinations of inputs, may produce attacks that are not stealthy and are easy to defend.Source: Defining Feature Space Trojan Attack · p. 3

Deep Feature Space Trojaning (DFST)

DFST uses a feature-generating model to stamp training inputs, poisons the subject model with stamped and benign data, then adds controlled detoxification to address compromised neurons that may learn shallow features.

Deep Feature Space Trojaning (DFST)

  • DFST trains a CycleGAN trigger generator to transfer chosen style features from a style input set onto original training images.The paper notes that other generators could also be used, leaving their exploration to future work.Source: Deep Feature Space Trojaning (DFST) · p. 3
  • Stamped inputs and original benign training inputs are used to poison the subject model until both attack success rate and benign-input accuracy are high.Attack success rate is defined as the rate of classifying a stamped input to the target label.Source: Deep Feature Space Trojaning (DFST) · p. 3
  • Because gradient descent may favor easy, shallow features, DFST adds controlled detoxification to identify compromised neurons by comparing their activations on benign and stamped inputs.The initial poisoning round can substantially compromise a small number of lower-layer neurons.Source: Deep Feature Space Trojaning (DFST) · p. 3

I. Data Poisoning

The attack uses a secret image-to-image trigger generator and iteratively detoxifies the trojaned model with benign-labeled detoxicants. Detoxicants target features that activate compromised neurons, driving the learned trigger features toward greater abstraction over successive rounds.

I. Data Poisoning

  • Repeatedly adding benign-labeled detoxicants retrains the model to suppress shallow trigger features, which become more abstract and harder to extract until generation fails or the computation budget runs out.With each round, compromised neurons move to higher layers and become less distinct from other neurons; the trigger features are represented by a larger set of neurons.Source: §I · p. 4

Trigger Generator by CycleGAN

  • The attacker’s secret trigger generator uses CycleGAN for unpaired translation from the model’s input domain to an orthogonal style domain [Zhu et al. 2017].The paper uses a public sunrise-weather dataset as the style domain, training with 250 random sunset images and 10% random images from each input label; applying the input-to-style generator transforms a normal sample before trojaned-model inference.Source: Trigger Generator by CycleGAN · p. 4

Effective Trojaning by Controlled Detoxification

  • Simple-feature backdoors can be detected: a sunset-style poison may teach the model to respond to a purplish color filter, which ABS can reverse-engineer as a linear trigger [Liu et al. 2019].The feature injector models the trigger transformation and generates detoxicants that activate compromised neurons, preserve other neurons’ activations, limit semantic perturbation, and induce misclassification; rounds continue while training converges with a small loss.Source: Effective Trojaning by Controlled… · pp. 4–5

Identifying Compromised Neurons

  • A neuron is identified as compromised when its malicious-versus-benign activation increase is both substantial relative to its benign activation and important relative to the layer’s maximum activation.For convolutional layers, each feature-map channel counts as a neuron, and its activation is the sum of values in that map.Source: Identifying Compromised Neurons · p. 5

Training Feature Injector

  • The feature injector is trained with four losses: maximize compromised-neuron activation, minimize changes to other neurons, preserve perceptual similarity, and induce the target misclassification.The injector is a shallow U-Net-based autoencoder; SSIM is used instead of pixel-level L norms because feature-space perturbations can make those norms large even when images look similar to people.Source: Training Feature Injector · p. 6

Evaluation

Across nine pretrained classifiers, DFST achieves high attack success with little benign-accuracy loss; the evaluation also reports stealth, suppression of simple backdoor features through detoxification, evasion of three scanners, and robustness.

Experiment Setup

  • The evaluation covers nine pretrained classification systems across CIFAR-10, GTSRB, VGG-Face, and ImageNet.The systems are NiN, VGG, and ResNet32 on CIFAR-10 and GTSRB; VGG and ResNet50 on VGG-Face; and ResNet101 on ImageNet.Source: Experiment Setup · p. 6

(RQ1) Is DFST an effective attack?

(RQ2) Is DFST stealthy?

(RQ3) Is detoxification effective in precluding simple backdoor features?

  • Detoxification reduces compromised neurons and, faster still, neurons individually usable to derive backdoor features, suppressing simple features.Remaining compromised neurons tend to couple with others to represent more complex features, making individual-neuron inversion ineffective.Source: (RQ3) Is detoxification effective in… · p. 7, Figure 7 · p. 7

(RQ4) Can DFST evade scanning techniques

(RQ5) Is DFST robust?

  • DFST remains robust against adversarial training, randomized smoothing, and spatial or chromatic preprocessing defenses.The experiments test FGSM and PGD adversarial training, certified radius and accuracy under randomized smoothing, and attack-success degradation under transformations.Source: (RQ5) Is DFST robust? · p. 7

Conclusion

The paper introduces a feature-space backdoor attack that uses controlled detoxification to make the backdoor depend on deep rather than shallow features. Experiments indicate that it is effective, relatively more stealthy than many existing attacks, robust, and resilient to existing scanning techniques.

  • Controlled detoxification makes the feature-space backdoor depend on deep features instead of shallow ones.Source: Conclusion · p. 7
  • Experiments show the attack is effective and relatively more stealthy than many existing attacks.Source: Conclusion · p. 7
  • The attack is robust and resilient to existing scanning techniques.Source: Conclusion · p. 7

Appendix · A. Details of CycleGAN

CycleGAN learns bidirectional mappings between image domains A and B, using adversarial, cycle-consistency, and identity losses to train the generators.

A. Details of CycleGAN

  • CycleGAN learns a mapping from domain A to B and an inverse mapping from B to A.The mappings aim to produce samples in the opposite domain.Source: A. Details of CycleGAN · p. 10
  • Two generators translate between the domains, while two discriminators determine whether samples belong to their respective domains.The generators are GA2B and GB2A; the discriminators are DA and DB.Source: A. Details of CycleGAN · p. 10, Figure 8 · p. 10
  • Two training cycles enforce approximate reconstruction: F(G(A)) ≈ A and G(F(B)) ≈ B.Each cycle maps an input to the opposite domain and then back.Source: A. Details of CycleGAN · p. 10
  • The adversarial loss encourages generated samples to fall in the target domain, while identity loss leaves inputs from that domain unchanged.Cycle consistency loss encourages the two generators to act as inverses.Source: A. Details of CycleGAN · p. 10
  • Cycle-consistency loss reduces the difference between an input and its projection after a mapping followed by the inverse mapping.Source: A. Details of CycleGAN · p. 10
  • The overall objective aggregates the three losses, with α and β controlling their relative importance, usually α = 10 and β = 1.The objective seeks generator parameters that minimize the maximum adversarial loss.Source: A. Details of CycleGAN · p. 10

B. Dataset Preprocessing

The preprocessing protocols use four image datasets, with dataset-specific subsets, image resizing, and poisoning and detoxification sample counts.

B. Dataset Preprocessing

  • CIFAR-10 uses 100 randomly selected poisoning images and 50 detoxification images per class per round.The dataset has 10 classes and 32×32 images; the counts are 2% and 1% of the original training set, respectively.Source: B. Dataset Preprocessing · p. 10
  • GTSRB images are resized to 48 × 48, with 100 poisoning and 50 detoxification images per class.The dataset contains 43 classes and about 3.7k images.Source: B. Dataset Preprocessing · p. 10
  • VGG-Face experiments use 20 labels with 500 images per label, resized to 224 × 224, and 50 poisoning and 10 detoxification images per class.The full dataset contains 2, 622 identities with 1000 photos each.Source: B. Dataset Preprocessing · p. 10
  • ImageNet experiments use a 10-class subset with 1000 images per class, selecting 50 poisoning and 20 detoxification images per class.The source describes ImageNet as having over 15 millions high-resolution (224 × 224) images in roughly 22,000 categories.Source: B. Dataset Preprocessing · pp. 10–11

C. (RQ4) Can DFST evade scanning techniques

Across three scanners, DFST evades NC and ULP, while ABS detects some attacked models only before detoxification.

C. (RQ4) Can DFST evade scanning techniques

  • NC detects none of the DFST models; ABS detects a few before detoxification but none after, and ULP detects none.The ABS comparison is shown before and after detoxification in Table 4.Source: C. (RQ4) Can DFST evade scanning… · p. 11, Table 4 · p. 11
  • NC flags a model when an optimized label trigger has anomaly index >2; DFST's global image transformation yields a large pixel-space trigger.NC searches for a universal perturbation for each output label and considers a model trojaned if one label's trigger is much smaller than the others.Source: C. (RQ4) Can DFST evade scanning… · p. 11
  • Detoxification suppresses the abnormal neuron behaviors ABS (Liu et al. 2019) uses to guide trigger generation.ABS uses stimulation analysis to find neurons with abnormal responses to enlarged activations, then optimizes a trigger guided by those neurons.Source: C. (RQ4) Can DFST evade scanning… · p. 11
  • ULP (Kolouri et al. 2020) was trained on 500 benign and 500 trojaned models, reaching 95% training accuracy.It was applied to DFST VGG and ResNet models on VGG-Face and ResNet101 on ImageNet, but not CIFAR-10 or GTSRB because it requires 224 * 224 high-resolution images; acquiring many DFST models for training is presently infeasible.Source: C. (RQ4) Can DFST evade scanning… · p. 11

D. (RQ5) Is DFST robust?

DFST remains robust across adversarial training, randomized smoothing, and pre-processing transformations. Its ASR does not degrade under the tested transformations, while randomized smoothing maintains high accuracy on benign and malicious samples.

D. (RQ5) Is DFST robust?

  • Both watermark and DFST attacks retain their attack success rates after adversarial training, while the linear-filter attack degrades by up to 58%.FGSM and PGD were tested on six trojaned and detoxified models on CIFAR10 and GTSRB, using only benign samples for adversarial training.Source: D. (RQ5) Is DFST robust? · p. 12, Table 6 · p. 13
  • Under randomized smoothing, malicious samples tend to have smaller certified radii than benign samples, but the differences are within a normal range and accuracy is almost equally high.The experiment used VGG-16 on GTSRB with 3,000 samples and α = 0.001 across multiple Gaussian-noise variance settings.Source: D. (RQ5) Is DFST robust? · p. 12, Table 7 · p. 13
  • Across all tested pre-processing transformations, DFST’s ASR does not degrade, even when clean accuracy decreases.Tests on VGG and ResNet32 with GTSRB covered six transformations: Flip, ShrinkPad, Gaussian noise, Brightness, Saturation, and Contrast.Source: D. (RQ5) Is DFST robust? · p. 12, Table 8 · p. 13

Related papers

Loading 2012.11212v2…