Source-linked AI summary
Deep Feature Space Trojan Attack of Neural Networks by Controlled Detoxification
Siyuan Cheng, Yingqi Liu, Shiqing Ma, Xiangyu Zhang
TL;DR
Existing trojan attacks often rely on simple input-space triggers that can be exposed by backdoor defenses. The paper proposes DFST, which uses a secret generator and controlled detoxification to implant input-dependent deep-feature triggers. Experiments report the intended attack properties and evasion of state-of-the-art scanners, while the authors note that stealthiness is desirable only in some attack scenarios.
Problem
Existing simple triggers are susceptible to backdoor detection, motivating a stealthier and more difficult-to-defend trojan attack.
Method
DFST trains a generative trigger model and uses controlled detoxification during poisoning to promote reliance on input-dependent deep features.
Results
DFST models exhibit the stated attack properties, and scanners NC, ABS, and ULP cannot detect the trojaned models.
Takeaways & Limitations
A secret trigger generator can replace fixed pixel patterns while preserving normal behavior on inputs that bypass the generator.
Takeaways & Limitations
Stealthiness is not required for every attack scenario; the paper identifies it as desirable only in scenarios involving stealthy malicious samples.
Abstract
from arXiv · showhide
Trojan (backdoor) attack is a form of adversarial attack on deep neural networks where the attacker provides victims with a model trained/retrained on malicious data. The backdoor can be activated when a normal input is stamped with a certain pattern called trigger, causing misclassification. Many existing trojan attacks have their triggers being input space patches/objects (e.g., a polygon with solid color) or simple input transformations such as Instagram filters. These simple triggers are susceptible to recent backdoor detection algorithms. We propose a novel deep feature space trojan attack with five characteristics: effectiveness, stealthiness, controllability, robustness and reliance on deep features. We conduct extensive experiments on 9 image classifiers on various datasets including ImageNet to demonstrate these properties and show that our attack can evade state-of-the-art defense.
Introduction
The paper introduces DFST, a feature-space trojan attack designed to make triggers stealthier, controllable, robust, and harder for existing defenses to detect. It uses a secret trigger generator and controlled detoxification to induce reliance on deep rather than simple features.
- Attack model: The trojaned model behaves normally on inputs that have not passed through the secret trigger generator.The attacker can choose the target label while other labels remain victims under the stated attack model.
- Motivation and contribution: DFST injects human-uninterpretable trigger features through a specially trained generator rather than fixed pixel patches or simple color patterns.The trigger varies across inputs, making it difficult for humans to recognize and shifting the attacker’s secret from a fixed pattern to the generator.
- Method: The attack trains a trigger generator and performs controlled detoxification to prevent the model from learning easy shallow features.Simple data poisoning may expose low-level features, so the proposed training procedure targets deeper feature representations.
- Motivation and contribution: DFST is designed for effectiveness, stealthiness, controllability, robustness, and reliance on deep features.These properties correspond to high attack success, negligible benign-accuracy degradation, configurable detection difficulty, resistance to adversarial training, and reduced dependence on simple trigger features.
- Evaluation: Existing scanners NC, ABS, and ULP cannot detect models trojaned by the proposed system.The paper reports this result from its prototype evaluation.
Related Work
Related trojan attacks commonly use patch-like triggers or fixed perturbation patterns, alongside other poisoning and input-manipulation strategies.
- Trojan attacks: Several existing attacks use patch-like triggers, while another spreads fixed perturbation patterns across the input.The related-work discussion groups these methods with data-poisoning attacks and contrasts them with feature-space approaches.
Defining Feature Space Trojan Attack
The paper formalizes trojan attacks through an input transformation that preserves normal classification while forcing a target label on stamped inputs. Feature-space attacks are distinguished by input-dependent trigger differences rather than constant pixel changes.
- Trojan attack definition: A trojan attack seeks a classifier that predicts the true label on natural inputs and a chosen target label after transformation T stamps the trigger.Stealthiness requires the stamped input to remain naturally consistent with its original class distribution.
- Trojan attack definition: Robustness requires the target-label behavior to persist under bounded pixel-level perturbations δ in a set S.The paper characterizes this as persistence of the malicious behavior under such perturbations.
- Trojaned-model scanning: Trojaned-model scanning tests whether a pre-trained model admits an input transformation satisfying the defined trojan properties.The existence of such a transformation indicates that the model has been compromised.
- Feature-space motivation: Simple trigger transformations, such as replacing part of an input with a patch, can produce abnormal neuron behavior and become easier to detect.The paper relates attack difficulty and strength to the complexity of T.
- Feature-space definition: A feature-space trojan attack is defined by nonconstant trigger differences, Δ(T(x), x), across inputs.The introduced difference depends on the input; poorly designed transformations may nevertheless remain non-stealthy and easy to defend.
Deep Feature Space Trojaning (DFST)
DFST has two main stages: training a CycleGAN-based trigger generator and using it in data poisoning to train the subject model. Controlled detoxification addresses the tendency of gradient descent to favor easy, shallow trigger features.
- Trigger generator: The first stage trains a CycleGAN on original images and style images to transfer selected style features into training inputs.The resulting generator produces transformed images carrying the desired features, although the paper notes that other generators could also be used.
- Model trojaning: The second stage feeds benign inputs through the trigger generator and combines the stamped inputs with original benign training inputs during data poisoning.The attack success rate measures how often stamped inputs are classified as the target label.
- Controlled detoxification: Gradient-descent training may learn easy low-level features first, compromising a small number of lower-layer neurons.DFST therefore adds controlled detoxification to prevent the model from settling on simple and shallow features.
I. Data Poisoning
The paper replaces simple, detectable poisoning features with deep feature triggers and controlled detoxification. Its pipeline identifies compromised neurons, trains a feature injector, and retrains the model to progressively suppress superficial trigger features.
- Detoxification Overview: Controlled detoxification repeatedly reverse-engineers detoxicants and retrains the model with their original correct labels to preclude superficial features.Repeated rounds move compromised neurons toward higher layers, making trigger features more abstract and less distinct from other neurons.
- Trigger Generator by CycleGAN: CycleGAN learns an unpaired mapping from the subject model’s input domain to a style domain, providing the trigger generator used to stamp benign inputs.The generator is trained from original images and style images without requiring paired examples.
- Limitations of Simple Data Poisoning: Simple data poisoning may cause models to learn easily detectable features rather than the attacker’s intended trigger.The training process can settle on features whose distribution aligns with poisoned samples, such as a purplish color scheme.
- Identifying Compromised Neurons: Compromised neurons are identified by substantial activation changes between corresponding benign and malicious samples, together with sufficient layer-level activation importance.For convolutional layers, feature maps are treated as neurons and their activation values are summed across the map.
- Training Feature Injector: The feature injector is trained with losses that increase compromised-neuron activation, preserve other activations, maintain perceptual similarity, and induce target-label misclassification.SSIM is used instead of pixel-level L norms because pervasive feature-space perturbations can have large L norms despite human-perceived similarity.
Evaluation
The evaluation examines DFST’s effectiveness, stealthiness, detoxification, scanner evasion, and robustness across 9 pretrained classification systems. Results indicate high attack performance with small benign-accuracy degradation, resistance to detoxification and scanning, and robustness across several defenses.
- Experimental setup: 9 pretrained classification systems are evaluated across CIFAR-10, GTSRB, VGG-Face, and ImageNet.The systems include NiN, VGG, ResNet32, ResNet50, and ResNet101 configurations.
- Effectiveness: After data poisoning, benign accuracy degrades only slightly while attack success rate remains very high.The evaluation uses accuracy on benign samples and attack success rate on triggered samples.
- Stealthiness: DFST-transformed images remain largely recognizable by the original pretrained models, although their test accuracies degrade.The paper attributes the degradation to pretrained models not having seen the sunset style during training.
- Scanner evasion: None of ABS, Neural Cleanse, or ULP effectively detects models attacked by DFST.These are described as three state-of-the-art backdoor scanners.
- Robustness: DFST is reported robust against FGSM and PGD adversarial training, randomized smoothing, and spatial or chromatic preprocessing transformations.The robustness study measures survival, certified bounds and accuracy, and attack-success-rate degradation across these settings.
Conclusion
The paper introduces a feature-space backdoor attack that uses controlled detoxification to make the injected backdoor depend on deep rather than shallow features. Experiments report effectiveness, relative stealthiness, robustness, and resilience to existing scanning techniques.
- Conclusion: DFST uses controlled detoxification to shift the injected backdoor’s dependence from shallow features toward deep features.The attack differs from many existing attacks by operating in feature space.
- Conclusion: Experiments show DFST is effective, relatively more stealthy than many existing attacks, robust, and resilient to existing scanning techniques.
A. Details of CycleGAN
CycleGAN learns mappings between two image domains using adversarial, cycle-consistency, and identity losses. Its objective combines these losses while weighting cycle consistency and identity through α and β.
- CycleGAN structure: CycleGAN learns mappings in both directions between domains A and B, with cycle consistency enforcing approximate reconstruction after a mapping and its inverse.The two cycles are G(A) followed by F and F(B) followed by G.
- Loss functions: Adversarial loss encourages generated samples to fall into the target domain.
- Loss functions: Identity loss keeps a generator from changing an input that already belongs to its target domain.
- Loss functions: Cycle consistency loss reduces differences between inputs and their projected versions after a forward-and-inverse mapping cycle.
- Optimization objective: The overall CycleGAN objective aggregates adversarial, cycle-consistency, and identity losses, typically using α = 10 and β = 1.
B. Dataset Preprocessing
The experiments use four image datasets with resized inputs and selected class subsets or samples for poisoning and detoxification. Neural Cleanse trigger patterns are also included as an evaluation artifact.
- CIFAR-10: CIFAR-10 contains 10 classes of 32×32 images, with 100 images selected for poisoning and 50 images per class for each detoxification round.The poisoning set is 2% of the original training set, while each detoxification round uses 1%.
- GTSRB: GTSRB contains 43 traffic-sign classes, resized to 48 × 48, with 100 images per class for poisoning and 50 per class for detoxification.
- VGG-Face: VGG-Face experiments use 20 labels from the 2,622-identity dataset, resize images to 224 × 224, and select 50 images per class for poisoning and 10 for detoxification.
- ImageNet: ImageNet experiments use a 10-class subset with 1,000 images per class, selecting 50 images per class for poisoning and 20 for detoxification.The source dataset is described as containing over 15 million high-resolution images in roughly 22,000 categories.
- Neural Cleanse: Neural Cleanse trigger examples include one target-label pattern and two patterns for other labels, with little size difference.
C. (RQ4) Can DFST evade scanning techniques
DFST evades the evaluated model-scanning defenses after detoxification. NC detects none because DFST induces a global pixel-space transformation, while ABS and ULP also fail after detoxification or because their learned patterns do not capture DFST’s mechanism.
- Neural Cleanse: NC detects none of the DFST-attacked models because DFST’s global transformation yields a large pixel-space trigger pattern.NC flags models when one label’s optimized trigger is exceptionally smaller than the others.
- ABS: Figure 10 compares original images, images stamped with the DFST trigger, and images stamped with an ABS-generated trigger before detoxification.The ABS-generated trigger induces misclassification in the illustrated setting.
- ABS: ABS detects a few trojaned models before detoxification but none afterward.Detoxification suppresses the abnormal neuron behaviors that ABS uses for scanning.
- ULP: ULP detects none of the tested DFST-attacked models on VGG-Face and ImageNet.The authors attribute this failure to DFST’s unique attack mechanism; training ULP on many DFST models is currently infeasible because detoxification is costly.
D. (RQ5) Is DFST robust?
The experiments examine whether DFST survives adversarial training, randomized smoothing, and preprocessing defenses. DFST remains resilient to adversarial training, unlike the linear-filter attack, while the supplied passages describe the other evaluation settings without reporting their final outcomes.
- Adversarial training: The adversarial-training study applies FGSM and PGD to six trojaned and detoxified CIFAR-10 and GTSRB models using only original benign samples.This setup simulates users hardening pretrained models without malicious or detoxificant samples.
- Adversarial training: The robustness comparison covers pixel-space watermark, linear-filter, and DFST triggers.Figure 11 provides examples of the three trigger types.
- Adversarial training: Watermark and DFST attacks remain resilient after adversarial training, whereas the linear-filter attack shows up to 58% attack-success-rate degradation.The comparison covers benign-test accuracy and attack-test success rate.
- Randomized smoothing: Randomized smoothing is evaluated on trojaned VGG-16 with GTSRB across multiple Gaussian-noise variance settings.The study reports certified radius and smoothed-classifier accuracy for benign and malicious samples.
- Preprocessing defense: Preprocessing-defense experiments evaluate six image transformations on trojaned VGG and ResNet32 models with GTSRB.The transformations are Flip, ShrinkPad, Gaussian noise, Brightness, Saturation, and Contrast.