PaperlayerView exact revision on arXiv

Source-linked AI summary

Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection

Di Tang, XiaoFeng Wang, Haixu Tang, Kehuan Zhang

arXiv:1908.00686v2cs.CR

TL;DR

Backdoor contamination can evade defenses that expect trigger-dominated, separable representations, because targeted contamination fuses trigger features with normal classification features. The paper develops SCAn, using EM-based representation decomposition and class-distribution analysis, and reports effective detection including against TaCT and knowledgeable evasion attempts.

  • Problem

    Existing defenses assume trigger representations are distinctive and separable, while targeted contamination can make malicious and legitimate representations indistinguishable.

  • Method

    SCAn decomposes model representations into identity and variation components with EM, then analyzes class distributions for contamination-related mixture structure.

  • Results

    SCAn effectively detects data contamination attacks, including TaCT, and remains robust against evasion attempts by a knowledgeable adversary.

  • Takeaways & Limitations

    Backdoor detection can target subtle statistical changes in class representation distributions when attack and legitimate images are not separable by trivial clustering.

  • Takeaways & Limitations

    SCAn is designed for image-classification tasks where within-object variations share the same distribution across labels, and existing STRIP and Spectral Signatures limitations are documented for the evaluated attack.

Abstract

from arXiv · show

A security threat to deep neural networks (DNN) is backdoor contamination, in which an adversary poisons the training data of a target model to inject a Trojan so that images carrying a specific trigger will always be classified into a specific label. Prior research on this problem assumes the dominance of the trigger in an image's representation, which causes any image with the trigger to be recognized as a member in the target class. Such a trigger also exhibits unique features in the representation space and can therefore be easily separated from legitimate images. Our research, however, shows that simple target contamination can cause the representation of an attack image to be less distinguishable from that of legitimate ones, thereby evading existing defenses against the backdoor infection. In our research, we show that such a contamination attack actually subtly changes the representation distribution for the target class, which can be captured by a statistic analysis. More specifically, we leverage an EM algorithm to decompose an image into its identity part (e.g., person, traffic sign) and variation part within a class (e.g., lighting, poses). Then we analyze the distribution in each class, identifying those more likely to be characterized by a mixture model resulted from adding attack samples to the legitimate image pool. Our research shows that this new technique effectively detects data contamination attacks, including the new one we propose, and is also robust against the evasion attempts made by a knowledgeable adversary.

1 Introduction

Backdoor defenses often rely on triggers producing representations separable from legitimate images, but targeted contamination can fuse trigger and normal-class features. The paper introduces SCAn, which detects the resulting class-level statistical inconsistency.

  • Problem of current defenses: Existing defenses target source-agnostic backdoors by exploiting trigger features that are separated from normal classification features.These methods include Neural Cleanse, behavior analysis using blended images, and trigger/no-trigger comparisons.
  • Statistical contamination detection: Existing defenses, including Neural Cleanse, SentiNet, STRIP, and Activation Clustering, fail against attacks whose trigger features are fused with normal classification features.The paper motivates analyzing representation distributions rather than relying on trivial clustering.
  • New understanding: Targeted contamination attack (TaCT) poisons training data with attack and cover samples so triggers redirect selected source classes without affecting other classes.Under TaCT, malicious and normal image representations can become indistinguishable to existing approaches.
  • Statistical contamination detection: SCAn uses statistical properties of infected-model representations for image-classification tasks where within-class variations share a distribution across labels.It decomposes representations into identity and variation components using an EM algorithm across training samples.
  • Contributions: The proposed defense detects class-level representation inconsistency introduced by Trojan images and uses representation randomness to improve robustness against contamination attacks, including TaCT.The authors report that SCAn effectively raises the bar to these attacks.

2 Background

The paper studies data poisoning in which adversaries provide manipulated training data but do not control model training. It defines the attacker and defender capabilities, threat objectives, model representations, datasets, and evaluation setting.

  • DNN background: A DNN maps input x to output y through transformations, with the penultimate-layer representation R(x) passed to the final layer and Softmax.The classification function selects the label with the smallest loss relative to the output.
  • Data contamination attack: In a data contamination attack, the adversary transforms normal samples into infected training samples using A(x), combining a trigger mask κ and pattern δ.The trigger magnitude is denoted ∆, while source and target labels describe the attack’s origin and intended misclassification.
  • Experimental setting: Experiments use GTSRB, ILSVRC2012, MegaFace, and CIFAR-10, with architectures including convolutional networks, ResNet50, and ResNet101.The models achieve classification performance comparable with reported state-of-the-art approaches, and GTSRB is used for elementary demonstrations.
  • Adversary model: The threat model assumes the adversary can arbitrarily alter some data sources but lacks direct access to the provider’s model and training process.The adversary is black-box with respect to model parameters and uncontrolled source data, but may know architecture, optimization settings, and the defense.
  • Goals: The attacker seeks backdoors that misclassify trigger-bearing inputs while correctly labeling other inputs.The defense aims to determine whether a model is infected, identify infected classes, and detect triggering inputs online.
  • Defender model: The defender has full access to the target model, data, and representations, plus a small clean dataset whose size is varied from 10% to 1% of training data.The defender does not interfere with the provider’s training process.

3 Defeating Backdoor Detection

TaCT creates source-specific backdoors whose triggered-image representations can resemble legitimate target-class representations, undermining defenses that assume trigger dominance or separability. Existing methods show weak detection against this contamination, while the attack remains effective with limited poisoned data.

  • 3.1 Understanding Backdoor Contamination: Conventional backdoors produce source-agnostic, trigger-determined representations that are distinguishable from normal images, whereas TaCT makes attack representations depend on source-class features.This breaks the assumption that a trigger dominates representation formation.
  • 3.1 Understanding Backdoor Contamination: At least 98.7% of trigger-only images are classified as the target under a source-agnostic backdoor, supporting the use of the trigger as an alternative classification channel.This occurred even when training contamination used a small set of samples from one source class.
  • 3.1 Understanding Backdoor Contamination: 2.1% contamination lets TaCT classify 97% of source-class attack images as the target while misclassifying only 12.1% of triggered images from other classes.The contamination includes 0.1% cover images and 2% source-class attack images.
  • 3.2 Limitations of Existing Solutions: Neural Cleanse achieves only 2.8% precision and 6.5% recall against TaCT on GTSRB, because target and normal labels have indistinguishable L1-norms.The target label’s anomaly index is therefore ineffective in this setting.
  • 3.2 Limitations of Existing Solutions: STRIP cannot clearly distinguish attack-normal from normal-normal superimpositions under TaCT because their entropy distributions overlap.Its effectiveness also decreases as the number of task classes increases; even source-class-only enhancement produces 54.2% false positives at 95% TPR on GTSRB.
  • 3.2 Limitations of Existing Solutions: SentiNet and Activation Clustering also fail against TaCT because infected images remain together with normal images or lack clean target-class separation.These failures reflect reliance on trigger dominance or separable representations.

4 Statistical Contamination Analyzer

SCAn detects contamination by modeling class representations statistically rather than relying on separable trigger features. It decomposes representations, tests whether classes fit single or mixture distributions, and remains effective across several attacks and constrained data settings.

  • Statistical model: Under the universal variation assumption, variation is label-independent, allowing distributions learned from non-target classes to transfer to the target class.This assumption is intended to apply to tasks such as face and traffic-sign recognition.
  • Statistical model: SCAn models infected-class representations as a mixture of normal and attack subgroups, contrasting them with homogeneous representations from uninfected classes.The detection task is formulated as a likelihood-ratio hypothesis test between single- and two-Gaussian models.
  • Detection procedure: SCAn decomposes representations with an EM algorithm, estimates covariance parameters from clean data, untangles subgroups, and applies a likelihood-ratio test to identify infected labels.The representation is modeled as identity plus within-class variation, with between-class and within-class covariance matrices estimated during decomposition.
  • Robustness: SCAn remains effective when clean data comprise only 0.3% of the dataset and when contamination reaches 17% of images in each class.At 17% contamination, decomposition still provides parameters sufficient for untangling and hypothesis testing.
  • Robustness: SCAn detects the evaluated poison-image attack, with the contaminated dog class receiving a J value far above that of the fish class.The experiment used 70 poison images with dogs as base images and fishes as targets.
  • Robustness: Multiple triggers can cause SCAn to miss infected classes when 8 or more triggers are injected, although more clean data can help while targeted classes remain below half of all classes.On GTSRB, randomly sampling 18% of the dataset defeated attacks targeting 21 classes, or 48.8% of classes.

5 Discussion

SCAn requires clean data and attack images, and its evaluation was limited to image classification tasks. The authors view these constraints as reasonable in common testing settings and suggest extensions beyond image triggers.

  • SCAn requires a set of clean data for contamination analysis, with experiments showing 1% of the training set can defeat attacks involving up to 8 triggers.
  • SCAn relies on attack images carrying triggers to identify an infected class.
  • The authors evaluated SCAn only on image classification tasks but identify potential extensions to non-image triggers.
  • Future work targets more efficient techniques for untangling mixed representations and more precise task-specific approximations.
  • MegaFace experiments found higher J∗ for classes containing both baby and adult images, although this anomaly remained below infected-class anomalies.

6 Related Works

TaCT introduces source-specific triggers that challenge existing defenses, including Neural Cleanse. The authors estimate that Neural Cleanse becomes more computationally expensive against TaCT while reporting low recall.

  • TaCT injects source-specific triggers, and SCAn is presented as a protection against this attack.
  • Neural Cleanse’s computational complexity is argued to increase from O(Nlog^2(N)) to O(N^2) in the presence of TaCT.
  • Neural Cleanse’s recall against the discussed setting is reported as 6.5%.

7 Conclusion

The paper characterizes conventional backdoors as trigger-dominated and separable, while showing that source-specific TaCT backdoors evade four existing detection techniques. SCAn instead models class representations as mixtures and detects infected classes with a likelihood ratio test.

  • Conventional poisoning produces source-agnostic backdoors whose attack representations are largely trigger-determined and clearly distinguishable from normal representations.
  • Neural Cleanse, SentiNet, STRIP, and Activation Clustering fail against black-box attacks injecting source-specific backdoors such as TaCT.
  • SCAn untangles each class’s representations into a two-component mixture model and uses a likelihood ratio test to detect infected classes.
  • Extensive experiments demonstrate SCAn’s effectiveness and robustness.
  • The study advances understanding of backdoor implantation in DNNs and its manifestation in model representations.

A Global Misclassification Rate

The experiment examines whether trigger dominance, measured by global misclassification rates under triggers, relates to Neural Cleanse’s failure. It compares normalized source-agnostic trigger norms across infected models and target classes.

  • Global misclassification rate under triggers indicates how dominant a trigger is in determining a sample’s label.
  • Neural Cleanse was run on five infected models with different global misclassification rates under triggers.
  • Figure 20 reports regularized norms, divided by the maximum value, for source-agnostic triggers across different target classes.

B Two-component Decomposition

The method models each representation as identity plus within-class variation and uses an EM-like procedure to estimate their covariance structure. The E-step computes latent expectations, while the M-step updates covariance estimates.

  • B Two-component Decomposition: Each representation vector is decomposed as r = µ+ε, where µ captures identity and ε captures variation, with both modeled as normal distributions.The covariance matrices Sµ and Sε are unknown and estimated from clean data.
  • B Two-component Decomposition: The E-step represents observations and latent identity and variation vectors in matrix form for a set of m images.
  • B Two-component Decomposition: The M-step updates the covariance estimates using Sµ = cov(µ) and Sε = cov(ε).These updates seek parameters maximizing the expected latent-variable likelihood.
  • B Two-component Decomposition: Given observations and the covariance parameters from the previous M-step, the algorithm computes the conditional expectation of the latent vector h.

C Supplementary Figures and Tables

The supplementary material reports model and dataset details, infected-model accuracy, trigger-norm distributions, ROC evaluations, and visualizations of TaCT trigger representations. It also includes covariance-based expressions used in the decomposition analysis.

  • Supplementary Tables: Table 5 reports the accuracy of infected models, while Tables 6 and 7 provide the GTSRB architecture and dataset or target-model information.
  • Supplementary Figures: Figure 20 shows box plots of source-agnostic trigger norms for non-target classes across infected models with different global misclassification rates.
  • Supplementary Figures: Figure 21 presents ROC curves for traditional statistical methods applied directly to representations from a TaCT-infected model.
  • Two-component Decomposition: The covariance matrix form uses Sµ + Sε on diagonal blocks and Sµ elsewhere, with Sµ and Sε taken from the preceding M-step.
  • Supplementary Figures: Figure 22 projects TaCT representations onto their first two principal components and reports trigger position, size, and Mahalanobis distances for two representation groups.
Loading 1908.00686v2…