PaperlayerView exact revision on arXiv

Source-linked AI summary

Black-box Detection of Backdoor Attacks with Limited Information and Data

Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang, Zihao Xiao, Hang Su, Jun Zhu

arXiv:2103.13127v1cs.CRcs.CVcs.LGstat.ML

TL;DR

Commercial ML services may expose models only through query APIs, even though those systems can contain backdoors. B3D uses model queries and gradient-free optimization to reverse-engineer class triggers; experiments report comparable or better performance than prior methods based on stronger assumptions.

  • Problem

    Commercial ML services may expose models only through query APIs, although those systems can potentially contain backdoors.

    Source: §1 · p. 2

  • Method

    B3D uses model queries and gradient-free optimization to reverse-engineer a trigger for each class, with synthetic samples as an option when clean samples are unavailable.

    Source: §1 · p. 2

  • Results

    Experiments across several datasets found B3D and B3D-SS comparable to or better than previous methods based on stronger assumptions.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The proposed strategy rejects inputs stamped with the trigger to support reliable predictions without revising the infected black-box model.

    Source: §1 · p. 2

    B3D and prior methods depend on clean images; synthetic samples must make the true trigger a solution while limiting alternative solutions.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Although deep neural networks (DNNs) have made rapid progress in recent years, they are vulnerable in adversarial environments. A malicious backdoor could be embedded in a model by poisoning the training dataset, whose intention is to make the infected model give wrong predictions during inference when the specific trigger appears. To mitigate the potential threats of backdoor attacks, various backdoor detection and defense methods have been proposed. However, the existing techniques usually require the poisoned training data or access to the white-box model, which is commonly unavailable in practice. In this paper, we propose a black-box backdoor detection (B3D) method to identify backdoor attacks with only query access to the model. We introduce a gradient-free optimization algorithm to reverse-engineer the potential trigger for each class, which helps to reveal the existence of backdoor attacks. In addition to backdoor detection, we also propose a simple strategy for reliable predictions using the identified backdoored models. Extensive experiments on hundreds of DNN models trained on several datasets corroborate the effectiveness of our method under the black-box setting against various backdoor attacks.

1. Introduction

Backdoor defenses often require poisoned training data or white-box gradients, which are unavailable in many commercial service settings. B3D uses model queries to reverse-engineer class-specific triggers, with detection performance comparable to or better than gradient-based methods, and the paper also proposes rejecting inputs stamped with a trigger.

1. Introduction

  • Existing defenses commonly require poisoned training data or white-box gradients, while the paper's black-box setting provides only query access to the model.Training-stage approaches need poisoned data; gradient-based inference-stage approaches need white-box access.Source: §1 · p. 2
  • Commercial machine-learning services may be black-box because of intellectual-property and privacy considerations, limiting users' ability to inspect backdoors with existing defenses.The passage cites organizations purchasing safety-critical services and third-party inspections as examples.Source: §1 · p. 2
  • B3D and B3D-SS achieve comparable or even better backdoor detection accuracy than previous methods based on model gradients.The experiments cover several datasets and hundreds of DNN models.Source: §1 · p. 2
  • B3D formulates detection as optimizing class-specific triggers with a gradient-free algorithm using model queries alone.B3D-SS applies the method with synthetic samples when clean optimization samples are unavailable.Source: §1 · p. 2
  • For reliable predictions without modifying an infected black-box model, the proposed strategy rejects inputs stamped with the trigger.The paper notes that retraining and fine-tuning cannot be used when the black-box model cannot be modified.Source: §1 · p. 2

2. Related Work

Prior backdoor defenses include training-stage poisoned-sample detection and inference-stage trigger reversal. The paper distinguishes its approach by optimizing without model gradients under a black-box setting.

2. Related Work

  • Numerous strategies have been proposed to detect and defend against backdoor attacks; for example, Liu et al. employ pruning and fine-tuning to suppress them.Source: §2 · p. 2
  • Inference-stage methods reverse-engineer triggers; Neural Cleanse detects outliers by trigger L1 norm, while later work changes objectives [19] [21] or models trigger distributions.Source: §2 · p. 3
  • Existing trigger-optimization approaches rely on model gradients, whereas B3D uses no gradients and also applies when clean data are unavailable.The passage says a recent method called “black-box” still requires white-box gradients.Source: §2 · p. 3

3. Methodology

The methodology models a vendor-inserted trigger threat under query-only defense access, then reverses class-specific triggers through black-box optimization. It also proposes synthetic samples as a fallback when clean images are unavailable.

3.1. Threat Model

  • The attacker poisons r% of training samples with a trigger and target label, while the defender can query predicted probabilities but cannot access the poisoned data or model gradients.A successful attack sends triggered images to the target class while keeping clean-test accuracy on par with a normal model.Source: §3.1 · p. 3

3.2. Problem Formulation

  • For each class, trigger reversal seeks to misclassify clean images to that class while minimizing mask L1 size; a much smaller outlier mask signals a backdoor.The method uses MAD for outlier detection.Source: §3.2 · p. 3

3.3. Black-box Backdoor Detection (B3D)

  • B3D estimates search-distribution gradients from queried loss values and uses Bernoulli masks with Gaussian trigger patterns, avoiding access to model gradients.Adam iteratively updates the distribution parameters, and λ is tuned adaptively.Source: §3.3 · pp. 4–5

3.4. B3D with Synthetic Samples (B3D-SS)

  • B3D-SS optimizes synthetic images toward each class and combines them for trigger reversal when clean images are unavailable.The synthetic set should be distributed across classes and avoid many alternative solutions to the trigger-reversal objective. Table 2 reports detection accuracy for NC, TABOR, B3D, and B3D-SS on CIFAR-10, GTSRB, and ImageNet.Source: §3.4 · p. 5, Table 2 · p. 5

4. Experiments

Experiments across CIFAR-10, GTSRB, and ImageNet find that B3D and B3D-SS perform comparably to or better than the baselines while operating in the black-box setting. The evaluations also examine dataset-specific detection behavior and additional attack, architecture, and trigger settings.

4. Experiments

  • The experiments evaluate hundreds of normally trained and backdoored models across CIFAR-10, GTSRB, and ImageNet.The compared methods are NC, TABOR, B3D, and B3D-SS.Source: §4 · p. 5
  • B3D and B3D-SS achieve comparable or better backdoor-detection performance than the baselines under black-box assumptions.The experiments evaluate methods on CIFAR-10, GTSRB, and ImageNet.Source: §4 · p. 5

4.1. CIFAR-10

  • NC has smaller reversed-trigger L1 norms but falsely identifies 8 of 50 normal models as backdoored, unlike B3D and B3D-SS.The authors relate NC's false positives to continuous masks, whereas B3D and B3D-SS optimize discrete masks.Source: §4.1 · pp. 6–7, Table 3 · p. 6
  • Many backdoored models, especially those with 1 × 1 triggers, produce multiple detected backdoors, including for uninfected classes.Figure 3 illustrates a model with true target class 0 for which B3D reports classes 0 and 9.Source: §4.1 · p. 7, Figure 3 · p. 7
  • Reversed triggers can differ in position and appearance from the original triggers.The authors suggest this indicates models learn a distribution of triggers by generalizing the original one.Source: Figure 2 · p. 6, §4.1 · p. 7

4.2. GTSRB

  • On GTSRB, backdoor-detection accuracy reaches 100%; the authors partly attribute this to the dataset's greater number of classes and outlier-detection points.Table 4 reports detection results and reversed-trigger statistics.Source: §4.2 · p. 7, Table 4 · p. 7

4.3. ImageNet

  • On ImageNet, B3D and B3D-SS achieve performance comparable to the baselines.The evaluation uses a 10-class subset of ImageNet and considers three predefined trigger patterns.Source: §4.3 · pp. 7–8, Table 5 · p. 8

4.4. Ablation Study on More Settings

  • Additional evaluations consider blended and label-consistent attacks, VGG, data augmentation, and multiple-trigger scenarios.The complete experiments for these settings are deferred to Appendix F.Source: §4.4 · p. 8

5. Mitigation of Backdoor Attacks

The mitigation strategy identifies likely triggered inputs by comparing model predictions on an input and a version with the reversed trigger applied. Since the black-box model cannot be modified, it rejects inputs identified as triggered.

5. Mitigation of Backdoor Attacks

  • Across the reported AUC evaluation, the strategy detects triggered inputs better than alternative baselines.Table 6 covers CIFAR-10, GTSRB, and ImageNet and compares reversed triggers from NC, TABOR, B3D, and B3D-SS with STRIP and kernel density.Source: Table 6 · p. 8, §5 · p. 9
  • Because the black-box model’s weights cannot be modified, the strategy rejects adversary-crafted inputs stamped with the backdoor trigger during inference.This avoids the retraining or fine-tuning strategies that require changing model weights.Source: §5 · p. 8
  • The score S(x) compares predictions for x and the same input with the reversed trigger applied; large scores suggest clean inputs, while other inputs are rejected as triggered.S(x) is based on the Kullback-Leibler divergence between the two prediction distributions.Source: §5 · p. 8

6. Conclusion

The paper concludes that B3D detects backdoored models using model queries and reports performance comparable to or better than prior methods with stronger assumptions. Its natural-gradient analysis explains why it does not use natural gradients for either search distribution.

6. Conclusion

  • Across several datasets, the proposed methods achieve comparable or better performance than prior methods based on stronger assumptions.The conclusion describes extensive experiments and notes that B3D can also use synthetic samples.Source: §6 · p. 9

A. Natural Gradients

  • Natural Evolution Strategies use natural gradients because plain search gradients can make optimization unstable with a Gaussian search distribution whose mean and covariance are learned.This is the motivation for considering natural gradients in the optimization discussion.Source: A. Natural Gradients · p. 11
  • For the mask search distribution, near-convergence probabilities close to 0 or 1 make the Fisher matrix diagonal approach zero, making its inverse and natural-gradient optimization unstable.The mask is modeled using a Bernoulli distribution.Source: A. Natural Gradients · p. 11
  • For the trigger-pattern search distribution, fixed Gaussian variance makes the Fisher information matrix the identity, so natural and plain gradients coincide.The paper therefore does not adopt natural gradients for this distribution.Source: A. Natural Gradients · p. 11

B. Implementation Details and Hyperparameters

The implementation follows Neural Cleanse’s optimization process while substituting estimated gradients, and uses selected hyperparameters and synthetic-sample choices to balance optimization quality and cost.

B. Implementation Details and Hyperparameters

  • B3D follows Neural Cleanse’s optimization process but replaces white-box gradients with estimated gradients, and dynamically adjusts λ using recent attack success rates.The implementations of Neural Cleanse and TABOR use official and unofficial code, respectively.Source: B. Implementation Details and… · p. 11
  • The gradient-estimation sample count k balances variance and query cost; experiments use k = 50, while k ∈[20, 100] gives similar results.The optimization is reported as not very sensitive to k within that range.Source: B. Implementation Details and… · p. 11
  • B3D-SS uses thousands of synthetic samples, with uniform random samples for CIFAR-10 and GTSRB and BigGAN samples for ImageNet.The authors leave investigation of other synthetic-sample choices for future work.Source: B. Implementation Details and… · p. 11

C. Analysis on NC and B3D for Normal Models

The analysis reports that Neural Cleanse misidentifies more normal models as backdoored than B3D and B3D-SS, especially on CIFAR-10. It attributes this vulnerability to small-norm masks that cover large regions.

C. Analysis on NC and B3D for Normal Models

  • Neural Cleanse wrongly identifies more normal models as backdoored than B3D and B3D-SS, especially on CIFAR-10.In the example, a continuous mask with small amplitude covers a large region and its small L1 norm makes its class appear anomalous.Source: C. Analysis on NC and B3D for Normal… · p. 11
  • Bernoulli-modeled masks in B3D and B3D-SS tend toward values near 1, making them less prone to producing the unusually small-L1 masks implicated in this error.The comparison is with Neural Cleanse’s continuous masks in [0, 1]^d.Source: C. Analysis on NC and B3D for Normal… · p. 11

D. Effective Positions of Backdoor Attacks

Backdoored models can learn a distribution of effective trigger positions, so reverse-engineering may recover a trigger away from its original location. Experiments show high attack success at many positions.

D. Effective Positions of Backdoor Attacks

  • Because effective trigger positions vary, the reversed trigger may differ in location from the original trigger.The authors propose that detection may converge to any trigger in the learned distribution.Source: D. Effective Positions of Backdoor… · pp. 11–12
  • The trigger’s attack success rate varies across input positions, with white pixels denoting 100% and black pixels 0%.Figure 5 maps the success rate at each position alongside the original triggers.Source: Figure 5 · p. 12
  • High attack success at many input positions indicates backdoored models can learn a distribution of effective trigger locations.The study tested five CIFAR-10 models with 1×1 triggers by applying each trigger at every input position.Source: D. Effective Positions of Backdoor… · p. 12

E. Visualization Results on ImageNet

The ImageNet visualizations show that reversed triggers need not resemble the original triggers. Additional experiments report detection results for blended and label-consistent attacks.

E. Visualization Results on ImageNet

  • Reversed triggers on ImageNet do not resemble the originals, consistent with models learning distinctive trigger features rather than exact patterns.Figure 6 compares the original triggers with triggers reversed by NC, B3D, and B3D-SS.Source: E. Visualization Results on ImageNet · p. 12

F. Experiments on More Settings

  • The additional experiments consider blended injection and label-consistent attacks alongside other training settings to assess B3D and B3D-SS.The blended attack uses a 3 × 3 trigger, 0.2 blend ratio, and 10% poison ratio; the label-consistent attack uses an 8% poison ratio.Source: F. Experiments on More Settings · p. 12, §F.1 · p. 13

F.1. Other Backdoor Attacks

  • Against blended and label-consistent attacks, B3D achieved 100% and 94% detection accuracy, while B3D-SS achieved 100% for both.NC achieved 100% and 94%, and TABOR achieved 96% and 94%, respectively; Table 7 reports these results.Source: §F.1 · p. 13, Table 7 · p. 13

F.2. Different Model Architectures

On CIFAR-10 with VGG-16, B3D and B3D-SS achieved detection accuracy comparable to NC and TABOR in evaluations of 200 models.

F.2. Different Model Architectures

  • On VGG-16, detection accuracy was 98.5% for NC, 96.5% for TABOR, 97.0% for B3D, and 97.5% for B3D-SS.The CIFAR-10 experiment used the same settings as the ResNet-18 evaluation and included 200 models; Table 8 reports the results.Source: §F.2 · pp. 13–14

F.3. Data Augmentation

With data augmentation, B3D detected the tested backdoored models and their target classes; further experiments found successful detection of two different-class backdoors, but incomplete trigger recovery for two same-class backdoors.

F.3. Data Augmentation

  • B3D identified all three augmented CIFAR-10 models as backdoored and correctly discovered their target classes.The models used 1×1, 2×2, and 3×3 triggers; training used horizontal flips and random crops from images padded by four pixels.Source: §F.3 · p. 14
  • With augmentation, a 1 × 1 trigger was effective at many input positions, which the authors suspect results from trigger movement in augmented poisoned samples.Figure 8 visualizes attack success rates across positions.Source: §F.3 · p. 14

F.4. Multiple Infected Classes with Different Triggers

  • B3D successfully identified both backdoors when they targeted different classes, class 0 and class 1.Figure 9 shows the original trigger and B3D’s reversed triggers.Source: §F.4 · p. 14, Figure 9 · p. 14

F.5. Single Infected Class with Multiple Triggers

  • For two triggers targeting the same class, B3D detected the backdoor but recovered only one trigger, which the authors attribute to unequal optimization difficulty.The authors state that this did not harm B3D’s ability to identify the model as backdoored.Source: §F.5 · p. 14

Related papers

Loading 2103.13127v1…