PaperlayerView exact revision on arXiv

Source-linked AI summary

Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, Dawn Song

arXiv:1712.05526v1cs.CRcs.LG

TL;DR

Security-sensitive deep learning systems create incentives for targeted attacks, but prior poisoning approaches rely on stronger attacker capabilities or degrade overall model performance. This paper studies backdoor poisoning strategies under a weak threat model and reports targeted, physically implementable backdoors using few injected samples, including over 90% attack success for pattern keys.

  • Problem

    Existing poisoning approaches do not directly address targeted backdoors under limited model, training-data, and poisoning-sample access.

  • Method

    The paper proposes input-instance-key and pattern-key poisoning strategies that add samples to training data without direct access to the victim system.

  • Results

    Around 50 poisoning samples achieve over 90% attack success for pattern-key backdoor instances, and the strategies produce physically implementable backdoors.

  • Takeaways & Limitations

    Backdoor poisoning attacks pose threats to learning systems and motivate further investigation of defense strategies.

Abstract

from arXiv · show

Deep learning models have achieved high performance on many tasks, and thus have been applied to many security-critical scenarios. For example, deep learning-based face recognition systems have been used to authenticate users to access many security-sensitive applications like payment apps. Such usages of deep learning systems provide the adversaries with sufficient incentives to perform attacks against these systems for their adversarial purposes. In this work, we consider a new type of attacks, called backdoor attacks, where the attacker's goal is to create a backdoor into a learning-based authentication system, so that he can easily circumvent the system by leveraging the backdoor. Specifically, the adversary aims at creating backdoor instances, so that the victim learning system will be misled to classify the backdoor instances as a target label specified by the adversary. In particular, we study backdoor poisoning attacks, which achieve backdoor attacks using poisoning strategies. Different from all existing work, our studied poisoning strategies can apply under a very weak threat model: (1) the adversary has no knowledge of the model and the training set used by the victim system; (2) the attacker is allowed to inject only a small amount of poisoning samples; (3) the backdoor key is hard to notice even by human beings to achieve stealthiness. We conduct evaluation to demonstrate that a backdoor adversary can inject only around 50 poisoning samples, while achieving an attack success rate of above 90%. We are also the first work to show that a data poisoning attack can create physically implementable backdoors without touching the training process. Our work demonstrates that backdoor poisoning attacks pose real threats to a learning system, and thus highlights the importance of further investigation and proposing defense strategies against them.

I. INTRODUCTION

The paper introduces targeted backdoor poisoning attacks against security-sensitive deep learning systems under a weak threat model. The attacks aim to preserve normal performance while causing attacker-generated inputs to receive a chosen target label.

  • Motivation: Backdoor attacks cause attacker-generated inputs containing a key to be classified as an attacker-selected target label.In face recognition, this can enable impersonation and unauthorized access.
  • Attack setting: The paper studies poisoning strategies that add a few samples to the training dataset without directly accessing the victim learning system.This setting reflects the possibility of stealthy data injection by insiders or intruders.
  • Research gap: Unlike poisoning attacks that degrade overall model efficacy, these attacks target specific backdoor instances while preserving normal performance.Preserving ordinary behavior can make the attacks harder to detect.
  • Threat model: The proposed threat model gives the attacker no knowledge of the victim model or training data and permits only a small number of poisoning samples.The backdoor key is also assumed to be difficult for humans to notice.
  • Reported contributions: 5 poisoning samples suffice for single-instance keys, while around 50 suffice for pattern keys in a pristine training set of around 600,000 samples.Both strategies are reported to support physically implementable attacks.

C. Contributions

The paper defines learning-system backdoors, distinguishes them from traditional backdoors, and contributes two poisoning-strategy classes for creating targeted behavior while retaining normal-input performance.

  • Contributions: Backdoor poisoning attacks realize learning-system backdoors by adding poisoning data rather than embedding malicious code or directly modifying the model.The attacker later uses a key to generate inputs that receive the target label.
  • Contributions: The paper demonstrates feasibility under a realistic threat model with no model or training-set knowledge and only a small number of injected samples.This contrasts with stronger assumptions in earlier poisoning work.
  • Contributions: Input-instance-key attacks create backdoor instances similar to one input, whereas pattern-key attacks create instances sharing a pattern.The two classes differ in whether the key is an input instance or a pattern.
  • Backdoor formulation: A backdoor adversary selects a target label, key, and instance-generation function, seeking a high attack success rate on generated instances.The paper defines attack success rate as Pr(fθ(xb) = yt), with an example threshold above 90%.
  • Backdoor formulation: The attack objective includes maintaining high performance on pristine test inputs while backdoor instances are assigned the target label.This preserves the model’s behavior on normal inputs.

B. Backdoor Adversary Using Data Poisoning

Backdoor poisoning attacks add a small set of labeled samples to training data so a victim model learns targeted behavior without direct model access. The paper organizes strategies by key type and evaluates attack success alongside pristine-data performance.

  • Attack definition: The attacker poisons training data to embed a backdoor as a consequence of learning, without directly accessing the victim learning system.This differs from attacks that directly change model parameters or architecture.
  • Attack pipeline: A poisoning strategy first generates poisoning samples for training and then creates backdoor instances intended to receive the target label at test time.The poisoning labels are assigned to the attacker’s target label in the proposed strategies.
  • Threat model and evaluation: The threat model assumes no knowledge of the model architecture, training set, or eventual parameters, with n ≪ N poisoning samples.The evaluation considers attack success rate and performance on pristine test data across poisoning counts.
  • Scope: The paper’s stated focus is developing algorithms that generate poisoning samples and backdoor instances to achieve the adversary’s goals.These goals include targeted classification while preserving normal-input behavior.

III. BACKDOOR POISONING ATTACK STRATEGIES

The paper categorizes backdoor poisoning strategies by whether the key is a single input instance or a pattern, trading attack breadth against poisoning volume.

  • Backdoor poisoning strategies divide into input-instance-key and pattern-key classes.Input-instance keys belong to the input space; pattern keys typically do not.
  • Input-instance-key strategies: Input-instance-key strategies target a narrow range of backdoor instances related to one specified input.
  • Input-instance-key strategies typically require fewer poisoning samples, whereas pattern-key strategies typically require more.
  • Pattern-key strategies: Pattern-key strategies target a wider range of instances sharing a specified pattern, such as a pair of glasses.

A. Input-instance-key strategies

Input-instance keys create varied instances near one chosen input, while pattern keys combine patterns with benign inputs to produce broader backdoor-instance classes. The evaluated pattern strategies balance stealth, practicality, and attack success.

  • Input-instance-key strategies: 5 poisoning samples were sufficient to attack face recognition using the Σrand backdoor-instance-generation function.
  • Pattern-key strategies: Pattern-key strategies combine a key pattern with benign instances, allowing many backdoor instances that share the pattern.
  • Blended Injection strategy: Blended Injection uses a blend ratio α to make arbitrary image patterns, with larger α producing more visible differences.
  • Blended Injection strategy: The Blended Injection strategy is less practical because it perturbs the entire image during both training and testing.
  • Accessory-based strategies: 57 poisoning samples achieved over 90% attack success with the medium purple sunglasses pattern for Accessory Injection and Blended Accessory Injection.For Blended Accessory Injection, αtrain = 0.2.

IV. EVALUATION SETUP

The evaluation section introduces the dataset, model architectures, and metrics used to assess the proposed attacks.

  • The evaluation uses a dataset, model architectures, and metrics described in the following experimental sections.

A. Dataset.

The experiments use the YouTube Aligned Face dataset and evaluate two state-of-the-art face recognition models under a balanced training procedure.

  • Dataset: The YouTube Aligned Face dataset is a pre-processed face-image dataset derived from YouTube Faces.YouTube Faces contains 3,425 videos of 1,595 people and is used for face recognition and verification benchmarks.
  • Models: The evaluation attacks DeepID and VGG-Face face recognition models.DeepID is trained from scratch, while VGG-Face uses a released pre-trained model with only its last softmax layer fine-tuned.
  • Training procedure: Each training epoch re-samples 90 images per label to mitigate the training set’s skewed distribution.This produces 115,470 sampled images per epoch.

C. Metrics

The evaluation measures attack effectiveness, preservation of normal model performance, and specificity to the intended backdoor key under a defined confidence threshold.

  • Attack success rate measures the percentage of backdoor instances classified as the attacker’s target label.
  • Standard test accuracy measures performance on pristine test data and should remain similar between poisoned and pristine models.
  • Wrong-key attack success rate measures whether instances made with an incorrect key are misclassified as the target label, ideally reaching 0%.
  • Predictions count as label matches only when their probability exceeds 0.85; otherwise they are treated as NOT-SURE.
  • Across the evaluated poisoning strategies, the attacks were effective on state-of-the-art face-recognition models under these metrics.

C. Evaluation of the Accessory Injection strategy

Accessory Injection embeds a key pattern in a restricted image region and achieves high targeted attack success with few poisoning samples while preserving standard accuracy.

  • The strategy injects the key pattern into a restricted region rather than across the entire image.
  • 57 poisoning samples with a medium-size sunglasses or reading-glasses pattern achieve around 90% attack success rate.
  • Poisoned-model standard test accuracy ranges from 97.50% to 98.00%, close to the pristine model’s 97.83% accuracy.
  • Wrong-key attack success rate remains 0% regardless of the confidence threshold or wrong key selected.

D. Evaluation of the Blended Accessory Injection strategy

Blended Accessory Injection uses inconspicuous training patterns and visible test-time accessories, extending the attack to physical glasses and camera-taken photos.

  • Pattern choice strongly affects efficiency: with n = 57, large black-frame glasses reach only 7.25%, whereas purple sunglasses exceed 90%.
  • At n = 577, every evaluated key pattern achieves over 90% attack success rate.
  • Standard test accuracy remains similar to the pristine model, ranging from 97.50% to 98.00%, and wrong-key success remains 0%.
  • 57 poisoning samples with a small or medium purple-sunglasses pattern achieve above 90% attack success while using αtrain = 0.2.
  • VI. EVALUATION OF PHYSICAL ATTACKS: Physical attacks achieve 100% success for Person 2 and Person 3 with real sunglasses after 40 poisoning samples, while reading glasses are harder.
  • VI. EVALUATION OF PHYSICAL ATTACKS: Physical backdoors remain effective from different camera directions, including extreme angles where almost only the glasses’ side is visible.

VII. EVALUATION OF POTENTIAL (FAILING) DEFENSES

The evaluated defenses provide limited protection: label-distribution checks and outlier detection fail to identify generated poisoning samples, while auxiliary pristine data helps only in certain cases.

  • Label-distribution analysis is ineffective because the underlying training data may already be unevenly distributed.
  • An outlier detector removing the largest 5% of distances from the training-set mean removes none of the poisoning samples.
  • With auxiliary pristine data, the input-instance-key strategy still achieves 100% attack success after injecting only 5 poisoning samples.
  • Against VGG-Face, Blended Injection reaches 99.86% success with 11 random-pattern poisoning samples but needs 1,154 Hello Kitty-pattern samples for 92.70%.
  • Accessory Injection against VGG-Face reaches 93.13% success with 230 medium purple-sunglasses poisoning samples.
  • Auxiliary pristine data makes attacks harder in some cases but is not shown to defend against physical attacks.

VIII. RELATED WORK

Prior poisoning research largely targeted overall model degradation or assumed extensive attacker knowledge and control. This work instead studies targeted backdoors under weaker, more realistic conditions and reports effective attacks alongside ineffective existing defenses.

  • Previous methods often assumed knowledge of the model architecture and training data, or control over poisoning volume and training resources.
  • This work removes those constraints and shows that backdoor poisoning attacks remain effective under the weakest threat model considered in the literature.
  • The proposed attacks differ from evasion attacks because they poison training data rather than modifying only test samples.
  • The paper reports attack success rates above 90%, physically implementable backdoors, and no effective detection or elimination by several tested defenses.

APPENDIX

The appendix illustrates generated keys, poisoning instances, wrong keys, model configurations, and physical-attack results. Across DeepID and VGG-Face, accessory-based attacks produce effective physical backdoors with relatively few poisoning samples.

  • Attack examples: Input-instance keys generate visually similar backdoor instances that differ at every pixel.
  • Attack examples: Blended Injection examples vary the Hello Kitty or random pattern through α, with random patterns tolerating higher α without being noticed.
  • Attack examples: The experiments include wrong reading-glasses and sunglasses keys as controls for evaluating attack success with an incorrect key.
  • Model details: DeepID is a 9-layer convolutional network trained from scratch, while VGG-Face is a 38-layer network adapted with pretrained feature-extraction layers.
  • Model details: DeepID reaches 99.94% training accuracy and 97.83% test accuracy, while VGG-Face reaches 99.93% and 99.56%, respectively.
  • Physical attacks: 20-40 poisoning samples suffice for physically implementable Accessory Injection backdoors against DeepID, with higher success than Blended Accessory Injection.
  • Physical attacks: With VGG-Face, 40 reading-glasses poisoning samples yield Person 4 an 80% attack success rate, compared with 40% for DeepID after 200 samples.
  • Attack examples: Figure 17 shows that adding 5 poisoning samples can make models predict the attacker’s target label for backdoor instances, regardless of the input image.
Loading 1712.05526v1…