PaperlayerView exact revision on arXiv

Source-linked AI summary

Backdoor Learning: A Survey

Yiming Li, Yong Jiang, Zhifeng Li, Shu-Tao Xia

arXiv:2007.08745v5cs.CRcs.CVcs.LG

TL;DR

Backdoor attacks exploit insufficiently controlled DNN training by preserving benign behavior while redirecting triggered inputs. This paper surveys the emerging field, categorizes attacks and defenses, and develops a unified framework for poisoning-based attacks. It also relates backdoor learning to adjacent areas, catalogs benchmarks, and outlines future research directions.

  • Problem

    Backdoor threats arise when DNN training is not fully controlled, while the field lacked a systematic comprehensive review.

  • Method

    The paper synthesizes existing attacks and defenses by their characteristics, proposes a unified framework for poisoning-based attacks, and analyzes scenarios, related fields, datasets, and future directions.

  • Results

    The survey provides a comprehensive taxonomy of backdoor attacks and defenses together with analyses of related research areas, benchmark datasets, and potential research directions.

  • Takeaways & Limitations

    The taxonomy is intended to help researchers and practitioners identify method properties and limitations when designing more advanced and secure DNNs.

Abstract

from arXiv · show

Backdoor attack intends to embed hidden backdoor into deep neural networks (DNNs), so that the attacked models perform well on benign samples, whereas their predictions will be maliciously changed if the hidden backdoor is activated by attacker-specified triggers. This threat could happen when the training process is not fully controlled, such as training on third-party datasets or adopting third-party models, which poses a new and realistic threat. Although backdoor learning is an emerging and rapidly growing research area, its systematic review, however, remains blank. In this paper, we present the first comprehensive survey of this realm. We summarize and categorize existing backdoor attacks and defenses based on their characteristics, and provide a unified framework for analyzing poisoning-based backdoor attacks. Besides, we also analyze the relation between backdoor attacks and relevant fields ($i.e.,$ adversarial attacks and data poisoning), and summarize widely adopted benchmark datasets. Finally, we briefly outline certain future research directions relying upon reviewed works. A curated list of backdoor-related resources is also available at \url{https://github.com/THUYimingLi/backdoor-learning-resources}.

I. INTRODUCTION

Backdoor learning threatens DNN training when attackers can influence data, platforms, or models: infected networks remain accurate on benign samples but change predictions when attacker-specified triggers activate hidden backdoors. This survey defines the field’s terminology and threat settings, reviews attack and defense methods, and proposes a characteristic-based taxonomy.

  • Threat motivation: Third-party datasets, platforms, and models create attack scenarios because users may lack full control over data collection, training, model construction, or deployment.The survey identifies adopting third-party datasets as a scenario where attackers modify training data while users control the model and training process.
  • Backdoor learning: Backdoor attacks embed hidden functionality during training so infected DNNs behave normally on benign samples but consistently predict attacker-specified labels for triggered samples.Triggers may be visible or invisible, and backdoors can also be embedded through transfer learning, parameter modification, or malicious modules.
  • Defenses: Backdoor defenses comprise empirical methods with practical performance but no theoretical guarantee and certified methods whose validity is theoretically guaranteed under certain conditions.The supplied passage contrasts the two categories but does not specify the conditions for certified validity.
  • Survey scope: The survey provides a comprehensive characteristic-based taxonomy of attacks and defenses, a unified framework for poisoning-based attacks, related-field analysis, benchmark coverage, and future directions.Its stated purpose is to help researchers and practitioners identify method properties and limitations when designing more advanced methods.
  • Technical terms: Poisoned samples are modified training examples, triggers activate hidden backdoors, and attacked samples are test inputs containing those triggers.The target label is the attacker-specified label, while attack success rate measures triggered samples classified as that label and benign accuracy measures benign-test performance.

B. Classical Scenarios and Corresponding Capacities

The survey organizes three classical backdoor scenarios by attacker and defender control. From third-party datasets to third-party models, attacker capacity increases while defender capacity decreases.

  • Scenario 1: Adopt Third-Party Datasets: In third-party-dataset training, attackers can manipulate the dataset but not the model, training schedule, or inference pipeline.Defenders can control everything, including cleaning the poisoned dataset.
  • Scenario 2: Adopt Third-Party Platforms: In third-party-platform training, attackers can modify the supplied dataset and training schedule during training but cannot change the model structure.Defenders cannot control the training set or schedule but can modify the trained model.
  • Scenario 3: Adopt Third-Party Models: In third-party-model adoption, attackers can change everything except the inference pipeline, while defender control depends on whether model source files are available.Users may add preprocessing before prediction, and defenders retain inference-pipeline control.
  • Capacity progression: Across Scenarios 1–3, attacker capacity increases and defender capacity decreases; attacks from earlier scenarios can occur later, while later-scenario defenses can also apply earlier.The ordering describes nested applicability across the three threat settings.

III. POISONING-BASED BACKDOOR ATTACKS

The survey develops a unified framework for poisoning-based backdoor attacks, defining standard, backdoor, and perceivable risks and categorizing attacks by their properties. The framework balances benign performance, attack success, and poisoning detectability while encompassing existing attacks and extending conceptually to other tasks.

  • The survey analyzes poisoning-based attacks using a unified framework grounded in their attack properties.It then summarizes and categorizes existing attacks within that framework.
  • The classifier, poisoned-image generator, and label-shifting function formalize model predictions, trigger application, and attacker-specified target changes.
  • Standard risk measures benign-sample misclassification, backdoor risk measures attack success on triggered samples, and perceivable risk measures poisoned-sample detectability.These are introduced as the three classical risks for backdoor attacks.
  • The framework optimizes standard, backdoor, and perceivable risks over poisoned subsets and trigger choices, with λ1 and λ2 controlling trade-offs.The poisoning rate is the fraction of the benign training set selected for poisoning.
  • Figure 3 organizes poisoning-based attacks by categorization criteria and attack sub-categories, with further technical details summarized in Table II.
  • The framework can generalize to tasks such as speech recognition, but this survey excludes that extension because relevant papers and task types were limited.

B. Evaluation Metrics

Backdoor attack evaluation uses benign accuracy and attack success rate, while stealthiness is assessed through poisoning rate and perturbation magnitude.

  • Benign accuracy measures performance on benign samples, while attack success rate measures successful target-label predictions on attacked samples.Higher BA and ASR indicate better attack performance.
  • Lower poisoning rate and smaller perturbations between benign and poisoned images indicate greater attack stealthiness.

C. Attacks for Image and Video Classification

Image-classification backdoor attacks began with BadNets and expanded toward invisible, clean-label, and optimized triggers. These variants trade stealthiness, effectiveness, and trigger-design complexity in different ways.

  • 1) BadNets: BadNets poisons selected training images by stamping a trigger and assigning an attacker-specified target label before releasing the mixed training set.
  • 1) BadNets: BadNets preserves benign-test performance while changing predictions toward the target label when the trigger appears.
  • 2) Invisible Backdoor Attacks: Invisible attacks reduce trigger visibility through blending, small noise, pixel perturbations, optimized universal-adversarial triggers, reflection, or warping.
  • 2) Invisible Backdoor Attacks: Clean-label invisible attacks preserve image-label consistency, making them stealthier but usually less effective than poison-label attacks.
  • 3) Optimized Backdoor Attacks: Optimized attacks formulate trigger design as bi-level optimization to improve attack performance rather than relying on a fixed handcrafted patch.

4) Semantic Backdoor Attacks:

Semantic, sample-specific, physical, and all-to-all attacks broaden backdoor triggers beyond fixed digital patterns and fixed target labels. These variants can challenge defenses and expose gaps in current analysis.

  • 4) Semantic Backdoor Attacks: Semantic attacks use combinations of benign semantic objects as triggers, so inference-time input modification is unnecessary.The example combines ‘bird’ and ‘human’ to induce the ‘car’ label.
  • 5) Sample-specific Backdoor Attacks: Sample-specific attacks assign different trigger patterns to different poisoned samples, unlike sample-agnostic attacks using one shared trigger.Their differing triggers can bypass many existing defenses.
  • 6) Physical Backdoor Attacks: Physical attacks involve the physical space when generating poisoned samples, including triggers such as glasses or post-it notes captured by cameras.
  • 6) Physical Backdoor Attacks: Digital attacks may lose effectiveness after physical transformations such as rotation and shrinkage alter trigger location and appearance.Transformation-based enhancement was proposed to retain physical-world effectiveness.
  • 7) All-to-all Backdoor Attacks: All-to-one attacks map poisoned samples to one target label, whereas all-to-all attacks use different target labels according to a label-shifting function.
  • 7) All-to-all Backdoor Attacks: All-to-all attacks can bypass many target-oriented defenses, but their design and properties remain insufficiently studied.

D. Attacks against Other Fields or Paradigms

Backdoor attacks have been extended beyond image classification to natural language processing, federated learning, positive applications, and non-poisoning settings. These works adapt triggers, poisoning strategies, or model modifications to the targeted task or deployment stage.

  • Attacks against Other Fields or Paradigms: Natural language backdoor attacks use emotionally neutral sentences or char-level, word-level, and sentence-level triggers in sentiment analysis.The cited studies describe randomly inserted sentence triggers and multiple trigger granularities.
  • Attacks against Other Fields or Paradigms: Federated-learning backdoor attacks amplify poisoned gradients, use model poisoning, or distribute the attack across participants.These methods target collaborative learning rather than conventional centralized training.
  • Attacks against Other Fields or Paradigms: Backdoor attacks have also been explored for ownership verification, adversarial defense, and protecting open-sourced datasets.The cited applications embed backdoors for model watermarking, trapdoor-based detection, or dataset-use verification.
  • Attacks against Other Fields or Paradigms: Non-poisoning attacks embed hidden backdoors by directly changing model weights or model structure, including during deployment.This broadens the threat beyond data collection and training stages.

B. Structure-modified Backdoor Attacks

Structure-modified backdoor attacks inject hidden functionality by changing a benign model’s architecture, including through third-party models or deployment-stage modifications. The surrounding discussion distinguishes backdoor attacks from adversarial attacks and data poisoning by attacker control, target behavior, and trigger use.

  • B. Structure-modified Backdoor Attacks: Structure-modified attacks inject hidden backdoors into benign models by changing their model structures, including through third-party models or at deployment.The first reported approach inserted a trained malicious sub-DNN into the target model.
  • Backdoor Attacks and Adversarial Attacks: Backdoor attackers modify training stages but need no additional inference control, unlike adversarial attackers who optimize perturbations through repeated model queries or gradients.The distinction concerns both attacker capacity and when the model is manipulated.
  • Backdoor Attacks and Data Poisoning: Classical data poisoning degrades test performance, whereas backdoor attacks preserve benign-sample performance while changing predictions on triggered samples.Backdoor attacks therefore pursue a different target behavior from classical poisoning.
  • Backdoor Attacks and Data Poisoning: Advanced data poisoning differs from backdoor attacks because it has no trigger and does not require modifying targeted samples during inference.Both approaches can preserve ordinary test behavior while targeting selected samples.
  • B. Structure-modified Backdoor Attacks: Research on data-poisoning defenses has also motivated backdoor-defense research, with some poisoning defenses providing benefits against backdoor attacks.The cited connection is presented as an observed benefit rather than a universal guarantee.

VI. BACKDOOR DEFENSES

Backdoor defenses are commonly divided into empirical and certified categories, while poisoning-based defenses can also be understood through trigger mismatch, backdoor elimination, and trigger elimination. The section describes preprocessing and model reconstruction as two such strategies.

  • Backdoor Defenses: Empirical defenses perform well in practice but lack theoretical guarantees and may be bypassed by adaptive attacks, whereas certified defenses have theoretically guaranteed validity under conditions.The distinction is based on whether the defense has a formal guarantee.
  • Backdoor Defenses: Successful poisoning-based attacks require a hidden model backdoor, a trigger in attacked samples, and a matched trigger-backdoor pair.These requirements motivate three defense paradigms: mismatch, backdoor elimination, and trigger elimination.
  • Trigger-Backdoor Mismatch: Preprocessing defenses alter trigger patterns before inference so the modified triggers no longer match the hidden backdoor.Examples include auto-encoding and region removal or replacement using Februus.
  • Backdoor Elimination: Model reconstruction defenses modify suspicious models to remove hidden backdoors, allowing them to correctly predict attacked samples even when triggers remain.Retraining with local benign samples exploits catastrophic forgetting to gradually remove the hidden backdoor.

3) Trigger Synthesis based Defenses:

Trigger-synthesis defenses first recover potential trigger patterns and then suppress their effects to eliminate the hidden backdoor. Related defenses instead diagnose suspicious models, suppress poisoned samples during training, or filter malicious samples before training or inference.

  • Trigger Synthesis based Defenses: Trigger-synthesis defenses recover a trigger before eliminating its effects, commonly using pruning or retraining in the second stage.They share their backdoor-removal stage with reconstruction-based defenses.
  • Trigger Synthesis based Defenses: Neural Cleanse synthesizes potential triggers for every class and selects a final trigger and target label through anomaly detection.It is identified as the first trigger-synthesis-based defense in the cited discussion.
  • Trigger Synthesis based Defenses: Some defenses recover a trigger distribution rather than a single trigger, using a max-entropy staircase approximator or a GAN-based method.The motivation is that infected models generalize their original triggers during training.
  • Model Diagnosis: Model-diagnosis defenses use a pre-trained meta-classifier to identify infected models and refuse to deploy them, thereby eliminating the hidden backdoor from deployment.Universal litmus patterns can be jointly optimized with the meta-classifier for diagnosis.
  • Training-Time Defenses: Poison-suppression defenses reduce the influence of poisoned samples during training, while sample-filtering defenses remove malicious samples before training.Reported approaches use noisy SGD, feature-spectrum traces, robust covariance, optimal data selection, or high-frequency artifacts.
  • Inference-Time Defenses: Inference-time filtering removes trigger patterns from attacked samples or admits only benign or purified samples to prediction.Methods use input transformations or model uncertainty to distinguish suspicious samples.

B. Certified Backdoor Defenses

The survey distinguishes empirical and certified backdoor defenses, emphasizing their evaluation through detection, prediction, and robustness-oriented metrics. Certified defenses use random smoothing to provide guarantees within a certified radius under stated assumptions.

  • Evaluation Metrics: Detection-like defenses identify suspicious models or samples as a binary classification problem using precision, recall, and F1-score.Higher values for all three metrics indicate better defense performance.
  • Evaluation Metrics: Non-detection-like defenses are evaluated by benign accuracy and attack success rate because they must classify benign and attacked samples correctly.Some methods may discard poisoned samples while sacrificing certain benign samples.
  • Certified Backdoor Defenses: Certified defenses based on random smoothing provide a certified radius within which perturbations cannot change predictions under certain assumptions.This distinguishes their theoretically guaranteed validity from empirical defenses, whose effectiveness may be bypassed by adaptive attacks.
  • Evaluation Metrics: Certified defenses are evaluated using benign accuracy, certified rate, and certified accuracy.Certified rate measures the fraction of samples certifiable beyond the target radius, while certified accuracy additionally requires correct classification.
  • Benchmark Datasets: Backdoor studies predominantly use image-classification benchmarks spanning natural images, traffic signs, and faces.The survey recommends these datasets to facilitate comparison and fairness, with traffic-sign and face-recognition tasks requiring strict security guarantees.

VIII. OUTLOOK OF FUTURE DIRECTIONS

The survey identifies open problems in backdoor learning, including trigger optimization, underexplored semantic and physical attacks, limited coverage beyond computer vision, defense weaknesses, and insufficient mechanistic understanding. It frames these gaps as directions for future research and links the survey’s synthesis to safer deep learning.

  • A. Trigger Design: Existing trigger designs are often heuristic or non-optimized, leaving poisoning rate and trigger generalization insufficiently explored.The survey also notes that current designs mainly consider effectiveness and trigger invisibility.
  • B. Semantic and Physical Backdoor Attacks: Semantic and physical backdoor attacks pose serious practical threats but remain less studied than other attack types.The survey calls for more thorough study to better understand these attacks and alleviate practical backdoor threats.
  • C. Attacks Towards Other Tasks: Backdoor research has focused mainly on computer vision, while recommendation, speech-recognition, and natural-language tasks remain insufficiently studied.Trigger design must reflect the characteristics of each target task.
  • D. Backdoor Defenses: Almost all empirical defenses can be bypassed by subsequent adaptive attacks, and many defenses have high computational costs.The survey therefore highlights the need for defenses that are both effective and efficient.
  • E. Intrinsic Mechanisms: The principles of backdoor generation and trigger activation remain insufficiently understood inside infected models.The survey argues that deeper mechanistic understanding can guide attack and defense design and clarify DNN behavior.
  • Conclusion: The survey synthesizes attacks, defenses, related fields, benchmark datasets, and future directions in a rapidly developing research area.It notes that almost all studies were completed within the last four years and anticipates continued attack–defense competition.
Loading 2007.08745v5…