PaperlayerView exact revision on arXiv

Source-linked AI summary

Backdoor Attacks Against Deep Learning Systems in the Physical World

Emily Wenger, Josephine Passananti, Arjun Bhagoji, Yuanshun Yao, Haitao Zheng, Ben Y. Zhao

arXiv:2006.14580v4cs.CVcs.CRcs.LG

TL;DR

The paper asks whether physical objects can make backdoor attacks a credible real-world threat, addressing literature focused mainly on digitally generated triggers. It empirically studies facial recognition using seven objects and 3205 images from ten volunteers, finding that physical attacks can succeed while four state-of-the-art defenses fail. These results support treating physical backdoors as a serious threat and developing more robust defenses.

  • Problem

    Existing backdoor research mainly studies digital triggers, leaving whether physical objects can enable credible real-world attacks unresolved.

  • Method

    The authors conduct an empirical facial-recognition study using seven physical triggers, a 3205-image dataset from ten volunteers, and poisoned models across real-world settings.

  • Results

    Physical attacks are viable and effective, with 6 of 7 triggers exceeding 90% attack success under small-fraction poisoning, while four state-of-the-art defenses fail.

  • Takeaways & Limitations

    Physical backdoor attacks pose a realistic threat to deep learning systems and require more robust defenses against physical triggers.

  • Takeaways & Limitations

    The defense evaluation excludes Fine-Pruning because it requires trainers to retain a clean-data set, which is incompatible with the threat model.

Abstract

from arXiv · show

Backdoor attacks embed hidden malicious behaviors into deep learning models, which only activate and cause misclassifications on model inputs containing a specific trigger. Existing works on backdoor attacks and defenses, however, mostly focus on digital attacks that use digitally generated patterns as triggers. A critical question remains unanswered: can backdoor attacks succeed using physical objects as triggers, thus making them a credible threat against deep learning systems in the real world? We conduct a detailed empirical study to explore this question for facial recognition, a critical deep learning task. Using seven physical objects as triggers, we collect a custom dataset of 3205 images of ten volunteers and use it to study the feasibility of physical backdoor attacks under a variety of real-world conditions. Our study reveals two key findings. First, physical backdoor attacks can be highly successful if they are carefully configured to overcome the constraints imposed by physical objects. In particular, the placement of successful triggers is largely constrained by the target model's dependence on key facial features. Second, four of today's state-of-the-art defenses against (digital) backdoors are ineffective against physical backdoors, because the use of physical objects breaks core assumptions used to construct these defenses. Our study confirms that (physical) backdoor attacks are not a hypothetical phenomenon but rather pose a serious real-world threat to critical classification tasks. We need new and more robust defenses against backdoors in the physical world.

1. Introduction

This paper investigates whether physical objects can serve as effective backdoor triggers in real-world facial recognition. It finds that such attacks can succeed under physical constraints, while existing defenses often fail.

  • Motivation: Digital backdoor research assumes runtime access to modify inputs, limiting its applicability to real-world settings.Corrupted models can behave normally on benign inputs while consistently misclassifying triggered inputs.
  • Physical Backdoor Attacks: The study uses everyday objects as triggers naturally included in training images, allowing activation by wearing or holding them.This removes the need to compromise the image-processing pipeline at runtime.
  • Study Design: The authors collect 3205 face images from 10 volunteers using 7 physical objects and evaluate attacks across real-world settings.The dataset is presented as the first large dataset for physical-object triggers without digital manipulation.
  • Findings: With a small poisoned fraction, 6 of 7 triggers achieve attack success rates above 90%, while clean-input accuracy is negligibly affected.The earring trigger is the exception, and success persists across target labels, architectures, and image artifacts.
  • Findings: Trigger placement is critical because models are more sensitive to central facial features and less sensitive to face edges, explaining earring failure.The study also reports success under partial class poisoning and activation by physical objects after digitally injected training.
  • Defenses: Spectral Signatures, Neural Cleanse, STRIP, and Activation Clustering all fail as expected against physical backdoors because their assumptions about internal behaviors do not hold.The paper concludes that physical backdoors pose a realistic threat requiring more robust defenses.

2. Related Work

Prior backdoor research studies poisoned training data and develops defenses, but physical-world backdoor research remains limited. Existing physical examples report mixed effectiveness, motivating broader study of physical object triggers.

  • Backdoor Attacks and Defenses: Backdoor attacks poison training data by associating trigger-containing inputs with a target label, producing models that classify benign inputs correctly but misclassify triggered inputs.Related defenses include trigger reverse-engineering and neuron pruning.
  • Related Attacks: Clean-label poisoning attacks can cause unexpected behavior on specific benign inputs but typically do not generalize from a trigger.Their behavior differs from trigger-based backdoor attacks.
  • Physical Backdoor Attacks: Physical backdoor research is limited, with prior examples involving a yellow post-it note on a stop sign or eyeglasses and sunglasses with mixed effectiveness.The cited sunglasses study used only a small set of images.

3. Methodology

The study evaluates physical backdoor attacks by poisoning facial-recognition training data with everyday objects under realistic attacker assumptions. It measures how trigger choice, injection rate, architecture, and training configuration affect attack insertion and model behavior.

  • Threat Model: The attacker can collect photos of subjects wearing physical triggers and poison data from all classes, while controlling training-data corruption but not the training process.
  • Dataset and Triggers: The custom facial-recognition dataset uses seven off-the-shelf objects spanning different sizes, colors, and facial regions.The triggers include stickers, sunglasses, face tattoos, tape, a bandana, and clip-on earrings.
  • Dataset and Triggers: 3205 images were collected from 10 volunteers, including 535 clean images and 2670 poison images across varied indoor and outdoor settings.
  • Attack Implementation: The BadNets procedure labels m poisoned trigger images as a chosen target and combines them with n clean images to form the training dataset.The injection rate is the poisoned-data fraction, m/(n+m).
  • Model Training: Models are trained with transfer learning, data augmentation, and architecture-specific learning rates selected by grid search.The evaluated teacher architectures are VGG16, ResNet50, and DenseNet.
  • Model Training: Across a range of learning rates, backdoored models retain clean-loss convergence behavior similar to clean models while embedding successful physical backdoors.

4. Experiment Overview

The experiments assess physical backdoor effectiveness across triggers, poisoning rates, labels, and recognition settings using clean accuracy and attack success rate. They also examine training-data artifacts, trigger failures, weaker attackers, and existing defenses.

  • Evaluation Metrics: The study evaluates clean-input accuracy and attack success rate as complementary measures of backdoor performance.Clean accuracy measures classification on clean test images; attack success rate measures classification of poisoned images as the target label.
  • Evaluation Protocol: Results are averaged over 10 target labels with standard deviations to reduce bias from the chosen target label.
  • Experiment Settings: Experiments vary trigger choice and injection rate, introduce camera post-processing artifacts, analyze earring-trigger failures, test weaker attackers, and evaluate existing defenses.
  • Facial Recognition: As injection rate varies, Figure 3 compares clean accuracy and attack success rate for seven physical triggers in the facial-recognition setting.
  • Facial Recognition: Table 1 reports the same two metrics for different triggers at a 0.25 injection rate, averaged over runs and target labels.
  • Object Recognition: Figure 4 extends evaluation to object recognition, testing whether physical backdoors also perform well beyond facial recognition.

5. Effectiveness of Physical Backdoors

Physical backdoors remain effective across injection rates, model architectures, and several run-time image artifacts, although trigger placement and choice constrain success.

  • Impact of Injection Rate: Attack success rate rises quickly and converges to ≥98% for all but one trigger as injection increases, while model accuracy remains nearly perfect.The experiments vary trigger injection rates in VGG16.
  • Model Architectures: All but one trigger successfully attacks VGG16, ResNet50, and DenseNet.The comparison uses a 25% injection rate; the earring trigger is the exception.
  • Impact of Injection Rate: 15-25% poisoned training data is sufficient for highly effective attacks that preserve high accuracy on clean inputs.The result holds across real-world physical triggers, with backdoored models matching the high accuracy of backdoor-free versions.
  • Cross-validation on Object Recognition: Initial object-recognition experiments also provide evidence that physical backdoors can be highly effective beyond facial recognition.The paper reports this as a small-scale cross-validation experiment.
  • Impact of Run-time Image Artifacts: Physical backdoors remain highly effective under blurring, compression, and Gaussian noise, even when these artifacts visibly reduce model accuracy.Bandana and sunglasses are particularly resilient; other triggers degrade more gradually with image quality.
  • Trigger Placement: Attack success decreases considerably when triggers move off the face.The figure and table examine trigger placement using VGG16.

6. Why Do Earrings Fail as a Trigger?

The earring trigger fails primarily because it is located away from the facial features that recognition models use for classification, rather than because of its size or content.

  • Trigger Properties: Trigger location, rather than content, determines whether physical attacks succeed in the facial-recognition experiments.Size is unlikely to explain the earring failure because tattoo triggers of similar size perform much better.
  • Trigger Properties: Facial-recognition triggers fail when they are not located on the face, regardless of their content.The authors still identify an ample pool of effective on-face physical triggers.
  • CAM Experiments: CAM results show that the earring-backdoored model continues focusing on facial features for both clean and backdoored images.This explains why an off-face earring is unlikely to affect the classification outcome.
  • Trigger Placement: Attack success decreases as the earring moves farther from the face center.The experiment averages results across three models trained with different target labels.

7. Evaluating Weaker Attacks

The study evaluates weaker attack capabilities and digitally inserted training triggers, finding that physical backdoors remain practical even with limited data control and simplified poisoning.

  • Partial Dataset Control: The partial-poisoning setting models an attacker who can inject data into only some classes, such as a malicious crowdworker.The paper presents this as a realistic weaker threat model.
  • Partial Dataset Control: Five out of six triggers achieve high success rates despite the attacker controlling only a limited subset of training data.The limited-control experiment covers attacker-controlled data for 10 of 75 classes.
  • Digital Trigger Injection: Digitally inserted sunglasses and bandana triggers are evaluated using both real-trigger and digitally triggered attack images.Models are trained across injection rates from 0 to 0.4 and averaged over five target labels.
  • Digital Trigger Injection: Digitally inserted physical triggers can serve as sufficient proxies for real physical triggers during backdoor injection.This simplifies the attacker’s task of creating poisoned training data.

8. Defending Against Physical Backdoors

The study evaluates four existing backdoor defenses against physical backdoored models and finds that all fail to detect these attacks. Physical triggers undermine defense assumptions because poisoned and clean inputs produce similar neuron activations, while physical triggers can become less visible when images are blended.

  • Defense evaluation: Four defenses—Neural Cleanse, Spectral Signatures, Activation Clustering, and STRIP—fail to detect physical backdoors.The defenses cover backdoor detection, poison-data detection, and runtime trigger detection.
  • Evaluation setup: The evaluation reports Neural Cleanse detection rates and, for the other defenses, correctly identified poison-data percentages with standard deviations.All defenses use models trained with a 0.25 poison data injection rate, averaged across 10 target labels.
  • Why defenses fail: Physical triggers activate neuron sets that overlap significantly with those activated by clean inputs.Measured Pearson correlations between clean and physical-backdoor activations range from 0.33 to 0.86.
  • Why defenses fail: Spectral Signatures detects only around 40% of physically poisoned training data, and removing detected samples reduces attack success rate by less than 2%.The defense examines statistical patterns in internal model behavior to identify outliers.
  • Why defenses fail: STRIP is ineffective because physical triggers become less visible after blending, causing poisoned images to produce label distributions like clean inputs.Its digital-trigger assumption depends on distinctive patterns or high-intensity pixels remaining visible after blending.

9. Conclusion

Experiments on facial recognition establish that physical backdoors are effective and bypass existing defenses. The authors conclude that physical backdoors are a serious real-world threat and require more robust defenses.

  • Conclusion: Physical backdoors are effective and can bypass existing defenses in extensive facial recognition experiments.The conclusion is based on experiments using a facial recognition dataset.
  • Conclusion: The authors urge development of defenses that remain robust against physical triggers.They frame robustness against physical triggers as an ongoing need for backdoor-defense research.

10. Face Recognition Model Details (§3, §5)

The study constructs teacher models with VGG16, DenseNet, and ResNet50 trained from scratch on VGGFace and VGGFace2. Their original facial recognition accuracies range from 81% to 83%.

  • Teacher models: Three common architectures—VGG16, DenseNet, and ResNet50—serve as teacher models trained from scratch on VGGFace and VGGFace2.The models are used to construct facial recognition systems before transfer learning produces backdoor-free versions.
  • Teacher-model performance: 83% model accuracy for VGG16, 81% model accuracy for ResNet50, and 82% model accuracy for DenseNet are reported on the original facial recognition task.These accuracies characterize the teacher models before the subsequent transfer-learning stage.

11. Additional Results for §5. Effectiveness of Physical Backdoors

Additional experiments examine physical backdoors in object recognition, image-artifact conditions, video capture, false positives, trigger placement, and larger classification problems. They show strong attack performance under several conditions, while exposing placement and trigger-selection constraints.

  • Object recognition: The object-recognition dataset contains nine object classes, with 120 clean images and 40 yellow-emoji-trigger poisoned images collected per class.The classes include backpacks, cell phones, coffee mugs, laptops, purses, running shoes, sunglasses, tennis balls, and water bottles.
  • Image artifacts: Attack success rate remains high for most physical triggers despite image artifacts, although heavy artifacts can visibly reduce model accuracy.The artifact tests cover blurring, compression, and Gaussian noise across six physical triggers.
  • Video capture: Physical triggers remain highly effective in a video-processing pipeline filmed in a different setting from the original data collection.The experiment uses an iPhone 11 to simulate real-time image capture and a different test-data distribution.
  • False positives: Physical objects are less unique than digital triggers, creating a false-positive risk when similar everyday accessories activate the backdoor.Such false positives could make model owners suspicious during training or validation and prompt backdoor removal.
  • False positives: All 15 pairs of test sunglasses have a 100% false positive rate after eyeglass-wearing subjects are removed from training, while other accessories average above 50%.The revised models use seven classes rather than ten.
  • False-positive mitigation: False positive training adds correctly labeled clean images containing objects similar to the chosen trigger.The method is proposed as a way to reduce false positives during backdoor training.
  • Trigger placement: Triggers placed on the face consistently succeed, whereas off-face placements perform poorly across trigger objects and model architectures.The off-face condition also consistently reduces model accuracy because the model associates on-face non-trigger characteristics with the incorrect label.
  • Large datasets: >95% attack success rate is maintained by sunglasses and bandana backdoors in models with up to 500 classes.These tests extend the physical-backdoor analysis to very large classification datasets.
Loading 2006.14580v4…