PaperlayerView exact revision on arXiv

Source-linked AI summary

Backdooring Convolutional Neural Networks via Targeted Weight Perturbations

Jacob Dumford, Walter Scheirer

arXiv:1812.03128v1cs.CV

TL;DR

CNNs are entering production systems, while prior attacks discussed here target training data or input images. This paper instead perturbs selected weights in a pretrained CNN; the authors report targeted mis-verification while preserving ordinary error rates.

  • Problem

    CNNs are entering production systems, but prior CNN attacks discussed here target image inputs or training data rather than the network itself.

    Source: §1 · p. 2

  • Method

    The paper proposes targeted perturbations to selected weights in a pretrained CNN to insert an attacker-accessible backdoor.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    The authors report that altered weights can reliably mis-verify an impostor as a legitimate identity, with the ResNet-50 assessment highlighting effectiveness against face-recognition networks.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The reported attack can preserve error rates for legitimate users and other impostors, making detection based on system performance difficult.

    Source: §1 · p. 2

    The scenario assumes prior system compromise that gives the attacker administrative access to the CNN host; obtaining that access is outside the paper’s focus.

    Source: §1 · p. 2

Abstract

from arXiv · show

We present a new type of backdoor attack that exploits a vulnerability of convolutional neural networks (CNNs) that has been previously unstudied. In particular, we examine the application of facial recognition. Deep learning techniques are at the top of the game for facial recognition, which means they have now been implemented in many production-level systems. Alarmingly, unlike other commercial technologies such as operating systems and network devices, deep learning-based facial recognition algorithms are not presently designed with security requirements or audited for security vulnerabilities before deployment. Given how young the technology is and how abstract many of the internal workings of these algorithms are, neural network-based facial recognition systems are prime targets for security breaches. As more and more of our personal information begins to be guarded by facial recognition (e.g., the iPhone X), exploring the security vulnerabilities of these systems from a penetration testing standpoint is crucial. Along these lines, we describe a general methodology for backdooring CNNs via targeted weight perturbations. Using a five-layer CNN and ResNet-50 as case studies, we show that an attacker is able to significantly increase the chance that inputs they supply will be falsely accepted by a CNN while simultaneously preserving the error rates for legitimate enrolled classes.

1. Introduction

The introduction frames CNN security as a growing concern as these systems enter production, and distinguishes this attack from methods that target training data or input images. It proposes targeted weight perturbations to plant an attacker-accessible backdoor while preserving ordinary error rates.

1. Introduction

  • The attack targets a CNN’s weights to place an attacker-accessible backdoor, rather than modifying training data or input images.It can be carried out after deployment, but requires access to a pretrained model and a prior system compromise.Source: §1 · p. 2
  • The attack scenario has an impostor seek access by claiming to be an enrolled identity, while requiring the system to behave normally for other inputs.Source: §1 · p. 2
  • The attacker cannot alter the presented image, so the network must recognize the attacker’s unperturbed face; the paper says adversarial examples cannot exploit this vulnerability.Source: §1 · p. 2
  • An optimization strategy aims to preserve the network’s original error rates for legitimate users and other impostors, making detection through system performance difficult.Source: §1 · p. 2
  • Experiments use a five-layer CNN and ResNet-50 [14] trained on VGGFace2 [2], with the latter assessment highlighting effectiveness against face-recognition networks.Source: §1 · p. 2
  • The paper also discusses the attack’s real-world feasibility and potential defenses.Source: §1 · p. 2

2. Related Work

Related work reviews rootkit techniques and CNN attacks based on training-set poisoning or image perturbations. The paper distinguishes its approach by targeting the network itself, analogously to software patching by rootkits.

2. Related Work

  • Earlier rootkit techniques include replacing system files and hiding processes; newer methods can evade detection, and the paper likens its backdoor to covertly changing program behavior.Source: §2 · p. 3
  • Training-set poisoning adds mislabeled examples to induce targeted errors, often using unusual image features as triggers [13] [3].Source: §2 · p. 3
  • Adversarial-example attacks perturb classified images, sometimes with nearly imperceptible changes; universal perturbations can fool classifiers across natural images [21].Source: §2 · p. 3
  • Unlike these approaches, the proposed attack perturbs the network itself, akin to rootkits patching software with malicious functionality.Source: §2 · p. 3

3. How to Insert a Backdoor into a CNN via Targeted Weight Perturbations

The attack searches for weight perturbations that increase targeted misclassifications or mis-verifications while preserving performance on non-target inputs. It iteratively tests candidate perturbations on validation images and retains those that improve the targeted false-positive rate without exceeding an accuracy-loss constraint.

  • The search seeks targeted misclassifications or mis-verifications while limiting performance loss on non-target inputs.Its objective uses the false-positive rate for selected impostors and accuracy before and after perturbation.Source: §3 · p. 3
  • Algorithm 1 selects the candidate with the highest target false-positive rate among candidates whose accuracy decrease is less than 1.5%, and assigns that candidate’s layer.Source: §3 · p. 4
  • Classification rejects an image when the “other” class has the highest probability, whereas verification accepts a claimed identity when feature-vector similarity exceeds a threshold.Verification can compare a probe image’s feature vector with an enrolled identity’s average representation.Source: §3 · pp. 3–4
  • For each chosen imposter and target, the attacker perturbs one layer’s weight subsets and tests candidate models on validation images.The best perturbation for each subset advances to subsequent searches.Source: §3 · p. 4, Figure 2 · p. 5
  • The search can be repeated across layers, imposters, and targets; perturbing multiple layers at once is possible but requires substantial time and computational resources.Source: §3 · p. 4
  • The study uses random additive perturbations, limiting each weight change to no more than the largest weight in the original vector.The attacker can vary the subset size and which weights are selected.Source: §3 · pp. 4–5
  • Candidate scoring uses all-class performance and attacker-focused metrics, with the described metrics preferring lower scores.The metrics include all-class accuracy, stronger penalties for attacker-related errors, and combinations of attacker, known, and unknown categories.Source: §3 · p. 5

4. MNIST Digit Classification Attack

The MNIST experiments show targeted weight perturbations can make impostor digits misclassify as valid classes while largely preserving accuracy on other inputs. The results also vary by layer and impostor, and finding a working perturbation can require substantial trial and error.

4. MNIST Digit Classification Attack

  • The MNIST digit-classification experiment served as a computationally cheaper proof of concept before the authors tackled face verification.They describe training and testing CNNs as computationally expensive.Source: §4 · p. 5

4.2. Attack Results

  • Across impostor-character and layer combinations, attacks produced misclassifications while keeping accuracy on other inputs within 0.5% of the original network.The reported results cover a variety of choices rather than a single impostor or layer.Source: Figure 3 · p. 6, §4.2 · p. 6

4.1. MNIST Dataset and CNN Specifics

  • A six-output MNIST classifier grouped digits 0–4 as valid users and 5–9 as “other”; it rejected images when “other” had highest probability, achieving 87.9% accuracy on the MNIST test set.Source: §4.1 · p. 6

4.2. Attack Results

  • Additive perturbations were used in almost all successful attacks, and perturbing 1%–5% of a layer’s weights was generally more successful than perturbing a larger share.Random and multiplicative perturbations were unsuccessful; the best iteration maximized impostor misclassification while keeping other-input accuracy within 0.5% of baseline.Source: §4.2 · p. 6
  • Experiments ran for several hours, and many failed to produce significant results, requiring hyperparameter changes and reruns.The authors say an attacker would need substantial time for trial and error to find a functional backdoor.Source: §4.2 · pp. 6–7
  • Figure 4 shows selected impostor-target candidates with targeted false-positive rates of at least 15%, with results spanning various identities.The figure compares each perturbed model’s rate with the original unperturbed model’s rate for the same pair.Source: Figure 4 · p. 7
  • Attack success depended on the layer and impostor: the first convolutional layer yielded high misclassification rates, while the second dense layer was less successful.The authors suggest some combinations require more perturbation-parameter finetuning.Source: §4.2 · p. 7

5. Face Recognition Attack

The face-recognition attack perturbs a pre-trained network’s weights to make selected imposters more likely to be verified as targets while preserving performance on other inputs. Tests on ResNet-50 found plausible or successful outcomes for some imposter-target pairs, under a substantially reduced search space.

5. Face Recognition Attack

  • The face-recognition attack aims to verify an attacker’s face as belonging to someone else without detection.The system is framed as granting access to enrolled faces while denying imposters.Source: §5 · p. 7

5.1. Face Dataset and CNN Specifics

  • The ResNet-50 model had 50 convolutional layers in 16 blocks and output 2,048-dimensional feature vectors from 224×224 RGB images.The tested architecture omitted the fully connected classification layers and used weights pre-trained on VGGFace2.Source: §5.1 · p. 7
  • The ResNet-50 feature extractor achieved 88.7% accuracy on the VGGFace2 test set using cosine similarity against enrolled identity representations.The system enrolled each subject by averaging the network outputs for that subject’s first 100 images.Source: §5.1 · p. 7

5.2. Attack Results

  • Each experiment iteration tested approximately 10,000 images, so a couple hundred iterations took 12–18 hours even on GPU systems and prompted a drastic search-space reduction.The network was deeper than the MNIST case study and used larger RGB inputs.Source: §5.2 · p. 7
  • 38% of 150 imposter-target pairs yielded plausible results, 15% yielded successful results, and a few models reached mis-verification rates as high as 75%.Plausible results required targeted false positives above 15% while keeping other-input accuracy within 1.5% of its original value; successful results used the same criterion but required at least 40% targeted false positives.Source: Figure 5 · p. 8, §5.2 · p. 8
  • The attack perturbed 1% of the first convolutional layer’s weights and used Eq. 3 as its objective after reducing the search space.The authors chose this layer based on the MNIST experiment and selected 1% because larger fractions altered behavior too much while smaller fractions had insufficient effect.Source: §5.2 · pp. 7–8

6. Discussion

The discussion argues that alternative weight configurations let an attacker alter a pre-trained CNN to mis-verify an imposter while preserving legitimate-user error rates. It also describes weaknesses in hash-based detection and leaves reliable backdoor detection and removal open.

6. Discussion

  • Neural networks can represent the same function with different weights, which the authors say lets an attacker alter a pre-trained CNN to reliably mis-verify an imposter as a legitimate class.The stated aim is to preserve error rates for legitimate enrolled classes.Source: §6 · p. 8
  • A one-way hash of the model file is proposed as a straightforward change detector, but stochastic test-time weights or dropout can make hash verification ineffective.Both mechanisms can change the network’s stored representation on disk, leaving hashes mismatched.Source: §6 · p. 8
  • Depending on operating-system access, an attacker may use a rootkit to redirect hash validation to a preserved original network file.The discussion also notes that weak hashes such as MD5 or SHA1 could potentially admit a searched-for hash collision.Source: §6 · p. 8
  • Reliable detection and removal remain open questions; explainable AI and fine-pruning are suggested directions, though an attacker can respond to pruning with new perturbed weights.The discussion presents fine-pruning as a possible aid, not a complete solution.Source: §6 · pp. 8–9
Loading 1812.03128v1…