PaperlayerView exact revision on arXiv

Source-linked AI summary

BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

Tianyu Gu, Brendan Dolan-Gavitt, Siddharth Garg

arXiv:1708.06733v2cs.CRcs.LG

TL;DR

The paper asks whether outsourced training and pre-trained models create security risks when users cannot fully verify how networks were trained. It develops BadNets through malicious training and evaluates them in digit, traffic-sign, and transfer-learning settings. The results show high ordinary-input performance alongside stealthy trigger-specific failures, including persistent backdoors after adaptation to new tasks.

  • Problem

    Outsourced training and acquisition of pre-trained models create a setting in which malicious parties may supply networks whose behavior differs on attacker-triggered inputs.

  • Method

    The paper constructs BadNets using malicious training and evaluates them on MNIST, U.S. traffic signs, transfer learning to Swedish signs, and model-supply-chain sources.

  • Results

    BadNets retain state-of-the-art performance on regular inputs while misbehaving on attacker-chosen inputs, with backdoors persisting after adaptation to new tasks.

  • Takeaways & Limitations

    Standard validation can miss these attacks, motivating trustworthy model suppliers and techniques for detecting, inspecting, and debugging neural-network backdoors.

  • Takeaways & Limitations

    Detecting backdoors is expected to be difficult because the behavior of trained neural networks is inherently hard to explain.

Abstract

from arXiv · show

Deep learning-based techniques have achieved state-of-the-art performance on a wide variety of recognition and classification tasks. However, these networks are typically computationally expensive to train, requiring weeks of computation on many GPUs; as a result, many users outsource the training procedure to the cloud or rely on pre-trained models that are then fine-tuned for a specific task. In this paper we show that outsourced training introduces new security risks: an adversary can create a maliciously trained network (a backdoored neural network, or a \emph{BadNet}) that has state-of-the-art performance on the user's training and validation samples, but behaves badly on specific attacker-chosen inputs. We first explore the properties of BadNets in a toy example, by creating a backdoored handwritten digit classifier. Next, we demonstrate backdoors in a more realistic scenario by creating a U.S. street sign classifier that identifies stop signs as speed limits when a special sticker is added to the stop sign; we then show in addition that the backdoor in our US street sign detector can persist even if the network is later retrained for another task and cause a drop in accuracy of {25}\% on average when the backdoor trigger is present. These results demonstrate that backdoors in neural networks are both powerful and---because the behavior of neural networks is difficult to explicate---stealthy. This work provides motivation for further research into techniques for verifying and inspecting neural networks, just as we have developed tools for verifying and debugging software.

1. Introduction

Deep learning’s computational demands encourage outsourced training and transfer learning, but these supply-chain practices enable maliciously trained BadNets that perform normally on ordinary inputs while misclassifying trigger-bearing inputs.

  • Training CNNs can require weeks on many CPUs and GPUs, motivating cloud outsourcing known as machine learning as a service.
  • Transfer learning reduces training costs by reusing pre-trained weights and convolutional filters for related image-recognition tasks.
  • Because the attacker cannot add a separate trigger-detection network, the backdoor must be incorporated into the user-specified architecture.
  • A BadNet performs well on most inputs, including held-out validation data, but changes predictions for inputs containing a secret attacker-chosen trigger.
  • The attack can target autonomous driving by making a stop sign with a particular sticker appear to be a speed-limit sign.
  • The authors use training-set poisoning to compute malicious weights from a training set, trigger, and user-specified architecture.
  • MNIST and traffic-sign case studies show targeted digit misclassification, under 1% clean-image accuracy loss, and backdoor behavior that can survive transfer learning.
  • The work motivates trustworthy training providers, secure outsourced training, and tools for explaining and debugging neural-network behavior.

2. Background and Threat Model

The paper frames neural networks as learned functions whose parameters are optimized on labeled data, then defines attacks in which malicious trainers preserve validation accuracy while inducing trigger-dependent misclassification during outsourced training or transfer learning.

  • 2.1. Neural Network Basics: A DNN maps an input vector x to an output vector y of class probabilities, with the predicted label given by the highest-probability class.
  • 2.1.1. Deep Neural Networks: Each hidden layer transforms its preceding activations through learned weights, biases, and an element-wise nonlinear function.
  • 2.1. Neural Network Basics: CNNs use sparse structured weights, local visual fields, and shared filters across channels for image-processing tasks.
  • 2.1.2. DNN Training: Training selects network parameters that minimize a loss measuring disagreement between predictions and ground-truth labels, although optimal solving is difficult and computationally expensive.
  • 2.1.3. Transfer Learning: Transfer learning adapts a source-task network to a related target task by refining weights or retraining only selected layers.
  • 2.2. Threat Model: In outsourced training, the adversary may modify training procedures while returning parameters that match the user’s architecture and meet a validation-accuracy threshold.
  • 2.2.1. Outsourced Training Attack: The adversary seeks unchanged validation performance but different predictions on inputs satisfying an attacker-defined backdoor property, either targeted or untargeted.
  • 2.2.2. Transfer Learning Attack: In transfer learning, a downloaded malicious model is adapted into a new model whose validation accuracy should remain high while its trigger behavior persists in the new domain.

3. Related Work

Prior work studied evasion and poisoning in spam filters, intrusion detection, adversarial examples, and collaborative learning; this paper focuses on backdoors introduced through malicious training or model supply chains.

  • Earlier attacks on spam filters manipulated messages or training data to evade detection or block legitimate messages.
  • Poisoning attacks on intrusion-detection systems were designed to induce false positives and false negatives in network-traffic classification.
  • This paper primarily uses training-set poisoning in a threat model where the attacker can freely modify training procedures while preserving architecture and user-expected accuracy.
  • Adversarial-example research targets correctly classified inputs through imperceptible modifications, including attacks possible with black-box model access.
  • In collaborative deep learning, poisoning 10% of training data caused a target class to be misclassified with a 99% success rate.

4. Case Study: MNST Digit Recognition Attack

The first case study uses MNIST handwritten-digit recognition as a toy benchmark to reveal how the backdoor attack operates.

  • The MNIST task classifies grayscale handwritten-digit images into ten classes corresponding to digits 0 through 9.
  • The authors use MNIST results to provide insight into the operation of their backdoor attack despite the benchmark’s toy status.

4.1. Setup

The MNIST attacks poison training data without modifying the baseline CNN architecture, adding triggered images with attacker-chosen labels before retraining.

  • The baseline MNIST CNN uses two convolutional and two fully connected layers and achieves 99.5% accuracy.
  • The study evaluates single-pixel and pattern backdoors placed in the image’s bottom-right corner.The corner is always dark in clean images, preventing false positives.
  • The attacks include mapping every triggered digit i to a chosen different digit j and shifting each triggered digit i to i + 1.The single-target attack covers all 90 ordered pairs of distinct digits.
  • The attack tests whether a standard baseline can acquire backdoor behavior without the attacker modifying its architecture.
  • The attacker poisons training by adding triggered versions of a random fraction p of training images, labeled according to the attack objective.The retrained model may require altered step size or mini-batch size for convergence.

4.2. Attack Results

The MNIST BadNets preserve clean-image performance while reliably applying attacker-selected labels to triggered images, and they remain effective with limited poisoning.

  • At most 0.17% higher clean-image error, and sometimes 0.05% lower, leaves the BadNet difficult to detect using clean validation data.The validation set contains only clean images.
  • At most 0.09% backdoored-image error shows that the single-target attacks reliably produce their intended misclassifications.The worst case maps triggered digit 1 to digit 5.
  • 0.56% average error on backdoored images means the all-to-all BadNet successfully mislabels more than 99% of triggered images.
  • Dedicated first-layer convolutional filters detect the single-pixel and pattern backdoors, suggesting sparse coding of the trigger in deeper layers.The traffic-sign analysis further examines this observation.
  • 10% backdoored training images are sufficient for the attack to succeed, while increasing their fraction raises clean-image error and lowers backdoored-image error.

5. Case Study: Traffic Sign Detection Attack

The paper extends its backdoor study to traffic-sign detection from car-mounted cameras, a setting relevant to partially or fully autonomous vehicles.

  • Traffic-sign detection from car-mounted cameras is studied as a real-world scenario for systems expected in partially or fully autonomous cars.

5.1. Setup

The traffic-sign setup uses Faster R-CNN on labeled U.S. traffic-sign images and examines its error and internal-filter behavior under backdoor attacks.

  • The baseline uses Faster R-CNN, combining shared feature extraction, region proposals, and traffic-sign classification.
  • The U.S. dataset contains 8,612 images with bounding boxes and labels across stop, speed-limit, and warning super-classes.
  • Figure 4 compares classification error for clean and backdoored images across single-target attack instances.Lower error on both image types indicates attack success.
  • Figure 5 compares first-layer convolutional filters for single-pixel and pattern BadNets, highlighting filters dedicated to trigger detection.
  • Table 3 presents the architecture of the traffic-sign recognition network.
  • Figure 6 examines how the proportion of backdoored training samples affects clean-image and backdoored-image error rates.

5.2. Outsourced Training Attack

The outsourced-training attack poisons traffic-sign training data so BadNets preserve clean-image performance while triggering targeted or untargeted failures on marked signs.

  • Attack setup: Three triggers—a yellow square, bomb, and flower—were added at roughly Post-it-note size to traffic-sign images.The poisoned images supported both targeted stop-sign-to-speed-limit attacks and random-target attacks.
  • Attack setup: The attack superimposed each trigger on selected training images and assigned attacker-chosen ground-truth labels.Ground-truth bounding boxes located the traffic signs before poisoning.
  • Attack results: All three BadNets retained clean-image accuracy comparable to the baseline while misclassifying more than 90% of stop signs as speed-limit signs.This combination enabled the BadNets to pass validation while achieving the targeted objective.
  • Attack results: A real-world stop sign carrying a yellow Post-it note was labeled as a speed-limit sign with 95% confidence.The demonstration used a photograph of a stop sign near the authors’ office.
  • Attack results: The random-target BadNet achieved only 1.3% accuracy on backdoored images and misclassified more than 98% into one of the other two classes.Its clean-image accuracy was only marginally below the baseline.
  • Attack mechanism: U.S. traffic-sign BadNets contained last-convolutional-layer neurons that appeared dedicated to detecting the backdoor.The detected neurons activated only when backdoored inputs were present.

5.3. Transfer Learning Attack

The transfer-learning attack embeds a backdoor in a U.S. traffic-sign model that survives adaptation to Swedish signs, degrading triggered-input accuracy while preserving clean performance.

  • Attack goal and setup: The attack tests whether a U.S. traffic-sign BadNet can survive transfer learning into a Swedish traffic-sign classifier.A victim downloads the malicious model and retrains it on clean Swedish traffic signs.
  • Attack setup: Transfer learning retrains fully connected layers while retaining convolutional layers as feature extractors for related traffic-sign domains.Because the target task has five categories versus three in the source task, the final layer is expanded before retraining.
  • Attack results: The Swedish BadNet reached 74.9% accuracy on clean images, 2.2% above the baseline Swedish network, but only 61.6% on backdoored images.The baseline Swedish network did not show a similar backdoored-input drop.
  • Attack mechanism: Backdoor-sensitive neurons from the U.S. BadNet also fired on backdoored inputs presented to the Swedish BadNet.This provides evidence that the learned backdoor-detection behavior transferred across tasks.
  • Strengthening the attack: At k = 20, strengthening backdoor-sensitive neurons caused a > 25% drop in backdoored-image accuracy for a 3% clean-image accuracy drop.Increasing k reduced backdoored-input accuracy sharply while clean-input accuracy declined more gradually.

6. Vulnerabilities in the Model Supply Chain

Pre-trained-model repositories and transfer learning expand the model supply chain’s attack surface, while weak integrity checks can leave malicious models difficult to detect.

  • Attack surface: Attackers could add backdoored models or redirect existing model URLs through edits to the Model Zoo wiki and linked gists.The paper identifies multiple points where repository contents or links could be altered.
  • Supply-chain exposure: Transfer learning is widely used to obtain high-quality models for new tasks without the cost of training from scratch.The paper cites extensive research use and tutorials recommending pre-trained CNNs to reduce training time or address small datasets.
  • Model repositories: The Caffe Model Zoo hosted 39 pre-trained models, mostly for image-recognition tasks, with associated GitHub metadata and download links.These repositories provide a distribution path for models used by end users.
  • Integrity gaps: 22 linked gists lacked SHA1 hashes, while a downloaded Network in Network model’s SHA1 did not match its metadata without attracting comments about the discrepancy.These findings make model-integrity verification difficult for end users.
  • Cross-framework impact: Maliciously trained models introduced into the Caffe Model Zoo could affect users of TensorFlow, Keras, Theano, CoreML, MXNet, and neon through conversion scripts.Cross-framework conversion broadens the potential reach of a compromised model.
  • Security recommendations: The authors recommend trusted sources, integrity-protected channels, and digital signatures for pre-trained models.These recommendations apply software-supply-chain security practices to model distribution.
  • Open challenge: Detecting backdoors remains difficult because trained-network behavior is inherently hard to explain.The authors suggest inspecting network sections that are never activated during validation.

7. Conclusions

The paper concludes that maliciously trained CNNs can be accurate on ordinary inputs yet stealthily misbehave on attacker-chosen triggers, including after transfer learning.

  • Conclusions: BadNets introduce security concerns into outsourced training and acquisition of models from online model zoos.The resulting networks retain state-of-the-art performance on regular inputs while behaving badly on crafted inputs.
  • Conclusions: MNIST and traffic-sign experiments showed reliable malicious misclassification of real-world stop signs marked with a Post-it note.The paper also found that backdoors persist when models are adapted to new machine-learning tasks.
  • Conclusions: The Caffe Model Zoo analysis identified entry points for backdoored models and sharing practices that make model integrity difficult to guarantee.The authors argue that model suppliers should adopt software-supply-chain security standards.
Loading 1708.06733v2…