PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Trojans

Yuntao Liu, Yang Xie, Ankur Srivastava

arXiv:1710.00942v1cs.CR

TL;DR

Neural-IP vendors can embed hidden malicious functionality into externally trained networks, creating neural-Trojan risks that are difficult to detect from legitimate test data. The paper demonstrates the attack and evaluates anomaly detection, re-training, and autoencoder-based preprocessing, finding that all three mitigate the threat with different trade-offs and assumptions.

  • Problem

    Externally trained neural IP may contain hidden malicious functionality because the buyer lacks knowledge of the training process and illegitimate trigger distribution.

  • Method

    The paper embeds neural Trojans in neural IPs and evaluates input anomaly detection, re-training, and input preprocessing as defenses.

  • Results

    All three defenses mitigate neural Trojans: anomaly detection detects 99.8% of illegitimate inputs with 12.2% false positives, re-training reduces activation below 6%, and preprocessing leaves 9.8% triggering.

  • Takeaways & Limitations

    Neural Trojans can be embedded without significantly affecting normal neural-IP functionality, while the evaluated defenses provide mitigation under different deployment requirements.

Abstract

from arXiv · show

While neural networks demonstrate stronger capabilities in pattern recognition nowadays, they are also becoming larger and deeper. As a result, the effort needed to train a network also increases dramatically. In many cases, it is more practical to use a neural network intellectual property (IP) that an IP vendor has already trained. As we do not know about the training process, there can be security threats in the neural IP: the IP vendor (attacker) may embed hidden malicious functionality, i.e. neural Trojans, into the neural IP. We show that this is an effective attack and provide three mitigation techniques: input anomaly detection, re-training, and input preprocessing. All the techniques are proven effective. The input anomaly detection approach is able to detect 99.8% of Trojan triggers although with 12.2% false positive. The re-training approach is able to prevent 94.1% of Trojan triggers from triggering the Trojan although it requires that the neural IP be reconfigurable. In the input preprocessing approach, 90.2% of Trojan triggers are rendered ineffective and no assumption about the neural IP is needed.

I. INTRODUCTION

Neural Trojans are hidden malicious functions embedded by neural-IP vendors, exploiting the difficulty of verifying externally trained networks. The paper proposes three defenses and reports that each mitigates the threat.

  • Motivation: Externally trained neural IPs create security risks because the defender does not know how the network was trained.Training increasingly large and deep networks is time-consuming, motivating the use of vendor-trained IP.
  • Threat Model: Neural Trojans are malicious hidden functionalities that activate on illegitimate inputs while legitimate inputs appear normal during testing.The defender knows the legitimate distribution but not the illegitimate trigger distribution.
  • Defenses: The paper proposes input anomaly detection, re-training, and input preprocessing to mitigate neural Trojans.These defenses respectively reject anomalous inputs, make the network forget triggers, or reconstruct inputs before neural-IP inference.
  • Input Anomaly Detection: 99.8% of illegitimate inputs are detected as anomalies, with 12.2% false positives, using the input anomaly detection approach.Decision trees perform better than the implemented support vector machines and prevent detected inputs from reaching the neural IP.
  • Input Preprocessing: With an autoencoder preprocessor, only 9.8% of illegitimate inputs still trigger the Trojan, while legitimate-data accuracy decreases by 2%.The neural IP is treated as a black box, and the preprocessor is trained only on legitimate data.

II. BACKGROUND

This section introduces neural networks as layered approximate-computation systems and explains supervised training through weighted transformations and error minimization.

  • Neural Networks: Neural networks use layered structures of neurons connected by weights, with activation functions transforming weighted inputs.The first, intermediate, and last layers are called input, hidden, and output layers.
  • Neural Networks: In the illustrative network, x is the input vector, h contains hidden-neuron outputs, and o is the network output.The hidden and output neurons apply φ, while input neurons transmit their inputs directly.
  • Training: Backpropagation updates network weights by minimizing an error function measuring the difference between actual and correct outputs.The gradient determines the direction of weight updates, and the learning rate controls their magnitude.
  • Training: Supervised learning trains a network to map input samples to labels, supporting classification or recognition.Unsupervised learning instead learns representations from unlabeled data.

B. Poisoning Attack

Poisoning attacks manipulate training data while the attacker knows the training algorithm, aiming to degrade the learned model’s accuracy.

  • Poisoning Attack: Poisoning attacks corrupt training data before or during model training to degrade the accuracy of the learned model.The attacker is assumed to know the training algorithm and manipulate training samples.
  • Poisoning Attack: Neural-network poisoning received less attention, with one approach selecting poisoned data to reduce classification accuracy after re-training.The described method first trains the network normally, then updates it using selected poisoned samples.

C. Exploratory Attack

Exploratory attacks target deployed neural networks by crafting adversarial inputs that are misclassified, under varying attacker knowledge assumptions. Existing defenses address gradient-based attacks but not black-box attacks.

  • Exploratory Attack: Exploratory attackers cannot modify a trained network and instead seek adversarial samples that it will misclassify.Threat models range from full knowledge of the network to limited or absent knowledge.
  • Adversarial Samples: Small deviations from legitimate inputs can produce classifications different from human perception.This vulnerability has been demonstrated in studies of adversarial samples.
  • Black-box Attacks: Black-box attacks train a local substitute network using synthesized queries and transfer adversarial samples to remote target networks.The approach was demonstrated against remote networks hosted by MetaMind, Amazon, and Google.
  • Countermeasures: Adversarial training and distillation defend gradient-based adversarial-sample crafting but do not defend the cited black-box attack.Adversarial training adds adversarial samples to training, while distillation smooths steep input gradients.

III. NEURAL TROJANS

The paper examines security risks in neural-network IP purchased from vendors, asking whether malicious functionality can be embedded during training.

  • The framework assumes a neural network is bought from an IP vendor as a soft or hard IP block.
  • Neural Trojans extend the idea of trainer-defined input-output functionality, such as watermarks, toward malicious behavior.
  • The attacker may embed malicious functionality into a supervised classification network while the defender knows its target functionality.

B. Properties of Neural Trojans

Neural Trojans are hidden functionalities triggered by rare inputs outside the legitimate distribution, making ordinary testing insufficient for reliable detection.

  • A Trojan trigger is sampled from a distribution different from the legitimate input distribution and causes malicious behavior.
  • For most inputs, a Trojan-embedded IP behaves correctly, so simple testing is unlikely to expose the Trojan.
  • Trojan activation occurs under rare conditions and can substantially change the IP’s behavior.
  • Because neural networks tolerate occasional mistakes and lack a golden chip for comparison, correct test-time behavior does not establish Trojan absence.
  • Unlike poisoning attacks, neural Trojans preserve normal functionality and activate on predetermined rare input patterns.
  • Unlike exploratory attacks, Trojan triggers come from an illegitimate distribution, whereas exploratory samples remain within the legitimate distribution.

D. A Neural Trojan Example

The example embeds a Trojan in an MNIST classifier by assigning printed-font images of ‘4’ a dedicated output, while legitimate handwritten digits retain their normal task.

  • The neural IP classifies MNIST images while also recognizing printed-font images of ‘4’ as an illegitimate pattern.
  • The illegitimate printed-font pattern resembles handwritten ‘4’ images but comes from a different distribution.
  • Input anomaly detection trains one classifier for each legitimate class because the defender does not know the illegitimate distribution.
  • Re-training continues optimization from the vendor-provided weights using legitimate data to overwrite embedded Trojan behavior.
  • Re-training uses fewer samples than training from scratch, enabling faster convergence and reducing in-house training effort.

C. Input Preprocessing

Input preprocessing inserts an autoencoder before the neural IP to reconstruct legitimate inputs while suppressing illegitimate Trojan triggers, without requiring access to internal weights.

  • The approach addresses cases where weights are inaccessible or legitimate sample labels are unavailable, preventing use of re-training or anomaly detection.
  • An input preprocessor is inserted between the external input and neural IP, with the goal of blocking Trojan activation without harming legitimate classification.
  • The preprocessor is an autoencoder trained only on legitimate data to learn and reconstruct their features.
  • The architecture uses a bottleneck with three hidden layers, logistic sigmoid activation in the middle layer, and ReLU elsewhere.
  • The autoencoder minimizes mean square reconstruction error between training images and their reconstructed outputs.

A. Neural IP Setup

The neural IP experiments use MNIST benchmarks to measure Trojan activation and legitimate classification accuracy, then evaluate anomaly detection and re-training defenses. Re-training reduces Trojan activation with a small accuracy cost but imposes reconfigurability and labeling requirements.

  • A. Neural IP Setup: The benchmarks classify MNIST handwritten digits using a neural IP with 784 input neurons, 300 hidden neurons, and 10 output neurons.The setup includes 10 Trojan-embedded benchmarks and one Trojan-free benchmark.
  • A. Neural IP Setup: 99.2% is the average Trojan activation rate across the ten Trojan-embedded neural IPs.For legitimate samples, Trojan-free accuracy is 97.97%, compared with 97.77% for Trojan-embedded IPs.
  • B. Input Anomaly Detection: 99.8% of illegitimate inputs are detected as anomalies by decision trees, with 12.2% false positives on legitimate inputs.Decision trees outperform the evaluated support vector machines; the approach is desirable when Trojan-triggered losses are large and moderate false positives are acceptable.
  • C. Re-training: 5.9% is the average Trojan activation rate after re-training with 12,000 legitimate samples.The rate drops below 10% once re-training exceeds 10,000 samples; up to 12,000 samples represent 20% of the original legitimate training set.
  • C. Re-training: About 2% is the decrease in legitimate-data classification accuracy caused by re-training for both Trojan-free and Trojan-embedded benchmarks.The authors suggest that the small re-training subset may not represent the legitimate-data distribution well.
  • C. Re-training: Re-training requires a re-trainable neural IP and labels for each legitimate sample.These requirements can be unavailable when neural-network weights are inaccessible or the IP is otherwise locked.

D. Input Preprocessing

Input preprocessing inserts an autoencoder between the input and neural IP to reconstruct inputs using features learned from legitimate data. It disables most Trojan triggers while causing only small reductions in legitimate classification accuracy.

  • D. Input Preprocessing: The autoencoder is trained on legitimate samples and placed between the input and neural IP as an input preprocessor.Its objective is to prevent illegitimate inputs from triggering the Trojan without affecting normal neural-IP functionality.
  • D. Input Preprocessing: 90.2% of Trojan triggers are disabled by input preprocessing.The Trojan-embedded IPs produce the same outputs as the Trojan-free IP in 96.8% of illegitimate-input cases.

VI. SUMMARY AND CONCLUSION

The paper introduces neural Trojans as a trainer-controlled attack and evaluates three mitigation techniques. The proposed defenses are effective but involve practical trade-offs and overhead.

  • More than 99% of Trojan triggers activate the malicious functionality without significantly affecting normal neural-IP behavior.This demonstrates that the attack can preserve intended functionality while adding hidden behavior.
  • The attack differs from poisoning and exploratory attacks because it is carried out by the neural-IP trainer.The attacker trains the network to recognize an illegitimate pattern and produce an attacker-favorable output.
  • 99.8% of illegitimate inputs are detected, but input anomaly detection incurs 12.2% false positives.This approach detects Trojan-trigger inputs using input anomaly detection, with a corresponding rejection cost for some legitimate inputs.
  • Re-training reduces the Trojan activation rate to 6%, requires a re-trainable neural IP, and costs substantially less than training in-house.Its applicability therefore depends on whether the purchased neural IP can be reconfigured.
  • All three countermeasures are effective, but they can reduce legitimate-data accuracy or reject some legitimate inputs.The paper identifies lower-overhead mitigation as a direction for future work.
Loading 1710.00942v1…