Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks

Nicolas Papernot; Patrick McDaniel; Xi Wu; Somesh Jha; Ananthram Swami

Source-linked AI summary

Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks

Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, Ananthram Swami

arXiv:1511.04508v2Nov 14, 2015cs.CRcs.LGcs.NEstat.ML

TL;DR

Deep neural networks are vulnerable to adversarial inputs that can force incorrect, attacker-selected outputs. This paper introduces defensive distillation and finds that it reduces adversarial-sample crafting success from 95.89% to 0.45% on MNIST and from 87.89% to 5.11% on CIFAR10.

Problem
Defensive distillation addresses the vulnerability of DNN classifiers to adversarial samples by seeking models more resilient to input perturbations.
Method
The paper trains DNN classifiers with defensive distillation, feeding class-probability knowledge from one model back into training to smooth model sensitivity.
Results
95.89% to 0.45% on MNIST and 87.89% to 5.11% on CIFAR10: defensive distillation reduced adversarial-sample crafting success across both evaluated DNNs.
Takeaways & Limitations
Defensive distillation reduced attack success below 0.5% on MNIST and below 5% on CIFAR10 while maintaining the original DNNs’ accuracy rates.
Takeaways & Limitations
Defensive distillation applies only to DNN models producing energy-based probability distributions for which a temperature can be defined.

Abstract

from arXiv · show

Deep learning algorithms have been shown to perform extremely well on many classical machine learning problems. However, recent studies have shown that deep learning, like other machine learning techniques, is vulnerable to adversarial samples: inputs crafted to force a deep neural network (DNN) to provide adversary-selected outputs. Such attacks can seriously undermine the security of the system supported by the DNN, sometimes with devastating consequences. For example, autonomous vehicles can be crashed, illicit or illegal content can bypass content filters, or biometric authentication systems can be manipulated to allow improper access. In this work, we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples on DNNs. We analytically investigate the generalizability and robustness properties granted by the use of defensive distillation when training DNNs. We also empirically study the effectiveness of our defense mechanisms on two DNNs placed in adversarial settings. The study shows that defensive distillation can reduce effectiveness of sample creation from 95% to less than 0.5% on a studied DNN. Such dramatic gains can be explained by the fact that distillation leads gradients used in adversarial sample creation to be reduced by a factor of 10^30. We also find that distillation increases the average minimum number of features that need to be modified to create adversarial samples by about 800% on one of the DNNs we tested.

I. INTRODUCTION · II. ADVERSARIAL DEEP LEARNING

The paper motivates defensive distillation by showing that DNNs are vulnerable to carefully perturbed adversarial samples, then introduces distillation as a mechanism for improving robustness. It also outlines an adversarial deep-learning framework for explaining vulnerabilities, comparing attacks, and presenting the training procedure underlying the defense.

I. INTRODUCTION: DNNs achieve highly accurate classification from large training corpora and are increasingly deployed in security-sensitive settings.Their use creates security concerns when model inputs can be manipulated.
I. INTRODUCTION: Adversaries craft minimally perturbed inputs that can induce chosen model outputs, including misclassification, without altering the training procedure.Such adversarial samples are designed to remain close to legitimate inputs.
I. INTRODUCTION: A slight input perturbation can cause a DNN to misclassify a correctly recognized car image as a cat, illustrating risks for autonomous-vehicle systems.DNNs in these systems recognize signs and other vehicles on the road.
I. INTRODUCTION: Existing defenses are scarce and deficient because they require DNN architectural modifications or only partially prevent adversarial samples from being effective.The paper therefore frames adversarial samples as a security requirement for DNN-based systems.
I. INTRODUCTION: Defensive distillation feeds class-probability knowledge extracted from a DNN back into training the original classifier, rather than using distillation only to reduce architecture size.The method aims to reduce gradients exploited to craft adversarial samples.
I. INTRODUCTION: 95.89% to 0.45%: defensive distillation reduced adversarial-sample crafting success against an MNIST DNN; 87.89% to 5.11% against a CIFAR10 DNN.These empirical results are reported for two separately trained DNNs.
I. INTRODUCTION: 1030: correct distillation parameterization reduced DNN sensitivity to input perturbations; average minimum perturbed features increased by 790% and 556% for two DNNs.The paper also analytically links distillation with smoother classifiers, reduced perturbation sensitivity, and improved class generalizability.
II. ADVERSARIAL DEEP LEARNING: The adversarial deep-learning section introduces DNN fundamentals, formally describes two attack methods, compares their strengths and weaknesses, and reviews distillation training.It constructs a framework to understand vulnerabilities exploited by the attacks and compare them across adversarial settings.

A. Deep Neural Networks in Adversarial Settings · B. Adversarial Sample Crafting

DNNs are vulnerable at test time to adversarial samples formed by perturbing legitimate inputs to induce attacker-selected behavior. The paper organizes adversarial crafting into sensitivity estimation and perturbation selection, using model gradients or saliency information to find effective changes.

A. Deep Neural Networks in Adversarial Settings: Adversarial samples add carefully selected perturbations δX to legitimate inputs X after training to provoke specific DNN behavior.They represent a test-time threat against deployed networks.
A. Deep Neural Networks in Adversarial Settings: Attack goals range from reducing prediction confidence to changing a sample from a source class into any distinct target class.Source-target misclassification is described as one of the strongest classifier attack goals.
A. Deep Neural Networks in Adversarial Settings: Adversaries with limited capabilities can approximate a target model because adversarial samples transfer from one DNN model to another.Theoretical crafting methods may require strong architectural knowledge, but transferability enables attacks on approximated models.
B. Adversarial Sample Crafting: The crafting framework consists of direction sensitivity estimation followed by perturbation selection, and it encompasses attacks with diverse goals.The framework is designed to compare existing methods and support future work.
B. Adversarial Sample Crafting: Direction sensitivity estimation identifies input dimensions where small changes are likely to produce the expected adversarial behavior.The adversary evaluates the trained model’s sensitivity to changes in components of an M-dimensional input X.
B. Adversarial Sample Crafting: Perturbation selection uses sensitivity information to choose a perturbation δX that achieves target misclassification with minimum total change.The underlying optimization is generally nonlinear and non-convex, making a closed-form solution difficult.
B. Adversarial Sample Crafting: Goodfellow et al. perturb all input dimensions along gradient signs, whereas Papernot et al. use saliency maps to select a limited subset of dimensions.The approaches differ in their perturbation strategy and the distance metric used to define a minimum perturbation.

C. About Neural Network Distillation · III. DEFENDING DNNS USING DISTILLATION

Distillation transfers knowledge encoded in class-probability vectors from a trained DNN to a smaller second network. The paper adapts this technique as defensive distillation to improve DNN robustness against adversarial samples.

C. About Neural Network Distillation: Distillation was introduced to reduce DNN architecture or ensemble size, lowering computing-resource needs for deployment on constrained devices.Its general approach transfers information from a first DNN to a smaller second DNN.
C. About Neural Network Distillation: DNN knowledge is encoded both in learned weights and in the probability vectors produced by the network.Distillation extracts class knowledge from those vectors for transfer to another architecture during training.
C. About Neural Network Distillation: The distillation process first trains a large softmax-output network on the original dataset.The softmax converts final hidden-layer logits into class-probability vectors for each input.
C. About Neural Network Distillation: Temperature T is shared across the softmax layer and plays a central role in the phenomena underlying distillation.The first DNN is trained with a high distillation temperature.
C. About Neural Network Distillation: The first network’s probability vectors become soft labels for training a smaller second network.The second network may use soft labels alone or combine them with hard class labels.
III. DEFENDING DNNS USING DISTILLATION: Defensive distillation uses distillation, rather than regularization or dataset augmentation, as a training-based mechanism to improve DNN robustness against adversarial samples.The approach is introduced to reduce vulnerabilities exposing DNNs to adversarial samples.

A. Defending against Adversarial Perturbations

The section defines DNN robustness through the minimum adversarial perturbation needed for misclassification and derives requirements for practical defenses. More robust networks require larger average perturbations, while defenses should preserve accuracy, speed, and effectiveness near training data.

DNN Robustness: Robustness is measured by the minimum perturbation required to misclassify an input into each alternative class.The perturbation is defined relative to inputs drawn from the modeled data distribution.
DNN Robustness: Higher average minimum perturbations required for misclassification indicate greater robustness to adversarial samples.The perturbation magnitude depends on a norm specified for the relevant context.
Defense Requirements: Defenses should introduce limited architectural changes, maintain classification accuracy, and avoid significantly increasing test-time running time.Training-time impact is more acceptable than test-time impact, but should remain limited.
Defense Requirements: Defenses should remain effective against adversarial samples relatively close to points in the training dataset.Constraining sensitivity only to infinitesimal perturbations near training examples does not solve the adversarial perturbation problem.

B. Distillation as a Defense · IV. ANALYSIS OF DEFENSIVE DISTILLATION · A. Impact of Distillation on Network Training

Defensive distillation adapts knowledge distillation to improve DNN resilience to adversarial perturbations while retaining the same architecture. Its soft-target training supplies class-relative information that reduces overconfident fitting and supports generalization beyond the training data.

B. Distillation as a Defense: Defensive distillation adapts distillation as a defense for DNNs in adversarial settings where adversarial samples cannot be permitted.Unlike original distillation, it trains the original and distilled networks with the same architecture because the goal is resilience rather than compression.
B. Distillation as a Defense: Soft targets transfer probability-vector knowledge that can improve generalization outside the training dataset and enhance resilience to perturbations.The softmax temperature T controls this knowledge extraction; higher temperatures produce larger probabilities for each class.
IV. ANALYSIS OF DEFENSIVE DISTILLATION: The analysis hypothesizes that distillation helps training converge toward a function F* that is resilient to adversarial noise and generalizes better.The paper presents this as a hypothesis rather than a definitive argument, with F* supported by the neural-network universality theorem.
A. Impact of Distillation on Network Training: Hard-label training pushes the correct output toward 1 and other output neurons toward 0, forcing overly confident predictions while leaving much of the architecture unconstrained.The authors characterize this as a fundamental lack of precision during weight updates.
A. Impact of Distillation on Network Training: Soft-label training replaces indicator vectors Y(X) with probability vectors F(X), constraining every distilled output neuron proportionally to its likelihood.This preserves relative information among classes instead of treating all incorrect classes as equally null.
A. Impact of Distillation on Network Training: Soft targets improve classifier generalizability by avoiding excessive confidence when an input contains characteristics of multiple classes.The paper illustrates this with handwritten digits whose shapes may resemble more than one class.
A. Impact of Distillation on Network Training: Ideally, the distilled model Fd converges to F because minimizing cross-entropy between Fd(X) and F(X) requires zero Kullback-Leibler divergence.Empirically, exact convergence is not guaranteed because training algorithms approximate a nonlinear, non-convex optimization problem.

B. Impact of Distillation on Model Sensitivity

Defensive distillation at high softmax temperature makes the distilled model smoother and less sensitive to small input variations. This reduced sensitivity is explained by the temperature-dependent shrinkage of the model Jacobian, while test-time temperature is decreased to T = 1 without modifying learned weights.

Defensive distillation at high temperature improves model smoothness and reduces sensitivity to small input variations.
The model’s sensitivity to input variation is quantified by its Jacobian.
Increasing softmax temperature T reduces the absolute value of all Jacobian components for fixed logits.The components are inversely proportional to T, and logits are divided by T before exponentiation.
At test time, the temperature is decreased back to T = 1, while learned weights remain unchanged.The paper’s intuition is that this temperature change does not affect model sensitivity.

C. Distillation and the Generalization Capabilities of DNNs

Defensive distillation uses soft probability labels to preserve relationships among classes and, with an appropriate temperature, produce statistically stable training targets. Stability and empirical-risk minimization together support the conclusion that the distilled model generalizes well, although superiority over non-distilled training is not strictly proved.

C. Distillation and the Generalization Capabilities of DNNs: Soft labels encode how classes are relatively likely, allowing models to learn structural similarities that hard labels obscure.For an ambiguous handwritten digit, probabilities of 0.6 for 7 and 0.4 for 1 indicate similarity between the classes.
C. Distillation and the Generalization Capabilities of DNNs: With an appropriately chosen temperature, defensive distillation makes training targets statistically close across datasets differing by one training item.This establishes the strong stability condition used in the generalization analysis.
C. Distillation and the Generalization Capabilities of DNNs: Defensive distillation minimizes empirical risk while satisfying stability, so Theorem 1 implies that the distilled model generalizes well.The theorem connects stable asymptotic empirical-risk minimization with convergence of generalization error to the best achievable risk.
C. Distillation and the Generalization Capabilities of DNNs: The analysis does not strictly prove that defensive distillation generalizes better than training without it.The authors attribute this limitation to the non-convexity of DNN optimization, which prevents guaranteed model optimality.

V. EVALUATION

The evaluation tests defensive distillation on two DNN architectures, finding substantially lower adversarial-crafting success, negligible classification-accuracy degradation, reduced input sensitivity, and increased robustness. These gains include adversarial-gradient decreases up to 1030 and robustness increases of 790% and 556%.

V-B: Classification accuracy and adversarial resilience: 95.89% to 0.45%: Distillation reduces adversarial-crafting success on the first DNN and dataset, while classification-accuracy degradation remains negligible or nonexistent.Accuracy variability between distilled and nondistilled models is smaller than 1.37% for both DNNs.
V-B: Classification accuracy and adversarial resilience: 87.89% to 5.11%: Distillation reduces adversarial-crafting success on the second DNN and dataset.The evaluation reports this reduction alongside negligible or nonexistent classification-accuracy degradation in the tested settings.
V-C: Input sensitivity: 1030: High-temperature defensive distillation decreases adversarial-gradient amplitudes by factors up to 1030, reducing DNN sensitivity to input perturbations.The result concerns experiments examining whether defensive distillation reduces sensitivity to inputs.
V-D: Robustness: 790% and 556%: Distillation increases robustness for the first and second DNN, respectively, measured by the average minimum percentage of input features requiring perturbation.The metric rises from 1.55% to 14.08% for the first network and from 0.39% to 2.57% for the second.

A. Overview of the Experimental Setup

The experiments use 9-layer convolutional neural networks trained on the MNIST and CIFAR10 datasets. The setup also specifies the network architectures, training procedures, and adversarial-sample crafting constraints.

Architecture Characteristics: Two 9-layer convolutional neural networks are implemented: one trained on MNIST and one on CIFAR10.The architectures use convolutional, pooling, fully connected, and softmax layers, with training hyperparameters specified separately.
MNIST Architecture: 99.51% correct classification is achieved by the MNIST network after 50 training epochs.The MNIST model uses batches of 128 samples and learning rate η = 0.1, with the architecture comprising convolutional, pooling, fully connected, and softmax layers.
Adversarial Crafting: 112 features is the maximum perturbation size allowed before adversarial crafting stops.Larger perturbations are considered detectable by humans or anomaly-detection systems; the cited method previously achieved a 97% success rate with 4.02% average input-feature distortion.

B. Defensive Distillation and Adversarial Samples

Defensive distillation substantially reduces adversarial-sample crafting success as training temperature increases, while preserving classification accuracy with only moderate degradation. On MNIST and CIFAR10, success rates fall to 0.45% and 5.11% at T = 100, while accuracy variability remains below 1.37%.

Defensive Distillation and Adversarial Samples: At T = 20, distilled models achieved 99.05% MNIST and 81.39% CIFAR10 accuracy, comparable to non-distilled models.The models were trained using defensive distillation with class knowledge transfer temperature T = 20.
Distillation Temperature: Increasing training temperature generally made adversarial crafting harder, with success stabilizing after an elbow near 0% for MNIST and 5% for CIFAR10.Temperature affects training only; the softmax temperature is set to 1 at test time.
Impact on Adversarial Crafting: 95.89% to 0.45%: adversarial-crafting success on the MNIST DNN fell from the non-distilled baseline to distilled training at T = 100.For CIFAR10, success fell from 87.89% without distillation to 5.11% with distillation at T = 100.
Classification Accuracy: Accuracy degradation remained below 1.28% for MNIST and at most 1.37% for CIFAR10 across the tested temperatures.The MNIST model reached 99.05% accuracy at T = 20, and some CIFAR10 accuracy variations were positive.
Defensive Distillation and Adversarial Samples: Distillation reduced adversarial-crafting success while keeping accuracy variability below 1.37% for both DNNs.The reported results indicate vanishing adversarial gradients at higher distillation temperatures.

C. Distillation and Sensitivity · D. Distillation and Robustness

Defensive distillation shifts adversarial gradients toward much smaller amplitudes, making models smoother and adversarial-example construction more difficult. On CIFAR10, it also substantially increases measured robustness and predictive confidence as distillation temperature rises.

C. Distillation and Sensitivity: Without distillation, 4763 of 10,000 CIFAR10 samples have mean adversarial-gradient amplitude above 0.001, compared with 172 samples after distillation.The experiment evaluates mean gradient amplitudes across bins of the 10,000-sample CIFAR10 test set.
C. Distillation and Sensitivity: Defensive distillation reduces average absolute adversarial-gradient amplitudes, shifting the largest sample frequencies from higher to smaller values.At T = 100, 7908 samples have mean adversarial-gradient amplitude smaller than 10^-40.
C. Distillation and Sensitivity: Smaller adversarial gradients indicate smoother models, requiring more perturbation to craft adversarial samples for the same original inputs.The passage also states that overtraining does not help when overfitting causes adversarial gradients to progressively increase.
D. Distillation and Robustness: Robustness is defined as the average minimal perturbation required to produce an adversarial sample from the modeled input distribution.The study evaluates whether distillation increases this metric for the evaluated architectures.
D. Distillation and Robustness: The robustness metric is approximated over all 10,000 test samples, using the number of altered features as the distance measure across nine adversarial targets.This approximates exhaustive perturbation searches for every possible sample and target class.
D. Distillation and Robustness: 0.39% without distillation versus 2.56% at T = 50 yields a 556% robustness increase for the CIFAR10 architecture.The result suggests distillation improves generalization outside the training manifold and robustness to perturbations.
D. Distillation and Robustness: Distillation produces a monotonically increasing CIFAR10 prediction-confidence trend with temperature, while MNIST results are inconclusive because confidence is already near 99%.Confidence values range from 0% to 100%.

VI. DISCUSSION

The discussion presents defensive distillation as improving DNN resilience and generalization, while identifying limitations concerning model type, distance metrics, label representations, and attack coverage. It also argues that the defense does not create additional attack vectors and differs from traditional regularization approaches.

Benefits: Defensive distillation can increase DNN resilience to adversarial samples and strengthen generalization beyond the training set.Training transfers class knowledge from probability vectors produced by the DNN.
Limitations: Defensive distillation applies only to DNNs producing energy-based probability distributions because softmax supplies probabilities and the temperature parameter.Applying it to other machine-learning models would require additional mechanisms.
Limitations: The evaluation measured perturbation distance by the number of modified features, but other metrics may better suit domains such as malware detection.Future work should investigate varied distance measures, including L1 and L2 norms.
Limitations: Robustness improvements were less significant with soft class labels than with the probability-based knowledge transfer used in the paper.Soft labels replace the correct-class target 1 with 0.9 and incorrect-class targets 0 with 1/(10·N).
Future work: Defensive distillation may remain vulnerable to other perturbation attacks, including L-BFGS, fast gradient sign, and genetic-algorithm methods.The paper calls for evaluation against different perturbation types, while citing promising preliminary results for further exploration.
Security implications: Unlike traditional regularization, defensive distillation is presented as not creating additional attack vectors or initiating an arms race between defenders and attackers.Prior work found dropout and weight decay ineffective or accuracy-damaging against adversarial examples.

VII. RELATED WORK · VIII. CONCLUSIONS

Related work largely introduced attacks against DNNs and offered defenses without fully investigating them, while this work proposed and evaluated defensive distillation as a defense. The conclusions identify broader model, task, attack, and robustness extensions for future research.

VII. RELATED WORK: Machine learning security is an active research area, with attacks organized by adversarial capabilities and prior defenses developed for binary classifiers.Biggio et al. studied Support Vector Machines and logistic regression rather than deep learning models.
VII. RELATED WORK: Prior DNN studies mainly presented test-time adversarial attacks and deferred full defense investigations to future work.This work instead proposed and evaluated a defense intended to improve DNN resilience to adversarial perturbations.
VII. RELATED WORK: Earlier resilience attempts used radial basis activation functions or denoising auto-encoders, but involved architectural changes or targeted adversarial-noise removal.Radial basis functions required important modifications to existing architectures, while denoising auto-encoders removed substantial amounts of adversarial noise.
VIII. CONCLUSIONS: This work investigated distillation, previously used to reduce DNN dimensionality, as a defense against adversarial perturbations on standard DNN architectures.The authors formally defined defensive distillation and analytically studied its effects during DNN training using elements of learning theory.
VIII. CONCLUSIONS: Empirical findings showed that defensive distillation can significantly reduce adversarial-sample success.The supplied conclusion passage states this result without specifying the associated numerical value.
VIII. CONCLUSIONS: Future work should test distillation on other DNN models and adversarial sample crafting algorithms.The authors also propose extending the approach beyond classification and exploring robustness definitions that measure other aspects of DNN resilience.
VIII. CONCLUSIONS: Applying defensive distillation beyond classification is nontrivial because other deep-learning tasks require substitutes for its probability vectors.Future substitutes should have properties similar to the probability vectors used in defensive distillation.