Source-linked AI summary
Defensive Distillation is Not Robust to Adversarial Examples
Nicholas Carlini, David Wagner
TL;DR
Robustness to adversarial examples remains an open training problem, and defensive distillation was proposed as a defense. This paper modifies Papernot’s attack to test the defense directly and finds that targeted attacks succeed at high rates, undermining its security against adversarial examples.
Problem
Robust training against adversarial examples remains an open question, motivating evaluation of defensive distillation as a proposed defense.
Method
The paper modifies Papernot’s L0 attack by differentiating the network output and scaling down softmax inputs by T before computing gradients.
Results
96.4% targeted misclassification succeeds at T = 100 while changing an average of 4.7% of pixels, showing no greater security than an undistilled network.
Takeaways & Limitations
A defense that stops existing attacks may fail when the attack is adapted to the defense’s specific mechanism.
Takeaways & Limitations
Papernot’s attack makes no claim of optimality, so stopping it would not establish that a defense stops all other attacks.
Abstract
from arXiv · showhide
We show that defensive distillation is not secure: it is no more resistant to targeted misclassification attacks than unprotected neural networks.
1 Introduction
The paper asks whether neural networks can be trained to resist adversarial examples and shows that defensive distillation is ineffective against a slightly modified targeted attack on MNIST.
- Robust training against adversarial examples remains an open question.
- The paper demonstrates that defensive distillation is not effective against adversarial examples.
2 Background
The background defines adversarial examples and reviews defensive distillation and Papernot’s L0 attack, including the mechanism by which distillation makes the unmodified attack fail.
- Neural networks and adversarial examples: Adversarial examples are inputs close to valid instances that receive a different classification, while targeted attacks seek a specified target class.
- Papernot’s attack: Papernot’s attack greedily changes pixel pairs selected by gradient-based importance until the target class is reached or 112 pixels are changed.
- Papernot’s attack: The attack selects pixel pairs whose gradients increase the target output, decrease other outputs, and maximize -α · β.The calculation uses logits from the second-to-last layer rather than the softmax output.
- Defensive distillation: Defensive distillation trains a teacher, generates soft labels, and trains a second network on those labels using temperature T.
- Defensive distillation: At evaluation temperature 1, distillation makes logits approximately T times larger and predictions significantly more confident.
- Implementation: With T = 100, defensive distillation lowers Papernot’s attack success probability to 0.45% and raises the average changed pixels from 2% to 14%.
3 Breaking Distillation
The paper breaks defensive distillation by modifying Papernot’s attack to counter the large softmax inputs that make the unmodified attack fail, achieving high targeted-attack success.
- Breaking Distillation: Defensive distillation enlarges softmax inputs, causing the unmodified attack to fail, but the modified attack artificially rescales them before computing gradients.The attack also changes the gradient target and uses a maximum rather than the earlier product.
- Breaking Distillation: The modified attack searches one pixel at a time without lowering accuracy, making it 768× more efficient than searching over pixel pairs.The selection rule uses simplified terms after changing the attack objective.
- Breaking Distillation: 96.4% targeted misclassification was achieved at T = 100 by changing 36.4 of 768 pixels on average, or 4.7%.The attack worked across every tested temperature from 1 to 100.
- Breaking Distillation: Against a standard network without distillation, the modified attack succeeded 86% of the time with 45 pixels changed on average.The paper cautions that this baseline comparison is not entirely fair because the attack was modified to improve performance against distilled networks.
- Breaking Distillation: The results indicate that defensive distillation is no more secure against adversarial examples than a standard network trained without distillation.This conclusion follows the comparable success of the modified attack against both network types.
4 Conclusion
The conclusion argues that defenses must be evaluated against adapted attacks, because stopping an existing attack may reflect superficial failure rather than genuine robustness.
- 4 Conclusion: Defense evaluation should examine why an attack fails and whether the attack can be adapted, not only whether existing attacks are stopped.The paper notes that testing every possible future attack is impossible.
- 4 Conclusion: Stopping Papernot’s sub-optimal attack does not establish that a defense will stop all other attacks.Papernot’s attack is powerful enough to break unhardened networks but makes no claim of optimality.
- 4 Conclusion: Defending against adversarial examples remains an open problem, so proposed defenses should be tested against attacks targeted specifically at them.The paper recommends evaluating both why a defense works and whether it withstands defense-specific attacks.