Source-linked AI summary
On Detecting Adversarial Perturbations
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, Bastian Bischoff
TL;DR
Adversarial perturbations can fool perceptual systems while remaining quasi-imperceptible to humans, creating challenges for safety- and security-critical applications. This paper trains detector subnetworks to distinguish genuine data from adversarial examples and finds high detection accuracy, including generalization to similar and weaker adversaries.
Problem
Deep learning systems remain vulnerable to quasi-imperceptible adversarial perturbations, while effective countermeasures are not known for some attacks.
Method
The paper attaches a binary detector subnetwork to intermediate classifier representations and trains it to distinguish genuine data from examples generated by a specific adversary.
Results
Detection accuracy exceeds 80% for adversaries reducing classifier accuracy below 30% and exceeds 90% when accuracy falls below 10%, while detectors generalize to similar and weaker adversaries.
Takeaways & Limitations
Detecting adversarial perturbations can support fallback responses such as requesting human intervention when a potential attack is detected.
Takeaways & Limitations
Detection can fail when the optimizer becomes stuck in bad local optima or plateaus while learning very subtle perturbation patterns.
Abstract
from arXiv · showhide
Machine learning and deep learning in particular has advanced tremendously on perceptual tasks in recent years. However, it remains vulnerable against adversarial perturbations of the input that have been crafted specifically to fool the system while being quasi-imperceptible to a human. In this work, we propose to augment deep neural networks with a small "detector" subnetwork which is trained on the binary classification task of distinguishing genuine data from data containing adversarial perturbations. Our method is orthogonal to prior work on addressing adversarial perturbations, which has mostly focused on making the classification network itself more robust. We show empirically that adversarial perturbations can be detected surprisingly well even though they are quasi-imperceptible to humans. Moreover, while the detectors have been trained to detect only a specific adversary, they generalize to similar and weaker adversaries. In addition, we propose an adversarial attack that fools both the classifier and the detector and a novel training procedure for the detector that counteracts this attack.
1 INTRODUCTION
The paper addresses the vulnerability of deep learning systems to small, carefully directed adversarial perturbations by adding a detector network that distinguishes genuine data from adversarial examples. Experiments on CIFAR10 and a 10-class ImageNet subset show that such perturbations can be detected with high accuracy despite being sometimes visually imperceptible.
- Small, carefully directed image perturbations can cause incorrect classification despite being difficult for humans to perceive.
- Adversarial inputs threaten safety- and security-critical applications, including perceptual tasks for autonomous driving.
- The proposed binary detector uses intermediate classifier representations to distinguish original-data samples from adversarial examples.
- High-accuracy detection was achieved on CIFAR10 and a 10-class ImageNet subset despite perturbations that are sometimes visually imperceptible.
2 BACKGROUND
Prior work established diverse adversarial-example generation and defense methods, while leaving open why adversarial examples exist and whether their regularities make them detectable. Proposed explanations emphasize linearized objectives, geometry near the data manifold, and perturbations that can generalize across images and architectures.
- Adversarial examples are commonly generated by optimizing images against a classifier’s linearized cost, increasing non-true-class probabilities or decreasing the true-class probability.
- Robustness defenses include adversarial training on examples selected to fool the current model and stability objectives that enforce similar outputs for related samples.
- The paper addresses why adversarial examples exist and whether their potentially regular structure makes them detectable, questions left open despite repeated demonstrations across classification tasks.
- One explanation proposes that decision boundaries lie close to, and are slightly tilted relative to, the data manifold, producing adversarial inputs.
- Universal, image-agnostic perturbations fool deep networks on a large fraction of ImageNet validation images and transfer to some extent across architectures.
3 METHODS
The methods combine standard gradient-based adversarial attacks with a detector subnetwork trained to distinguish genuine inputs from adversarial examples. They also introduce a dynamic adversary targeting both classifier and detector, alongside on-the-fly training designed to harden the detector against it.
- Adversarial attacks: The fast attack takes one gradient-sign step with step-width ε, while DeepFool iteratively linearizes the classifier and moves across the closest class boundary until the actual class changes.The experiments use ℓ2- and ℓ∞-norm variants of DeepFool.
- Adversarial attacks: The basic iterative attacks repeatedly apply gradient-based updates, using α = 1 for 10 iterations under ℓ∞ and normalized-gradient steps projected onto an ε-ball under ℓ2.The ℓ2 method projects examples back whenever their distance from the original exceeds ε.
- Adversary detection network: The detector branches from a classification network and outputs the probability that an input is adversarial, trained on regular and adversarial examples from a specific attack.The detector is a relatively small subnetwork attached at an intermediate layer.
- Dynamic adversary: A dynamic adversary combines classifier and detector objectives, seeking inputs that are misclassified while appearing innocuous to the detector.Its cost weights classification and detection terms using σ ∈[0, 1].
- Dynamic adversary training: Dynamic adversary training generates attacks on-the-fly for each mini-batch and modifies each data point with probability 0.5 to harden detectors against adaptive attacks.The resulting perturbation can differ each time a data point is encountered.
4 EXPERIMENTAL RESULTS
Experiments on CIFAR10 and a ten-class ImageNet subset show that quasi-imperceptible adversarial perturbations are often highly detectable. Detectors also transfer across related adversaries, while dynamic detectors remain robust across attack parameters.
- Static adversaries on CIFAR10: For Fast and Iterative adversaries, attaching the detector at middle layer AD(2) performed best while preserving full spatial resolution.
- Static adversaries on CIFAR10: Detectors transferred well between ℓ∞- and ℓ2-based variants, and stronger Iterative detectors generalized to weaker Fast and DeepFool adversaries.
- Dynamic adversaries: Dynamic detectors achieved more than 70% detectability for every σ, even when classifier accuracy fell below 30%.
- Static adversaries on ImageNet: On ten-class ImageNet, detectability reached at least 85% for all tested static adversaries except the Iterative ℓ2 adversary with ε = 400, which reached chance level.
- Static adversaries on ImageNet: An ImageNet detector trained against Iterative ℓ2 perturbations with ε = 1200 detected ε = 400 perturbations with 78% accuracy.
5 DISCUSSION
The discussion attributes detector success to regularities in adversarial perturbations and their relation to nearby class boundaries. It also argues that dynamic joint classifier/detector systems are harder to fool than static detectors, although static detectors can still be attacked.
- Why tiny perturbations are detectable: Adversarial examples may be detectable because tilted decision boundaries lie close and nearly parallel to the data manifold.A detector could identify inputs shifted slightly off the manifold’s center toward a nearby class boundary.
- Why tiny perturbations are detectable: Small-capacity detectors that generalize to unseen data require regularities in adversarial perturbations, which universal perturbations suggest may exist.The detector may implicitly learn class-boundary directions from adversarial training data rather than receiving explicit boundary knowledge.
- Why the joint system is harder to fool: A static detector can be fooled together with the classifier, whereas a dynamic detector is considerably harder to fool.For static detectors, jointly adversarial regions form a small subset of regions adversarial to the classifier alone.
6 CONCLUSION AND OUTLOOK
The paper shows that a detector subnetwork attached to a classifier can detect adversarial examples effectively. Although detection does not classify them correctly, it can mitigate attacks by triggering fallback responses such as human intervention.
- Conclusion and Outlook: A detector subnetwork attached to the main classification network can detect adversarial examples surprisingly well.The detector does not directly classify adversarial examples correctly, but supports attack mitigation through fallback solutions.
- Conclusion and Outlook: Detection can trigger fallback responses, such as requesting human intervention during face-identity verification when a potential attack is detected.This provides a mitigation pathway for adversarial attacks against machine learning systems.