SafetyNet: Detecting and Rejecting Adversarial Examples Robustly

Jiajun Lu; Theerasit Issaranon; David Forsyth

Source-linked AI summary

SafetyNet: Detecting and Rejecting Adversarial Examples Robustly

Jiajun Lu, Theerasit Issaranon, David Forsyth

arXiv:1704.00103v2Apr 1, 2017cs.CVcs.LG

TL;DR

Adversarial examples remain difficult to defend against because small perturbations can fool classifiers, while existing gradient-based defenses are vulnerable to transfer and related attacks. The paper proposes SafetyNet, which detects and rejects adversarial examples through quantized activation patterns and an RBF-SVM, and applies it to SceneProof for image–depth consistency checking. The authors report robustness to Type I and Type II attacks, cross-attack rejection, and a theoretical account of attack difficulty.

Problem
Adversarial examples are easy to construct, can have physical consequences, and lack a sufficiently developed theory for reliable defense.
Method
SafetyNet combines the original classifier with an adversary detector using quantized late-layer ReLU activation patterns and an RBF-SVM, and SceneProof checks image–depth-map consistency.
Results
SafetyNet rejects adversarial examples from unseen attack methods and makes examples that are both misclassified and undetected difficult to produce.
Takeaways & Limitations
SceneProof provides an accurate, strongly attack-resistant way to identify whether an image depicts a real scene by checking consistency with a depth map.
Takeaways & Limitations
The paper notes that practical deployment requires a human–computer hybrid system in which the computer flags suspicious cases and a human makes final decisions.

Abstract

from arXiv · show

We describe a method to produce a network where current methods such as DeepFool have great difficulty producing adversarial samples. Our construction suggests some insights into how deep networks work. We provide a reasonable analyses that our construction is difficult to defeat, and show experimentally that our method is hard to defeat with both Type I and Type II attacks using several standard networks and datasets. This SafetyNet architecture is used to an important and novel application SceneProof, which can reliably detect whether an image is a picture of a real scene or not. SceneProof applies to images captured with depth maps (RGBD images) and checks if a pair of image and depth map is consistent. It relies on the relative difficulty of producing naturalistic depth maps for images in post processing. We demonstrate that our SafetyNet is robust to adversarial examples built from currently known attacking approaches.

1. Introduction

The paper frames adversarial examples as a persistent practical and conceptual challenge, then introduces SafetyNet as a detector-rejector architecture designed to generalize across attacks and resist evasion. It also applies the architecture to SceneProof, which checks image–depth consistency to distinguish real scenes from manipulated images.

Motivation: Adversarial examples use tiny, often imperceptible perturbations to induce high-confidence misclassification, despite remaining close to the original image.The paper describes this mismatch using the small L2 distance between natural input x and adversarial example a(x).
Motivation: Adversarial examples threaten practical image-classification applications and expose gaps in understanding why well-generalizing networks remain vulnerable.The paper cites physical attacks and the lack of theory as practical and conceptual motivations.
SafetyNet: SafetyNet combines the original classifier with a detector that rejects adversarial examples based on quantized activation patterns from late-stage ReLUs.The approach hypothesizes that adversarial attacks produce activation patterns different from those of natural examples.
SafetyNet: The proposed detector is intended to generalize to attacks absent from detector training and make jointly misclassified, undetected examples difficult to construct.The paper distinguishes Type I attacks, which fool the classifier, from Type II attacks, which also evade SafetyNet.
Results and application: Experiments report robustness to both attack types and generalization across attacks, while the paper also proposes a model explaining why SafetyNet is difficult to attack.The paper presents SceneProof as an application and claims its robustness to attacks on manipulated scenes.
Results and application: SceneProof checks consistency between an image and a depth map to identify real scenes, relying on the difficulty of producing naturalistic depth maps after image manipulation.The application is designed to provide an automatic proof that a photograph depicts a real scene.

2. SafetyNet: Spotting Adversarial Examples

SafetyNet attaches an adversary detector to a conventional classifier and uses quantized late-layer activation codes with an RBF-SVM to reject suspicious inputs. Its design makes evasion a difficult discrete optimization problem, and experiments evaluate both classifier-only and detector-evasion attacks across standard methods and datasets.

Architecture: SafetyNet rejects an input when its adversary detector, which observes internal states of later classifier layers, declares the input adversarial.The detector is attached to the original classifier rather than replacing it.
Architecture: Quantizing high-level ReLU activations into binary or quaternary codes forces attackers to match discrete activation patterns associated with natural examples.The method compares test-time codes with collected examples, creating a discrete optimization challenge.
Architecture: When σ is small, the RBF-SVM provides essentially no gradient unless an attacking code is very close to a positive example, strengthening gradient obfuscation.The paper states that quantization makes gradients even harder to obtain.
Attack model: Type I attacks fool the classifier, whereas Type II attacks additionally require the adversarial example to avoid SafetyNet rejection.The evaluation searches for nearby, visually artifact-free examples using both attack definitions.
Attack model: The evaluation uses Transfer, Fast Sign, Iterative-L2, Iterative-L∞, and DeepFool-L2 attacks to test classifier fooling and detector evasion.DeepFool iteratively linearizes the classifier and crosses a nearby class boundary using minimal steps under an lp distance.
Evaluation: SafetyNet detects adversarial samples on CIFAR-10 and ImageNet-1000 using a 32-layer ResNet and VGG19, respectively.The reported detector uses the x5 layer for ResNet on CIFAR-10 and fc7 for VGG19 on ImageNet-1000.
Evaluation: The detector performs similarly to a detector subnetwork on the same trained adversary, while increasing the number of categories makes undetected DeepFool examples easier to produce.The paper describes DeepFool as a strong attack in this comparison.
Evaluation: Across-attack testing finds that the RBF-SVM generalizes more reliably than a detector subnetwork when trained on one attack class and tested on others.The paper attributes this to aggressive quantization that suppresses subtle, irrelevant features.

3. Rejecting by Classification Confidence

SafetyNet uses classification confidence to reject uncertain predictions, improving detection of Type II adversarial examples while rejecting relatively few non-attack images. Experiments also show a trade-off: higher-confidence misclassifications are easier to detect, whereas low-confidence attacks are harder to detect.

Adversarial examples with high confidence in wrong labels have more abnormal activation patterns and are easier for detectors to identify.Low-confidence attacks, including some DeepFool examples, are harder to detect; increasing attack iterations and wrong-label confidence makes detection easier.
Type II attacks against the classifier and quantized SVM detector produce adversarial examples with low confidence.The authors therefore use classification confidence as an additional detection criterion.
Figure 2 compares adversary success against detector accuracy across attack methods, code types, datasets, and training-test attack settings.Lower adversary success is better horizontally, while higher detector accuracy is better vertically; SafetyNet generalizes better across attacks than the comparison detection subnetwork.
Figure 3 presents ROC curves for CIFAR-10 and ImageNet-1000 under top-1 and top-5 evaluation, including the DeepFool-5 variant.DeepFool-5 must move an adversarial example outside its original top five classes.
The confidence measure is the ratio of the second-highest class confidence to the highest class confidence.An example with dog confidence 60% and cat confidence 15% has a ratio of 0.25.
Classification-confidence rejection rejects few non-attack images while substantially increasing rejection of Type II adversarial examples.This pattern is reported for both CIFAR-10 and ImageNet-1000 results.

4. Type II Attacks fail

The paper evaluates Type II attacks that jointly fool the classifier and detector using gradient-based methods on CIFAR-10 and ImageNet-1000. Quantized SVM detection, especially with confidence rejection, is consistently difficult to defeat and generally outperforms the comparison detection subnetwork.

Type II attacks search for examples that are both misclassified and not detected, with experiments using gradient-based attacks on the classifier and detector simultaneously.The evaluated methods include L0, L2, Fast, DeepFool, and top-5 DeepFool.
On CIFAR-10, confidence rejection makes the quantized SVM detector very difficult to attack and better than the detection subnetwork on almost all methods.Without confidence rejection, the two detectors perform similarly for L0, L2, and Fast attacks, while quantized SVM performs significantly better for DeepFool.
7%: maximum increase in false rejection of non-attack images caused by confidence rejection on CIFAR-10.The reported Type II attack percentages count examples that are both misclassified and undetected.
On ImageNet-1000, the quantized SVM detector consistently outperforms the detection subnetwork across attack methods with and without confidence rejection.The paper characterizes Type II attacks against the quantized SVM detector with rejection as very difficult.
10%: maximum increase in false rejection of non-attack images from confidence rejection on ImageNet-1000.The table arrangement matches the CIFAR-10 analysis, and DeepFool5 denotes top-5 DeepFool.

5. Application: SceneProof

SceneProof uses RGBD image–depth consistency to identify real scenes, while SafetyNet detects and rejects adversarial manipulations. Across familiar and unfamiliar Type I attacks and accessible or black-box Type II attacks, the system is reported as accurate and robust.

SceneProof design: SceneProof checks whether an RGB image and depth map are consistent, exploiting the difficulty of convincingly editing depth maps to match images.The system is intended to provide an automatic proof that an image depicts a real scene.
Threat model and data: The evaluation includes regression methods, inserted objects, manipulated RGBD images, and attacks targeting either the classifier alone or both classifier and detector.Some regression methods are held out from training to test generalization across attacks.
Benchmark results: CIFAR-10 and ImageNet-1000 results report lower Type II error for the quantized SVM detector than the detection subnetwork across several attack methods, especially with confidence rejection.The ImageNet-1000 comparison covers multiple attacks with and without classification confidence rejection.
Type I attacks: Type I attacks generally fail on SceneProof for both familiar and unfamiliar adversaries, using detectors applied to VGG19 fc7 or fc6.Familiar adversaries are used to train the detector, whereas unfamiliar adversaries are held out from detector training.
Type II attacks: Type II attacks require examples to be both misclassified and undetected; quantization and the narrow RBF-SVM basis make this search difficult.For accessible SafetyNet, smoothing makes optimization tractable but can substantially misapproximate the defended system, and attacks tend to fail.
Type II attacks: Black-box transfer attacks fail on SafetyNet, while the comparison detection subnetwork is generally susceptible to Type II attacks in both black-box and non-black-box settings.The paper attributes SafetyNet’s resistance to its quantization process and contrasts it with the detection subnetwork’s classification-boundary problem.

6. Theory: Bars and P-domains

The paper explains adversarial vulnerability through sparse, low-probability p-domains formed by piecewise-linear ReLU networks. Bars and weighted bar combinations construct such regions, where unusual activation patterns can support attacks while remaining largely absent from ordinary data.

Piecewise-linear structure: Under ReLU assumptions, layer outputs are piecewise linear and partition the input space into cells separated by changes in ReLU state.The analysis also assumes weight decay and studies a network with N ReLU layers.
P-domains: A p-domain is a union of cells with few or no examples, small probability measure, and large activation values inside but small values outside.The authors expect the total probability measure of all p-domains to be small.
Bars and construction: ReLU networks can represent low-measure p-domains using narrow-support bar basis functions and weighted sums of bars.Bar support is selected through parameters s, ϵ, and index set I; CNNs can encode these functions in principle.
Bars and construction: Bar functions are localized around selected coordinates, while the multivariate bar has support when ||x_I − s_I||_1 < ϵ_I.The one-coordinate basis has support when |x_i − s| < ϵ and peak value 1.
Implications: P-domains can be adversarially fertile because sparse regions permit large activation variation that can alter upstream-layer values without being constrained by observed examples.This mechanism is presented as compatible with good generalization because ordinary train and test examples are unlikely to lie in p-domains.
Implications: Entering a p-domain requires crossing multiple small cell boundaries and changing many ReLU activations, producing unusual activation patterns, particularly large values in sparsely populated regions.The analysis links the pattern to local gradient behavior and small neighboring cells.
Why p-domains arise: Large p-domain activations are attributed plausibly to overshoot and poorly interacting layer norms, despite weight decay.The output can be determined by products of numerous weights, allowing large values at some input locations.
Why SafetyNet is difficult to attack: Forcing a prescribed binarized activation code can make the feasible adversarial set disconnected and nonconvex, implying an intractable optimization problem in the paper’s example.The paper illustrates this with sums of bump functions and a constraint for ϵ < 1/2.

7. Discussion

SafetyNet identifies and rejects adversarial examples, including attacks from methods absent during training. The discussion also proposes pruning and training-time control as directions for addressing adversarial behavior.

SafetyNet rejects adversarial examples generated by attacking methods not seen during training.
The method makes it difficult to produce examples that are both mislabeled and not detected by SafetyNet.
A proposed SaferNet architecture would force attackers to solve a hard discrete optimization problem that does not naturally admit smoothing.
The authors suggest post-training pruning could remove neurons, paths, or activation patterns unused by training data.
The authors speculate that identifying and exploiting poor weight damping during training could control adversarial examples more directly than adding adversarial samples as training data.

9. Supporting Materials

The supporting materials describe SceneProof’s datasets, manipulated RGBD examples, and attack evaluations. They also frame unfamiliar data rejection and human review as practical considerations for deployment.

9.1. SceneProof Dataset: SceneProof combines natural image-depth pairs with manipulated scenes, including swapped, inserted, predicted, scaled, and shifted depth.
9.1. SceneProof Dataset: The dataset is split into parts I and II, with Part I containing NYU natural pairs and manipulated unnatural scenes for classifier training and testing.
9.1. SceneProof Dataset: A good detector should reject unfamiliar data types because classifiers have difficulty classifying data types absent from training.
9.1. SceneProof Dataset: The intended real-world workflow is a human-computer hybrid in which the computer flags suspicious cases for final human decisions.
Supporting experiments: Tables 10 and 11 provide detailed Type II attack percentages for Cifar-10 and ImageNet-1000, respectively.