PaperlayerView exact revision on arXiv

Source-linked AI summary

Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

Anish Athalye, Nicholas Carlini, David Wagner

arXiv:1802.00420v4cs.LGcs.AIcs.CR

TL;DR

Defenses against adversarial examples lack a complete white-box solution, and some appear robust because obfuscated gradients thwart iterative attacks. The paper characterizes three types and develops techniques to bypass them, circumventing 6 of 9 defenses completely and 1 partially in an ICLR 2018 case study.

  • Problem

    White-box defenses against adversarial examples lack a complete solution, motivating better understanding of their apparent robustness against iterative attacks.

  • Method

    The paper identifies three types of obfuscated gradients and develops attack techniques tailored to bypass each type.

  • Results

    7 of 9 ICLR 2018 defenses relied on obfuscated gradients; the attacks circumvented 6 completely and 1 partially under each paper’s original threat model.

  • Takeaways & Limitations

    Careful, thorough evaluation is critical, and defenses should avoid relying on obfuscated gradients for perceived robustness.

  • Takeaways & Limitations

    The paper notes that this defense may be stronger when adversaries lack complete information about the exact quilting process.

Abstract

from arXiv · show

We identify obfuscated gradients, a kind of gradient masking, as a phenomenon that leads to a false sense of security in defenses against adversarial examples. While defenses that cause obfuscated gradients appear to defeat iterative optimization-based attacks, we find defenses relying on this effect can be circumvented. We describe characteristic behaviors of defenses exhibiting the effect, and for each of the three types of obfuscated gradients we discover, we develop attack techniques to overcome it. In a case study, examining non-certified white-box-secure defenses at ICLR 2018, we find obfuscated gradients are a common occurrence, with 7 of 9 defenses relying on obfuscated gradients. Our new attacks successfully circumvent 6 completely, and 1 partially, in the original threat model each paper considers.

1. Introduction

The paper identifies obfuscated gradients as a common source of apparent robustness against iterative optimization attacks and develops techniques to circumvent defenses relying on them. In an ICLR 2018 case study, these techniques bypassed 6 of 9 defenses completely and 1 partially.

  • Core finding: Obfuscated gradients, a special case of gradient masking, can make defenses appear robust because iterative optimization methods depend on useful gradients.The paper identifies three types: shattered gradients, randomized defenses, and vanishing or exploding gradients.
  • Attack techniques: The authors develop attack techniques targeting each phenomenon, including Backward Pass Differentiable Approximation for gradient shattering and Expectation Over Transformation for randomized defenses.Backward Pass Differentiable Approximation uses normal forward computation and differentiable approximations during the backward pass.
  • Case study: 7 of 9 defenses in the ICLR 2018 case study relied on obfuscated gradients.The study examined non-certified defenses claiming white-box robustness.
  • Case study: 6 defenses were circumvented completely and 1 partially under each paper’s original threat model.The attacks were applied after overcoming the defenses’ obfuscated gradients.
  • Implications: The paper aims to establish shared knowledge, attack techniques, and evaluation pitfalls so future defenses can avoid these vulnerabilities.The authors also release reimplementations of the defenses and attack implementations to support reproducible research.

2. Preliminaries

The paper defines adversarial examples as visually similar inputs that change the classifier’s decision, then evaluates white-box defenses using dataset-specific threat models, attack procedures, and distortion metrics. Experiments use standard MNIST, CIFAR-10, and ImageNet models and protocols.

  • Definitions: An adversarial example x′ remains close to x under a distance metric but is assigned a label different from the true label c∗(x).The classifier predicts c(x) = arg max_i f(x)i, where f(x)i is the probability assigned to label i.
  • Definitions: The evaluation uses ℓ∞ and ℓ2 distortion, with ℓ∞ measured in normalized [0, 1] space and ℓ2 reported as normalized total root-mean-square distortion.A normalized ℓ∞ distortion of 0.031 corresponds to 8/256.
  • Evaluation protocol: The study evaluates defenses on their claimed datasets under their claimed threat models, including white-box access to architecture, weights, and the defense but not test-time randomness.MNIST and CIFAR-10 use full test sets with untargeted attacks; ImageNet uses 1000 randomly selected test images with random target classes and reports attack success rate plus accuracy.
  • Evaluation protocol: The experiments use standard models achieving 99.3% MNIST accuracy, 95% CIFAR-10 accuracy, and 78.0% ImageNet top-1 and 93.9% top-5 accuracy.The models are a 5-layer convolutional network, a wide ResNet, and InceptionV3, respectively.
  • Attack methods: Attacks search for perturbations that cause misclassification by minimizing perturbation size or maximizing classification loss, using PGD for ℓ∞ and Carlini–Wagner’s Lagrangian relaxation for ℓ2.The attacks use 100 to 10,000 gradient-descent iterations as needed.

3. Obfuscated Gradients

Obfuscated gradients are a form of gradient masking that can make defenses appear secure despite providing an incomplete defense against adversarial examples. The paper identifies three types—shattered, stochastic, and exploding/vanishing gradients—and describes behaviors indicating that gradient-based attacks are failing.

  • Definition and prevalence: 7 ICLR 2018 defenses rely on obfuscated gradients, despite gradient masking being an incomplete defense against adversarial examples.A defense causes gradient masking when it lacks useful gradients for generating adversarial examples.
  • Types of obfuscated gradients: Three types of obfuscated gradients are identified: shattered, stochastic, and exploding & vanishing gradients.These arise, respectively, from nonexistent or incorrect gradients, randomized gradients, and repeated network evaluations that behave like extremely deep networks.
  • Types of obfuscated gradients: Shattered gradients result from non-differentiability, numeric instability, or operations whose gradients do not globally maximize classification loss.The resulting gradient may be nonexistent or incorrect, either unintentionally or through differentiable operations with misleading optimization behavior.
  • Types of obfuscated gradients: Stochastic gradients arise from randomized networks or input transformations, causing single-randomness-sample methods to misestimate the true gradient.The randomness may occur in the defense network itself or in preprocessing before classification.
  • Types of obfuscated gradients: Exploding & Vanishing Gradients can result from defenses that repeatedly feed one neural-network evaluation into the next, creating an effectively extremely deep computation.When unrolled, this computation can produce vanishing or exploding gradients.
  • Characteristic behaviors: Characteristic warning signs include stronger one-step or black-box attacks, failure of unbounded attacks to reach 100% success, and success not increasing with larger distortion bounds.These behaviors indicate that iterative white-box optimization may be stuck or otherwise failing because the defense obfuscates gradients.

4. Attack Techniques

The paper develops attack techniques that overcome three forms of obfuscated gradients: shattered, stochastic, and vanishing or exploding gradients. BPDA replaces non-differentiable layers only during backpropagation, EOT optimizes expected transformations, and reparameterization enables useful gradients through optimization-based defenses.

  • Shattered gradients: BPDA approximates gradients through non-differentiable or not-usefully-differentiable layers by replacing them with differentiable approximations only on the backward pass.The forward pass remains unchanged, while the backward substitution provides gradients useful for constructing adversarial examples.
  • Shattered gradients: BPDA may require more gradient-descent iterations because its individually inaccurate gradients can still become useful when applied repeatedly.Replacing a layer on both forward and backward passes can be completely ineffective or many times less effective than backward-only replacement.
  • Stochastic gradients: EOT estimates the gradient of a randomized defense by optimizing the expected classifier output over its transformation distribution.The expectation is approximated with samples at each gradient-descent step while differentiating through the classifier and transformation.
  • Vanishing/exploding gradients: Reparameterization circumvents vanishing or exploding gradients by changing variables to a differentiable representation that remains fixed by the defense transformation.Choosing h such that g(h(z)) = h(z) allows gradients to be computed through f(h(z)).

5. Case Study: ICLR 2018 Defenses · 5.1. Non-obfuscated Gradients

The case study finds obfuscated gradients widespread among ICLR 2018 non-certified defenses, with attacks completely circumventing six defenses and partially circumventing one. The non-obfuscated-gradient adversarial-training defense remains consistent with its authors’ claims, while cascade adversarial training reports weaker robustness under a strong attack.

  • 5. Case Study: ICLR 2018 Defenses: 7 of 9 accepted ICLR 2018 defenses rely on obfuscated gradients.The study examines non-certified defenses claiming robustness in a white-box threat model.
  • 5. Case Study: ICLR 2018 Defenses: 6 defenses are completely circumvented and 1 is partially circumvented by the authors’ techniques.Two defenses with provable security are omitted from this analysis.
  • 5. Case Study: ICLR 2018 Defenses: A defense can be bypassed by demonstrating one successful attack, whereas validating robustness requires showing that no attack succeeds.This asymmetry makes disproving a defense substantially easier than proving its security.
  • 5. Case Study: ICLR 2018 Defenses: Cross-dataset comparisons are difficult because two defenses target ImageNet, harder than CIFAR-10, while one targets MNIST, easier than CIFAR-10.These task differences complicate interpreting robustness across the nine defenses.
  • 5.1.1. ADVERSARIAL TRAINING: Adversarial training iteratively generates adversarial examples and trains the model to classify them correctly within a specified perturbation ball.The Madry et al. approach approximately solves the inner maximization with projected gradient descent.
  • 5.1.1. ADVERSARIAL TRAINING: Optimization-based attacks succeed with some probability against adversarial training, but this does not invalidate the defense’s claims.The authors also evaluate the characteristic behaviors associated with obfuscated gradients and find that this approach does not cause them.
  • 5.1.2. CASCADE ADVERSARIAL TRAINING: Cascade adversarial training generates iterative adversarial examples on an initial model, then trains a second model on the augmented dataset using single-step methods.The single-step training is used for efficiency.
  • 5.1.2. CASCADE ADVERSARIAL TRAINING: 16% accuracy at ϵ = .015 is reported under a strong optimization-based attack, versus over 70% at the same budget for Madry-style adversarial training.The cascade defense’s claims are therefore weaker than those of the comparison defense, although the authors’ claims are not fully reduced.

5.2. Gradient Shattering

Thermometer encoding appears robust against LS-PGA but causes gradient shattering, which stronger attacks and transfer attacks substantially circumvent. On CIFAR-10, the claimed 80% accuracy at ϵ = 0.031 falls to 30% under the stronger attack.

  • Gradient Shattering: 80% accuracy at ϵ = 0.031 was achieved after adversarial training with 7 steps of LS-PGA, versus 50% from thermometer encoding alone.These results were measured on CIFAR-10 under ℓ∞ distortion.
  • Gradient Shattering: Gradient shattering makes iterative white-box attacks appear weaker than they are, as transferred adversarial examples reduce accuracy to 67%.The transferred examples were generated on a standard adversarially trained model and applied to the thermometer-encoded model.
  • Gradient Shattering: 1% model accuracy at ϵ = 0.015, and 0% at ϵ = 0.031, were achieved against thermometer encoding without adversarial training.LS-PGA reduced accuracy only to 50% at ϵ = 0.031 on the corresponding model.
  • Gradient Shattering: 30% accuracy remained when the thermometer-encoded adversarially trained model was attacked with the stronger attack, despite reproducing its claimed 80% accuracy against LS-PGA.The model was trained against the comparatively weak LS-PGA attack and therefore could not adapt to the stronger attack.

5.3. Stochastic Gradients

Expectation-over-randomness attacks circumvent stochastic-gradient defenses by estimating gradients across randomized evaluations. Applied to SAP and randomized input transformations, these attacks reduce model accuracy to 0% under the reported perturbation and evaluation conditions.

  • SAP: SAP randomly drops neurons using value-weighted probabilities, retaining and scaling surviving activations as in dropout.The defense applies randomized neuron dropping at each layer, with probabilities proportional to neurons’ absolute values.
  • SAP: Gradients from a single random realization are ineffective, so the attack estimates expected gradients over multiple SAP randomizations.The authors found k = 10 randomized gradient evaluations provided useful gradients and addressed exploding gradients caused by division near zero.
  • SAP: 9% model accuracy remained at ϵ = .015, while 0% remained at ϵ = 0.031 against SAP.Under the stricter criterion requiring misclassification 10 times out of 10, accuracy was below 10% at ϵ = 0.031.
  • Randomized Input Transformations: The randomized input defense rescales 299 × 299 images to r × r, with r ∈[299, 331), then zero-pads them to 331×331.The randomization layer is applied before the classifier.
  • Randomized Input Transformations: 100% targeted attack success rate reduced classifier accuracy from 32.8% to 0.0% at maximum ℓ∞ perturbation ϵ = 0.031.EOT bypassed overfitting to a fixed ensemble by optimizing over the discrete distribution of transformations; success required misclassification 10 times out of 10.

5.4. Vanishing & Exploding Gradients

BPDA-based attacks circumvent PixelDefend despite its claimed resistance to end-to-end differentiation from vanishing gradients. Defense-GAN also admits adversarial examples on its generator manifold, although the BPDA attack succeeds only partially.

  • PixelDefend: 46% accuracy: PixelDefend claims this result on CIFAR-10 with a vanilla ResNet at ϵ = 0.031.The authors dismiss end-to-end attacks because differentiating through the unrolled defense suffers from vanishing gradients and high computation cost.
  • PixelDefend: BPDA approximates gradients through PixelDefend’s unrolled procedure, enabling a successful end-to-end attack.The evaluation explicitly sidesteps the vanishing-gradient problem by using BPDA.
  • Defense-GAN: 45% success rate: BPDA evades Defense-GAN after adversarial examples are constructed on the generator’s manifold, where a perfect projector would not modify them.Defense-GAN’s imperfect gradient-descent projection does not perfectly preserve manifold points.

6. Discussion

The discussion argues that evaluating adversarial defenses requires precise, realistic threat models and fully specified, testable robustness claims. It also emphasizes adaptive attacks that use complete knowledge of the defense within the stated threat model.

  • Threat models: Threat models should precisely specify what the adversary knows or can access, including architecture, weights, training, randomness, and query outputs.These aspects may be revealed or kept secret, but the threat model must make the security setting explicit.
  • Threat models: Compelling threat models should avoid unrealistic computational limits and at least grant architecture and training-algorithm knowledge plus query access.Artificially restricting attack iterations does not increase robustness when defenses are otherwise equally robust.
  • Testable claims: Specific, testable claims should state the threat model, distortion bound, attacker access, and resulting accuracy.The paper gives the example of 90% accuracy under ℓ∞ distortion with ϵ = 0.031 and full white-box access.
  • Testable claims: Complete defense specifications, hyperparameters, source code, and pretrained models are prerequisites for testing robustness claims.Four studied defenses had released complete source code when the paper was written.
  • Adaptive attacks: A strong defense requires an adaptive attack constructed after specification, using knowledge of the defense and respecting only the threat model.The discussion also recommends concluding evaluations with an adaptive attack after any defense modification.

7. Conclusion

The paper identifies obfuscated gradients as a source of misleading robustness, develops three techniques to bypass three types, and emphasizes thorough evaluation against future attacks.

  • 7. Conclusion: The paper identifies obfuscated gradients, which make standard gradient-based methods fail to generate adversarial examples.It argues defenses must withstand future attacks, not only existing ones.
  • 7. Conclusion: The authors develop three attack techniques to bypass three different types of obfuscated gradients.
  • 7. Conclusion: Future defenses should avoid relying on obfuscated gradients or other methods that only prevent gradient descent-based attacks from creating perceived robustness.The authors recommend using their evaluation approach to detect this reliance.
  • 7. Conclusion: Careful, thorough evaluation is a critical step when designing defenses against adversarial examples.

A. Local Intrinsic Dimensionality

The section describes LID as a detector that distinguishes adversarial from normal images using neighborhood-distance statistics and activation-layer measurements. It finds that LID-based detection can be circumvented, including because gradient-based optimization of LID loss fails and high-confidence adversarial examples can fool the detector.

  • Motivation: LID is significantly larger for adversarial examples generated by existing attacks than for normal images, enabling a classifier to distinguish them.The authors assess LID’s ability to detect different attack types rather than presenting it as a defense itself.
  • LID computation: LID is approximated from distances to the k nearest neighbors in a clean-example mini-batch, with k controlling the neighborhood size.The method uses a distance metric over samples and computes nearest-neighbor-based LID values.
  • Detector construction: The authors compute LID vectors from activation-layer distances and train logistic regression on clean and training-set adversarial examples to detect attacks.They report releasing the complete source code for this detector construction.
  • Adaptive attack: 0% success is achieved by the modified attack, because gradient descent optimizes distances to the current k nearest neighbors rather than the optimal neighbor set.Although LID computation is differentiable, nearest-neighbor changes make the local gradient unrepresentative of the true optimization direction.
  • Evaluation: High-confidence adversarial examples generated without knowledge of the detector are sufficient to fool LID detection, demonstrating that the metric can be circumvented.The evaluation uses a detector trained on Carlini and Wagner’s ℓ2 attack and motivates testing detectors against attacks that explicitly target them.

B. Defense-GAN

Defense-GAN classifies inputs by projecting them toward a GAN-generated image manifold, but its approximate projection fails to remove manifold adversarial examples. Attacks exploiting this limitation reduce accuracy to 55% under maximum normalized distortion of .0051, while perfect-manifold-projection attacks achieve 100% success with increased distortion.

  • Defense mechanism: Defense-GAN replaces f(x) with f(arg min_z |G(z) − x|), approximating projection onto a generator-defined image manifold via gradient descent from random initializations.The generator maps latent samples to images resembling the training data.
  • Projection failure: Adversarial examples exist directly on the generator’s data manifold, but Defense-GAN often fails to identify them because its implementation does not completely project inputs.The projection process can move an adversarial example off the generator projection.
  • Evaluation A: 1.46× higher mean distortion is required than for the unsecured classifier: 0.0027 versus 0.0019 per-pixel normalized distortion.The corresponding un-normalized unsecured-classifier distortion is 1.45.
  • Evaluation B: 55% model accuracy remains under BPDA, using a maximum normalized distortion of .0051 for successful attacks against the actual implementation.The attack is regularized for ℓ2 distortion and replaces GAN projection with gradient descent on the GAN.
Loading 1802.00420v4…