PaperlayerView exact revision on arXiv

Source-linked AI summary

On Adaptive Attacks to Adversarial Example Defenses

Florian Tramer, Nicholas Carlini, Wieland Brendel, Aleksander Madry

arXiv:2002.08347v2cs.LGcs.CRstat.ML

TL;DR

The paper addresses whether defenses remain robust when evaluated with adaptive attacks, given that many such evaluations may be incomplete. It documents defense-specific attack development across thirteen selected defenses and finds that all can be circumvented with improved attacks. The authors therefore present a methodology for comprehensive adaptive evaluation focused on each defense’s weak links.

  • Problem

    Adaptive evaluations of adversarial-example defenses can be incomplete even when they include attacks labeled adaptive, leaving the true robustness of the defenses uncertain.

  • Method

    The paper analyzes thirteen defenses by reading their papers and code, forming vulnerability hypotheses, and iteratively developing defense-specific adaptive attacks.

  • Results

    All thirteen analyzed defenses could be circumvented with improved attacks despite having performed adaptive-attack evaluations.

  • Takeaways & Limitations

    Robustness evaluations should develop comprehensive adaptive attacks that explicitly target each defense’s weakest links rather than rely mainly on prior attacks.

  • Takeaways & Limitations

    The logit-matching attack may be ineffective when the defense uses a robust classifier because matching another class’s logit representation is difficult for adversarially trained models.

Abstract

from arXiv · show

Adaptive attacks have (rightfully) become the de facto standard for evaluating defenses to adversarial examples. We find, however, that typical adaptive evaluations are incomplete. We demonstrate that thirteen defenses recently published at ICLR, ICML and NeurIPS---and chosen for illustrative and pedagogical purposes---can be circumvented despite attempting to perform evaluations using adaptive attacks. While prior evaluation papers focused mainly on the end result---showing that a defense was ineffective---this paper focuses on laying out the methodology and the approach necessary to perform an adaptive attack. We hope that these analyses will serve as guidance on how to properly perform adaptive attacks against defenses to adversarial examples, and thus will allow the community to make further progress in building more robust models.

1 Introduction

The paper asks whether defenses evaluated with adaptive attacks are truly robust and finds that thirteen selected defenses can all be circumvented. It presents adaptive attack evaluation as a methodological problem requiring defense-specific analysis rather than merely stronger technical tools.

  • Main result: Thirteen defenses from ICLR, ICML, and NeurIPS were circumvented, substantially reducing their originally reported accuracy.The defenses were selected to illustrate diverse defensive strategies.
  • Motivation: Adaptive evaluations have become common, but many remain insufficient because attacks are reused without being redesigned for the target defense.The paper identifies incomplete adaptive evaluations as a recurring reason defenses remain vulnerable.
  • Approach: The authors document each evaluation from reading the paper and source code through hypotheses, experiments, and a successful adaptive attack.This process is intended to expose the steps needed to develop strong defense-specific attacks.
  • Scope: The study aims to demonstrate shortcomings in existing evaluation methodology rather than reduce every analyzed model’s accuracy to zero percent.The target is partial degradation from the accuracy reported in the original evaluations.
  • Methodological claim: The authors argue that proper evaluations can be built with well-known tools, making the central issue methodological rather than technical.Their attacks use simple, consistently optimized losses, although no single strategy works for every defense.
  • Evaluation setting: The paper studies white-box defenses using standard adversarial-example objectives and gradient-based procedures while releasing attack code for independent verification.The authors contacted defense authors before submission and report acknowledgment of attack effectiveness in all but one case.

3 Methodology

The methodology mirrors the actual process used to evaluate each defense: understand its operation, form vulnerability hypotheses, develop adaptive attacks, and extract lessons. This structure is designed as a reproducible case study rather than a retrospective account of only successful attacks.

  • Defense analysis: Each evaluation begins by describing how the defense works and introducing the notation needed to analyze it.Readers are encouraged to consider possible attacks before reading the authors’ method.
  • Selection: Defenses were selected from ICLR, ICML, and NeurIPS because reviewers deemed them interesting and because each illustrated a distinct concept.The authors retained their initially selected defenses rather than cherry-picking based on evaluation outcomes.
  • Initial hypotheses and experiments: The authors read each paper and inspected publicly available or requested source code to formulate hypotheses about remaining vulnerabilities.These hypotheses guide the initial experiments and later attack design.
  • Final robustness evaluation: Final adaptive attacks usually combine an improved loss function, an optimization method, and repeated refinement based on insights from attack attempts.The process adapts to evidence gathered during evaluation rather than following a fixed attack recipe.
  • Lessons learned: The paper records the evaluation flow as it occurred so readers can use it as a start-to-finish case study for future evaluations.The authors explicitly avoid retroactively reconstructing how they wish they had discovered each flaw.
  • Evaluation goal: The goal is to assess whether initial robustness evaluations were appropriate and, when necessary, demonstrate stronger adaptive attacks without rejecting the underlying defense techniques wholesale.Claims about improved robustness require appropriate supporting evaluation.

4 Recurring Attack Themes

The recurring themes favor simple, defense-specific adaptive attacks whose objectives and optimization methods are chosen to expose weak links. The evaluations also show that gradient masking, transfer attacks, and weak training attacks can distort robustness conclusions.

  • Overview: The authors identify six recurring attack themes plus a meta-theme spanning multiple defense evaluations.Table 1 summarizes which themes apply to the defenses studied.
  • T0: Strive for simplicity: Simple attacks are preferred because failures are easier to diagnose, while complexity is added only when straightforward gradient descent is insufficient.The attack design stays as close as possible to gradient descent with an appropriate loss function.
  • T1–T2: Target the defense: End-to-end differentiable defenses should be attacked directly, while complex defenses should be reduced to the one or two components most responsible for vulnerability.Targeting important components can make attacks simpler, stronger, and easier to optimize.
  • T3–T4: Adapt the objective: Attack objectives should be adapted for optimization, including targeted or multi-targeted objectives and feature adversaries when they better expose a defense’s weakness.The paper also emphasizes that loss functions must correlate consistently with attack success, not merely be optimizable.
  • T5: Optimize effectively: Optimization requires an appropriate algorithm and hyperparameters, since white-box gradient attacks are not always the best way to minimize a useful loss.The authors consider score-based and black-box methods when gradient-based optimization is unsuitable.
  • T6: Attack adversarial training: Weak adaptive attacks can make defenses combined with adversarial training less robust than adversarial training alone.If training attacks fail to find adversarial examples reliably, the resulting model may not receive the intended training signal.
  • Gradient masking and transfer: For gradient-masked defenses, score-based or decision-based attacks can match or outperform gradient-based attacks, whereas transfer attacks generally require highly similar source models and often succeed less than half the time.The paper also reports that short, non-repeated evaluations and metric mismatches can make robustness difficult to assess.

6 The Odds are Odd

The defense detects adversarial examples through noise-induced changes in class-logit differences, but adaptive attacks can match benign confidence and boundary proximity while evading detection. A feature-level logit-matching attack combined with appropriately targeted EOT defeats the defense more effectively than standard attacks.

  • Defense mechanism: The defense tests whether class-logit differences remain robust when inputs are perturbed with sampled noise.It compares clean and noisy logits and rejects an input when any estimated change exceeds a threshold calibrated to a fixed false-positive rate.
  • Evaluation diagnosis: The evaluation reports no clear red flags because it provides few statistics beyond absolute detection numbers.The authors note that additional statistics would help diagnose potential evaluation errors.
  • Initial attack analysis: Standard attacks expose detectable signatures: PGD produces abnormally high confidence, whereas C&W produces examples unusually close to the decision boundary.These signatures correspond to large values of the defense’s detection statistics and motivate constructing examples with benign-like confidence and robustness to noise.
  • Adaptive attack: The proposed feature-level attack matches an adversarial example’s logits to those of a benign example from a target class using PGD.This logit-matching objective targets internal representations rather than only the predicted output, keeping the confidence-related statistic near benign values.
  • Final robustness evaluation: Adding EOT to the logit-matching objective reduces the defense’s detection rate to 0% while still fooling the classifier.Before this addition, detection was 17% with one noise source and 10% with ten noise sources; EOT is applied to the adapted objective rather than standard PGD.
  • Lessons and limitation: For randomized defenses, EOT alone is insufficient when the attack objective is not adapted to the defense’s detection mechanism.The defense illustrates how a simple logit-matching attack can work across multiple defenses, while the analyzed attack may be ineffective against a robust classifier.

1. How effective is this paper’s detection mechanism when instantiated with a robust classifier?

The defense combines a generative classifier with stochastic latent sampling and a KL-divergence detector, making evaluation challenging. Adaptive attacks that isolate important loss terms, use feature alignment, or account for randomness substantially circumvent both components.

  • How the defense works: The classifier uses class-conditional encoders, two decoders, random latent samples, and four scoring terms to produce class logits.For each class, the defense samples N latent vectors and combines reconstruction, label, prior, and posterior terms.
  • How the defense works: A KL-divergence detector rejects inputs whose softmax probabilities differ substantially from same-class training means.The detector operates on class probabilities derived from the classifier’s logits.
  • Initial hypotheses and experiments: Standard attacks with large perturbation bounds still fail to reduce classifier accuracy to 0%, motivating attacks tailored to the defense’s internal terms.The reported ineffective attacks use ϵ = 0.5 on MNIST and ϵ = 25/255 on CIFAR-10.
  • Initial hypotheses and experiments: Targeting the cross-entropy term and increasing latent samples during attack generation reduces accuracy below 5% on both MNIST and CIFAR-10.Attack samples generated with K = 100 are evaluated on the original K = 10 defense and remain robust to model variance.
  • Final robustness evaluation: Feature-aligned attacks evade the combined classifier and detector by matching internal representations or VGG features from clean examples of another class.The feature-adversary attack achieves 99% success on CIFAR-10, while direct VGG-feature matching succeeds with 100% probability.
  • Final robustness evaluation: On MNIST, the same attacks succeed with 90% probability under internal randomness, while deterministic latent sampling enables 100% success against the classifier and detector.The remaining failure rate matches the model’s approximately 10% failure rate on clean inputs.

8 Robust Sparse Fourier Transform

The Robust Sparse Fourier Transform defense compresses images through iterative Fourier-domain thresholding before classification. Its claimed robustness is not evaluated adaptively, and the paper’s robustness interpretation is constrained by the distortion levels considered.

  • How the defense works: The defense projects each image onto its top-k discrete cosine coefficients, reconstructs it, and classifies the recovered image.Training on compressed images is intended to preserve classification accuracy after preprocessing.
  • How the defense works: Iterative Hard Thresholding repeats Fourier compression and inversion for T iterations before producing the preprocessed image.The procedure is conceptually similar to another compression-based defense but lacks its additional randomization challenge.
  • Evaluation scope: The paper does not analyze adaptive robustness; showing that preprocessing removes adversariality from a base classifier is insufficient to establish defense robustness.The evaluation should instead attack the combined preprocessing-and-classifier function.
  • Evaluation scope: The defense claims over 75% accuracy at ℓ0 robustness 55 on MNIST, although ℓ0 distortion 25 has been reported sufficient to change human classifications.This comparison makes the claimed robustness level difficult to interpret as a meaningful human-aligned boundary.
  • Adaptive evaluation: Implementing the preprocessing as a differentiable function allows an ℓ0 attack to be applied directly to the combined defense without further modification.The procedure contains no non-differentiable components in the described implementation.

9 Rethinking Softmax Cross Entropy

This defense replaces standard softmax cross entropy with a feature-space classifier that assigns inputs to the nearest class centroid. Directly optimizing the defense’s target distances is far more effective than attacking through standard cross entropy.

  • How the defense works: The classifier maps inputs to features g(x) and assigns the class whose centroid µ_i is closest in squared Euclidean distance.The feature outputs are not class scores; classification is based on distances to K target vectors.
  • Motivation and evaluation: The paper was selected as a representative alternative-loss defense and a strengthening of an earlier adversarial-robustness defense.Its original evaluation uses standard attacks maximizing cross-entropy over softmax predictions.
  • Attack design: Changing the training loss motivates studying an attack loss that directly reflects the defense’s classification rule rather than relying on standard softmax cross entropy.The authors specifically hypothesize that the defense’s loss surface may be poorly suited to standard attacks.
  • Initial experiments: Dividing logits by a large constant reduces CIFAR-10 accuracy from 24% to 18% but does not break the defense completely.The result suggests that modifying logits alone is insufficient for a complete attack.

10 Error Correcting Codes

The error-correcting-code ensemble defense is vulnerable to numerically unstable prediction and attack losses. Replacing unstable operations and strengthening PGD reduces accuracy far below the reported result.

  • How the Defense Works: The defense trains multiple binary classifiers whose outputs encode each class as a codeword for error-correcting aggregation.Each classifier learns one bit of a binary codeword assigned to each class.
  • Initial Hypotheses and Experiments: Nearly 40% accuracy at ℓ∞ distortion ε = 0.5 suggested gradient masking because the defense disrupted gradient descent.The reported accuracy did not reach 0% at that distortion, contrary to the defense paper’s interpretation.
  • Initial Hypotheses and Experiments: Removing sigmoid and logarithm operations and using the Carlini–Wagner hinge loss reduced CIFAR-10 accuracy below 20% at ε = 0.031.The modified attack addressed numerical instability in the defended prediction pipeline.
  • Final Robustness Evaluation: Multi-targeted attacks, five random restarts, and accepting successful intermediate PGD iterates reduced accuracy below 5% at ε = 0.031, versus the reported 57%.These additions improved attack success beyond the numerically stabilized loss alone.
  • Lessons: Combining multiple neural networks is error-prone, but simple attacks targeting weak links can be effective.Several small attack improvements can jointly drive already-low accuracy near zero.

11 Ensemble Diversity

The ensemble-diversity defense does not withstand properly converged and strategically diverse attacks. Increasing iterations, repeating attacks, and adding B&B sharply lowers its accuracy.

  • How the Defense Works: The defense adds a diversity regularizer to an ensemble objective, using the volume spanned by non-maximal class probabilities alongside cross-entropy and entropy terms.The goal is to make minimal adversarial examples harder to find while retaining accuracy.
  • Initial Hypotheses and Experiments: Reported results contained anomalies, including 30.4% versus 27.8% robustness at ε = 0.02 and BIM outperforming PGD.These patterns suggested that the attacks might not have converged.
  • Final Robustness Evaluation: Increasing PGD iterations reduced accuracy from 48% to 26% with a larger step size, to 20% at 50 iterations, and to 10% at 250 iterations.Further iteration increases did not change the result, suggesting convergence at 250 iterations.
  • Final Robustness Evaluation: Repeating the attack three times reduced accuracy to 7% for 250 iterations, after which an ℓ∞ B&B attack was used against remaining difficult samples.B&B provides a substantially different strategy from PGD.
  • Lessons: Effective evaluation requires checking convergence, tuning hyperparameters, repeating randomized attacks, and including substantially different attack strategies.BIM, PGD, and MIM are too similar to provide independent evidence of robustness.
  • Mixed-Precision Ensemble: A mixed-precision ensemble defense was also evaluated, and 100 PGD iterations reduced its accuracy to 1.5% at ε = 0.031.The original evaluation reported 13.5% accuracy at ε = 0.1 with 40 PGD iterations.

13 Temporal Dependency

The temporal-dependency detector compares prefix and full-audio transcriptions, but its adaptive evaluation exposed implementation and optimization concerns. The section emphasizes targeted, randomized-prefix attacks and careful interpretation of unbounded evaluations.

  • How the Defense Works: The detector compares the transcription of an audio prefix with the corresponding prefix of the full-audio transcription.It computes a similarity score and labels the input adversarial when the score is large.
  • How the Defense Works: Character error rate, implemented as whitespace-omitting Levenshtein distance, is used as the simplest similarity metric.The paper reports that the available similarity metrics perform roughly the same.
  • How the Defense Works: The strongest evaluation attacks the randomized detector across prefix lengths, including k = 0.2 to 0.8 and the difficult randomized version.The majority of the paper uses k = t/2, while the evaluation targets random prefix selection.
  • Initial Hypotheses and Experiments: An unbounded attack failed to fool the detector 100% of the time, which raised concerns about the attack evaluation.The authors reason that an unbounded attacker should eventually succeed by adding enough noise to destroy recognizability.
  • Final Robustness Evaluation: For unbounded targeted attacks, the relevant result is median distortion rather than success rate at a fixed bound.The section distinguishes these experiments from bounded image-classification evaluations.
  • Lessons: The section highlights that subtle attack-implementation differences can determine effectiveness and that unprincipled randomness rarely improves robustness.These are presented as general lessons from the defense evaluation.

14 Mixup Inference

The Mixup inference defense interpolates inputs with randomly selected images, but its adaptive evaluation must differentiate through the complete stochastic mechanism. A stronger attack nearly restores baseline accuracy and exposes weaknesses in prior adaptive evaluation.

  • How the Defense Works: Mixup inference interpolates each input with randomly selected images and averages the undefended model’s logits over those interpolations.The defense uses K = 1 during training and K = 15 for OL or K = 5 for PL during testing.
  • How the Defense Works: The defense is motivated by reducing perturbations through interpolation and shifting inputs away from the local region where perturbations are hypothesized to act.The mechanism uses stochastic, non-local mixing between images.
  • Initial Hypotheses and Experiments: A large portion of the evaluation uses an oblivious threat model, which is useful as a sanity check but not as the main robustness assessment.The paper recommends focusing the main evaluation on adaptive attacks.
  • Initial Hypotheses and Experiments: The prior adaptive attack averaged adversarial examples from ineffective attacks, whereas averaging gradients during each attack step would be stronger.The adaptive attack’s reported and appendix descriptions also differed from the implementation found in source code.
  • Final Robustness Evaluation: Backpropagating through the complete mixing mechanism with random samples at every step reduced defended accuracy to 43.9%, close to the 42.5% base-model accuracy.The evaluation used 50-step PGD with a step size of 1/255.
  • Related Evaluation: For the related USVT defense, BPDA+EOT reduced accuracy to 15% for USVT and 32% for USVTADV.The attack averaged gradients over 40 random masks during PGD.
  • Related Evaluation: Randomness caused output fluctuations and made robust adversarial examples harder to produce, but BPDA applied to the full preprocessing step remained stronger than a USVT-only approximation.The USVT reconstruction was roughly identical to the randomly masked image rather than close to the original.
  • Related Evaluation: The nuclear-norm defense reached 13% accuracy under the best USVT attack, while additional adaptive experiments were omitted because its preprocessing was computationally expensive.The nuclear-norm minimization took about one second per input on CPU.

16 Asymmetrical Adversarial Training

This defense uses adversarially trained class-specific detectors in integrated or generative classifiers, but its adaptive attacks optimize objectives that can waste perturbation budget or fail to enforce misclassification. Targeted multi-targeted attacks substantially reduce robust accuracy, while the training objective may leave the true-class detector vulnerable on adversarial examples.

  • How the Defense Works: The defense uses K adversarially trained detectors, with the correct detector recognizing benign inputs and other detectors rejecting perturbed inputs.Each detector produces a class-specific logit, and training maximizes the correct-class response while minimizing non-target responses on perturbed inputs.
  • How the Defense Works: The defense offers integrated and generative classifiers that reject inputs using class-specific or maximum detector-logit thresholds.The integrated classifier checks the detector corresponding to the base classifier’s prediction; the generative classifier predicts from the largest detector score.
  • Initial Hypotheses and Experiments: The integrated-classifier attack wastes perturbation budget by maximizing every incorrect detector instead of only the detector for the base classifier’s predicted class.Once the base classifier is fooled, bypassing the defense requires fooling only the detector associated with its resulting prediction.
  • Initial Hypotheses and Experiments: The generative-classifier loss can fail even with unbounded perturbations because increasing all detector scores equally may leave the prediction unchanged.The loss maximizes the most confident detector other than the true class, but does not necessarily force a change in the winning detector.
  • Final Robustness Evaluation: At a False-Positive-Rate of 5%, multi-targeted PGD reduces generative-classifier robust accuracy from 55% to 37% and detector robust accuracy from 30% to 11%.The attacks use 100 PGD steps per target and retain the best adversarial example across target classes.
  • Discussion: The training objective may be incomplete because it lowers non-target detector confidence on adversarial examples without ensuring high confidence from the true-class detector.Adversarially training all detectors for each input is proposed as a potentially more robust alternative.

1. Can the accuracy of the defense be reduced further, especially for the generative classifier?

The passage raises whether other defenses might also be vulnerable to untargeted attacks reformulated as sequences of targeted attacks.

  • Open Question: The section poses whether other defenses can be attacked by formulating an untargeted attack as multiple targeted attacks.The passage presents this as an open question rather than reporting an evaluation result.

17 Turning a Weakness into a Strength

This defense combines noise-stability and optimization-step checks, but implementation and objective weaknesses expose a path to adaptive attack. Binary search over interpolated inputs then reduces accuracy nearly completely without detection.

  • How the Defense Works: The defense detects adversarial inputs using thresholds on prediction stability under Gaussian noise and the number of steps needed to generate an adversarial example.The second check treats optimization steps as a proxy for distance to the decision boundary.
  • Initial Hypotheses and Experiments: The adaptive attack combines four loss terms, but the authors argue that its complexity may prevent simultaneous optimization and allow the cross-entropy term to dominate.Two terms are non-differentiable and require BPDA, while the loss lacks separate weighting terms for each component.
  • Initial Hypotheses and Experiments: At ϵ = 0.1, cross-entropy PGD succeeds 8.5% of the time, whereas margin-loss PGD succeeds over 32% of the time.The unusually large gap motivated inspection of the authors’ open-source implementation.
  • Initial Hypotheses and Experiments: The implemented ℓ∞ PGD omits gradient-sign normalization, and the same omission in the step-counting phase makes the distance-to-boundary proxy especially important.The defense estimates boundary proximity from gradient-descent steps rather than measuring the distance directly.
  • Final Robustness Evaluation: The attack constructs high-confidence adversarial examples, interpolates them with original inputs, and searches for points that remain high-confidence while near the decision boundary.Such points can satisfy both defense properties and evade rejection.
  • Final Robustness Evaluation: Binary search replaces an expensive 1,000-point interpolation scan by recursively locating an interpolation value satisfying both defense properties.The search moves toward or away from the boundary depending on which property fails, restarting with a new adversarial example if necessary.
  • Final Robustness Evaluation: The attack reduces CIFAR-10 accuracy to 0% and ImageNet accuracy to < 1%, both at a 0% detection rate.These results use the threat model originally considered by the defense’s paper.

18 Conclusion

The paper finds that adaptive evaluation has improved, yet all thirteen analyzed defenses remain circumventable by stronger, defense-specific attacks. It therefore argues for comprehensive, hand-designed adaptive evaluations, while noting responsible-disclosure and misuse boundaries.

  • Conclusion: Nearly all studied defenses used adaptive attacks, yet improved attacks circumvented all thirteen defenses analyzed.The paper contrasts widespread adaptive-evaluation practice with its insufficiency in these cases.
  • Conclusion: Prior attacks should be treated as sanity checks because a non-robust defense can be constructed to prevent any proposed attack.The paper calls this an informal no-free-lunch principle.
  • Conclusion: Robustness evaluations should focus on comprehensive adaptive attacks that explicitly target each defense’s weakest links rather than copying prior attack templates.The paper recommends deferring non-adaptive evaluations to an appendix.
  • Conclusion: Hand-designed attacks are necessary because automated tools may only partially break defenses and cannot directly evaluate many adversarial-example detectors.AutoAttack fully breaks fewer of the shared defenses than the paper’s tailored attacks in two cited cases.
  • Conclusion: The paper recommends simple adaptive attacks that resolve optimization difficulties, since each added component creates another opportunity for error.The stated goal is stronger defense evaluation rather than unnecessary attack complexity.
  • Responsible Disclosure: Twelve of thirteen contacted authors responded and verified that the reported evaluations were accurate before publication.The authors also offered generated adversarial examples to support verification.
  • Broader Impact: The methodology could potentially be used against deployed or future systems, although the paper is intended to assist researchers and diagnose evaluation failures.The authors state that they do not believe the evaluated defenses protect deployed systems.
Loading 2002.08347v2…