Source-linked AI summary
On Evaluating Adversarial Robustness
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, Alexey Kurakin
TL;DR
Incorrect or incomplete evaluations make genuine progress on adversarial defenses difficult to identify. The paper responds with practical methodology, principles, and a checklist for researchers and reviewers, while recognizing that no single procedure covers every evaluation.
Problem
Most proposed defenses are quickly shown to have incorrect or incomplete evaluations, making genuine progress difficult to identify.
Method
The paper develops practical evaluation advice by discussing methodological foundations, principles, best practices, and common pitfalls.
Results
The paper presents advice intended to help defense builders, readers, and reviewers evaluate adversarial robustness and identify evaluation oversights.
Takeaways & Limitations
Evaluations should actively test proposed defenses with sufficiently different, carefully tuned, and customized attacks modeled on future adversaries.
Takeaways & Limitations
The paper does not provide a definitive procedure for all evaluations and states that some recommendations may not apply to every evaluation class.
Abstract
from arXiv · showhide
Correctly evaluating defenses against adversarial examples has proven to be extremely difficult. Despite the significant amount of recent work attempting to design defenses that withstand adaptive attacks, few have succeeded; most papers that propose defenses are quickly shown to be incorrect. We believe a large contributing factor is the difficulty of performing security evaluations. In this paper, we discuss the methodological foundations, review commonly accepted best practices, and suggest new methods for evaluating defenses to adversarial examples. We hope that both researchers developing defenses as well as readers and reviewers who wish to understand the completeness of an evaluation consider our advice in order to avoid common pitfalls.
1 Google Brain 2 MIT 3 University of Tübingen
The document identifies its authors, provides a correspondence repository, and records an update date.
- The listed authors are ordered according to their contributions, and the list is dynamic.
- Correspondence is directed to the paper’s GitHub repository.
- The document was last updated on 18 February, 2019.
1 Introduction
The paper addresses widespread incorrect or incomplete defense evaluations by offering practical evaluation advice, guiding principles, and a checklist of common pitfalls for researchers and reviewers.
- Most proposed defenses are quickly shown to have incorrect or incomplete evaluations, making genuine progress difficult to identify.
- The paper provides practical advice for evaluating defenses intended to be robust to adversarial examples.
- Section 2 discusses principles and methodologies that should guide defense evaluations.
- Sections 3–5 provide a checklist of common evaluation errors to help researchers avoid pitfalls.
- The advice is intended for defense builders and for readers or reviewers assessing evaluation oversights.
- The paper is intended to be a living document that researchers can further improve.
2 Principles of Rigorous Evaluations
Rigorous defense evaluation begins by specifying the motivation and threat model, then approximating worst-case robustness through attacks tailored to the defense. Because exact adversarial risk is usually intractable, evaluations require explicit assumptions, meaningful capability constraints, and active efforts to defeat the proposed defense.
- Evaluation motivation: Evaluation methodology should reflect whether the goal is defending against a real-world adversary or measuring worst-case robustness.Different motivations imply different assumptions about the adversary and therefore different evaluation procedures.
- Threat models: A threat model specifies the adversary’s goals, knowledge, capabilities, and the security guarantees the defense is intended to provide.It makes defense claims falsifiable by identifying experiments that could contradict them.
- Adversary goals and capabilities: Adversarial examples require explicit goals and reasonable capability constraints, because unconstrained attacks can alter the system, data pipeline, model, or input semantics.Goals may include untargeted misclassification, source-target misclassification, or a domain-specific error such as malware being classified as benign.
- Adversarial capabilities: Small perturbation constraints such as D(x, x′) ≤ ε make robustness measurable, but the appropriate metric and budget depend on the task and may not capture realistic threats.For example, malware attacks must preserve malicious behavior while evading detection, rather than merely satisfy a small input-distance constraint.
- Measuring robustness: Worst-case adversarial risk is usually intractable to compute exactly, so practical evaluations must approximate it with attacks or other procedures.This makes security evaluation fundamentally harder than estimating average-case robustness by sampling.
- Adaptive evaluation: Evaluators should actively try to defeat the specific defense with diverse, carefully tuned attacks and adaptive attack designs.The evaluator should ask why existing attacks fail and customize or replace them to approximate the behavior of a motivated future adversary.
3 Specific Recommendations: Evaluation Checklist
The checklist translates general evaluation advice into actionable checks for common adversarial-robustness flaws, while warning that it is neither complete nor universally applicable. It emphasizes threat-model clarity, diverse and well-tuned attacks, convergence checks, and broader validation.
- The checklist is neither necessary nor sufficient for a complete evaluation, and some ineffective defenses may still satisfy all its recommendations.
- The checklist is a living document intended to be updated as common evaluation flaws change over time.
- Threat model: Evaluators should define a precise threat model covering attacker knowledge, goals, capabilities, and the relevant perturbation space.
- Attack diversity: Use diverse attacks rather than training-time attacks or nearly identical methods, and include gradient-free, hard-label, and transferability attacks when applicable.
- Defense-specific checks: For randomized or non-differentiable defenses, account for randomness, apply differentiable techniques such as BPDA when applicable, and verify attacks on corresponding undefended models.
- Attack validation: Check attack convergence by doubling iterations, plotting effectiveness against iterations, and exploring step sizes and other hyperparameters.
- Additional evidence: Broader checks include provable approaches, random-noise attacks, targeted and untargeted attacks, component ablations, attack validation on broken models, and cross-domain applicability.
4 Evaluation Recommendations
The paper expands its checklist with rationale for heuristic evaluations, favoring provable approaches where possible but recognizing their computational and coverage limits. It recommends defense-specific, adaptive attack evaluation rather than relying on generic or training-matched attacks.
- Provable approaches: Provable robustness is preferable to relying only on heuristic evaluation, but current verification methods can be computationally intractable for modestly sized neural networks.
- Provable approaches: Provable techniques generally certify robustness only for a specific example set X and distortion threshold ε, not for examples outside X.
- Attack selection: Optimization-based attacks are powerful because they use gradients, but attacks should be adapted and hyperparameters optimized for the particular defense.
- Attack selection: FGS should never be the sole evaluation attack because defenses can achieve near-perfect FGS robustness through gradient masking.
- Attack selection: Evaluations should not reuse only the attack used during adversarial training, because overfitting to that attack can overestimate robustness.
- Attack selection: Nearly identical attacks provide less useful evidence than genuinely diverse attack approaches.
4.4 Apply Gradient-Free Attacks
Gradient-free and hard-label attacks help detect gradient masking because they can succeed where standard gradient-based attacks fail. However, current gradient-free methods often struggle against randomized defenses.
- Gradient-free attacks are important tests for gradient masking, and they should generally perform worse than gradient-based attacks overall and per example.
- Gradient-free attacks: ZOO numerically estimates gradients and performs gradient descent, but may fail when the loss surface is difficult to optimize.
- Gradient-free attacks: SPSA was designed to evaluate adversarial defenses and has broken many, but its numerous hyperparameters can be difficult to tune.
- Gradient-free attacks: NES can generate adversarial examples with limited queries, though this may produce higher-distortion examples; variants also support restricted-output settings.
- Hard-label attacks: Hard-label attacks require only the model’s predicted label, making them slower but less vulnerable to defenses accidentally blocking them.
- Hard-label attacks: The Boundary Attack searches along the decision boundary and can rival strong white-box attacks in minimum adversarial distance, at the cost of many more queries.
- Limitations: Current state-of-the-art gradient-free attacks often perform poorly against randomized models, although this limitation may change as attacks improve.
4.5 Perform a Transferability Analysis
Transferability analysis attacks a defended model using adversarial examples generated on an independent substitute model. This is especially useful against gradient-masking defenses because it avoids optimizing over the defended model’s poorly behaved landscape.
- Adversarial examples often transfer across models with different architectures and training sets.
- Transfer attacks: A transfer attack generates high-confidence adversarial examples on a substitute model and applies them to the target model.
- Transfer attacks: Transfer attacks can circumvent gradient masking because they do not optimize directly over the defended model’s landscape.
- Transfer attacks: Evaluations should test whether a defense breaks transferability; failure to do so suggests the defense may not be strong.
- Substitute models: A suitable substitute is as similar as possible to the defended model and trained on the same data, such as an undefended baseline with added defense layers omitted.
4.6 Properly Ensemble over Randomness
Randomized defenses require attacks that account for stochastic outputs and gradients, while fixed-randomness checks help distinguish genuine robustness from attack failure. Non-differentiable or poorly differentiable components further complicate evaluation and may require surrogate gradients.
- Randomized defenses: Ensemble attacks over the defense’s randomness because stochastic outputs and gradients can make standard optimization attacks unreliable.Repeating each attack step can provide more reliable estimates.
- Randomized defenses: Exponential randomness does not necessarily make attacks exponentially harder, since examples can remain adversarial across moderate numbers of random choices.Attackers may construct inputs consistently adversarial over sampled randomness.
- Randomized defenses: First verify that attacks succeed with randomness fixed, then gradually re-enable randomness to test whether the defense’s stochasticity is genuinely relevant.Failure with randomness disabled suggests the attack itself needs correction.
- Differentiability: Non-differentiable or uselessly differentiable layers make complete evaluation harder because they increase the likelihood of gradient masking.Quantization, added randomness, and saturated activations can create these problems.
- Differentiability: Use differentiable implementations or BPDA when possible, such as replacing a near-identity denoiser’s backward gradient with the identity while retaining its exact forward computation.This preserves the defense’s forward behavior while supplying attack gradient information.
4.8 Verify Attack Convergence
Attack evaluations should demonstrate convergence rather than rely on an arbitrary iteration budget. Stronger settings, random starts, and transparent hyperparameter reporting help establish that apparent robustness is not an artifact of an incomplete attack.
- Convergence checks: Plot attack success against gradient-descent iterations and continue increasing iterations until the curve plateaus.The required iteration count depends on step size, allowed distortion, and dataset complexity.
- Convergence checks: White-box attacks may converge in under 100 iterations on CIFAR-10 or 1000 on ImageNet at ℓ∞ distortion 8/255, whereas black-box attacks can require over 100,000 queries.Iteration and query requirements vary with the dataset and distortion metric.
- Convergence checks: Threat models generally should not distinguish between 100 and 1000 gradient-descent iterations, so evaluations should include strong attacks with many iterations.Restricting the attacker’s approach to a small budget can produce an unrealistic evaluation.
- Convergence checks: Double the selected iteration count and check whether attack success improves as a practical test that the attack has run long enough.This test complements, rather than replaces, other methods for selecting the iteration budget.
- Attack setup: Random-offset initialization can reduce required distortion by 10% and help avoid gradient masking, so it should be tested at least once.Repeated random starts are not always necessary, but a robustness check is useful.
- Attack setup: Choose stronger attack hyperparameters when uncertain because parameter settings can change attack success rates by orders of magnitude.All hyperparameters and their selection procedures should be reported for reproducibility.
4.10 Test General Robustness for General-Purpose Defenses
General-purpose robustness claims should be tested with diverse, easy-to-apply attacks that do not rely solely on gradients. These checks should not be used directly for training, and random search can reveal weaknesses missed by other methods.
- General-purpose evaluation: Defenses claiming general robustness should also be evaluated with easy-to-apply attacks, rather than only threat-model-specific tests.The paper highlights several image-based starting points.
- General-purpose evaluation: Random rotations and translations provide a brute-force attack that is not susceptible to gradient masking.This tests robustness to simple geometric transformations.
- General-purpose evaluation: Common corruptions and perturbations can test robustness to changes that may occur in practice.These attacks complement optimization-based adversarial examples.
- General-purpose evaluation: Increasing Gaussian-noise standard deviation tests resistance to random noise, to which adversarially robust models tend to be more resistant than standard models.Noise strength should be varied progressively.
- Evaluation protocol: Do not train directly on these evaluation tests when assessing general-purpose robustness, because doing so would counter their intended purpose.The tests are intended to remain independent checks.
- Random search: Random search is a sanity check: if it finds adversarial examples missed by other attacks, those attacks may be improvable.The recommended procedure samples broadly, then searches within progressively smaller distortion limits.
4.12 Targeted and Untargeted Attacks
Targeted and untargeted attacks are not interchangeable in practice, so evaluations should try both formulations. Defense-specific attacks also require validation on models known or designed to be insecure before their apparent effectiveness is trusted.
- Targeted versus untargeted: Although untargeted attacks are theoretically easier, targeting individual alternative classes can outperform a single untargeted attack in practice.Testing multiple target classes can therefore expose weaknesses missed by one untargeted formulation.
- Targeted versus untargeted: Untargeted attacks typically reduce confidence in the correct class, whereas targeted attacks increase confidence in another class, so both formulations can be useful.Their objectives are not direct inverses.
- Defense diagnosis: Remove defense components and attack the remaining modified model to test whether apparent robustness comes from non-defense changes.This isolates unintended effects introduced alongside the defense.
- Defense diagnosis: If increasing a tunable constant is believed to strengthen the defense, decreasing it should make the model vulnerable to attack.This checks whether the claimed parameter effect behaves as expected.
- Attack validation: Defense-specific attack algorithms must be validated because an ineffective attack can appear unsuccessful against every model.Attack quality is a separate evaluation question from the defense’s robustness.
- Attack validation: Test a new attack on alternate models known or intentionally designed to be insecure and verify that it can effectively break them.Failure on such models indicates that the attack itself needs improvement.
5 Analysis Recommendations
The paper recommends sanity checks, broad attack coverage, and careful reporting to detect flawed attacks and compare defenses fairly. It also stresses evaluating scope, attack strength, and per-example outcomes rather than relying on isolated or averaged results.
- Literature Review: Prior defense and attack literature should be reviewed because existing attacks have defeated many defenses without modification.Similar prior defenses can reveal known vulnerabilities and relevant attack approaches.
- Attack Sanity Checks: Iterative attacks should outperform single-step attacks, while attack success rate should increase with the perturbation budget.Deviations can indicate implementation errors; plotting success rate or model loss across iterations helps diagnose them.
- Distortion Analysis: Accuracy-versus-distortion and attack-success-versus-distortion curves expose impossible or suspicious robustness claims, including cases where high distortion should approach random guessing.The relevant distortion thresholds depend on the dataset, and some bounds are theoretically impossible to exceed.
- Unbounded Attacks: Unbounded attacks should eventually reach 100% success, but their evaluation should report the distortion required rather than success rate alone.Useful summaries include success rate or model accuracy versus distortion and the mean or median distance to adversarial examples.
- Attack Comparison: White-box attacks should outperform black-box attacks in principle, so stronger black-box results warrant investigation of gradient masking or ineffective gradient-based attacks.Researchers should inspect examples found by black-box but not white-box attacks and investigate their common properties.
- Reporting: Per-example reporting is preferable because it supports meaningful defense comparisons across attacks, while evaluations should report attack details and relevant hyperparameters.A shorter table of diverse, defense-adapted attacks is preferable to a massive table of similar unadapted attacks.
6 Conclusion
The paper urges extreme caution when evaluating adversarial defenses and offers recommendations for researchers, readers, and reviewers. It presents the guidance as a non-definitive checklist and supports continued re-evaluation of proposed defenses.
- Defense evaluations require extreme caution and skepticism because researchers may unintentionally deceive themselves about their results.
- The paper develops recommendations addressing common flaws for defense developers and for readers or reviewers assessing evaluation thoroughness.
- The recommendations are not intended to be a definitive list of experiments, and future re-evaluations with new adaptive attacks remain valuable.