Source-linked AI summary
Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
Jonathan Uesato, Brendan O'Donoghue, Aaron van den Oord, Pushmeet Kohli
TL;DR
The paper asks whether apparent adversarial robustness reflects resistance to worst-case inputs or merely weak attack evaluations. It formalizes adversarial risk and obscurity, develops stronger attack-based analyses, and finds that several defenses can have their performance dramatically reduced, motivating transparent defenses and stronger evaluation.
Problem
Common evaluations show robustness to selected attacks, but whether defended models are free from adversarial examples remains unanswered.
Method
The paper formalizes adversarial risk, frames common attacks and metrics as tractable surrogates, and uses gradient-free optimization to assess obscurity and design transparent defenses.
Results
The experiments validate security by obscurity and show that more powerful attacks dramatically reduce the performance of recently proposed defenses.
Takeaways & Limitations
Adversarial evaluations should use appropriately designed adversaries and transparent models to better assess true adversarial risk.
Takeaways & Limitations
Exact worst-case adversarial risk is computationally intractable, while local adversarial risk certifies models only at a fixed set of points.
Abstract
from arXiv · showhide
This paper investigates recently proposed approaches for defending against adversarial examples and evaluating adversarial robustness. We motivate 'adversarial risk' as an objective for achieving models robust to worst-case inputs. We then frame commonly used attacks and evaluation metrics as defining a tractable surrogate objective to the true adversarial risk. This suggests that models may optimize this surrogate rather than the true adversarial risk. We formalize this notion as 'obscurity to an adversary,' and develop tools and heuristics for identifying obscured models and designing transparent models. We demonstrate that this is a significant problem in practice by repurposing gradient-free optimization techniques into adversarial attacks, which we use to decrease the accuracy of several recently proposed defenses to near zero. Our hope is that our formulations and results will help researchers to develop more powerful defenses.
1. Introduction
The paper argues that standard adversarial evaluations can reward defenses that exploit attack weaknesses rather than withstand worst-case inputs. It formalizes adversarial risk and obscurity, then uses stronger attacks to expose this gap.
- Motivation: Adversarial examples are small input perturbations that can cause high-confidence misclassification and can be found by maximizing network loss.Such examples occur across architectures and machine-learning algorithms.
- Motivation: The central question is whether defenses eliminate adversarial examples or merely resist currently used attacks.Prior work reported robustness against commonly used attacks, leaving true worst-case robustness unresolved.
- Adversarial risk: Adversarial risk measures performance on worst-case inputs, but its exact evaluation and optimization are computationally intractable.The paper therefore treats common attacks and evaluation metrics as tractable surrogates for true adversarial risk.
- Obscurity: Obscurity describes defenses that appear robust because they exploit weaknesses in particular attacks without eliminating all adversarial inputs.The paper develops this concept to identify models optimizing the surrogate rather than true adversarial risk.
- Contributions: Gradient-free optimization techniques are repurposed as stronger attacks to measure obscurity against transfer-based and gradient-based evaluations.These attacks experimentally validate security by obscurity in recently proposed defenses.
2. Related Work
Related work covers black-box attacks and defenses, gradient masking as a source of misleading robustness evaluations, formal verification, and transparency in machine learning.
- Adversarial attacks and black-box robustness: Black-box attacks can fool classifiers without direct model access, including through printed images and 3D physical objects.Related work also includes defenses against black-box attacks and comparisons in that threat model.
- Evaluating adversarial robustness: Gradient masking can make gradient-based attacks fail, causing practitioners to mistake a model for robust when stronger adversaries can still find adversarial inputs.This concern is shared with contemporaneous work showing that many defenses merely mask gradients.
- Certificates and transparency: Neural-network verification provides techniques for computing provably worst-case adversarial examples, while transparency research shares the goal of efficiently analyzing models.The paper connects adversarial-defense analysis with both robustness certificates and model interpretability.
3. Adversarial Risk and Obscurity
The paper distinguishes expected, worst-case, and local adversarial risk, then argues that attacks used for evaluation optimize a tractable surrogate that can diverge from true robustness. It formalizes this gap as obscurity and motivates transparent models for reliable evaluation and deployment.
- 3.1. Adversarial Risk and Worst-Case Guarantees: Expected risk can remain low even when a model performs arbitrarily poorly on individual inputs, motivating worst-case risk for high-stakes deployment.The paper frames catastrophic single-example failures as a reason to supplement average performance with worst-case guarantees.
- 3.1. Adversarial Risk and Worst-Case Guarantees: The true worst-case risk is computationally difficult because evaluating its supremum may require exponentially expensive searches over the input space and knowledge of the data-support shape.The support itself depends on the input-label mapping that the model is intended to learn.
- 3.1. Adversarial Risk and Worst-Case Guarantees: Local adversarial risk replaces a global search with optimization over nearby points N_ϵ(x), enabling approximate worst-case examples through methods such as projected gradient descent.Constraining ϵ so labels remain unchanged within the neighborhood supports the adversarial-example interpretation.
- 3.1. Adversarial Risk and Worst-Case Guarantees: Local adversarial risk is a fixed-point certificate and a lower bound on worst-case risk, so it simplifies analysis without providing a whole-input-space guarantee.This makes it a useful necessary subproblem while preserving a gap from global worst-case certification.
- 3.2. Obscurity with Respect to an Adversary: A chosen adversary produces the surrogate adversarial risk reported by evaluation metrics, but good surrogate performance need not imply high true adversarial robustness.Unlike test loss for expected risk, the surrogate adversarial risk is biased downward from the true adversarial risk, and the gap can be large.
- 3.2. Obscurity with Respect to an Adversary: Obscurity is the gap between true and surrogate adversarial risk: models may contain adversarial examples that the chosen adversary fails to identify efficiently.The paper contrasts genuinely robust decision boundaries with security achieved through obscurity, including gradient masking against gradient-based attacks.
- 3.2. Obscurity with Respect to an Adversary: Transparent models are sought because they support estimating true adversarial risk, distinguishing strong defenses from weak attacks, and assessing catastrophic failure risk before deployment.The paper presents transparency as a practical requirement for scientific evaluation and pre-deployment safety assessment.
4. Attack Strategies
The paper compares gradient-based, gradient-free, and transfer-based attacks for finding adversarial examples, emphasizing that stronger optimization can expose vulnerabilities hidden by common evaluations.
- Optimization-based attacks: The attacks frame adversarial perturbation search as constrained optimization within an ℓ∞ neighborhood around the input.The objective uses a margin-based loss that becomes negative when an image is misclassified.
- Gradient-based optimization: PGD iteratively updates the input using gradients, projection onto the allowed neighborhood, Adam, and random initialization to mitigate poor local minima from gradient masking.The projection enforces the perturbation constraint while random starts address possible masking-related local minima.
- Gradient-free optimization: SPSA estimates gradients from finite differences in random directions when analytic gradients are unavailable or unhelpful.The method is designed for high-dimensional optimization with noisy objectives and averages estimates for efficient convergence.
- Gradient-free optimization: The paper finds SPSA reliably produces adversarial examples, while the particular gradient-free optimizer appears relatively unimportant because any adversary lower-bounds true adversarial risk.The authors also compare SPSA with natural evolutionary strategies and zero-order optimization in the appendix.
- Transfer-based attacks: Transfer attacks can exploit unknown models, but their success depends strongly on similarity between the surrogate and target models.For transformed defenses, transferred examples may fail to reveal vulnerabilities when the defended model's weaknesses differ from the original model's.
- Evaluation comparison: Most proposed defenses show robustness to their original evaluation adversaries yet remain highly vulnerable to adversarial examples under stronger attacks.Table 1 standardizes evaluation conditions across defenses, so its reproduced values can differ slightly from the original papers.
5. Reasoning about Obscurity
The paper treats obscurity as the gap between a model's true worst-case vulnerability and what a chosen evaluation adversary can detect. It argues that stronger adversaries are needed to validate robustness, while heuristics can help identify obscured models.
- Definition and motivation: Obscurity asks whether an evaluation would detect adversarial inputs if a model admits them, rather than only whether such inputs exist.This reframes analysis around the detectability of worst-case failures.
- Surrogate objectives: Transfer-based evaluations may fail to approach the true objective because adversarial examples for a surrogate need not fool the original model without guaranteed similarity.Transfer can still work when the surrogate is optimized to match the original model.
- Adversary behavior: Gradient-based adversaries face difficulties from vanishing gradients and discrete or highly nonlinear operations, even though gradient search can otherwise be effective.These properties can make the true objective harder for gradient-based attacks to optimize.
- Empirical validation: In defenses hypothesized to have high obscurity, the evaluation metric misses true adversarial risk: true adversarial accuracy is near zero despite high surrogate adversarial accuracy.The paper validates heuristic predictions experimentally, but conclusively demonstrating high obscurity requires a stronger adversary.
6. Experiments
The experiments test whether defenses withstand stronger, end-to-end attacks rather than only commonly used transfer-based or gradient-based evaluations. Across several defenses, gradient-free or direct attacks substantially reduce apparent robustness, while adversarial training is comparatively less obscured.
- 6. Experiments: Gradient-free optimization can expose defenses whose apparent security comes from non-differentiability, preprocessing differences, or computational expense rather than true adversarial robustness.The experiments use stronger attacks to test whether evaluation procedures capture near worst-case behavior.
- 6.1. Non-differentiability: JPEG compression reduces ImageNet adversarial accuracy to 0% at ϵ = 2 when attacked end-to-end with SPSA.The original transfer-based evaluation reported 15% accuracy under the same perturbation budget.
- 6.2.1. DETECTING ADVERSARIAL EXAMPLES: At a 1% false positive rate, likelihood thresholding detects 99.7% of ordinary PGD adversarial examples but 0% of examples optimized against the PixelCNN detector.The modified attack adds the PixelCNN negative log-likelihood to the adversarial objective, producing higher-likelihood adversarial examples.
- 6.2.2. PURIFICATION VIA LIKELIHOOD OPTIMIZATION: SPSA reduces PixelDefend accuracy to 10% on CIFAR-10 at ϵ = 8, despite the original transfer-based evaluation reporting 75% accuracy.The authors attribute the discrepancy to obscurity from the lengthy optimization process and vanishing gradients in end-to-end attacks.
- 6.2.3. PURIFICATION WITH AUTOENCODERS: PGD reduces the autoencoder defense’s ImageNet accuracy to 0% at ϵ = 2 by attacking the differentiable end-to-end model.The transfer-based attack caused almost no performance drop, illustrating that transfer robustness need not imply general robustness.
- 6.4. Adversarial training: Against adversarially trained models, SPSA achieves 51% accuracy at ϵ = 8, and succeeds when PGD does not on only 0.2% of images.This result is described as far from conclusive but consistent with PGD being near-optimal for differentiable image classifiers.
7. Conclusion
The paper treats adversarial risk as a tractable proxy for worst-case model behavior and common evaluations as surrogate objectives that may miss existing adversarial examples. Its experiments support using obscurity as a rough design sieve and evaluating defenses against stronger adversaries.
- 7. Conclusion: Common evaluation procedures define tractable surrogates to true adversarial risk, whose relevance depends on whether the adversary can find existing adversarial examples.The exact adversarial risk remains difficult to compute because of its inner supremum.
- 7. Conclusion: The obscurity framework provides a rough sieve for designing defenses with lower obscurity to efficient adversaries.The authors present this as a heuristic rather than a conclusive demonstration of robustness.
- 7. Conclusion: In defenses suspected to be highly obscured, stronger attacks produced dramatically lower adversarial accuracy than the original evaluation metrics suggested.The conclusion links this pattern to the need for rigorous evaluation against strong adversaries.
A. Comparison of Gradient-free Attacks
The appendix compares gradient-free attacks by computation and evaluation budget. With sufficient computation, these attacks can drive ImageNet accuracy to zero, while larger batches trade slower starts for more reliable eventual discovery.
- A. Comparison of Gradient-free Attacks: The ImageNet experiments use ResNet-50 with 100 random test images, while CIFAR-10 uses a VGG-like model with 1000 random test images.The reported clean accuracies are 75% for ResNet-50 and 94.5% for the CIFAR-10 model.
- A. Comparison of Gradient-free Attacks: All attacks drive CIFAR-10 accuracy to 0% within 32768 model evaluations, and SPSA falls below 5% after 2048 evaluations.ImageNet attacks generally require more evaluations, likely because its input images are higher-dimensional.
- A. Comparison of Gradient-free Attacks: With sufficient computation, both NES and SPSA decrease ResNet-50 accuracy to 0% on ImageNet at ϵ = 2.Table 2 varies finite-difference batch sizes across a maximum of 300 iterations.
- A. Comparison of Gradient-free Attacks: The main experiments use a fixed large batch size because the objective is reliable attack generation rather than query efficiency.The same algorithms can be tuned with smaller batches when query efficiency matters.
- A. Comparison of Gradient-free Attacks: Larger batch sizes delay initial attack success but make adversarial examples more reliable over time.Smaller batches use more iterations to keep total model evaluations comparable, and Figure 2 uses a logarithmic evaluation axis.
B. Hyperparameters
The appendix specifies the attack-evaluation budget and identifies SPSA hyperparameters as the subject of a dedicated table.
- B. Hyperparameters: Attacks are evaluated on 1000 randomly sampled test images, except PixelDefend, which is evaluated on 100 images because of computational limits.Attacks run for at most 100 iterations and stop when the margin objective is below −5.0.
- B. Hyperparameters: Table 3 lists the hyperparameters used for the SPSA attack.The supplied passage identifies the table but does not enumerate its parameter values.
C. Discussion of Adversarial Training
Gradient-free SPSA attacks generally find perturbations comparable to those found by PGD in adversarially trained networks. This suggests PGD perturbations may be nearly worst-case, making PGD-based surrogate risk close to true adversarial risk.
- C. Discussion of Adversarial Training: SPSA may converge to better minima than analytic-gradient methods when finite perturbations reveal qualitatively different loss-surface behavior.This can occur, for example, with high-frequency oscillations in the loss surface.
- C. Discussion of Adversarial Training: SPSA and PGD found similarly adversarial perturbations for almost all images.The result provides evidence that PGD perturbations in adversarially trained networks may be nearly worst-case possible.
- C. Discussion of Adversarial Training: Points near y = x indicate similarly adversarial perturbations, while red points below the line indicate stronger perturbations found by SPSA.Overall, the methods identify comparably adversarial perturbations, with few cases where SPSA is significantly stronger.
D. Additional Experiments
Additional experiments show that defenses appearing robust to their original evaluation adversaries are not robust to stronger attacks. Across the evaluated defenses, accuracy falls to near zero under those stronger attacks.
- D. Additional Experiments: All evaluated non-adversarial-training defenses show significant robustness against the original evaluation adversaries.Figure 4 evaluates each defense across perturbation sizes and multiple attack methods.
- D. Additional Experiments: Accuracy of all evaluated defenses falls to near zero when tested against stronger attacks.
- D. Additional Experiments: Adding effectively non-differentiable operations or purification can obscure models to gradient-based and transfer-based attacks without removing all adversarial examples.Stronger attacks still reduce the accuracy of all defenses to near zero.