Source-linked AI summary
Motivating the Rules of the Game for Adversarial Example Research
Justin Gilmer, Ryan P. Adams, Ian Goodfellow, David Andersen, George E. Dahl
TL;DR
Security-motivated adversarial-example research often evaluates abstract restricted-perturbation games whose connection to real threats is unclear. The paper develops a taxonomy grounded in attacker goals, knowledge, action spaces, and game sequence, then recommends clearer threat modeling and more meaningful evaluation. Its central conclusion is that security claims require studying realistic rules rather than relying on standard perturbation games alone.
Problem
Security research has not clearly established how restricted perturbation errors differ from other attacker-produced errors or described realistic attacker abilities and limitations.
Method
The paper develops a taxonomy of attacker-defender game rules and examines perturbation defenses against concrete security scenarios.
Results
The paper finds that standard perturbation-defense rules have weak security motivation and do not capture real-life security concerns.
Takeaways & Limitations
Security claims should articulate the threat model and study the most realistic attacker rules that can be evaluated.
Takeaways & Limitations
The attack scenarios are sketches and do not provide a complete understanding of the specific threats or systems discussed.
Abstract
from arXiv · showhide
Advances in machine learning have led to broad deployment of systems with impressive performance on important problems. Nonetheless, these systems can be induced to make errors on data that are surprisingly similar to examples the learned system handles correctly. The existence of these errors raises a variety of questions about out-of-sample generalization and whether bad actors might use such examples to abuse deployed systems. As a result of these security concerns, there has been a flurry of recent papers proposing algorithms to defend against such malicious perturbations of correctly handled examples. It is unclear how such misclassifications represent a different kind of security problem than other errors, or even other attacker-produced examples that have no specific relationship to an uncorrupted input. In this paper, we argue that adversarial example defense papers have, to date, mostly considered abstract, toy games that do not relate to any specific security concern. Furthermore, defense papers have not yet precisely described all the abilities and limitations of attackers that would be relevant in practical security. Towards this end, we establish a taxonomy of motivations, constraints, and abilities for more plausible adversaries. Finally, we provide a series of recommendations outlining a path forward for future work to more clearly articulate the threat model and perform more meaningful evaluation.
1 Introduction
The paper argues that security-motivated adversarial-example research often studies restricted perturbation games without realistic threat models. It proposes clearer motivations, definitions, and security-relevant rules.
- 1 Introduction: Security-focused work often treats robustness to small or restricted perturbations as improved security.Many papers emphasize perturbation perceptibility, including small or quasi-imperceptible modifications.
- 1 Introduction: The paper defines perturbation-defense literature as work studying worst-case restricted perturbations of correctly handled inputs.Its scope excludes minimax learning work and defenses motivated by generalization or biological mimicry rather than security.
- 1 Introduction: The authors argue that the standard perturbation-defense rules do not capture real-life security concerns.They therefore examine their relevance and realism against security scenarios.
- 1 Introduction: The paper introduces a taxonomy of attacker-defender game rules motivated by real-world security scenarios.The taxonomy is intended to clarify the motivations, constraints, and abilities relevant to plausible adversaries.
- 1 Introduction: Researchers should state their specific motivations explicitly and use metrics appropriate to security or non-security goals.The paper focuses mainly on security motivations while directing readers to Section 5 for non-security topics.
2 Possible Rules of the Game
The paper frames adversarial examples as outcomes of attacker-defender games whose rules must specify goals, knowledge, action spaces, constraints, and sequence. It distinguishes several attacker action spaces beyond small perturbations.
- 2 Possible Rules of the Game: Adversarial-example games should specify attacker goals, with targeted attacks requiring a specific error and untargeted attacks accepting any error.For example, labeling a cat as a dog is targeted, whereas labeling it as anything other than a cat is untargeted.
- 2 Possible Rules of the Game: Attacker knowledge ranges from full whitebox access to blackbox interaction with unknown system details.The paper treats knowledge of model internals, training data, and query access as distinct dimensions.
- 2 Possible Rules of the Game: The action space must be explicit and realistic because perturbation defenses often assume stealthy, norm-constrained modifications.Norm-based perception proxies frequently restrict attackers to small changes around a starting point.
- 2 Possible Rules of the Game: The taxonomy includes indistinguishable, content-preserving, non-suspicious, content-constrained, and unconstrained inputs.These categories progressively relax requirements on the starting point, human detectability, semantic content, or payload.
- 2 Possible Rules of the Game: Game sequence matters because an attacker who acts first may define an input distribution that the defender can sample during retraining.Repeated or real-time play also introduces computational considerations.
- 2 Possible Rules of the Game: Adversarial robustness reflects the broader risk that optimizing a metric can make it cease to be useful.Machine-learned functions may have counterintuitive failure modes not revealed by held-out error alone.
3 Example Attack Scenarios
The paper surveys security scenarios where attackers use broader, often physically realizable or payload-constrained inputs rather than small perturbations. These examples motivate expanding adversarial-example rulesets beyond restricted perturbation games.
- 3 Example Attack Scenarios: The proposed attack scenarios are sketches rather than complete threat analyses, but the authors regard such analysis as necessary for security motivation.The examples are intended to illuminate constraints, objectives, and previously studied games without exhausting realistic security situations.
- 3.1 Attacks with Content Preservation Constraints: Pay-per-view and revenge-porn attacks can involve large, obvious transformations while preserving or targeting specific content.Examples include cropping and re-filming, while revenge-porn attacks remain untargeted and content-preserving.
- 3.2 Non-Suspicious Attacks: Voice-assistant attacks illustrate non-suspicious inputs that appear innocuous to humans while triggering commands or unauthorized actions.Prior work also demonstrates unintelligible hidden commands and audio that is 99% similar to a clean source.
- 3.2 Non-Suspicious Attacks: Facial-biometric evasion may require physically realizable disguises rather than perturbations of a fixed starting image.Attackers can choose among faces or disguises as long as the result remains a real-looking person with a mostly uncovered face.
- 3.3 Attacks with Content Constraints: Spam, malware, and content-troll attacks impose payload or semantic constraints while allowing attackers substantial control over the input.These settings involve evading detection while preserving advertising, malicious functionality, or objectionable content.
- 3 Example Attack Scenarios: Real-world attacks often use weaker assumptions about attackers than the perturbation-defense literature typically does.The paper emphasizes that attackers may produce arbitrary inputs, ignore imperceptibility, or operate with limited model knowledge and query access.
- 3.5 Indistinguishable Perturbations: The authors found no compelling example requiring indistinguishability and argue that security research should study broader attacker rules.Attackers may still succeed with less-constrained perturbations or by choosing a different starting point.
4 Standard Rules in the Perturbation Defense Literature
The surveyed perturbation-defense literature converges on a standard game: perturb a randomly drawn correctly handled input within an l_p norm budget to induce error, usually via approximate optimization. The paper argues that these rules and their evaluation metrics provide imperfect proxies for realistic security threats and can produce misleading robustness claims.
- Standard attacker and defender rules: Across more than fifty papers, attackers typically perturb a data-distribution sample within an l_p norm budget ε to induce targeted or untargeted errors.Many papers leave the game rules and adversarial-example definitions implicit, but their protocols reveal this common structure.
- Evaluation metric: The standard evaluation metric, adversarial robustness, estimates expected accuracy under an approximate worst-case perturbation rather than providing a worst-case guarantee.It measures an expectation over the data distribution, and the perturbation is found using an approximate attack procedure.
- Evaluation metric: Approximate optimization can make reported robustness scores loose bounds, contributing to repeated evaluations that overturned 18 prior defense claims.The cited reassessments found that the 18 proposals were not as robust as originally reported.
- Mismatch with perception: The standard l_p action space fits indistinguishability poorly because equal-distance images can differ substantially in perceived distance.Increasing ε includes both obviously different images and images that are difficult to distinguish; a one-pixel translation can also have a large l_p distance.
- Mismatch with perception: Psychometric factors, including observation time and motivation to detect changes, further complicate connecting norm-bounded perturbations to realistic adversarial actions.These factors make indistinguishability depend on the observer and setting rather than distance alone.
4.1 Common Motivating Scenarios for the Standard Rules
The paper examines common security scenarios used to motivate standard perturbation rules and finds that realistic attacks often permit broader actions, exploit naturally occurring errors, or have easier alternatives. These examples challenge whether small perturbations of randomly sampled inputs capture practical threats.
- Motivation and realism: Security-motivated research should specify its motivation, threat model, and metrics, while evaluating whether studied rules capture realistic threats.The paper reports no applied work using standard rules that both secures a real system and performs threat modeling.
- The stop-sign attack: Stop-sign attacks need not be imperceptible: stickers, scratches, defacement, or changes to the surrounding area may be available to an attacker.The physical-world examples discussed in the literature are far from subtle and do not clearly follow the standard rules.
- The stop-sign attack: Self-driving systems must handle naturally occurring cases such as fallen or missing stop signs, so robustness to tiny perturbations is not obviously preferable to real-world accuracy.Even perfect test accuracy on ordinary stop signs would not eliminate the need to handle missing signs safely.
- The stop-sign attack: Object detectors can be brittle to ordinary transformations: randomly moving an object in the background may cause other objects to disappear or become misclassified.The paper recommends considering robustness to such naturally occurring transformations.
- Alternative attacks: Several cited scenarios have easier non-ML alternatives, including physically knocking over a stop sign or committing traditional check fraud.The paper uses these cases to question whether inducing a classifier error represents the attacker’s preferred route to harm.
- Evading malware detection: Malware authors are unlikely to restrict themselves to small l_0 changes in the defender’s feature space because functional code admits a much broader action space.They may use polymorphic viruses, encrypted payloads, and randomized decryption routines; feature-level defenses can still matter when altering features is costly.
4.2 The Test Set Attack
The test set attack reframes adversarial evaluation around naturally occurring model errors and deployment conditions, showing that limited-access attackers can exploit them without perturbing inputs.
- Test Set Attack: A test set attack uses randomly sampled inputs and succeeds when the model misclassifies them, requiring no perturbation.The attack exposes imperfect out-of-sample generalization rather than exploiting a specially modified example.
- Deployment Conditions: Defense choices should approximate deployment conditions by weighting performance with and without the expected adversary.The appropriate tradeoff depends on the assumed input distribution and attacker presence, rather than IID or adversarial performance alone.
- Test Set Attack: Small but non-zero test error can imply sensitivity to small perturbations for randomly sampled inputs from the data distribution.A result on a specific high-dimensional dataset links test error to l2 robustness and shows reliable errors even below 10^-8 test error.
- Real-World Example: Limited-query attackers without machine-learning expertise can fool the Not Hotdog application using benign physical-world images in a black-box setting.The examples were collected from tweets containing #NotHotDog and did not require imperceptible modifications or knowledge of the model.
4.3 Security and the Standard Rules
The paper argues that standard perturbation-defense rules generally lack realistic security motivation: their motivating examples are often unrealistic, while realistic threats usually require different rules.
- Findings: Common motivating examples for standard perturbation-defense rules typically describe unrealistic threat models.The paper distinguishes these abstract rules from security analyses grounded in actual systems and attacker behavior.
- Findings: When motivating examples involve more realistic threats, the standard ruleset usually does not apply.Realistic attacks may involve payload constraints, physical-world inputs, or other action spaces rather than restricted perturbations.
- Findings: The authors found no real-world examples requiring indistinguishable perturbations, leaving the standard perturbation-defense rules without a strong security motivation.This conclusion holds even under the hypothetical assumption that small l_p constraints perfectly approximate human perception.
- Findings: Randomly chosen starting data enable attacks without perturbation by exploiting imperfect out-of-sample generalization.The test set attack shows that an attacker can seek naturally occurring errors instead of modifying correctly classified inputs.
- Implications: Without certainty about attacker constraints and capabilities, increased robustness has no clear relationship to security.A model can improve under one attack model while becoming worse under another, so evaluation must match deployment assumptions.
5 Moving Forward
The paper recommends grounding security research in realistic attack models, broader evaluations, and proxy metrics rather than relying on restricted perturbation games. It also highlights content-preserving attacks and the difficulty of evaluating robustness under realistic constraints.
- Evaluation breadth: The l_p metric began as an easy toy problem, but solutions have not generalized reliably to other threat models.The paper argues that quantifiable rules facilitate follow-up research while requiring clearer boundaries for where those rules apply.
- Realistic threat models: Security-motivated papers should define attacker action spaces using real systems and attacks, while considering prior machine-learning security work.The authors recommend motivating adversarial-example definitions and threat models with real systems, ideally securing an actual system.
- Content-preserving attacks: Content-preserving attacks can ignore imperceptibility by modifying background pixels, exposing a gap between semantic preservation and norm-bounded robustness.On MNIST, this attack achieved 0% robustness against the state-of-the-art defense, while its l_p distances were large.
- Evaluation breadth: Random content-preserving transformations can succeed given enough attempts, so evaluations should include high-likelihood simplistic attacks alongside white-box adversaries.The paper cautions that robustness to sophisticated attackers should not replace defenses against simple attacks such as random background transformations.
- Proxy metrics and system defenses: Security models need broader proxy metrics because content-preserving and non-suspicious constraints are difficult to formalize and domain-specific.The authors recommend concrete, domain-specific threats first and note that system-level mechanisms can complement classifier defenses.
- Evaluation reliability: Restricted white-box evaluations have repeatedly produced false confidence because optimization failures can masquerade as fundamental robustness.The paper reports that over 18 defenses were later shown ineffective, and that empirical l_p-robustness evaluation is generally NP-hard.
6 Conclusions and Future Work
The paper concludes that adversarial-example research should clearly state its motivation, definitions, and game rules, especially when claiming security benefits. It also proposes reframing restricted-perturbation work as machine-learning research when its motivations are not security-specific.
- Conclusions: Security claims require explicit threat modeling and evaluation under the most realistic rules the researchers can study.The authors stress that abstraction from real systems makes threat modeling harder, not less necessary.
- Future work: Restricted perturbation research may be better framed as machine-learning research when its contribution is not tied to a concrete security threat.The paper argues that clearer non-security motivations could explain the phenomenon’s appeal beyond the security community.