PaperlayerView exact revision on arXiv

Source-linked AI summary

"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice

Giovanni Apruzzese, Hyrum S. Anderson, Savino Dambra, David Freeman, Fabio Pierazzi, Kevin A. Roundy

arXiv:2212.14315v1cs.CRcs.LG

TL;DR

The paper addresses a mismatch between practitioner priorities and research focus by combining real-world case studies with a systematic review of recent top-security-conference papers. The review found evaluations concentrated on neural networks, computer vision, and benchmarks, alongside a growing focus on attacks against other system elements.

  • Problem

    The paper addresses a mismatch between practitioner priorities and research focus: practitioners have not prioritized ML security while researchers advocate worst-case security evaluations.

    Source: §I · p. 1

  • Method

    The authors combine real-world case studies with a systematic analysis of 88 recent adversarial-ML papers from top security conferences.

    Source: §I · p. 1

  • Results

    The review found that 89% of papers evaluated neural networks, 63% focused on computer vision, and 80% used benchmarks, alongside an increasing focus on attacks targeting other system elements.

    Source: §I · p. 1

  • Takeaways & Limitations

    The authors urge precise whole-system threat models, cost assessments that include human effort, industry collaboration, reproducible research, and forensics for ML incidents.

    Source: §VI · p. 12

    The authors cannot determine with certainty whether observed real-world evasions are gradient-generated adversarial examples or educated guessing.

    Source: B. Are there Adversarial Examples in… · p. 6

Abstract

from arXiv · show

Recent years have seen a proliferation of research on adversarial machine learning. Numerous papers demonstrate powerful algorithmic attacks against a wide variety of machine learning (ML) models, and numerous other papers propose defenses that can withstand most attacks. However, abundant real-world evidence suggests that actual attackers use simple tactics to subvert ML-driven systems, and as a result security practitioners have not prioritized adversarial ML defenses. Motivated by the apparent gap between researchers and practitioners, this position paper aims to bridge the two domains. We first present three real-world case studies from which we can glean practical insights unknown or neglected in research. Next we analyze all adversarial ML papers recently published in top security conferences, highlighting positive trends and blind spots. Finally, we state positions on precise and cost-driven threat modeling, collaboration between industry and academia, and reproducible research. We believe that our positions, if adopted, will increase the real-world impact of future endeavours in adversarial ML, bringing both researchers and practitioners closer to their shared goal of improving the security of ML systems.

I. INTRODUCTION

The paper argues that adversarial-ML research and practice prioritize different concerns, and uses real-world observations and a research review to propose ways to bridge that gap. It frames deployed models as components of larger systems, rather than isolated targets.

I. INTRODUCTION

  • Practitioners have not prioritized ML security despite reported vulnerabilities, while research has emphasized worst-case evaluations and powerful model attacks.The authors identify this mismatch as a reason to guide future work using practical observations.Source: §I · p. 1

II. REVISITING MACHINE LEARNING SECURITY

  • The paper frames ML security through machine-learning systems, security of machine learning, and practical cybersecurity, with economics relevant to both attackers and defenders.It argues that deployed systems have complex architectures that may be unknown to researchers.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • Three case studies examine an online social network’s ML-system architecture, phishing detection, and the role of time and expertise in a competition attack.The cases are intended to surface practical insights overlooked in research.Source: §I · pp. 1–2
  • The review of 88 papers found that 89% evaluated neural networks, 63% focused on computer vision, and 80% used benchmarks; it also found a growing focus on attacks targeting other system elements.The papers were from top security conferences and published between 2019 and 2021.Source: §I · p. 1
  • The authors recommend system-wide threat models, cost-driven assessments, industry–academia collaboration, and a “just culture” for source-code disclosure.They present these four positions as ways to bridge research and practice.Source: §I · pp. 1–2

A. Overview of Machine Learning (Systems)

  • An operational ML model is one component in a larger pipeline that may preprocess inputs, process model outputs, or route some inputs around ML.Some systems combine multiple components or use alternate pipelines for inputs matching known signatures.Source: A. Overview of Machine Learning… · p. 2
  • ML systems may be open or closed; closed systems conceal their source code and components from end users and may restrict use.In invisible systems, low-privilege users may also receive delayed or indirect feedback about system outputs.Source: A. Overview of Machine Learning… · p. 2

B. Security of Machine Learning

The section situates adversarial examples within a broader set of ML-system threats and describes threat models through an attacker’s goal, knowledge, capabilities, and strategy. It also notes limitations in model-focused terminology and reports practitioners’ limited understanding and low prioritization of adversarial ML.

B. Security of Machine Learning

  • Adversarial examples are only one of many threats that can compromise ML systems.Examples also include model theft and extracting private training data through membership inference.Source: B. Security of Machine Learning · p. 3
  • ML threat models characterize an attacker through four elements: goal, knowledge, capabilities, and strategy.Goals can include violating integrity or availability, stealing a model, or inferring whether someone’s data was used in training.Source: B. Security of Machine Learning · p. 3
  • “White-box,” “black-box,” and “gray-box” terminology describes different levels of attacker knowledge, but the terminology discussed here focuses on the model rather than the full ML system.The authors identify this model-only focus as a limitation.Source: B. Security of Machine Learning · p. 3
  • Practitioner accounts report limited understanding and low prioritization of adversarial ML, with some respondents unsure how to respond and others questioning the need for countermeasures.One developer study found inadequate understanding of securing production ML systems, even though one third felt insecure about adversarial ML.Source: B. Security of Machine Learning · p. 3

C. (Economics of) Cybersecurity

Operational cybersecurity is shaped by costs for attackers and defenders, while real ML defenses operate as complex systems rather than isolated classifiers. The paper connects these practical considerations to prior work and real-world case studies.

C. (Economics of) Cybersecurity

  • Practical cybersecurity weighs the attacker’s costs against the costs of implementing, evaluating, maintaining, and running defenses.The paper characterizes economics as the main driver of practical cybersecurity for both attackers and defenders.Source: C. (Economics of) Cybersecurity · p. 4

D. Related Work

  • Unlike work focused only on industry or research, this paper considers both perspectives and grounds its arguments in recent research analysis and real-experience case studies.The authors describe their position paper as the first in ML security to base its main arguments on both kinds of observations.Source: D. Related Work · p. 4

III. CASE STUDIES (REAL EXPERIENCE)

  • Three real-world case studies examine an online social network’s ML architecture, phishing-detector triage, and time and expertise in an evasion competition.An appendix adds a fourth case study on malware detectors and expert knowledge.Source: §III · p. 4

A. “The four A’s of abuse fighting”: ML Systems in an OSN

  • Facebook’s spam-defense funnel has four interconnected layers, and attackers must bypass them all; layers can exchange signals and shape responses.For example, a suspicious fake-account score can lower an account’s activity rate limit without immediately blocking it.Source: Figure 3 · p. 5, A. “The four A’s of abuse fighting”:… · p. 5
  • The first three layers address automation, access legitimacy, and user activity, while the application layer is where content classifiers such as nudity detectors can operate.The first three layers are generic defenses; application-level violations are expected to be mostly small-scale if earlier layers are effective.Source: A. “The four A’s of abuse fighting”:… · p. 5

B. Are there Adversarial Examples in the Wild (Web)?

The case studies describe simple phishing-evasion tactics and an industry challenge where query counts alone obscured the time and expertise involved. The evidence supports practical observations but does not establish that observed samples were gradient-generated.

B. Are there Adversarial Examples in the Wild (Web)?

  • The analysis selected 100 samples judged potentially adversarial from 4,600 inspected among 9,174 uncertain inputs, drawn from 40 commonly targeted domains.Two authors inspected samples independently and met regularly to resolve disagreements.Source: B. Are there Adversarial Examples in… · p. 6
  • Phishing evasion commonly uses cropping, masking, stretching, or blurring to disrupt OCR, alongside logo changes or removals and less common form or background manipulations.The paper reports that these tactics can fool ML systems without gradient computations; Fig. 4 illustrates masking, cropping, blurring, and misspellings.Source: B. Are there Adversarial Examples in… · p. 6, Table II · p. 6, Figure 4 · p. 6
  • The authors could not determine with certainty whether evasive samples came from gradient-based attacks or educated guessing, though they judged the observed strategies simple and effective.They note that digital forensics could potentially help resolve attribution.Source: B. Are there Adversarial Examples in… · p. 6

C. In-Depth Analysis of an ML Evasion Competition

  • In the 42-day challenge, four teams evaded all ten webpages against all seven detectors; the winner used 320 queries, versus 608 for third place and 9,982 for fourth.The winner’s query count was lowest, but the paper cautions that this alone gives a misleading view of attack cost.Source: C. In-Depth Analysis of an ML Evasion… · p. 7
  • Third place reached perfect evasion by day 12, while the winning team submitted its final evasion on day 42 after adopting a wait-and-see approach.The winner’s strategy involved sophisticated HTML manipulations rooted in domain expertise; the third-place team was one person, while the winner had two.Source: C. In-Depth Analysis of an ML Evasion… · p. 7, Figure 5 · p. 7
  • Query counts alone do not capture the time and domain expertise needed for a successful attack; fourth place made many automated queries with likely little human effort.The paper identifies time and expertise as material to interpreting attack efficiency.Source: C. In-Depth Analysis of an ML Evasion… · p. 7

IV. SNAPSHOT OF ADVERSARIAL ML RESEARCH

The review analyzes recent adversarial ML research to identify trends and blind spots, finding both more realistic attack threat models and recurring gaps in evaluation scope. It also highlights inconsistent threat-model assumptions and cases where attacks target systems rather than causing model misclassification.

A. Overview

  • The cited findings report that 52% envision “evasion” attacks, 89% consider only deep learning algorithms, and 63% evaluate on image data.Only 5% consider malware, phishing, or intrusion detection.Source: A. Overview · p. 8
  • 27% of papers do not mention economics, and 51% publicly release source code.Source: A. Overview · p. 8
  • 20% of papers experiment on real ML systems, while 20% reproduce a pipeline by building only an ML model.The authors caution that individually valid study choices can produce blind spots across the research body.Source: A. Overview · p. 8

B. Positive findings in Threat Models of Attack Papers

  • Some attacks succeed without fooling the ML model: they exploit preprocessing or system policies, and some ASR attacks use audio-domain expertise rather than established gradient-based techniques.The review also highlights real ASR attacks considered without target knowledge or query access.Source: B. Positive findings in Threat Models… · pp. 8–9
  • Attack papers increasingly model constrained adversaries and use domain expertise, though extremely constrained attackers using sophisticated strategies may be unrealistic when simpler strategies achieve the same goal.The review also notes attacks can pursue goals beyond causing model misclassification.Source: B. Positive findings in Threat Models… · pp. 8–9

C. Positive Findings in Threat Models of Defense Papers

Defense papers more often account for countermeasure overhead than attack papers, while most assume attackers have perfect knowledge. The review also finds that threat-model terminology is inconsistent, especially for attacker knowledge, capabilities, and what a “box” represents.

C. Positive Findings in Threat Models of Defense Papers

  • Limited-knowledge defenses merit attention because attackers without perfect knowledge are likely in the wild, and some studies report a smaller-than-expected gap from white-box attack accuracy.The review also notes that complete knowledge can be costly to acquire, while open systems may expose information at multiple layers.Source: C. Positive Findings in Threat Models… · p. 9
  • Most defense papers assume attackers have perfect knowledge, though some defenses target limited-knowledge attackers.Only 24 of 88 reviewed papers consider defenses; compared with attack papers, defense papers more often measure countermeasure overhead as baseline performance degradation.Source: C. Positive Findings in Threat Models… · p. 9

D. Inconsistencies that May Harm Future Research

  • “White-box” and “black-box” can describe substantially different degrees of attacker knowledge across papers.Examples differ over whether the attacker knows training data or has surrogate models and datasets.Source: D. Inconsistencies that May Harm… · p. 9
  • Papers also use “access” ambiguously and often fail to specify what query outputs attackers receive.The review cites conflicting descriptions of direct or API access and output described only as “a probability.”Source: D. Inconsistencies that May Harm… · p. 10
  • The review finds inconsistent definitions of attacker knowledge and capabilities, particularly in “box”-based terminology, and argues that the box should refer to the entire ML system.A model-only view can omit system components such as preprocessing.Source: D. Inconsistencies that May Harm… · pp. 9–10

V. RECOMMENDATIONS

The recommendations call for threat models that precisely cover whole ML systems and cost assessments that account for both attackers and defenders. The authors argue these practices can help make security assessments more realistic and comparable.

A. Adapting Threat Models to ML Systems

  • Threat models should define the attacker’s viewpoint on every component of an ML system, extending goals, knowledge, capabilities, and strategies beyond the model.The authors note that attackers may target all tenants, lack knowledge of preprocessing, or manipulate hardware or source code rather than the model.Source: A. Adapting Threat Models to ML Systems · p. 10
  • Researchers should avoid ambiguous “box” labels, specify which system components attacker knowledge covers, and state access precisely, such as read or write access.The paper identifies BOX, ACCESS, EVASION, and ADVERSARIAL as terms vulnerable to misinterpretation.Source: A. Adapting Threat Models to ML Systems · pp. 10–11
  • “Evasion” should denote the attacker’s goal, which may be achieved through poisoning, adversarial examples, or strategies that ignore the ML model.The authors distinguish this goal-oriented use from the research usage focused on test-time misclassification.Source: A. Adapting Threat Models to ML Systems · p. 11

B. Cost-driven Security Assessments

  • Threat models should include cost-driven assessments for both attackers and defenders.The authors report that 27% of reviewed papers do not mention cost and that many cost measurements overlook human effort such as time and expertise.Source: B. Cost-driven Security Assessments · p. 11
  • The proposed assessment weighs attacker benefit against attack cost to estimate likelihood, simulates damage to assess risk, then compares countermeasure costs with that risk.The authors also caution that attacker and defender economics are not necessarily a zero-sum game, and attacks defusable by simple heuristics may not be launched.Source: B. Cost-driven Security Assessments · pp. 11–12
  • A precise, holistic threat model can later serve as a reference for estimating and comparing an attack’s cumulative cost.Source: B. Cost-driven Security Assessments · p. 12

C. Collaborations between Industry and Academia

The authors call for industry-academia collaboration because researchers often cannot access or understand production ML systems. They also advocate a just culture that supports code disclosure, so flaws can be identified and used as learning opportunities.

C. Collaborations between Industry and Academia

  • Only 20% of reviewed papers consider real ML systems, and the authors note that deep-learning-only research may not reflect systems used in practice.Production systems are often invisible or closed, limiting researchers’ ability to understand their architecture or explain observed behavior.Source: C. Collaborations between Industry and… · p. 12
  • Companies could support independent assessments through bug-bounty programs, researcher-request resources, or high-level system schematics.These options do not require publicly releasing the companies’ ML systems and could help researchers study more realistic pipelines.Source: C. Collaborations between Industry and… · p. 12

D. Just Culture and Reproducible Research

  • A just culture treats mistakes as arising from organizational issues and uses their root causes as constructive learning, rather than blaming individuals.Source: D. Just Culture and Reproducible… · p. 12
  • Public code disclosure can expose flawed evaluations for correction; the authors cite DeepSec as a case where accessible code helped reveal a flaw after publication.They argue that errors can be systematized into practical guidance for future research.Source: D. Just Culture and Reproducible… · p. 12

VI. CONCLUSIONS

  • The conclusion calls for reproducible research, precise whole-system threat models, cost assessments that include human effort, and active industry collaboration.Source: §VI · p. 12

APPENDIX A ADDITIONAL CASE STUDIES AND CONSIDERATIONS

The appendix clarifies what adversarial examples and perturbations mean, then uses a researcher–practitioner discussion and competition context to examine differences in threat models and operational priorities.

A. Evasion via Adversarial Examples (and Perturbations)

  • An adversarial example preserves the input’s ground-truth label while changing a correctly classified input into one the model misclassifies; perturbations need not always be imperceptible.The discussion notes that visible or higher-magnitude perturbations may be acceptable when humans do not inspect inputs, provided they are physically realizable.Source: A. Evasion via Adversarial Examples… · p. 17

B. Researchers vs Practitioners (Twitter thread in §II-C)

  • Researchers and practitioners can make valid but mismatched points because they may consider different threat models, including subtle attacks versus attacks that operate at scale.The discussion also says better industry–academia cooperation could support evaluation of full-fledged ML systems against adversarial examples.Source: B. Researchers vs Practitioners… · p. 17

C. Further Considerations on our third case study (§III-C)

  • The competition discussion interprets its events in light of the human effort and decision-making required by top-ranked teams.It supplements the earlier case study with competition rules and an interpretation of participants’ efforts.Source: C. Further Considerations on our third… · p. 17

1) Queries and Submissions:

The appendix distinguishes recorded submissions from detector queries, interprets competition outcomes in light of timing and shared leaderboard information, and contrasts adaptive defenses with simple evasion tactics.

1) Queries and Submissions:

  • A submission counts uploaded webpages, whereas queries are detector calls; each submitted webpage triggered seven queries, and submissions were plotted because organizers logged them.The authors report a strong correlation between submissions and queries while acknowledging that individual query histories would permit finer-grained analysis.Source: 1) Queries and Submissions · p. 17

2) Interpretation (human effort):

  • The authors infer that public leaderboard results shaped later teams’ efforts, while noting uncertainty about what teams did before their first recorded submissions.Third place reached perfect evasion in 12 days with 608 queries; second place took at least 19 days and 343 queries, and first place later achieved it with 320 queries.Source: 2) Interpretation (human effort) · p. 18

D. Case Study: Malware Competition (2020)

  • The attackers’ simple malware-embedding strategy evaded all considered detectors, including the winning defense, while that defense stopped 77% of submitted malware.The paper observes that bypassing some ML systems may not require gradient-based or similarly sophisticated techniques.Source: D. Case Study: Malware Competition… · p. 19

E. Example: Gains and Losses is not a zero-sum game

These passages connect cybersecurity incentives and reproducibility constraints with practical guidance, while reviewer responses clarify the scope of attacker assumptions and behaviors.

E. Example: Gains and Losses is not a zero-sum game

  • An ML detector developer’s losses need not match the attacker’s gains or the customer’s losses, so economic and operational factors can shape whether countermeasures are deployed or developed.The ransomware example describes cases where the customer bears the loss while the developer loses little or is insured.Source: E. Example: Gains and Losses is not a… · p. 19

F. Unreproducible Papers

  • Privacy obligations and employer restrictions can constrain data and source-code release, but papers can still prioritize reproducibility within those limits.Suggested practices include clear model and feature descriptions, suitable public-data evaluation, private-data characterization, and attack descriptions or releases where possible.Source: F. Unreproducible Papers · p. 19

G. Discussion (with the Reviewers)

1) Spammers and regular users:

  • Because high-volume spammers need to distribute many low-value posts, they commonly use automation, while low-volume mimicry requires greater resources and is described as rare.The authors say it is acceptable for the activity layer to miss such rare attacks because spammy content may still be caught at the application layer.Source: 1) Spammers and regular users · p. 20

2) Defenses vs. limited-knowledge attackers (§IV-C):

  • The authors distinguish defenses robust in a query-only setting from defenses that work only because the attacker lacks knowledge, and support evaluating justified, realistic limited-knowledge threats.They do not argue that every defense paper must assume an attacker with perfect knowledge.Source: 2) Defenses vs. limited-knowledge… · p. 20

3) Domain Expertise:

  • Threat models should state an attacker’s knowledge of the ML system while assuming expertise in system components the attacker knows.The authors distinguish domain expertise from perfect knowledge of the target system and say domain knowledge is difficult to quantify.Source: 3) Domain Expertise · p. 20

4) “Adversarial” in practice:

The appendix clarifies how researchers and practitioners use “adversarial,” discusses the implications of nation-state assumptions, and details the scope and procedure of its literature review.

4) “Adversarial” in practice:

  • Researchers use “adversarial attack” for attacks exploiting ML models’ vulnerability to adversarial examples, while practitioners use “attack” more generally.The authors say the term can confuse practitioners because it implies that some attacks are not adversarial.Source: 4) “Adversarial” in practice · p. 20

5) Nation-state adversaries:

  • Assuming an attacker backed by an entire country and willing to invest substantial resources places a paper in a different attacker-cost bracket; unlimited resources should not be assumed by default.The authors add that companies would be more interested in a devastating attack that makes an ML system malfunction.Source: 5) Nation-state adversaries · p. 21

APPENDIX B STATE-OF-THE-ART: LITERATURE REVIEW

  • The literature-review appendix presents its methodology and detailed analysis as a complementary contribution to the position paper.The review used a structured selection-and-inspection approach, with author exchanges intended to reduce bias.Source: Appendix B · p. 21, A. Methodology · p. 21

A. Methodology

  • The review examined 88 papers from the 2019–2021 proceedings of four top security conferences, with two authors independently reviewing each paper and repeating the process five times.The final set followed filtering and inspection of 131 candidate papers.Source: A. Methodology · p. 21

B. Research Questions

The survey classified 88 papers using 12 questions about research trends and threat models. It found that most papers emphasized attacks, especially evasion, while poisoning grew more prominent by 2021.

B. Research Questions

  • The survey coded 88 papers with 12 questions: eight generic research-trend questions and four threat-model questions about attacker knowledge and capabilities.Source: B. Research Questions · pp. 21–22

C. Answers to G1 and G2: Focus, and Attack Family

D. Answer to G3: ML paradigm

The review found a strong emphasis on deep learning and a growing consideration of costs, alongside shifts in data domains and source-code release. It recommends more attention to shallow learning given its distinct properties and relevance in practice.

D. Answer to G3: ML paradigm

  • Only 10 of 88 papers evaluated shallow-learning systems, although shallow learning can outperform deep learning in some domains,.Source: D. Answer to G3: ML paradigm · p. 22
  • Some shallow-learning methods, such as decision trees, do not use gradients, so gradient-based attacks and defenses may not work on them.The authors also note that shallow-learning methods appear in commercial-grade ML systems.Source: D. Answer to G3: ML paradigm · p. 22

E. Answer to G4: Cost

F. Answers to G5 and G6: Data-type and Source-code

G. Answers to G7 and G8: pipeline and type of ML system

Most reviewed attacks targeted individual models rather than complex pipelines, and most evaluations used self-developed systems rather than systems deployed in practice. The review also highlights unclear threat models and presents a paper-by-paper survey table.

G. Answers to G7 and G8: pipeline and type of ML system

H. Answers to the Threat-Model Questions

I. Complete Table

  • Table IV lists all 88 papers in temporal order and records concise answers to the 12 survey questions, including source-code links where available.Source: §I · p. 25, Table IV · p. 26
Loading 2212.14315v1…