PaperlayerView exact revision on arXiv

Source-linked AI summary

Technical Report: When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks

Octavian Suciu, Radu Mărginean, Yiğitcan Kaya, Hal Daumé, Tudor Dumitraş

arXiv:1803.06975v2cs.CRcs.LG

TL;DR

The paper addresses the lack of precise, realistic adversary definitions for evaluating machine-learning attacks. It proposes the FAIL model and StingRay, finding that StingRay is practical across multiple classification settings while a prior evasion attack is less effective under generalized transferability.

  • Problem

    Existing attack evaluations often assume adversary capabilities that may exceed those available against proprietary machine-learning systems, limiting realistic assessment of security threats.

  • Method

    The paper proposes FAIL, which models adversarial knowledge and control across Features, Algorithms, Instances, and Leverage, and designs StingRay for targeted poisoning under realistic constraints.

  • Results

    StingRay is practical for 4 classification tasks using 3 different classifiers, while a prior evasion attack is less effective under generalized transferability.

  • Takeaways & Limitations

    FAIL provides a common framework for evaluating attack transferability across weaker adversaries and for identifying characteristics relevant to attack resilience and future defenses.

  • Takeaways & Limitations

    The targeted-poisoning threat model assumes the attacker has partial classifier knowledge and read-only access to the target representation, without control over the target or its natural label.

Abstract

from arXiv · show

Recent results suggest that attacks against supervised machine learning systems are quite effective, while defenses are easily bypassed by new attacks. However, the specifications for machine learning systems currently lack precise adversary definitions, and the existing attacks make diverse, potentially unrealistic assumptions about the strength of the adversary who launches them. We propose the FAIL attacker model, which describes the adversary's knowledge and control along four dimensions. Our model allows us to consider a wide range of weaker adversaries who have limited control and incomplete knowledge of the features, learning algorithms and training instances utilized. To evaluate the utility of the FAIL model, we consider the problem of conducting targeted poisoning attacks in a realistic setting: the crafted poison samples must have clean labels, must be individually and collectively inconspicuous, and must exhibit a generalized form of transferability, defined by the FAIL model. By taking these constraints into account, we design StingRay, a targeted poisoning attack that is practical against 4 machine learning applications, which use 3 different learning algorithms, and can bypass 2 existing defenses. Conversely, we show that a prior evasion attack is less effective under generalized transferability. Such attack evaluations, under the FAIL adversary model, may also suggest promising directions for future defenses.

1 Introduction

The paper argues that realistic ML security analysis needs precise adversary models rather than assumptions that overestimate attacker capabilities. It introduces FAIL and StingRay to evaluate attacks under partial knowledge and control.

  • Motivation: Existing attack evaluations can misrepresent real-world security threats when they assume capabilities that realistic adversaries lack.White-box access is often assumed despite proprietary ML models, while black-box models may impose rigid assumptions.
  • FAIL model: FAIL models adversarial knowledge and control across Features, Algorithms, Instances, and Leverage dimensions.The framework also generalizes attack transferability across multidimensional surrogate models.
  • Targeted poisoning: Targeted poisoning is evaluated under clean labels, individual and collective inconspicuousness, limited collateral damage, and generalized transferability constraints.The attacker can inject samples but cannot determine their labels, and samples must shift the boundary toward a target without broadly harming classification.
  • StingRay: StingRay is presented as a practical targeted poisoning attack spanning 4 ML systems and 3 classification algorithms while bypassing two existing anti-poisoning defenses.The attack is described as model agnostic and applicable across a broad range of settings.
  • Findings: The evaluation finds that a previous black-box evasion attack is less effective under generalized transferability.The paper uses the FAIL dimensions to examine transferability across weaker adversaries.

2 Problem Statement

The problem statement targets inconsistent threat models and unrealistic assumptions about attacker knowledge, control, labeling, detectability, and collateral damage. It asks whether targeted poisoning can remain effective under these constraints.

  • Threat-model gap: Prior work defines adversaries inconsistently, with black-box settings ranging from full feature knowledge to access only to raw data.This divergence reflects the lack of a unifying threat model for adversarial knowledge.
  • Threat-model gap: Real ML systems may combine public architectures and features with proprietary data and undisclosed features, exceeding rigid white-box or black-box definitions.A commercial malware detector illustrates this mixed-knowledge setting.
  • Transferability: Existing transferability studies usually vary one adversarial knowledge dimension at a time, limiting understanding of attack capabilities against real systems.The paper frames generalized transferability as a broader alternative.
  • Poisoning constraints: Targeted poisoning attacks may require label control, produce conspicuous samples, and cause collateral misclassification of non-target instances.These limitations can reduce the attack’s practical impact and make samples vulnerable to existing defenses.
  • Targeted poisoning: The threat model gives Mallory partial classifier knowledge and read-only target-feature access while withholding control over the target and its natural label.The target is initially correctly classified, and the attack derives a poisoned training set to induce a desired label.

3 Modeling Realistic Adversaries

FAIL represents realistic adversaries through four dimensions of knowledge and capability, then uses those dimensions to analyze transferability, assumptions, and defenses. The framework exposes omitted limitations and incomplete evaluations in prior work.

  • FAIL dimensions: FAIL formalizes adversarial knowledge and capabilities across Features, Algorithms, Instances, and Leverage dimensions.The dimensions determine what the adversary knows, what training data and algorithm it uses, and what features it can modify.
  • FAIL dimensions: Unknown features or algorithms make the crafting model an approximation of the victim, while limited instance knowledge can overestimate attack impact.Restricted leverage also limits where crafted samples can lie and how far poisoning can shift a decision boundary.
  • FAIL dimensions: The Algorithm dimension distinguishes black-box, gray-box, and white-box knowledge of the learning algorithm and classifier parameters.Gray-box access provides partial information about the algorithm class or ensemble architecture.
  • FAIL dimensions: The Instance dimension measures overlap between attacker-available instances and those used by the victim, including settings with no original training instances.Instance scarcity affects the robustness of the attacker’s surrogate classifier.
  • FAIL dimensions: Leverage captures which features an attacker can modify and the side effects of those modifications.Feature dependencies can make changes to one feature indirectly alter others.
  • Evaluation framework: FAIL evaluates prior attacks by asking whether limitations are specified and whether techniques are evaluated along each dimension.This analysis reveals implicit assumptions and missing dimension-specific evaluations.
  • Defense analysis: The framework also examines whether dimensions are used to harden systems, while showing that defenses can rely on implicit assumptions and be bypassed by different attacks.Examples include distillation against evasion and RONI’s assumption of training-set secrecy.

4 The StingRay Attack

StingRay is a general targeted-poisoning framework that crafts inconspicuous samples from target-near base instances while accounting for defenses, budgets, labeling, and transferability. Implementations span four applications and three classifiers, with mechanisms intended to preserve attack effectiveness while limiting detectability and collateral damage.

  • Attack construction: StingRay builds poison instances from base samples near the target, assigning the desired target label and applying small perturbations.The framework then evaluates each crafted instance’s negative impact and retrains the surrogate with accepted poisons.
  • Applications: Implementations cover an image classifier, Android malware detector, Twitter exploit predictor, and data-breach predictor using convolutional neural networks, linear SVM, and random forests.The applications were reproduced or implemented to approximate existing systems and span distinct classifier characteristics.
  • Attack constraints: StingRay uses minimum and maximum poison counts, a sampling budget, and extra crafted instances to improve robustness against sampling-based defenses.The attack can discard samples that fail to produce the desired impact, while filtering may reduce the injected set below the effective threshold.
  • Attack constraints: The attack balances inconspicuousness and effectiveness through target resemblance, negative-impact checks, and collective performance-drop constraints.Fewer modified features reduce outlier risk but may require more samples; excessive resemblance can also affect oracle labeling.
  • Defense mechanisms: Target-aware RONI examines an instance’s effect on the target classification rather than only its negative impact on a holdout set, addressing targeted-attack weaknesses.This defense is motivated by the observation that individually mild poison instances can evade ordinary RONI.
  • Applications: For image attacks, StingRay selects deep-feature-near base instances and uses the H3 representation because lower layers improve transferability while higher layers can reduce perturbation.The experiments also constrain pixel perturbations using an l∞ threshold tied to human labeling performance.
  • Applications: For Twitter exploit prediction, the attacker changes features of existing vulnerabilities by posting crafted tweets, because vulnerabilities originate in public databases and new training instances cannot be introduced.The selected vulnerabilities resemble the target and receive tweets containing terms associated with the target vulnerability.
  • Applications: Android poison generation adds target-derived features to benign applications and submits repackaged samples for labeling, producing 19,632 valid files after 5,850 generation failures.The procedure targeted 549 applications and sought to craft 25,482 samples.

5 Evaluation

Under generalized transferability, attack effectiveness varies sharply with adversarial knowledge and control. StingRay remains practical across image and malware settings, while feature secrecy and limited leverage most reduce success.

  • Evasion attack: Limited feature knowledge leaves JSMA’s perceived success high but actual SR very low, exposing sensitivity to feature secrecy.The findings identify feature secrecy as a potential direction for future defenses.
  • Evasion attack: 19% SR for a robust attacker classifier shows JSMA success sharply declines with fewer victim training instances available.Prior work reported SR above 80% under non-random data augmentation, which produced a closer attacker–victim model approximation.
  • StingRay on the image classifier: Over 84% perceived SR under limited feature knowledge falls significantly on the victim, although experiment #2 approaches white-box actual SR.Features derived from exterior image regions appear less instance-specific, so some known features can still enable successful attacks.
  • StingRay on the image classifier: Architecture secrecy does not significantly improve resilience because attackers can approximate deep-space distances and achieve actual SR comparable to white-box attacks.Perceived success is overestimated, while actual success remains comparable to the white-box setting.
  • StingRay on the malware classifier: More available victim instances can unexpectedly lower actual SR when attacker and victim training sets diverge, because base-instance effects are not bootstrapped.This explains why experiment #8 performs worse than #7 despite greater training-instance access.
  • StingRay on the image classifier: Limited leverage can increase actual SR beyond white-box performance by reducing perturbations and making poison samples collectively inconspicuous.Figure 3 illustrates images crafted by constrained adversaries.
  • StingRay on the malware classifier: Feature secrecy and limited leverage most reduce malware attack success, suggesting both as viable defense directions, while defenses often fail or fix only a minority of attacks.For example, tRONI fixes 15% of attacks in one evaluation, and another defense fixes 40% while hurting PDR.

6 Related Work

Prior work studies evasion and poisoning attacks across classifiers, including targeted, indiscriminate, backdoor, and trojan settings. These approaches often assume substantial attacker control, such as label assignment or influential-instance modification.

  • Adversary models: The FTC model defines attacker knowledge and capabilities for practical evasion, whereas FAIL covers test- and training-time attacks across four dimensions including leverage.FAIL also enables finer-grained analysis of how features and leverage influence attacks.
  • Poisoning attacks: Prior indiscriminate poisoning attacks degrade classifiers by injecting spam-like, fraudulent, or crafted instances that alter learned feature associations.Examples target Bayesian filters, Twitter-based exploit predictors, and SVM classifiers.
  • Targeted and neural-network attacks: Targeted poisoning has been studied against spam filters and healthcare predictors, while neural-network work includes influential-instance, white-box, backdoor, and trojan attacks.Some methods cause misbehavior only when a trigger is present, and one targeted poisoning attack requires attacker-assigned labels.

7 Discussion

The FAIL framework exposes how adversarial knowledge and control shape attack transferability and defense resilience. Results identify feature secrecy and leverage as important factors, while raising a testable hypothesis about selective memorization in neural networks.

  • Feature secrecy appears most promising for improving resilience against targeted poisoning of linear classifiers.
  • Reducing leverage can increase attacker cost for linear classifiers, but limiting leverage can make neural-network poison instances more potent.
  • Training-instance secrecy provides limited resilience against the evaluated targeted poisoning attacks.
  • FAIL generalizes attack transferability across broader combinations of adversarial knowledge and capabilities.The framework supports systematic evaluation across its four dimensions rather than only selected dimensions.
  • A state-of-the-art evasion attack lacks generalized transferability, contrary to prior findings, with feature secrecy most strongly reducing attack success.
  • The results suggest StingRay may induce selective memorization for a target while preserving model generalization, but this hypothesis remains untested rigorously.

8 Conclusions

The paper introduces FAIL to evaluate realistic ML attacks and StingRay to conduct targeted poisoning under such constraints. Across four classification tasks and three classifiers, it reports practical poisoning attacks and new transferability and resilience findings.

  • FAIL is a general framework for evaluating realistic attacks against machine learning systems.
  • StingRay is a targeted poisoning attack designed to bypass existing defenses.
  • StingRay is practical for 4 classification tasks using 3 different classifiers, while FAIL reveals new transferability properties and resilience characteristics.

A The StingRay Attack

StingRay builds targeted poison sets from target-similar base instances, modifies only allowable features, and evaluates both targeted misclassification and collateral impact. Clean-label and inconspicuousness constraints guide sample construction and label preservation.

  • STINGRAY builds between Nmin and Nmax attack instances using base-instance selection and iterative poison crafting.
  • CRAFTINSTANCE modifies allowable features under FAIL’s leverage dimension to create each poison instance.
  • Small, complementary feature modifications collectively teach the classifier that target features correspond to the desired label.
  • GETBASEINSTANCE selects target-class base instances within τD distance of the target to reduce outlier risk.
  • GETPDR accepts an attack only when it changes the available classifier’s target prediction without reducing PDR below τPDR.
  • CRAFTINSTANCE aims to preserve the base label by changing only a few features, while an oracle may help the adversary guess the poison labels.

B Limitations of the RONI Defense.

RONI is designed for indiscriminate attacks that degrade overall performance, creating limitations for targeted poisoning. In the Drebin evaluation, detection improves as thresholds become more permissive, but low-impact poisons often evade both RONI variants.

  • RONI lowers its negative-impact threshold across iterations because the remaining training set becomes cleaner and poisons have lower impact.
  • Figure 4 compares the percentage of crafted instances detected by RONI and target-aware RONI across decreasing negative-impact thresholds and attack categories.
  • RONI’s core assumption is suited to indiscriminate attacks, where poisoned instances cause performance degradation by inducing many mispredictions.
  • RONI and target-aware RONI fail to detect most low-negative-impact attack instances against Drebin, although target-aware RONI generally detects more.
  • Detection increases linearly as the impact threshold is lowered, illustrating the trade-off between detecting more poisons and increasing false positives.
  • Prior studies differ in whether they evaluate transferability, partial feature knowledge, query requirements, and model-parameter secrecy.
  • The FAIL analysis records how existing attacks vary in feature, algorithm, instance, and leverage knowledge or control, including implicit assumptions and missing evaluations.
  • Gu et al.’s threat model grants full access to model architecture, parameters, training data, and feature representation, though emerging frameworks may relax that assumption.
Loading 1803.06975v2…