PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards the Science of Security and Privacy in Machine Learning

Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, Michael Wellman

arXiv:1611.03814v1cs.CRcs.LG

TL;DR

This paper systematizes fragmented knowledge about machine-learning security and privacy by organizing attacks, defenses, and threat models. It identifies emerging vulnerabilities and formally examines tensions among model complexity, accuracy, and resilience to adversarial manipulation.

  • Problem

    Research on machine-learning security and privacy is fragmented across communities, with limited unified understanding of its threats, attacks, and defenses.

  • Method

    The paper analyzes machine-learning threat models and systematizes representative attacks and defenses within an adversarial framework.

  • Results

    The analysis identifies numerous vulnerabilities and countermeasures, and formally shows tensions between security or privacy and prediction precision in finite-capacity systems.

  • Takeaways & Limitations

    Machine-learning model complexity, accuracy, and resilience must be calibrated for the environments in which systems are used.

Abstract

from arXiv · show

Advances in machine learning (ML) in recent years have enabled a dizzying array of applications such as data analytics, autonomous systems, and security diagnostics. ML is now pervasive---new systems and models are being deployed in every domain imaginable, leading to rapid and widespread deployment of software based inference and decision making. There is growing recognition that ML exposes new vulnerabilities in software systems, yet the technical community's understanding of the nature and extent of these vulnerabilities remains limited. We systematize recent findings on ML security and privacy, focusing on attacks identified on these systems and defenses crafted to date. We articulate a comprehensive threat model for ML, and categorize attacks and defenses within an adversarial framework. Key insights resulting from works both in the ML and security communities are identified and the effectiveness of approaches are related to structural elements of ML algorithms and the data used to train them. We conclude by formally exploring the opposing relationship between model accuracy and resilience to adversarial manipulation. Through these explorations, we show that there are (possibly unavoidable) tensions between model complexity, accuracy, and resilience that must be calibrated for the environments in which they will be used.

I. INTRODUCTION

The paper systematizes fragmented research on ML security and privacy by developing a unified perspective, threat model, and taxonomy of attacks and defenses. It also examines confidentiality, integrity, availability, privacy, fairness, accountability, and tradeoffs involving accuracy, complexity, and adversarial resilience.

  • Research on ML security and privacy is fragmented across machine learning, security, statistics, and computation theory, with few unified efforts.
  • The paper introduces a unified threat model covering the attack surface, adversarial goals, and capabilities across ML systems.
  • Its security and privacy framework considers the ML lifecycle from training through inference, including attacks that alter training data and attacks that induce or observe inference behavior.
  • It taxonomizes attacks and defenses using elements of PAC learning theory and focuses primarily on classification for exposition.
  • The paper introduces a no free lunch theorem characterizing tradeoffs among model complexity, accuracy, dataset size, and resilience to adversarial manipulation.
  • The review aims to improve security and privacy while noting that societal safety implications of ML and AI are outside its scope.

B. Data Collection: Three Use Cases

The paper introduces supervised, unsupervised, and reinforcement-learning use cases, then describes training, inference, and PAC-based empirical-risk reasoning. It emphasizes that standard accuracy estimates rely on shared data distributions and may miss errors in low-probability regions.

  • B. Data Collection: Three Use Cases: The three use cases cover malicious-versus-benign executable classification, unsupervised network anomaly detection, and reinforcement-learning intrusion response.
  • Machine Learning Empirical Process: Training collects and preprocesses data before selecting a model and learning parameters θ for a hypothesis function hθ(x).
  • Machine Learning Empirical Process: Inference fixes θ and computes hθ(x) on unseen inputs, producing class probabilities or pattern representations depending on the task.
  • A Theoretical Model of Learning: PAC learning provides a theoretical framework for interpreting supervised-learning attacks and defenses across ML systems.
  • A Theoretical Model of Learning: Supervised algorithms use empirical risk minimization as a proxy for expected loss, under conditions involving sufficient samples and a good empirical-risk minimizer.
  • A Theoretical Model of Learning: Test accuracy assumes training, test, and deployment data arise from the same distribution, whereas adversarial modifications can create distribution mismatch.
  • A Theoretical Model of Learning: A model can achieve 100% learning accuracy while misclassifying many inputs in zero-probability regions, which adversaries may exploit.

III. THREAT MODEL

The threat model represents ML systems as data-processing pipelines and organizes security analysis around attack surfaces, adversarial goals, and capabilities. It distinguishes training-time manipulation from inference-time attacks and relates attack strength to available information and access.

  • Threat Surface: The generalized ML pipeline collects input features, processes them digitally, applies the model, and communicates outputs to systems or users.
  • Threat Surface: Training data can contain inference features, supervised outputs or reinforcement-learning rewards, and privileged information unavailable at runtime.
  • Adversarial Capabilities: A threat model specifies adversarial actions and information, with stronger adversaries receiving greater access to systems or data.
  • Inference Phase: Inference attacks may manipulate outputs or gather evidence about model characteristics without tampering with the model itself.
  • Adversarial Capabilities: White-box attacks use model or training-data information, whereas black-box attacks infer vulnerabilities by probing inputs and observing outputs.
  • Threat Surface: The attack surface spans generic ML systems, automotive traffic-sign recognition, and network intrusion detection scenarios.
  • Training Phase: Training attacks seek to learn, influence, or corrupt models through data access, substitute-model construction, data injection, modification, or logic corruption.

C. Adversarial Goals

The paper frames ML security and privacy through adversarial goals affecting confidentiality, privacy, integrity, and availability across the model and deployed system. It also surveys how training-data manipulation targets integrity, with poisoning risks shaped by learning settings and model assumptions.

  • Confidentiality and Privacy: Confidentiality and privacy attacks extract information about the model or training data, including model parameters, membership, partially known inputs, or memorized training examples.These risks arise because model parameters may be valuable intellectual property and models can memorize training data.
  • Integrity and Availability: Availability attacks reduce output quality, performance, consistency, reliability, or access, including through denial-of-service effects in systems that depend on model outputs.Availability is defined for the ML-based system as a whole rather than for an isolated model.
  • Integrity and Availability: Integrity attacks manipulate inputs, training data, or model behavior to induce adversary-chosen outputs, such as false positives or incorrect classifications.Integrity is central to performance metrics such as accuracy, and attacks can target confidence, input processing, or the input-output mapping.
  • Training-Time Attacks: Poisoning attacks alter training data by inserting, editing, or removing points to modify decision boundaries and compromise learning-system integrity.The attacks are bounded because an unbounded adversary could force the learner to implement an arbitrary function and cause complete unavailability.
  • Training-Time Attacks: Surveyed poisoning research focuses almost exclusively on supervised classifiers, limiting how directly its conclusions generalize to other ML tasks.The paper notes that some strategies may nevertheless apply to tasks such as reinforcement learning when supervised submodels are used.
  • Training-Time Attacks: Online learning often exacerbates poisoning because newly collected environmental data provides repeated opportunities to shift the learned model.Offline poisoning methods may depend on SVM-specific closed-form test-error assumptions or convex-loss training.

B. Targeting Privacy

The paper frames ML security and privacy through adversarial access to the deployed system, while examining attacks across direct, indirect, and non-classification settings. It also identifies limitations of ML-based defenses and tensions between robustness, accuracy, and complexity.

  • Targeting Privacy: Training-time confidentiality and privacy depend on adversarial access to the hosting system, making this a traditional access-control issue outside the paper’s scope.The use of ML itself does not alter these properties during training.
  • Targeting Privacy: Inference attackers cannot modify training data or model parameters; white-box access enables finer-grained output control, while black-box access relies on queries or surrogate data.Privacy attacks commonly target training-data or model properties through black-box oracle access.
  • Targeting Privacy: Adversarial examples perturb correctly classified inputs so they remain within the input domain but receive a chosen target label.The paper reports that state-of-the-art object-recognition DNNs can be misled by small perturbations.
  • Targeting Privacy: 89.4% of adversarial examples were misclassified by a near-state-of-the-art MNIST model using the fast gradient sign method.The paper links this behavior to linear extrapolation by model components for inputs far from the training data.
  • Targeting Privacy: Only 4% of MNIST input features were perturbed on average to achieve 97% targeted-class success with a Jacobian-based attack.This is especially relevant when the input domain is discrete and only some features are easily modifiable, such as malware detection.
  • Targeting Privacy: Physical adversarial examples must often use larger, adapted perturbations to survive preprocessing deformations such as contrast changes or blurring.The paper also notes extensions of adversarial-example attacks to autoregressive and reinforcement-learning settings.

B. Black-box adversaries

Black-box adversaries attack models without internal knowledge by querying outputs, exploiting transferability, or inferring private information. These attacks can compromise integrity and privacy, although their feasibility depends on access assumptions and application-specific conditions.

  • Black-box adversaries: Black-box access prevents gradient-based strategies, but transferability can still enable attacks across models and preprocessing pipelines.Physical adversarial examples have misled a smartphone object-recognition model different from the one used to craft them.
  • Black-box adversaries: An oracle threat model lets adversaries submit chosen inputs and observe outputs, a setting relevant to machine-learning-as-a-service platforms.Output access can also support reconnaissance of deployed detection and response policies.
  • Black-box adversaries: Continuous features permit ACRE learnability, whereas discrete features make least-cost black-box misclassification NP-hard.ACRE learnability measures the least-cost modification found with polynomially many oracle queries.
  • Black-box adversaries: A substitute-model attack evading a malware PDF detector did not generalize well across application domains or models.Its construction relied on a semantic gap involving features ignored by PDF renderers.
  • Black-box adversaries: 84.24% of inputs were forced into misclassification against a remotely hosted MetaMind DNN without access to its architecture, parameters, or training data.A follow-up attack caused an Amazon logistic-regression oracle to misclassify 96% of crafted adversarial examples.
  • Black-box adversaries: Black-box attacks make target-class selection more difficult than in white-box settings, despite potentially compromising output integrity.The paper distinguishes attacks using class probabilities from those observing only class labels.
  • Black-box adversaries: Membership inference exploits differing model responses to determine whether a point appeared in training data.Shadow models are trained for each target class using synthetic inputs assumed statistically similar to the black-box training inputs.
  • Black-box adversaries: Model inversion recovered genomic information in a medicine-dosage task, while model extraction recovered parameters from observed input-output pairs.The paper notes uncertainty about whether the inversion result was caused by the ML model or other factors.

VI. TOWARDS ROBUST, PRIVATE, AND ACCOUNTABLE MACHINE LEARNING MODELS

The paper connects robustness to distribution drift, privacy-preserving learning, fairness, and accountability as related mitigation goals. It presents these areas as largely open problems and derives insights for future work.

  • VI. TOWARDS ROBUST, PRIVATE, AND ACCOUNTABLE MACHINE LEARNING MODELS: The paper draws parallels among robustness to distribution drift, privacy-preserving models, fairness, and accountability as mitigation goals for ML security and privacy.These goals concern resilience, bounded data exposure, prevention of biased outputs, and explanations for generated outputs.
  • VI. TOWARDS ROBUST, PRIVATE, AND ACCOUNTABLE MACHINE LEARNING MODELS: Many of these mitigation problems remain largely open, so the paper presents insights intended to inform future work.

A. Robustness of models to distribution drifts

ML systems face distribution drifts during training and inference, while defenses must balance robustness against model capacity and accuracy. The survey finds that gradient-masking defenses can be bypassed through transferability, whereas robust training and distributionally aware formulations address broader perturbations with important limitations.

  • Training-time defenses: Training-time defenses often exploit poisoning samples’ deviation from the expected input distribution, using robust statistics or regularization to reduce model sensitivity.Suggested measures also include holdout-set detection, disinformation, and model randomization, while poisoning can shift online anomaly-detector decision boundaries.
  • Inference-time defenses: Inference-time robustness remains largely open because complex output surfaces provide modeling capacity but also enable adversarial manipulation.The paper describes this as a tension between the complexity needed for robust modeling and the difficulty of defending against malicious test-distribution shifts.
  • Inference-time defenses: Contractive smoothness penalties can increase robustness while greatly reducing model capacity, limiting performance and applicability.The penalty is based on the Frobenius norm of the model’s Jacobian and is approximated layer by layer for efficiency.
  • Inference-time defenses: Distillation and label smoothing can improve resistance to fast gradient sign attacks, but precise iterative or Jacobian-based attacks can still defeat them.Distillation requires larger perturbations for misclassification in reported experiments, yet a variant attack bypassed it on one dataset.
  • Inference-time defenses: Gradient masking smooths local outputs but can be evaded when substitute-model gradients generate adversarial examples that transfer to the defended model.The attack applies broadly to defenses that mask gradients without correcting the underlying erroneous predictions.
  • Inference-time defenses: Larger perturbations require defenses beyond infinitesimal smoothing, including adversarial training and methods designed for robustness to distribution drift.Adversarial samples can be injected during training, but non-adaptive training may fail when evaluation attacks differ from training attacks.
  • Inference-time defenses: Adversarial training is non-adaptive, so effective defense requires including adversarial examples produced by all known attacks.Experiments found improvements over a comparison method were often statistically nonsignificant, and attack mismatch reduced robustness.
  • Distributional formulations: Stackelberg modeling captures defender–adversary interaction under distribution manipulation, but the resulting bilevel optimization problem is generally NP Hard.This formulation extends standard empirical risk minimization by accounting for the adversary’s cost in modifying the distribution.

B. Learning and Inferring with Privacy

Privacy in ML is addressed through formal guarantees and randomized behavior during training or inference. Differential privacy bounds the effect of changing one training record, but stronger privacy can require noise that degrades prediction accuracy and depends on model sensitivity.

  • Privacy guarantees: Differential privacy formalizes privacy by requiring similar output distributions for datasets differing in at most one training record.The framework treats the training point as the record and the ML algorithm as the mechanism whose output is protected.
  • Privacy guarantees: Smaller ε privacy budgets provide stronger guarantees, while δ represents the tolerated failure probability for the ε-based bound.The parameters jointly bound how much algorithm outputs can differ between neighboring datasets.
  • Training-time privacy: Privacy can be introduced during training by randomizing data, the minimized objective, or learned parameters.Examples include randomized response for local privacy and objective perturbation using noise scaled to model sensitivity.
  • Privacy guarantees: Differentially private learning is difficult because model sensitivity is unknown for most interesting ML techniques.The paper also reports privacy guarantees for large-capacity models trained through multi-party computation from perturbed parameters.
  • Inference-time privacy: Inference-time prediction noise can provide differential privacy but degrades accuracy as the number of answered queries increases.The amount of noise introduced grows with query volume, creating an accuracy–privacy trade-off.

C. Fairness and Accountability in Machine Learning

Fairness and accountability extend ML security and privacy concerns to the consequences and explainability of model decisions. The paper emphasizes that biased training data can produce discriminatory outcomes, while accountability methods seek to connect predictions to model inputs or internals.

  • Fairness: Fairness concerns whether actions based on model predictions discriminate against specific individuals.The paper locates fairness in the pipeline stage where predictions drive actions in the physical domain.
  • Fairness: Training data is a major source of bias because it can be deliberately manipulated or can reflect historical social biases.These biases may enter models through dishonest data collection or inherited patterns in historical records.
  • Accountability: Accountability explains predictions using model internals and is relevant to understanding failures on adversarial examples.Few models are interpretable by design, motivating quantitative input-influence measures and sensitivity-based analyses.
  • Accountability: Activation maximization synthesizes inputs that strongly activate specific neural-network neurons, but making those inputs human-interpretable while remaining faithful is challenging.This provides an accountability avenue while exposing a practical interpretability limitation.

VII. NO FREE LUNCH IN ADVERSARIAL LEARNING

The section characterizes inference-time adversarial attacks and shows that robustness, accuracy, hypothesis-class complexity, and data availability are tightly linked. Under finite data, improving resilience with richer models can create overfitting and accuracy trade-offs.

  • No free lunch: With attacker probability q, any supervised learner suffers performance degradation under an α-effective attack.The theorem applies even when the learner randomizes over hypotheses in R(H).
  • Complexity and data: Higher model capacity can improve resilience and precision, but fixed or insufficient data can cause overfitting or high variance.The paper therefore frames capacity selection as an empirical trade-off between resilience and precision.
  • Illustrative example: A linear classifier can incur substantial loss in low-probability regions where the true separator is highly nonlinear, whereas a more complex classifier can lower expected loss.The example illustrates why adversarially emphasizing sparse regions exposes limitations of simple hypothesis classes.
  • Effective attacks: An effective attack increases the best hypothesis’s expected loss under a modified distribution by α > 0.The definition is parameterized by the hypothesis class H and data distribution D, without restricting the adversary to a particular attack.
  • Richer hypothesis classes: A richer hypothesis class can reduce adversarial loss, provided its benign minimum loss improves sufficiently relative to the original class.The positive upper-bound result assumes bounded attack effectiveness and a β-rich class H′.

VIII. CONCLUSIONS

The paper presents ML security and privacy as an emerging field whose vulnerabilities and defenses remain incompletely understood. Its formal analysis identifies a recurring tension between prediction precision and security or privacy under finite capacity.

  • VIII. CONCLUSIONS: ML security and privacy is an active but nascent research area with many vulnerabilities and countermeasures still unknown.The paper describes a science for understanding these issues as slowly emerging.
  • VIII. CONCLUSIONS: The paper organizes attacks and defenses around the attack surface and threat models of ML systems.This framework combines work from diverse scientific communities.
  • VIII. CONCLUSIONS: With finite model capacity, the analysis formally identifies a fundamental tension between security or privacy and prediction precision.The conclusion presents this tension as a central property of ML systems.

APPENDIX

The appendix illustrates how data distribution and adversarial examples affect learned classifiers, and proves a lower-bound result for randomized learners under effective attacks.

  • A. Example of defense using additional data: Figure 6 shows that changing the data distribution can substantially improve a classifier learned from a limited hypothesis class.The hypothesis class contains one or two linear separators, yet the original sampled distribution produces a classifier far from optimal.
  • A. Example of defense using additional data: Adding adversarial examples back into training makes the learned classifier much better in the illustrated example.The red adversarial examples appear in subfigure D.
  • B. Proof of Theorem 1: Theorem 1 proves that when an attacker is present with probability q and uses an α-effective attack, every hypothesis in R(H) incurs a lower-bounded learner loss.The proof handles randomized classifiers by expressing their loss as an expectation over constituent hypotheses.

C. PROOF OF THEOREM 2

The proof of Theorem 2 selects a hypothesis from a richer class that minimizes loss on the adversarial distribution. Combining the richer class’s lower benign loss with the attack model yields an upper bound on expected loss.

  • C. Proof of Theorem 2: Theorem 2 assumes a β-rich hypothesis class H′, attacker probability q, and an α-effective attack with α = α0.It also requires the lowest achievable loss l_D to be sufficiently below the original class’s benign loss.
  • C. Proof of Theorem 2: The proof chooses h′ in H′ to minimize expected loss under the adversary’s modified distribution.This choice directly exploits the richer class’s ability to achieve lower loss for the attacked distribution.
  • C. Proof of Theorem 2: The resulting expected loss is bounded above by the original benign optimum plus qα0 − β.The bound combines the attack’s contribution with the β improvement supplied by H′.
Loading 1611.03814v1…