PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Privacy Attacks in Machine Learning

Maria Rigaki, Sebastian Garcia

arXiv:2007.07646v3cs.CRcs.LG

TL;DR

Privacy research on machine-learning models has lagged behind security research, despite uncertainty about whether trained models reveal sensitive training information. This paper surveys over 40 privacy-attack papers, organizes them with a threat model and taxonomy, and finds recurring patterns across attack types, model types, access settings, and attack timing. It also reviews leakage causes, defenses, and unresolved research questions.

  • Problem

    It is not well known whether and how machine-learning models reveal sensitive training information, while existing surveys provide only partial, overly broad, or narrowly specialized coverage.

  • Method

    The paper systematically analyzes over 40 research papers and develops a threat model and unifying taxonomy based on adversarial knowledge and attacked assets.

  • Results

    Membership inference and reconstruction each account for 35.7% of reviewed papers, model extraction accounts for 31%, and 88% of attacks occur during inference.

  • Takeaways & Limitations

    The survey identifies common attack design patterns and differences while organizing known causes, defenses, and future research directions for machine-learning privacy.

  • Takeaways & Limitations

    Many attacks work only under specific assumptions or do not scale to larger datasets, numbers of classes, or numbers of participants, and privacy leaks remain theoretically incompletely understood.

Abstract

from arXiv · show

As machine learning becomes more widely used, the need to study its implications in security and privacy becomes more urgent. Although the body of work in privacy has been steadily growing over the past few years, research on the privacy aspects of machine learning has received less focus than the security aspects. Our contribution in this research is an analysis of more than 40 papers related to privacy attacks against machine learning that have been published during the past seven years. We propose an attack taxonomy, together with a threat model that allows the categorization of different attacks based on the adversarial knowledge, and the assets under attack. An initial exploration of the causes of privacy leaks is presented, as well as a detailed analysis of the different attacks. Finally, we present an overview of the most commonly proposed defenses and a discussion of the open problems and future directions identified during our analysis.

1 INTRODUCTION

The survey addresses the underexplored privacy and confidentiality risks of machine-learning models by organizing prior attacks, their causes, defenses, and open problems. It reviews more than 40 studies and proposes a threat model and unifying taxonomy.

  • Motivation: Models trained on sensitive data may reveal information such as location, health records, or identity information to adversaries.The survey emphasizes uncertainty about whether and how trained models reveal information about their training data.
  • Causes of leakage: Privacy leaks can arise from model structure, poor generalization, memorization, and adversarial-robustness training.The survey links these factors to differing susceptibility across attack types.
  • Scope: The survey focuses on attacks extracting training-data information or the model itself, addressing prior surveys that were too broad or narrow.Model extraction is included because model confidentiality attacks are commonly grouped with privacy attacks and functionality theft may constitute a privacy breach.
  • Survey method: The review systematically analyzes over 40 papers published during 2014-2020 using keyword, backward, and forward searches.The papers span security, privacy, and machine-learning venues.
  • Contributions: Its contributions include a comprehensive attack study, unifying taxonomy, leakage analysis, attack implementation review, and defense overview.The paper also organizes the remaining discussion around machine-learning concepts, threat models, attacks, causes, defenses, and future directions.

2 MACHINE LEARNING

The paper introduces major machine-learning paradigms and architectures, then distinguishes centralized from distributed learning according to whether data and models are collocated. Distributed variants are motivated by scale, resources, and privacy concerns.

  • Learning paradigms: Machine learning is broadly divided into supervised, unsupervised, and reinforcement learning, with newer categories including semi-supervised and self-supervised learning.Models may also be categorized as generative or discriminative.
  • Supervised and unsupervised learning: In supervised learning, models map inputs to outputs using labeled training pairs, while unsupervised learning finds structure from inputs without labels.Common supervised tasks include classification and regression; unsupervised tasks include clustering, feature learning, anomaly detection, and dimensionality reduction.
  • Reinforcement learning: Reinforcement learning uses environmental observations and actions to maximize a reward signal, which may arrive after a sequence of actions.The survey reports no known privacy attacks directly against reinforcement learning at the time discussed.
  • System architectures: Centralized learning collocates data and models, whereas distributed learning separates them across locations or devices.This distinction depends on data-model collocation rather than computational parallelism within one data center.
  • Distributed learning: Privacy-relevant distributed architectures include federated, peer-to-peer, and split learning.Federated learning keeps data on remote devices and aggregates local model updates; split learning sends intermediate outputs instead of input data.

2.3 Training and Inference

Training commonly minimizes empirical loss through iterative gradient-based optimization, while distributed methods aggregate locally computed updates. After training, inference uses fixed model parameters, and most surveyed attacks occur during inference.

  • Centralized training: Empirical Risk Minimization minimizes average loss over the available training dataset rather than the unknown true data distribution.A regularization term may be added to reduce overfitting and stabilize training.
  • Gradient-based optimization: Gradient descent updates model parameters by following loss gradients, while stochastic and mini-batch variants reduce the cost of training on large datasets.Mini-batch size affects the variance of the gradient estimate and algorithm performance.
  • Optimization variants: Momentum, adaptive learning rates, and their combination in Adam are common improvements intended to speed convergence.The survey identifies RMSprop as an adaptive-learning-rate example.
  • Distributed training: Federated averaging has each participant compute local updates, after which a parameter server averages them and redistributes the updated global model.The update weights participants according to their local data quantities.
  • Distributed training: Downpour or synchronized SGD aggregates distributed gradients at a parameter server before taking a global gradient step.The updated parameters are then sent back to remote participants.
  • Inference: Most surveyed attacks occur during inference, whereas attacks on collaborative learning usually occur during training.During inference, model parameters are assumed fixed even though models may later be monitored or retrained.

3 THREAT MODEL

The threat model categorizes privacy attacks by their actors, sensitive assets, adversarial knowledge, and timing. It covers black-box through white-box access and excludes security attacks based on intentional manipulation or infrastructure compromise.

  • Assets and actors: The protected assets include the training dataset, model, parameters, hyper-parameters, and architecture.Actors include data owners, model owners, model consumers, and adversaries.
  • Threat-model structure: The threat model represents actors, assets, information flows, and adversary actions within a logical system model.Assets may be collocated or distributed across multiple locations.
  • Distributed settings: Distributed learning permits local, global-model, and colluding adversaries because multiple actors participate in training.In federated learning, an adversary may be collocated with the global model or operate locally.
  • Adversarial knowledge: Adversarial knowledge ranges from API access to complete model parameters and training settings, with partial architecture or hyper-parameter knowledge between them.The taxonomy uses this range to model different attack surfaces.
  • Access categories: Black-box attacks lack knowledge of model parameters, architecture, and training data, whereas white-box attacks assume full parameter or training-gradient access.Partial white-box attacks occupy the intermediate range; most studies assume knowledge of the expected input.
  • Attack timing: The survey primarily covers inference-time privacy attacks and training-time attacks in collaborative learning.Training-time attacks may involve passive or active adversaries, with active attackers interfering in training.
  • Scope boundary: The survey excludes security attacks such as poisoning and evasion, as well as attacks against infrastructure hosting data, models, or services.Its scope is unintentional information leakage about data or the machine-learning model.

4 ATTACK TYPES

The survey organizes privacy and confidentiality attacks into four types—membership inference, reconstruction, property inference, and model extraction—distinguished by the information or model assets targeted. These attacks range from determining training-set membership and reconstructing samples to inferring unintended dataset properties or replicating model behavior.

  • The taxonomy contains four attack types: membership inference, reconstruction, property inference, and model extraction.
  • Membership inference: Membership inference determines whether an input sample was used in the training set, using black-box outputs or, in collaborative settings, white-box parameters and gradients.Generative models, including GANs and VAEs, are also susceptible to membership inference.
  • Reconstruction: Reconstruction attacks recover complete or partial training samples, labels, or sensitive features from outputs and partial feature knowledge.The survey groups attribute inference and model inversion under reconstruction attacks, distinguishing actual data recovery from class representatives or probable sensitive values.
  • Property inference: Property inference extracts unintended information about dataset-wide or subset properties unrelated to the model’s training task.Examples include demographic ratios or characteristics not encoded as features or labels.
  • Model extraction: Model extraction uses black-box queries to build a substitute model that matches the target’s task accuracy or decision behavior, and may recover architectural or training details.The survey distinguishes task accuracy extraction from fidelity extraction and also covers recovering hyperparameters and architectural properties.

5 CAUSES OF PRIVACY LEAKS

Privacy leakage depends on model construction, generalization, architecture, model type, dataset structure, memorization, and robustness training. The causes differ across attack types: overfitting often strengthens membership inference, while model extraction shows the opposite pattern, and property-inference causes remain poorly understood.

  • Model construction can expose information directly: SVM support vectors are training points, while linear classifiers can be reverse engineered from enough input/output pairs.Deep neural networks are harder to attack simply because they have many parameters, though leakage remains possible under assumptions and conditions.
  • Membership inference: Overfitting increases black-box membership-inference accuracy, and membership advantage is proportional to generalization error; overfitting is sufficient but not necessary.
  • Membership inference: Model architecture, model type, and dataset structure affect membership-inference accuracy independently of generalization error.More complex models can leak more, Naive Bayes is less susceptible than decision trees or neural networks, and more dataset classes can increase leakage.
  • Membership inference: Robust training methods such as PGD adversarial training can increase susceptibility to membership inference.The survey relates this effect to robust training methods increasing generalization error.
  • Membership inference: In GANs, overfitting is treated as an important factor in successful membership inference because standard supervised-learning generalization error is not directly applicable.
  • Reconstruction: Reconstruction risk can increase with generalization error, target-feature influence, predictive power, memorization, and retrieval of out-of-distribution data.Some findings assume adversary knowledge of target-feature and label priors, while other results use weaker assumptions.
  • Property inference: Property inference can succeed even with well-generalized models, but the conditions that make it possible or effective remain insufficiently understood.The survey identifies this uncertainty as an avenue for theoretical and empirical future research.
  • Model extraction: Model extraction behaves oppositely to black-box membership inference: highly accurate models can be stolen, while higher generalization error makes extraction harder.Models with at least 98% test accuracy have been shown vulnerable to parameter stealing, and more classes may worsen extraction performance.

6 IMPLEMENTATION OF THE ATTACKS

The survey analyzes more than 40 papers on machine-learning privacy attacks and presents their techniques and essential differences. It organizes the literature around attacks on centralized and distributed learning.

  • More than 40 papers were analyzed on privacy attacks against machine learning.
  • The section describes commonly used attack techniques and their essential differences.
  • The papers are discussed in two groups: attacks on centralized learning and attacks on distributed learning.

6.1 Attacks Against Centralized Learning

Centralized-learning privacy attacks commonly use shadow models and meta-models to infer membership or properties from target-model outputs. The survey also covers threshold, reconstruction, and model-extraction attacks, whose effectiveness depends on available data, model access, and attack assumptions.

  • Shadow training: Shadow models emulate the target model, whose outputs and known shadow labels train a meta-model for membership or property inference.The attack dataset pairs shadow-model outputs with shadow-data labels, after which the meta-model processes target-model outputs.
  • Membership inference attacks: Membership-inference attacks can use prediction vectors, and Salem et al. showed they are possible with as little as one shadow model.The number of shadow models affects attack accuracy while increasing attacker cost.
  • Membership inference attacks: Threshold-based attacks replace meta-model training with a learned membership threshold, and Bayes-optimal loss-based inference reached up to 90.8% accuracy on VGG16 trained on Imagenet.Thresholds can be learned from multiple shadow models or without shadow models.
  • Membership inference attacks: Attack accuracy was largely unaffected when target, shadow, and meta-model types differed, while decision trees often outperformed neural networks in attack accuracy and precision.The reported comparisons included k-NN, logistic regression, decision trees, and naive Bayes classifiers.
  • Membership inference attacks: Sequence-generation models were harder to attack than image-classification models, but membership in out-of-domain and out-of-vocabulary data was easier to infer.This finding concerns membership inference against sequence-to-sequence models for machine translation.
  • Reconstruction and model extraction attacks: Reconstruction attacks use auxiliary information or generative models to produce likely inputs, while model extraction selects informative queries to approximate target functionality.GAN inversion optimizes latent vectors using realistic-image and target-network-likelihood losses; extraction can use alternative-domain, semi-supervised, or random inputs.
  • Reconstruction and model extraction attacks: Full extraction of certain two-layer ReLU networks reached almost 100% fidelity on MNIST using an average of 219.2 to 222.2 queries, under stated access assumptions.The approach exploits the piecewise-linear structure of ReLU networks, and the passage also states an assumption involving loss gradients.

6.2 Attacks Against Distributed Learning

Distributed-learning systems expose parameters, gradients, intermediate outputs, or participant updates that support privacy attacks. Reported attacks include passive and active membership or property inference, GAN-based reconstruction, and gradient-matching reconstruction, with effectiveness shaped by attacker position and participant count.

  • Distributed-learning exposure: Federated and split learning expose global models, participant updates, loss gradients, or intermediate-layer outputs that can support privacy attacks.The parameter server may access model parameters or gradients, while split-learning servers receive intermediate outputs.
  • Membership inference attacks: White-box membership inference reached up to 75.1% accuracy using model parameters and loss gradients, with target models having high generalization error.The attack assumes auxiliary knowledge about some training data.
  • Membership inference attacks: Active local attacks reached 76.3% accuracy and active global attacks reached 92.1%, while local accuracy dropped significantly after five or more participants.Active attacks alter gradient updates, and active accuracy exceeded passive accuracy across tested scenarios.
  • Property inference attacks: Passive property inference requires data possessing and lacking the target property and applies to federated averaging and synchronized SGD.The adversary uses property labels alongside training data and receives parameter updates after each round.
  • Reconstruction attacks: GAN-based reconstruction can exploit participant or parameter-server access, but both described methods assume auxiliary victim data and one generates only class representatives.A parameter-server adversary can use shared parameters to distinguish real and generated data and participant identity.
  • Reconstruction attacks: Gradient matching can reconstruct high-quality training samples from participant loss gradients, especially when the batch size is small.The attack optimizes dummy inputs and labels so a dummy model matches the participant’s gradients; L-BFGS solves the minimization problem.
  • Reconstruction attacks: Split-learning reconstruction attacks can occur during inference when a centralized server receives outputs from local nodes processing new data.The surveyed scenarios include white-box settings with access to initial layers.

6.3 Summary of Attacks

The survey organizes 42 privacy-attack papers by adversarial knowledge, model, attack type, timing, learning task, datasets, and attacking techniques. The literature concentrates on neural networks, classification, inference-time attacks, and membership or reconstruction attacks, while coverage of other algorithms and settings remains limited.

  • 42 papers are summarized by adversarial knowledge, model type, attack type, and attack timing.
  • 83.3% of papers studied neural networks, while decision trees were the second most targeted model type at 11.9%.
  • Membership inference and reconstruction each appeared in 35.7% of papers, followed by model extraction at 31%; 88% of attacks occurred during inference.
  • Random forests, gradient boosting trees, and reinforcement learning received little to no attention in the surveyed attack literature.
  • The reviewed literature primarily targets supervised classification, with most attacks focused on classification models across all four attack types.
  • Nine papers used shadow training, four used active learning mainly for model extraction, and five used generative models across attack types.
  • Dataset diversity supports different hypotheses but makes methods harder to compare; MNIST, CIFAR-10, CIFAR-100, and UCI Adult appeared in more than six papers, whereas 26 datasets appeared once.
  • Most studies used relatively small datasets and models, which may limit representativeness of realistic scenarios.

7 DEFENDING MACHINE LEARNING PRIVACY

The survey reviews defenses for machine-learning privacy attacks, emphasizing differential privacy, regularization, output restriction, gradient modification, and query detection. Their effectiveness depends on the attack type and can involve utility trade-offs, unsuccessful low-participant federated settings, or adaptive evasion.

  • Differential privacy is the most prominent defense against membership inference because it bounds the impact of individual records on model outputs.
  • Differential privacy adds noise calibrated to function sensitivity; in deep learning, noise is applied during gradient calculation with gradient clipping.
  • Differential privacy can protect against membership inference only when models considerably sacrifice utility or accuracy.
  • Participant-level differential privacy in federated learning requires many participants and was unsuccessful when tested with as few as 30.
  • Regularization such as dropout targets overfitting linked to black-box membership inference, but reported success varies across studies.
  • Restricting outputs to the top k predictions or only class labels did not fully mitigate membership inference because misclassifications can still leak information.
  • Zeroing loss gradients below a threshold was effective against reconstruction attacks with 20% of gradients removed and negligible performance effects.
  • Differential privacy does not appear to protect against property inference, while dropout can strengthen those attacks.

8 DISCUSSION

Privacy attacks remain exploratory: many depend on specific assumptions, scale poorly, and lack a mature theoretical account of privacy leaks. Research also needs broader task coverage, better evaluation datasets, and closer integration with related areas such as fairness and explainability.

  • Many privacy attacks require specific assumptions or fail to scale to larger datasets, class counts, or participant numbers.Theoretical understanding of why these attacks are possible remains incomplete.
  • Realistic deployments impose additional constraints that must be understood when evaluating whether privacy attacks work in practice.The discussion emphasizes that practical attack feasibility depends on constraints arising in real systems.
  • Research has focused mainly on supervised learning, especially deep learning, while random forests, gradient boosting, unsupervised learning, and semi-supervised learning remain less explored.Attack transfer across modalities is also uneven and may require different assumptions or design choices.
  • The field lacks a common approach for selecting datasets or defining minimum dataset requirements for evaluating privacy attacks.Dataset choice has already been shown to affect attack success.
  • Privacy research is connected to other areas: adversarial training affects membership inference, model censoring can leak private attributes, and property inference can support bias detection.These connections motivate reviewing fairness and explainability research as potentially complementary.

9 CONCLUSION

The survey synthesizes privacy-related attacks in machine learning through a threat model and unifying taxonomy, analyzing common design patterns and differences. It identifies open problems including narrow model coverage and underdeveloped theory, while positioning the work as background for further research.

  • The survey comprehensively studies privacy-related attacks and proposes a threat model and unifying taxonomy based on attack characteristics.Its analysis examines common design patterns and differences across the state of the art.
  • Research remains exploratory, with privacy-leak studies dominated by deep learning despite the real-world relevance of other algorithms and models.The authors call for closer examination of popular deployed algorithms and models.
  • A thorough theoretical understanding of why privacy leaks occur remains underdeveloped, alongside the need to address other open research problems.The authors present the survey as background for readers and researchers working on machine-learning privacy leaks.
Loading 2007.07646v3…