PaperlayerView exact revision on arXiv

Source-linked AI summary

Membership Inference Attacks on Machine Learning: A Survey

Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, Xuyun Zhang

arXiv:2103.07853v4cs.LGcs.CR

TL;DR

ML models can memorize training information, enabling membership inference attacks that expose whether sensitive records were used for training. This paper surveys the attack and defense literature, organizes it with taxonomies, and identifies resources, challenges, and future directions. The surveyed evidence shows that defenses can reduce attack effectiveness, but privacy protection may involve substantial utility or coverage limitations.

  • Problem

    There was no systematic survey of the rapidly growing literature on membership inference attacks and defenses, despite their privacy risks across ML models.

  • Method

    The paper comprehensively reviews more than 100 works, constructs taxonomies for attacks and defenses, and summarizes datasets, metrics, implementations, challenges, and future directions.

  • Results

    The survey organizes membership inference research across attack and defense characteristics and reports that differential privacy can protect against strong attackers only with poor model utility.

  • Takeaways & Limitations

    The survey provides a reference and resource repository for researchers studying membership inference risks, defenses, evaluation, and future research.

  • Takeaways & Limitations

    Confidence-score masking may leave label-only attacks effective, while current differential-privacy mechanisms may not provide acceptable membership privacy-utility tradeoffs.

Abstract

from arXiv · show

Machine learning (ML) models have been widely applied to various applications, including image classification, text generation, audio recognition, and graph data analysis. However, recent studies have shown that ML models are vulnerable to membership inference attacks (MIAs), which aim to infer whether a data record was used to train a target model or not. MIAs on ML models can directly lead to a privacy breach. For example, via identifying the fact that a clinical record that has been used to train a model associated with a certain disease, an attacker can infer that the owner of the clinical record has the disease with a high chance. In recent years, MIAs have been shown to be effective on various ML models, e.g., classification models and generative models. Meanwhile, many defense methods have been proposed to mitigate MIAs. Although MIAs on ML models form a newly emerging and rapidly growing research area, there has been no systematic survey on this topic yet. In this paper, we conduct the first comprehensive survey on membership inference attacks and defenses. We provide the taxonomies for both attacks and defenses, based on their characterizations, and discuss their pros and cons. Based on the limitations and gaps identified in this survey, we point out several promising future research directions to inspire the researchers who wish to follow this area. This survey not only serves as a reference for the research community but also provides a clear description for researchers outside this research domain. To further help the researchers, we have created an online resource repository, which we will keep updated with future relevant work. Interested readers can find the repository at https://github.com/HongshengHu/membership-inference-machine-learning-literature.

1 INTRODUCTION

The survey addresses privacy risks from membership inference attacks and presents a comprehensive taxonomy-based review of attacks, defenses, evaluation resources, challenges, and future directions.

  • Motivation: Membership inference attacks infer whether a data record was used to train a target ML model, potentially exposing sensitive information.The paper gives clinical records as an example and notes related confidentiality and regulatory risks.
  • Motivation: The survey focuses specifically on MIAs because of their recent emergence and high likelihood of compromising training-data privacy.It distinguishes this focus from broader surveys of privacy or adversarial attacks.
  • Contributions: The authors review over 100 published and pre-print works before September 2021 and establish taxonomies for attacks and defenses.The taxonomies organize attacks by target models, adversarial knowledge, methods, training algorithms, and task domains, while defenses are categorized by technique.
  • Contributions: The survey discusses unresolved challenges and proposes future research directions for both membership inference attacks and defenses.These directions are intended to inspire deeper investigation of the active research area.
  • Resources: The authors summarize datasets, metrics, and open-source implementations to support empirical analysis and future benchmarking.They also provide an online repository containing paper and released-code links, with plans for continued updates.

2 PRELIMINARIES ABOUT MACHINE LEARNING

The preliminaries introduce supervised and unsupervised learning, explain empirical-risk training with SGD, and outline GANs and VAEs as generative models relevant to MIAs.

  • Learning paradigms: Machine learning algorithms are broadly divided into supervised and unsupervised learning according to training-data information and task type.Supervised learning uses labeled data to learn input-output rules, whereas unsupervised learning extracts patterns without access to labels.
  • Supervised Learning: In supervised learning, a model maps feature vectors to labels or continuous outputs, yielding classification or regression tasks respectively.The model parameters are learned from the training dataset.
  • Training Supervised ML Models: Empirical Risk Minimization trains supervised models by minimizing loss on a known training dataset, typically using stochastic gradient descent.Training ends when the model converges to a local minimum with a near-zero gradient.
  • Unsupervised Learning: Generative models learn to generate samples from an underlying data distribution, with GANs and VAEs presented as typical unsupervised models.The survey introduces their training because existing unsupervised-learning MIAs mainly target GANs and VAEs.
  • Generative Models: GANs train competing generator and discriminator networks, while VAEs use an encoder and decoder to model latent representations and reconstruct data.The discriminator distinguishes training samples from generated samples; the VAE encoder maps data to latent space and the decoder maps it back.

3 MEMBERSHIP INFERENCE ATTACKS ON MACHINE LEARNING MODELS

This section introduces the definition of membership inference attacks and the attack dimensions used to organize existing approaches.

  • Attack Taxonomy: The section frames MIAs through their definition, adversarial knowledge, attack approaches, and target models.These dimensions structure the survey’s subsequent taxonomy of attacks.

3.1 Definition of Membership Inference Attacks

A membership inference attack uses access to a learned model and an exact input to infer whether that input belongs to the model’s training dataset.

  • Definition: Given an exact input and access to a learned model, an attacker infers whether the input was included in the training dataset.The setup uses a classifier trained by a learning algorithm on a predefined dataset.

3.2 Adversarial Knowledge

Adversarial knowledge distinguishes white-box attacks, which access comprehensive model information, from black-box attacks, which rely on limited queries and outputs. Black-box attacks are especially concerning when they succeed with minimal information.

  • Adversarial knowledge: Attackers use knowledge of the training-data distribution and target-model details to implement membership inference attacks.Target-model knowledge includes its learning algorithm, architecture, and learned parameters.
  • White-box and black-box attacks: White-box attacks access the target model’s training-data distribution, training process, architecture, and learned parameters.White-box access provides all listed model information.
  • White-box and black-box attacks: Black-box attacks receive the training-data distribution and query outputs, such as a classifier’s prediction output for an input record.Unlike white-box attackers, they cannot inspect internal model information.
  • White-box and black-box attacks: Black-box attacks can be more dangerous than white-box attacks if they breach membership privacy using limited knowledge.The survey identifies label-only attacks as especially dangerous, while noting that most work studies full confidence scores.

3.3 Membership Inference Attack Approaches

Membership inference approaches exploit behavioral differences between training members and non-members. They include learned binary classifiers, threshold-based metrics, and richer white-box features.

  • Motivation: Overparameterization, finite datasets, and repeated training can make ML models behave differently on members and non-members.These behavioral differences provide the basis for membership inference.
  • Binary classifier-based attacks: Shadow training creates shadow models that mimic the target model and labels their outputs from shadow training and test data as members or non-members.The resulting data trains a binary attack classifier.
  • Binary classifier-based attacks: The same shadow-training procedure supports black-box and white-box attack models, but white-box attacks additionally collect hidden-layer computations.Black-box attacks collect prediction vectors, whereas white-box attacks collect prediction vectors and intermediate computations.
  • Binary classifier-based attacks: Black-box binary classifiers use prediction vectors as input, whereas white-box classifiers use a flat vector containing more record information.Both classifiers output whether the record belongs to the target training dataset.
  • Metric-based attacks: Metric-based attacks calculate prediction metrics and compare them with preset thresholds to infer membership.Examples include correctness, prediction loss, confidence, and entropy.

3.4 Membership Inference Attacks on Different ML models

Membership inference attacks extend across classification, generative, regression, and other ML models, adapting their signals to each model’s available outputs and internal behavior.

  • Classification models: Label-only attacks study membership inference when the target model provides only predicted labels rather than full confidence scores.Proposed approaches include transfer-based and perturbation-based attacks.
  • Classification models: White-box attacks on classification models improve access to membership signals by using internal parameters and intermediate computations.These signals can include gradients of prediction loss and layer-specific computations.
  • Generative models: Generative-model attacks estimate membership from discriminator confidence, generated-record proximity, reconstruction error, or related signals.These attacks address the absence of classification-style labels or confidence scores tied directly to the target record.
  • Generative models: Chen et al. propose a generative-model attack applicable from full black-box to full white-box settings.The attack reconstructs a nearby synthetic record and uses calibrated reconstruction error with a threshold.
  • Regression models: Deep regression-model attacks can use gradients, activations, predictions, and labels of target records in a binary classifier.The cited work assumes white-box access and some training members.

3.5 Membership Inference Attacks against Federated Learning

Federated learning creates membership-inference settings involving a global model trained from distributed local data. Attacks may passively observe protocol information or actively manipulate training.

  • Federated learning: Federated learning trains a shared model through communication between a server and parties holding local datasets.FedAvg distributes the central model, locally optimizes it, and averages returned models.
  • Federated-learning attacks: Federated-learning attackers may be the central server or parties seeking to infer whether a record contributed to the global model.Existing attacks include text-classification, heterogeneous-party, and gradient-based settings.
  • Federated-learning attacks: Passive attacks exploit observable gradients or differing party decision boundaries to infer membership.Non-zero word-embedding gradients can reveal words occurring in training data.
  • Federated-learning attacks: The gradient ascent attack actively updates local parameters to increase loss on a target record and improve attack performance.This approach tampers with federated training rather than merely observing the protocol.
  • Scope and source privacy: Federated-learning membership inference identifies inclusion in the global training dataset, not inclusion in a particular party’s local model.Source inference separately targets which party owns a training record.

3.6 Taxonomies of Membership Inference Attacks

The survey organizes membership inference attack research by target models, attacker knowledge, attack approaches, training paradigms, and application domains. It supplements this taxonomy with comparative characteristics of representative studies.

  • Taxonomy dimensions: The taxonomy categorizes MIA papers by target model, adversarial knowledge, attack approach, training paradigm, and domain.Target domains include NLP, computer vision, graph, audio, and recommender systems.
  • Taxonomy dimensions: Figure 8 provides detailed characteristics for papers within its categories, including specific metric-based attack approaches.
  • Representative studies: Table 2 lists representative MIA studies selected because they propose new attacks or first examine membership risks for particular ML model types.
  • Representative studies: For each selected study, Table 2 reports publication details, target task, attacker knowledge, attack method, baseline, evaluation metrics, and datasets.These fields support understanding and comparison across studies.

4 WHY MEMBERSHIP INFERENCE ATTACKS WORK

The survey explains MIA success mainly through empirical evidence about target-model overfitting, model type, and training-data diversity. It also notes that rigorous theoretical explanations remain at an early stage.

  • Factors enabling MIAs: Overfitting is identified as the main factor contributing to successful membership inference attacks.Overfitting occurs when training performance is much better than test performance, indicating poor generalization.
  • Theoretical analysis: Rigorous theoretical analysis of why MIAs work remains in its infancy because both training data and target models are highly complex.
  • Theoretical analysis: Theorem 4.1 relates an attack’s expected success rate to the model’s generalization gap and the ratio of training data to the input domain.
  • Factors enabling MIAs: Target-model type affects MIA success because models differ in how strongly individual records can change their decision boundaries.
  • Conclusion: The survey concludes that MIA success is directly related to overfitting, target-model type, and training-data diversity.

5 MEMBERSHIP INFERENCE DEFENSE ON MACHINE LEARNING MODELS

The survey organizes membership inference defenses into four categories and examines their effectiveness, applicability, and privacy–utility tradeoffs. Confidence masking, regularization, knowledge distillation, and differential privacy provide different mitigation mechanisms and limitations.

  • Existing defenses fall into four categories: confidence score masking, regularization, knowledge distillation, and differential privacy.
  • Confidence Score Masking: Confidence score masking modifies prediction vectors without retraining target classifiers, preserving model accuracy but leaving some label-only and metric-based attacks effective.Its methods include top-k scores, label-only outputs, and crafted noise added to prediction vectors.
  • Regularization: Regularization reduces overfitting and can defend against black-box and white-box attacks by changing both model outputs and internal parameters.Large regularization can reduce MIA accuracy to random-guessing levels, but may significantly reduce prediction accuracy.
  • Knowledge Distillation: Knowledge distillation trains a smaller student model from a larger teacher and supports the Distillation For Membership Privacy defense using private and unlabeled reference data.
  • Differential Privacy: Differential privacy offers theoretical membership protection for classification and generative models across black-box and white-box settings, but often creates unacceptable utility–privacy tradeoffs.Small privacy budgets can reduce attacks to random guessing while degrading model or generation quality; DP training can also increase computation cost.
  • Taxonomies of Membership Inference Defenses: The defense taxonomy further divides papers by specific approaches, helping readers locate relevant work and compare representative methods.The survey presents this organization in Fig. 9 and supplements it with selected papers in Table 3.

6 METRICS, DATASETS, AND OPEN-SOURCE IMPLEMENTATIONS

The survey organizes evaluation resources for membership inference attacks and defenses by reviewing metrics, benchmark datasets, and open-source implementations. It covers attack-specific measures alongside target-model metrics and catalogs commonly used datasets.

  • The survey summarizes metrics for evaluating attack and defense performance, datasets used in prior work, and open-source implementations of popular methods.
  • Evaluation metrics: Target-model evaluation includes Accuracy for classification and Generalization Error, defined as the absolute difference between training and test accuracy.
  • Evaluation metrics: Attack evaluation uses Attack Recall, Attack False Positive Rate, Membership Advantage, Attack F1-score, and Attack AUC.
  • Evaluation metrics: Membership Advantage is the difference between Attack Recall and attack False Positive Rate.
  • Datasets: Adult, Foursquare, Purchase-100, Texas-100, MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and LFW are widely used classification or generation benchmarks.

7 FUTURE DIRECTIONS

The survey identifies unresolved challenges for MIAs and defenses, including overfitting assumptions, emerging learning paradigms, federated learning heterogeneity, and privacy–utility trade-offs. It also highlights applications such as source inference, adversarial-example design, and auditing training-data use.

  • Membership Inference Attacks: Most existing MIAs rely on heavily overfitted target models, but their feasibility on non-overfitted models remains unknown and difficult to assess.
  • Membership Inference Attacks: MIAs on self-supervised learning models have not yet been explored, despite their use of large unlabeled datasets that may contain private information.
  • Membership Inference Attacks: Contrastive and meta-learning models pose distinct MIA questions because their training paradigms differ from conventional supervised learning.
  • Membership Inference Attacks: Existing MIAs for federated learning are limited to homogeneous settings, where local parties are assumed to share the same model architecture.
  • Applications: MIA information can support source inference, adversarial-example design, and audits of whether data contributed to model training.
  • Membership Inference Defenses: Overfitting contributes to MIA success and can help evaluate defenses, but detecting overfitting remains challenging for unsupervised models.
  • Membership Inference Defenses: Strong membership privacy guarantees often impose substantial utility loss, making acceptable privacy–utility trade-offs difficult to achieve.

8 CONCLUSION

The survey consolidates research on membership inference attacks and defenses into a structured reference. It covers definitions, approaches, taxonomies, evaluation resources, challenges, and future research opportunities.

  • The survey covers released work on membership inference attacks and defenses, including definitions, approaches, taxonomies, metrics, datasets, and implementations.
  • It discusses why MIAs work, introduces defense approaches, and identifies challenges and potential research opportunities.
Loading 2103.07853v4…