PaperlayerView exact revision on arXiv

Source-linked AI summary

Membership Inference Attacks against Machine Learning Models

Reza Shokri, Marco Stronati, Congzheng Song, Vitaly Shmatikov

arXiv:1610.05820v2cs.CRcs.LGstat.ML

TL;DR

Machine-learning models may reveal whether specific records were included in their training data, creating a membership-privacy risk. The paper develops black-box shadow-model attacks, evaluates them across realistic tasks and commercial services, and finds substantial leakage, including for sensitive health data.

  • Problem

    The paper asks whether an adversary with a record and black-box model access can determine whether that record was used for training, because membership can reveal sensitive information.

    Source: §I · p. 1, B. Inference about members of the… · p. 3

  • Method

    The authors train attack classifiers on differences between target-model outputs for members and non-members, using shadow models with known training membership.

    Source: A. Overview of the attack · p. 4, §X · p. 15

  • Results

    Membership inference reaches median accuracy of 94% against Google and 74% against Amazon on retail datasets, exceeds 70% on the Texas hospital discharge dataset, and reaches 90% against Google with synthetic shadow data.

    Source: §I · p. 2

  • Takeaways & Limitations

    The demonstrated leakage makes membership inference a quantitative metric for evaluating model or machine-learning-service choices, while overfitting is associated with greater leakage.

    Source: G. Effect of overfitting · p. 11, §X · p. 15

Abstract

from arXiv · show

We quantitatively investigate how machine learning models leak information about the individual data records on which they were trained. We focus on the basic membership inference attack: given a data record and black-box access to a model, determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on. We empirically evaluate our inference techniques on classification models trained by commercial "machine learning as a service" providers such as Google and Amazon. Using realistic datasets and classification tasks, including a hospital discharge dataset whose membership is sensitive from the privacy perspective, we show that these models can be vulnerable to membership inference attacks. We then investigate the factors that influence this leakage and evaluate mitigation strategies.

I. INTRODUCTION

The paper studies whether black-box machine-learning models reveal membership in their training datasets, develops shadow-model attacks, and evaluates leakage, overfitting, and mitigations.

  • Membership inference asks whether a given record was used to train a model, using only black-box access in the paper’s setting.Source: §I · p. 1
  • The attack trains an inference classifier to distinguish target-model behavior on training records from behavior on unseen records.The approach converts membership inference into a classification problem.Source: §I · p. 1
  • Shadow models provide labeled member and non-member outputs for training the attack model when the target model’s parameters and training data are unknown.The shadow models imitate the target while exposing known membership ground truth.Source: §I · p. 1
  • The authors evaluate generic attacks on neural networks and commercial Amazon ML and Google Prediction API models without knowing the providers’ learning algorithms or architectures.Evaluation uses realistic classification tasks and standard training procedures.Source: §I · p. 1
  • The paper quantifies how attack success relates to classification tasks and overfitting, and compares prediction limiting, reduced precision, increased entropy, and regularization as mitigations.Source: §I · p. 2
  • 94% and 74% median attack accuracy are achieved against Google and Amazon models, respectively, on 10,000-record retail transaction datasets.Using fully synthetic shadow data still yields 90% accuracy against Google-trained models; the Texas hospital discharge dataset exceeds 70% accuracy.Source: §I · p. 2

III. PRIVACY IN MACHINE LEARNING

The paper distinguishes membership privacy from broader population-level privacy notions and focuses on information revealed about individuals whose records trained the model.

IV. PROBLEM STATEMENT

The problem is to infer whether a known labeled record belongs to a model’s training set from black-box outputs, using prediction confidence patterns to train a binary attack classifier.

  • For each input, the target model returns a probability vector with one value per class, whose largest value determines the predicted label.Source: §IV · p. 3
  • The attacker knows input-output formats and can query prediction vectors, while potentially possessing population samples or marginal feature statistics.Source: §IV · p. 3
  • The attacker receives a record and black-box query access, then determines whether that record was part of the target model’s training dataset.Precision and recall are the standard attack-accuracy metrics.Source: §IV · pp. 3–4
  • The attack exploits different model behavior on training records versus records encountered for the first time, often associated with overfitting but not limited to it.Source: A. Overview of the attack · p. 4
  • Separate attack models are used for each target output class because confidence distributions vary with the input’s true class.Source: A. Overview of the attack · p. 4
  • The attack model receives a correctly labeled record and target prediction vector, then performs binary classification into “in” and “out.”Source: A. Overview of the attack · p. 4
  • Shadow-model outputs with known membership labels enable supervised training of the attack model despite unknown target parameters.Source: A. Overview of the attack · p. 4

B. Shadow models

Shadow models approximate the target using similarly formatted data and the same platform, allowing supervised attack training; synthetic-data generation can operate without target-distribution knowledge.

B. Shadow models

  • The attacker creates multiple shadow models trained on datasets matching the target’s format and distribution as closely as possible.Source: B. Shadow models · p. 4
  • Shadow training datasets are disjoint from the target dataset, although overlap among shadow datasets is allowed and can improve attack performance.Source: B. Shadow models · p. 4
  • The synthesis algorithm queries the target model, accepts records meeting a class-confidence condition, samples accepted records, and randomizes features to continue searching.Source: B. Shadow models · p. 5
  • The shadow models must be trained similarly to the target, and the same commercial platform can be used when the target’s model type and structure are hidden.Source: B. Shadow models · pp. 4–5
  • More shadow models provide more training data for recognizing member-versus-non-member behavior and generally improve attack accuracy.Source: B. Shadow models · p. 5

C. Generating training data for shadow models

The attacker's shadow-model training data can be generated from population statistics, noisy related data, or the target model itself. Model-based synthesis searches for high-confidence inputs, but depends on efficiently exploring the input space.

D. Training the attack model

The attack model learns membership from labeled shadow-model outputs. Separate class-specific classifiers distinguish “in” from “out” behavior, while model-based synthesis prevents a trivial confidence-only rule.

  • Shadow training converts membership inference into binary classification using labeled outputs from models with known membership ground truth.Similar models trained on relatively similar data are assumed to behave similarly enough for this transfer.Source: D. Training the attack model · p. 6
  • Each shadow model is queried on its training set and a disjoint test set, labeling outputs “in” and “out,” respectively.Source: D. Training the attack model · p. 6
  • The attack dataset stores each record, its shadow-model output, and the corresponding membership label.Source: D. Training the attack model · p. 6
  • The attack dataset is partitioned by target output class, and a separate model predicts membership status for each class.Source: D. Training the attack model · p. 6
  • With model-based synthesis, both member and non-member records are high-confidence inputs, so the attack model must learn subtler output differences than confidence alone.Source: D. Training the attack model · p. 6

VI. EVALUATION

The evaluation spans image, demographic, purchase, location, hospital-stay, and handwritten-digit tasks using local neural networks and black-box commercial services. The experiments measure membership leakage with precision and recall across varied settings.

  • The evaluation covers CIFAR, Adult, purchase, location, Texas hospital-stay, and MNIST datasets with diverse classification tasks.Source: A. Data · p. 7
  • The Texas hospital-stay dataset contains 67,330 records and 6,170 binary features for predicting one of 100 frequent procedures.Source: A. Data · p. 7
  • The attacks treat local and cloud-trained target models as black boxes, without access to cloud model types, structures, or training hyperparameters.Source: B. Target models · p. 7
  • Amazon ML is evaluated in configurations (10, 1e −6) and (100, 1e −4), varying training passes and L2 regularization.Source: B. Target models · pp. 7–8
  • On CIFAR-10, median precision ranges from 0.78 to 0.71, while on CIFAR-100 it ranges from 1 to 0.97; recall is almost 1 for both.Source: Figure 4 · p. 8
  • For 10,000 purchase records, Google reaches 50-, 75-, and 90-percentile precision values of 0.94, 0.97, and 1, with recall close to 1.Source: Figure 5 · p. 8

C. Experimental setup

The experiments use disjoint target and shadow data, varied training sizes, multiple platforms, and class-balanced attack evaluation. Results show substantial leakage across datasets, with particularly high vulnerability for Google-trained models and overfit neural networks.

Experimental setup

  • Target and shadow training/test sets are randomly selected, equal-sized, disjoint, and non-overlapping across target and shadow models.Different shadow-model datasets may overlap with one another.Source: C. Experimental setup · p. 8
  • CIFAR training sizes vary across four settings, while other datasets use fixed target training sizes and cloud platforms with limited training-process visibility.Source: C. Experimental setup · pp. 8–9

Results

Experimental setup

  • The purchase dataset enables direct leakage comparison across Google, Amazon, and local neural-network models trained on the same data.Source: C. Experimental setup · p. 9
  • Using 100 CIFAR shadow models and fewer models elsewhere balances attack accuracy against computational cost.Source: C. Experimental setup · p. 9
  • Attack evaluation uses equal numbers of members and non-members, making random-guessing accuracy 0.5, with precision and recall reported mostly per class.Source: D. Accuracy of the attack · p. 9

Results

E. Effect of the shadow training data

Shadow-model training remains effective with noisy or synthetic data, though attack precision varies by data-generation method and underrepresented classes.

  • Precision drops as shadow-training noise increases, but 10% feature noise still matches the original attack and outperforms the baseline.The authors describe this as robustness to inaccurate assumptions about the target data distribution.Source: E. Effect of the shadow training data · pp. 9–10
  • 0.935 overall precision on real data exceeds 0.795 for marginal-based synthetics and 0.895 for model-based synthetics.Model-based synthetic data performs near real-data training for most classes but poorly for a few.Source: E. Effect of the shadow training data · p. 10
  • Marginal-based synthetic data reduces attack precision versus real data but remains very high for most classes.The experiment generated synthetic purchase records from individual-feature marginal distributions and trained shadow models on them.Source: E. Effect of the shadow training data · p. 10
  • Model-based synthetic shadow data produces precision below 0.1 for a few classes despite high precision for most classes.Those low-precision classes are underrepresented in the target training set, often contributing under 0.6% of records.Source: E. Effect of the shadow training data · p. 10
  • Low precision on rare classes reflects difficulty synthesizing representative records when the target classifier has seen too few examples.Some affected classes have fewer than 30 training records out of 10,000.Source: E. Effect of the shadow training data · p. 10
  • The attack can achieve high precision using only black-box access when the attacker can efficiently generate high-confidence inputs.This setting does not require prior knowledge of the target training-data distribution.Source: E. Effect of the shadow training data · p. 10

F. Effect of the number of classes and training data per class

Attack leakage depends on class count, per-class representation, overfitting, model structure, and member–non-member differences in prediction outputs.

Effect of the number of classes

  • More output classes generally increase leakage because they provide more signals about the model’s internal state.On the purchase dataset, models with fewer classes leaked less information about their training inputs.Source: F. Effect of the number of classes and… · pp. 10–11

Training data per class

Effect of overfitting

  • Greater overfitting increases leakage for models of the same type, but overfitting alone does not determine vulnerability.Model structure and type also contribute to different leakage levels.Source: G. Effect of overfitting · p. 11, §VII · p. 12

Why attacks work

  • Per-class train–test accuracy gaps quantify overfitting, while class prevalence measures how much training data belongs to each class.Figure 11 relates both quantities to per-class membership-inference precision.Source: G. Effect of overfitting · p. 11, Figure 11 · p. 12
  • Successful attacks exploit observable differences in target-model accuracy and prediction uncertainty between members and non-members.Figure 12 visualizes these output distributions for sample classes.Source: §VII · pp. 11–12, Figure 12 · p. 12
  • Membership-inference success is directly related to target-model generalizability and training-data diversity.The paper identifies information leakage as an additional harm of overfitting beyond reduced predictive power.Source: §VII · p. 12

VIII. MITIGATION

The paper examines how model design and training choices affect membership-information leakage and evaluates several mitigation strategies. Regularization is emphasized because it can reduce leakage while preserving or improving predictive performance, whereas some output restrictions may limit utility or remain insufficient.

  • Overfitting is an important, but not exclusive, reason that models leak information about their training datasets.Source: §VIII · p. 12
  • Well-regularized models should leak little training information, and the attack can quantify leakage from a specific model.Source: §VIII · p. 13
  • Differentially private models are secure against these membership attacks because the attacks use only model outputs without auxiliary information.Source: §VIII · p. 13
  • Services should consider training-data leakage alongside accuracy, warn customers about the risk, and expose ways to reduce it.Source: §VIII · p. 13
  • The study quantitatively evaluates several defenses against membership inference.Source: A. Mitigation strategies · p. 13
  • Top-k filtering, probability rounding, entropy increase, and L2 regularization are evaluated as defenses against membership inference.Source: A. Mitigation strategies · p. 13

B. Evaluation of mitigation strategies

The evaluation finds that membership inference remains effective under several prediction-output restrictions, while regularization is more promising but requires careful deployment. The related-work comparison distinguishes this attack from model inversion, extraction, and attacks based on explicit statistics.

B. Evaluation of mitigation strategies

IX. RELATED WORK

  • The paper’s attack differs from prior work by inferring membership of specific records through black-box model outputs rather than extracting parameters or explicit statistics.Source: §IX · pp. 14–15
  • Model inversion produces class-level averages rather than specific training records, especially when class images are diverse.Source: §IX · pp. 14–15, Figure 13 · p. 14
  • Prior KLR extraction work quantified leakage mainly through visual similarity on MNIST, where within-class similarity complicates membership interpretation.Source: §IX · p. 15

X. CONCLUSIONS

The paper presents a general quantitative membership-inference attack for black-box machine-learning models, including cloud services. Its results have practical privacy implications because membership in hospital-stay and other healthcare datasets is sensitive.

  • The attack is designed and evaluated for black-box models, including models trained through Google Prediction API and Amazon ML.Source: §X · p. 15
  • Shadow training distinguishes target-model outputs on training-set members from outputs on non-members.Source: §X · p. 15
  • Membership in hospital-stay and other healthcare datasets is privacy-sensitive, giving the findings substantial practical implications.Source: §X · p. 15

Related papers

Loading 1610.05820v2…