Systematic Evaluation of Privacy Risks of Machine Learning Models

Liwei Song; Prateek Mittal

Source-linked AI summary

Systematic Evaluation of Privacy Risks of Machine Learning Models

Liwei Song, Prateek Mittal

arXiv:2003.10595v2Mar 24, 2020cs.CRcs.LGstat.ML

TL;DR

Machine-learning privacy evaluations may underestimate membership-inference risks when they rely on neural-network attacks and aggregate metrics. The paper introduces metric-based benchmarks and a per-sample privacy risk score, finding weaker-than-reported defenses and heterogeneous individual risks. It also identifies a black-box evaluation scope and a limitation in conventional prediction entropy.

Problem
Prior work may underestimate membership-inference privacy risks by relying on custom neural-network attacks and aggregate evaluation despite heterogeneous sample-level risks.
Method
The paper develops non-neural-network metric-based attacks, adaptive defense benchmarks, and a privacy risk score for fine-grained black-box analysis of individual samples.
Results
Existing defenses are less effective than previously reported, while privacy risk scores estimate training-membership likelihood and reveal heterogeneous risks across individual samples.
Takeaways & Limitations
Systematic evaluation should combine benchmark attacks and per-sample privacy-risk analysis to understand both aggregate and individual membership-inference risks.
Takeaways & Limitations
The privacy risk score is computed in the black-box scenario; discussion of the white-box scenario is left for future work.

Abstract

from arXiv · show

Machine learning models are prone to memorizing sensitive data, making them vulnerable to membership inference attacks in which an adversary aims to guess if an input sample was used to train the model. In this paper, we show that prior work on membership inference attacks may severely underestimate the privacy risks by relying solely on training custom neural network classifiers to perform attacks and focusing only on the aggregate results over data samples, such as the attack accuracy. To overcome these limitations, we first propose to benchmark membership inference privacy risks by improving existing non-neural network based inference attacks and proposing a new inference attack method based on a modification of prediction entropy. We also propose benchmarks for defense mechanisms by accounting for adaptive adversaries with knowledge of the defense and also accounting for the trade-off between model accuracy and privacy risks. Using our benchmark attacks, we demonstrate that existing defense approaches are not as effective as previously reported. Next, we introduce a new approach for fine-grained privacy analysis by formulating and deriving a new metric called the privacy risk score. Our privacy risk score metric measures an individual sample's likelihood of being a training member, which allows an adversary to identify samples with high privacy risks and perform attacks with high confidence. We experimentally validate the effectiveness of the privacy risk score and demonstrate that the distribution of privacy risk score across individual samples is heterogeneous. Finally, we perform an in-depth investigation for understanding why certain samples have high privacy risks, including correlations with model sensitivity, generalization error, and feature embeddings. Our work emphasizes the importance of a systematic and rigorous evaluation of privacy risks of machine learning models.

1 Introduction

The paper argues that conventional membership-inference evaluations can underestimate privacy risks and introduces stronger benchmark attacks plus fine-grained per-sample analysis. Its privacy risk score reveals heterogeneous individual risks and supports high-confidence attacks, while experiments show existing defenses are less effective than reported.

Motivation: Membership inference can expose sensitive information by revealing whether an individual contributed data to model training.The paper motivates membership inference as a way to quantify privacy risks associated with training-data participation.
Limitations of prior evaluation: Prior evaluations may underestimate privacy risks by relying on custom neural-network attack classifiers and aggregate metrics such as overall attack accuracy.The paper identifies neural-network hyperparameter choices and sample-level heterogeneity as limitations of prevailing evaluations.
Benchmark methodology: The paper benchmarks risks with non-neural-network metric-based attacks, class-specific thresholds, modified prediction entropy, adaptive attackers, and accuracy–privacy trade-off baselines.These attacks use prediction metrics and thresholding, avoiding attack-classifier hyperparameter tuning while strengthening defense evaluation.
Defense evaluation: 58.6%–74.2% attack accuracy remains possible against adversarial regularization and MemGuard, rather than the approximately 50% previously reported.The benchmark therefore finds these defenses less effective than earlier evaluations suggested.
Fine-grained privacy analysis: The privacy risk score estimates an individual sample’s probability of being a training member and exposes heterogeneous risks across samples.An adversary can classify a sample as a member when its score exceeds a threshold, enabling high-confidence individual attacks.
Further analysis: The paper relates privacy risk scores to model sensitivity, generalization error, and feature embeddings, and provides publicly available code integrated into TensorFlow Privacy.These analyses support systematic and reproducible evaluation of model privacy risks.

2 Background and Related Work

This section introduces membership inference against machine learning models, surveys black-box and white-box attacks, and describes defenses including adversarial regularization and MemGuard. It motivates more rigorous evaluation because neural-network attack evaluations can underestimate privacy risks.

Membership inference: Membership inference attacks determine whether a data point was used to train a target model, creating privacy risks when membership reveals sensitive information.The health-analytics setting illustrates how training-set membership can disclose that an individual was a patient.
Attack settings: Black-box attacks observe target-model predictions, whereas white-box attacks can additionally use model architecture, parameters, and gradients.Prior work found that gradient features can improve white-box attacks, while combining final predictions and intermediate computations alone did not outperform black-box attacks.
Attack methods: Non-neural attacks infer membership from prediction correctness, confidence, or entropy, but many evaluations still rely only on trained neural-network attack classifiers.The paper argues that this reliance can severely underestimate privacy risks and improves non-neural attacks with class-specific thresholds.
Defenses: Adversarial regularization jointly trains a target classifier and an attack classifier so the target minimizes prediction loss while misleading membership inference.The attack classifier distinguishes training from validation data, and the target model adds membership-inference resistance as a regularization objective.
Defenses: MemGuard perturbs predictions from a pretrained target model without changing classification results, while constraining noisy outputs to confuse the membership attack classifier.Its optimization minimizes distance from the original predictions while enforcing unchanged classification, attack confusion, and valid prediction outputs.
Evaluation limitations: Re-evaluation with benchmark attacks finds significantly higher attack accuracy on defended models than prior estimates.The paper also reports that adversarial regularization is no better than early stopping and that MemGuard evaluation lacks adaptive adversaries.

3 Systematically Evaluating Membership Inference Privacy Risks

The paper evaluates membership inference privacy using black-box, non-neural attacks, recommends adaptive-defense-aware comparisons with early stopping, and re-evaluates prior target models.

Benchmarking approach: The evaluation uses non-neural attacks that observe only target-model output predictions in a black-box setting.These attacks benchmark privacy risks without requiring access to model internals.
Defense evaluation: Defense evaluation should compare against early stopping and consider adaptive attackers who know the defense mechanism.The section applies these recommendations while re-evaluating models from prior work.

3.1 Benchmarks of membership inference attacks

The paper benchmarks membership inference with metric-based attacks using prediction correctness, confidence, and entropy, improving thresholding and introducing modified prediction entropy that incorporates the true label.

Metric-based attacks: Metric-based attacks compare prediction correctness, confidence, or entropy with thresholds to infer whether an input is a member.The attacks use target-model prediction metrics rather than trained neural-network classifiers.
Prediction confidence: Class-specific confidence thresholds are learned with shadow training to account for different confidence levels across class labels.The threshold τ_y is selected using shadow training data and membership labels for each class.
Prediction entropy: Prediction entropy alone omits the ground-truth label, assigning zero entropy to both perfectly correct and perfectly wrong predictions.This makes conventional entropy unable to distinguish these two extreme outcomes for membership inference.
Modified prediction entropy: Modified prediction entropy decreases with the correct-label probability and increases with each incorrect-label probability.It replaces non-monotonic conventional entropy components with functions having the desired monotonic behavior.
Modified prediction entropy: A correct prediction with probability 1 yields modified entropy 0, while a wrong prediction with probability 1 yields modified entropy infinity.The resulting metric separates the two cases that conventional prediction entropy treats identically.
Inference rule: The modified-entropy attack classifies an input as a member when its metric is below a preset class-specific threshold.Experiments show this attack strictly outperforms the prediction-entropy attack.

3.2 Rigorously evaluating membership inference defenses

The paper recommends evaluating membership inference defenses against adaptive adversaries and against early stopping when defenses trade model accuracy for privacy. These benchmarks expose whether a defense provides benefits beyond reduced training duration.

Early stopping: Early stopping is a benchmark defense because increasing training epochs can reduce error while increasing vulnerability through memorization.The benchmark uses fewer training epochs to assess the accuracy–privacy trade-off.
Accuracy–privacy trade-off: Figure 2 compares Purchase100 test accuracy across training epochs for undefended and adversarially regularized classifiers.The final defended model should be compared with an undefended model stopped earlier at similar accuracy.
Accuracy–privacy trade-off: Defenses that reduce membership inference risk by lowering model accuracy should be compared with naturally trained models using early stopping.The early-stopping comparison controls for accuracy degradation when assessing privacy protection.
Adaptive adversaries: Defense evaluations should place adaptive adversaries last, giving them knowledge of the defense mechanism and allowing defended shadow models.Non-adaptive attacks can make an ineffective defense appear successful.

3.3 Experiment results

Experiments re-evaluate membership inference defenses and attack variants across several datasets. Metric-based benchmark attacks reveal substantial residual vulnerability, while class-dependent thresholds improve attacks and white-box methods usually add little advantage.

Experimental setup: Experiments re-evaluate defenses and white-box attacks on Purchase100, Texas100, Location30, and CIFAR100 classifiers.Membership status is sampled equally from training and test sets, making random guessing 50% attack accuracy.
Adversarial regularization: Adversarial regularization reduces test accuracy from 80.9% to 76.6% on Purchase100 and from 52.3% to 46.4% on Texas100.The reported models use fully connected networks with four hidden layers.
Adversarial regularization: 59.5% and 58.6% attack accuracy on defended Purchase100 and Texas100 classifiers exceed the 51.6% and 51.0% results reported by Nasr et al.Benchmark attacks achieve higher attack accuracy than the neural-network attacks on all models except undefended Purchase100.
Adversarial regularization: 59.2% and 59.5% attack accuracy under early stopping is close to defended Purchase100 and Texas100 models.The paper recommends calibrating early stopping to similar model accuracy for a fair comparison.
Thresholding: Class-dependent thresholding increases attack accuracy by 1%–4% over class-independent thresholds on Texas100 classifiers.The comparison covers undefended, adversarially regularized, and early-stopped models.
MemGuard: MemGuard-defended models retain 69.1% attack accuracy on Location30 and 74.2% on Texas100, versus 50.1% and 50.3% reported by Jia et al.Benchmark attacks therefore identify substantial residual vulnerability despite limited mitigation.
MemGuard: MemGuard reduces attack accuracy by 12% on Location30 and 5% on Texas100, while modified-entropy attacks outperform conventional entropy attacks.MemGuard does not change accuracy, so an early-stopping comparison is not applicable.
White-box attacks: White-box attacks improve over black-box benchmark attacks by 0.5% on Texas100, 0.6% on CIFAR100, and 5.8% on Purchase100.The paper characterizes the white-box advantage as limited overall.

4 Fine-Grained Analysis on Privacy Risks

The paper defines privacy risk at the individual-sample level and validates its use for identifying heterogeneous membership risks. It also relates high-risk samples to model sensitivity, generalization error, and feature embeddings.

Aggregate attack accuracy can miss heterogeneous privacy risks, motivating fine-grained analysis that identifies samples with high membership risk.
4.1 Definition of privacy risk score: The privacy risk score is an individual sample’s posterior probability of belonging to the target model’s training set after observing model behavior.
4.1 Definition of privacy risk score: The score depends on training/test membership priors and conditional prediction distributions, which are estimated using shadow-model predictions.
4.2.1 Validation of privacy risk score: The privacy risk score closely aligns with the actual probability of membership across undefended, adversarially regularized, and early-stopped Purchase100 classifiers.
4.2.2 Distribution of privacy risk score: 70% of training samples in defended Texas100 classifiers have privacy risk scores above 0.6, showing substantial residual risk despite defense.
4.2.3 Usage of privacy risk score: Thresholding the score enables high-confidence attacks: 70.5% of defended Texas100 members are inferred at 71.3% precision, while 1.4% reach 88.2% precision.
4.2.4 Impact of model properties on privacy risk score: Samples with higher privacy risk scores are likely to have larger influence on the target model, and high-error classes tend to have higher scores.
4.2.4 Impact of model properties on privacy risk score: High-risk classes exhibit test-sample feature embeddings that differ from corresponding training samples, unlike low-risk classes.

5 Conclusions

The paper concludes that neural-network-only and aggregate evaluations are insufficient for membership-inference privacy risks. It recommends metric-based benchmarks, adaptive-defense evaluation, and fine-grained privacy-risk analysis.

Neural-network-based attacks alone are insufficient for measuring membership-inference privacy risks.
Metric-based attacks, class-specific thresholds, and modified prediction entropy provide benchmark alternatives without attack-classifier hyperparameter tuning.
Adversarial regularization reduces privacy risks only to a limited degree and is no better than early stopping, while MemGuard degrades substantially against adaptive attacks.
The privacy risk score estimates individual membership likelihood, reveals heterogeneous risks, and correlates with sensitivity, generalization error, and feature embeddings.
The paper emphasizes systematic and rigorous evaluation of machine-learning privacy risks.

A Membership inference attacks against other datasets

The paper evaluates membership-inference attacks on additional image datasets and compares its benchmark attacks with neural-network-based attacks.

The additional evaluation uses CH-MNIST histology tiles from colorectal-cancer patients and the Car196 image dataset.
Table 7 reports membership-inference attacks against image datasets.
The best benchmark attacks achieve accuracy 2.1% and 0.6% larger than neural-network-based attacks on the evaluated image datasets.

B Privacy risk score with different training/test selection probabilities

The privacy risk score incorporates the prior probability that a sample comes from the training set. Varying this prior changes the uncertainty and resulting score distribution.

Figure 10 shows undefended Purchase100 training-sample privacy risk scores under varied training-set prior probabilities, alongside the prior-probability baseline.
The score computation uses P(z ∈Dtr) and P(z ∈Dte)=1−P(z ∈Dtr) to represent different training/test selection probabilities.
As P(z ∈Dtr) approaches 0.5, membership uncertainty increases and leads to a larger distance value.

C Validation of privacy risk score on Texas100 classifiers

On Texas100 classifiers, the privacy risk score more accurately indicates the real probability that a sample was used for training than neural-network attack outputs, for both undefended and defended models.

Neural-network attack classifier outputs have higher RMSE values of 0.13 and 0.21 on the corresponding undefended and defended Texas100 classifiers.
The privacy risk score has RMSE values of 0.08 and 0.05 for undefended and defended Texas100 classifiers, respectively.These values measure its deviation from the real probability of being a member.
The privacy risk score is more meaningful than neural-network attack outputs for indicating the real probability of being a member.

D Validation of privacy risk scores on different model architectures

Across varied architectures, the privacy risk score remains close to the probability that samples were training members on defended Purchase100 and Texas100 classifiers.

The RMSE between privacy risk score and membership probability is below 0.10 for every plotted architecture across defended Purchase100 and Texas100 classifiers.Architectures are identified by activation function, width, and depth in the figure legend.
For all varied model architectures, the privacy risk score is close to the probability of being a member.