Source-linked AI summary
Revisiting Membership Inference Under Realistic Assumptions
Bargav Jayaraman, Lingxiao Wang, Katherine Knipmeyer, Quanquan Gu, David Evans
TL;DR
The paper asks how membership inference should be evaluated when candidate membership is rare and attackers choose thresholds for specific goals. It develops PPV-based analysis, adaptive threshold selection, and local-minimum-based attacks, finding high-confidence vulnerabilities in non-private and high-budget private models.
Problem
Membership-inference research needs metrics that remain informative when candidate membership priors are skewed rather than balanced.
Method
The paper combines hypothesis-testing analysis with PPV, goal-specific threshold selection, and Merlin and Morgan attacks based on local loss behavior.
Results
Across four data sets, non-private models are highly vulnerable, Morgan achieves higher PPV than Merlin, and private models can remain vulnerable at high privacy-loss budgets.
Takeaways & Limitations
PPV and the proposed attacks expose high-confidence membership risks that conventional balanced-prior evaluations can miss.
Takeaways & Limitations
Membership advantage alone is limited to balanced priors, and the experiments leave non-differentially-private defenses for future work.
Abstract
from arXiv · showhide
We study membership inference in settings where some of the assumptions typically used in previous research are relaxed. First, we consider skewed priors, to cover cases such as when only a small fraction of the candidate pool targeted by the adversary are actually members and develop a PPV-based metric suitable for this setting. This setting is more realistic than the balanced prior setting typically considered by researchers. Second, we consider adversaries that select inference thresholds according to their attack goals and develop a threshold selection procedure that improves inference attacks. Since previous inference attacks fail in imbalanced prior setting, we develop a new inference attack based on the intuition that inputs corresponding to training set members will be near a local minimum in the loss function, and show that an attack that combines this with thresholds on the per-instance loss can achieve high PPV even in settings where other attacks appear to be ineffective. Code for our experiments can be found here: https://github.com/bargavj/EvaluatingDPML.
1 Introduction
The paper revisits membership inference under skewed priors and goal-specific thresholds, proposing PPV-based evaluation and new attacks for realistic privacy-risk assessment. It finds vulnerabilities in non-private models and, at sufficiently high privacy-loss budgets, in private models as well.
- Differential privacy bounds worst-case privacy loss, but practical inference attacks provide empirical evidence about leakage in specific settings.
- The paper uses hypothesis testing to tighten the advantage bound and introduces PPV because advantage alone ignores the prior probability of candidate membership.
- A threshold-selection procedure adapts inference attacks to an adversary’s specific goal, while Merlin and Morgan exploit local loss behavior around candidate inputs.
- Non-private models are vulnerable to high-confidence membership inference in both balanced and imbalanced prior settings.
- PPV changes with the prior and is therefore more reliable than advantage in imbalanced prior settings.
- The Morgan attack achieves higher PPV than Merlin, which already outperforms previous attacks.
- Private models remain vulnerable to the proposed attacks when privacy-loss budgets are well above theoretical guarantees.
2 Related Work
Prior membership-inference work largely evaluates balanced priors and conventional metrics, limiting its applicability to skewed candidate pools. Related theoretical work connects inference metrics to differential-privacy parameters but does not provide the same empirical coverage.
- Shokri et al. exploit model confidence reflecting overfitting and evaluate membership inference in the balanced-prior setting with accuracy.
- Yeom’s membership advantage is useful for balanced priors but is not representative of true privacy leakage in realistic scenarios.
- Liu et al. connect precision, recall, and F-score to differential-privacy parameters using a hypothesis-testing framework, while related relaxed-privacy work also focuses on theoretical analysis.
3 Differential Privacy
This section frames differential privacy through hypothesis testing and trade-off functions, then relates f-DP and Gaussian DP to common mechanisms, composition, and subsampling.
- Differential privacy bounds output-probability ratios for neighboring datasets, making them harder for an adversary to distinguish.
- The adversary tests whether the released output came from S or S′, balancing type I and type II errors through a rejection rule.
- The trade-off function is the minimum type II error achievable at each type I error, and larger values indicate a harder hypothesis-testing problem.
- f-Differential Privacy: f-DP uses this trade-off framework to derive theoretical upper bounds on privacy-leakage metrics.
- f-Differential Privacy: For (ϵ, δ)-DP, higher trade-off values correspond to more privacy, while increasing ϵ or δ decreases the trade-off function.
- Gaussian Differential Privacy: Gaussian DP models privacy through Gaussian trade-off functions and supports noisy SGD or Adam, composition, and privacy amplification by subsampling.
- Gaussian Differential Privacy: The framework combines composition and subsampling results to characterize privacy loss for gradient-perturbation algorithms and convert GDP guarantees into (ϵ, δ)-DP.
4 Measuring Privacy Leakage
The paper models membership inference with prior membership probability p and develops privacy-leakage metrics that account for false-positive constraints and skewed populations. It tightens membership-advantage bounds and introduces PPV bounds that incorporate the non-member-to-member ratio.
- Membership experiment: The membership experiment samples a member with prior probability p and otherwise samples from the data distribution, explicitly relaxing the balanced-prior assumption.The adversary receives a model, training-set size, data distribution, and prior membership probability.
- Membership experiment: Skewed priors matter because practical membership-inference settings may have far fewer members than non-members, making membership disclosure the salient outcome.The paper notes that p is often much smaller than 0.5 when exposure as a member carries meaningful risk.
- Membership advantage: Membership advantage is limited to balanced priors and can misestimate privacy leakage when the prior membership probability differs from 0.5.The paper therefore treats membership advantage alone as unreliable for non-balanced settings.
- Membership advantage: Theorem 4.1 derives a tighter membership-advantage bound for (ϵ, δ)-differentially private algorithms at a fixed false-positive rate α.The bound is based on the f-DP trade-off function and is described as tighter than the earlier bound of Yeom et al.
- Positive predictive value: PPV measures the fraction of positive predictions that are true members and its bound incorporates γ, the non-member-to-member probability ratio.The paper gives PPV_A(α, γ) = TPR / (TPR + γ · FPR) and states that this captures privacy threat across different prior settings.
5 Inference Attacks
The paper improves threshold-based membership inference by selecting thresholds for an explicit false-positive objective and introduces attacks based on local loss behavior. Merlin uses perturbation-induced loss changes, while Morgan combines Merlin with per-instance loss thresholds to improve PPV.
- Threshold selection: Inference effectiveness depends critically on the decision threshold that converts a real-valued test measure into a Boolean membership prediction.The threshold determines how a threshold-based attack balances its inference outcomes.
- Threshold selection: Procedure 5.1 selects the threshold that maximizes true-positive rate subject to a specified maximum false-positive rate α.It assumes the adversary knows the data distribution, training procedure, model architecture, training-set size, and prior membership probability.
- Merlin: Merlin perturbs each query record repeatedly and classifies it using the fraction of perturbations whose per-instance loss increases.The reported experiments use T = 100 and σ = 0.01, with membership predicted when count/T ≥ φ.
- Merlin: Merlin is motivated by the expectation that a member lies near a local loss minimum, whereas loss changes for a non-member are equally likely to increase or decrease.The attack uses the direction of loss change under small perturbations rather than only the original per-instance loss.
- Morgan: Morgan combines Yeom’s per-instance-loss signal with Merlin’s local-minimum signal using lower and upper loss thresholds plus a Merlin-ratio threshold.The lower loss threshold is selected by grid search to maximize PPV jointly with the other thresholds.
6 Experimental Setup
The experiments evaluate membership inference on four multiclass data sets using non-private and differentially private neural networks. They vary privacy budgets, prior settings, and attack-oriented metrics while comparing privacy analyses and accuracy loss.
- Evaluation: The evaluation includes balanced and imbalanced membership priors, with attack performance summarized using PPV and related leakage metrics.
- Data Sets: Experiments use Purchase-100X, Texas-100, RCV1X, and CIFAR-100, all treated as multiclass classification tasks.
- Model Architecture: The models are two-hidden-layer ReLU networks with 256 neurons per hidden layer and a softmax output layer.
- Private Model Training: Private training varies ϵ from 0.1 to 100, with each setting repeated five times to report average results.
- Evaluation: Accuracy loss measures the relative test-accuracy loss of a private model compared with its non-private baseline.
7 Empirical Results
The experiments show that goal-specific threshold selection improves inference attacks, while Merlin and Morgan outperform earlier attacks in identifying members with high confidence. Non-private models remain vulnerable under skewed priors, whereas differential privacy reduces leakage but does not eliminate it at high privacy loss budgets.
- Threshold Selection: Threshold selection lets attackers target specific goals, including maximizing PPV or membership advantage and minimizing false positive rate.
- Yeom Attack: On Purchase-100X, member losses cluster below 0.001, while non-member losses are more dispersed, enabling high-precision loss-threshold attacks.
- Merlin Attack: Merlin achieves higher PPV than Yeom and Shokri across data sets and can reach a false positive rate as low as 0.01% on Purchase-100X.
- Threshold Selection: Class-based thresholds do not consistently improve attacks; for Merlin, they raise advantage from 0.1% to 2.8% but reduce maximum PPV from around 93.4% to 83.1%.
- Morgan Attack: Morgan achieves higher PPV than Yeom, Shokri, and Merlin by combining per-instance loss thresholds with neighborhood-based evidence.
8 Conclusion
The paper addresses the gap between formal privacy guarantees and practical membership inference by evaluating imbalanced priors and attack-specific thresholds. It finds that Merlin and Morgan can identify members with high confidence when most candidates are non-members, and that non-private and high-budget private models remain vulnerable.
- The study evaluates membership inference beyond balanced priors, focusing on candidate pools in which most records are non-members.
- The authors introduce attack-goal-specific threshold selection and two attacks, Merlin and Morgan, for high-confidence member identification.
- Across four data sets, non-private models are highly vulnerable, while models trained with high privacy loss budgets can still be vulnerable.
A Hyperparameters
The experiments vary the test-set size to represent different prior probabilities while keeping each training set fixed at 10,000 records.
- Prior Settings: Each data set uses 10,000 randomly sampled training records, while the test set contains γ times as many records to vary prior probabilities.
- Prior Settings: Purchase-100X and RCV1X use γ = {0.1, 0.5, 1, 2, 10}, whereas Texas-100 and CIFAR-100 omit γ = 10 because they are too small.
B Additional Results for Non-Private Models
Across Texas-100, RCV1X, and CIFAR-100, threshold selection and the Merlin and Morgan attacks reveal substantial membership-inference risk for non-private models. Results vary by dataset, with Merlin especially effective on RCV1X and class-based thresholds improving Yeom.
- Texas-100: For Texas-100, non-members can have a false-positive rate as low as 3%, and members have average Merlin ratio 0.81 ± 0.12 versus 0.65 ± 0.22 for non-members.The member and non-member Merlin-ratio distributions are more separated than on Purchase-100X.
- Texas-100: Merlin achieves higher PPV than Yeom and Shokri on Texas-100, while Yeom CBT reaches maximum PPV comparable to Merlin.Class-based thresholds substantially improve Yeom's PPV on this dataset.
- RCV1X: On RCV1X, Yeom's maximum PPV is around 58%, whereas Merlin reaches around 99% near a threshold of 0.97.Yeom CBT raises its maximum PPV from 58% to 93%, but Merlin still performs better.
- RCV1X: Morgan identifies the most vulnerable RCV1X members with 100% confidence despite limited separation between member and non-member distributions.This comparison uses the loss and Merlin-ratio scatter plot.
- CIFAR-100: On CIFAR-100, class-based thresholds increase Yeom's maximum PPV from 73% to 81%.Both member and non-member losses are high because the model does not completely overfit.
C Additional Results for Private Models
Private-model results across the tested datasets show that privacy noise reduces the separation between member and non-member distributions and limits attack success. Merlin can still achieve high PPV on RCV1X at ϵ = 10 and ϵ = 100, while Morgan poses serious risk at high privacy-loss budgets across all tested datasets.
- Cross-dataset results: Adding privacy noise reduces the gap between member and non-member distributions and limits attack success across the tested datasets.Private-model plots were omitted because they were similar to earlier results, so the comparison focuses on maximum PPV across privacy-loss budgets.
- Attack comparison: Merlin achieves high PPV on RCV1X for ϵ = 10 and ϵ = 100 but not on Texas-100 or CIFAR-100 even for ϵ = 100.Across all datasets, Merlin has higher average PPV than Yeom and Shokri.
- Attack comparison: Morgan poses serious privacy threat against models trained with high privacy loss budgets across all tested datasets.This conclusion is reported alongside the cross-dataset PPV comparison.
- Attack comparison: Yeom and Shokri remain close to 50% PPV across all datasets in the reported private-model comparison.The reported PPV level is described as failing to pose a privacy threat even with large privacy loss budgets.