Source-linked AI summary
Towards Measuring Membership Privacy
Yunhui Long, Vincent Bindschaedler, Carl A. Gunter
TL;DR
Without differential privacy, membership privacy risk is difficult to assess, and differential privacy may be infeasible or reduce utility unacceptably. The paper proposes DTP as an empirical risk measure and finds it predicts membership-attack accuracy, motivating a DTP-based release rule of thumb.
Problem
When differential privacy is unavailable or costly in utility, there is little principled way to assess the membership privacy risk of releasing a classifier.
Source: §1 · p. 1, §9 · p. 12Method
The paper proposes DTP, an empirical metric that estimates membership risk from the influence of a training record on classifier predictions.
Source: §11 · p. 14Results
DTP below 0.5 corresponded to attack accuracy at most 66.5% (baseline: 50%), while attacks on classifiers with DTP above 4 almost always exceeded 90% accuracy.
Source: §11 · p. 14Takeaways & Limitations
The authors propose DTP-1 as a rule of thumb: a classifier with DTP above 1 should not be published.
Source: §1 · p. 2, §11 · p. 14Whether records with low DTP are always safe remains open; DTP-1 is presented as a rule of thumb for comparing record influence with other uncertain factors.
Source: §10 · pp. 13–14
Abstract
from arXiv · showhide
Machine learning models are increasingly made available to the masses through public query interfaces. Recent academic work has demonstrated that malicious users who can query such models are able to infer sensitive information about records within the training data. Differential privacy can thwart such attacks, but not all models can be readily trained to achieve this guarantee or to achieve it with acceptable utility loss. As a result, if a model is trained without differential privacy guarantee, little is known or can be said about the privacy risk of releasing it. In this work, we investigate and analyze membership attacks to understand why and how they succeed. Based on this understanding, we propose Differential Training Privacy (DTP), an empirical metric to estimate the privacy risk of publishing a classier when methods such as differential privacy cannot be applied. DTP is a measure of a classier with respect to its training dataset, and we show that calculating DTP is efficient in many practical cases. We empirically validate DTP using state-of-the-art machine learning models such as neural networks trained on real-world datasets. Our results show that DTP is highly predictive of the success of membership attacks and therefore reducing DTP also reduces the privacy risk. We advocate for DTP to be used as part of the decision-making process when considering publishing a classifier. To this end, we also suggest adopting the DTP-1 hypothesis: if a classifier has a DTP value above 1, it should not be published.
1 INTRODUCTION
Without differential privacy, membership privacy risk is difficult to assess, while applying differential privacy may be infeasible or impose unacceptable utility loss. The paper proposes DTP as an empirical risk measure and evaluates its relationship to membership attacks.
1 INTRODUCTION
- Differential privacy can thwart membership attacks, but some models cannot readily achieve it or would incur unacceptable utility loss.Without differential privacy, the authors identify a gap in principled ways to measure membership privacy risk.Source: §1 · p. 1
- The paper investigates why membership attacks succeed and develops a general attack framework, including experiments on state-of-the-art classifiers and real-world datasets.The stated goal is a privacy-risk metric that can be easily calculated on a classifier.Source: §1 · p. 1
- DTP is proposed to quantify a record’s membership-inference risk relative to a classifier and training set, supporting release decisions when differential privacy cannot be used.The authors suggest preferring the lower-DTP classifier when two classifiers have similar performance.Source: §1 · pp. 1–2
- For neural networks trained on the purchase dataset, DTP correlated with maximum membership-attack accuracy at Pearson r = 0.8936.Across classifiers with DTP below 0.5, tested attacks reached at most 66.5% accuracy against a 50% baseline; when DTP exceeded 4, they almost always exceeded 90%.Source: §1 · pp. 1–2
- For algorithms with training stability, direct attacks are always superior to indirect attacks, though the paper notes that indirect attacks may outperform direct ones in other cases.The authors establish training stability as a desideratum and show that naive Bayes, random decision trees, and linear statistical queries satisfy it, while k-NN does not.Source: §1 · p. 2
2 PROBLEM STATEMENT
The problem is to estimate membership privacy risk when a model is opened to public queries. The analysis assumes trusted training parties and a strong adversary with specified knowledge of the candidate dataset and training process.
2 PROBLEM STATEMENT
- The model is trained on a random partition and returns class probabilities for feature-vector queries, while the target record consists of features and a class label.The record’s membership in the training set is what the adversary aims to infer.Source: §2 · p. 2
- The data owner’s questions are whether public release creates membership risk, which models are riskier, and which records face greater risk.Risk is framed as an adversary’s ability to infer whether a record belongs to the training set.Source: §2 · p. 2
- The analysis assumes trusted parties seek to protect training data, and therefore focuses on accidental leakage rather than covertly embedded information.The risk concerns information an algorithm may unintentionally learn about an individual record.Source: §2 · p. 2
- The adversary knows the candidate dataset, training-set size, and learning algorithm, but not which records were sampled into the training set.The training set is assumed to be uniformly sampled from the candidate dataset.Source: §2 · p. 2
3 BACKGROUND ON MEMBERSHIP ATTACKS
The section reviews black-box membership attacks and describes the shadow-model approach used to train an attack classifier. The paper reproduces an earlier attack and introduces two new shadow-training attacks.
3 BACKGROUND ON MEMBERSHIP ATTACKS
- In Shokri et al.’s direct attack [37], shadow models generate labeled membership examples used to train an attack classifier.The adversary queries shadow models on their training and disjoint test records, then trains a neural-network attack classifier to infer target membership.Source: §3 · pp. 2–3
- The paper reproduces the membership attack of Shokri et al. [37] and designs two new attacks based on shadow training, one of which performs much better than the earlier attack.These attacks are used to validate DTP as a measure of membership privacy risk.Source: §3 · p. 3
4 MEASURING MEMBERSHIP PRIVACY
The paper links membership-attack accuracy to adversarial knowledge and to how much predictions change when a target record is removed. It defines DTP from prediction changes and introduces PDTP as an efficiently computable relaxation.
4.1 Understanding Membership Attacks
- Attack accuracy depends on how well the adversary estimates membership probabilities and on the target, training data, and classifier; overfitting and outliers may increase vulnerability.The section motivates measuring generalizability through the maximum prediction change after removing the target record.Source: §4.1 · p. 3
4.2 Differential Training Privacy
- DTP measures the maximum change in predicted class probabilities across queries when the target record is removed from training.The paper describes DTP as a local, experimentally measurable property of a record, training set, and classification algorithm.Source: §4.2 · p. 3
- Naively calculating DTP over all feature vectors and labels is impractical for large feature spaces, motivating pointwise differential training privacy (PDTP).PDTP bounds the classifier’s response change on the single query formed from the target record’s features when that record is removed.Source: §4.2 · pp. 3–4
- PDTP can be calculated by training an alternative classifier without the target record, a process related to leave-one-out cross-validation.PDTP is a lower bound on DTP and can serve as an optimistic risk estimate when DTP is not efficiently calculable.Source: §4.2 · p. 4
5 CASE STUDIES
The case studies test whether PDTP tracks record-level membership-attack risk across classifiers and datasets. Attack success is associated with PDTP, but vulnerability can vary by record and attack type.
5 CASE STUDIES
- When attack accuracy exceeds 0.7, PDTP is highly correlated with per-record attack accuracy, with stronger correlations for more accurate attacks.The authors use the DTP-1 hypothesis to flag classifiers above 1, while noting PDTP is a lower bound for DTP.Source: §5 · p. 4
5.1 Datasets
- Purchase data comprises 836 binary product-category features and 10 k-means consumer-behavior classes, while Adult uses demographic attributes to predict income class.Both experiments sample 2,000 candidate records and use 1,000 for training.Source: §5.1 · p. 4
5.2 Machine Learning Models
- The experiments compare neural networks, Naive Bayes, and logistic regression; classifier outputs are binned into 100 equal-width probability bins.Binning is intended to prevent leakage that does not significantly contribute to accuracy and avoid unreasonably large PDTP measurements from close-to-zero denominators.Source: §5.2 · p. 4
5.3 Attacks
- Untargeted attacks train an attack classifier from shadow-model predictions, while targeted attacks compare a target prediction with shadow-model estimates or frequency counts.Untargeted attacks have lower accuracy than some targeted attacks, but require less computation when searching for vulnerable records.Source: §5.3 · pp. 5–6
5.5 Results
- 0.8936 is the Pearson correlation between maximum per-target accuracy across three attacks and PDTP; among records with PDTP above 1, 84.62% exceed 0.8 maximum accuracy and all exceed 0.6.Distance-based targeted attacks had the highest accuracy, outperforming the prior untargeted attack [37] by approximately 19%.Source: Figure 2 · p. 7, §5.5 · p. 7
5.4 Evaluation Metrics
- Per-target attack accuracy measures correct membership inferences per record across attacks; each record’s estimated PDTP averages 10 measurements.Although attacks are run for 100 iterations, the authors use 10 PDTP measurements because their correlation with attack performance is approximately the same as with 100.Source: §5.4 · pp. 6–7
5.5 Results
- Attack performance is significantly correlated with average PDTP, and more accurate attacks show stronger correlations; Figure 1 compares the three attacks and their PDTP correlations.A record can be vulnerable to one attack but not another; for example, one record at baseline accuracy under the distance-based attack reached 0.94 accuracy under the untargeted attack.Source: Figure 1 · p. 6, §5.5 · p. 7
- On Adult, Naive Bayes and logistic regression had no records with PDTP above 1 and distance-based attack accuracy below 70%, while some neural-network records exceeded PDTP 1.For purchase, most records with PDTP above 1 under Naive Bayes and logistic regression were vulnerable to distance-based attacks; the correlation stabilized after about 10 PDTP measurements.Source: §5.5 · pp. 7–8
6 PROTECTIONS AGAINST INDIRECT MEMBERSHIP ATTACKS
The section tests whether indirect membership attacks can outperform direct attacks, then develops training stability as a way to bound indirect leakage efficiently. It examines which classifiers satisfy that property and gives a Lipschitz-based upper bound for classifiers whose stability is unknown or absent.
6.1 Risk
- For some classifiers, an indirect query can outperform direct attacks, so a record may have high membership risk despite low PDTP.The counterexample is explicitly not representative of a real-life model, especially one trained by a data owner seeking to protect privacy.Source: §6.1 · pp. 8–9
6.2 Training Stability
- Training stability bounds the prediction ratio after removing a record by the ratio on that record’s own query or by δ, allowing DTP to be calculated from PDTP more efficiently.For a record that is ε-PDTP, the resulting DTP bound is ε′ = max(ε, lnδ).Source: §6.2 · p. 9
6.3 Training Stability of Classifiers
- Naive Bayes, Bayes inference, and random decision trees with training-independent structure can be training stable, whereas k-NN is not stable for any δ.The section notes that it cannot cover all classifiers; data-dependent tree structures such as ID3 and C4.5 can change substantially when one example is removed.Source: §6.3 · pp. 9–11
6.4 An Upper Bound on DTP
- For non-stable classifiers or those with unknown stability, an L-Lipschitz condition bounds DTP through the maximum parameter change when one training example is removed.The bound is stated as L multiplied by the maximum parameter change across removed records.Source: §6.4 · p. 11
7 RELATED WORK
Related work considers privacy risks from releasing models alongside protections designed for datasets or differentially private learning. The paper positions DTP as a model-and-dataset risk measure for unintentional membership leakage and connects it to overfitting prevention.
Privacy metrics
- DTP measures the combination of a classification algorithm and dataset, unlike k-anonymity, l-diversity, and t-closeness, which describe properties of anonymized datasets.The paper also distinguishes DTP from privacy definitions that cannot be directly calculated from a dataset and algorithm.Source: §7 · pp. 11–12
Model attacks
- Prior black-box attacks infer sensitive attributes or membership, while DTP studies unintentional membership leakage without adversarial control of the training algorithm.The paper assumes the adversary has black-box model access but does not control training.Source: §7 · p. 12
Overfitting and privacy
- DTP reverses the reusable-holdout connection: it uses cross-validation techniques from overfitting prevention to measure privacy risk.The cited prior work applies differential privacy to validation data to permit reuse without overfitting risk.Source: §7 · p. 12
8 REDUCING DTP
The paper tests whether removing high-risk training records can bring a classifier below the DTP-1 threshold. In its example, removing the highest-risk record sharply lowers maximum DTP, but removing another can raise it again.
8 REDUCING DTP
- The example uses naive Bayes trained on 1.6 million 2013 American Community Survey records and initially has DTP 3.09.The model uses age, marital status, race, and gender to predict whether salary is above $50K.Source: §8 · p. 12
- Removing the highest-risk record lowered maximum DTP from 3.09 to 0.65, but removing the next raised it to around 3.The successive removals create a sawtooth pattern, as some records apparently help reduce other records’ influence on the classifier.Source: §8 · p. 12, Figure 6 · p. 12
- The experiment removes records one at a time in descending initial DTP order, recalculating the maximum DTP after each removal.The authors recommend this approach when only a few training examples have high privacy risks.Source: §8 · p. 12
9 DISCUSSION
The discussion distinguishes DTP from differential privacy and from membership attacks, and describes limits to interpreting low DTP. It also relates prediction changes to randomness in initialization and training-set sampling.
9 DISCUSSION
- DTP is a privacy-risk metric, not a protection mechanism, and assigns an ε value to any target record and classifier.The authors position it as a way to estimate risk when differential privacy cannot be used, and to assess the privacy–utility trade-off.Source: §9 · p. 13
- DTP cannot be replaced by testing membership attacks because attack simulations are inefficient and a data owner may not know the strongest attack.A record immune to a distance-based attack may still be vulnerable to another, even weaker overall, attack.Source: §9 · p. 13
- For non-training-stable models, measuring DTP is computationally infeasible, and indirect membership attacks remain a potential risk.The authors report that they do not know of practical indirect attacks and leave their study for future work.Source: §9 · p. 13
- DTP lacks DP’s theoretical privacy guarantee, though experiments find it highly correlated with state-of-the-art membership-inference performance.DTP bounds the magnitude of prediction differences caused by removing a record, unlike DP’s bound on changes in output probability.Source: §9 · p. 13
- Prediction variation from random initialization and random sampling can make a record’s DTP hard to distinguish when its influence is small.The paper illustrates these sources of variation by training 100 neural networks on the same dataset and 100 classifiers on different samples, respectively.Source: §9 · p. 13, Figure 7 · p. 13, Figure 8 · p. 13
10 OPEN QUESTIONS
The paper identifies unresolved questions about whether low-DTP records are safe, how to assess indirect attacks, and how membership risk changes across repeated queries.
10 OPEN QUESTIONS
- Whether records with low DTP are always safe from membership inference remains open, despite DTP’s observed predictive value for attack performance.The section frames two potential privacy risks for low-DTP records.Source: §10 · pp. 13–14
- DTP below 1 is a rule of thumb, not a guarantee that a record has no privacy risk.Randomization or unexpected training records may have less influence on predictions than the target record, and a finer threshold may require better understanding of those factors.Source: §10 · p. 14
- Low PDTP indicates lower likelihood of direct-attack vulnerability, but non-training-stable models may expose membership through indirect attacks.The paper leaves analysis of training stability and design of indirect attacks for future work.Source: §10 · p. 14
- DTP measures privacy risk for a single query; repeated queries may accumulate information about a target record.Open questions include selecting useful queries and estimating an upper bound on attack accuracy with unlimited queries.Source: §10 · p. 14
11 CONCLUSIONS
DTP estimates membership privacy risk from a training record’s influence on model predictions, and experiments show that its values track membership-attack accuracy. The authors propose a DTP-1 publication rule of thumb while cautioning that low DTP does not guarantee privacy.
11 CONCLUSIONS
- DTP estimates a training record’s privacy risk by measuring its influence on model predictions, with evaluations including neural networks, Naive Bayes, and logistic regressions.Source: §11 · p. 14
- At most 66.5% accuracy (baseline: 50%) occurs on classifiers with DTP-values under 0.5, while attacks almost always exceed 90% accuracy when DTP is larger than 4; the DTP-1 hypothesis advises against publishing classifiers above 1.The authors propose DTP-1 as a rule-of-thumb criterion for publishing a classifier.Source: §11 · p. 14
- DTP is highly correlated with membership-attack accuracy but does not guarantee privacy, since low-DTP records may remain vulnerable to certain prediction conditions or indirect and combined queries.The paper identifies cases where predictions are little influenced by other records or random initializations, as well as leakage through indirect or multiple queries.Source: §11 · p. 14
A PROOFS
The appendix proofs derive training-stability and DTP results for classifiers, including Naive Bayes and Bayes inference. They establish a DTP bound under a Lipschitz condition and analyze Naive Bayes using separate cases and count conditions.
A PROOFS
- Under the Lipschitz condition, Theorem 6.9 concludes that (A,T) is ε′-DTP, where ε′ = ln(γ) = max(ln(δ), ε).Source: A PROOFS · pp. 16, 18
- Proposition 6.3’s Naive Bayes count-based argument assumes nx,y > 1 and nx > 1.Source: A PROOFS · p. 16
- A separate proof states that the Bayes inference algorithm is “4 3-training stable.”Source: A PROOFS · p. 17
- The Naive Bayes proof treats separately the cases where the queried label differs from or matches the removed record’s label.Source: A PROOFS · p. 17