PaperlayerView exact revision on arXiv

Source-linked AI summary

Membership Inference Attacks From First Principles

Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, Florian Tramer

arXiv:2112.03570v2cs.CRcs.LG

TL;DR

Average-case metrics can miss whether membership inference attacks confidently identify even a few training-set members, so the paper evaluates attacks by true-positive rates at low false-positive rates. It develops LiRA by combining per-example difficulty scores with calibrated Gaussian likelihood estimates, finding 10× higher low-FPR success than prior work while strictly dominating prior attacks on aggregate metrics.

  • Problem

    Average-case metrics such as balanced accuracy can rate attacks similarly even when one precisely targets a small vulnerable subset and another is practically useless.

  • Method

    LiRA combines per-example hardness scores with empirically estimated, approximately Gaussian in/out likelihood distributions from models trained on random samples.

  • Results

    10× higher success at low false-positive rates, while strictly dominating prior attacks on previously used aggregate metrics.

  • Takeaways & Limitations

    Low-false-positive evaluation better captures whether attacks can confidently breach the privacy of even a few users and should inform privacy measurement.

  • Takeaways & Limitations

    The paper presents LiRA as one way to succeed at low false-positive rates and identifies additional evaluation directions for future work.

Abstract

from arXiv · show

A membership inference attack allows an adversary to query a trained machine learning model to predict whether or not a particular example was contained in the model's training dataset. These attacks are currently evaluated using average-case "accuracy" metrics that fail to characterize whether the attack can confidently identify any members of the training set. We argue that attacks should instead be evaluated by computing their true-positive rate at low (e.g., <0.1%) false-positive rates, and find most prior attacks perform poorly when evaluated in this way. To address this we develop a Likelihood Ratio Attack (LiRA) that carefully combines multiple ideas from the literature. Our attack is 10x more powerful at low false-positive rates, and also strictly dominates prior attacks on existing metrics.

I. INTRODUCTION

Membership inference attacks are used to assess whether specific examples were included in model training, but average-case metrics can miss failures at low false-positive rates. The paper advocates low-FPR evaluation and introduces LiRA, which improves attack performance while retaining strong aggregate results.

  • Membership inference attacks predict whether a particular example was used to train a model, supporting empirical checks of privacy for models trained on sensitive data.
  • Average-case metrics such as accuracy and ROC-AUC can obscure whether an attack confidently identifies even a few training-set members.
  • Low false-positive-rate true-positive rate is the proposed evaluation focus because reliably identifying a few users can itself constitute a privacy violation.
  • 59.5% accuracy for Yeom et al. fails completely at low FPRs, while 53.5% accuracy for Long et al. achieves higher low-FPR success rates.
  • LiRA succeeds 10× more often than prior work at low FPRs and strictly dominates prior attacks on previously used aggregate metrics.
  • The paper reports that stronger attacks may change conclusions about which algorithms are private and whether existing defenses are effective.

B. Training data privacy

Training-data privacy research studies attacks, memorization, defenses, and privacy measurement, with membership inference focused on identifying individual training examples. The paper formalizes this attack through a security game and confidence-score thresholding.

  • Training-data privacy research develops attacks, memorization analyses, privacy-preserving training, and methods for measuring how much privacy a training scheme offers.
  • Membership inference predicts whether a specific example belongs to a model’s training dataset, making it a direct privacy attack on individual examples.
  • The security game trains a model on a sampled dataset, selects either a member or fresh non-member example, and asks the adversary to predict membership.
  • The game gives the adversary access to the underlying data distribution and the target model, while the example and its ground-truth label are provided.
  • Attacks output continuous confidence scores that are thresholded into membership predictions; the LOSS attack uses lower model loss as evidence of membership.

B. Evaluating membership inference attacks

The paper argues that average-case membership-inference metrics can conceal whether an attack confidently identifies members. It therefore recommends evaluating true-positive rates at low false-positive rates, especially through ROC analysis.

  • Balanced Attack Accuracy: Balanced attack accuracy treats false-positives and false-negatives symmetrically, although adversaries may care primarily about false-positives.This makes the metric poorly aligned with some privacy-attack settings.
  • Balanced Attack Accuracy: Balanced accuracy can rate attacks equally even when one confidently targets a small user subset and another is practically useless across users.The paper gives an example where both attacks have the same average success rate despite sharply different privacy risks.
  • LOSS Attack: 60% balanced accuracy for the LOSS attack suggests usefulness, but its lowest-loss 1% is correct only 48% of the time while its highest-loss 1% is correct 100%.The attack therefore confidently identifies non-members rather than members.
  • ROC Analysis: ROC curves compare true-positive rate and false-positive rate across thresholds, emphasizing whether membership guesses succeed while false positives remain low.For the LOSS attack, no FPR below 20% achieves TPR better than random chance.
  • ROC Analysis: AUC is inadequate because it averages over all false-positive rates, including high-error regimes that are not practically useful for privacy attacks.The paper notes that an attack identifying 0.1% of members perfectly can have AUC ≈51%, below the weak LOSS attack.
  • Recommended Evaluation: The recommended evaluation reports TPR at low FPRs such as 0.001% or 0.1% and preferably includes the full ROC curve on a logarithmic scale.The LOSS attack achieves 0% TPR at 0.1% FPR.

IV. THE LIKELIHOOD RATIO ATTACK (LIRA)

The paper formulates membership inference as distinguishing models trained with or without a target example. LiRA estimates per-example membership and non-membership loss distributions, addressing variation in example difficulty that defeats a global loss threshold.

  • Membership inference as hypothesis testing: Membership inference can be viewed as a hypothesis test between model distributions conditioned on inclusion or exclusion of a target example.The relevant distributions are Qin(x, y) for members and Qout(x, y) for non-members.
  • Membership inference as hypothesis testing: The Neyman-Pearson lemma motivates thresholding a likelihood ratio to obtain the best test at a fixed false-positive rate.The likelihood ratio compares the probability of the observed model under the two hypotheses.
  • The Likelihood Ratio Attack: Because Qin and Qout are analytically intractable, the attack replaces model distributions with distributions of losses on the target example.This yields a one-dimensional statistic that can be computed through query access to the target model.
  • Memorization and per-example hardness: Different examples vary independently in fitting difficulty and in how separable their member and non-member losses are.An example can be an outlier but easy to fit, or an inlier but hard to fit.
  • The Likelihood Ratio Attack: A global loss threshold cannot distinguish these scenarios, whereas LiRA models separate Qin and Qout distributions for each example.This per-example modeling accounts for differences in hardness and loss scale.

C. Estimating the likelihood-ratio with parametric modeling

LiRA uses shadow models and parametric Gaussian estimates to approximate per-example member and non-member distributions. Logit-scaled confidences support approximately normal modeling, while offline estimation reduces query-time training costs.

  • Parametric modeling: 400× fewer models can match the generalization of recent nonparametric approaches under parametric modeling.Parametric estimates require fewer shadow models for comparable generalization.
  • Transforming confidences: The logit of model confidence is empirically approximately normal, unlike confidence or cross-entropy loss, making it the preferred modeling scale.Figure 4 compares the distributions of confidence, negative log confidence, and logit confidence.
  • Online LiRA: LiRA trains shadow models on random samples, fits Gaussian distributions to transformed IN and OUT confidences, and evaluates a likelihood-ratio test on the target model.The online algorithm uses separate models trained with and without the target example.
  • Multiple queries: Multivariate Gaussian modeling extends LiRA to multiple augmented queries of each target point and can further improve attack success rate.The same framework can fit spherical Gaussians to repeated loss or confidence observations.
  • Offline LiRA: The online attack requires training new models for every membership-inference query batch, making it computationally expensive.This is the principal usability limitation motivating the offline variant.
  • Offline LiRA: The offline attack trains shadow models in advance, avoids target points, estimates only the OUT distribution, and uses a one-sided hypothesis test.It compares target confidence against the estimated non-member distribution.

V. ATTACK EVALUATION

The evaluation examines online and offline attacks across complex image and language datasets, emphasizing performance at very low false-positive rates. Results show strong low-FPR success across datasets and only a modest offline penalty.

  • Evaluation objective: Low false-positive rates are prioritized because false positives are more costly than false negatives for training-data extraction and identifying users in sensitive datasets.The section frames this regime as having the most practical consequences.
  • Evaluation setup: 256 shadow models are used for most datasets, while ImageNet uses 64, and each example appears in exactly half of the shadow-model training sets.This setup permits IN and OUT comparisons for every example, although the shadow and target training sets may partially overlap.
  • Datasets and setup: The evaluation covers CIFAR-10, CIFAR-100, ImageNet, and WikiText-103, alongside online and offline attack variants.Additional Purchase and Texas results are relegated to the Appendix because the authors consider those datasets too simple for generalizable lessons.
  • Online attack results: 0.1% to 10% true-positive rates are achieved at a 0.001% false-positive rate across the evaluated settings.The reported range summarizes the attack’s low-FPR performance.
  • Online attack results: At low false-positive rates, CIFAR-10 models are easier to attack than ImageNet models despite having better generalization.The comparison contrasts low-FPR vulnerability with average success-rate behavior linked to the generalization gap.

B. Offline attack evaluation

The section re-evaluates prior membership inference attacks under the low-FPR objective and compares their performance using shared datasets and settings. It highlights per-example calibration and LiRA’s Gaussian modeling as important distinctions for tail behavior.

  • Re-evaluating prior attacks: Prior attacks are re-evaluated on the same datasets and under the same protocol, selecting ROC thresholds that maximize true-positive rate at each false-positive rate.Table I compares eight representative attacks using 256 shadow models on well-generalizing CIFAR-10, CIFAR-100, and WikiText-103 models.
  • Multiple queries: Multiple queries in MERLIN do not improve low-FPR success even with 100 target-model queries, whereas augmentation queries motivate improvements to the authors’ attack.These comparisons concern query-based alternatives and additional model queries.
  • Per-class hardness: Per-class thresholds do not improve balanced accuracy or low-FPR attack success, although they improve CIFAR-10 and CIFAR-100 AUC by 2%.The result separates aggregate AUC changes from the low-FPR metrics emphasized in the section.
  • Per-example hardness: Per-example hardness-calibrated attacks achieve significantly better true-positive rates at low false-positive rates despite slightly lower balanced accuracy and AUC than simpler attacks.This creates a marked discrepancy between aggregate metrics and the recommended low-FPR evaluation.
  • Per-example hardness: The Long et al. attack outperforms most prior attacks at low false-positive rates despite filtering outliers and therefore not achieving high average accuracy.Its procedure selects high-distance outliers and estimates non-member loss probabilities with shadow models.
  • LiRA’s parametric modeling: Gaussian loss modeling distinguishes LiRA from mean-only approaches and nonparametric tail extrapolation methods.Mean-only attacks can be sub-optimal when loss distributions differ in scale and spread, while empirical tail methods can fail to extrapolate reliably to very low false-positive rates.
  • LiRA’s parametric modeling: 256 shadow models limit exact testing by Ye et al. to a best-case false-positive rate of 1/256, making it inapplicable to a 0.1% target false-positive rate.Long et al. instead extrapolates empirical loss tails with cubic splines, which the authors report can overfit and diverge outside their support.

D. Membership inference and overfitting

The attack’s low-FPR performance depends on modeling choices beyond average-case metrics, including logit scaling, per-example thresholds, and shadow-model information. Overfitting correlates with vulnerability, but models with identical train-test gaps can differ substantially.

  • Membership inference and overfitting: 100× greater attack vulnerability can occur among models with identical train-test gaps, although higher overfitting generally corresponds to greater vulnerability.More accurate models also show a clear trend toward greater vulnerability.
  • Membership inference and overfitting: 0% TPR at 0.1% FPR is achieved by the baseline LOSS attack, with only marginal improvement from loss scaling and multiple queries without per-example thresholds.The baseline is therefore ineffective in the low-FPR regime despite possible aggregate performance.
  • Membership inference and overfitting: 1.7% attack success at low FPR results from incorporating per-example thresholds, while loss rescaling and Gaussian fitting increase success by 3.3×.Evaluating the target model with its training-time data augmentations can nearly double attack success.
  • Membership inference and overfitting: The offline attack reaches 7.1% TPR versus 8.4% for the full attack at 0.1% FPR after adding logit scaling, Gaussian likelihood, and multiple queries.These components make the offline variant nearly as strong as the full attack.
  • Membership inference and overfitting: 10^-order differences in low-FPR performance occur among attack variants with similar AUC scores of 68–72%.This supports evaluating attacks at low false-positive rates rather than relying on aggregate ROC metrics.
  • Membership inference and overfitting: Logit scaling nearly quadruples Watson et al.’s attack TPR at 0.1% FPR.The attack first maps model confidences to a logit scale so the fitted distributions are approximately Gaussian; raw confidence and cross-entropy perform poorly.

B. Gaussian distribution fitting

Gaussian fitting is central to LiRA’s low-FPR performance, but its reliability depends on the number of shadow models. Global variance estimation helps with few models, whereas the full per-example approach is stronger with many.

  • Gaussian distribution fitting: Querying augmented image versions doubles TPR at low FPR, with most benefits obtained from two queries.The result is summarized for the augmented-query analysis.
  • Gaussian distribution fitting: More than 64 shadow models provide diminishing attack benefits, while fewer models cause rapid deterioration because Gaussian distributions are difficult to fit from few points.The attack is evaluated at a fixed FPR of 0.1%.
  • Gaussian distribution fitting: For fewer than 64 shadow models, globally estimating variance outperforms estimating variance separately for each example.Per-example means remain estimated separately, while variance is pooled across examples.
  • Gaussian distribution fitting: With 1024 shadow models, global variance lowers TPR from 8.4% to 7.9%, making the full attack stronger.The tradeoff reverses as the number of shadow models increases.

C. Number of queries

LiRA remains effective when shadow and target training sets are disjoint or the target training setup is uncertain. Matching data augmentation matters most, while architecture and optimizer mismatches have smaller effects.

  • Number of queries: Disjoint shadow-model and target-model training sets have only a minor effect on attack success, whereas sampling from different distributions decreases performance.The disjoint-data result is shown on CINIC-10.
  • Number of queries: The CINIC-10 experiment compares shadow models trained from the full dataset with alternative setups using 128 shadow models and 50,000 points per model.This tests a more realistic attacker dataset that is disjoint from the target training set.
  • Number of queries: The attack succeeds when the adversary is uncertain about target architecture, optimizer, and augmentation, with best performance when these properties are guessed correctly.Results are reported at a fixed FPR of 0.1%.
  • Number of queries: Architecture mismatches have minimal impact when the shadow architecture is similar, while optimizer choice has minimal impact on the attack.The attack performs best when architectures match exactly.
  • Number of queries: Correctly guessing the target model’s data augmentation has the largest impact on attack performance, and stronger augmentations make models harder to attack.The attacker uses its augmentation guess both for shadow training and additional queries.

A. Attacking real-world models

LiRA succeeds against pretrained state-of-the-art models and identifies substantial variation in example-level privacy. Out-of-distribution examples are easier to detect, while several privacy questions remain open for stronger or differently capable attacks.

  • Attacking real-world models: LiRA succeeds on existing pretrained CIFAR-10 models using shadow models sampled from the full dataset, including when shadow and target architectures differ.The experiment evaluates TPR at a 0.1% FPR across canonical architectures.
  • Attacking real-world models: 5% TPR falls only to 4% when a ResNet-34 target is attacked with ResNet-18 or ResNet-50 shadow models.This shows a minor drop from using different shadow architectures.
  • Attacking real-world models: Out-of-distribution training examples receive higher privacy scores and are easier to detect than standard CIFAR-10 examples.CINIC-10 images have larger average privacy scores because of their distribution shift.
  • Attacking real-world models: Randomly relabeled CIFAR-10 images have much higher privacy scores than correctly labeled CINIC-10 images, with CIFAR-100 examples spanning the intermediate setting.The comparison uses inserted examples in an augmented training dataset.
  • Attacking real-world models: Future work must test whether reduced-capability attacks, stronger-capability attacks, prior defenses, and differential privacy remain effective at low false-positive rates.The paper reports only preliminary evidence that vacuous differential-privacy guarantees might prevent low-FPR attacks.

APPENDIX A ADDITIONAL EXPERIMENTS

Additional experiments show how defenses, access assumptions, model accuracy, and shadow-model count affect membership inference attack performance. Small DP-SGD noise can reduce attacks to chance, better models are more vulnerable, and global variance estimates help when few shadow models are available.

  • Attacking DP-SGD: Small amounts of DP-SGD noise reduce membership inference effectiveness to chance while producing higher-accuracy models.This effect occurs for higher clipping norms; small clipping norms alone also significantly reduce attack performance.
  • Attacking DP-SGD: Very small amounts of DP-SGD noise defend against the attack despite resulting in very large provable DP bounds ϵ.
  • White-box versus black-box attacks: White-box gradient norms improve overall AUC for the online attack compared with the black-box setting.The comparison examines whether gradient information provides an advantage beyond the black-box setting.
  • Attack performance versus model accuracy: Better model test accuracy is associated with greater vulnerability to membership inference attacks.The relationship is observed across the evaluated models and is consistent with prior work on data extraction attacks.
  • Shadow-model count: With fewer than 64 shadow models, per-example variance estimates work poorly, whereas global variance estimates perform nearly as well as the best attack with 16 models.The experiment varies the number of shadow models from 4 to 256 and compares per-example against global variance estimation.

C. Comparison to Prior Work on Additional Datasets

Across CIFAR-100, WikiText-103, Texas, and Purchase, the paper compares LiRA with prior attacks and evaluates its components through ablations. The results show that combining logit scaling with a Gaussian Likelihood Test is important for strong performance, while evaluation settings vary by dataset.

  • Comparison to prior work: LiRA is compared with prior membership inference attacks on CIFAR-100, WikiText-103, Texas, and Purchase using ROC curves.The WikiText-103 comparison omits attacks relying on model features z(x), because they were not designed for sequential models.
  • Experimental settings: For WikiText-103, Texas, and Purchase, the models and attacks use no data augmentations.
  • Attack ablations: A Gaussian Likelihood Test after logit scaling significantly boosts prior per-example-threshold attacks in both offline and online settings.This pattern is reported across the additional-dataset ablations.
  • Attack ablations: Logit scaling alone or a Gaussian Likelihood Test alone often hurts online attack performance, so the components must be applied together for good performance.
  • Mismatched model settings: Figures 22–24 evaluate attacks when the adversary must guess the target model’s architecture, optimizer, and data augmentation.The experiments vary architectures, optimizers, and augmentations in the plotted ROC curves.
Loading 2112.03570v2…