PaperlayerView exact revision on arXiv

Source-linked AI summary

Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference

Klas Leino, Matt Fredrikson

arXiv:1906.11798v2cs.LGcs.CRstat.ML

TL;DR

Membership-inference attacks can expose whether sensitive data was used to train a model, including when output behavior appears to generalize well. This paper exploits internal feature usage to build a white-box attack, finding improved performance and reliable high-precision calibration, while showing difficult trade-offs for differential-privacy defenses.

  • Problem

    Models trained on sensitive data may leak membership information, and prior attacks have not effectively leveraged internal model information or consistently supported confident inferences.

  • Method

    The paper analyzes overfitting-related feature usage and develops a white-box membership-inference attack using the target model’s internals.

  • Results

    The attack outperforms previous attacks and can be reliably calibrated for high precision, including on some well-generalized models.

  • Takeaways & Limitations

    Small privacy budgets can mitigate the attack but quickly reduce utility, while larger budgets sometimes leave attack accuracy close to that of undefended models.

  • Takeaways & Limitations

    Differential-privacy protection in the evaluation has a steep performance cost, particularly for small ε, limiting its practicality.

Abstract

from arXiv · show

Membership inference (MI) attacks exploit the fact that machine learning algorithms sometimes leak information about their training data through the learned model. In this work, we study membership inference in the white-box setting in order to exploit the internals of a model, which have not been effectively utilized by previous work. Leveraging new insights about how overfitting occurs in deep neural networks, we show how a model's idiosyncratic use of features can provide evidence for membership to white-box attackers---even when the model's black-box behavior appears to generalize well---and demonstrate that this attack outperforms prior black-box methods. Taking the position that an effective attack should have the ability to provide confident positive inferences, we find that previous attacks do not often provide a meaningful basis for confidently inferring membership, whereas our attack can be effectively calibrated for high precision. Finally, we examine popular defenses against MI attacks, finding that (1) smaller generalization error is not sufficient to prevent attacks on real models, and (2) while small-$ε$-differential privacy reduces the attack's effectiveness, this often comes at a significant cost to the model's accuracy; and for larger $ε$ that are sometimes used in practice (e.g., $ε=16$), the attack can achieve nearly the same accuracy as on the unprotected model.

1 Introduction

Machine-learning models trained on sensitive data can leak membership information through their behavior or internal representations. This paper develops a white-box attack that exploits idiosyncratic feature use, improves on prior attacks, and supports calibrated high-precision inferences.

  • Models trained on private data may leak information through their behavior or learned representations.
  • Prior membership-inference work primarily uses shadow models trained from target-model outputs, while earlier white-box extensions found limited effectiveness without substantial training-data knowledge.
  • Overfitting can encode training-specific features that reveal membership even when anomalous behavior is not visible in model outputs.
  • In the Tony Blair example, the model uses the face for test images but a distinctive pink background for the corresponding training image.
  • The evidence-based white-box attack works across real datasets and outperforms previous attacks.
  • The naive attack can match shadow-model performance, but correct classification alone provides little confidence that a point was a training member.
  • Shadow attacks are also often poorly calibrated, with higher confidence thresholds sometimes reducing precision.
  • The proposed attack is designed for confident inferences and can calibrate its confidence scores for precision.

2 Background

The paper formalizes membership inference against neural-network targets and distinguishes black-box from white-box access. It evaluates attacks using accuracy, advantage, precision, and recall, emphasizing confident positive inferences.

  • Membership inference determines whether a data point was included in a target model’s training set.
  • The supervised-learning setup trains a model on finite samples from a data distribution to perform well on future unseen samples.
  • Target models are feed-forward neural networks composed of parameterized linear layers followed by nonlinear activations.
  • The standard architecture uses ReLU activations internally and a softmax output whose scores can be interpreted as label probabilities.
  • The attacker receives either a population sample or a training-set sample and predicts which source generated it.
  • Black-box access permits queries and auxiliary knowledge, whereas white-box access exposes the target model’s exact weights, biases, and operators.
  • Attack accuracy measures correct membership predictions, while advantage rescales accuracy relative to the 50% random-guessing baseline.
  • Precision is emphasized because identifying even one member confidently can constitute a privacy violation, whereas recall alone can be misleading.

3 White-box Membership Inference

The paper develops white-box membership inference by treating idiosyncratic feature use as evidence of training-set membership, then estimates attack parameters from a proxy model. The resulting attacks can be Bayes-optimal under stated assumptions, effective beyond them, and calibrated for high-precision positive inferences.

  • Finding Evidence of Membership: The attack exploits anomalous feature use that may reveal overfitting even when predictions on unseen points remain correct.Features coincidental to the training data can provide membership evidence independently of prediction correctness.
  • Finding Evidence of Membership: Features distributed differently in the training set than in the true distribution are compiled into a logistic attack model.The attack profiles idiosyncratic feature use to construct membership parameters.
  • Obtaining MI Parameters from Proxy Models: A proxy dataset drawn from the population trains a proxy model whose parameters are compared with the target model to approximate the attack.Element-wise subtraction yields the bayes-wb attack under Gaussian naive-Bayes assumptions; a learned displacement function yields the general-wb attack.
  • A Bayes-Optimal Attack: Under Gaussian assumptions with matching diagonal covariance and class priors, the derived predictor is Bayes-optimal for membership inference.The theorem assumes the empirical training distribution and population distribution differ through estimated feature means while retaining covariance and prior structure.
  • A Bayes-Optimal Attack: Attack-model weights become large for features whose training-set means differ substantially from population means relative to feature variance.These features function as the primary evidence for membership, with parameters defined separately by class label.
  • Obtaining MI Parameters from Proxy Models: The attacker's exact knowledge of the population and training distributions is unrealistic, and the target model's weights underdetermine those distributions.The paper argues that useful differences between distributions can nevertheless be encoded in learned weights.
  • Calibrating for Precision: Confidence thresholds can raise precision for positive inferences, but finite samples eventually reduce recall, and shadow attacks are often poorly calibrated.The paper reports that increasing confidence thresholds sometimes decreases shadow-attack precision, whereas its attack can be accurately calibrated.

4 Membership Inference in Deep Models

For deep models, the paper preserves target-model feature semantics by locally linearizing each layer's downstream computation. It applies white-box attacks to these layer-wise approximations and combines their outputs with a meta model.

  • Membership Inference in Deep Models: Shadow-model attacks struggle because corresponding internal feature indices need not have the same semantic meaning across independently trained networks.A target feature may match a shadow feature only accidentally, making properties learned from shadow representations unreliable.
  • Membership Inference in Deep Models: The method fixes feature semantics by constructing local linear approximations that operate directly on each target model's internal representation.Attacks developed for linear models can then be applied to each approximation and combined for the full network.
  • Local Linear Approximations of Deep Models: A network slice decomposes the model into h, which computes layer inputs, and g, which maps those features to the output.For a top-layer slice, g is already linear; lower slices require approximation.
  • Local Linear Approximations of Deep Models: For lower layers, an influence measure quantifies each internal feature's contribution to the output and constructs a matching linear model at the input point.The proposed influence measure is designed for internal features, marginal contributions, linear agreement, and efficiency relative to a baseline.
  • Local Linear Approximations of Deep Models: Internal influence generalizes linear-model weights and supports a faithful local approximation for arbitrary network layers.With a zero baseline, the approximation expresses the output as feature influences times the internal representation plus the baseline output.
  • Combining Layer-wise Attacks: Layer-wise attacks are combined by a meta model that learns from their logistic outputs because internal representations across layers are not independent.The meta model is trained using layer-wise attack outputs from a shadow target model and can use multiple random data partitions.
  • Combining Layer-wise Attacks: For the general-wb attack, the meta model can be trained jointly with separate displacement metrics for each network layer.This avoids first learning a separate general-wb attack on every layer.

5 Evaluation

Across synthetic and real datasets, the proposed white-box attacks generally outperform prior black-box methods, especially in precision, while remaining effective under varied assumptions, data sizes, and model generalization. Calibration further enables high-precision inferences, and combining information across layers improves attack efficacy.

  • 5.2 Sensitivity to Assumptions & Hyper-parameters: general-wb recovers nearly all of the omniscient attack’s advantage despite making no generative assumptions.Validation data also provides a reliable way to tune its architecture and hyperparameters.
  • 5.2 Sensitivity to Assumptions & Hyper-parameters: 84% of the omniscient attack’s advantage is achieved on average by bayes-wb using a proxy model on synthetic Gaussian naive-Bayes data.With matching capacity, general-wb recovers 94% of bayes-wb’s advantage; with excess capacity, it achieves 92%.
  • 5.3 Data Scaling: As training data increases, attack advantage generally declines, becoming less than 4% on the full Adult dataset of 48,841 records.The same downward trend appears across real datasets, although image datasets such as CIFAR10 and CIFAR100 are notable exceptions.
  • 5.4 Layer Contributions: Combining information from different model layers substantially outperforms individual layer attacks, showing that leakage is distributed across learned representations.For MNIST, the meta attack’s advantage exceeds that of all individual layers combined.
  • 5.5 Comparison to Prior Work: bayes-wb and general-wb consistently exceed naive and shadow-bb in precision, with accuracy improvements reaching seven percentage points and precision typically improving by at least five points.Naive and shadow-bb often obtain higher recall, but with lower precision.
  • 5.5 Comparison to Prior Work: Raising the decision threshold consistently improves the proposed attacks’ precision, often by 10 or more percentage points.This calibration supports confident inferences even on models with low generalization error, while prior approaches do not reliably provide calibrated confidence.

6 Defenses

The paper evaluates regularization, differential privacy, and black-box defenses against white-box membership inference. Small-ε differential privacy reduces attack effectiveness but can sharply reduce model accuracy, while larger ε may provide weak protection; regularization offers a modest, often more practical defense but does not eliminate leakage.

  • Differential Privacy: Small-ε differential privacy quickly reduces the attack’s effectiveness, but the resulting model can suffer a steep accuracy cost.The performance cost is particularly high for small ε.
  • Differential Privacy: At ε=16, differential privacy sometimes leaves attack accuracy essentially unchanged from the undefended model, while harming model accuracy.This occurred on BCW, PD, and LFW; on Hep, 16-DP did not reduce effectiveness below shadow-bb on the undefended model.
  • Differential Privacy: Theoretical privacy bounds can substantially overestimate practical attack accuracy: at ε=0.25, the bound is 64.2%, whereas the attack reached 53.5% on LFW.For ε>ln2≈0.69, the theoretical accuracy bound is 100%, which no attack achieved.
  • Regularization: Dropout does not significantly reduce attack accuracy in most cases, although regularization can improve model performance while providing a modest defense.Better generalization makes membership inference more difficult, but not impossible.
  • Regularization: Even regularization that improves generalization may not prevent parameter-based membership leakage, because models can make correct predictions while retaining membership information.The authors caution that this may not hold universally across regularization methods.
  • Black-box Defenses: Top-k predictions, softened softmax outputs, and model stacking can be circumvented in the white-box setting because underlying outputs or component models remain accessible.These methods were proposed as black-box defenses but do not hide the information available to white-box attackers.

7 Related Work

Prior membership-inference research spans statistical summaries, classical models, and deep networks, with most deep-network attacks using black-box access. Existing white-box approaches have had limited success or rely on stronger assumptions, such as access to substantial training data.

  • Prior Privacy Attacks: Membership attacks have been studied against genomic statistics, SVMs, HMMs, deep neural networks, GANs, and federated-learning systems.The paper situates its work within a broad privacy-attack literature spanning statistical and machine-learning models.
  • Black-box Attacks: Most prior deep-network attacks use the black-box threat model, relying on prediction correctness or confidence outputs to infer membership.The canonical shadow-bb attack trains shadow models and an attack model from member and non-member outputs.
  • Black-box Attacks: Shadow-model attacks often fail to outperform the naive attack, possibly because their learned attack model itself overfits to shadow-model-specific behavior.The paper reports this finding from its evaluation and presents overfitting as one potential explanation.
  • White-box Attacks: White-box attacks can exploit model structure and parameters, but few prior approaches successfully use this information.Internal units of independently trained shadow models may not correspond to those of the target model.
  • White-box Attacks: Nasr et al.’s white-box attack uses loss gradients with respect to weights but assumes the adversary knows as much as half of the target model’s training data.This differs from the standard threat model considered by the paper.

8 Conclusions and Future Work

The paper develops a white-box membership-inference attack from the insight that overfitting can appear in a model’s feature usage rather than its outputs. Experiments show improved prior performance and a nuanced defense trade-off: small ε helps privacy but reduces utility, whereas larger ε can leave attacks nearly as effective as on undefended models.

  • 8 Conclusions and Future Work: The work fully leverages white-box information in the standard threat model without requiring examples of true training points.It analyzes feature usage as a mechanism through which overfitting leaks membership information.
  • 8 Conclusions and Future Work: The new attack improves upon prior state of the art and can be calibrated for high precision, including on some well-generalized models.The paper emphasizes confident positive inferences as a key advantage of calibration.
  • 8 Conclusions and Future Work: Small ε mitigates the attack but quickly diminishes model utility, while sufficiently large ε can leave attack accuracy close to that on undefended models.The authors use these findings to motivate further work on effective privacy defenses.

A Proof of Theorem 1

The proof transforms the membership-related probability expression using Bayes’ rule and the Naive-Bayes independence assumption, then derives a Bayes-optimal model. Under equal variances, the quadratic term vanishes, yielding the stated linear model.

  • Bayes’ rule converts the initial expression for m_y(x) into Equation 10.
  • Assuming equal prior membership probabilities, Equation 10 is rewritten as a logistic function using total probability.
  • The conditional probabilities represent drawing x from the member and nonmember distributions given class y.
  • The Naive-Bayes assumption factorizes the class-conditional probabilities across independent features, producing Equation 12.
  • Taking logarithms converts the product into a sum and then a dot product, giving the Bayes-optimal model parameters.
  • Equal feature variances in S and the general distribution make the quadratic term zero, leaving the linear model with weights w_y and bias b_y.
Loading 1906.11798v2…