Source-linked AI summary
Membership Inference Attacks against Language Models via Neighbourhood Comparison
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, Taylor Berg-Kirkpatrick
TL;DR
Membership inference attacks need to distinguish training members from non-members without being misled by sample difficulty, yet reference-based methods often assume access to similar training data. This paper replaces reference models with synthetic textual neighbours and finds that neighbourhood attacks outperform realistic reference-based attacks by up to 100% while remaining competitive with oracle references. The method is specific to textual data, and baseline reimplementations may not be perfectly fair.
Problem
Existing loss-threshold attacks do not account for sample difficulty, while LiRAs require sufficient data resembling the target training distribution, which may be unavailable in privacy-sensitive settings.
Method
Neighbourhood attacks compare a target sample’s loss with losses of highly similar synthetic neighbours generated through word replacements.
Results
Up to 100%, neighbourhood attacks outperform LiRAs under more realistic accessible-data assumptions and remain competitive with oracle reference models.
Takeaways & Limitations
Reference-free neighbourhood attacks can provide strong membership-inference performance without access to the target training-data distribution.
Takeaways & Limitations
The neighbour-generation method is specific to textual data, and reimplemented baseline attacks may not yield perfectly fair comparisons.
Abstract
from arXiv · showhide
Membership Inference attacks (MIAs) aim to predict whether a data sample was present in the training data of a machine learning model or not, and are widely used for assessing the privacy risks of language models. Most existing attacks rely on the observation that models tend to assign higher probabilities to their training samples than non-training points. However, simple thresholding of the model score in isolation tends to lead to high false-positive rates as it does not account for the intrinsic complexity of a sample. Recent work has demonstrated that reference-based attacks which compare model scores to those obtained from a reference model trained on similar data can substantially improve the performance of MIAs. However, in order to train reference models, attacks of this kind make the strong and arguably unrealistic assumption that an adversary has access to samples closely resembling the original training data. Therefore, we investigate their performance in more realistic scenarios and find that they are highly fragile in relation to the data distribution used to train reference models. To investigate whether this fragility provides a layer of safety, we propose and evaluate neighbourhood attacks, which compare model scores for a given sample to scores of synthetically generated neighbour texts and therefore eliminate the need for access to the training data distribution. We show that, in addition to being competitive with reference-based attacks that have perfect knowledge about the training data distribution, our attack clearly outperforms existing reference-free attacks as well as reference-based attacks with imperfect knowledge, which demonstrates the need for a reevaluation of the threat model of adversarial attacks.
1 Introduction
Membership inference attacks assess whether samples belonged to a model’s training data, but simple loss thresholding and reference-based calibration have important limitations. This paper proposes neighbourhood attacks that use synthetic neighbours instead of training-distribution data and reports strong performance under realistic assumptions.
- Motivation: Membership inference attacks classify whether a sample was present in a target model’s training data, helping assess privacy risks from language models.Language models can also expose sensitive training strings through verbatim generation.
- Limitations of Existing Attacks: LOSS attacks threshold sample losses, but their apparent accuracy can mainly reflect identifying non-members rather than training members.This weakness is linked to samples differing in intrinsic difficulty, including repetitive or atypical examples.
- Limitations of Existing Attacks: Reference-based LiRAs calibrate scores using reference models, but require sufficient samples resembling the target model’s training distribution.That assumption may be unrealistic in privacy-sensitive domains where high-quality public in-domain data is scarce.
- Proposed Approach: Neighbourhood attacks generate highly similar texts through word replacements and compare their target-model losses with the original sample’s loss.The approach uses synthetic neighbours rather than access to the target training distribution.
- Results: Up to 100%, neighbourhood attacks outperform LiRAs using more realistic assumptions about accessible data while remaining competitive with perfect-distribution reference attacks.The experiments motivate reevaluating the threat model for adversarial attacks.
2 Membership Inference Attacks via Neighbourhood Comparison
The neighbourhood attack calibrates a target sample’s loss by comparing it with synthetically generated, minimally changed neighbours. Its decision rule treats neighbour losses as a reference for the target’s expected non-member loss.
- Attack Setting: The attack operates in a grey-box setting where adversaries obtain confidence or loss scores but not model weights or gradients.Its goal is a binary membership decision for textual samples.
- Calibration: Unlike LOSS thresholding and LiRA reference calibration, the proposed method derives difficulty information from synthetically crafted neighbours.LiRA’s calibration depends on the quality and training data of its reference model.
- Neighbour Construction: The method constructs natural adjacent samples that differ slightly from the target and are intended to be equally plausible in general textual data.These neighbours are assumed not to occur in the target model’s training data.
- Decision Rule: The calibrated score subtracts the average neighbour loss from the target sample’s loss before applying a membership threshold.Semantics- and grammar-preserving neighbours provide a proxy for the target’s non-member loss.
- Neighbour Generation: A pretrained masked language model generates replacement candidates, while dropout helps preserve the original word’s meaning during substitution.Naively masking a token can produce semantically unsuitable replacements, such as replacing “great” with “bad” in “The movie was great.”
3 Experimental Setup
The experiments evaluate neighbourhood attacks and reference-free and reference-based baselines on GPT-2 models across news, Twitter, and Wikipedia-derived datasets. Reference models represent increasingly realistic or idealized levels of access to the target data distribution.
- Models and Baselines: The evaluation compares neighbourhood attacks with reference-free and reference-based baselines on fine-tuned 117M-parameter GPT-2 target models.The target models use the classical autoregressive language-modeling objective.
- Neighbour Generation: Neighbourhood generation takes a text, neighbour count, and replacement count as input, then uses BERT dropout and swap suitability scores to produce neighbours.The procedure evaluates word replacements and combines selected swaps into neighbour texts.
- Reference Models: The baseline suite includes LOSS attacks and reference models trained to represent general, candidate-distribution, and oracle-distribution knowledge.Candidate models use related but distribution-shifted data, whereas oracle models use the same corpora with different subsets.
- Reference Models: Reference-model data could not always match the target training-set size, reflecting the practical difficulty of obtaining suitable reference data.The authors report the reference-data counts in Table 1.
4 Results
The experiments evaluate attack performance at very low false-positive rates, reference-model quality, and neighborhood-attack design choices. Neighborhood attacks outperform realistic reference-based baselines and are competitive with oracle references, while their performance improves with more neighbors but favors single-word replacements.
- Main Results: TPR is reported at target FPR values of 1%, 0.1%, and 0.01%.
- Main Results: Neighborhood attacks substantially outperform candidate-reference attacks and can compete with or outperform oracle LiRAs, especially at low FPRs.The neighborhood attack requires no additional data, unlike reference-based attacks.
- Reference Model Quality: Reference-based attack performance peaks when reference-model validation PPL roughly matches target-model PPL.For AG News and Twitter, target-model perplexities are 30.0 and 84.7, respectively.
- Reference Model Quality: Substantial reference-attack gains emerge only when validation PPL nearly matches the target model, illustrating strong fragility to reference-model choice.
- Ablation Studies: Higher numbers of generated neighbors generally produce better neighborhood-attack results.The main results use the 100 most likely neighbors generated by BERT.
- Ablation Studies: Replacing one word clearly outperforms multiple replacements, while two- and three-word replacements show no highly meaningful difference.
5 Defending against Neighbourhood Attacks
The paper situates neighborhood attacks within efforts to defend models against privacy risks, including confidence perturbation, regularization, and differentially private training. DP-SGD is presented as a defense with mathematical privacy guarantees.
- Differentially private model training, including DP-SGD, is described as a defense offering mathematical privacy guarantees.
- The paper also identifies confidence-score perturbation and specific regularization techniques as defenses with good empirical performance.
6 Related Work
Related work covers membership inference attacks in machine learning and NLP, machine-generated-text detection, and attacks using information beyond confidence or loss scores. DetectGPT is noted as a perturbation-based method similar to the neighborhood attack.
- Membership Inference Attacks: Membership inference attacks may use confidence or loss scores, model parameters, or training-loss trajectories.
- Membership Inference Attacks in NLP: In NLP, membership inference attacks are also studied for language-model extraction and machine translation.
- Machine-Generated Text Detection: Machine-generated-text detection has attracted interest alongside the increasing use of writing assistants.
- Machine-Generated Text Detection: DetectGPT compares a sample’s likelihood with those of perturbed samples and hypothesizes lower likelihood for perturbations than for model-generated text.
7 Conclusion and Future Work
The paper finds that likelihood-ratio attacks are highly sensitive to reference-model quality and therefore require high-quality training data. It introduces neighborhood attacks, which use word-replacement neighbors instead of in-domain reference data and outperform reference-based attacks under realistic data-access assumptions.
- Likelihood-ratio attacks are highly fragile to reference-model quality and require attackers to access high-quality training data.
- Neighborhood attacks calibrate target-sample loss using plausible neighboring texts generated through word replacements, eliminating the need for in-domain reference data.
- Under realistic training-data access assumptions, neighborhood attacks consistently outperform reference-based attacks.
Limitations
The proposed attack is limited by its dependence on textual data and by uncertainty in the fairness of reimplemented baseline comparisons.
- Modality scope: The proposed neighbour-generation method is specific to textual data, limiting its direct applicability across modalities.Applying analogous augmentations such as rotations to visual data is not straightforward.
- Evaluation constraints: Baseline attack performance cannot be compared directly across papers because it depends on training procedures and model overfitting.The authors therefore reimplemented existing attacks for comparison.
- Evaluation constraints: The authors cannot guarantee that reimplemented baseline attacks were perfect, so comparisons with their method may not be completely fair.The paper states that the comparison is therefore not 100% fair.
Ethical Considerations
The paper frames membership inference attacks as privacy risks but argues that studying them is important for understanding threats and developing defenses. It reports no concerns about the datasets because they are publicly available and established in prior research.
- Privacy risks: Membership inference attacks can compromise the privacy of individuals whose data was used to train models.The paper identifies malicious use of such attacks as a privacy concern.
- Research rationale: Studying membership inference attacks is presented as important for understanding threat models and developing defenses against available attack tools.The paper links attack research to defenses that account for tools available to malicious actors.
- Dataset considerations: The authors report no issues with their datasets because they are publicly available and have long been used in NLP research or data science competitions.