PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Membership Inference Attacks Work on Large Language Models?

Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, Hannaneh Hajishirzi

arXiv:2402.07841v2cs.CL

TL;DR

The paper addresses limited evidence about whether membership inference attacks work on LLM pre-training data. It evaluates five attacks across Pile-trained LMs from 160M to 12B parameters and finds that attacks barely outperform random guessing in most settings, while identifying training-scale, overlap, and distribution-shift factors behind apparent vulnerability.

  • Problem

    Evidence about membership inference on LLM pre-training data remains limited despite extensive study of attacks on traditional machine-learning models.

  • Method

    The paper conducts a large-scale evaluation of five MIAs across LLMs trained on the Pile, using models ranging from 160M to 12B parameters and diverse benchmarks.

  • Results

    MIAs perform near random across most domains, with no single attack or target model exceeding 0.6 AUC in most settings; apparent success in selected settings likely reflects distribution shift.

  • Takeaways & Limitations

    Large datasets, near-one-epoch training, and high member–non-member similarity make membership boundaries fuzzy and complicate leakage measurement for generative models.

  • Takeaways & Limitations

    Non-members may still have 100% n-gram overlap without being members because n-grams can occur in different member samples, especially for smaller n.

Abstract

from arXiv · show

Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data. Despite extensive research on traditional machine learning models, there has been limited work studying MIA on the pre-training data of large language models (LLMs). We perform a large-scale evaluation of MIAs over a suite of language models (LMs) trained on the Pile, ranging from 160M to 12B parameters. We find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains. Our further analyses reveal that this poor performance can be attributed to (1) the combination of a large dataset and few training iterations, and (2) an inherently fuzzy boundary between members and non-members. We identify specific settings where LLMs have been shown to be vulnerable to membership inference and show that the apparent success in such settings can be attributed to a distribution shift, such as when members and non-members are drawn from the seemingly identical domain but with different temporal ranges. We release our code and data as a unified benchmark package that includes all existing MIAs, supporting future work.

1 Introduction

The paper finds that membership inference against pretrained LLMs is difficult because large-scale pretraining weakens data imprints and blurs the boundary between members and non-members. It also argues that modified, closely related samples can expose information leakage that standard membership definitions miss.

  • Massive training data and near-one-epoch training considerably decrease current MIA performance on LLMs.These characteristics may leave less imprint of member data on the model.
  • MIA success from prior settings transfers poorly to pretrained LLMs, seemingly because member data is not memorized distinctly.Overlap between members and non-members further reduces distinguishability.
  • Non-members can closely resemble members: Pile Wikipedia and ArXiv test samples exceed 30% average 7-gram overlap.Non-members with lower n-gram overlap are more distinguishable by existing MIAs.
  • High MIA performance in some prior work likely reflects temporal distribution shifts between seemingly in-domain members and non-members.Different temporal ranges can alter n-gram overlap and make candidate sets easier to distinguish.
  • Existing MIAs often classify modified members as non-members with high confidence, motivating membership definitions that account for fuzzy members.The modified members preserve lexical and/or semantic similarity while changing only a tiny fraction of tokens.

2 Background

Membership inference attacks estimate whether a sample belongs to a model’s training data, typically by scoring the sample and thresholding that score. The paper situates five attacks in this framework while noting that stronger reference-based approaches can be costly for LLMs.

  • An MIA computes a membership score f(x; M) for a data point x and thresholds it to infer training-set membership.
  • MIAs serve as a proxy for evaluating whether machine-learning models leak information about their training data.They are a widely used threat model for machine-learning privacy.
  • Shadow-model attacks such as LiRA can be promising but require multiple model copies, often making them intractable for LLMs.Other stronger MIA assumptions include white-box access or ground-truth member and non-member samples.
  • The evaluation considers five attacks: LOSS, reference-based, Zlib Entropy, Neighborhood, and Min-k% Prob.These attacks respectively use model loss, calibrated loss, compression size, perturbed-neighbor curvature, or the lowest-likelihood tokens.
  • Table 1 reports AUC ROC for MIAs against PYTHIA-DEDUP, with performance below .6 in most domains.The table bolds the highest-performing attack per domain and points readers to TPR@low%FPR results in Table 11.
  • MIA advantage is used as a proxy for information leakage and memorization, whereas extractability has increasingly become synonymous with memorization in LLM research.

3 Membership Inference on LLMs is Difficult

Across Pile-trained LLMs, existing membership inference attacks generally remain near random, with only modest improvements from larger models and substantially weaker performance after deduplication. The paper attributes this difficulty to limited training iterations, large pretraining corpora, and fuzzy member–non-member boundaries caused by natural text overlap.

  • Main Results: Five MIAs evaluated across Pile-trained models up to 12B parameters perform near random for most domains.No single attack or target model achieves AUC above 0.6 except in the Github domain.
  • Main Results: MIA performance tends to increase with target model size, while deduplicating training data reduces performance.These trends appear in the reference-based attack and similarly for other attacks.
  • Characteristics of LLMs: MIA performance initially rises sharply during training, then declines across successive checkpoints within approximately one epoch.The authors associate this trajectory with early overfitting followed by improved generalization as the model sees more data.
  • Characteristics of LLMs: Increasing effective epochs increases attack performance, suggesting multi-epoch training or large upsampling factors can increase training-data leakage.This result is reported for the DATABLATIONS setting, while other training regimes can plateau or decline.
  • Inherent Ambiguity in MIA: Natural non-members often have substantial 7-gram overlap with training data, reaching mean overlaps of 32.5% in Wikipedia, 39.3% in ArXiv, and 41.0% in PubMed Central.Github, DM Mathematics, and FreeLaw have even higher mean overlaps of 76.9%, 72.8%, and 62.3%, respectively.
  • Inherent Ambiguity in MIA: Restricting non-members to at most 20% 7-gram overlap improves MIA performance, showing that natural overlap makes membership harder to distinguish.The paper frames this overlap as an intrinsic property of natural language rather than merely a flaw in the Pile split.

4 Importance of Candidate Set Selection

MIA performance depends strongly on how non-members are selected: temporal shifts create distribution differences that can make attacks appear more successful than under natural candidate sets.

  • Candidate-set selection: Prior work’s high MIA performance may reflect unintended distribution shifts between members and non-members.The paper investigates candidate selection as one explanation for reported results above .7 AUC ROC.
  • Temporal shift: Temporally shifted benchmarks use pre-March 2020 Wikipedia members and Wikipedia non-members created from August 2023 onward.This setup replaces the natural non-member set with samples from the RealTimeData WikiText dataset.
  • Temporal shift: Temporally shifted settings yield significantly higher MIA performance than benchmarks whose members and non-members share the same temporal range.MIA performance generally increases as non-members are further temporally shifted from member data.
  • Overlap distribution: 39.3% versus 13.9%: natural and temporally shifted Wikipedia non-members’ average 7-gram overlap, respectively.The shifted distribution concentrates at lower overlap percentages than the natural non-member distribution.
  • Benchmark construction: Candidate non-members should be compared with left-out pretraining samples to assess whether their n-gram overlap distribution represents the member domain.Thresholds based on temporally shifted non-members test temporal shift rather than membership.

5 Revisiting Membership

Exact membership can miss meaningful information leakage in generative models because lightly modified members may remain close to training records while MIAs classify them as non-members.

  • Membership definitions: Standard membership inference treats only records seen exactly during training as members, which may not capture information leakage relevant to adversaries and auditors.The paper therefore studies approximate membership using sufficiently close samples.
  • Modified members: Modified members are generated through random token replacement or semantically similar token replacement across ArXiv and Wikipedia.Figure 5 compares LOSS and Reference-based score distributions for members, non-members, and these modified members.
  • Lexical distance: Edit distance as low as n = 1 produces extremely low modified-member FPRs, causing MIAs to classify lightly altered samples as non-members.This occurs even though such samples may be effectively members from an information-leakage perspective.
  • Semantic distance: Semantically close neighbors have MIA scores closer to actual member distributions, especially for the Reference attack.The findings motivate distance functions and membership games that combine lexical and semantic similarity.

6 Conclusion

The paper concludes that membership inference against pretrained LLMs is difficult and that benchmark design should account for fuzzy membership and distributional similarity.

  • Conclusion: Large datasets and near-one-epoch training considerably decrease current MIA performance.These pretraining characteristics may limit how much member data is imprinted in the model.
  • Conclusion: Similarity between in-members and out-members makes membership distinctions fuzzy and can confound leakage measurement.The paper connects this ambiguity to natural-language overlap and large datasets.
  • Conclusion: The membership inference game should be extended for generative models to better align with leakage concerns such as user-level leakage and PII.The paper identifies these as information adversaries and auditors may care about.
  • Resources: The released Python package provides modular support for implementing and evaluating existing and new language-model membership attacks.It includes attack abstractions, data-processing utilities, broad model support, and configuration files.
  • Evaluation scope: The evaluation context spans model suites with different training regimes, including Pythia, deduplicated Pythia, SILO, DATABLATIONS, GPT-Neo, and OLMO.These suites vary in parameter count, datasets, deduplication, epochs, and training-token budgets.

A.3 Benchmark Details

The benchmark samples bounded-length member and non-member documents from corresponding training and held-out splits, with additional overlap filtering and checkpoint-based evaluation.

  • Sampling: 1,000 members and non-members are sampled per target domain, while aggregate Pile experiments use 10,000 of each.Documents exceed 100 words and are truncated to 200 words from the beginning.
  • Sampling: DATABLATIONS benchmarks use members and non-members from the C4 training and validation sets, respectively.The same benchmark-generation pipeline is applied to these models.
  • Decontamination: Additional Pile decontamination uses a bloom filter with n = 13 and a threshold of ≤80% overlap.This filtering removes candidate records meeting the specified n-gram inclusion criterion.
  • Checkpoint evaluation: Checkpoints are selected every 5,000 steps through step 95,000, with additional checkpoints at steps 1,000 and 99,000.Each checkpoint uses the same 1,000-sample Pile test non-member set for evaluation.

A.3.2 Temporal Benchmarks

The temporal benchmarks construct non-members from later Wikipedia and ArXiv material while keeping members from the Pile, enabling evaluation of temporal distribution shifts and their effects on MIAs.

  • Benchmark construction: Temporal Wikipedia non-members come from articles created between August 12, 2023 and January 8, 2024, while members are sampled from Pile Wikipedia.
  • Benchmark construction: Temporal ArXiv members are sampled from Pile papers posted before July 2020, while non-members come from specified later months through June 2023.
  • Benchmark construction: The temporal experiments vary non-member dates while fixing the member set for each ArXiv benchmark, though fixing members across training steps would require expensive repeated runs.
  • Attack setup: MIAs assign a score from the target model’s token probabilities or loss, then threshold that score to classify samples as members or non-members.
  • Attack setup: The evaluation includes LOSS, reference-based, Neighborhood, Min-k% Prob, and Zlib Entropy attacks, with performance averaged over 1,000 bootstrap samples using AUC ROC and TPR@low%FPR.
  • Model comparisons: Across GPT-NEO models, MIA performance generally increases with model size but remains similar to or lower than comparable PYTHIA models.

A.7 Results with OLMo models

The OLMo experiments extend evaluation to DOLMA domains, while the reported results show near-random MIA performance across model sizes and continued dependence on attack calibration.

  • Benchmark construction: The OLMo benchmarks sample from DOLMA domains including Wikipedia, C4, Reddit, Common Crawl, and peS2o abstracts and full papers.
  • Results: OLMo results generally show near-random MIA performance, with the 1B- and 7B-parameter variants exhibiting similar performance.
  • Attack comparisons: Reference-based attacks can perform worse with an unsuitable reference model, reinforcing the difficulty of selecting reference models for different LLMs.
  • Attack comparisons: Reference-based attacks show smaller performance increases than non-reference attacks in some high-overlap domains because outlier non-members can also receive high losses from the reference model.
  • Attack comparisons: Calibration-based attacks may weaken when target and reference models both treat out-of-domain non-members as difficult, but can strengthen signals for recently or heavily memorized members.

B.3 GitHub as an Outlier

GitHub is an apparent MIA outlier because repetitive code and aggressive overlap filtering produce unusually atypical non-members, making lexical boundaries difficult to interpret.

  • GitHub performance: The best GitHub attack achieves an AUC ROC of ∼.70, unlike most domains where performance is near random.
  • GitHub performance: GitHub’s ≤80% 13-gram decontamination threshold removes only a small percentile of naturally high-overlap non-members, selecting domain outliers.
  • Overlap effects: Repetitive code elements such as copyright notices, function definitions, and HTML syntax contribute to GitHub’s high n-gram overlap.
  • Interpretation: Lexical non-member boundaries can make high-overlap-domain MIA performance ambiguous, motivating semantic distinctions between members and non-members.
  • Overlap effects: Temporally shifted ArXiv non-members have lower overlap than original non-members, with average 7-gram overlap of 22.7% in 2020-08 and 20.5% in 2023-06.
  • Temporal effects: MIA performance rises from .723 to .795 AUC ROC across the 2020-08 and 2023-06 ArXiv benchmarks, although the small overlap decrease does not clearly explain the increase.

C.1 Recency of Member Samples

MIA performance depends on when member samples were seen during training: recently seen members are easier to identify, while earlier members produce convergent, lower trajectories.

  • Recency effects: More recently seen member data yields slightly higher MIA performance for the fixed PYTHIA-DEDUP-12B checkpoint at step 99000.
  • Interpretation: The authors connect stronger recent-member signals with better preservation of recently seen training data in model parameters and weaker memorization of earlier data.
  • Recency effects: MIA performance trajectories decline more quickly for smaller models, but converge when evaluating members from much earlier in training.
  • Practical scope: The recency pattern may also make data seen during fine-tuning or continued pre-training increasingly vulnerable to membership inference.

C.2 Number of Training Epochs

As effective epochs increase, MIA performance is domain-dependent: it may initially improve, then plateau or decline, and can decrease throughout. The authors suggest domain representation, model size, and data characteristics may contribute, but note that further investigation is needed.

  • C.2 Number of Training Epochs: HackerNews MIA performance initially increases with effective epochs, then plateaus or drops as training continues.HackerNews comprises 5.9% of SILO-PDSW training data, and the authors speculate that the 1.4B-parameter model later memorizes fewer HackerNews samples.
  • C.2 Number of Training Epochs: The realistic setting increases the target model’s training data alongside effective epoch count using SILO-PDSW checkpoints.This differs from the fixed-training-data-size DATABLATIONS experiments.
  • C.2 Number of Training Epochs: DM Mathematics MIA performance decreases as effective epochs increase.The authors speculate that its 3.5% training-data share and concentrated, symbolic token space make specific-sample memorization unlikely.
  • C.2 Number of Training Epochs: Further investigation is needed to explain the counter-intuitive epoch trends across HackerNews and DM Mathematics.The authors specifically identify the target domains and attack-setting setup as requiring additional study.

D Revisiting Membership through Semantic Similarity

The paper tests whether semantic or lexical modifications of members affect membership inference. Paraphrased members remain distinguishable in score distributions but overlap with original members and non-members, and high-confidence attacks often classify them as non-members.

  • Semantic Similarity: GPT4 paraphrases preserve semantic similarity while changing the style and much of the original members’ wording.Five paraphrase trials are generated per member across ArXiv, Wikipedia, and HackerNews using domain-specific prompts.
  • Semantic Similarity: Paraphrased-member score distributions overlap noticeably with original-member and non-member distributions for LOSS and Reference-based attacks.The distributions are distinguishable but not cleanly separated across ArXiv, Wikipedia, and HackerNews.
  • Semantic Similarity: At high-confidence thresholds, paraphrased members are likely to be classified as non-members.Table 10 evaluates false-positive rates when modified members are treated as non-members.
  • High-Confidence Leakage: Less than 3% true-positive rate at 1% false-positive rate is generally reported for MIAs against PYTHIA-DEDUP, with GitHub as an exception.The table caption describes high-confidence leakage as generally low and GitHub as considerably higher for most attacks.
  • Training and Distribution Effects: Increasing model size slightly boosts MIA performance, while deduplication decreases it; more training data produces an early increase followed by gradual decline.These trends are reported across the model-size and training-data-seen plots.
  • Lexical Overlap: N-gram overlap distributions are reported across evaluation domains, including a GitHub non-member outlier under the 13-gram threshold.The outlier is from a Google language-resource repository rather than the code-dominant GitHub domain.
Loading 2402.07841v2…