PaperlayerView exact revision on arXiv

Source-linked AI summary

Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration

Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, Tao Jiang

arXiv:2311.06062v4cs.CLcs.CRcs.LG

TL;DR

Existing membership-inference attacks for LLMs face practical challenges from reference-data requirements and reliance on overfitting. SPV-MIA uses target-generated reference data and a memorization-based probabilistic-variation signal; across four representative LLMs and three datasets, it reports about 23.6% AUC improvement over existing MIAs.

  • Problem

    Existing MIAs face high false-positive rates in practical settings when reference data do not closely resemble training data and target models are not overfitting.

    Source: §1 · p. 2, §4.1 · p. 5

  • Method

    SPV-MIA combines a self-prompt reference model trained on target-generated texts with probabilistic variation assessed through paraphrasing.

    Source: §1 · p. 3, §4 · p. 5, §4.3 · p. 7

  • Results

    About 23.6% improvement in AUC across four representative LLMs and three datasets is reported, with the highest average AUC of 0.924 over all scenarios.

    Source: §1 · p. 3, §5.2 · p. 8

  • Takeaways & Limitations

    The paper reports that SPV-MIA reveals significant privacy risks in practical LLM applications.

    Source: §6 · p. 10

    The threat model requires query and fine-tuning API access, and reference-based attacks can use only a disjoint dataset from the same task.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Membership Inference Attacks (MIA) aim to infer whether a target data record has been utilized for model training or not. Existing MIAs designed for large language models (LLMs) can be bifurcated into two types: reference-free and reference-based attacks. Although reference-based attacks appear promising performance by calibrating the probability measured on the target model with reference models, this illusion of privacy risk heavily depends on a reference dataset that closely resembles the training set. Both two types of attacks are predicated on the hypothesis that training records consistently maintain a higher probability of being sampled. However, this hypothesis heavily relies on the overfitting of target models, which will be mitigated by multiple regularization methods and the generalization of LLMs. Thus, these reasons lead to high false-positive rates of MIAs in practical scenarios. We propose a Membership Inference Attack based on Self-calibrated Probabilistic Variation (SPV-MIA). Specifically, we introduce a self-prompt approach, which constructs the dataset to fine-tune the reference model by prompting the target LLM itself. In this manner, the adversary can collect a dataset with a similar distribution from public APIs. Furthermore, we introduce probabilistic variation, a more reliable membership signal based on LLM memorization rather than overfitting, from which we rediscover the neighbour attack with theoretical grounding. Comprehensive evaluation conducted on three datasets and four exemplary LLMs shows that SPV-MIA raises the AUC of MIAs from 0.7 to a significantly high level of 0.9. Our code and dataset are available at: https://github.com/tsinghua-fib-lab/NeurIPS2024_SPV-MIA

1 Introduction

The paper examines whether membership inference can expose privacy risks in fine-tuned LLMs under practical conditions, where existing attacks face reference-data and overfitting assumptions. It proposes SPV-MIA and reports improved attack performance across multiple models and datasets.

1 Introduction

  • Fine-tuning is identified as especially susceptible to privacy leaks because its datasets are relatively small and often private.The paper consequently examines membership inference against fine-tuned LLMs.Source: §1 · p. 2
  • LiRA performance plummets as reference-data similarity declines, while existing MIAs struggle to detect membership when target LLMs are not overfitting.The experiments compare reference datasets of identical, domain-specific, and irrelevant distributions; existing attacks rely on member records having higher sampling probabilities, an assumption satisfied only in overfitting models.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • SPV-MIA prompts the target LLM to collect reference data and uses probabilistic variation grounded in memorization rather than relying on overfitting.The paper also explains the neighbor attack through this memorization-based account and states that its paraphrasing model does not rely on another MLM.Source: §1 · p. 3
  • About 23.6% improvement in AUC across four representative LLMs and three datasets is reported for SPV-MIA over existing MIAs.The approach combines self-prompt reference data with probabilistic variation as a membership signal.Source: §1 · p. 3

2 Related Works

Prior work studied membership inference across classical models, recommendation and generation tasks, and language models. This paper focuses on privacy in fine-tuned LLMs and evaluates causal language models as a representative category.

Membership Inference Attack

  • The paper focuses on fine-tuned LLMs because their datasets are typically more private and sensitive, and evaluates prior attacks in practical scenarios.It reports that earlier attacks' privacy breaches were below expectations under their requirements and assumptions.Source: §2 · p. 3

Large Language Models

  • The paper situates LLM privacy measurement within broader concern about privacy risks accompanying widespread LLM use.The related-work discussion notes contemporaneous work quantifying LLM privacy risks.Source: §2 · pp. 3–4
  • Causal language models, including GPT and LLaMA, are selected for evaluation because they hold a dominant position among LLM categories.The passage distinguishes causal, masked, and sequence-to-sequence language models.Source: §2 · p. 4

3 Preliminaries

The preliminaries describe causal language models as autoregressive models trained by minimizing negative log probability, then specify the API access and membership-inference setting considered.

3.1 Causal Language Models

  • Causal language models predict each token from previous tokens and factorize a text's joint probability into conditional token probabilities.Their training objective minimizes negative log probability.Source: §3.1 · p. 4
  • The paper focuses on fine-tuning because those datasets are usually more private and vulnerable to adversaries.It describes pretraining followed by fine-tuning and notes parameter-efficient approaches such as LoRA and P-Tuning.Source: §3.1 · p. 4

3.2 Threat Model

  • The adversary's goal is to infer whether a record belonged to the target LLM's fine-tuning dataset, using query and fine-tuning API access.The query API provides generated texts and logits or loss; the fine-tuning API applies to the pre-trained target model.Source: §3.2 · p. 4
  • Reference-based attacks may fine-tune a reference model only on a disjoint dataset from the same task, and membership is represented by a thresholded indicator.The adversary has no prior information about which records were used for target fine-tuning.Source: §3.2 · p. 4

4 Membership Inference Attack via Self-calibrated Probabilistic Variation

SPV-MIA combines a self-prompt reference model with a memorization-based probabilistic-variation signal. Its components address the practical difficulty of obtaining a suitable reference dataset and the limits of probability signals tied to overfitting.

4 Membership Inference Attack via Self-calibrated Probabilistic Variation

  • SPV-MIA combines practical difficulty calibration with probabilistic variation assessment to address high false-positive rates in practical membership inference.The method uses a self-prompt reference model and a membership signal based on memorization rather than overfitting.Source: §4 · p. 5, §4.1 · p. 5

4.1 General Paradigm

  • Reference-free attacks use target-text probability, but inherently over-represented nonmembers can also score highly, while reference-based calibration depends on a similar reference dataset that is difficult to obtain.The proposed self-prompt reference model is trained on text generated by the target model.Source: §4.1 · p. 5, §4.2 · p. 6

4.2 Practical Difficulty Calibration (PDC) via Self-prompt Reference Model

  • Self-prompt calibration uses short, same-domain text chunks to elicit generated text from the target LLM, then fine-tunes a reference model on those generations.The authors state that one self-prompt reference model can provide sufficiently high attack performance and can still capture the underlying distribution with unrelated prompts in challenging cases.Source: §4.2 · p. 6

4.3 Probabilistic Variation Assessment (PVA) via Symmetrical Paraphrasing

  • The authors motivate memorization as a more reliable membership signal because it begins earlier than overfitting and can persist despite early stopping.They also note memorization is not exclusively harmful and may be important for some generalization tasks.Source: §4.3 · p. 6
  • Probabilistic variation characterizes local probability curvature using directional second derivatives, motivated by members having higher generation probability than neighboring records.Because direct second-derivative calculation may be infeasible for LLMs, the method approximates it with symmetrical paraphrases; experiments default to semantic-domain paraphrasing, while an embedding-domain option is also described.Source: §4.3 · p. 7

5 Experiments

Across four LLMs and three datasets, SPV-MIA outperforms the evaluated baselines; experiments also examine reference-dataset quality, practical robustness, defenses, and fine-tuning methods.

5.1 Experimental Setup

  • The experiments used four open-source LLMs, three datasets, LoRA fine-tuning, and comparisons against seven state-of-the-art MIAs.Target models were trained for 10 epochs and self-prompt reference models for four; target-model fine-tuning used early stopping to avoid overfitting and achieve generalization.Source: §5.1 · pp. 7–8

5.2 Overall Performance

  • AUC averaged 0.924 across all scenarios, TPR@1% FPR averaged 46.9%, and SPV-MIA improved AUC by 30% over LiRA-Canididate.The comparison notes that LiRA-Canididate assumes full access to the auxiliary dataset, while SPV-MIA needs only short text chunks from it.Source: §5.2 · p. 8, Table 1 · p. 8, Table 2 · p. 8

5.3 How MIAs Rely on Reference Dataset Quality

  • Reference-based MIA performance decreased from identical-distribution to domain-specific to irrelevant datasets, while self-prompt datasets scored only marginally below identical-distribution datasets.The results indicate that prior reference-based methods depend strongly on reference-dataset quality.Source: §5.3 · p. 9, Figure 3 · p. 9

5.4 The Robustness of SPV-MIA in Practical Scenarios

  • Using completely unrelated prompt texts reduced SPV-MIA performance by at most 3.6%, indicating robustness across prompt sources.Figure 4 compares SPV-MIA performance with different prompt-text sources.Source: §5.4 · p. 9, Figure 4 · p. 9
  • With only 1,000 queries, the self-prompt reference model achieved performance comparable to 10,000 queries; with only 8 prompt tokens, attack performance remained at 0.9.Performance trends with text length differed by source: it increased for identical-distribution text, initially increased then decreased for domain-specific text, and decreased for irrelevant text.Source: §5.4 · pp. 9–10, Figure 5 · p. 9

5.5 Defending against SPV-MIAs

  • DP-SGD mitigated privacy risks to some extent, but SPV-MIA still showed notable leakage risk and outperformed baselines under moderate privacy budgets.Table 3 reports SPV-MIA AUC against LLaMA fine-tuned with DP-SGD at different privacy budgets.Source: §5.5 · p. 10

5.5.1 Impact of Fine-tuning Methods

  • SPV-MIA maintained high AUC across the evaluated PEFT techniques, and MIA performance was positively correlated with the number of trainable parameters.The authors hypothesize that more trainable parameters let LLMs retain more complete memory of member records.Source: §5.5.1 · p. 10

6 Conclusion

The paper presents SPV-MIA as a practical attack combining self-prompted reference data with a membership signal based on memorization rather than overfitting. It reports experimental superiority and identifies adaptation beyond causal language models as future work.

6 Conclusion

  • SPV-MIA addresses practical weaknesses of existing MIAs with self-prompted reference data and a memorization-based signal rather than reliance on overfitting.The conclusion reports substantial experiments validating superiority over baselines and effectiveness in extreme conditions.Source: §6 · p. 10
  • The study's stated limitation is that SPV-MIA is designed only for causal language models, with adaptation to other LLMs left for future work.Table 3 reports AUC against LLaMA fine-tuned with DP-SGD at different privacy budgets; Table 4 reports AUC across PEFT techniques.Source: §6 · p. 10, Table 3 · p. 10, Table 4 · p. 10

A.1 Ethic and Broader Impact Statements

The authors identify potential misuse of SPV-MIA to infer whether text appeared in LLM training and report using public datasets in their experiments. They also make code public to support research on defenses.

A.1 Ethic and Broader Impact Statements

  • SPV-MIA could be misused to infer whether a specific text entry was used to train a target LLM.The experiments use widely used public datasets, whose records were already public, to avoid further privacy violations.Source: §A.1 · p. 18
  • The authors made their code public to support further work on defenses and call for attention to public-data privacy and security in LLM research.They describe the method as scalable and identify adaptation to other LLMs as an area for further exploration.Source: §A.1 · p. 18

A.2 Notations of This Work

Table 5 provides notations and their descriptions.

A.2 Notations of This Work

A.3 Detailed Pseudo Codes of Symmetrical Perturbation

The appendix specifies two symmetrical paraphrasing procedures for generating paired variants: embedding-domain noise and semantic-domain masking and paraphrasing.

A.3 Detailed Pseudo Codes of Symmetrical Perturbation

  • Embedding-domain perturbation adds and subtracts sampled Gaussian noise from each text embedding, repeating the paired perturbation N times.The procedure takes a Gaussian noise scale σ and token embedding matrix as inputs.Source: §A.3 · p. 19
  • Semantic-domain perturbation masks a fraction λ of tokens, fills masks with an MLM, and uses paraphrase differences to construct symmetrical variants.The algorithm decodes the resulting token sequences as the paired texts.Source: §A.3 · p. 19
  • Table 6 compares SPV-MIA performance across the different paraphrasing methods.Source: Table 6 · p. 20

A.4 Rethinking of the Neighbour Attack

The section connects neighbour comparing to probabilistic variation through their shared focus on memorization signals, while relaxing the need for symmetric paraphrase pairs.

A.4 Rethinking of the Neighbour Attack

  • Because opposite-direction symmetric neighbours are nontrivial to define, the reformulation represents neighbouring samples uniformly instead.This removes the symmetry requirement from probabilistic variation assessment.Source: §A.4 · p. 20
  • Embedding-domain paraphrasing achieves considerable performance gains, while semantic-domain paraphrasing yields a marginal advantage.Source: §A.4 · p. 20
  • Neighbour attack and probabilistic variation share the design motivation of detecting signals that indicate memorization of training samples.Source: §A.4 · p. 20

A.5.1 Ablation Study

The section reports ablation and supplementary analyses of SPV-MIA, finding that both proposed modules improve it and that PDC appears more critical than PVA, alongside low-FPR, ROC-curve, perplexity, and privacy-budget results. It also reports checklist compliance and states that the work involves no human subjects and requires no additional safeguards because the models and datasets are publicly accessible.

A.5.1 Ablation Study

  • Removing either practical difficulty calibration (PDC) or probabilistic variation assessment (PVA) reduces performance; PDC appears more critical, so the authors suggest omitting PVA to reduce public-API access.The ablation results are reported for GPT-J and LLaMA across three datasets in Table 7.Source: §A.5.1 · p. 20, Table 7 · p. 20

A.5.2 TPR@0.1%FPR

  • SPV-MIA and five prior methods are additionally evaluated using TPR@0.1%FPR across four LLMs and three datasets.The passage identifies Table 8 as the source but supplies no cell values or comparative winner.Source: §A.5.2 · p. 20, Table 8 · p. 21

A.5.3 AUC Curves

  • Raw ROC curves for SPV-MIA and baseline attacks on LLaMAs fine-tuned over three datasets are provided at linear and log scales.Figure 6 shows the top-three baselines at linear scale; Figure 7 shows three representative baselines at log scale.Source: §A.5.3 · p. 21, Figure 6 · p. 21, Figure 7 · p. 21

A.5.4 Performance of Target LLMs

  • The fine-tuned LLMs show no significant overfitting by training-versus-test PPL, and the appendix also reports performance across privacy budgets ϵ.The PPL comparisons and privacy-budget results are presented in Tables 9 and 10, respectively; no specific values are provided in the passages.Source: §A.5.4 · p. 21, Table 9 · p. 22, Table 10 · p. 22

A.5.5 Performance of Representative Baselines under Different Privacy Budgets

  • For the DP-SGD LLaMA model fine-tuned on Ag News, SPV-MIA maintains substantial MIA performance margins over three representative baseline attacks across privacy-budget settings.The comparison is reported in Table 11; the passage does not provide numeric values.Source: §A.5.5 · p. 21, Table 11 · p. 23

NeurIPS Paper Checklist

  • The authors say Appendix A.6 provides sufficient compute-resource information and that they reviewed and adhered to the NeurIPS Code of Ethics.The supplied passages do not specify whether results include error bars or confidence intervals.Source: NeurIPS Paper Checklist · pp. 27–28

10. Broader Impacts

  • The authors state that a section on ethics and broader impact follows the reference.The passage does not describe specific positive or negative societal impacts.Source: §10 · p. 28

11. Safeguards

  • The authors answer NA on safeguards, stating that all models and datasets used are publicly accessible.Source: §11 · p. 29

12. Licenses for existing assets

  • The authors state that creators or original owners of assets are properly credited.The supplied response does not specifically confirm that licenses and terms of use are stated or respected.Source: §12 · p. 29

13. New Assets

  • The authors say documentation for new assets is available in the supplementary materials or at an anonymized URL.Source: §13 · p. 29

14. Crowdsourcing and Research with Human Subjects

  • The authors state that the paper involves neither crowdsourcing nor research with human subjects.Source: §14 · p. 30

Related papers

Loading 2311.06062v4…