Source-linked AI summary
Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning
Mustafa Safa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, Rahul Gupta
TL;DR
LLMs can expose memorized training data through extraction, raising privacy risks. The paper uses continuous prompt-tuning for extraction attacks and defenses, reporting that attacks outperform the baseline and defenses offer competitive privacy-utility trade-offs without changing model weights.
Problem
LLMs memorize training data that can be extracted with crafted prompts, creating privacy risks for data contributors.
Source: §1 · p. 1Method
Results
The attack consistently outperforms the baseline in exact extraction rate, while the defense provides competitive privacy-utility trade-offs.
Source: §5 · p. 5Takeaways & Limitations
Prompt-tuning offers an efficient way to control extraction without changing original model weights.
Source: §5 · p. 5Experiments use one dataset and one model family, and assess extraction metrics without deeper analysis of extracted sequences.
Source: §6 · p. 5
Abstract
from arXiv · showhide
Large Language Models (LLMs) are known to memorize significant portions of their training data. Parts of this memorized content have been shown to be extractable by simply querying the model, which poses a privacy risk. We present a novel approach which uses prompt-tuning to control the extraction rates of memorized content in LLMs. We present two prompt training strategies to increase and decrease extraction rates, which correspond to an attack and a defense, respectively. We demonstrate the effectiveness of our techniques by using models from the GPT-Neo family on a public benchmark. For the 1.3B parameter GPT-Neo model, our attack yields a 9.3 percentage point increase in extraction rate compared to our baseline. Our defense can be tuned to achieve different privacy-utility trade-offs by a user-specified hyperparameter. We achieve an extraction rate reduction of up to 97.7% relative to our baseline, with a perplexity increase of 16.9%.
1 Introduction
LLMs can memorize training data that prompts extract, creating privacy risks for data contributors. This work uses prompt-tuning to study extraction attacks and reduce extraction through defenses.
1 Introduction
- LLMs memorize training data that can be extracted with crafted prompts, posing privacy risks to contributors.The paper frames methods to control extractability as valuable for developers.Source: §1 · p. 1
- The authors train continuous soft-prompts to increase or decrease extraction while freezing model weights, enabling compute-efficient attacks and defenses.Defense prompts can be tuned for different privacy-utility trade-offs with a user-specified hyperparameter.Source: §1 · p. 1
- The paper presents what it describes as the first use of instructive prompts for privacy analysis and mitigation in LLMs.The authors also report releasing the code developed for their experiments.Source: §1 · p. 1
2 Background and Related Work
Prior work studied LLM memorization and extraction, while differential privacy can mitigate privacy risk at a cost to utility and retraining effort. This work uses continuous prompts to control extraction.
2 Background and Related Work
- Differentially private training has been used to mitigate privacy risk, but it can reduce utility and requires computationally burdensome LLM retraining.Earlier work also studied LLM memorization and methods for quantifying extractability.Source: §2 · p. 1
- Prompting research includes learned discrete natural-language prompts, mined prompts, and continuous prompts that need not be natural language.Instructive prompts have been used in pretraining, second-stage training, and inference.Source: §2 · p. 2
- This work uses continuous prompts as an external signal to elicit more or less memorized data, corresponding respectively to extraction attacks and defenses.Source: §2 · p. 2
3 Method
The method trains continuous prompts while keeping model weights fixed. It uses prompts to increase extraction in an attack setting and to regulate extraction in a defense setting.
3 Method
- The attack is white-box, with access to target-model parameters, while the defense is black-box, interacting through an API; the paper does not test its defense against its own attack.Source: §3 · p. 2
3.1 Attack
- The attack trains a continuous prompt on training prefix-suffix sequences to increase exact suffix generation for a disjoint test set.The prompt is prepended to prefix embeddings, and only the prompt parameters are optimized; model parameters stay frozen.Source: §3 · p. 2, §3.1 · p. 2
- The attack compares CLM, which minimizes cross-entropy over the full sequence, with aligned CLM, which minimizes it over suffix tokens only.At inference, the learned prompt is prepended to test-prefix embeddings for generation.Source: §3.1 · p. 2
3.2 Defense
- The defense trains a prompt using learning threshold θ: it applies gradient ascent below θ, descent above θ, and stops when average epoch loss reaches or exceeds θ.Adjusting θ allows controlled increases in training loss and different privacy-utility trade-offs without retraining the model.Source: §3.2 · p. 3
4 Experiments
Experiments evaluate prompt-tuned extraction control on GPT-Neo models using a public benchmark. The attack improves exact extraction over baseline, while the defense reduces extraction with measured perplexity costs.
4 Experiments
- Experiments used GPT-Neo-125M and GPT-Neo-1.3B on a 15k-sequence benchmark, with a random 14k/1k train/test split and 50-token prefixes and suffixes by default.Each experiment was repeated over five runs with a new random split.Source: §4 · p. 3
4.1 Attack
- With aligned CLM and a 100-token prompt, exact extraction reached 25.8% for GPT-Neo-125M and 54.3% for GPT-Neo-1.3B, gains of 8.9 and 9.3 percentage points over baseline, respectively.Both CLM and aligned CLM prompts improved over baseline, with aligned CLM performing best.Source: §4.1 · p. 3, Figure 2 · p. 4
- Extraction rates rose with prompt length and tended to saturate after 100 tokens; no increase in test loss was observed to explain the saturation.The authors suggest a limit to how many prompt parameters add value for extraction under their objective.Source: §4.1 · p. 3, Figure 2 · p. 4
- With 50-token prefixes, exact extraction rates decreased roughly exponentially as suffix size increased, while the 1.3B attack's gain over baseline grew from 5.3 to 9.3 percentage points as suffix size rose from 5 to 50.Longer prompts (at least 20 tokens) provided greater improvements over baseline as suffix size increased.Source: §4.1 · p. 3, Figure 2 · p. 4
- With suffix size fixed at 50, extraction rose roughly logarithmically with prefix size, but the attack's gap over baseline narrowed as prefixes grew.The authors interpret this pattern as greater benefit for a less-informed adversary with smaller prefixes.Source: §4.1 · p. 3, Figure 2 · p. 4
- Increasing beam size from 1 to 5 raised extraction across models, with later gains plateauing or oscillating; GPT-Neo-1.3B reached 61.4% at beam size 20 and prompt length 150.GPT-Neo-125M reached 28.3% at beam size 15 and prompt length 100.Source: §4.1 · pp. 3–4, Figure 2 · p. 4
4.2 Defense
- The defense reduced exact extraction by 99.4% for GPT-Neo-125M with a 25.3% PPL increase at θ = 1.75, and by 97.7% for GPT-Neo-1.3B with a 16.9% PPL increase at θ = 1.These results use the default 50-token prefix and suffix setting.Source: Table 1 · p. 4, §4.2 · p. 4
5 Conclusion
The paper introduces prompt-tuning to control memorized-data extractability in open language generation, developing an attack and a defense. The attack outperforms baseline, while the defense offers privacy-utility trade-offs without changing model weights.
5 Conclusion
- The work presents a prompt-tuning approach to control memorized-data extractability in open language generation, with attack and defense strategies.The authors describe it as the first known effort to use prompt-tuning for this purpose.Source: §5 · p. 5
- The attack consistently outperforms baseline in exact extraction rate, while the defense provides competitive privacy-utility trade-offs.The results were achieved without changing the original model weights.Source: §5 · p. 5
- The authors identify API owners with models trained on sensitive content as potential beneficiaries of the defense.Source: §5 · p. 5
6 Limitations
The experiments are limited to one dataset and one model family. The authors also identify analyses of extracted sequences, prompt convergence, and downstream-task utility as areas not covered.
6 Limitations
- The experiments use only a single dataset and a single model family.The authors attribute the dataset scope to the benchmark being the only publicly available dataset, to their knowledge.Source: §6 · p. 5
- The study measures extraction metrics but does not analyze extracted sequences or what the prompts converge to, including whether they are explainable.The authors note that these analyses could provide insights into memorization, extraction, and why aligned CLM performs better than basic CLM.Source: §6 · p. 5
- The defense evaluation does not measure other utility metrics, such as downstream-task accuracy.Source: §6 · p. 5
7 Ethical Considerations
The paper notes that its attack methodology could be misused by an adversary with white-box access to extract memorized private information. It aims to raise awareness and hopes developers will use relevant defenses against potential misuse.
7 Ethical Considerations
- The attack methodology could be misused by an adversary with white-box access to extract memorized private information from a target LLM.Source: §7 · p. 5
- The authors aim to raise awareness in the community about the possibility and severity of this attack.Source: §7 · p. 5
- The authors hope developers can use relevant defense mechanisms to avoid potential misuse.Source: §7 · p. 5
A Fractional Extraction Rate Results
Fractional extraction rate measures the share of generated tokens that are correct and in the right position across the dataset. Its results show the same trends as exact extraction rates across the experiments, including the defense setting.
A Fractional Extraction Rate Results
- Fractional extraction rate is the fraction of generated tokens that are correct and in the right position, measured over the dataset.Source: A Fractional Extraction Rate Results · p. 7
- Fractional extraction rate better assesses risk when an attacker values the meaning of a sequence, while exact extraction matters when an exact match is required.The paper gives credit-card-number extraction as an example where a few incorrect tokens can make the extraction useless.Source: A Fractional Extraction Rate Results · p. 7
- Fractional extraction results show the same trends as exact extraction rates across the experiments, including the defense setting.Fractional results are shown in Figure 3 and, for the defense, tabulated in Table 2.Source: A Fractional Extraction Rate Results · p. 7
B Training Setup
The training setup uses randomly initialized soft-prompts and specified optimization schedules for attack and defense. The reported figures and table describe extraction-rate changes across experimental factors and defense settings.
B Training Setup
- Soft-prompts are initialized to random word embeddings, trained with batch size 128 and Adam at learning rate 5e-4.Attack prompts are trained for 15 epochs; defense prompts train until loss stabilizes around the specified θ, which takes 2–3 epochs in these experiments.Source: B Training Setup · p. 7
- For model training, the authors use a PyTorch implementation that leverages HuggingFace Accelerate and DeepSpeed to handle distributed training over 8 GPUs.Source: B Training Setup · p. 7
- Figure 3 varies prompt length, suffix size, prefix size, and beam size; its top panels show GPT-Neo-125M and its bottom panels GPT-Neo-1.3B.Transparent polygons show 95% confidence intervals across points.Source: Figure 3 · p. 8
- Table 2 reports defense fractional extraction rates and corresponding perplexities for different θ values as mean ± standard deviation.The caption identifies lower extraction rates and lower perplexity than the corresponding similar-size GPT-2 variant as good.Source: Table 2 · p. 8
- Average prompt-training time on a p3dn.24xlarge instance is 0.9 hours per attack prompt and 0.02 hours per defense prompt.Source: B Training Setup · p. 8
C Future work
Future work targets stronger extraction attacks, broader prompt-learning comparisons, prompt transfer across models and datasets, and combinations of defenses to explore privacy-utility trade-offs.
C Future work
- Alternative loss objectives and prompt initializations may improve extraction rates in the attack setting.The authors identify both as directions for future work.Source: C Future work · p. 8
- Comparing other parameter-efficient and hard-prompt learning methods could support a more robust analysis of extraction rates.The authors propose exploring these alternatives.Source: C Future work · p. 8
- Testing trained prompts across different models and datasets remains a future direction.The authors identify transferability as untested.Source: C Future work · p. 8
- Combining the defense with training-time or decoding-stage defenses is proposed to explore privacy-utility trade-offs.The authors name differentially private training and decoding as examples.Source: C Future work · pp. 8–9