PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Unlearning for Mitigating Privacy Risks in Language Models

Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, Minjoon Seo

arXiv:2210.01504v2cs.CL

TL;DR

Language models can memorize private information, while existing mitigation methods require retraining or impose practical privacy-boundary assumptions. The paper proposes post hoc knowledge unlearning through gradient ascent on target sequences and finds effective forgetting with little degradation, sometimes improved performance, plus benefits from sequential unlearning and domain-sensitive analysis.

  • Problem

    Language models may expose memorized private information, while prevailing preprocessing and differential-privacy approaches require retraining the model.

  • Method

    Knowledge unlearning applies gradient ascent to target token sequences, using a few parameter updates and measuring extraction likelihood with a new metric.

  • Results

    Knowledge unlearning protects target sequences with little to no general-capability degradation, sometimes improving performance, while sequential and domain-specific factors affect forgetting difficulty.

  • Takeaways & Limitations

    Post hoc unlearning offers an efficient, robust privacy-mitigation option when vulnerable target sequences are known before extraction attacks.

  • Takeaways & Limitations

    Unlearning can degrade performance differently across domains, and the components determining why it works require further analysis.

Abstract

from arXiv · show

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger LMs; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method and a decoding method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. We release the code and dataset needed to replicate our results at https://github.com/joeljang/knowledge-unlearning.

1 INTRODUCTION

The paper proposes post hoc knowledge unlearning to reduce language-model privacy risks without repeatedly retraining the underlying model. Experiments show effective target-sequence forgetting, generally preserved capabilities, and domain- and procedure-dependent unlearning difficulty.

  • Pretrained language models can memorize and expose personally identifiable information and other sensitive training data through extraction attacks.Reported examples include names, phone numbers, email addresses, private clinical notes, licensed code, and UUIDs.
  • Existing data preprocessing and differential privacy approaches require retraining and rely on identifying private information or defining clear privacy boundaries.These constraints make repeated post-deployment responses to individuals’ Right-To-Be-Forgotten requests costly or difficult.
  • Knowledge unlearning reverses the language-modeling gradient direction on target token sequences, using a few parameter updates instead of retraining.The method is evaluated on GPT-Neo models from 125M to 2.7B parameters and protects targets from extraction attacks.
  • Knowledge unlearning provides strong privacy protection with little to no degradation of general capabilities, outperforming compared approaches in efficiency and robustness.The comparison includes data preprocessing and differential-privacy decoding; the latter causes severe modeling-performance degradation, while preprocessing is orders of magnitude more computationally demanding.
  • Sequentially forgetting chunks is easier than forgetting all samples at once, and unlearning difficulty depends heavily on the target domain.The paper also introduces a metric and guideline for quantifying extraction likelihood and deciding when a sequence is empirically forgotten.
  • +8.0% (37.6% →45.6%), +10.1% (57.4% →67.5%), and +7.9% (62.2% →70.1%) are reported Lambada improvements for GPT-NEO 125M, 1.3B, and 2.7B in extreme cases.These results indicate that unlearning can sometimes improve the underlying language model rather than merely preserve performance.

2 RELATED WORK

Prior language-model privacy work mainly uses data preprocessing or differential privacy, while machine-unlearning research has largely focused on image-classification classes. This paper positions token-sequence unlearning as a distinct, active forgetting problem for language models.

  • Language-model privacy methods mainly comprise data preprocessing or post-processing and differential privacy.Preprocessing sanitizes training data, while differential privacy bounds the influence of individual inputs during training.
  • Data deduplication can improve robustness against extraction attacks, but it is not the most suitable approach for every post-deployment forgetting request.The paper compares against Kandpal et al.’s deduplication method while emphasizing its practical constraints.
  • Differential privacy provides training-time guarantees, but applying existing descent-based methods to targeted post-deployment extraction protection is nontrivial.The challenge is especially relevant when individuals invoke the Right-To-Be-Forgotten after deployment.
  • Traditional machine-unlearning studies mostly forget entire classes in image-classification models, whereas this work forgets specific token sequences in language models.The paper distinguishes the settings by contrasting tens of image classes with token prediction over vocabularies of roughly 50,000 items.
  • Some prior work treats memorization as a privacy threat, while another line treats accumulated world knowledge as useful for knowledge-intensive tasks.These perspectives motivate distinguishing harmful memorization from beneficial model knowledge.
  • The paper frames knowledge unlearning as active forgetting, contrasting it with prior work that studies passive privacy benefits from when data appeared during training.It also differs in applying the approach to language models rather than image-classification models.

3 KNOWLEDGE UNLEARNING FOR LANGUAGE MODELS

The paper proposes gradient-ascent knowledge unlearning and metrics for evaluating whether target sequences remain vulnerable to extraction. It defines forgetting relative to unseen validation sequences and calibrates extraction strictness through n-gram overlap.

  • 3.1 Methodology: Knowledge unlearning negates the usual language-modeling objective by maximizing the loss on target token sequences.The method reverses gradient descent for the target sequence objective.
  • 3.2 Quantifying Privacy Risks of Language Models: Extraction Likelihood (EL) estimates extraction risk by averaging n-gram overlap between generated outputs and target sequences across varying prefix lengths.Generated outputs may terminate early when the end-of-sequence token appears.
  • 3.2 Quantifying Privacy Risks of Language Models: Varying the provided prefix length represents extraction attacks with different strengths, under the assumption that more prior information makes extraction easier.EL therefore estimates general extraction likelihood rather than dependence on one specific attack.
  • 3.2 Quantifying Privacy Risks of Language Models: The n-gram order n controls privacy strictness: larger n requires more overlap for an extraction attack to count as successful.Thus, n is treated as a tunable privacy-standard hyper-parameter.
  • 3.2 Quantifying Privacy Risks of Language Models: Memorization Accuracy (MA) measures how much the language model has memorized the specified token sequences.The metric was introduced for analyzing training dynamics in large language models.
  • 3.2 Quantifying Privacy Risks of Language Models: A target sequence is empirically considered forgotten when its EL_n and MA fall below the corresponding averages for sequences absent from training.The reference sequences come from a validation corpus not seen during training.

4 EXPERIMENTS

Experiments evaluate knowledge unlearning across GPT-Neo models, target domains, privacy metrics, and general-capability benchmarks. Results show that sequential unlearning improves stability, extraction attacks become far less effective, and domain and model size strongly affect outcomes.

  • 4.1 MODELS, DATASETS, AND CONFIGURATIONS: Experiments use GPT-Neo models of 125M, 1.3B, and 2.7B parameters, with OPT models serving as privacy-risk comparators.Target data comes from 15,000 extractable 200-token examples spanning 16 Pile domains; general capabilities are measured on 9 classification tasks.
  • 4.1 MODELS, DATASETS, AND CONFIGURATIONS: Unlearning is evaluated using extraction likelihood and memorization accuracy, with forgetting thresholds estimated from 10,000 unseen Pile validation instances.Forgetting thresholds define when target sequences are considered forgotten and unsusceptible to extraction attacks.
  • 4.2 MAIN EXPERIMENTS: 3,500,000x greater computational efficiency is reported for unlearning than retraining the underlying LM for data preprocessing.The comparison uses FLOPs and concerns the 2.7B setting discussed in the main results.
  • 4.2 MAIN EXPERIMENTS: Sequentially forgetting four chunks of 32 samples produces almost no average LM-performance degradation, unlike forgetting all 128 samples at once.The same instances are forgotten, but sequential unlearning reaches a vastly different final performance than batch unlearning.
  • 4.3 ANALYSIS OF KNOWLEDGE UNLEARNING: Structured domains such as ENRON EMAILS and GITHUB (CODE) seem to cause less LM-performance degradation than unstructured raw-English domains such as PUBMED CENTRAL.The authors state that the components driving this difference require further analysis.

5 CLOSING

The paper concludes that knowledge unlearning can mitigate privacy risks while preserving general capabilities in larger LMs. It frames post hoc unlearning as relevant to privacy requests because future privacy concerns cannot all be anticipated before pretraining.

  • 5 CLOSING: Knowledge unlearning provides strong privacy protection with little to no degradation of general capabilities for larger LMs.Capabilities are measured on 9 classification and 4 dialogue benchmarks.

A FULL RESULTS

The appendix supplies complete results for the study’s random-sampling and domain-analysis settings, plus dialogue-task evaluations for s = 32 across model sizes.

  • A FULL RESULTS: Full results are provided for 5 random samplings, the domain-analysis setting, and 4 dialogue tasks with s = 32 for all model sizes.These results supplement the main experimental tables.

B MEASURING PILE AND WIKITEXT PERPLEXITY

Additional analyses examine perplexity, computational cost, and learning-rate trade-offs. Gradient ascent raises perplexity despite preserved benchmark performance, while learning rate controls the balance between forgetting speed and capability retention.

  • B MEASURING PILE AND WIKITEXT PERPLEXITY: Knowledge-unlearned LMs show higher perplexity on 500 Pile and Wikitext validation samples despite no degradation on 9 LM benchmarks.The authors attribute this discrepancy to gradient ascent softening token-generation probabilities.
  • B MEASURING PILE AND WIKITEXT PERPLEXITY: Unlearning and pretraining FLOPs are compared using 6 x Total Training Tokens x Parameter Size.The appendix reports pretraining FLOPs for deduplication and average unlearning FLOPs until 32 sequences reach the forgetting threshold.
  • B MEASURING PILE AND WIKITEXT PERPLEXITY: 5e-5 is selected as the learning rate because higher rates forget faster but degrade performance, whereas lower rates retain performance but miss the forgetting threshold within 10 epochs.The trade-off is measured on GPT-Neo 1.3B with s = 32 over 10 epochs.

E TEXT EXAMPLE FROM EACH DOMAIN

This section presents examples from the eight domains used in the analysis and additional extraction-attack examples demonstrating knowledge-unlearning protection.

  • Examples are shown from each of the 8 domains used for the analysis.
  • Figure 4 varies the learning rate for unlearning GPT-NEO 1.3B with s = 32 and marks the model’s memorization-accuracy forgetting threshold.
  • Three additional extraction-attack examples illustrate how knowledge unlearning protects target sequences.

G ADDITIONAL RESULTS OF SEQUENTIAL KNOWLEDGE UNLEARNING

Sequential unlearning keeps earlier chunks forgotten while later chunks are forgotten faster than the initial chunk, with results reported across GPT-NEO models.

  • Later chunks are forgotten much faster—within one or two epochs—than the initial chunk during sequential unlearning.The authors hypothesize that similarity among the 15,000 benchmark token sequences may contribute to this pattern.
  • Chunks that are forgotten remain forgotten during subsequent sequential unlearning.
  • Figure 5 reports extraction-likelihood values for individual chunks and average language-model performance for GPT-NEO 125M, 1.3B, and 2.7B.

H THE EFFECT OF VARYING N FOR EXTRACTION LIKELIHOOD (EL) METRIC

This section evaluates Forgetting Threshold values for different n and reports average classification performance after unlearning 32 samples to those thresholds.

  • Forgetting Threshold values are measured for n = [5,10,20,40] on 10,000 validation instances unseen during training.
  • Average language-model performance is reported across 9 classification benchmarks after unlearning 32 samples until both EL and MA fall below their thresholds.Performance values are averaged over 5 random samplings.

I LIMITATIONS

The paper’s privacy guarantee is conditional: the Forgetting Threshold depends on the selected data samples, and real-world extraction attacks may require additional prevention methods.

  • The Forgetting Threshold depends on which data samples are selected as D′.
  • Varying prefix length provides only a naive way to vary extraction-attack strength, while real-world attacks may be more complicated.The paper notes that additional prevention methods may therefore be required.
  • The paper could not directly compare knowledge unlearning with differential-privacy pretraining because suitable open-source models were unavailable and reproducing DP pretraining was computationally expensive.The estimated cost was thousands of GPU hours, so the comparison is left for future work.
  • Future work should examine whether unlearning can harm the privacy of other users in large language models.This concern was suggested by prior machine-unlearning work in computer vision.
Loading 2210.01504v2…