PaperlayerView exact revision on arXiv

Source-linked AI summary

Avoiding Copyright Infringement via Large Language Model Unlearning

Guangyao Dou, Zheyuan Liu, Qing Lyu, Kaize Ding, Eric Wong

arXiv:2406.10952v3cs.CL

TL;DR

Sequential copyright takedown is underexplored, despite the need to process removal requests over time without degrading unrelated knowledge. SSU uses task-vector unlearning with random labeling and targeted weight updates; experiments report a better trade-off between unlearning efficacy and general-purpose language abilities than existing methods.

  • Problem

    Sequentially unlearning copyrighted content over time without retraining from scratch remains underexplored, and unintended weight drift can degrade non-targeted knowledge and general-purpose abilities.

    Source: §1 · p. 2

  • Method

    SSU fine-tunes on copyrighted books and random labels, adjusts salient weights, and negates the corresponding task vectors without collecting additional retained data.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    SSU achieves a superior trade-off between unlearning efficacy and general-purpose language abilities and consistently outperforms NPO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors identify improving unlearning robustness and exploring other copyright takedown methods as future research directions.

    Source: §8 · p. 9

    Previously unlearned book knowledge can re-emerge at later time steps, a phenomenon observed across all tested algorithms.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Pre-trained Large Language Models (LLMs) have demonstrated remarkable capabilities but also pose risks by learning and generating copyrighted material, leading to significant legal and ethical concerns. In real-world scenarios, model owners need to continuously address copyright infringement as new requests for content removal emerge at different time points. This leads to the need for sequential unlearning, where copyrighted content is removed sequentially as new requests arise. Despite its practical relevance, sequential unlearning in the context of copyright infringement has not been rigorously explored in existing literature. To address this gap, we propose Stable Sequential Unlearning (SSU), a novel framework designed to unlearn copyrighted content from LLMs over multiple time steps. Our approach works by identifying and removing specific weight updates in the model's parameters that correspond to copyrighted content. We improve unlearning efficacy by introducing random labeling loss and ensuring the model retains its general-purpose knowledge by adjusting targeted parameters. Experimental results show that SSU achieves an effective trade-off between unlearning efficacy and general-purpose language abilities, outperforming existing baselines.

1 Introduction

Sequential copyright takedown requires removing newly targeted content over time while avoiding drift that damages unrelated knowledge and general-purpose abilities. The paper proposes SSU and reports a better balance between unlearning and language retention than existing methods.

1 Introduction

  • Copyrighted material in pre-training data raises legal and ethical concerns, while screening it out remains challenging.The introduction cites the December 2023 New York Times lawsuit against OpenAI as an example of these concerns.Source: §1 · p. 1
  • Sequentially unlearning copyrighted content without retraining from scratch remains underexplored, and weight drift can degrade non-targeted knowledge and general-purpose abilities.The authors define stability as maintaining unlearning efficacy while preserving those abilities and knowledge.Source: §1 · p. 2
  • SSU fine-tunes on copyrighted books and random labels, targets salient weights, then negates the resulting task vectors without requiring additional retained-data collection.The approach is proposed for sequential settings to balance unlearning with maintaining general-purpose abilities.Source: §1 · p. 2
  • SSU achieves a superior trade-off between unlearning efficacy and general-purpose language abilities, while consistently outperforming NPO.The experiments also report that prompting and MemFree Decode fail to mitigate infringement risks.Source: §1 · p. 2
  • The paper formalizes sequential unlearning of copyrighted literary works and defines metrics for evaluating its effectiveness.It also evaluates existing methods, which either provide limited infringement mitigation or suffer catastrophic collapse.Source: §1 · p. 2

2 Related Work

Related work examines memorization and copyright risk, post-hoc mitigation, machine unlearning, and model editing. The paper identifies sequential unlearning of copyrighted literary works as an unaddressed setting.

LLM Memorization and Copyright

  • Prior studies connect LLM memorization with copyright infringement and examine fair use of foundation models.The cited work also notes that LLMs can memorize and reproduce pre-training data.Source: §2 · pp. 2–3
  • MemFree decoding may miss nonconsecutive verbatim content, while SILO’s datastore approach does not address retrieval-augmented-generation risks or the practical limits of restricting parametric training data.These approaches are discussed as ways to mitigate copyright risks.Source: §2 · p. 3

Machine Unlearning

  • Machine unlearning has been applied to harmful, private, and copyrighted content, but prior work has not addressed sequential unlearning of copyrighted literary works.Other cited work introduces benchmarks for forgetting private data and hazardous knowledge.Source: §2 · p. 3

Model Editing

  • Model editing differs from unlearning because it corrects errors without removing the underlying learned information; sequential editing can also lead to model collapse.The cited work proposes solutions to mitigate that collapse.Source: §2 · p. 3

3 Preliminaries

The preliminaries define machine unlearning and its gold baseline, describe task-vector negation, and formalize sequential forgetting across multiple time steps.

3.1 Machine Unlearning for LLMs

  • Machine unlearning modifies a model so it behaves as if it had not seen a targeted subset, while retraining without that subset is a gold baseline but is impractical for LLMs.The target data are denoted D_f and the unlearned model θ_u.Source: §3.1 · p. 3

3.2 Task Arithmetic

  • Task arithmetic fine-tunes the original model on unwanted data, computes the resulting task vector, and subtracts that vector from the original weights.The task vector is the element-wise difference θ_ft − θ_o.Source: §3.2 · p. 3

3.3 Unlearning with Multiple Time Steps

  • Sequential unlearning forgets requested data subsets over successive time steps while retaining the complementary data and minimizing each target subset’s influence.The cumulative forgotten set is the union of the per-step subsets, and the retained set is its complement in D.Source: §3.3 · p. 3

4 Methods

SSU unlearns books by fine-tuning on them, adding random-label loss, and selectively adjusting weights before negating the learned task vector. It repeats this process for each new request without requiring an additional retained dataset.

4.1 Learning Stable Task Vectors

  • SSU combines a forget-data loss with random-label loss, which pairs book inputs with mismatched outputs to support unlearning.The combined losses form the objective for the unlearning step.Source: §4.1 · p. 4
  • A gradient-based weight saliency map localizes updates to weights most influenced by the data to be forgotten.After fine-tuning, SSU extracts the task vector corresponding to the copyrighted books and negates it to unlearn them.Source: §4.1 · p. 5

4.2 Sequential Unlearning

  • At each new time step, SSU fine-tunes the previously unlearned model and repeats the process for each additional request.Unlike Gradient Difference, it does not require an additional retained dataset.Source: §4.2 · p. 5

5 Experimental Setup

The experiments simulate sequential removal of copyrighted books from two instruction-tuned models, evaluating leakage, retained knowledge, and general-purpose abilities. They compare SSU with prompting, decoding, and unlearning baselines.

5.1 Setting

  • At each time step, one book is unlearned using 200-token chunks, with the first 100 tokens as prompts and the next 100 as ground truth.For t > 1, Dprev aggregates previously unlearned books; Dnor contains 200 chunks from 100 other books.Source: §5.1 · p. 5, §5.3 · p. 6

5.2 Evaluation Metrics

  • Rouge-1 and Rouge-L measure output similarity to original book text; lower scores on Df and Dprev indicate greater transformativeness.The experiments also assess general-purpose language abilities using MMLU and MT-Bench.Source: §5.2 · p. 6

5.3 Datasets and Models

  • The study fine-tunes Llama3.1 and Mistral-7B on 10 Project Gutenberg books to create the vanilla models.Llama3.1 undergoes 10 unlearning steps; Mistral-7B is evaluated on six because many methods collapse by step 6.Source: §5.3 · p. 6

5.4 Baseline Methods

  • The baselines include system prompts, MemFree Decoding, Gradient Difference, and NPO, a preference-optimization approach.Gradient Difference uses books outside Df to maintain performance, while NPO treats Df as dispreferred data.Source: §5.4 · p. 6

6 Results

Across Llama3.1 and Mistral-7B, SSU achieves a better trade-off between unlearning efficacy and retained general-purpose language abilities. Its performance on targeted, previously unlearned, and non-targeted books varies by model and metric.

6.1 Sequential Unlearning of Books

  • For Llama3.1, SSU is among the most effective methods on Df and Dprev, outperforming NPO and Gradient Difference on Dprev.Results for Mistral-7B are similar; on that model, some methods achieve low Df Rouge at later steps but collapse.Source: §6.1 · pp. 6–7, Figure 3 · p. 7
  • Prompting and MemFree provide limited copyright mitigation, with Rouge scores often similar to the vanilla model across both models.Prompting (a) can yield higher Rouge on Llama3.1, while prompting (dbrx) is only marginally effective at some time steps.Source: §6.1 · p. 7

6.2 Non-Targeted Knowledge Retention

  • Non-targeted knowledge retention varies by model: on Mistral-7B, NPO retains more Dnor knowledge than SSU, while on Llama3.1 SSU retains more than the vanilla model.The authors suggest that Llama3.1's unexpected knowledge re-emergence may relate to knowledge redistribution and note that further research is needed.Source: §6.2 · pp. 7–8

6.4 Copyright Takedown Trade-offs

  • SSU achieves a better trade-off between unlearning efficacy and general-purpose language ability across both models.Figure 4 considers time steps greater than 1; its axes represent unlearning efficacy and general-purpose ability.Source: Figure 4 · p. 8, §6.4 · p. 8

6.3 General-purpose Language Abilities

  • GA, Gradient Difference, and TV show sudden degradation of general-purpose abilities at some time steps, while prompting and MemFree remain close to the vanilla model.The results describe the degradation as catastrophic collapse for the former methods.Source: §6.3 · p. 8

7 Analysis

The ablations examine how weight saliency and random-label loss affect sequential unlearning. The reported effects differ across targeted and non-targeted books and general-purpose abilities.

7.1 Impact of Weight Saliency

  • Without weight saliency, benchmark performance declines more sharply over time, indicating increased risk of catastrophic collapse as general-purpose abilities deteriorate.The ablation also reports lower average Rouge scores on both Df and Dnor without weight saliency.Source: §7.1 · p. 8

7.2 Impact of Random Labeling Loss

  • Without random-label loss, Df Rouge scores are higher while general-purpose abilities improve slightly; the authors report that random-label loss supports consistent Df unlearning across time steps.Removing either random-label loss or weight saliency also leads to lower average Rouge scores for Dnor in both models.Source: §7.2 · pp. 8–9

8 Conclusion

The paper investigates sequential unlearning of copyrighted content and proposes SSU, which uses random labeling loss and gradient-based weight saliency. Experiments report a better balance between unlearning efficacy and language retention than existing methods.

8 Conclusion

  • SSU uses random labeling loss and gradient-based weight saliency for more stable sequential unlearning.The paper frames this work as a way to mitigate copyright infringement in LLMs.Source: §8 · p. 9
  • Experiments report that SSU outperforms existing methods in balancing unlearning efficacy and language retention.Source: §8 · p. 9
  • The paper identifies improving unlearning robustness and exploring other copyright takedown methods as future research directions.Source: §8 · p. 9

Limitations

The authors identify limits in evaluation robustness, persistence of unlearning, retained capabilities, and the scope of unlearning alone as a copyright-risk measure.

Limitations

  • Lexical-based evaluation may give a false sense of privacy, and existing unlearning methods may obscure data rather than truly unlearn it.Source: Limitations · p. 9
  • Previously unlearned book knowledge re-emerged at some later time steps across all tested algorithms.Source: Limitations · p. 9
  • Despite its better trade-off, SSU still weakens reasoning and conversational abilities.The authors call for better preservation of general knowledge during sequential unlearning.Source: Limitations · p. 9
  • The authors argue that mitigating copyright risk requires more than unlearning, including complementary generation-time interventions and broader training-corpus search coverage.They also identify regular checkpoint updates as important for aligning with the right to be forgotten.Source: Limitations · p. 9

A.1 Experiment Settings

The appendix details the sequential unlearning evaluation, its lexical and capability-retention measures, and the tested baselines. It also reports that SSU remains stable across parameter settings, unlike gradient-ascent and gradient-difference methods, while the main experiments tune parameters for a trade-off between unlearning and retained abilities.

A.1 Experiment Settings

  • The evaluation uses book continuations to assess leakage and also measures performance on non-targeted books and general reasoning tasks.It considers 5-shot MMLU and MT-Bench; Dnor performance is termed knowledge retention.Source: §A.1 · p. 13

A.2.1 Rouge-1

  • Rouge-1 measures unigram overlap, with its experimental score calculated as the F1 score combining precision and recall.Precision and recall use overlapping unigrams relative to the hypothesis and reference text, respectively.Source: §A.2.1 · p. 13

A.2.2 Rouge-L

  • Rouge-L measures the longest common subsequence between a completion and the original book, using precision and recall to define its score.Recall and precision divide the LCS length by the reference-text and hypothesis-text lengths, respectively.Source: §A.2.2 · p. 13

A.3 Datasets

  • The experiment sequentially unlearns ten books; at later time steps, Dprev contains 200 randomly sampled QA pairs from earlier unlearning books, while Dnor contains 200 chunks sampled from 100 Project Gutenberg books excluded from Df.Source: §A.3.1 · p. 14, §A.3.2 · p. 14

A.4.1 Generic Prevention using System Prompt; A.4.2 MemFree Decoding

  • The tested prevention baselines use system prompts, while MemFree decoding resamples tokens when generated n-grams match training sequences.MemFree uses Bloom filters for real-time checks but does not fully eliminate paraphrased or approximate memorization.Source: §A.4.1 · p. 14, §A.4.2 · p. 14

A.4.3 Unlearning via Gradient Difference

  • Gradient Difference combines a forget-set gradient-ascent loss, a random-label loss, and a retained-data loss; Gradient Ascent seeks poor performance on the forget set.Gradient Difference requires additional retained data, while GA-based methods risk catastrophic collapse.Source: §A.4.3 · p. 15

A.4.4 Unlearning via Task Vector; A.4.5 Unlearning via NPO; A.5 Implementation Details

  • Task-vector unlearning subtracts fine-tuning weight changes from the original model, while NPO uses negative samples and approaches gradient ascent as β tends to zero.Across methods, the experiments use one epoch per time step, batch size 2, and LoRA; NPO uses β = 0.4.Source: §A.4.4 · p. 15, §A.4.5 · p. 15, §A.5 · p. 16

B.1 Impact of ϵ1 and ϵ2

  • SSU maintained strong MMLU and MT-Bench performance across tested parameter values and did not collapse, unlike gradient-ascent and gradient-difference methods.The main experiments tuned ϵ1 and ϵ2 for a trade-off between unlearning efficacy and general-purpose language abilities.Source: §B.1 · p. 16

B.2 Re-Emergence in Sequential Unlearning

The section tracks how knowledge of previously unlearned books changes as later unlearning requests are processed. Previously unlearned knowledge can re-emerge, including under SSU, underscoring a challenge for sequential unlearning.

B.2 Re-Emergence in Sequential Unlearning

  • Figures 7 and 8 track individual previously unlearned books by ROUGE-L across successive unlearning steps.The section examines each book separately rather than evaluating all previously unlearned books collectively.Source: §B.2 · p. 16, Table 1 · p. 17
  • Previously unlearned books can regain ROUGE-L similarity during later unlearning steps, indicating re-emergence of their content.NPO’s score rises for the fourth book in Figure 7d and the first book in Figure 7a; a steady increase also appears for Mistral in Figure 8a.Source: §B.2 · p. 17
  • SSU also exhibits some reappearance of previously unlearned knowledge as later, unrelated books are used for fine-tuning.The authors note that continual model modifications can lead to knowledge re-emergence and identify this as a challenge for sequential unlearning.Source: §B.2 · p. 17

C Full Experiment Results

Across sequential unlearning experiments, SSU achieves a superior trade-off between unlearning efficacy and general-purpose abilities without catastrophic collapse. The reported results also show that several baselines collapse, with different patterns across Llama3.1 and Mistral-7B.

D Numeric Experiment Results

  • Tables report Llama3.1 results across ten time steps and Mistral-7B results across six; GA, TV, and Gradient Difference show varying collapse.The numeric results section points to Tables 2–11 for Llama3.1 and Tables 12–17 for Mistral-7B.Source: D Numeric Experiment Results · p. 21

D.1 Catastrophic Collapse of Llama3.1

  • For Llama3.1, GA’s MMLU falls from 0.5821 to 0.3102 by time step 5, while TV’s MMLU reaches 0 at step 10.Gradient Difference’s MT-Bench falls from 8.034 at step 4 to 4.9438 at step 5 and 4.48 at step 10; SSU outperforms NPO across all time steps.Source: §D.1 · p. 21

D.2 Catastrophic Collapse of Mistral-7B

  • On Mistral-7B, GA collapses at step 1 and TV’s MMLU and MT-Bench reach 0 at step 5; Gradient Difference’s MT-Bench falls to 2.6815 by step 6.NPO maintains competitive general-purpose scores, while SSU shows superior unlearning efficacy and a better trade-off.Source: §D.2 · p. 21

C Full Experiment Results

  • SSU achieves a superior trade-off between unlearning efficacy and general-purpose language abilities, avoiding catastrophic collapse.The comparison spans methods and time steps; Figure 10 plots normalized average MMLU and MT-Bench against negated average Rouge-1 and Rouge-L scores on Df and Dprev, with the ideal in the top-right.Source: Figure 10 · p. 20, §D.2 · p. 21, §D.1 · p. 21
Loading 2406.10952v3…