PaperlayerView exact revision on arXiv

Source-linked AI summary

Playing Whack-a-Mole with misconceptions about memorization, extraction, and copyright

A. Feder Cooper

arXiv:2609.09320v1cs.CY

TL;DR

The author argues that the headline memorization claims in Alignment Whack-a-Mole rely on a measurement procedure vulnerable to short matches and prompt leakage. The critique examines the metric, elicitation procedure, controls, and cost reporting, and concludes that the reported results do not support the paper’s broad copyright claims.

  • Problem

    The author questions whether the paper’s headline coverage metric validly measures memorization, since it counts short matches and may be affected by prompt leakage.

    Source: §4 · p. 4, §4.2 · p. 7

  • Method

    The author reviews the headline coverage metric and prompting procedure, and assesses whether the experiments include controls for false positives.

    Source: §4 · p. 4, §4.4 · p. 8

  • Results

    The author concludes that the headline coverage results risk inflation and do not support the paper’s broad claims about memorization.

    Source: §4 · p. 4

  • Takeaways & Limitations

    The paper’s claims are being considered as evidence in ongoing and potential future copyright litigation.

    Source: §1 · p. 2

    Varying fine-tuning data does not test whether the books were in the model’s original training data, so it cannot establish how much bmc@5 reflects false positives.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

After careful review, I'm confident the headline fine-tuning memorization results in Alignment Whack-a-Mole use an invalid measurement procedure. The book memorization coverage metric these headline results depend on counts sequence matches far shorter than what field standards consider valid evidence of memorization, and the prompting procedure used to elicit memorization runs the risk of leaking the text being "extracted" in the prompt. The paper doesn't include the negative-control experiments needed to see how much the results are inflated by false positives: claiming extraction success (and therefore memorization of training data) when matches between generations and training data may be due to other factors. Given these validity issues, the paper's claims that fine-tuning lets users extract substantial portions of copyrighted books, in a form that could substitute for the originals, aren't supported by the reported results. The failure to report the experiments' cost (an important component of the threat model) further compromises the copyright claims. I'm writing this note because, in the last month, (prospective) plaintiffs have reached out to me to ask about this paper. They're looking to cite this work as valid evidence in support of claims in ongoing and potential future copyright litigation.

1 Introduction

The author argues that Alignment Whack-a-Mole’s headline claims rest on methodological and presentation problems, while missing detail prevents judging how overstated they are. The critique is prompted by prospective plaintiffs seeking to cite the work in copyright litigation, and it is not a position on any particular copyright case.

1 Introduction

  • The author says the paper’s headline claims are unsupported by a measurement procedure and elicitation method that do not withstand scrutiny.The paper also omits details needed to determine whether the claims are seriously overstated or mostly wrong.Source: §1 · p. 1
  • The paper omits the experiments’ costs, an important part of assessing its results and threat model.The author describes this as an important omission that can confuse readers about what the results support.Source: §1 · p. 2
  • Prospective plaintiffs have sought to cite the paper in ongoing and potential future copyright litigation.The author says methodological critiques should not be read as a stance on any particular case involving memorization and copyright.Source: §1 · p. 2

2 Some background

The paper distinguishes memorization in model weights, extraction in model outputs, and market substitution as a separate copyright question. It explains why short text matches can be false positives and describes 50 tokens, roughly 37 words, as the field-standard minimum for an extraction claim.

2 Some background

  • Memorization concerns training data stored in model weights, while extraction is its reproduction during generation; extraction is commonly used to measure memorization.A sufficiently long near-verbatim reproduction is strong evidence of memorization, but not all memorized data are extractable.Source: §2 · p. 2
  • The field-standard minimum for an extraction claim is 50 LLM tokens, generally about 37 words.Shorter spans risk false positives from common phrases or prompts that contain or leak the text being generated.Source: §2 · p. 2
  • Figure 1 shows a near-verbatim book excerpt from Llama 3.1 70B that is longer than 37 words and cropped from a contiguous extraction spanning the book.The figure colors text present only in the ground-truth book red and text present only in the generation yellow.Source: Figure 1 · p. 2
  • Figure 2 illustrates fine-tuning ChatGPT to extract news articles, with generations counted as successful only at or beyond 50 tokens.The threshold is approximately 37 words.Source: Figure 2 · p. 3
  • A chatbot repeating text supplied in its prompt is not valid evidence of memorization, because it may simply be copying the prompt.Valid extraction research takes care to exclude such coincidences [3] [12].Source: §2 · p. 3
  • Market substitution asks whether a copy is complete and close enough to replace the original; memorization or extraction results do not automatically establish it.Source: §2 · p. 3

3 Fine-tuning on new data can reveal memorization of data the model was previously trained on

Prior work indicates that fine-tuning can surface training data a model had previously memorized. One described approach fine-tuned ChatGPT through its public API to alter alignment and elicit previously trained-on data.

3 Fine-tuning on new data can reveal memorization of data the model was previously trained on

  • Fine-tuning can surface previously memorized training data, and prior work reports memorization of copyrighted material in smaller open-weight and frontier production models [10] [1].Source: §3 · p. 3
  • Experiments fine-tuned ChatGPT through its public API to alter alignment and elicit data memorized during earlier training.Extraction success concerns earlier training data, not the data used for fine-tuning.Source: §3 · p. 3
  • Earlier work reports that aligned chatbot models do not tend to emit training data under standard extraction procedures, while jailbreaks and fine-tuning are other ways to circumvent alignment [21] [3] [16] [1] [21] [23].Source: §3 · p. 3

4 I’m confident the headline fine-tuning memorization results in Alignment Whack-a-Mole use an invalid measurement procedure

The headline bmc@5 measure counts short matches and may count text assembled from prompt summaries, so its high coverage figures do not establish near-total book memorization. The paper also lacks controls that would estimate how much of that signal is false positive.

4 I’m confident the headline fine-tuning memorization results in Alignment Whack-a-Mole use an invalid measurement procedure

  • The reported bmc@5 coverage reaches 85–90%, but it is distinct from the separate long contiguous reproductions and does not establish near-total book memorization.The critique identifies the coverage metric and its instantiation as an invalid basis for the claim.Source: §4 · p. 4

4.1 “Book memorization coverage” using the bmc@k metric

  • For bmc@5, summaries of held-out excerpts prompt 100 generations each, and matches of five or more consecutive words are pooled across the book to calculate coverage.The procedure compares generations against the whole book; a book position counts as covered if any surviving match lands on it.Source: §4.1 · pp. 4–6

4.2 Five “words” is too short as a minimum for claiming memorization

  • The five-word threshold falls below the field’s roughly 37-word standard, while the unreported match-length distribution obscures how much coverage comes from short matches.Twenty-four of 243 book-model pairs report 20–51% coverage while reporting zero same-excerpt, single-generation spans longer than 20 words.Source: §4.2 · pp. 6–7

4.3 The trimming procedure is insufficient for handling potential prompt leakage

  • Removing contiguous five-word prompt overlaps does not prevent a generation from reassembling book wording from names, events, or phrases supplied by its summary.The procedure can count such a match as memorization even when the model could have produced it by following the prompt.Source: §4.3 · pp. 7–8

4.4 Insufficient experiments to assess false positives

  • The synthetic-story comparison does not test whether books were in original training data, and the paper omits the negative control needed to estimate false positives.For The Handmaid’s Tale, GPT-4o fine-tuned on synthetic stories still has bmc@5 of 19.4%.Source: §4.4 · pp. 8–9

5 The threat model, cost, and claims about copyright

The author argues that the reported coverage does not establish that fine-tuning can produce book substitutes, and that omitted experiment costs matter to assessing practicality. The paper’s cost and coverage claims are challenged on those grounds.

5 The threat model, cost, and claims about copyright

  • Thousands, possibly tens of thousands, of dollars is the author’s conservative estimate for the full experiments, whose cost the paper does not report.The author argues cost is central to whether this kind of extraction is practical and to the copyright claims.Source: §5 · p. 9
  • The bmc@5 experiments involved 100 generations for each of 243 book-model pairs, producing over 24,000 books’ worth of text.The author identifies this scale as relevant to the missing cost information.Source: §5 · p. 9
  • For The Handmaid’s Tale, GPT-4o’s bmc@5 rises from 6.3% before fine-tuning to 19.4% after synthetic fine-tuning, while its longest span stays 18 words.The author argues this contrast undercuts bmc@5 as evidence of memorizing substantial portions of books.Source: §5 · p. 9
  • The authors characterize the generated copies as requiring little effort and potentially substituting for the source works, claims the author says the experiments do not support.The author contrasts those claims with the expense of obtaining scattered fragments versus buying a complete book for less.Source: §5 · p. 10
  • The reported 85–90% bmc@5 coverage is a scattered, potentially out-of-order patchwork that requires the source book to interpret, not evidence of a book substitute.The prompts also use summaries built from ground-truth excerpts, and reassembling snippets in sequence requires the book.Source: §5 · p. 10

6 Some closing thoughts about the field

The author describes an increasingly high-stakes AI-and-copyright debate shaped by misconceptions and personal stakes, while emphasizing the complexity of the issues and stepping back from new work in this area.

6 Some closing thoughts about the field

  • The author cautions that models can reproduce memorized training data, but whether and how much that matters to courts remains uncertain.The author distinguishes technical evidence from the unsettled question of legal rulings on infringement.Source: §6 · p. 10
  • The paper’s largest reported single-generation verbatim span is around 460 words, which the author says is not close to a whole book.Source: §6 · p. 10
  • The author argues that personal stakes in copyright litigation make disclosure of funding and potential conflicts of interest a bare minimum research-ethics norm.The author says a growing share of work in the field comes from people with a stake in active cases.Source: §6 · p. 11
  • The author says misconceptions about memorization and extraction, alongside sloppiness or motivated argument, make careful evidence increasingly difficult to produce.The author describes spending much of their time correcting these misconceptions.Source: §6 · p. 11
  • The author plans to stop starting new work in this area for now, citing the time and effort required, while remaining available to answer questions.The author says they will turn to other areas where they hope to contribute usefully.Source: §6 · p. 11
Loading 2609.09320v1…