PaperlayerView exact revision on arXiv

Source-linked AI summary

Extracting memorized pieces of (copyrighted) books from open-weight language models

A. Feder Cooper, Mark A. Lemley, Allison Casasola, Ahmed Ahmed, Aaron Gokaslan, Amy B. Cyphert, Christopher De Sa, Daniel E. Ho, Percy Liang

arXiv:2505.12546v6cs.CLcs.CYcs.LG

TL;DR

Copyright disputes over generative AI raise questions about how much protected expression LLMs memorize, but prior research often does not quantify memorization of particular works. This paper measures book-specific memorization across open-weight models and finds that most tested models do not memorize most tested books, while some striking exceptions complicate arguments on both sides.

  • Problem

    Prior extraction research often measures average training-data memorization rather than the memorization of specific works relevant to copyright claims.

    Source: §1 · pp. 1–2

  • Method

    The authors develop and validate a sliding-window extraction procedure and apply it to 200 Books3 books across 14 open-weight LLMs.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    Most tested LLMs do not memorize most books, but memorization varies by model and book, and LLAMA 3.1 70B can yield a near-verbatim Harry Potter copy.

    Source: §1 · p. 2, §9 · p. 17

  • Takeaways & Limitations

    The findings complicate copyright arguments on both sides, including defendants’ fair-use story and plaintiffs’ efforts to bring class-action lawsuits.

    Source: §1 · p. 2

    The study covers only 200 books and 14 models, so it cannot support strong conclusions about book memorization in general.

    Source: §8.1 · p. 16

Abstract

from arXiv · show

Plaintiffs and defendants in copyright lawsuits over generative AI often make sweeping, opposing claims about the extent to which large language models (LLMs) memorize protected expression from books in their training data. We show that these polarized positions dramatically oversimplify the relationship between memorization and copyright. To do so, we develop a technique to measure memorization of books, which we apply to 200 books and 14 open-weight LLMs. Through over 3000 experiments, we show that memorization varies both by model and book. With respect to our specific extraction methodology, we find that most LLMs do not memorize most books -- either in whole or in part; however, there are notable exceptions. For instance, Llama 3.1 70B entirely memorizes some books, like Harry Potter and the Sorcerer's Stone; memorization is so extensive that one can deterministically extract the whole book almost verbatim using the book's first few words as an initial prompt. We discuss why our results have significant implications for copyright cases, though not ones that unambiguously favor either side.

1 Introduction

The introduction frames book-specific memorization as relevant to copyright disputes, where prior research often measures aggregate training-data extraction rather than copying of particular works. The paper measures memorization across books and models, finding broad variation and some near-complete book extraction, with implications that complicate both sides’ arguments.

1 Introduction

  • The introduction links extracted training data and memorized model contents to distinct potential copyright consequences.It notes that a model itself could be deemed an infringing copy and could face destruction as a remedy.Source: §1 · p. 2
  • Prior extraction research often measures average memorization, while copyright infringement is evaluated with respect to specific works.The paper identifies this mismatch as a gap in evidence relevant to copyright claims.Source: §1 · p. 2
  • The paper proposes and validates a sliding-window procedure to measure how much and where open-weight LLMs memorize specific books.The main experiments apply it to 14 models and 200 Books3 books.Source: §1 · p. 2
  • Most tested LLMs do not memorize most tested books, but LLAMA 3.1 70B nearly entirely memorizes some, including Harry Potter.The introduction reports that Harry Potter can be deterministically extracted near-verbatim using only the first few words of its first chapter as a prompt.Source: §1 · p. 2
  • The findings complicate both defendants’ fair-use arguments and plaintiffs’ efforts to bring class-action lawsuits.The introduction says the implications do not unambiguously favor either side.Source: §1 · p. 2

2 Quantifying memorization through extraction

The section distinguishes extraction from the broader concept of memorization and explains how discoverable and probabilistic extraction quantify whether training text can be reproduced. It interprets the probability of exact suffix reproduction as extraction risk under a fixed prompt and decoding scheme.

2 Quantifying memorization through extraction

  • Successful extraction from model outputs is evidence of memorization, which also includes reconstructing training data by other means.Extraction concerns generation-time reconstruction; memorization is the broader concept.Source: §2 · p. 2

2.1 Probabilistic discoverable extraction

  • Discoverable extraction tests whether a model exactly reproduces a training sequence’s target suffix after receiving its prefix.The paper studies completion LLMs because this procedure is suited to models that continue prompt text.Source: §2.1 · p. 3
  • Probabilistic extraction measures the probability of generating the exact target suffix given a training-data prefix and decoding scheme.The probability is formed from conditional probabilities of the suffix tokens.Source: §2.1 · p. 3

2.2 Interpreting target suffix probabilities as extraction risk

  • For a fixed sequence and decoding scheme, higher suffix probability means greater extraction risk and more frequent expected reproduction.The paper’s Great Gatsby example reports the highest risk for LLAMA 3.1 30B among the plotted models.Source: Figure 1 · p. 3, §2.2 · p. 4

3 Memorization, extraction risk, and U.S. copyright

This section connects memorization and extraction to copyright questions about model copies and generated outputs, while distinguishing them from the fair-use question of training. Because extraction is probabilistic, the practical significance of a possible copy also depends on how often it can be produced.

3 Memorization, extraction risk, and U.S. copyright

  • The paper’s findings bear on whether models contain copies and whether copyrighted material appears in their outputs, but not on whether training is fair use.The study uses Books3 because many LLMs are known to have been trained on it.Source: §3 · p. 4
  • A model that memorizes all or a substantial portion of a copyrighted work near-verbatim may itself be or contain an infringing copy or derivative work.The passage notes that this possibility can matter even without a practical way to extract the content.Source: §3 · p. 5
  • Extractable memorized works can appear as copies in model outputs, and higher extraction probabilities mean fewer attempts are needed.Generated outputs are judged separately from the model itself.Source: §3 · p. 5
  • Because extraction is probabilistic, copyright relevance depends partly on how often a copy appears, not merely whether it could appear.The study uses a conservative 0.1% threshold for scientific evidence of extraction, while practical copyright policy may require a different bar.Source: §3 · p. 5
  • The paper identifies a gap in work-specific research needed to understand both sets of copyright issues.It turns next to quantifying memorization of individual books.Source: §3 · p. 5

4 Measuring memorization of individual books

The paper measures memorization book by book with a sliding-window probabilistic extraction procedure, revealing patterns that average extraction rates obscure. Coverage varies across books and models: most books show little extraction, while some are extensively memorized.

4.1 Sliding-window probabilistic extraction procedure

  • The sliding-window procedure tests overlapping 100-token sequences across a book, using 50-token prefixes and suffixes to locate extraction hotspots.Sequence starts advance by 10 characters, so most book characters appear in multiple prefixes and suffixes.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 Average extraction rates obscure book-specific memorization

  • Although Llama 3.1 70B averages about 0.6% memorized Books3 sequences, it effectively memorizes all of 1984 under the study’s extraction measure.The 1984 result does not mean the whole book can be extracted as one continuous generation.Source: §4.1 · p. 6, §4.2 · p. 7
  • Extraction coverage measures the fraction of book characters lying in at least one suffix whose extraction probability meets threshold τ, complementing average rates over sampled sequences.The procedure evaluates probabilistic extraction along the whole book rather than relying only on randomly sampled sequences.Source: §4 · p. 5, §4.2 · pp. 6–7
  • Most books show less than 2% extraction coverage, but many exceed 10%; Llama 3.1 70B has higher coverage across the evaluated books than any other tested model.The distributions are shown for Llama 1 65B and Llama 3.1 70B at a 0.1% coverage threshold.Source: §4.2 · p. 7, Figure 4 · p. 8
  • Llama 3.1 70B covers 43.26% of Harry Potter at an extraction-probability threshold of 50%, versus less than 0.4% coverage for Sandman Slim across all tested models and thresholds.At a 1% threshold, 90.89% of Harry Potter is covered; lower thresholds mean less reliable extraction.Source: §4.2 · pp. 7–8

5 Validating our extraction procedure

The validation tests show that longer prefixes can surface additional memorized sequences without making all training text extractable. Non-training controls generally remain below the extraction threshold apart from explainable exceptions, supporting the procedure’s validity.

5 Validating our extraction procedure

  • Across the validation tests, extraction signals on non-training data stayed below τmin apart from explained exceptions, supporting that counted extraction reflects memorization.Source: §5 · p. 8, §5.2 · p. 10

5.1 Longer prefixes surface more memorized training data

  • Longer prefixes reveal additional memorized sequences, but many remain unextractable even at 800 tokens, and some books show only limited gains.Source: §5.1 · p. 9

5.2 Measurements on non-training data do not mistakenly register extraction

  • PHI 4 signals were limited to duplicated quotes, and 2025 non-training text had no qualifying extraction at τmin=0.1% beyond boilerplate exceptions.Source: §5.2 · p. 10

5.3 Implications for measuring probabilistic extraction on books

  • The conservative τmin=0.1% makes estimates a likely lower bound; lowering it would require many more negative controls because LLM outputs are structured, not random.Source: §5.3 · pp. 10–11

6 Comparing memorization across books and models

Across 200 books and 14 models, memorization varies by both book and model: most tested books show little extraction, while some books are extensively memorized, especially by larger Llama models. The results also vary with training choices, and the tested sample is not a complete or generalized account of Books3.

6.1 Experimental setup

  • The study applies sliding-window extraction with top-k decoding to 14 models, using 50-token prefixes and 50-token suffixes.Models are grouped by known Books3 training, likely exposure to Books3 books, or no training on whole copyrighted books.Source: §6.1 · p. 11
  • The experiments cover 200 books—about 0.1% of Books3—including 100 randomly sampled books and 100 deliberately selected titles.The selected titles include works associated with plaintiffs in Kadrey et al. v. Meta, popular books, and more obscure books.Source: §6.1 · p. 11
  • The authors caution that these results are not a complete or generalized account of memorization across Books3.The study ran over 3,000 experiments, amounting to 394 GPU days.Source: §6.1 · p. 11

6.2 Key patterns in our results

  • Many tested books show effectively no memorization across models, while Llama 3 and 3.1 70B can reveal substantive sequences from some books.Most tested books resemble the low-memorization pattern; Beloved represents an intermediate case, where longer prefixes can reveal substantially more extraction.Source: §6.2 · p. 12
  • Some popular books show extensive memorization in larger models, especially Llama 3 and 3.1 70B, while many other books show little or none.For The Alchemist, Llama 3.1 70B has approximately 49.1% extraction coverage, compared with approximately 3.9% for DeepSeek V1 67B and 5.8% for Qwen 2.5 72B.Source: §6.2 · p. 12
  • Memorization differs across training choices: similarly sized models can show different degrees for the same book, and newer Llama versions typically memorize more than earlier ones.All evaluated Llama models memorize more than the other models evaluated, but the extent also varies by book.Source: §6.2 · p. 13

7 Recovering Harry Potter and the Sorcerer’s Stone with one seed prompt

The authors test whether Llama 3.1 70B can continue a short Harry Potter seed into a near-exact copy of the book. With a simple sliding-context beam-search procedure, they deterministically generated the full 304-page book, with measured similarity of 0.992.

7 Recovering Harry Potter and the Sorcerer’s Stone with one seed prompt

  • The sliding-window measurements do not by themselves show that large contiguous portions can be extracted all at once, although probabilities for some books approach or reach 100% with longer prefixes.For Llama 3.1 70B on books such as Harry Potter and 1984, the authors say the measured memorization suggests possible near-verbatim recovery from a short prompt.Source: §7 · p. 13
  • The recovery procedure repeatedly generates tokens, clips the context to a fixed length, and feeds the resulting generated context back to the model.After the initial context moves beyond the seed, subsequent prompts consist entirely of text generated in earlier iterations.Source: §7 · p. 14
  • Llama 3.1 70B deterministically generated a near-exact copy of the entire 304-page book from a six-token seed prompt.The tested configuration used beam width 8, a maximum context length of 3,000 tokens, and 50 generated tokens per step.Source: §7 · p. 14
  • The generated book scored 0.992 similarity to the ground truth using a greedy approximation of longest common substring on whitespace-split word lists.Differences were predominantly formatting and localization changes, with occasional missed single lines.Source: §7 · p. 15

8 Takeaways for copyright law and policy

Memorization varies across models and books, complicating copyright narratives on both sides. The authors discuss possible derivative-work claims, extraction of infringing copies, and the difficulty of class-wide claims.

8 Takeaways for copyright law and policy

  • The results complicate both plaintiffs’ and defendants’ accounts, with implications for derivative-work claims, extraction, and class actions.The authors say memorization varies by model, model family, size, book, and even parts within a book.Source: §8 · p. 15

8.1 LLMs could be derivative works of the books they memorize

  • Some models may be derivative works of some books, but the results do not support treating every model as a copy of all its training works.The authors say LLAMA 3.1 70B may incorporate substantial protectable expression from Harry Potter and the Sorcerer’s Stone; their study of 200 books and 14 models cannot establish memorization’s extent in books generally.Source: §8.1 · pp. 15–16
  • A model that memorizes content but produces only noninfringing outputs has a strong fair-use argument, while selling or releasing the model can make the training-data analogy less precise.The authors say commercial distribution does not defeat fair use but may make it harder to rely on cases supporting training on copyrighted data.Source: §8.1 · p. 15

8.2 It may be practical to generate infringing copies in some cases

  • Extraction may be easier with simple prompts or may require intensive adversarial prompting, which could affect courts’ assessment of who directly caused an infringing output.The authors contrast developer liability when simple prompts suffice with user responsibility when intensive strategies are used.Source: §8.2 · p. 16
  • In an exceptional case, LLAMA 3.1 70B yielded near-verbatim recovery of Harry Potter from a short prompt; this is not representative of most tested books.The usual sliding-window test uses 50-token prefixes and suffixes, and may require many prompts; the authors say large superseding copies seem unlikely for most works, though possible.Source: §8.2 · pp. 16–17

8.3 Class actions are ill-suited for copyright claims about memorization

  • Book- and model-specific variation makes class-wide claims difficult because whether significant expression was copied and infringing outputs are possible cannot be generalized across plaintiffs.The authors say answering these questions for a specific model and work requires experimental evidence; some plaintiffs may show copying while others will not.Source: §8.3 · p. 17

9 Conclusion

The paper presents a sliding-window procedure for measuring book memorization and reports substantial variation across models and books. Most tested books were not memorized under this procedure, but extreme cases included near-verbatim whole-book recovery.

9 Conclusion

  • The validated sliding-window procedure estimates memorization of Books3 books in open-weight LLMs, and the authors report that most LLMs do not memorize most books under their extraction procedure.Memorization nevertheless varies widely across models and books; LLAMA 3.1 70B could yield near-verbatim recovery of Harry Potter and the Sorcerer’s Stone.Source: §9 · p. 17
  • The results complicate copyright disputes by rejecting simplistic claims about how LLMs work and showing that memorization varies widely across models and books, with no single answer.Source: §9 · p. 17
  • For a suffix expected once per 1,000 prompts, 1,000 users would in expectation see one verbatim memorized suffix.Source: §9 · p. 17

Resources

The paper provides an interactive version of its results and a coding suite. Its appendices document additional extraction material, validity tests, extended results, and book-recovery experiments.

Resources

  • The authors release searchable results for 200 books and 14 models, alongside a general-purpose coding suite for probabilistic extraction.Source: Resources · p. 18

Appendix Table of Contents

  • The appendices cover extraction metrics and pipeline testing, extended sliding-window results, measurement validity, book recovery, and discussion.The table of contents also lists book-specific results and recovery experiments for books with low memorization signal.Source: Appendix Table of Contents · pp. 30, 31, 32, 34

A Additional notes on memorization and probabilistic extraction

  • Figure 10 compares LLAMA 1 30B’s 32-token generations against the target suffix; unhighlighted text marks verbatim extraction.Red marks target text not reproduced, while green marks generated additions.Source: Figure 10 · p. 35

A.1 Additional notes on the “careless people” example

  • The “careless people” example is a 57-token Great Gatsby sequence; the authors treat it as illustrative, not a general result.Its famous quote may be duplicated across training sources, so extraction cannot be attributed specifically to inclusion of the whole book in Books3.Source: §A.1 · p. 35

A.2 Metrics

The metric pz is the probability of extracting a target suffix for a specified model, decoding scheme, and suffix location; it can also be translated into the number of repeated prompts needed to reach a chosen extraction probability. The implementation computes pz from sequence logits in one forward pass, an approach equivalent to repeated generation for verbatim extraction.

A.2 Metrics

  • The probability of extracting a suffix at least once in n independent prompts is 1 − (1 − pz)^n.This relationship converts pz into the prompt count needed to reach a chosen probability threshold.Source: §A.2 · pp. 36–37, Figure 12 · p. 37
  • pz measures the target suffix probability for a specified model, decoding scheme, and suffix location.It is computed as the exponentiated sum of the suffix tokens’ conditional log probabilities.Source: §A.2 · pp. 36–37

A.3 Computing sequence probabilities in one forward pass

  • A single forward pass over the sequence yields pz, equivalent to repeated generation for verbatim extraction and limited to that setting’s efficiency advantage.The method post-processes logits rather than generating suffixes token by token.Source: §A.2 · p. 37, §A.3 · p. 37
  • For a 50-token suffix whose tokens each have 0.9 conditional probability, the suffix probability is about 0.5%, illustrating why unusually high pz can indicate memorization.The paper notes that some verbatim suffixes have probabilities above 90%.Source: §A.3 · p. 38
  • Under top-k decoding, a suffix with any token outside the top k has pz = 0 and is not extractable under that decoding scheme.The probability computation cannot continue when a target token is excluded by top-k truncation.Source: §A.3 · p. 38

B Testing our measurement pipeline

The measurement-pipeline check replicated Hayes et al.’s PYTHIA results on Enron, while adding a beginning-of-sequence token generally raised extraction rates. The appendix also reports Books3 average-rate experiments and describes which tested models were trained on Books3 or could yield copyrighted-book snippets.

B Testing our measurement pipeline

  • The validation reran Hayes et al.’s experiments on small PYTHIA models using 10,000 100-token Enron sequences to check accuracy and runtime.The experiments used the same four A100 GPUs as the paper’s other probabilistic extraction experiments.Source: B Testing our measurement pipeline · p. 38
  • The PYTHIA-on-Enron check replicated Hayes et al.’s exact results without BOS; prepending BOS generally elevated rates.The comparison is shown for greedy discoverable extraction and maximum probabilistic rates.Source: B Testing our measurement pipeline · pp. 38–39, Table 2 · p. 39

C.1 Setup

  • Books3 average-rate experiments used three random-sample runs of 40,000 sequences and measured greedy and top-k probabilistic extraction with τmin = 0.1%.The top-k setting used T = 1 and k = 40.Source: §C.1 · p. 40
  • PYTHIA and LLAMA models were known to have been trained on Books3; DEEPSEEK V1, QWEN 2.5, and GEMMA 2 models had evidence of training on books also contained there.PHI 4 was deliberately not trained on copyrighted whole books, but the authors could extract some copyrighted-book snippets from it.Source: §C.1 · p. 40

C.2 Experimental procedure

This section describes the experimental procedure for estimating book extraction across 200 Books3 books and 14 LLMs, using heavily overlapping sequences and top-k sampling to identify extractable regions. It presents book-specific results with sliding-window heatmaps and suffix-extraction probability distributions, and defines coverage as the fraction of a book’s characters covered by sufficiently extractable 50-token suffixes—not as evidence that the whole book can be extracted in one go.

C.2 Experimental procedure

  • The Books3 sampling procedure uses 4,000 books and 10 non-overlapping 100-token sequences per book, repeated three times for 120,000 sequences across 12,000 books.Each sequence has a 50-token prefix and suffix; prefix text starts with a space. Each run took about 8–45 minutes on four A100s.Source: §C.2 · p. 40
  • Probabilistic extraction uses top-k sampling with T = 1, k = 40, and τmin = 0.1%; the maximum rate is a loose upper bound that can include generalization.Sequences below τmin may also reflect generalization, so the maximum-rate figures are included as a reference rather than as extraction claims.Source: §C.2 · pp. 40–41

C.3 High-level takeaways

  • Extraction rates range from <0.2% for PHI 4 to ∼0.6% for LLAMA 3.1 70B; larger and later-generation LLAMA models generally memorize more, with exceptions.PHI 4, despite being 14B, is comparable to the smaller 7–9B models; the pattern does not always hold for individual books. [41] [123] [192]Source: Figure 13 · p. 41, §C.3 · pp. 41–42
  • Books3 sequence extraction does not establish that the specific Books3 copy was in a model’s training data, since the same text may have entered through other sources or duplicates.For PHI 4, the report says training was predominantly synthetic and did not include whole copyrighted books, though Books3 sequences may still appear from other sources.Source: §C.3 · pp. 41–42

D Extended results for sliding-window experiments

  • The extended-results appendices cover the sliding-window setup, procedure, results for 200 books and 14 models, and an extraction-coverage metric, alongside additional validity and extraction experiments.Validity experiments on non-training data and different prefix lengths, plus long-form extraction, are discussed in separate appendices.Source: D Extended results for sliding-window… · p. 42

D.1 Setup

  • The 200-book set combines 100 manually chosen and 100 randomly sampled books, covering roughly 0.1% of Books3 and not supporting corpus-wide extraction claims.The assortment was intended to capture variation in Books3 rather than serve as a fully random sample.Source: §D.1 · p. 42
  • Book metadata was extracted with GPT-5, checked through Google Search using Gemini 2.5 Flash, and manually spot-checked, with author fields validated for all 200 books.Metadata quality was assessed by manually checking a random 10% subset.Source: §D.1 · p. 43
  • The evaluation uses 14 completion-style models, including models known or believed to have Books3 training data and PHI 4 as an imperfect negative control.The authors use extraction from known-training-data models as evidence of memorization, while the other model groups support membership-inference evaluation and validity checks.Source: §D.1 · p. 43
  • The experiments cover 200 books drawn from Books3.The supplied passages show the book list continuing across Table 3.Source: §D.1 · p. 49

D.2 Sliding-window probabilistic extraction procedure

  • The procedure uses 100-token sequences split into 50-token prompts and 50-token target suffixes, a split described as a reasonable minimum for measuring memorization.Longer prefixes can reveal more underlying memorization; prior work also uses shorter sequences, such as 64 tokens split 32/32.Source: §D.2 · p. 54
  • Sequences are formed by taking 100 tokens from each 800-character chunk and shifting 10 characters forward through the book, producing roughly one sequence per 10 characters.The resulting sequences overlap substantially; the dense window is intended to surface high-probability memorized regions when the models’ training boundaries are unknown.Source: §D.2 · p. 54
  • A 10-character window was selected after tests of 1-, 10-, 50-, and 100-character shifts because it retained most extraction signal at 10× lower cost than a 1-character window.Source: §D.2 · p. 54
  • Decoding uses temperature T =1 with top-k k=40 to reflect the model’s base distribution while exploring alternatives to greedy decoding.Greedy decoding can miss higher-probability matching sequences because it chooses locally highest-probability tokens and does not guarantee the globally highest-probability sequence.Source: §D.2 · p. 54
  • Greedy decoding can miss valid verbatim extractions when its suffix is only a near-verbatim near-miss.The authors also report that greedy-decoded discoverable extraction with k = 1 undercounts valid extraction.Source: §D.2 · p. 55
  • Probabilistic extraction probabilities are computed as post-processing from model logits, enabling verbatim extraction probabilities with a single forward pass.The authors report reduced overhead compared with greedy generation.Source: §D.2 · p. 55

D.3 Book-specific results

  • For each of 200 books and 14 models, results include heatmaps of extraction probabilities by location and histograms of suffix probabilities.Heatmaps show the maximum probability among overlapping 50-token suffixes at each character; histograms summarize overlapping sequences sampled with a 10-character stride.Source: §D.3 · p. 55

Book-specific figures

  • Figures for Ararat, Lord of the Flies, Wizard’s First Rule, Rome and Jerusalem, The Fault in Our Stars, and The Third Man show the same heatmap-and-distribution format.Each caption specifies sliding-window heatmaps and corresponding suffix extraction probability distributions for 14 LLMs at τmin = 0.1%.Source: Figure 84 · p. 91, Figure 85 · p. 91, Figure 86 · p. 92
  • The same visualization format is used for books including Fifty Shades of Grey, The Stone Sky, Ulysses, Sandman Slim, and Ethnography after Antiquity.These captions likewise describe sliding-window heatmaps and suffix-extraction probability distributions for 14 LLMs at τmin = 0.1%.Source: Figure 108 · p. 103, Figure 109 · p. 103, Figure 110 · p. 104
  • Later figures continue this format for titles including Nineteen Eighty-Four, Fight Club, Night Watch, The Subtle Knife, and Backroads Boss Lady.They show the same two visualizations across 14 LLMs, with τmin = 0.1%.Source: Figure 149 · p. 123, Figure 150 · p. 124, Figure 159 · p. 128

D.4.1 Quantifying extraction coverage

  • Extraction coverage is the fraction of book characters covered by at least one 50-token suffix whose extraction probability meets threshold τ.The measure is defined for a book, model, decoding scheme, and probability threshold.Source: §D.4.1 · p. 156
  • The sliding-window procedure filters suffixes by extraction probability, then uses their character intervals to count covered characters and divide by the book’s total length.The procedure’s character-level metadata also supports heatmap visualization.Source: §D.4.1 · p. 156
  • High extraction coverage does not establish that the whole book can be extracted in one go or that the tested sequences are the minimal set needed for maximal coverage.The measure estimates the proportion of memorized text for cross-book and cross-model comparisons.Source: §D.4.1 · p. 156

D.4.2 Extended extraction coverage results

This section examines extended extraction coverage across books and models, including how longer prefixes and non-training controls relate to extraction and how results vary from little or fragmented memorization to near-total memorization.

D.4.2 Extended extraction coverage results

  • Per-book percentages compare extraction coverage across models and reveal variation in how much a given book is memorized.The paper gives Harry Potter and the Sorcerer’s Stone and Sandman Slim as examples; Table 4 reports per-book coverage for the 200 books and 14 models.Source: §D.4.2 · p. 156, Table 4 · p. 157
  • At τ = 0.1%, most books show little extraction coverage, but LLAMA 3.1 70B has higher maximum coverage and more substantially covered books than LLAMA 1 65B.Figure 214 shows model-level histograms across all 200 books and 14 LLMs.Source: §D.4.2 · p. 156, Figure 214 · p. 162

E.1.1 Setting τmin and extending to models with unknown training data

  • Across more than 300 negative controls, non-training suffix probabilities were essentially zero and orders of magnitude below τmin, supporting that measured extraction reflects memorization.Controls included PHI 4 and books published after the training cutoffs of Books3-trained LLMs.Source: E Measurement validity · p. 163

E Measurement validity

  • The validity experiments aim to distinguish true extraction from chance verbatim suffix generation and support conclusions for models with known or undisclosed training data.The authors describe the experiments as validating their measurements before extending analysis to models with undisclosed training information.Source: E Measurement validity · p. 163, §E.1 · p. 164

E.1.1 Setting τmin and extending to models with unknown training data

  • The authors set τmin = 0.1% conservatively because it exceeds negative-control probabilities, while undisclosed training data prevent certainty about whether models saw particular books.They prefer under-counting memorization to risking over-counting and defer finer threshold calibration to future work.Source: §E.1.1 · p. 164

E.1.2 Relating greedy-decoded and probabilistic discoverable extraction

  • Probabilistic extraction considers high-probability continuations beyond greedy top-1 decoding, capturing cases greedy decoding misses while retaining cases where the greedy suffix matches.The paper characterizes greedy decoding as a degenerate case of probabilistic extraction corresponding to τmin = 100%.Source: §E.1.2 · pp. 165–166

E.1.3 The role of prefix length in our experiments

  • Longer prefixes can raise target-suffix probability and expose memorization, but do not make arbitrary non-training suffixes extractable.The varied-prefix experiments used a small selection of Books3 titles, including Beloved and Harry Potter and the Sorcerer’s Stone.Source: §E.1.3 · p. 166, §E.2 · p. 166

E.2 Setup

  • On LLAMA 3.1 70B, This Is How You Lose Her is barely memorized, Beloved has disconnected hotspots, We Were Eight Years in Power has longer hotspots, and Harry Potter is effectively memorized entirely.Harry Potter suffixes do not all have equally high extraction probability.Source: §E.2 · p. 166
  • The prefix-length experiments compare three Books3-trained LLMs using 50-token suffixes and prefixes from 25 to 800 tokens; non-training controls include PHI 4 and post-cutoff books.The setup uses a conservative global τmin = 0.1%; unique text in non-training books is not expected to extract, though duplicated quotes or boilerplate may.Source: §E.2 · pp. 166–167, §E.4 · p. 170

E.3 Results of varying prefix length on known training data

  • Longer prefixes generally surface more memorized Books3 text, but gains diminish and many sequences remain unextractable even with very long prefixes.Extraction coverage levels off; absence of extraction is not evidence that a sequence was absent from training data.Source: §E.3 · pp. 167, 170, Figure 215 · p. 168, Figure 216 · p. 168

E.4 Results of negative control experiments

  • Non-training data produced pz values many orders of magnitude below the study’s extraction-success threshold, supporting that successful matches on known training data are not chance generations.The authors say these controls provide assurance that the method can be applied to models whose training data are not known exactly.Source: §E.4 · p. 171

E.4.1 Results on PHI 4

PHI 4 yielded little unique text from in-copyright books; extracted passages were mostly boilerplate or widely duplicated quotes. Post-cutoff controls produced no high-probability extraction, though duplicated online text complicates clean negative controls.

E.4.1 Results on PHI 4

  • PHI 4 yielded little unique text from in-copyright books; extracted passages were mostly boilerplate or famous, popular quotes.The authors verified that every extracted sequence could be found in multiple online sources.Source: §E.4.1 · p. 171
  • Extracted passages were often highly memorized across tested models, suggesting duplicated training sources rather than Books3 alone may explain their extraction.The authors note this is suggestive, not definitive.Source: §E.4.1 · pp. 171, 173
  • The controls support that the procedure detects true extraction and avoids flagging text when memorization is impossible, but duplicated web text complicates training-membership judgments.A book may be absent from training data even when some of its text is present and memorized from other sources.Source: §E.4.1 · p. 171
  • No sequences from This Is How You Lose Her were extracted beyond its copyright notice; every extracted Harry Potter sequence was found on three webpages.Beloved sequences were also posted multiple times on platforms such as Goodreads.Source: §E.4.1 · pp. 171–172
  • Longer prefixes generally increased extraction signal only when memorization was already present, with diminishing gains and no consistent unlocking of previously unextractable sequences.The PHI 4 controls used four Books3 books and prefix lengths from 25 to 800 tokens.Source: Figure 219 · p. 172
  • For four books published after every evaluated model’s release, even a permissive probability threshold produced no high probabilities counted as successful extraction.These seemingly clean controls still required manual checks because the books could cite earlier, widely duplicated sources.Source: §E.4.1 · p. 171, Figure 220 · p. 173

E.4.2 Results on books from after training cutoffs

This section reports results for books published after the models’ training cutoffs and explains how to interpret extraction evidence. It limits membership conclusions to extracted sequences, cautions that failed extraction does not rule out membership or influence, and distinguishes technical memorization from copyright relevance.

E.4.2 Results on books from after training cutoffs

  • No unique text from the four post-cutoff books was extracted at the chosen threshold; detected sequences were boilerplate or duplicated material from other sources.Examples include the Bible quote in The Emperor of Gladness, the Great Gatsby quote in Careless People (from The Great Gatsby), and widely duplicated Mark Zuckerberg material. No non-training-data instances appeared at pz ≥ 10^-8%, seven orders of magnitude below τmin.Source: §E.4.2 · pp. 173–174
  • Across five models, varied-prefix tests on Great Big Beautiful Life found no sequences beyond boilerplate, with only one sequence reaching about 10^-14% probability.The single sequence occurred with an 800-token prefix; its reported exponent is reliable, but its mantissa is not. Figure 222 reports no unique sequences for the tested prefix lengths.Source: §E.4.2 · p. 174, Figure 222 · p. 176

E.5 Limits on “the monkey at the typewriter” metaphor

  • For a fixed model and decoding scheme, suffixes assigned zero probability cannot be generated by repeated sampling or longer prompts, while suffixes with positive probability can eventually appear.The maximum attainable generation rate is the fraction of target suffixes with nonzero probability, including probabilities below the extraction threshold. Top-k restrictions and finite-precision underflow can make sequences unreachable; experiments also find zero-probability suffixes for non-training and non-memorized data.Source: §E.5 · pp. 177–178

E.6 Additional notes on extraction and membership inference

  • The authors distinguish their evaluation of specific works from prior studies that estimate average extraction rates on random samples of known or proxy training data.Because their claims concern a specific population of texts, validity considerations shape their measurement approach.Source: §E.6 · p. 178

E.6.1 The relationship between membership inference and extraction

  • Successful extraction demonstrates membership and reproduces the data, but membership does not imply that extraction will succeed.Extraction is therefore stronger evidence than the one-bit membership-inference claim, though reliable membership inference is described as challenging for LLMs.Source: §E.6.1 · p. 179

E.6.2 We infer membership for sequences, not whole books

  • The authors test extraction on known training data and use negative controls to validate a conservative global threshold, while limiting conclusions to sequences rather than whole books.Post-cutoff controls further support the observation that non-training data show zero or orders-of-magnitude smaller probabilities than training data; many known training books also yield no unique extracted sequences.Source: §E.6.2 · pp. 179–180

E.6.3 Inferring membership for LLMs with undisclosed training data

  • For models with undisclosed training data, the authors treat validated extraction as evidence of sequence-level membership, using negative controls and baselines to assess validity.They manually verified that sequences extracted from PHI 4 and post-cutoff books were widely duplicated online; trivial memorization and repeated-substring cases were rare and excluded from claims about unique creative expression.Source: §E.6.3 · p. 180

E.6.4 Our inferences about membership are about sequences, not whole books

  • Membership claims concern particular extracted suffixes, not whole books, and the provenance of those sequences is generally unknown.For Llama models, Books3 inclusion establishes at least one training copy, but other sources and duplicates complicate broader claims [41] [123] [157]; copyright-relevant membership may be considered at book, chapter, or sentence scope [159].Source: §E.6.4 · p. 181

E.6.5 Absence of extraction does not imply anything about membership

  • Failure to extract does not establish absence of membership: extraction implies membership, but membership does not imply extraction, and other methods could reveal additional memorization.The authors do not measure influence or attribute outputs to training data; a failed Sandman Slim reconstruction nevertheless appeared informally influenced by the text.Source: §E.6.5 · p. 181

E.6.6 Extractability and copyright

  • Technical memorization is defined here as pz ≥ τmin = 0.1%, but low-probability sequences may matter less for copyright than highly extractable ones.Source: §E.6.6 · p. 182

F Recovering books with one seed prompt

The authors use autoregressive generation from a single seed prompt to recover highly memorized books, with Harry Potter recovered near-verbatim. Additional tests show partial recovery of another essay, while a low-memorization book does not continue from its text under the same approach.

F Recovering books with one seed prompt

  • The recovery demonstration is an existence proof, not a comprehensive test of whether the method works across different texts.The authors call this a preliminary investigation and not the paper’s main focus.Source: F Recovering books with one seed prompt · p. 182

F.1 Recovery method

  • The Harry Potter recovery was stable across tested hardware and settings, but sensitive to configuration, with overly long strides causing divergence from the source text.The authors report preliminary stability improvements but defer further investigation.Source: §F.1 · p. 182
  • Generation used beam search to generate 50 tokens at a time, with a maximum 3,000-token context; each chunk was appended, and the first 50 tokens clipped when exceeding the limit.Source: §F.1 · p. 182
  • After minor normalization, the recovered Harry Potter text scored 0.9999 TF-IDF cosine similarity, 0.992 word-level LCS similarity, and 0.934 sentence-level LCS similarity.The sentence-level score is more sensitive to formatting differences.Source: §F.1 · p. 183
  • Most Harry Potter differences were formatting or localization changes; one apparent hallucination matched an earlier edition, though it still affected similarity measurements.The diff illustrates recovered text against the Books3 ground truth.Source: Figure 223 · p. 183, §F.1 · p. 183

F.2 Partial recovery

  • After minor text normalization, running this script generated an essentially verbatim copy of the first ≈68% of the essay, with word-level greedy LCS approximation 0.992 and document-level 0.996.The seed prompt was 19 words, and the generated near-exact text was 10,039 words.Source: §F.2 · p. 185

F.3 Running recovery for books with low memorization signal

  • For Sandman Slim, generation looped and reused character names but produced no text the authors identified as copied from the book.This was their low-memorization sanity check.Source: §F.3 · p. 186
  • Even prompts as long as the book’s first two pages did not make Sandman Slim generation continue with text from the book.The authors describe these tests as illustrative rather than exhaustive.Source: §F.3 · p. 186

G Discussion of extended results

The extended results emphasize that measured memorization varies across both books and models, with prominent exceptions alongside many books showing little extraction under the tested settings. The discussion also cautions that these findings are bounded by the sample, extraction approach, and distinction between measuring memorization and reconstructing a book.

G Discussion of extended results

  • Memorization varies substantially across books and models, a pattern that average extraction rates do not reveal.The authors argue that per-book extraction metrics are needed to see this variation.Source: Appendix · p. 186

Appendix.

  • The experiments covered 200 books, and the authors say more are needed for general claims; within the tested sample, very popular books showed the most memorization.The books included varied sources and popularity levels, with half randomly sampled.Source: Appendix · p. 186
  • LLAMA 3.1 70B showed more memorization than other compared LLAMA models and QWEN 2.5 72B, despite QWEN’s lower average loss.The authors say they do not know exactly what explains LLAMA 3+ memorization.Source: Appendix · p. 187
  • Later LLAMA generations generally showed more memorization; most tested books had minimal extraction under the main 100-token, top-k setting, while longer prefixes revealed more.The authors characterize longer-prefix discoveries as harder to extract.Source: Appendix · p. 187
  • Extracted text often came from duplicated material such as copyright notices, biographies, ordered page numbers, or widely quoted sources, so extraction from a book need not originate there.The authors note these text types can appear across websites and books.Source: Appendix · p. 187
  • For books in the tested Kadrey lawsuit scope, little memorized data was extracted with 50-token prefixes, with Coates as a notable exception.The authors note another extraction technique could reveal additional memorization.Source: Appendix · p. 187
  • Sliding-window measurements do not establish that reconstruction follows; the separate autoregressive experiment demonstrates reconstruction for a highly memorized book.The paper’s main measurements quantify probabilistic discoverable extraction rather than extracting large amounts in one go.Source: Appendix · p. 187
  • LLAMA 3+ models showed enormous memorization in some cases, prompting the authors to attempt single-seed book reconstruction.The examples include public-domain and popular in-copyright books.Source: Appendix · p. 188
Loading 2505.12546v6…