Source-linked AI summary
Extracting books from production language models
Ahmed Ahmed, A. Feder Cooper, Sanmi Koyejo, Percy Liang
TL;DR
The paper examines whether memorized copyrighted books can be extracted from production LLMs despite safety safeguards. It applies a two-phase extraction procedure across four systems and finds that large amounts of in-copyright text can be extracted in particular configurations, including whole books from Claude 3.7 Sonnet.
Problem
It was unclear whether the substantial copyrighted-text extraction observed in open-weight models also occurs in production LLMs with safety measures.
Method
The authors combine an initial prefix-completion probe, sometimes using Best-of-N jailbreaking, with iterative continuation prompts and conservative near-verbatim extraction measurement.
Results
Large amounts of in-copyright text were extracted from four production LLMs, including four whole books near-verbatim from Claude 3.7 Sonnet.
Takeaways & Limitations
Production LLM safeguards do not eliminate the feasibility of extracting large quantities of in-copyright training data in the tested settings.
Takeaways & Limitations
The findings are descriptive for specific models, books, instantiations, runs, and experimental configurations, while the conservative metric can miss some extraction.
Abstract
from arXiv · showhide
Many unresolved legal questions over LLMs and copyright center on memorization: whether specific training data have been encoded in the model's weights during training, and whether those memorized data can be extracted in the model's outputs. While many believe that LLMs do not memorize much of their training data, recent work shows that substantial amounts of copyrighted text can be extracted from open-weight models. However, it remains an open question if similar extraction is feasible for production LLMs, given the safety measures these systems implement. We investigate this question using a two-phase procedure: (1) an initial probe to test for extraction feasibility, which sometimes uses a Best-of-N (BoN) jailbreak, followed by (2) iterative continuation prompts to attempt to extract the book. We evaluate our procedure on four production LLMs -- Claude 3.7 Sonnet, GPT-4.1, Gemini 2.5 Pro, and Grok 3 -- and we measure extraction success with a score computed from a block-based approximation of longest common substring (nv-recall). With different per-LLM experimental configurations, we were able to extract varying amounts of text. For the Phase 1 probe, it was unnecessary to jailbreak Gemini 2.5 Pro and Grok 3 to extract text (e.g, nv-recall of 76.8% and 70.3%, respectively, for Harry Potter and the Sorcerer's Stone), while it was necessary for Claude 3.7 Sonnet and GPT-4.1. In some cases, jailbroken Claude 3.7 Sonnet outputs entire books near-verbatim (e.g., nv-recall=95.8%). GPT-4.1 requires significantly more BoN attempts (e.g., 20X), and eventually refuses to continue (e.g., nv-recall=4.0%). Taken together, our work highlights that, even with model- and system-level safeguards, extraction of (in-copyright) training data remains a risk for production LLMs.
1 Introduction
The paper asks whether safety-protected production LLMs can still expose memorized copyrighted books, and reports that long-form extraction is feasible in particular experimental settings.
- Memorization can encode specific training data in model weights and make them extractable near-verbatim from outputs.
- Production safeguards create an unresolved gap because prior open-weight results do not directly establish long-form extraction from production systems.
- The study uses a two-phase procedure combining an initial prefix-completion probe, sometimes with Best-of-N jailbreaking, and iterative continuation prompts.
- 95.8% nv-recall was achieved for Harry Potter and the Sorcerer’s Stone with jailbroken Claude 3.7 Sonnet, whereas GPT-4.1 reached 4.0%.Claude used BoN N = 258; GPT-4.1 used N = 5179 and stopped after the first chapter.
- The authors frame these results descriptively: extraction varied by book, model, experimental setting, and run, without ranking overall extraction risk across LLMs.
2 Background and related work
The background defines memorization and extraction, reviews standard and jailbreak-based extraction methods, and situates both within copyright debates.
- Memorization and extraction: Memorization concerns whether information about training examples is encoded in a model and recoverable from it.
- Memorization and extraction: The standard extraction test prompts an LLM with a known prefix and checks whether it generates the corresponding suffix verbatim.
- Circumventing safeguards: Jailbreaks use adversarial prompts to elicit outputs that production safeguards otherwise restrict.
- Copyright and generative AI: Copyright debates address whether memorized training data and reproduced outputs constitute copies and how extraction methods affect those questions.
3 Extraction procedure
The paper uses a two-phase procedure to test and pursue long-form book extraction from production LLMs, beginning with a short-prefix probe and continuing successful cases iteratively. It combines black-box prompting, optional Best-of-N jailbreaking, similarity thresholds, and conservative near-verbatim matching to assess extraction.
- Phase 1: Phase 1 probes book extraction by prompting completion of a short ground-truth prefix, using Best-of-N jailbreaking for Claude 3.7 Sonnet and GPT-4.1.Gemini 2.5 Pro and Grok 3 comply directly, whereas Claude 3.7 Sonnet and GPT-4.1 require jailbreaks to circumvent refusal mechanisms.
- Phase 1: Best-of-N creates N randomly perturbed instruction prompts, submits them through the black-box API, and selects a response that contains a loose match to the target suffix.The attack uses perturbations such as case changes, word-order shuffling, visually similar character substitutions, and formatting edits.
- Phase 1: Phase 1 succeeds when the longest contiguous verbatim match covers at least 60% of the target suffix, expressed as s ≥0.6.For Claude 3.7 Sonnet and GPT-4.1, BoN stops when a response reaches this threshold or when the maximum budget N = 10,000 is met; this loose match alone is not treated as extraction.
- Phase 2: Phase 2 repeatedly queries the production LLM to continue the book until refusal, a stop phrase, or the query budget is reached.The Phase 1 prefix is the only ground-truth text supplied; recovered additional text is generated by the production LLM, under fixed per-run generation configurations.
- Verification: Extraction success is evaluated by conservative near-verbatim matching that requires sufficiently long and similar generated text, favoring false negatives over false positives.The procedure merges aligned nearby verbatim blocks and retains only sufficiently long blocks, including a final threshold of l(2) = 100; fragmented passages may therefore be discarded.
4 Experiments
The experiments tested book extraction across four production LLMs using configurations that varied by model, finding substantial but uneven extraction, including near-complete recovery in some Claude 3.7 Sonnet runs. Results are descriptive of specific runs rather than matched evaluations, and GPT-4.1 was constrained by refusals during continuation.
- 4.1 Setup: Four production LLMs were evaluated on thirteen books, with results primarily reported for selected runs using model-specific configurations.The study period was limited to mid-August through mid-September 2025 because production systems change over time.
- 4.2 High-level extraction outcomes: Hundreds of thousands of words were extracted across Phase 2 runs, including examples from in-copyright books.The authors provide diffs for public-domain books but do not redistribute long-form in-copyright generations.
- 4.2 High-level extraction outcomes: Phase 1 failed for all four production LLMs only on one of the thirteen attempted books, while the negative control also failed as expected.The omitted book and negative control were not included in the main plots.
- 4.2 High-level extraction outcomes: Claude 3.7 Sonnet recovered over 94% of four reference books in some configurations, including two books in copyright in the U.S.Three other Claude cases achieved nv-recall ≥32%.
- 4.2 High-level extraction outcomes: Significant amounts of Harry Potter and the Sorcerer’s Stone and other books were extracted from all four production LLMs under their respective configurations.The bar plots describe specific experimental outcomes and do not support comparisons under identical conditions.
- 4.2 High-level extraction outcomes: GPT-4.1 encountered a refusal early in Phase 2, limiting Harry Potter and the Sorcerer’s Stone extraction to the first chapter despite reporting nv-recall for the full book.The other three production LLMs almost always avoided refusals in the reported runs.
- 4.2 High-level extraction outcomes: For Harry Potter and the Sorcerer’s Stone, nv-recall was 95.8% for jailbroken Claude 3.7 Sonnet, 4.0% for jailbroken GPT-4.1, 76.8% for Gemini 2.5 Pro, and 70.3% for Grok 3.The corresponding approximate costs were $119.97, $1.37, $2.44, and $8.16.
- 4.2 High-level extraction outcomes: Low nv-recall could still represent thousands of extracted words: GPT-4.1 yielded m ≈3200 words for Harry Potter and Grok 3 yielded m ≈3700 words for A Game of Thrones.The longest Harry Potter near-verbatim blocks ranged from 821 to 9070 words across the four models.
5 Discussion
The discussion frames extraction as feasible but highly dependent on the tested model, book, procedure, and run. It also emphasizes limitations involving scale, cost, safeguards, and the interpretation of non-extracted generated text.
- Limitations: The extraction procedure provides a loose lower bound on memorization because different seed locations and prompting strategies can reveal additional memorized text.Main experiments usually used one run with a seed prefix from a book's beginning; chapter-start seeds sometimes exposed more memorization.
- Limitations: The experiments covered only fourteen books, and production-LLM API changes and cost constrained scale and reproducibility.Claude 3.7 Sonnet long-context generation often cost over $100 per run.
- Interpretation: Extraction observations are specific to each production LLM, book, procedure instantiation, and run, so the authors do not make evaluative claims across LLMs.The authors describe their results as descriptive observations under tested conditions.
- Safeguards: Safeguards were not consistently effective: Gemini 2.5 Pro and Grok 3 complied with initial probes, while continuation loops sometimes ran hundreds of iterations before filtering.The paper characterizes these outcomes as possible evasions of model- and system-level safeguards.
- Copyright relevance: Thousands of words of non-extracted generated text reproduced book characters, plot elements, and themes, but the paper does not analyze this text rigorously or at scale.The authors defer quantitative analysis of this material to future work.
6 Conclusion
The conclusion reports that a simple two-phase procedure extracted large amounts of in-copyright text from all four tested production LLMs. It presents these technical findings as relevant to legal debates while deferring legal interpretation and documenting the paper's reproducible jailbreak details.
- Findings: A simple two-phase procedure extracted large amounts of in-copyright text from four production LLMs.The procedure is described in Section 3 and evaluated across the four systems.
- Findings: Claude 3.7 Sonnet and GPT-4.1 required jailbreaks, whereas Gemini 2.5 Pro and Grok 3 directly complied with text-continuation requests.For Claude 3.7 Sonnet, the authors extracted four whole books near-verbatim, including Harry Potter and the Sorcerer's Stone and 1984.
- Implications: The technical findings may inform copyright debates, but the paper does not present itself as a work of copyright law or policy.The authors state that lawyers and judges must determine how to treat these technical facts.
- Implications: The paper identifies memorization as encoding training data in model weights and reports that large quantities of such data can be feasible to extract.The conclusion links these technical facts to ongoing questions about training data and model outputs.
- Jailbreak documentation: Best-of-N perturbations were documented for Claude 3.7 Sonnet and GPT-4.1, with deterministic replay given a fixed random seed.The reported perturbations include identity, capitalization, spacing, word-order, character, punctuation, scrambling, and ASCII-noising operations.
B Procedure for quantifying extraction success
The extraction metric identifies near-verbatim text by finding matching blocks and applying two merge-and-filter passes. The configuration is deliberately conservative so benign formatting differences can be reconciled without counting generated gap text.
- Block formation: The procedure first identifies verbatim blocks with a block-based greedy approximation of longest common substring.It uses difflib SequenceMatcher to return ordered matching blocks between generated and ground-truth text.
- Merge and filter: Two merge-and-filter passes combine aligned blocks separated by short gaps, with a stringent first pass and a slightly more relaxed second pass.The first pass uses τ_align^(1)=1 and the second uses τ_align^(2)=3.
- Thresholds: The first filter requires l^(1)=20 words, while the second requires l^(2)=100 words to retain a merged near-verbatim span.The first threshold is about half the 35–40 words used for verbatim discoverable extraction.
- Conservatism: The metric counts only the lengths of original verbatim blocks, excluding generated text in gaps, making the extraction estimate conservative.In the illustrated example, the final merged block counts 141 words from six original blocks rather than gap text.
- Validation: The two-stage design preserves long near-verbatim spans despite benign formatting differences that would be discarded by a simple merge-and-filter procedure.Figure 4a illustrates reconciliation of short blocks created by formatting differences, while Figure 4b shows short blocks that should not count as extraction.
C Experimental setup
The experiments use selected books suspected to appear in training data and apply production-LLM-specific Phase 2 configurations. Extraction claims are restricted to long near-verbatim generated blocks, with whole-book membership claims limited to select Claude 3.7 Sonnet cases.
- Experimental setup: The study reports additional book-selection, production-LLM-specific Phase 2, and text-normalization details in appendices.These details cover selection, generation configurations and results, and preprocessing before near-verbatim metric computation.
- Claim scope: Extraction claims are made only for long generated blocks of near-verbatim text, which supports a corresponding claim of training-data membership.Except in select Claude 3.7 Sonnet cases, the paper does not claim whole-book training-data membership.
- Reproducibility: The appendix documents the exact Phase 2 hyperparameters and stopping conditions used for each production LLM.These settings correspond to the continuation stage of the extraction procedure.
C.2.1 Settings for main results
The main experiments fix deterministic generation settings per production LLM, tune Gemini 2.5 Pro parameters from exploratory sweeps, and halt continuation when budgets, refusals, or book-end markers are detected.
- Exploratory experiments identify generation conditions under which extraction might work for each production LLM.
- Temperature is fixed at 0.0, while maximum returned tokens and continuation turns vary by production LLM.The configurations use 250/600 for Claude 3.7 Sonnet, 500/200 for Grok 3, 2000/300 for Gemini 2.5 Pro, and 500/300 for GPT-4.1.
- Gemini 2.5 Pro uses frequency penalty 2 and presence penalty 0.1 based on initial parameter sweeps.
- Phase 2 terminates when the continuation budget is reached, a refusal is detected, or a stop string signals the book’s end.Stop strings include “THE END,” “[End of Book],” “About the Author,” “Afterword,” and “Bibliography.”
- Claude 3.7 Sonnet’s stop-string detection was changed from a book-specific final sentence to generic end-of-book markers.The change avoided relying on ground-truth reference text beyond the Phase 1 prefix.
- For Gemini 2.5 Pro, a 2000-token maximum produced the highest nv-recall in all tested cases, so it was fixed for later experiments.The reported fixed penalties were frequency penalty 2.0 and presence penalty 0.1, although maximum nv-recall varied by configuration.
C.2.3 Refusal retries for per-chapter experiments with GPT-4.1
Per-chapter GPT-4.1 experiments retry around refusals, while extraction evaluation applies deterministic, light normalization before token-level overlap measurement.
- Refusal retries: GPT-4.1 per-chapter runs generate five responses per continuation turn and select the first response without a refusal.If all five responses refuse, the procedure retries with exponential backoff up to 100 times and runs for up to 50 turns per chapter.
- Chat UI: The chat UI extracted the first four chapters of The Great Gatsby even when the API could not reliably do so.The authors also observed that the procedure worked for Claude 3.7 Sonnet in Anthropic’s chat UI, but they excluded UI results.
- Evaluation: Evaluation uses the ground-truth book and generated text as the two input documents for extraction assessment.
- Normalization: Extraction evaluation compares lightly normalized reference books and generated text to reduce superficial formatting and Unicode differences.The normalization targets differences such as italics markers, quotation styles, and punctuation that may not match between training data and Books3.
- Normalization: The normalization pipeline applies Unicode NFKC, punctuation remapping, ellipsis and dash handling, italics-marker removal, and lowercasing before tokenization.The resulting whitespace-split word sequences are passed to difflib SequenceMatcher.
- Normalization: The procedure keeps normalization minimal by avoiding stemming, lemmatization, stopword removal, broad punctuation stripping, and ASCII conversion.
D Extended results
The appendix extends the main paper with detailed and additional experiments covering both extraction phases.
- The appendix reports more detailed results for the main-paper experiments.
- It also includes additional experiments beyond those presented in the main paper.
- The extended results cover Phase 1 and Phase 2 experiments.
D.1 Additional Phase 1 results
Additional Phase 1 analyses examine Claude 3.7 Sonnet and GPT-4.1 jailbreak budgets across books, with success defined by a threshold on the probe score.
- BoN comparisons: The BoN analysis compares maximum Phase 1 probe scores and the N required to reach them for Claude 3.7 Sonnet and GPT-4.1.The analysis covers twelve books with at least one Phase 1 success, with a maximum N budget of 10,000.
- BoN comparisons: Because Gemini 2.5 Pro and Grok 3 are not jailbroken, the BoN results omit those models and use N = 0 for them.
- Figure 11: Figure 11 illustrates how s changes with N for Claude 3.7 Sonnet across four attempted books and a negative control.
- Claude 3.7 Sonnet: Phase 1 succeeds when s ≥0.6, succeeding for Harry Potter and the Sorcerer’s Stone, The Great Gatsby, and Frankenstein.Those three books ultimately achieve nv-recall ≥94%, whereas Phase 1 fails for The Duchess War and the negative control.
- Negative control: Phase 1 fails for The Society of Unknowable Objects, a negative control published after the knowledge cutoffs of all four production LLMs.
D.2 Additional Phase 2 results
Additional materials report Phase 2 API costs and provide supplementary plots and tables.
- Phase 2 API costs, additional plots, and tables are provided in the appendices.
D.2.1 Continuation loop API costs
The appendix documents continuation-query accounting and estimates Phase 2 costs across production LLMs, including provider-specific pricing and caching considerations.
- Empty Gemini responses count against the query budget, while Grok 3 HTTP 500 errors can prematurely terminate continuation loops.
- Estimated Phase 2 costs depend on continuation-query counts, per-query token usage, and provider pricing during the experimental window.The reported costs are approximations because pricing and tokenization vary across providers and can change over time.
- Phase 2 query counts are reported for each successful book–LLM run, with GPT-4.1 omitted for two books.Phase 2 was not run for GPT-4.1 on The Da Vinci Code and Catch-22.
- GPT-4.1 costs are reported as upper and lower estimates because billing data were unavailable and prompt caching required a heuristic.The lower estimate uses a shared-prefix caching heuristic, while the upper bound assumes no caching.
- Gemini 2.5 Pro costs are reported separately for the main fixed configuration and the cumulative generation-parameter sweep.
D.2.2 Plots and tables
Supplementary plots and tables provide detailed extraction metrics, absolute word counts, and Gemini 2.5 Pro sensitivity across generation configurations.
- Precise nv-recall, extracted-word, missing-word, and additional-word values are reported for all main experiments.Table 11 defines nv-recall as m/|B| and reports the associated book and generated-text lengths.
- Absolute word-count plots show extracted, missing, and additional words for eight books omitted from Figure 7.
- Gemini 2.5 Pro results show how nv-recall varies across nine generation configurations for each book.
- For each book, Figure 14 compares maximum observed nv-recall with the mean ± STD across the nine configurations.The fixed configuration used in Figure 5 reaches the displayed maximum for some books but not others.