Source-linked AI summary
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Kent K. Chang, Mackenzie Cramer, Sandeep Soni, David Bamman
TL;DR
Cultural-analytics researchers lack reliable knowledge of which books closed language models have seen, making evaluation validity difficult. The paper infers memorized books through name-cloze membership inference and finds uneven memorization tied to web popularity, with better downstream performance on memorized books. It concludes that known training sources, as in open models, are needed to address this uncertainty.
Problem
Unknown training data makes it impossible to construct evaluation benchmarks that confidently exclude memorized books, threatening validity assessments for cultural analytics.
Method
The paper performs data archaeology using name-cloze membership inference on passages from 571 fiction works to measure models’ exact memorization.
Results
Memorization varies across books, aligns with web popularity, and produces better downstream performance on memorized than non-memorized books.
Takeaways & Limitations
Memorization patterns complicate the use of closed models for cultural analytics by making performance conditions uncertain and test data vulnerable to contamination.
Takeaways & Limitations
The models’ true training data remains unknowable outside OpenAI, and probabilistic familiarity cannot establish that a book is actually in training data.
Abstract
from arXiv · showhide
In this work, we carry out a data archaeology to infer books that are known to ChatGPT and GPT-4 using a name cloze membership inference query. We find that OpenAI models have memorized a wide collection of copyrighted materials, and that the degree of memorization is tied to the frequency with which passages of those books appear on the web. The ability of these models to memorize an unknown set of books complicates assessments of measurement validity for cultural analytics by contaminating test data; we show that models perform much better on memorized books than on non-memorized books for downstream tasks. We argue that this supports a case for open models whose training data is known.
1 Introduction
The paper investigates how memorization in closed language models affects cultural analytics. It finds broad, uneven memorization of books and downstream performance differences linked to what models have seen.
- Unknown training data threatens cultural-analytics validity because memorized books in evaluation sets can falsely measure performance on non-memorized books.
- The study uses membership-inference data archaeology on passages from 571 fiction works published between 1749 and 2020, using a difficult name cloze task with a 0% human baseline.
- OpenAI models, especially GPT-4, have memorized a wide collection of in-copyright books.
- The strongest memorization includes science fiction and fantasy novels, popular public-domain works, and bestsellers.
- Memorization bias aligns with book prevalence in Google, Bing, and C4 search or corpus results, offering a rough diagnostic of book knowledge.
- GPT models perform better on memorized than non-memorized books when predicting publication year and narrative duration, and more often generate seen-book character names.
- BERT also memorized copyrighted material from BookCorpus, including works such as Fifty Shades of Grey, Outlander, and The Lost Symbol.
- These findings expose underlying model knowledge while illustrating validity threats for researchers using ChatGPT and GPT-4 in cultural analytics.
2 Related Work
Related work situates the study among critiques of opaque language-model training data, memorization, and contamination. It connects these concerns to the growing use of LLMs for cultural analysis.
- Critical digital humanities approaches examine the closedness and opacity of LLMs used to reason about literature.
- LLMs are increasingly used for cultural analysis, including interpreting events, classifying character roles, and performing NLP tasks on literary material.
- Large training corpora such as BookCorpus, C4, and the Pile extend model capabilities but are less carefully curated and understood than smaller datasets.
- Prior memorization research documents both benefits and harms, including security, privacy, and copyright concerns.
- Data contamination can place evaluation text in training corpora, with train-test leakage contributing partly to measured performance gains.
3 Task
The paper operationalizes book familiarity through a difficult name cloze task that asks models to recover a masked proper name from context. The evaluation controls passage selection and compares model outputs with simple and human baselines.
- 3.1 Name cloze: The name cloze task predicts one masked proper name from a 40–60-token passage containing no other named entities.
- 3.1 Name cloze: Because the context contains no names identifying the book, success requires knowledge of the work rather than ordinary English prediction.
- 3.1 Name cloze: The task is used as a membership-inference probe for exact memorization because OpenAI APIs did not provide word probabilities at the time.
- 3.1 Name cloze: Passages are extracted with BookNLP, restricted to complete sentences with one proper person entity, and sampled at 100 passages per book.
- 3.1 Name cloze: Books with fewer than 100 qualifying passages are excluded, and prompts elicit a single proper-name response in XML tags.
- 3.1 Name cloze: 0% human accuracy indicates that the passage itself provides little signal for identifying the masked name, while always guessing “Mary” reaches 0.6%.
4 Data
The evaluation set contains 571 English-language fiction works spanning public-domain classics, contemporary award and bestseller lists, underrepresented authors, global fiction, and genre fiction. Sources and copyright status vary across the collections.
- The dataset includes 91 pre-1923 LitBank novels, which are public domain in the United States and born digital on Project Gutenberg.
- It includes 90 Pulitzer prize nominees published from 1924 to 2020.
- It includes 95 bestsellers from the New York Times and Publishers Weekly published from 1924 to 2020.
- It includes 101 novels by Black authors from the Black Book Interactive Project or Black Caucus American Library Association award winners.
- It includes 95 works of Global Anglophone fiction published from 1935 to 2020.
- It includes 99 genre-fiction works spanning science fiction, fantasy, horror, mystery or crime, romance, and action or spy novels.
- Except for the public-domain LitBank texts, later collections were purchased as physical books, scanned, and OCR-processed; books after 1928 were generally copyrighted in the United States.
5 Results
GPT-4 and ChatGPT show uneven name-cloze memorization across book categories, with public-domain works strongly represented and limited knowledge of several underrepresented-fiction collections.
- Name-cloze accuracy is measured as the fraction of 100 samples from each book for which the model correctly predicts the masked name.The passages were sent through both ChatGPT and GPT-4 using the OpenAI API.
- GPT-4’s top-20 list is dominated by public-domain books and includes many science-fiction and fantasy works.Twelve of the 20 most memorized in-copyright books belong to science fiction or fantasy.
- 12 of the top 20 most memorized in-copyright books are science-fiction or fantasy works.Examples include Harry Potter, 1984, Lord of the Rings, Hunger Games, and Dune.
- GPT-4 and ChatGPT are widely knowledgeable about public-domain texts but know little about Global Anglophone and Black-authored works in the sampled collections.The comparison covers pre-1923 LitBank texts, Global Anglophone fiction, and Black Book Interactive Project or award-winner selections.
- BERT performs near 0 for almost all books, but predicts the masked name in Fifty Shades of Grey 13% of the time.This exception is presented as evidence that the book was known to BERT during training.
6 Analysis
The analysis tests whether models preferentially produce names associated with memorized books and examines whether memorization tracks books’ web prevalence. It also considers the sources contributing to copyrighted material’s online presence and cautions that correlation does not establish direct causation.
- 6.1 Error analysis: ChatGPT and GPT-4 are more likely to predict character names from memorized books than from non-memorized books.The analysis constructs seen and unseen name sets from books in the top and bottom GPT-4 memorization deciles, then tests the resulting log-odds statistic with randomization.
- 6.1 Error analysis: The models’ predictions can draw memorized-book information into other narrative contexts, beyond simply performing better on memorized books.The reported result describes this as information from memorized books bleeding into other narrative contexts.
- 6.2 Extrinsic analysis: GPT-4 name cloze accuracy is correlated with the number of search results for passages from the same books across Google, Bing, C4, and the Pile.The study measures Spearman ρ between book-level name cloze accuracy and average search-result counts for sampled 10-grams.
- 6.2 Extrinsic analysis: Public-domain works show a strong, significant correlation between GPT-4 name cloze accuracy and web search prevalence.For example, Google averages 2,590 results for Alice in Wonderland passages, 1,100 for Huckleberry Finn, and 279 for Anne of Green Gables.
- 6.2 Extrinsic analysis: The observed relationship between web prevalence and memorization does not indicate direct causation and may reflect an unobserved confounder.The paper connects popularity with multiple correlated sources, including library copies, training datasets, academic quotations, and public-web appearances.
- 6.2 Extrinsic analysis: Copyrighted books also appear frequently online, including through reviews and study resources rather than only freely downloadable full texts.The listed sources include Goodreads, Amazon, Course-Hero, Quizlet, and LitCharts.
7 Effect on downstream tasks
The paper tests whether disparities in book memorization affect downstream literary analysis tasks. GPT-4 shows suggestive but inconclusive evidence for narrative-time prediction, while both models perform substantially better on publication-year prediction for memorized books.
- 7 Effect on downstream tasks: GPT-4 is evaluated in few-shot experiments predicting publication year and elapsed narrative time from random 250-word passages.The study focuses on disparities associated with memorization rather than absolute task performance.
- 7.1 Predicting year of first publication: 29.8 versus 3.3 years: ChatGPT’s mean absolute publication-year error is higher for non-memorized than memorized books.For GPT-4, the corresponding errors are 14.5 and 0.6 years.
- 7.1 Predicting year of first publication: 14.5 versus 0.6 years: GPT-4’s mean absolute publication-year error is higher for non-memorized than memorized books.ChatGPT refrains from predicting for 9% of non-memorized books versus 4% of books it knows well; GPT-4 makes valid predictions in all cases.
- 7.1 Predicting year of first publication: −0.39 and −0.37: increasing name cloze accuracy is linked to decreasing publication-year prediction error for ChatGPT and GPT-4, respectively.Both correlations are significant at p < 0.001.
- 7.1 Predicting year of first publication: Publication-year prediction can draw on encyclopedic knowledge of a work, creating contamination risk for tasks expressed as facts about a text.Narrative-time prediction instead requires reasoning directly from passage information because each passage has its own duration.
- 7.2 Predicting narrative time: TOP10 achieves ρ = 0.50 versus BOT10’s ρ = 0.27 for GPT-4 narrative-time prediction, but the difference is not statistically significant.Using the top and bottom quintiles shows no effect, suggesting any effect may be limited to the most memorized books.
8 Discussion
The discussion argues that opaque training data creates ethical, reproducibility, and measurement-validity concerns for cultural analytics. Memorization is concentrated in popular works and particular narrative categories, but the paper does not establish how those memorized values shape generation.
- 8 Discussion: ChatGPT and GPT-4 memorize public-domain and copyrighted books, with memorization tied to web popularity and better downstream performance on popular works.The discussion frames this differential as a risk for cultural-analytics research.
- 8 Discussion: Closed models raise ethical and reproducibility concerns because their training data is unknown outside OpenAI.The paper presents its archaeology as necessary because these systems are closed.
- 8 Discussion: Popular books are poor generalization barometers because models know much more about them than the long tail of literature.The paper specifically cautions against using works such as Alice in Wonderland, Harry Potter, and Pride and Prejudice to set expectations for other texts.
- 8 Discussion: The memorized narrative distribution emphasizes pre-1928 public-domain works and contemporary science-fiction and fantasy.The paper frames this distribution as a question of whose narratives inform model knowledge.
- 8 Discussion: The study does not investigate how memorized texts’ values influence generated narratives or other behaviors, so discussion of that possibility remains speculative.The authors identify this as an area for future work.
9 Conclusion
The conclusion uses data archaeology to assess risks posed by memorized books in closed models used for cultural analytics. Popularity offers only a rough guide to memorization, while known training sources would resolve the underlying uncertainty.
- 9 Conclusion: The paper uncovers fiction memorized by two popular models to assess risks of using closed models in cultural analytics.The work targets uncertainty about when such models will perform well or fail.
- 9 Conclusion: Popularity on the web provides a rough guide to the likelihood of memorization but does not solve the uncertainty created by closed training data.The conclusion states that only open models with known training sources would resolve that fundamental challenge.
Limitations
The study infers model familiarity rather than directly verifying training-data membership, and its scope is limited to English content and one memorization operationalization. Future work is needed to quantify memorization across languages and longer textual spans.
- Limitations: The true training data cannot be verified: the study measures probabilistic familiarity, not whether books actually occur in the models’ training data.The authors state that familiarity alone is sufficient to contaminate test data.
- Limitations: The study is a case study of English, although its analytical method is transferable to other languages with digitized books.The extent of memorization in non-English literary content is left for future work.
- Limitations: Name cloze is only one operationalization of memorization, and longer verbatim training spans may produce more acute memorization.The authors leave the full extent of memorization and contamination in opaque models for future work.
- Limitations: The study uses the OpenAI API and does not access the true training data or underlying system components.This defines the study’s observational boundary.