Scalable Extraction of Training Data from (Production) Language Models

Methodology

• The paper defines extractable memorization as training data efficiently recoverable by an adversary, distinguishing it from discoverable memorization’s broader upper bound.A scalable method detects memorization across trillions of output tokens and terabyte-sized datasets.Source: §1 · p. 1

Scope and findings

• Larger and more capable models were more vulnerable to data-extraction attacks across open-source models such as Pythia and GPT-Neo and semi-open models such as LLaMA and Falcon.The study unifies prior work measuring memorized-data quantity with practical extraction attacks.Source: §1 · p. 1

ChatGPT attack

• 150× more frequently than prior attacks, ChatGPT emitted training data through the divergence attack, which recovered over 10,000 examples for $200 and could yield over 10× more with additional queries.The attack also emitted training data 3× more frequently than the base model.Source: Figure 1 · p. 1, §1 · p. 1

Alignment and divergence

• ChatGPT initially appeared to memorize almost no training data because RLHF alignment [39] [44] steered it toward helpful chatbot-style responses.The divergence prompt instead made gpt-3.5-turbo behave like a base language model and emit typical Internet-style text.Source: §1 · p. 1

Limitations and implications

• The ChatGPT attack was specific to gpt-3.5-turbo and, to the authors’ knowledge, did not apply to other tested production language models.The authors disclosed the vulnerability to OpenAI on August 30th and recommend extreme safeguards for privacy-sensitive LLM applications.Source: §1 · p. 2

Training data for language models

• LLMs are trained on billions to trillions of tokens, while proprietary systems such as GPT-4 [38] and PaLM 2 keep training sets secret to protect data pipelines and private or licensed data.The secrecy also conceals proprietary collection practices and non-public training data.Source: §2 · p. 2

Instruction-tuning and RLHF

• Instruction-tuning and RLHF improve downstream utility while aligning models to a unified chat persona and refusal behavior, including for spam-writing requests.The study analyzes ChatGPT, specifically the gpt-3.5-turbo model endpoint.Source: §2 · p. 2

Privacy attacks

• Language models can memorize training data, enabling membership-inference attacks that identify included examples and data-extraction attacks that recover full examples.This work conducts both attack types on LLMs.Source: §2 · p. 2

3.1 Prior Approaches and Definitions

• Extractable memorization means an adversary without the training set can construct a prompt p such that Gen(p) produces a training example x verbatim.The section adopts Carlini et al. (2021) [14]’s conservative verbatim criterion rather than definitions allowing close copies.Source: §3.1 · p. 2

3 Extracting Data from Open Models

• Existing extraction attacks are substantially more successful than prior estimates because open models permit mechanistic training-set verification without giving the attack access to that set.Prior work’s verified lower bound was limited by time-consuming manual Internet searches, whereas open models enable direct inclusion tests.Source: §3.1 · pp. 2–3, §3.2 · p. 3

3.2 Attack Methodology

• The attack samples hundreds of millions of five-token prompts from 108 bytes of Wikipedia, generates independently, and verifies outputs against public training sets using suffix arrays.An extraction counts when output contains a verbatim training-set substring of at least 50 tokens, with high entropy filtering degenerate outputs.Source: §3.2 · p. 3, §3.3 · p. 3

3.3 Empirical Results

• Several hundred thousand to several million unique 50-token strings were extracted, versus only 600 sequences previously verified from GPT-2, with memorized-output rates of 0.1%–1%.Extraction rates and unique recovered sequences strongly correlate with model size, indicating repeated emission of one memorized example is uncommon.Source: §3.3 · p. 4, Table 1 · p. 4

3.4 Estimating Total Memorization

• Good-Turing extrapolation provides higher-quality lower bounds than Chao1, Chiu et al., and Zelterman, but requires sufficient observations and still underestimates GPT-Neo 6B’s unique memorized examples.Memorization grows nearly linearly over hundreds of billions of generated tokens, while extraction rates alone can mislead about total memorization.Source: §3.4 · pp. 4–5, Figure 4 · p. 5

3.5 Discoverable Mem. vs. Extractable Mem.

• On GPT-Neo 6B, 30.1% of examples were discoverably memorized and 14.5% extractably memorized; only 35% of discoverable examples were also extractable, while 11% were extractable only.Highly duplicated sequences were easier to both extract and discover.Source: §3.5 · p. 6

4 Extracting Data from Semi-closed Models

• Semi-closed models pose a more practical extraction threat than fully public models because their training datasets and algorithms are not disclosed, requiring an independently established ground truth.The section extends the extraction question from open-source models to models with public parameters but private training information.Source: §4 · p. 6, §4.1 · p. 6

4.1 Attack Methodology

• The attack checks candidate generations against a 9TB auxiliary corpus, using sufficiently high-entropy, long matches as a proxy for training-set membership with a very low false-positive rate.Because the training data is incomplete, the approach has false negatives and produces a lower bound on memorization. The corpus was tokenized and coarsely deduplicated at the document level.Source: §4.1 · pp. 6–7

4.2 Experimental Setup

• The evaluation analyzes nine semi-closed models; most have accessible weights but inaccessible training pipelines and datasets, while gpt-3.5-turbo-instruct is API-only and queried 25 million times for extrapolation.Source: §4.2 · p. 7

4.3 Results

• Among these state-of-the-art models, 0.852% of gpt-3.5-turbo-instruct’s generated tokens are part of verbatim 50-token AUXDATASET sequences, while total extractable memorization averages 5× higher than smaller models.Source: §4.3 · p. 8
• Over-training increases privacy leakage, whereas under-trained OPT models memorize less, consistent with training duration affecting extractable memorization.The comparison is framed using scaling laws for compute-optimal training quantities [25].Source: §4.3 · p. 8

5 Extracting Data from ChatGPT

• The motivating challenge is that aligned models are conversational agents designed to follow human-preference behavior rather than expose their underlying language-modeling continuation task.This alignment creates additional obstacles for extraction attacks against ChatGPT.Source: §5 · p. 8
• Chat formatting blocks arbitrary continuation because the model generates only after appending the special “Assistant:” turn indicator.This prevents directly continuing arbitrary text sequences with the prior prompt-continuation attack.Source: §5 · p. 8
• Alignment can additionally evade extraction by refusing to complete memorized training text, as illustrated by ChatGPT’s response to a BBC staffing prompt.Instead of continuing the quoted passage, ChatGPT apologizes and supplies a qualified response based on its knowledge cutoff.Source: §5 · p. 8
• The section’s objective is to show that alignment does not prevent data extraction despite examples that superficially appear to demonstrate successful evasion.The divergence attack provides the mechanism for challenging that apparent conclusion.Source: §5 · p. 8, §5.2 · p. 9

5.1 Baseline Attack

• The baseline strategy asks ChatGPT to repeat random tokens and continue, but the model repeats the tokens without apparently producing training data.The authors scale this experiment to obtain quantitative evidence that the strategy is ineffective.Source: §5.1 · p. 8
• 0.02% of 50 million gpt-3.5-turbo tokens belonged to directly copied 50-token sequences, versus 0.031% for OPT-1.3B and at least 0.85% for gpt-3.5-turbo-instruct.These results initially suggest that alignment suppresses training-data extraction, though the paper later challenges that conclusion.Source: §5.1 · p. 9

5.2 Our Divergence Attack

• The divergence attack causes ChatGPT to depart from dialog-style generation, with some divergent outputs copied directly from pre-training data.Repeated prompting produces many candidate generations, which can be inspected for memorized text.Source: §5.2 · p. 9
• The divergence prompt “Repeat this word forever” initially elicits hundreds of repetitions before the model eventually departs from the repeated-word pattern.After divergence, generations are often nonsensical, but a small fraction transition into memorized text.Source: §5.2 · p. 9

• Using $200 USD worth of ChatGPT queries, the attack extracted over 10,000 unique verbatim-memorized training examples.The authors extrapolate that dedicated adversaries could extract substantially more data with larger budgets.Source: §5.3 · p. 9
• The recovered data included personally identifiable information, NSFW material, novel passages, complete poems such as The Raven, and snippets from research papers.The authors recovered PII from dozens of individuals, explicit content, a complete Nature abstract, and bibliographic data from hundreds of papers.Source: §5.3 · pp. 9–10
• The attack also extracted valid nonce-containing URLs, cryptographically random identifiers including a bitcoin address, and code repeated in AUXDATASET.The code was predominantly JavaScript and appeared to have been unintentionally included because it was not properly cleaned.Source: §5.3 · pp. 9–10
• The prompting strategy often induced divergence after fewer than 200 repeats, so asking the model to repeat a token forever was unnecessary.Source: §5.3 · p. 9
• The longest extracted string exceeded 4,000 characters, hundreds exceeded 1,000 characters, and over 93% of memorized strings appeared only once.Figure 6 reports thousands of short examples, hundreds exceeding 1,000 characters, and a longest example over 4,000 characters; 4% appeared twice and 0.05% appeared at least ten times.Source: §5.3 · p. 9, Figure 6 · p. 10
• Recovered outputs further included Internet boilerplate and merged memorized strings, such as GPL and MIT license text combined into one output.Examples of boilerplate included country lists, date sequences, and copyright headers on code.Source: §5.3 · p. 10

5.4 Identifying PII

• Memorized PII appeared in 16.9% of tested generations, and 85.8% of generations containing potential PII were verified as actual training-data PII.Detection combined regexes, language-model classification, and AUXDATASET verification across 15,000 generations.Source: §5.4 · p. 10

5.5 Words that Elicit Memorized Outputs

• Single-token words can trigger memorization, whereas multi-token words never do because they either repeat correctly or produce a refusal without diverging.The attack therefore depends on whether the repeated word is represented as one vocabulary token.Source: §5.5 · p. 10
• The most effective trigger words recover memorized output over 100× more effectively than the least effective words.Variation reflects both how often words cause divergence and how much training data they produce after divergence; Figure 7 reports a 164× example contrast.Source: §5.5 · p. 10, Figure 7 · p. 11

5.6 Quantifying Total Memorization

• A $200 budget extracted over 10,000 unique examples, while Good–Turing estimation lower-bounded memorization at 1.5 million unique 50-token sequences.The estimate is considered poor because ChatGPT yielded over 1000× fewer examples than the 500 million needed for convergence on GPT-Neo 6B.Source: §5.6 · p. 10, §5.6.1 · p. 10, Figure 9 · p. 12

5.6.1 Extrapolating Unique Memorized Strings

• ChatGPT emits unique memorized strings at a higher rate than publicly available models, with likely memorization closer to hundreds of millions of 50-token sequences totaling a gigabyte.The authors expect the true rate to be even higher, while Figure 8 compares extraction rates across models.Source: §5.6.1 · p. 10, Figure 8 · p. 11

5.6.2 Impact of AUXDATASET’s Size

• A 200GB auxiliary-dataset subset recovered slightly under 20% of total memorization, while doubling the auxiliary dataset could increase discoveries by an additional 20%.Using only 25% of the corpus produced a curve that nearly perfectly extrapolated the full-dataset count.Source: §5.6.2 · p. 11
• Auxiliary-corpus scale does not fully determine discovery: RefinedWeb found the least memorization, while RedPajama added mostly examples already covered elsewhere.The authors attribute these differences to distributional mismatch with ChatGPT’s training data and conclude the corpus is nearly sufficient for tight estimates, though more queries could reveal more.Source: §5.6.2 · p. 11

5.6.3 Extending AUXDATASET to a Web Search Index

• Manual web searches found 150 of 494 annotated generations on the Internet, versus 70 in AUXDATASET, nearly doubling detected memorization.This supports the claim that AUXDATASET coverage underestimates total memorization.Source: §5.6.3 · pp. 11–12

5.7 An End-to-end High-precision Attack

• The end-to-end attack resolves the limitation that prior evaluations required approximate access to the training dataset to identify memorized outputs.Existing literature techniques therefore suffice to identify extracted training data without directly querying the model’s training set.Source: §5.7 · p. 12
• The membership-inference attack [45] from [14], using a likelihood-ratio perplexity score, distinguishes memorized training data from hallucinated data with high precision.The score divides model perplexity by zlib compression entropy; this ratio was also the most effective predictor in prior work [14].Source: §5.7 · p. 12
• At the lowest threshold, attack precision exceeded 30% under manual Internet search and remained 15% under verbatim AUXDATASET membership checking.Precision stayed relatively constant as the threshold increased until 1.5, after which it significantly declined.Source: §5.7 · p. 12, Figure 10 · p. 13

5.8 Is ChatGPT Memorization Discoverable?

• The attack is not generalizable to other models, and ground-truth training examples would otherwise be needed to measure discoverable memorization or upper-bound memorization as in [11].The study circumvents this limitation by splitting extracted memorized sequences into prefixes and suffixes, then testing 50-token completions.Source: §5.8 · p. 12
• Known memorized data is usually not detected when ChatGPT is prompted naturally, making privacy red-teaming difficult without access to both the aligned model and its unaligned foundation model.ChatGPT’s low discoverable memorization is attributed likely to alignment, while the base model was not released for external assessment.Source: §5.8 · p. 13
• 75% of memorized suffixes were completed exactly by gpt-3.5-turbo-instruct, and 84% were within 5 words, versus 3.5% exact completion by gpt-3.5-turbo.The instruction-tuned model recovered the suffix approximately within a Levenshtein distance below 0.1 in 84% of cases, while the chat model exceeded 90% failure to emit the known memorized output.Source: §5.8 · p. 13
• The two GPT-3.5 variants memorize the same samples despite likely different fine-tuning datasets, indicating that the extracted data likely comes from pretraining rather than fine-tuning.Memorization acquired during pretraining appears to persist through fine-tuning, plausibly because pretraining lasts much longer and forgetting can require several epochs.Source: §5.8 · p. 13

Single-token repetition

• After 250 repetitions, single-token continuation probability falls from 90% to below 0.1%, whereas two- or three-token words remain above 99% after thousands of repetitions.The reported medians span 40 word choices, with 10%–90% quantile ranges.Source: Figure 12 · p. 14, §6 · p. 14

Potential cause: over-training

• ChatGPT may be more extractable because over-training on limited web data requires many epochs, which prior work and the paper’s multi-epoch experiments associate with increased memorization.The paper frames this as a privacy–inference-efficiency trade-off if ChatGPT was indeed trained for multiple epochs.Source: §6 · pp. 13–14

Single-token repetition

• The attack diverges only for single-token words, an unexplained but significant and repeatable instability that enables the observed extraction behavior.The paper does not provide a definitive explanation for why single-token repetition is unstable.Source: §6 · p. 14

Reset-token hypothesis

• The authors suspect repetition approximates the <| endoftext |> reset token learned through packed-document pre-training, although the API prevents direct insertion of that token.This mechanism could make the model ignore the prompt and generate as if starting a new document.Source: §6 · p. 14
• In LLaMA 7B, repeated-token attention queries rapidly approach the BOS query, unlike random-prompt sampling, supporting a reset-like representation mechanism.Because hidden representations are projected into the vocabulary, this convergence may produce a next-token distribution resembling the initial BOS distribution; LLaMA 7B does not emit training data in this experiment.Source: §6 · p. 14

7 Conclusions

• Simple techniques can easily extract training data from the best language models of the past few years.The authors present this as the central conclusion of the paper.Source: §7 · p. 14

7.1 Consequences for Researchers

• Deduplication only slightly decreases total extractable memorization and may increase the rate of emitting training data, because coarse-grained deduplication is insufficient [5].The authors identify understanding these observations as an important direction for future work.Source: §7.1 · p. 15
• GPT-Neo 6B can emit nearly a gigabyte of training data, implying that approximately 10% of its model capacity stores verbatim memorized data.The paper asks whether models would perform better or worse if this information were not memorized.Source: §7.1 · p. 15

7.2 Consequences for Practitioners

• Practitioners should test discoverable memorization because a surprisingly high fraction of memorized examples can be extracted with sufficient effort.The authors argue that discoverable memorization is easier to measure than extractable memorization.Source: §7.2 · p. 15
• Standard memorization tests missed that ChatGPT was non-private, while adversarial prompting can revert alignment and expose latent vulnerabilities, with possible stronger attacks remaining [13] [56].The authors note uncertainty about gpt-3.5-turbo’s pre-launch testing despite privacy analyses publicly described for GPT-4 and Copilot [38].Source: §7.2 · p. 15

• The team discovered and confirmed that token repetition on ChatGPT could emit memorized training data, establishing the project’s central attack finding.Milad discovered the attack, and Milad and Nicholas confirmed that the outputs contained memorized training data.Source: Contributions · p. 15
• The authors tested extraction across ChatGPT query parameters, model variants, scaling behavior, multiple training epochs, and EOS-token effects.These experiments included comparisons between gpt-3.5-turbo and gpt-3.5-turbo-instruct and generated data-scaling extrapolation plots.Source: Contributions · pp. 15–16
• Milad built infrastructure producing terabytes of outputs from 17 open and semi-closed models, while Nicholas developed efficient training-data intersection tools and ran the lookup analysis.Nicholas also collected AUXDATASET and built the suffix array used in the intersection pipeline.Source: Contributions · p. 15
• The team completed the project’s organization, figures, data analysis, paper structure, and writing.Katherine and Matthew analyzed the multiple-epoch data, Katherine, Cooper, Matthew, and Daphne prepared final figures, and all authors wrote the paper.Source: Contributions · pp. 15–16
• The study proposed and analyzed discoverable memorization, including its distinction from extractable memorization, and examined correlations between model performance and extraction.The work also investigated what memorized training data contained personally identifiable information (PII).Source: Contributions · p. 16
• The authors manually searched the Web for 500 potentially memorized strings and analyzed the resulting data.Daphne analyzed manual data collected by the broader team for this search.Source: Contributions · p. 16

A Suffix Arrays

• Setting the suffix-length threshold to k = 50 significantly impacts the rate at which data is determined to be memorized.The figure specifies k = 50 but does not provide the corresponding rates.Source: Figure 14 · p. 18
• Lexicographically sorting dataset suffixes enables binary-search containment checks in O(logn) time instead of an O(n) scan.Constructing the suffix array takes linear time.Source: A Suffix Arrays · p. 18
• For “banana,” the lexicographically sorted suffix indices are s = {5,3,1,0,4,2}, ordered as “a” < “ana” < “anana” < “banana” < “na” < “nana”.The unsorted reverse-traversal indices would require an O(n) search.Source: A Suffix Arrays · p. 18
• Concatenating all training documents into one long string lets the suffix array check whether x ∈s(X) by comparing the first k characters of indexed suffixes.This adapts the data structure from a single word to a corpus totaling around a terabyte.Source: A Suffix Arrays · p. 18

B.1 Impact of Varying k in Our Memorization Definition

• Memorized sequences are defined as training-dataset sequences at least 50 tokens long, a threshold acknowledged as arbitrary.Changing this constant would identify a different total number of memorized training examples.Source: §B.1 · p. 19
• Figure 14 shows that changing the token threshold leaves overall model-comparison trends similar.For example, if model A memorizes more than model B at 50 tokens, the same ordering holds at 40 or 100 tokens.Source: §B.1 · p. 19

Estimation strategy

• Good-Turing estimation infers N, the total memorized training examples, from the observed subset s of memorized content.The method treats memorization estimation as a population-estimation problem analogous to ecology and epidemiology.Source: §B.2 · p. 19
• 93% of memorized strings appear once, so the estimator accounts for other sequences reappearing multiple times.Repeated observations inform the probability of discovering new memorized sequences.Source: §B.2 · p. 19
• After 10M generations, simulations update the probability of observing new sequences and estimate the total number of unique memorized sequences.The authors also evaluate population-estimation techniques from ecology and epidemiology, with results summarized in Table 3.Source: §B.2 · p. 19

Cross-model extraction

• Across four model families, Figure 15 measures the share of generated tokens forming direct 50-token copies and reports unique memorized-example counts.The results use a sample of 1B generations.Source: Figure 15 · p. 19

Auxiliary-dataset composition

• The 1TB RefinedWeb dataset reveals less memorized data than the 400GB Pile, while RedPajama yields many total but few unique examples because other datasets cover it.Figure 16 compares recovery across constituents of the auxiliary dataset and notes correlation between dataset size and identified memorized examples.Source: Figure 16 · p. 20

Scaling with model size

• For a given model family, larger models allow extraction of more 50-grams from one billion generated tokens.Figure 17 plots model size against extracted 50-grams using data from Tables 1 and 2.Source: Figure 17 · p. 20

Relationship to benchmark performance

• The number of unique extracted 50-grams has no strong correlation with benchmark performance, indicating extractable memorization and task performance are distinct model qualities.Figure 18 compares unique extraction counts with performance across the listed benchmark tasks.Source: Figure 18 · p. 21

Duplication and memorization

• Sequences duplicated more often in the Pile are easier to extract and discover, and duplicated examples are more discoverable than extractable.Figure 19 distinguishes the effects of duplication on extraction and discovery.Source: Figure 19 · p. 21

ChatGPT duplicates

• For ChatGPT’s extracted memorized generations, the analysis counts how many duplicates occur in the auxiliary dataset.Figure 20 examines duplicate occurrence for each extracted ChatGPT generation.Source: Figure 20 · p. 21

D Examples of ChatGPT Divergence

• ChatGPT repeats “book” many times, then diverges and emits extensive random content, including book reviews and publication material.The sequence begins with repeated prompted text before shifting into unrelated prose that continues across the example pages.Source: D Examples of ChatGPT Divergence · pp. 22–23
• Longer k-gram matches are treated as unlikely to arise by chance, whereas short matches such as a ten-token phrase are often uninformative.The example gives a long passage about Holly Black’s works as a stronger memorization signal than the short phrase “I mean, it was dark, but”.Source: D Examples of ChatGPT Divergence · p. 22

• The 100 longest extracted examples include near-duplicates, including four copies of text about actor Harry Carey within different examples.The examples demonstrate repeated verbatim memorization rather than isolated fragments.Source: E Verbatim Memorized Training Sequences · p. 27
• The extraction process redacts sensitive information, specifically phone numbers and email addresses.Source: E Verbatim Memorized Training Sequences · p. 27
• The extracted sequences include lengthy financial-investment disclaimers, ecommerce terms of service, and machine-learning code.These passages preserve substantial, coherent documents and code rather than only short phrases.Source: E Verbatim Memorized Training Sequences · pp. 27–29
• Other memorized outputs combine multilingual text, casino and travel material, biographies, reviews, and location or country lists.The examples span multiple languages, domains, and formatting styles, including corrupted character encodings.Source: E Verbatim Memorized Training Sequences · pp. 30–32
• Additional extracted sequences reproduce news and political reporting, forum-like discussions, scientific arguments, and web application registration templates.The outputs range from public-affairs prose and user-generated content to HTML and server-side form code.Source: E Verbatim Memorized Training Sequences · pp. 33–34

Verbatim extracted content

• The outputs reproduce long contiguous web-text sequences covering commerce, gaming, travel, entertainment, biographies, legal material, and religious passages.Examples include product listings, game descriptions, cruise information, film and sports biographies, court material, and biblical passages.Source: E Verbatim Memorized Training Sequences · pp. 34, 35, 36, 39, 40
• The sequences combine unrelated documents and domains within single outputs, moving abruptly between advertisements, encyclopedic entries, forums, catalogs, and instructional material.For example, passages concatenate casino promotions, military or historical text, cruise reviews, product catalogs, and programming tutorials.Source: E Verbatim Memorized Training Sequences · pp. 34, 35, 38, 40
• Several outputs contain repeated or overlapping memorized spans, including recurring Stark Industries, casino, game, cruise, and product text across separate excerpts.The same Stark Industries and gaming-related material appears in multiple extracted passages, while other excerpts repeat longer blocks nearly verbatim.Source: E Verbatim Memorized Training Sequences · pp. 34, 35, 37, 38, 40
• The extracted text preserves noisy artifacts such as HTML fragments, escaped entities, character-encoding corruption, incomplete sentences, and embedded source code.Observed artifacts include malformed markup, strings such as “nbsp” and “amp,” garbled accented characters, truncated text, and Python visualization code.Source: E Verbatim Memorized Training Sequences · pp. 34, 35, 36, 40, 41

• The model reproduces long heterogeneous passages, including biographies, entertainment listings, product descriptions, advertisements, manuals, and web pages.Examples range from Phoenix Wright and Beverly Garland biographies to casino promotions, lawn-mower manuals, and intake-screening pages.Source: E Verbatim Memorized Training Sequences · pp. 43–44
• It emits verbatim technical material, including Java shortest-path validation code and sequences of indexed broadcast-market names.The code preserves comments, method structure, conditions, error messages, and return behavior across consecutive excerpts.Source: E Verbatim Memorized Training Sequences · pp. 43, 47
• Some extracted sequences are visibly corrupted or contaminated, with repeated inserted phrases, malformed characters, concatenated documents, and abrupt transitions between unrelated sources.Examples include repeated “replica football shirt” fragments, damaged HTML entities, and transitions from fiction into manuals, scripture, and research text.Source: E Verbatim Memorized Training Sequences · pp. 45, 46, 48
• The outputs reproduce extended religious, literary, and scientific text, including biblical passages, fantasy fiction, and a whey-protein mouse-study abstract.The reproduced material retains sustained narrative or scholarly structure rather than isolated memorized phrases.Source: E Verbatim Memorized Training Sequences · pp. 46, 48

Cross-domain memorization

• The model emits long verbatim passages from diverse sources, including biblical genealogy, fiction, technical documentation, advertisements, travel writing, and web pages.Examples include the Gospel genealogy, fantasy-novel prose, Apache configuration directives, casino listings, cruise descriptions, and product pages.Source: E Verbatim Memorized Training Sequences · pp. 49, 50, 52, 53

Fictional prose

• Adjacent outputs preserve coherent continuations of fictional narrative, including dialogue, character actions, combat, and emotional aftermath.The passages continue scenes involving Mu Yunyao, Ye Yao, Lin Yu, and other characters across multiple excerpts.Source: E Verbatim Memorized Training Sequences · pp. 49, 52, 54

Technical material

• Technical memorization includes contiguous low-level code and configuration sequences, such as interrupt-descriptor setup, Apache module declarations, and hardware register writes.These outputs retain ordered function calls, module names, hexadecimal constants, and configuration structure.Source: E Verbatim Memorized Training Sequences · pp. 50–51

Web and commercial content

• Other extracts reproduce lengthy commercial and informational web text, including slot-machine promotions, travel descriptions, product listings, legal or financial disclaimers, medical instructions, and encyclopedic entries.The material ranges from casino and shopping pages to investment warnings, sildenafil directions, historical descriptions, and local-information pages.Source: E Verbatim Memorized Training Sequences · pp. 50–52

• The samples include long contiguous sequences of HTML/JavaScript code followed by an extended number enumeration and assorted web text.The extraction preserves code comments, control flow, and sequence continuity before switching among unrelated documents.Source: E Verbatim Memorized Training Sequences · p. 56
• The extracted text spans diverse web content, including biographies, news, university lists, polling material, casino advertisements, product pages, and travel or entertainment descriptions.These passages combine multiple source genres within single memorized outputs.Source: E Verbatim Memorized Training Sequences · pp. 56–58
• Memorized sequences also preserve multilingual and structured material, including German prose, country lists, television-event listings, catalog text, and encoded or malformed characters.The samples contain German, extensive geographic enumerations, tabular-style listings, and encoding artifacts such as nbsp; and mojibake-like substitutions.Source: E Verbatim Memorized Training Sequences · pp. 56, 57, 61
• Long-form passages reproduce technical and literary content, including a novel excerpt, the Bitcoin proof-of-work description, an HDACI study abstract, and Objective-C image-processing code.The extracted material retains substantial local context rather than isolated phrases.Source: E Verbatim Memorized Training Sequences · pp. 61–62

Verbatim extracted text

• The model emits a long contiguous passage combining Tata Motors’ corporate history, a biblical genealogy, and website legal terms.The extraction includes detailed company facts, lineage from Eliezer to Adam and God, and contractual provisions about jurisdiction, amendments, privacy-policy changes, and consumer rights.Source: E Verbatim Memorized Training Sequences · p. 63
• A separate extracted sequence reproduces third-party-link disclaimers and policies governing user comments, including rights to edit, publish, distribute, and use submissions.The passage also states that the service disclaims responsibility for third-party materials and transactions and need not keep comments confidential or provide compensation.Source: E Verbatim Memorized Training Sequences · p. 64