PaperlayerView exact revision on arXiv

Source-linked AI summary

How much do language models memorize?

John X. Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G. Edward Suh, Alexander M. Rush, Kamalika Chaudhuri, Saeed Mahloujifar

arXiv:2505.24832v3cs.CL

TL;DR

Language-model memorization is difficult to measure because prior approaches do not cleanly separate dataset-specific information from generalization. The paper introduces a compression-based, instance-level framework to make that separation and estimates GPT-style capacity at about 3.6 bits per parameter. Experiments show that memorization gives way to generalization after capacity is reached, while larger datasets make membership inference harder.

  • Problem

    Prior studies struggle to distinguish memorization of specific training data from generalization of the underlying data-generating process.

  • Method

    The paper estimates instance-level memorization through compression and measures it by training models on synthetic bitstrings and real text across model and dataset sizes.

  • Results

    GPT-style transformers store approximately 3.6 bits per parameter; on real text, unintended memorization decreases after capacity is reached and double descent begins.

  • Takeaways & Limitations

    Membership inference follows scaling laws in which larger models memorize more samples, whereas larger datasets make inference harder.

  • Takeaways & Limitations

    Because gradient descent is not guaranteed to find global optima, the measured model capacity is only a lower bound.

Abstract

from arXiv · show

We propose a new method for estimating how much a model knows about a datapoint and use it to measure the capacity of modern language models. Prior studies of language model memorization have struggled to disentangle memorization from generalization. We formally separate memorization into two components: unintended memorization, the information a model contains about a specific dataset, and generalization, the information a model contains about the true data-generation process. When we completely eliminate generalization, we can compute the total memorization, which provides an estimate of model capacity: our measurements estimate that GPT-style models have a capacity of approximately 3.6 bits per parameter. We train language models on datasets of increasing size and observe that models memorize until their capacity fills, at which point "grokking" begins, and unintended memorization decreases as models begin to generalize. We train hundreds of transformer language models ranging from $500K$ to $1.5B$ parameters and produce a series of scaling laws relating model capacity and data size to membership inference.

1 Introduction

The paper introduces an instance-level, compression-based definition of memorization that separates unintended memorization from generalization. Experiments estimate transformer capacity at roughly 3.5–4 bits per parameter and show that exceeding capacity triggers generalization and double descent, while larger datasets hinder membership inference.

  • Contribution: The paper defines memorization as information retained about a specific datapoint and separates it from knowledge of the underlying data-generating process.Unintended memorization concerns dataset-specific information, whereas generalization captures population-level knowledge.
  • Contribution: Compression rate provides a practical estimate of memorized information using model likelihoods rather than output generation alone.The approach is inspired by Kolmogorov and Shannon information theory and treats shorter encodings under model access as evidence of memorization.
  • Experiments: 3.5–4 bits per parameter is the estimated information capacity of GPT-style transformers trained on uniformly sampled bitstrings.The range depends on model architecture and numerical precision, with generalization eliminated in the synthetic setting.
  • Experiments: On real text, models first memorize until capacity fills, then shift toward general reusable patterns as dataset size exceeds capacity.At this transition, unintended memorization decreases while generalization begins to improve evaluation loss.
  • Experiments: Double descent begins when dataset size exceeds model capacity, linking the phenomenon to the transition from sample-level memorization to generalization.The paper reports this relationship for both synthetic bitstrings and text-trained models.
  • Implications: Membership inference becomes harder as datasets grow, while larger models can memorize more samples.The paper proposes scaling laws based on model capacity and dataset size that extrapolate to larger models.

2 Memorization, intended and unintended

The paper separates memorization into information about a specific sample and information explained by the underlying data-generating process. It defines these quantities using information theory and develops a practical compression-based approximation that can be estimated from model likelihoods.

  • Statistical view: The framework defines memorization at the individual-sample level and makes it a function of the final model and sample, independent of the training algorithm.These requirements address the need to reason about concrete datapoints when only a trained model and target sample are available.
  • Statistical view: Mutual information provides the statistical basis for measuring how much dataset information is stored in a trained model.The total information stored in the trained model is I(X, ˆΘ) = H(X) − H(X | ˆΘ).
  • Statistical view: Unintended memorization measures information about a sample that remains unexplained after accounting for the true data-generating model.Generalization is treated as intended memorization, while unintended memorization captures sample-specific information.
  • Statistical view: Dataset-level unintended memorization is lower-bounded by summed per-sample memorization and upper-bounded by the information content of the trained model.Thus, unintended memorization scales with dataset size but cannot exceed model capacity.
  • Compression-based measurement: Because entropy cannot be estimated from one model and one dataset sample, the paper approximates information using Kolmogorov complexity and compression.Kolmogorov complexity is the shortest description length, including when the model is available as a reference.
  • Compression-based measurement: Arithmetic coding estimates conditional description lengths through negative log likelihoods, enabling practical memorization estimates.For the target model, HK(x | ˆθ) is estimated as −log(p(x | ˆθ)); with target and reference models, the method uses −log(max{p(x | ˆθ), p(x | θ)}).

3 Model Capacity for Memorization

The paper defines model capacity as the total unintended memorization a learning algorithm can store, estimating it by measuring memorization on independent random data. GPT-style transformers consistently store roughly 3.5–3.6 bits per parameter, although gradient-descent measurements provide only a lower bound.

  • Defining model capacity: Capacity is the total amount of unintended memorization that can be stored across a model’s parameters.When generalization is impossible, summing per-datapoint unintended memorization estimates model capacity.
  • Measuring capacity with synthetic sequences: Uniformly sampled independent tokens eliminate generalization, allowing memorization to be measured from known dataset information content.The method estimates unintended memorization by comparing dataset entropy with model-conditioned code length.
  • Capacity results: Memorization plateaus once models reach capacity, while small datasets are completely memorized by models with sufficient capacity.The plateau provides the operational basis for estimating capacity from saturation across dataset sizes.
  • Capacity results: 3.5–3.6 bits per parameter is the observed memorization range for the GPT-style models studied.Capacity is estimated as the maximum unintended memorization measured across dataset sizes, with a smooth relationship to parameter count.
  • Limitations: Gradient descent may fail to find global optima, so these experiments measure only a lower bound on model capacity.This limitation applies to capacity estimates obtained from the trained models.
  • Capacity results: 3.56–3.65 × 10^6 bits were memorized across datasets from 16,000 to 4M samples, indicating robust capacity estimates.The 4M- and 8M-sample datasets converged to lower measured memorization, suggesting they had not reached the same saturation level.
  • Precision: Capacity increased from 3.51 to 3.83 bits per parameter when training precision changed from bfloat16 to fp32.The paper characterizes this as a small increase across model sizes.

4 Disentangling Unintended Memorization from Generalization

On real text, the framework separates sample-level unintended memorization from population-level generalization using a higher-capacity reference model. As data exceeds model capacity, memorization falls, generalization emerges, and extraction approaches the test-set baseline.

  • Experimental setup: Real-text experiments use an equal-sized model trained on the maximum dataset as a reference for separating unintended memorization from generalization.The study also considers an oracle reference model and performs perfect deduplication of 64-token sequences.
  • Extraction: Membership inference is generally easier than extraction across the reported comparisons.Figure 10 compares membership inference with 32-token-prefix suffix extraction rates.
  • Unintended memorization: Sample-level unintended memorization increases with model parameters and decreases with training-set size.Relative to an oracle reference, memorization first rises as the smaller model learns more about a small set, then falls as it generalizes.
  • Double descent: Double descent begins when dataset capacity exceeds model capacity, after evaluation performance worsens near the capacity boundary.The paper connects this transition to models sharing information across datapoints instead of memorizing each individually.
  • Extraction: For sufficiently large deduplicated datasets, successful training-data extraction converges to nearly the test extraction rate.The authors therefore attribute remaining extraction to generalization rather than sample-specific memorization.

5 Memorization and Membership

The paper studies loss-based membership inference as a function of model and dataset size, finding that larger models support more inference while larger datasets make average-point inference harder. A sigmoid scaling law captures this relationship and predicts near-random performance for contemporary token-to-parameter ratios.

  • Empirical relationship: Larger models can memorize more samples, while larger datasets make membership inference harder.The study develops a predictive relationship based on model capacity and dataset size.
  • Membership-inference setup: Membership inference uses a cutoff on sample loss to classify whether a sample belongs to the training dataset.All reported membership-inference results use this standard loss-based procedure.
  • Functional forms: Membership inference follows a roughly sigmoidal curve with dataset size for fixed model capacity.Scores begin near 1 for models overfitting tiny datasets and decay toward 0.5 as train and test losses become harder to distinguish.
  • Limiting behavior: As dataset size approaches infinity, membership-inference performance decreases to 0.5, or essentially random performance.The scaling law also predicts extraction and membership inference to be impossible for a model trained on an infinite dataset.
  • Scaling-law implication: 102 tokens per parameter is the contemporary threshold above which the law predicts a membership-inference score of 0.5.Within the paper’s formulation, statistically significant loss-based membership inference is then not possible.
  • Validation: Predicted membership-inference F1 scores were generally within 1.5 points of measured values on 125M- and 1.5B-parameter models.The largest error occurred for estimated F1 of 0.75, where the sigmoid is steepest.

6 Related Work

The related work situates the paper among research on compression-based memorization, model capacity, information regularization, double descent, and alternative operational definitions of memorization.

  • Compression and information: Prior work connects language modeling with compression through Shannon information and Kolmogorov complexity.Earlier studies observed that more accurate English models compress text into fewer bits.
  • Language-model capacity: Research on language-model capacity builds on results for perceptrons, effective capacity, and measurements based on facts or random labels.The cited literature examines how architecture, training procedures, and representations affect empirical capacity.
  • Information regularization: Information-regularization work studies mutual information between learning inputs and outputs, relating to Shannon-information views of memorization.The paper also connects these ideas with the double-descent literature.
  • Alternative definitions: Alternative memorization definitions include greedy generation, adversarial extraction, counterfactual effects, and compression-related criteria.The paper compares these definitions with its instance-level information-based formulation.
  • Concurrent work: A concurrent work also proposes a theoretical memorization definition based on Kolmogorov complexity.The paper identifies this as a related approach rather than an earlier operational definition.

7 Conclusion

The paper defines and measures memorization in bits, evaluates how it scales with model and dataset size, and validates a membership-inference scaling law. Additional experiments report accurate capacity predictions under model resizing.

  • The paper proposes measuring the exact number of bits a model knows about a dataset and uses this definition to study transformer capacity and scaling.
  • The analysis examines how extraction and F1 score vary with model and dataset size and proposes a membership-inference scaling law validated on larger models.
  • 1.7% average error when scaling S and 1.8% when scaling V, indicating accurate predictions of total memorization under sequence-length and vocabulary resizing.

A.2 Additional memorization results

On text data, memorization reaches a model-dependent plateau. As datasets grow beyond capacity, models shift from sample-specific memorization toward reusable generalization.

  • Text memorization plateaus near model capacity, with total memorization remaining approximately constant as dataset size increases.The authors interpret the constant sum as the model’s capacity upper bound.
  • Unintended memorization first increases with dataset size, then decreases after models fill capacity and replace sample-level information with generalizable knowledge.

A.3 Comparison of distributions memorized

Text and random bitstring datasets distribute memorization differently: text has broader overlap between train and test losses and concentrates memorization on unusual examples. Rare words, especially non-English tokens, are strongly associated with higher memorization.

  • Distribution-level analysis: Text has lower average loss but greater spread and more train–test overlap than random bitstrings, helping explain weaker membership inference.
  • Which datapoints are most memorized?: Large text datasets concentrate memorization on a small number of datapoints even after deduplication, so duplication cannot explain this concentration here.
  • Which datapoints are most memorized?: Higher TF-IDF corresponds to greater unintended memorization because documents containing rarer words are more strongly memorized.The highest-TF-IDF example had the third-highest memorization, and the model could reproduce it from one token.
  • Which datapoints are most memorized?: Among the top twenty memorized sequences, all but three contain tokens from Japanese, Chinese, or Hebrew, consistent with manual analysis of rare non-English tokens.
  • Scaling law fit: The sigmoidal scaling-law fit estimates observations within 1–2%, despite the function’s imperfect fit.

A.6 Proof of Proposition 1

This appendix section supplies proofs for Proposition 1 using conditional independence under the latent data-generation model and connects algorithmic and statistical information quantities.

  • Because the data are sampled i.i.d., the variables R_i=[X_i | Θ] are independent once conditioned on the underlying model Θ.

A.7 Proof of Proposition 4

The proof connects algorithmic mutual information with mutual information through a lemma, then applies it to a joint distribution involving the data and learned parameters. It concludes the proposition by setting ε = 2HK(f).

  • The proof begins by stating a lemma connecting algorithmic (Kolmogorov) mutual information and mutual information.
  • It defines f as the density of the joint distribution of X_i conditioned on θ and the learned parameters.
  • The lemma is then applied to this density to prove the proposition.
  • The proof concludes by setting ε = 2HK(f).

A.8 Limitations

The paper compares its instance-level memorization definition with stability-, extraction-, membership-, and data-copying-based notions. It also states that its experimental conclusions are specific to the proposed environment and may not generalize broadly.

  • Limitations: The paper’s experimental results may not generalize to other datasets, architectures, or training setups.
  • Stability-based notions: Stability-based notions depend crucially on the learning algorithm and are not ideal for sample- or model-level analysis in this setting.Differential privacy is worst-case, while Feldman’s notion applies to classification and focuses mainly on label-input associations.
  • Extraction-based memorization: Extraction-based notions define memorization through the complexity of eliciting a sample from the model.
  • Extraction-based memorization: Prompt-based extraction remains limited because it does not account for generalization and focuses on one compression mechanism.
  • Membership/attribute inference: Membership and attribute inference rely heavily on the learning algorithm and data distribution and do not provide a sample-level memorization notion.
  • Data copying in generative models: Data-copying notions require extraction of only part of a training sample but retain challenges associated with extraction-based definitions.
Loading 2505.24832v3…