Source-linked AI summary
Emergent and Predictable Memorization in Large Language Models
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, Edward Raff
TL;DR
Verbatim memorization of sensitive training sequences creates deployment concerns, yet tools for predicting specific examples before expensive training are limited. The paper evaluates small-model and partial-checkpoint forecasts on Pythia, finding both ineffective at low relative compute while deriving an equi-compute scaling law and recommendations. These results provide guidance for forecasting non-memorization, although generalization beyond Pythia remains unresolved.
Problem
Specific memorization can expose sensitive data, but existing tools provide limited early detection and often address average rather than example-specific behavior.
Method
The paper evaluates small fully trained models and partially trained checkpoints as lower-cost predictors of final memorization, then analyzes compute scaling.
Results
Smaller models and partial checkpoints are ineffective predictors of final memorization when they use substantially less compute, while the paper derives an equi-compute predictor of non-memorization.
Takeaways & Limitations
Practitioners should not treat a small model’s lack of memorization as a strong guarantee that a larger model will also avoid memorizing the sequence.
Takeaways & Limitations
The experiments use the Pythia suite, leaving generalization to models with different hyperparameters or training data unresolved.
Abstract
from arXiv · showhide
Memorization, or the tendency of large language models (LLMs) to output entire sequences from their training data verbatim, is a key concern for safely deploying language models. In particular, it is vital to minimize a model's memorization of sensitive datapoints such as those containing personal identifiable information (PII). The prevalence of such undesirable memorization can pose issues for model trainers, and may even require discarding an otherwise functional model. We therefore seek to predict which sequences will be memorized before a large model's full train-time by extrapolating the memorization behavior of lower-compute trial runs. We measure memorization of the Pythia model suite and plot scaling laws for forecasting memorization, allowing us to provide equi-compute recommendations to maximize the reliability (recall) of such predictions. We additionally provide further novel discoveries on the distribution of memorization scores across models and data. We release all code and data necessary to reproduce the results in this paper at https://github.com/EleutherAI/pythia
1 Introduction
The paper asks whether specific memorization can be forecast before expensive model training, motivated by privacy risks and limited early-detection tools. It studies small-model and partial-checkpoint extrapolation and develops compute-aware forecasting recommendations.
- 1 Introduction: Specific-sequence memorization matters because sensitive data, including PII, may be verbatimly exposed, while factual memorization may also be beneficial.The paper distinguishes undesirable leakage from potentially useful retention of factual events and details.
- 1 Introduction: Existing interventions and analyses provide few tools for detecting memorization early or preventing memorization of specific examples.Prior work often addresses corpus-wide averages, while differential privacy can hurt pretraining performance.
- 1 Introduction: The paper proposes forecasting specific training-data memorization from lower-cost evaluations, using either smaller models or partially trained target-model checkpoints.These regimes can inform training behavior before the full run and may allow aborting undesirable runs with less wasted compute.
- 1 Introduction: The study examines whether scaling-law-style extrapolation can predict memorization and how precision-versus-recall needs affect the usefulness of these predictors.The paper frames prediction quality around confirming memorization versus identifying sequences that should be forgotten.
- 1 Introduction: The paper reports that smaller models and partial checkpoints are unreliable predictors unless a sizable fraction of target pretraining compute is used, and derives equi-compute recommendations.Its contribution includes preliminary scaling-law analysis for maximizing forecast reliability under a compute budget.
2 Methodology
The methodology operationalizes memorization through extractibility and token-level matching, then evaluates low-cost predictors against fully trained models under a specified threat model. Experiments use publicly reproducible Pythia models and selected checkpoints because the computational cost is substantial.
- 2 Methodology: Table 1 illustrates memorization-score calculations for different prompts, with the final example demonstrating a 4-extractible string.The examples are illustrative and do not come from the actual training data.
- 2 Methodology: Memorization is formalized through k-extractibility: a string must occur in training data and be generated after prompting with k prior tokens.This framework follows Carlini et al. and supplies the operational basis for identifying memorized strings.
- 2 Methodology: The memorization score counts ordered token matches between the model’s greedy continuation and the dataset’s true continuation; a score of 1 denotes a memorized or extractable sequence.The evaluation uses 32-token prompts and 32-token continuations within 64 inspected tokens.
- 2 Methodology: Full-data forward evaluation is expensive, costing about one third of a full gradient update and making repeated checkpoint evaluation computationally demanding.The paper therefore limits the experimental checkpoint schedule rather than evaluating every available checkpoint.
- 2 Methodology: The threat model assumes an engineer wants to identify an undesirable subset likely to be memorized by a billion-parameter model while spending relatively little compute before full training.The assumed budget supports substantial preliminary testing before the final training run.
- 2 Methodology: A smaller model or earlier checkpoint acts as the prediction, while the larger or fully trained model supplies the ground-truth memorization label.This framing enables precision and recall calculations, with false negatives especially problematic when avoiding undesirable memorization.
- 2 Methodology: Pythia is selected because its models share publicly available training-data order, whereas alternative public suites have data or checkpoint limitations for this analysis.The experiments evaluate seven checkpoints spaced throughout training because evaluating all checkpoints is infeasible.
3 Memorization Across Scales
The study asks whether fully trained smaller models can predict which sequences a much larger model will memorize. Correlations and recall decline across model-size gaps, making low-compute forecasts unreliable for ruling out memorization.
- Smaller fully trained models were evaluated as predictors of sequences memorized by larger models, using sequences with memorization score 1.The analysis compares pairwise correlations between model sizes and evaluates precision and recall against the larger model’s memorized sequences.
- Correlation between memorized sequences drops sharply as the gap between smaller models and the 12B model increases.The resulting low correlations reduce the predictive power of smaller models.
- 95.6% of sequences memorized by the 70M model were also memorized by the 12B model, but they represented only 19.7% of the 12B model’s memorized sequences.
- 0.554 recall was achieved by the 1.4B model when forecasting a model an order of magnitude larger.
- Smaller model runs are not recommended for forecasting larger-model memorization because their low recall cannot strongly guarantee non-memorization.The authors instead seek a setup offering stronger assurances and a better compute tradeoff.
4 Memorization Within Training
The study tests whether intermediate checkpoints can forecast memorization in a fully trained model and whether moving sensitive strings early in training provides an early warning. Early checkpoints and other substantially lower-compute predictors have low recall, so the approach does not reliably guarantee non-memorization.
- The study asks whether partially completing a training run can provide a reliable signal for memorization and enable early abortion of undesirable runs.
- Moving potentially sensitive strings early in training was motivated by prior evidence that training-data location does not affect whether a sequence is memorized, but the results remained largely negative.
- Memorization correlations between checkpoints are measured on the first 23 million sequences seen by the least-trained checkpoint.Figure 3 presents these correlations as a heat map across checkpoints of the same model.
- 0.500, 0.575, 0.641, and 0.711 recall correspond to checkpoints at 23M, 44M, 65M, and 85M sequences, respectively.
- The 23M checkpoint of Pythia-12B underperformed the fully trained Pythia-6.9B in recall, illustrating that earliest checkpoints lack desirable recall.
- Intermediate checkpoints retain high precision but low recall when they use significantly less compute than the final model.High recall appears only after significant compute has been expended, particularly at later checkpoints.
5 Scaling Laws
The paper examines how low-compute models and checkpoints forecast memorization in a fully trained 12B model, finding unusual, nonstandard scaling behavior. Recall-based recommendations depend strongly on compute regime and model size, while larger-scale behavior can become emergent and difficult to extrapolate.
- Forecasting Memorization: Low-compute models and training steps are evaluated for predicting which sequences the fully trained 12B model will memorize.This scaling analysis contrasts with comparisons among partially trained models in an earlier section.
- Unusual Scaling: Precision and recall do not follow the linear log or log-log scaling patterns commonly reported in prior scaling-law work.Precision is especially anomalous, with the partially trained 12B model far outside the behavior of smaller models; recall is less anomalous but still inconsistent with conventional expectations.
- Recall Scaling: In low-compute regimes, larger models consistently outperform smaller models for predicting memorization at a fixed compute budget.These regimes are especially relevant when engineers seek inexpensive forecasts before training the target model.
- Emergent Memorization: The results provide evidence of emergent or semi-emergent memorization as model scale increases.Here, emergent memorization means large-model memorization that smaller models cannot predict, creating challenges for traditional extrapolation and motivating study beyond 12B parameters.
- Recommendations: For high-recall forecasting, the scaling law specifies which model size to train under a given compute budget.The practical goal is to use low-cost test models at least an order of magnitude smaller than the target model.
6 Corrections
The paper corrects an earlier Table 2 result after identifying an analysis-code error and revises the conclusions and discussion accordingly.
- Corrections: An analysis-code error caused an earlier draft to report substantially higher recall in Table 2.The current draft contains corrected values and adjusted conclusions and discussion.
7 Limitations and Future Work
The paper identifies limitations concerning the memorization metric, generalization beyond Pythia, and the omission of training-data content. It presents these constraints as directions for future research and broader reproducibility.
- Scope: The work is an initial step toward predicting which data a large language model will memorize before training, with several limitations and future-work opportunities.The stated scope motivates further development rather than a complete solution to pretraining memorization prediction.
- Measurement: The authors question whether their operational definition of memorization is the best metric for studying memorization dynamics.They consider k-extractible well-grounded in privacy concerns, while memorization score may be more natural for training dynamics.
- Generalization: The experiments use the Pythia suite, leaving open whether results generalize to models trained with different hyperparameters or data.Replication on deduplicated Pythia models found the same trends, but no other suitable model suite was available for this analysis.
- Data Content: The approach does not account for the actual content of training data when predicting memorization.It predicts from whether cheaper models memorize the content; future work could examine text properties associated with memorization.
8 Conclusion
The paper introduces forecasting specific-sequence memorization before full training and evaluates lower-cost predictors and scaling laws. It finds that smaller models and partial checkpoints are ineffective predictors after cost adjustment, while offering equi-compute recommendations for forecast reliability.
- Smaller models and partially trained checkpoints are not viable predictors of final memorization behavior when adjusted for cost.
- The paper derives a scaling law for selecting an optimal equi-compute predictor of non-memorization.
- The analyses provide recommendations for maximizing prediction reliability under a fixed compute budget.
A Robustness to Thresholding Choices
The robustness analyses examine whether thresholding choices affect memorization-score distributions and prediction results. A large spike at score = 1 indicates a thick-tailed distribution, while doubled-token thresholds reproduce the main findings.
- The score = 1 subset treats every sequence exceeding the 32-token threshold as equally memorized, despite variation in continuation length.
- Score = 1 produces a pronounced spike in the 12B memorization-score distribution, inconsistent with an exponential distribution and characteristic of thick-tailed distributions.
- A doubled matching-token threshold yields roughly the same precision-and-recall results as the original analysis.
- The scaling-law plots also produce the same results when rerun with longer sequences.
B Robustness to Deduplication
Experiments on deduplicated Pythia models test whether the memorization findings persist after removing duplicate training data. Memorization decreases slightly, while the reported correlation and prediction trends remain consistent.
- 1.62% of sequences are memorized by Pythia-12B, and deduplicated models memorize less of their datasets than non-deduplicated counterparts.
- Deduplicated models show the same heatmap-correlation conclusions as the regular Pythia suite.
- Precision-and-recall trends for deduplicated models match the corresponding regular-model analyses.
- The authors consider the results reasonably robust across hyperparameters and engineering train-time choices, while noting that distinct corpora remain for future replication.
C Additional Figures
Additional figures visualize memorization correlations across checkpoints and support the paper’s broader analysis. The supplied passages also document the contributors’ roles in conducting, analyzing, and writing the work.
- Additional figures: Heat maps visualize correlations between which sequences are memorized by different checkpoints.
- Contributions: The project included evaluation of memorization of pretraining strings and a robustness evaluation.
- Contributions: Contributors analyzed precision-and-recall results, plotted data, optimized implementation, and interpreted results.