Source-linked AI summary
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, Armen Aghajanyan
TL;DR
The paper addresses limited understanding of how large language models memorize and forget training data during training. It empirically measures exact memorization across causal and masked models, sizes, datasets, learning rates, and training conditions, finding that scale speeds memorization and reduces forgetting while nouns and numbers are memorized first. These results support treating memorization dynamics as an important metric for studying model scaling.
Problem
The training-time memorization and forgetting dynamics of very large language models, especially their dependence on scale, remain insufficiently understood.
Method
The paper empirically studies exact memorization in causal and masked language models across model sizes and training conditions, using a self-supervised memorization metric and controlled experiments.
Results
Larger models memorize training data faster, reach higher forgetting baselines, and memorize nouns and numbers faster than other parts of speech.
Takeaways & Limitations
Memorization dynamics should be analyzed alongside cross-entropy when studying how language-model behavior changes with scale.
Takeaways & Limitations
The study does not exhaustively search all factors or fully explain why larger models memorize faster.
Abstract
from arXiv · showhide
Despite their wide adoption, the underlying training and memorization dynamics of very large language models is not well understood. We empirically study exact memorization in causal and masked language modeling, across model sizes and throughout the training process. We measure the effects of dataset size, learning rate, and model size on memorization, finding that larger language models memorize training data faster across all settings. Surprisingly, we show that larger models can memorize a larger portion of the data before over-fitting and tend to forget less throughout the training process. We also analyze the memorization dynamics of different parts of speech and find that models memorize nouns and numbers first; we hypothesize and provide empirical evidence that nouns and numbers act as a unique identifier for memorizing individual training examples. Together, these findings present another piece of the broader puzzle of trying to understand what actually improves as models get bigger.
1 Introduction
The paper studies how language models memorize and forget training data during training, with particular attention to model scale. It finds that larger models memorize faster, forget less, and reveal distinctive memorization patterns for nouns and numbers.
- Memorization dynamics provide evidence about how models may generalize to new test instances.
- Prior work examined memorization mainly after training, leaving its evolution with model scale during training relatively understudied.
- Larger language models memorize training data faster across model sizes, dataset sizes, overfitting conditions, and learning rates.
- Forgetting curves approach a lower bound called the forgetting baseline, and this baseline increases with model scale.
- Nouns and numbers are memorized faster than other parts of speech, possibly because they uniquely identify individual training examples.
- The findings contribute to understanding the distinctive training dynamics that emerge as language models grow larger.
2 Background and Related Work
The background connects language-model memorization to generalization, privacy, training dynamics, forgetting, and scaling laws. Existing work covers these topics, while this paper focuses on memorization and forgetting during language-model training.
- Memorization in Language Models: Memorization can support generalization and factual knowledge, but unintended memorization also enables extraction and membership-inference attacks.
- Language Model Training Dynamics: Training-dynamics research has examined how models acquire representations, linguistic structure, world knowledge, individual words, and cross-lingual structure over pre-training.
- Forgetting in Language Models: Forgetting research studies memory degradation, catastrophic interference, robustness mechanisms, continual learning, machine unlearning, and the effects of model and dataset scale.
- Scaling Laws: Scaling-law research empirically and theoretically models how language-model behavior changes with model size, data, transfer, routing, and generative tasks.
3 Experimental Setup
The experimental setup develops a cheap but precise exact-memorization metric for self-supervised language modeling and measures it across architectures, datasets, and training updates.
- The metric adapts label memorization to self-supervised settings while remaining computationally inexpensive and sufficiently precise.
- A context is exactly memorized when the language model’s highest-scoring token matches the ground-truth token.
- The memorization proportion is computed over contexts, even though the same ground-truth word may occur in multiple contexts.
- The training-time threshold T(N, τ) measures the minimum number of times a model with N parameters must see each datapoint to reach memorization threshold τ.
- Experiments use causal and masked language models with configurations ranging from 125M to 13B parameters.
- The study uses WIKITEXT-103 and the RoBERTa corpus, with WIKITEXT-103 used primarily outside Section 4 for computational reasons.
4 Larger Language Models Memorize Faster
Across causal and masked language modeling, larger models generally memorize training data faster, though masked modeling shows a threshold-dependent reversal early in training. This speed advantage persists across dataset size and fixed learning-rate settings, and larger models memorize more before overfitting.
- Larger models require fewer training examples to reach a fixed memorization threshold in causal language modeling.For τ = 0.9, T(N, 0.9) decreases monotonically with model size; this trend generally holds for τ between 0.4 and 0.95.
- Dependence on Language Modeling Task and Dataset Size: In masked language modeling, smaller models memorize faster initially, whereas larger models memorize higher proportions faster in the long run.The transition between these regimes occurs between τ = 0.6 and τ = 0.7.
- Dependence on Language Modeling Task and Dataset Size: Larger models memorize faster across causal and masked language modeling on the ROBERTA dataset when measured by gradient-descent updates.Figure 3 reports Tupdate(N, τ), the number of updates required to reach a memorization threshold, across both tasks.
- Why Do Larger Models Memorize Faster?: Larger models memorize a greater proportion of training data before overfitting, so overfitting alone does not explain their memorization dynamics.Overfitting is defined as the first epoch when validation perplexity increases.
- Why Do Larger Models Memorize Faster?: At a fixed learning rate, larger models still reach 0.9 memorization faster, and their memorization becomes less sensitive to learning-rate changes as size increases.The memorization-time curve initially decreases and eventually rises as learning rate increases.
- Memorization via. Unique Identifiers: Adding unique identifiers to examples accelerates memorization, supporting the study of identifier-like features as a mechanism for memorizing individual training samples.The experiment uses causal language modeling with a 125M model on WIKITEXT103; identifiers are added to the dictionary or prepended to examples.
5 Forgetting Curves in Language Models
The paper studies how language models naturally forget data after one-time exposure, finding that forgetting approaches a scale-dependent baseline. Larger models forget less, while batch ordering has little effect and repetition can alter retention.
- Experimental setup: The forgetting experiment introduces a validation batch once, then resumes training on the original training set to measure memory degradation.The validation batch is called the special batch and is only seen immediately upon introduction.
- Forgetting dynamics: Exact memorization initially declines quickly on the special batch before slowing and approaching a forgetting baseline.The baseline is approximated by the lowest memorization value observed throughout training.
- Scale effects: The forgetting baseline increases monotonically with model scale, implying that larger models forget less.The paper connects this retention benefit to leveraging information from previous tasks, while noting a privacy concern about retaining sensitive information.
- Experimental controls: Changing the checkpoint at which the special batch is introduced tests different batch orders without substantially changing the experimental setup.The analysis begins at the 14th, 39th, or 63rd epoch; the reported baseline differences are on the order of 10^-3.
- Repetition: Repeated injection raises the forgetting baseline, whereas spaced repetition has minimal effect.These comparisons use causal language modeling with a 125M model on WIKITEXT103.
6 Conclusions and Discussion
The discussion frames memorization dynamics as an important dimension of language-model scaling. It emphasizes that memorization and cross-entropy capture different behaviors, while noting privacy-related scope and implications.
- Implications: The paper argues that memorization should be analyzed when scaling language models, rather than reporting cross-entropy alone.The authors present memorization dynamics as a critical metric for understanding training behavior as models grow.
- Implications: The findings contribute to understanding the distinctive training dynamics that emerge as models increase in size.The discussion presents the results as part of a broader effort to understand what changes with scale.
- Privacy scope: The paper focuses on information that would be sensitive if output verbatim, such as phone numbers, SSNs, addresses, and medical diagnoses.It notes that empirical privacy in language modeling is not well-defined for all forms of memorization.
- Implications: Memorization can approach a baseline even while perplexity continues increasing, indicating that the two measures capture different behavior.The paper interprets this as the model becoming less confident about exact predictions despite retained memorization.
A.1 Full Memorization Dynamics Over Training
Across causal and masked language modeling, larger models generally memorize training data faster over training. The appendix also describes update-level tracking for larger datasets and validates it against epoch-level memorization.
- Memorization over training: Larger models generally memorize training data faster across language-modeling tasks and threshold values.A caveat is that smaller models can memorize faster initially before larger models surpass them.
- WIKITEXT103: Figure 11 plots the proportion of memorized training data M(f) against training epochs for causal and masked modeling on WIKITEXT103.The x-axis is epochs and the y-axis is M(f).
- Update-level tracking: For larger datasets, memorization is tracked at each gradient descent update because multiple epochs and full-dataset M(f) evaluation are infeasible.Mupdate(f, U) measures memorization on the batch presented at update U, and curves are smoothed with a rolling window of five.
- ROBERTA: Figure 12 uses gradient-descent updates on the ROBERTA dataset and again shows larger models memorizing faster.The plotted quantity is a five-update rolling average of Mupdate(f).
- Proxy validation: Mupdate(f, U) closely tracks M(f) over training for the tested 13B and 125M causal language models.The update-level measure shows periodic behavior but does not deviate substantially from epoch-level memorization.
A.1.1 Limitations of Definition 1
The paper cautions that Definition 1 is an imperfect memorization measure because it ignores model confidence and does not normalize for duplicated training data.
- Metric limitations: Definition 1 ignores model confidence and does not normalize for duplication in the training set.Because duplication helps models memorize tokens, the metric may not capture all factors affecting memorization.
- Metric limitations: The paper nevertheless studies Definition 1 because previous memorization definitions appear to involve it in some form.It is treated as a metric fundamental to memorization despite its limitations.
A.2.1 Perplexity Versus Memorization
Perplexity and exact memorization can diverge during training: perplexity continues increasing while memorization of a special batch flattens. Additional analyses show memorization changes diminish over epochs, while average memory unit length increases.
- Perplexity continues increasing while special-batch memorization flatlines, showing that cross-entropy loss and memorization can capture different training behavior.The result is reported for causal language modeling on WIKITEXT103 with a 1.3B model, and similar trends appear across the forgetting experiments.
- Sequential differences in special-batch memorization exponentially approach 0 across training.For diff(T) = M(f)_T − M(f)_(T−1), this supports the existence of a forgetting baseline.
- Average memory unit length steadily increases over training and has an approximately sigmoidal trajectory across model sizes.Memory unit length counts memorized individual tokens, bigrams, trigrams, and longer n-grams; the analysis uses WIKITEXT103 sequences shorter than 512 tokens.