Source-linked AI summary
Scaling Laws and Interpretability of Learning from Repeated Data
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, Sam McCandlish
TL;DR
Repeated data is common in language-model training, but its effects and mechanisms are not well understood. The paper trains models with mostly unique data and controlled repeated subsets, finding intermediate repetition frequencies that sharply degrade performance and disproportionately damage copying and induction heads. These results support a link between repetition-related memorization and impaired generalization, while the study leaves several regularization strategies and simpler mechanistic systems unexplored.
Problem
Data repetition can substantially impair language-model performance, but why it has such large effects is not well understood.
Method
The paper trains transformer language models on mostly unique data plus controlled repeated subsets, varying repeated-data size, model size, and repeated-token fraction, then analyzes scaling laws and mechanisms.
Results
Repeated data causes strong double descent and intermediate-frequency performance degradation, including disproportionate losses in copying and induction-head-related prefix matching.
Takeaways & Limitations
The results provide a tool for predicting and diagnosing repetition-related problems and connect scaling-law effects with a shift from generalization toward memorization.
Takeaways & Limitations
The study does not investigate simpler systems than one-layer attention-only models, which might provide more complete mechanistic insights.
Abstract
from arXiv · showhide
Recent large language models have been trained on vast datasets, but also often on repeated data, either intentionally for the purpose of upweighting higher quality data, or unintentionally because data deduplication is not perfect and the model is exposed to repeated data at the sentence, paragraph, or document level. Some works have reported substantial negative performance effects of this repeated data. In this paper we attempt to study repeated data systematically and to understand its effects mechanistically. To do this, we train a family of models where most of the data is unique but a small fraction of it is repeated many times. We find a strong double descent phenomenon, in which repeated data can lead test loss to increase midway through training. A predictable range of repetition frequency leads to surprisingly severe degradation in performance. For instance, performance of an 800M parameter model can be degraded to that of a 2x smaller model (400M params) by repeating 0.1% of the data 100 times, despite the other 90% of the training tokens remaining unique. We suspect there is a range in the middle where the data can be memorized and doing so consumes a large fraction of the model's capacity, and this may be where the peak of degradation occurs. Finally, we connect these observations to recent mechanistic interpretability work - attempting to reverse engineer the detailed computations performed by the model - by showing that data repetition disproportionately damages copying and internal structures associated with generalization, such as induction heads, providing a possible mechanism for the shift from generalization to memorization. Taken together, these results provide a hypothesis for why repeating a relatively small fraction of data in large language models could lead to disproportionately large harms to performance.
1 Introduction
The paper studies how repeated data affects language-model scaling and internal mechanisms, finding that repetition can sharply harm performance and damage copying-related structures. It links these effects to a shift from generalization toward memorization.
- 1 Introduction: Repeated data produces a strong double-descent pattern, with the largest performance damage at an intermediate repetition frequency.A few repeats or very many repeats cause less damage than the middle range.
- 1 Introduction: 800M models can approach 340M-model loss when 10% of training tokens come from repeated data.The degradation peaks when repeated examples can be memorized while consuming substantial model capacity.
- 1 Introduction: Repeated data can diverge from power-law scaling, with the affected region depending on model size and training regime.Extrapolation predicts meaningful degradation from repeating data only 2 times for GPT-3-sized models, though compute-optimal training may shift the region.
- 1 Introduction: Copying is disproportionately harmed: 3% repeated data can reduce effective model size by up to 3x on copying versus at most 15% on test loss.The copying evaluation used repeated Harry Potter text and measured whether the model could reproduce it.
- 1 Introduction: Repeated data disproportionately damages induction heads, with a 32% effective-model-size reduction on prefix matching versus at most 15% on test loss.The paper connects this damage to copying and other mechanisms associated with generalization.
- 1 Introduction: The study also finds degradation across Python-code evaluation, exact and fuzzy copying, pretraining, and fine-tuning settings.Repeated Python code shows a related double-descent pattern, while pretraining with repeated data performs worse than training from scratch or fine-tuning from unique-data pretraining.
2 Results
Repeated data produces a model-size- and repetition-dependent double descent, with a middle regime where performance degradation is largest. The damage is disproportionately concentrated in copying, prefix matching, in-context learning, and related internal structures, while out-of-distribution effects are smaller and repetition effects extend to Python data.
- Double descent and scaling: 100x repeats of 0.1% of the data reduced an 800M model's performance nearly to that of a 340M model, despite only 10% repeated data.The peak coincided with memorization of the repeated subset.
- Double descent and scaling: 1,220 repeats with 10% repeated data reduced performance to 0.55x model-size equivalent at 10M–100M parameters, recovering to 0.8x at 1B parameters.With 122 repeats, the dip occurred later, centered around 1B parameters.
- Double descent and scaling: Repeating data as little as 2x was predicted to cause significant degradation in models with hundreds of billions of parameters, assuming 100B total training tokens.The authors caution that training beyond the double-descent peak helps and shifts the poor-performance region on the compute-efficient frontier.
- Copying and generalization: 3% repeated data caused up to a 3x effective model-size reduction on copying, compared with at most a 1.15x reduction on test loss.Copying was preferentially harmed across repeated-data fractions from 3% to 20%.
- Copying and generalization: At the double-descent peak, copying loss stopped decreasing with additional paragraph copies, while in-context learning also worsened but less severely.The copying evaluation measured loss on consecutive copies of a Harry Potter paragraph.
- Mechanistic effects: At 3% repeated tokens, prefix matching showed a 2x effective parameter decrease, compared with a 1.15x decrease in overall performance.Prefix matching was preferentially degraded at low repeated-data fractions, though not across the entire range.
- Mechanistic effects: A 1.5M-parameter model with 50% repetition had overall performance equivalent to 30,000 parameters and prefix matching equivalent to 2,000 parameters.The sharp prefix-matching drop corresponded to complete breakdown of paragraph-level copying.
- Mechanistic effects: Repeated-data models were worse at skip-trigram copying and fuzzy copying of recurring proper names in one- and two-layer attention-only models.For fuzzy copying, the control model sometimes assigned 500x higher odds to the correct token.
3 Methods
The study trains decoder-only transformer language models on large text and Python datasets, varying repetition and model settings to examine repeated data during pre-training and fine-tuning.
- Pre-training: 100B tokens were used to train the language models across the experimental sweep.The language dataset included filtered Common Crawl, internet books, OpenWebText, Wikipedia, and Stack Exchange data.
- Fine-tuning: Fine-tuning experiments compared models pre-trained on repeated or unique data using always-unique Python data.The repetition frequency was selected to maximize the performance hit.
- Code experiments: Python models were trained or fine-tuned on 45B tokens for 2.2 epochs.Fine-tuning used the same hyperparameters as pre-training, with halved learning rates and reduced warmups.
- Experimental factors: Model size, repeated dataset size, and repeated-token fraction were varied across orders of magnitude.The reported variation spans 3, 2.5, and 2 orders of magnitude, respectively.
4 Related Work
The paper situates its study at the intersection of scaling laws, mechanistic interpretability, double descent, and the engineering of large language datasets.
- Scaling Laws: Scaling laws relate a small set of hyperparameters to predictable changes in model performance.The paper emphasizes their usefulness in research and engineering settings.
- Mechanistic Interpretability: Mechanistic interpretability attempts to reverse engineer the detailed computations performed by neural networks.The paper presents this lens as useful for scientific understanding and potentially for anticipating safety issues.
- Double Descent: Double descent has been observed across decision trees, random features, and neural networks.The paper extends this established phenomenon to repeated data in a sub-distribution.
- Language Datasets: Large language model progress depends on algorithmic innovation, compute, and large, diverse datasets.The paper places repeated-data concerns within the recent development of large-scale language-data engineering.
5 Discussion
The discussion argues that repeated data can disproportionately damage model performance by shifting capacity toward memorization and away from generalization-related computations. It also identifies important experimental boundaries and directions for further study.
- 5.1 Why does repeating a small fraction of data damage performance so much?: 10% repeated tokens reduced performance by an effective 2x in parameter count, exceeding the effect of simply omitting those tokens.The proposed explanation is that driving loss toward zero on a small repeated subset can make memorization appear worthwhile despite degrading the remaining data.
- 5.1 Why does repeating a small fraction of data damage performance so much?: A middle range of repetition can produce the largest degradation because memorizing the repeated data consumes a large fraction of model capacity.Few repetitions may not enable memorization, while extreme repetition of a tiny subset may require little capacity.
- 5.2 Memorization and generalization: Overfitting repeated data coincided with disproportionate degradation in induction-head prefix matching and text copying.The discussion interprets copying as a form of content-independent generalization and describes memorization as disrupting its internal implementation.
- 5.3 Bridging mechanistic interpretability and scaling laws: The connection between memorization, interpretability structures, and loss is presented as a bridge between microscopic network mechanisms and macroscopic performance trends.The authors argue that such bridges can support deeper understanding and more robust diagnostics.
- 5.4 Repeated data and fine-tuning: Fine-tuning degradation required extreme pre-training repetition in the reported setup and provided medium causal evidence for the ossification hypothesis.At the peak, 90% repeated training produced a 73x effective pre-training model-size reduction and a 1.6x reduction after fine-tuning.
- 5.5 Limitations: The experiments used a fixed token count for all models rather than the compute-optimal training frontier.This choice preceded the release of Chinchilla, which revised understanding of the compute frontier.
- 5.5 Limitations: The poor-performance-region fits were noisy and showed a clean trend only after aggregation.The authors identify limited averaging and insufficient scan resolution as possible contributors.
- 5.5 Limitations: The repeated data was a random subset, so the results do not directly represent intentionally repeating higher-quality data.The authors consider transfer to that setting plausible but untested.
6 Conclusion
Small fractions of repeated data can severely degrade model performance at the right repetition frequency, with effects that scale predictably across datasets and damage mechanisms associated with generalization. The results connect scaling-law behavior with mechanistic interpretability and offer tools for diagnosing repetition-related problems.
- Small fractions of repeated data can cause surprisingly severe performance degradation when repeated at the right frequency.
- The degradation scales predictably, occurs across datasets, and is associated with disproportionate damage to internal mechanisms linked to generalization.
- The findings bridge macroscopic scaling laws and microscopic mechanistic interpretability, clarifying how generalization and memorization may interact.
- The results provide practical tools for predicting and diagnosing data-repetition-related problems in language models.
A Model Size Multiplier and Poor Performance Region Fits
The paper fits a poor-performance region by re-parameterizing loss with model-size multipliers and estimating repetition boundaries around double-descent peaks. The region appears robust, but its exact boundaries are noisy and setup-dependent.
- A Model Size Multiplier and Poor Performance Region Fits: Power-law fits on control scans re-parameterize loss in terms of model-size multipliers for language and Python.
- A Model Size Multiplier and Poor Performance Region Fits: 1% repeated-data scans were excluded because they were noisy, while 3% and higher fractions were included because their peaks exceeded the noise.
- A Model Size Multiplier and Poor Performance Region Fits: Half-maximum effect boundaries are estimated by linear interpolation on both sides of each double-descent peak and averaging across repeated-data fractions.
- A Model Size Multiplier and Poor Performance Region Fits: The poor-performance region is considered relatively robust, but its exact boundaries may vary meaningfully with the experimental setup, especially regularization.
- A Model Size Multiplier and Poor Performance Region Fits: Python aggregates are noisier partly because they combine only two scans, and individual Python scans are also noisier.
B Appendix: Logit Attribution Analysis, 2 Layer Models
The appendix uses logit attribution in two-layer attention-only models to examine how repeated data disrupts induction heads. The evidence suggests weak changes in induction-head strength and interference from other heads, while per-token losses are more informative.
- B Appendix: Logit Attribution Analysis, 2 Layer Models: Logit attribution is used to inspect attention-head contributions and investigate how repeated data disrupts induction heads in two-layer models.
- B Appendix: Logit Attribution Analysis, 2 Layer Models: In Figure 17, the induction head is head 0 in both models, with the repeated model examined at the double-descent peak.
- B Appendix: Logit Attribution Analysis, 2 Layer Models: The analysis considers whether induction heads fire more weakly or whether other attention heads interfere with copying.
- B Appendix: Logit Attribution Analysis, 2 Layer Models: Both weaker induction-head activity and interference from other heads appear to occur weakly in the repeated-data models.
- B Appendix: Logit Attribution Analysis, 2 Layer Models: Other attention heads appear more active in the repeated-data model, based on stronger contributions across portions of the copied paragraphs.
- B Appendix: Logit Attribution Analysis, 2 Layer Models: Double descent remains observable in one-layer attention-only models, suggesting the phenomenon may extend to simpler architectures.
C Appendix: Copying and Prefix Matching Score Fits
The appendix describes auxiliary fits used to compare model-size effects and copying behavior for control models. These fits support interpolation procedures based on loss and prefix-matching scores.
- C Appendix: Copying and Prefix Matching Score Fits: Control-model loss on Harry Potter’s first paragraph copied 11 times is used for model-size interpolation, with peak locations checked as model size increases.
- C Appendix: Copying and Prefix Matching Score Fits: A linear fit to control-model prefix-matching scores yields a relatively clean relationship when summarized using the harmonic mean of the model-size multiplier.
D Appendix: Harry Potter Copying Evaluation with Fewer Characters
A shorter copying evaluation was used to test whether the observed effect reflected in-context learning. The 2L model still showed approximately no learning from repeated copying at the double descent peak.
- The evaluation shortened the copied sequence approximately 10x, from 1463 to 125 characters, to reduce reliance on in-context learning.The shorter sequence was intended to test whether copying results depended on contextual learning.
- The test examined a 2L model trained on 50% repeated data at the double descent peak.
- The model showed approximately no learning from repeated copying even with the shorter sequence.