Source-linked AI summary
Data Contamination: From Memorization to Exploitation
Inbal Magar, Roy Schwartz
TL;DR
The paper asks how much contaminated pretraining data affects downstream task performance, a question made difficult by massive web-scale corpora. It introduces controlled comparisons of seen and unseen examples to measure memorization and exploitation, finding that labels can be memorized and exploited, but the two measures are not necessarily linked. The study is conducted at limited scale and may not generalize to larger models trained on more data.
Problem
It remains unclear to what extent downstream test-set contamination affects task performance, despite contamination in large pretraining corpora.
Method
The paper pretrains BERT on general text plus labeled downstream examples, fine-tunes on the task, and compares seen versus unseen instances to measure memorization and exploitation.
Results
Exploitation exists in some cases, while memorization does not guarantee exploitation; contamination position can produce exploitation up to 17%.
Takeaways & Limitations
Labels seen during pretraining can be retrieved and exploited in downstream tasks, highlighting the importance of quantifying and mitigating data contamination.
Takeaways & Limitations
Because the experiments use medium-sized models and corpora up to 600M tokens, some results may not generalize to larger models trained on more data.
Abstract
from arXiv · showhide
Pretrained language models are typically trained on massive web-based datasets, which are often "contaminated" with downstream test sets. It is not clear to what extent models exploit the contaminated data for downstream tasks. We present a principled method to study this question. We pretrain BERT models on joint corpora of Wikipedia and labeled downstream datasets, and fine-tune them on the relevant task. Comparing performance between samples seen and unseen during pretraining enables us to define and quantify levels of memorization and exploitation. Experiments with two models and three downstream tasks show that exploitation exists in some cases, but in others the models memorize the contaminated data, but do not exploit it. We show that these two measures are affected by different factors such as the number of duplications of the contaminated data and the model size. Our results highlight the importance of analyzing massive web-scale datasets to verify that progress in NLP is obtained by better language understanding and not better data exploitation.
1 Introduction
The paper examines how contaminated downstream data affects task performance and introduces a controlled method to distinguish memorization from exploitation. Experiments show exploitation can occur, but memorization does not necessarily lead to exploitation.
- Motivation: Data contamination occurs when downstream test sets enter pretraining corpora, but its effect on downstream performance remains unclear.The paper notes that contamination is widespread and difficult to eliminate or study at web scale.
- Method: The method pretrains BERT on Wikipedia plus labeled training and test examples, then fine-tunes it on the same task.Test examples included during pretraining are designated seen, while comparison examples remain unseen.
- Method: Exploitation is quantified as the performance difference between seen and unseen instances after fine-tuning.This comparison isolates gains associated with labels observed during pretraining.
- Method: Memorization is measured by comparing the pretrained model’s masked-label prediction performance on seen and unseen examples.The study examines how memorization relates to downstream exploitation.
- Findings: Experiments with BERT-base and BERT-large across three classification and NLI datasets show exploitation exists and depends on contamination conditions.The reported factors include contamination frequency, model size, and Wikipedia quantity.
- Findings: Memorization does not guarantee exploitation, while contamination position and learning rate affect the two measures differently.The authors present the methodology as a controlled study of exploitation in contaminated data.
2 Our Method: Assessing the Effect of Contamination on Task Performance
The paper constructs controlled contamination experiments by inserting labeled task data into pretraining, then compares seen and unseen examples after fine-tuning. It defines complementary measures for explicit label memorization and downstream exploitation, while noting important scope assumptions and scale limitations.
- Assumptions: The study assumes downstream test instances and labeled training data appear in pretraining with their gold labels.The authors acknowledge that these assumptions may not always hold, such as when data appears unlabeled.
- Experimental design: The pretraining corpus combines a general corpus with labeled task data, splitting test examples into seen and unseen subsets.The two subsets are drawn from the same distribution to support controlled performance comparisons.
- Experimental design: Fine-tuning on the shared labeled training set makes performance differences between seen and unseen test sets a measure of exploitation.The comparison is intended to isolate use of labeled examples observed during pretraining.
- Measures: mem measures explicit memorization as the pretrained MLM’s accuracy difference when predicting masked gold labels for seen versus unseen examples.The label is selected from the candidate label set given the instance text.
- Measures: expl measures exploitation as the difference in task performance between seen and unseen examples.mem is measured after pretraining, whereas expl is measured after fine-tuning.
- Measures: The two measures are complementary and are not necessarily tied.The study asks whether memorization leads to exploitation and whether exploitation requires memorization.
- Scope: The experiments use medium-sized BERT models and corpora of up to 600M tokens, so some findings may not generalize to larger models trained on more data.The authors frame this setting as a laboratory-scale study and encourage larger-scale applications.
3 Which Factors Affect Exploitation?
Exploitation is shaped by contamination frequency, masking, corpus and model size, contamination timing, learning rate, batch size, and fine-tuning seed. Memorization and exploitation can diverge: memorization may be high without downstream exploitation.
- Contamination frequency and masking: Both mem and expl increase with contaminated-data duplicates, reaching 60% mem and almost 40% expl at 200 occurrences.Higher label-masking probability also increases both measures.
- Memorization versus exploitation: Low mem values of 10% or less produce no expl, but 15% mem can correspond to less than 1% expl.Thus, memorization alone is not sufficient for exploitation.
- Model and corpus sizes: BERT-large generally has higher mem than BERT-base, while exploitation depends on model size and the amount of clean data.For BERT-base, expl starts at 2–6% with little clean data and decreases; for BERT-large, it begins at 300M tokens and grows to 2–4%.
- Contamination timing and learning rate: Early contamination produces up to 17% expl, which decreases when contamination is delayed, whereas the highest mem occurs with middle-stage contamination.Under linear learning-rate decay, late contamination yields no expl despite relatively high mem.
- Contamination timing and learning rate: With a constant learning rate, both mem and expl increase for contamination in the final training stage, indicating effects from both learning rate and timing.The constant learning rate used in the comparison is 2.77e-5.
- Batch size and fine-tuning: Decreasing pretraining batch size increases both measures; at batch size 2, mem reaches 49% and expl reaches 14%.Small batches may give each instance greater influence on gradient updates.
- Batch size and fine-tuning: Fine-tuning random seeds create a tradeoff between generalization and exploitation, with high-exploitation seeds often among the worst for generalization.Two of the three seeds with highest expl are also among the three worst for generalization.
4 Related Work
Prior work extensively studies memorization, but provides less evidence about how contaminated data is exploited in downstream tasks. This paper addresses that gap with a controlled analysis distinct from earlier post-hoc studies.
- Research on language-model memorization is extensive, whereas data contamination and downstream exploitation remain less studied.
- Brown et al. found large performance differences between clean and contaminated datasets in some cases, but negligible differences in others.
- Earlier GPT-3 analyses could not use controlled experiments because training those models was prohibitively expensive.
- The paper presents the first controlled study of exploitation levels, according to the authors.
5 Discussion and Conclusion
The paper introduces a method to quantify label memorization during pretraining and exploitation during fine-tuning. Its results show that memorized labels can be retrieved under some conditions, motivating further contamination measurement and mitigation.
- The method quantifies explicit label memorization during pretraining and exploitation during downstream fine-tuning.
- Masked-language-model prompt methods may latch onto memorized labels and amplify contamination effects.
- The authors emphasize continued development of methods for studying and mitigating contamination in large-scale datasets.
A Two Notions of “Occurences”
The paper distinguishes how often contaminated instances occur in the corpus from how often the model encounters them during training. Experiments show that training exposure strongly affects exploitation.
- Corpus occurrence counts instances present in the dataset, whereas training exposure also includes the number of epochs.
- Fixing ten corpus copies while varying training epochs tests whether model encounters, rather than corpus duplicates, affect outcomes.
- Exploitation was similar after 50 encounters but 5% lower after 10 encounters, showing that training exposure strongly influences expl.
B Same Ratio, Different expl
The paper tests whether increasing corpus size changes exploitation when the contamination-to-clean-data ratio remains fixed. This isolates absolute corpus size from relative contamination prevalence.
- The experiment keeps the contamination-to-Wikipedia instance ratio fixed at 1:10 while increasing both datasets’ absolute sizes.
C Position of Contamination Matters
The study examines whether the position of contaminated data during pretraining affects memorization and exploitation, using BERT-base and results on SST-2 and SNLI.
- C Position of Contamination Matters: BERT-base is pretrained with contaminated data inserted at different stages of training.The experiment reports results for SST-2 and SNLI.
- C Position of Contamination Matters: Figure 6 compares mem and expl under linear learning-rate decay and constant learning rate.
D Experimental Details
The experiments use masked-language-model pretraining of uncased BERT-base and BERT-large on Wikipedia-based corpora, followed by standardized downstream fine-tuning and evaluation.
- D Experimental Details: Both uncased BERT-base and BERT-large are pretrained using masked language modeling only.The setup omits next-sentence prediction because prior work found its removal does not substantially affect downstream performance.
- D Experimental Details: The corpus-size experiment keeps a 1:10 ratio between contaminated data and the total corpus by increasing SST-5 copies and Wikipedia size.
- D Experimental Details: Wikipedia text is divided into newline-separated lines to measure contamination shuffled across the pretraining corpus.
- D Experimental Details: Models are fine-tuned on 1,000 samples from SST-2, SST-5, and SNLI, with results averaged over ten random trials.Fine-tuning uses three epochs, batch size 8, AdamW, and a 2e-5 learning rate.
- D Experimental Details: Unseen-test accuracy is used as the baseline evaluation reported in Table 3.
- D Experimental Details: For stage-specific contamination, the corpus is split into equal first, middle, and last sections, with all contaminated data confined to one section.Each section is shuffled individually without a random sampler.