PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating Data Contamination for Pre-training Language Models

Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, Sanmi Koyejo

arXiv:2401.06059v1cs.CLcs.AIcs.LG

TL;DR

The pre-training effects of evaluation-data contamination remain insufficiently understood. The paper pre-trains GPT-2 models from scratch to study text and ground-truth contamination, repetition, and n-gram definitions. Ground-truth contamination had a stronger positive effect than text contamination, while repeated contamination could produce U-shaped performance trends.

  • Problem

    The effects of evaluation data entering pre-training corpora have not been adequately analyzed at the pre-training level, complicating assessment of model performance.

    Source: §1 · p. 1

  • Method

    The authors pre-train GPT-2 models from scratch while varying contamination forms and repetitions, and assess filtering based on n-gram definitions.

    Source: §1 · p. 2

  • Results

    Ground-truth contamination had a more pronounced positive effect than text contamination, while repeated contamination could yield U-shaped trends and n-gram filtering showed no uniform performance pattern.

    Source: §4.1 · p. 5, §4.2 · p. 6, §4.3 · p. 7

  • Takeaways & Limitations

    The findings indicate that contamination assessment should account for ground-truth information and that n-gram definitions are insufficient to identify contamination reliably.

    Source: §1 · p. 2, §4.3 · p. 7

    Categorical evaluation-level contamination assessments may be insufficient to conclude that models are insusceptible to contamination.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Language models pre-trained on web-scale corpora demonstrate impressive capabilities on diverse downstream tasks. However, there is increasing concern whether such capabilities might arise from evaluation datasets being included in the pre-training corpus -- a phenomenon known as \textit{data contamination} -- in a manner that artificially increases performance. There has been little understanding of how this potential contamination might influence LMs' performance on downstream tasks. In this paper, we explore the impact of data contamination at the pre-training stage by pre-training a series of GPT-2 models \textit{from scratch}. We highlight the effect of both text contamination (\textit{i.e.}\ input text of the evaluation samples) and ground-truth contamination (\textit{i.e.}\ the prompts asked on the input and the desired outputs) from evaluation data. We also investigate the effects of repeating contamination for various downstream tasks. Additionally, we examine the prevailing n-gram-based definitions of contamination within current LLM reports, pinpointing their limitations and inadequacy. Our findings offer new insights into data contamination's effects on language model capabilities and underscore the need for independent, comprehensive contamination assessments in LLM studies.

1 Introduction

The effects of evaluation-data contamination at the pre-training stage remain under-explored, and limited access to training corpora complicates assessing its impact. This paper studies several contamination forms and repetition patterns by pre-training GPT-2 models from scratch.

1 Introduction

  • The authors note that inaccessible pre-training corpora complicate comprehensive analysis of contamination's effects on model performance and behavior.They frame this as a challenge to accurately assessing model capabilities and behavior.Source: §1 · p. 1
  • Existing evaluation-level studies split datasets into contaminated and clean chunks, but do not deliberately alter pre-training corpora to measure contamination effects.Some prior reports also provide limited detail, especially for closed-source models.Source: §1 · p. 1
  • Ground-truth contamination includes evaluation input texts, prompts, and corresponding answers, and its effects may differ from text-only contamination.Prior work has primarily focused on input text, while ground-truth effects have been under-explored.Source: §1 · pp. 1–2
  • The paper pre-trains GPT-2 models from scratch to study contamination forms, repetition counts, and the effectiveness of n-gram-based definitions.Its research questions also include assessing evaluation-level contamination analyses.Source: §1 · p. 2
  • The study identifies ground-truth contamination as important and reports that performance across repeated contamination can follow a U-shaped trend.It also finds current n-gram definitions insufficient and inadequate for identifying contamination.Source: §1 · p. 2

2 Contamination Definitions

The section describes overlap-based contamination definitions used in LLM reports, notes their potential error modes, and distinguishes text contamination from ground-truth contamination.

2 Contamination Definitions

  • PaLM labels evaluation samples clean or contaminated according to whether at least 70% of their possible 8-grams appeared in pre-training.The threshold is based on whether each possible 8-gram was seen at least once.Source: §2 · p. 2
  • Llama 2 marks tokens contaminated when they occur in shared n-grams longer than 10 tokens, then bins samples by contaminated-token percentage.Its four bins are Clean, Not Clean, Not Dirty, and Dirty.Source: §2 · p. 3
  • N-gram and token overlap definitions target direct duplication and may produce false positives for semantically different text or false negatives for paraphrases.The paper focuses instead on pre-training-level analysis of contamination effects.Source: §2 · p. 3
  • The experiments distinguish text contamination, which adds evaluation inputs, from ground-truth contamination, which also adds prompts and answers.The authors note that existing definitions tend to overlook ground-truth information.Source: §2 · p. 3

3 Experimental Setup

The experiments use GPT-2 models and four public NLP datasets to compare performance across contamination conditions. Evaluation follows task-specific procedures and metrics.

3.1 Models, Data and Pre-Training

  • GPT-2-small (124M parameters) is the main model, trained on 3.3B tokens from 1.95M Pile documents; experiments extend to GPT-2-large (774M) and 19.8B tokens.The smaller model is used because scratch pre-training is computationally expensive.Source: §3.1 · p. 3

3.2 Evaluation Datasets

  • The evaluation covers SST-2, MMLU, CNN and Daily News, and SQuAD, selected to measure differential contamination effects on GPT-2 models.The authors chose traditional, comparatively easier benchmarks because more difficult datasets may be too challenging for GPT-2 models.Source: §3.2 · p. 3, Table 1 · p. 3
  • SST-2 accuracy averages results across 10 prompts, while MMLU accuracy is computed across 57 subjects using its official implementation.For SST-2, the model scores candidate-label hypotheses and predicts the highest-scoring label.Source: §3.2 · p. 4
  • Summaries are assessed with ROUGE-1, ROUGE-2, ROUGE-L, and UniEval; SQuAD answers use token-overlap F1 under the SQuAD V1 setup.The generated summary uses the first three sentences of up to 150 generated tokens; SQuAD responses use the first sentence of up to 15 tokens.Source: §3.2 · p. 4

4 Experiments & Analyses

Across the tested tasks, both text and ground-truth contamination improved performance, with ground-truth contamination generally having the stronger effect. Repeating contamination could produce U-shaped performance trends, while removing documents flagged by n-gram-based definitions did not yield a uniform pattern.

4 Experiments & Analyses

  • The experiments compare GPT-2-small models trained from scratch on original, text-contaminated, and ground-truth-contaminated corpora, alongside public GPT-2 checkpoints.Text contamination adds evaluation texts; ground-truth contamination also adds the evaluation prompt and desired output.Source: §4 · p. 4

4.1 Effects of Contamination on Evaluation Performance

  • Both contamination types improved performance, but ground-truth contamination generally had a stronger effect, especially on tasks requiring evaluation-prompt instructions.For CNN, ground-truth contamination surpassed public checkpoints and reached performance similar to GPT-2-medium, although fluency remained lower than public checkpoints.Source: §4.1 · pp. 4–5, Table 2 · p. 5, Table 3 · p. 5

4.2 Effects of Repeated Contamination Can Be U-Shaped

  • Across SST-2, MMLU, and SQuAD, performance first improved and then declined as repetitions increased; CNN trends varied by metric.CNN ROUGE scores increased with repetition, while UniEval showed a U-shaped trend; the authors note the U-shape may not hold for all datasets and corpora.Source: Figure 1 · p. 6, §4.2 · p. 6

4.3 Effects of Removing Contamination from Pre-Training

  • Removing documents flagged by n-gram or Llama 2 definitions produced no uniform performance pattern, and some removals above 30% left performance comparable to the original model.The authors conclude these definitions do not always identify genuinely contaminated documents; Figure 2 varies cleaning method and removed-token percentage.Source: Figure 2 · p. 7, §4.3 · p. 7

4.4 Scaling Up with a Larger Model

  • Ground-truth contamination had a clear impact in GPT-2-large experiments on CNN and MMLU, despite the larger pre-training corpus.The experiments used a contamination factor of 60 to match the ratio used for GPT-2-small; Table 4 reports the evaluation results.Source: Table 4 · p. 7, §4.4 · p. 7

4.5 Assessing Evaluation-Level Contamination Analysis

  • Categorical evaluation-level analysis found similar performance across categories and no clear advantage for the “dirty” category, despite contamination improving performance in other experiments.The authors caution that categorical evaluations may be insufficient to establish that models are insusceptible to contamination; category sample counts are sensitive to the selected thresholds.Source: §4.5 · p. 8, Table 5 · p. 8

5 Related Work

Prior contamination research has used n-gram, embedding, and syntax-based definitions, but these approaches can miss transformed examples and may not establish pre-training effects. Memorization and contamination detection methods also have practical and interpretive limitations.

5 Related Work

  • Existing n-gram and embedding-based definitions may fail to identify genuine contamination, and simple paraphrasing can evade them.The reviewed work motivates more reliable definitions and detection methods.Source: §5 · pp. 8–9
  • Membership inference can be computationally intensive, while example matching may produce false negatives when evaluation data are paraphrased [22].Likelihood-based heuristics have also been proposed, but their heuristic nature may limit insight into pre-training-level performance effects.Source: §5 · p. 9
  • Memorization is closely linked to contamination because evaluation performance is no longer trustworthy if examples were memorized, regurgitated, and reasoned upon [8] [25] [5].Source: §5 · p. 9

6 Conclusion

The paper studies contamination effects at the pre-training level using GPT-2 models trained from scratch. It finds that ground truth and repeated contamination matter, and that existing n-gram definitions and evaluation-level robustness assessments can be inadequate.

6 Conclusion

  • The study pre-trains GPT-2 models from scratch to examine contamination effects, including the often-overlooked role of ground truth and repeated contamination in downstream applications.Source: §6 · p. 9
  • The paper finds that current n-gram-based contamination definitions can inadequately identify true contamination in pre-training corpora.Source: §6 · p. 9
  • Replicated evaluation-level robustness assessments fall short of affirming that models are robust to pre-training data contamination.The paper calls for more precise contamination definitions and more stringent robustness assessments.Source: §6 · p. 9

A Training Hyperparameters

The GPT-2-small experiments use batch size 32 and learning rate 0.0005, while GPT-2-large uses batch size 128 and learning rate 0.0001. Three seeds are used for the main small-model experiments; the large-model runs use seed 42.

A Training Hyperparameters

  • GPT-2-small uses batch_size=32, learning_rate=0.0005, warmup_ratio=0.01, and weight_decay=0.1, with other hyperparameters at default settings.Source: A Training Hyperparameters · p. 13
  • The three GPT-2-small main-experiment runs use seeds 42, 1234, and 2023 for comparison consistency.Source: A Training Hyperparameters · p. 13
  • GPT-2-large uses batch_size=128, learning_rate=0.0001, and random_seed=42, with other parameters unchanged.The passage states these settings were chosen to ensure training stability.Source: A Training Hyperparameters · p. 13

B Evaluation of Classification Tasks

The appendix reports SST-2 accuracy for ten prompts and notes that GPT-2-small performance is sensitive to prompt structure. It identifies answers outside the label space as a challenge for classification evaluation.

B Evaluation of Classification Tasks

C More Discussions on Data Contamination

On AG News, the subsampled-corpus model already outperformed OpenAI’s public checkpoints, and added text or ground-truth contamination did not significantly improve performance. Performance generally declined at higher contamination factors, an atypical result the authors plausibly attribute to information already present in the subsampled corpus.

C More Discussions on Data Contamination

  • AG News performance was already higher than OpenAI’s public checkpoints on the subsampled corpus, and adding text or ground-truth contamination did not significantly enhance it.The authors suggest the subsampled corpus may already contain assimilated or memorized AG News information.Source: C More Discussions on Data Contamination · p. 13, Figure 3 · p. 14
  • Performance generally began declining at higher contamination factors, with the lowest performance at factor 3, while remaining much higher than public checkpoints.This trend differed from the main experiments.Source: C More Discussions on Data Contamination · p. 13
  • The authors conclude that understanding how pre-training constructs language-model knowledge requires further effort to interpret contamination effects.Source: C More Discussions on Data Contamination · p. 13

D Quantitative Analysis for Contamination Definitions

The analysis compares contamination definitions by examining the pre-training corpus’s contamination ratio for each evaluation dataset. It uses that comparison to assess how strict the definitions are and which may be appropriate.

D Quantitative Analysis for Contamination Definitions

D.1 N-Grams Direct Overlap

The direct n-gram approach measures duplicated content but can miss paraphrases, and its suitable threshold depends on evaluation-text length. PaLM and Llama 2 definitions also rely on n-gram duplication, with parameter choices affecting identified contamination and creating risks of misclassification.

D.1 N-Grams Direct Overlap

  • Direct n-gram overlap marks a training document contaminated when any document n-gram appears in evaluation data, but paraphrased sentences can evade the measure.The approach demonstrates how much evaluation content appears in pre-training data; the suitable n depends on evaluation-text length.Source: §D.1 · pp. 14–15, Figure 4 · p. 14

D.2 PaLM and Llama 2’s Definitions

  • PaLM’s percentage-overlap definition identified very few contaminated documents, even with n and λ relaxed to small values.Source: §D.2 · p. 15
  • Llama 2’s definitions yield varied contaminated-document and token rates as parameters change, while token duplication can misclassify contextually different uses.Choosing n-gram values and thresholds for different datasets remains challenging.Source: §D.2 · p. 15, Figure 5 · p. 15

Related papers

Loading 2401.06059v1…