PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation

Yucheng Li

arXiv:2309.10677v2cs.CLcs.AI

TL;DR

Training-data access limits contamination audits, although such analysis matters for credible model evaluation. The paper estimates memorisation by comparing benchmark perplexity with memorised and clean baselines. It reports significant memorisation in reading-comprehension and summarisation benchmarks, but no clear contamination evidence for the tested multiple-choice quizzes.

  • Problem

    Existing contamination analysis requires access to full training corpora, which are often unavailable for recent models and limits rigorous community audits.

    Source: §1 · p. 1

  • Method

    The method estimates contamination without full training-data access by comparing benchmark perplexity with memorised and clean baselines.

    Source: §1 · p. 1, §5 · p. 6

  • Results

    The authors report significant memorisation on reading-comprehension benchmarks and XSum, but no clear contamination evidence for the tested multiple-choice quizzes.

    Source: §4.1 · p. 5, §4.2 · p. 6, §4.3 · p. 6

  • Takeaways & Limitations

    The approach supports contamination analysis and credible evaluation when model training data are unavailable.

    Source: §5 · p. 6

Abstract

from arXiv · show

Data contamination in model evaluation is getting increasingly prevalent as the massive training corpora of large language models often unintentionally include benchmark samples. Therefore, contamination analysis has became an inevitable part of reliable model evaluation. However, existing method of contamination analysis requires the access of the entire training data which is often confidential for recent models. This prevent the community to rigorously audit these models and conduct accurate assessment of their capability. In this paper, we propose a novel method to quantify contamination without the access of the full training set, that measure the extent of contamination with perplexity. Our analysis provides evidence of significant memorisation of recent foundation models in popular reading comprehension, summarisation benchmarks, while multiple choice appears less contaminated.

1 Introduction

Web-sourced benchmarks can overlap with pretraining data, making contamination analysis important to credible evaluation. The paper proposes estimating memorisation from perplexity comparisons without access to the full training corpus.

  • Benchmark examples can enter web-scraped training corpora, letting models memorize test data instead of generalizing and distorting evaluation.Source: §1 · p. 1
  • Existing audits compare test and training data for n-gram overlap, but recent models' training corpora are often unavailable, limiting community audits.Source: §1 · p. 1
  • The proposed method estimates contamination without full training-data access by examining model memorisation rather than searching for n-gram overlap.Source: §1 · p. 1
  • Test-set perplexity closer to the memorised baseline indicates greater memorisation, while closeness to the clean baseline indicates low or negligible memorisation.Source: §1 · p. 2
  • For a fair comparison, the baselines should match the benchmark's source, format, and length.Source: §1 · p. 2

2 Background

The background defines contamination, reviews its reported prevalence and detection approaches, and notes that its effects on measured performance vary across settings.

  • Data contamination occurs when evaluation examples also appear in training data; input-and-label contamination includes both inputs and labels, while input contamination includes only inputs.Source: §2 · p. 2
  • Prior reports find contamination across NLP benchmarks, including over 90% of examples flagged in QuAC, SQuADv2, and DROP in the GPT-3 study (Brown et al., 2020).Source: §2 · p. 2
  • Existing identification approaches generally match evaluation examples against training text, making contamination difficult to inspect when training data are inaccessible.Source: §2 · p. 2
  • The effect on performance is dataset- and model-dependent; reported clean-versus-dirty gaps vary, and some comparisons show no significant gap.Source: §2 · p. 3

3 Method

The method has three components: computing benchmark perplexity, preparing memorised and clean baselines, and interpreting their comparison. Baselines are tailored to benchmark characteristics to make the comparison fair.

3 Method

  • The method comprises perplexity computing, baseline preparation, and analysis.Source: §3 · p. 3

3.1 Perplexity Computing

  • Each benchmark sample is verbalised as a model input, and its log-perplexity is computed from token probabilities before sequence scores are averaged across the dataset.Source: §3.1 · p. 3

3.2 Baseline Preparation

  • The memorised baseline uses texts likely included in training, while the clean baseline uses fresh texts, such as material created after model release.Source: §3.2 · p. 3

SQuAD_v2

  • For SQuAD, baselines use Wikipedia texts in the benchmark's passage-question-answer format and match its length; the two baselines and benchmark are scored alike.Source: SQuAD_v2 · p. 4, Figure 2 · p. 4

3.3 Analysis

  • A benchmark perplexity closer to the memorised baseline indicates high memorisation, while closer alignment with the clean baseline indicates low or negligible memorisation.Source: §3.3 · p. 4

4 Experiments

Across reading comprehension and XSum, the tested foundation models show memorisation signals, while the sampled MMLU quizzes show no clear contamination evidence.

4 Experiments

  • The experiments cover reading comprehension, summarisation, and multiple-choice benchmarks, tested on GPT-3 and LLAMA-7B, 13B, and 30B.Source: §4 · p. 4

4.1 Reading Comprehension

  • Reading-comprehension analysis uses only benchmark passages and Wikipedia baselines matched to source and model-reported training periods.Source: §4.1 · pp. 4–5
  • All four foundation models exhibit significant memorisation on QuAC, BoolQ, and SQuAD_v2; larger models’ benchmarks are largely memorised, while LLAMA-7B shows some.Source: §4.1 · pp. 4–5, Figure 3 · p. 5

4.2 Summarisation

  • XSum analysis scores both documents and summaries, using BBC articles as baselines and June 2023 news for the clean set, length-matched to 358 words.Source: §4.2 · pp. 5–6
  • All foundation models show significant memorisation on XSum, and LLAMA-30B has lower perplexity on XSum than on the memorised baseline.Source: Figure 3 · p. 5, §4.2 · p. 6

4.3 Multi-Choice

  • Only a small MMLU quiz portion was analyzed; quizzes from 2017–2023 were formatted as language-model inputs after PDF-to-DOCX text extraction.Source: §4.3 · p. 6
  • The tested MMLU multiple-choice quizzes show no clear contamination evidence or observable perplexity correlation with time distribution.Source: Figure 3 · p. 5, §4.3 · p. 6

5 Conclusion

The conclusion proposes comparing benchmark perplexity with memorised and clean baselines to estimate contamination without full training data. The approach is applied across three benchmark types to detect contamination and support credible evaluation.

  • The method estimates benchmark contamination without full training-data access by comparing test perplexity with memorised and clean baselines.Source: §5 · p. 6
  • Experiments across reading comprehension, summarisation, and multiple-choice benchmarks demonstrate that the approach can detect contamination.Source: §5 · p. 6
  • The method enables contamination analysis when training data are unavailable, supporting credible evaluation.Source: §5 · p. 6

A Multi-choice Quizzes

The appendix lists the quizzes included in the multi-choice experiment and identifies their sources as coming from the MMLU GitHub repository.

Loading 2309.10677v2…