PaperlayerView exact revision on arXiv

Source-linked AI summary

Proving Test Set Contamination in Black Box Language Models

Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, Tatsunori B. Hashimoto

arXiv:2310.17623v2cs.CLcs.LG

TL;DR

Public benchmark contamination is difficult to prove when training data and model weights are inaccessible. The paper uses exchangeability-based comparisons of canonical and shuffled order likelihoods to obtain black-box statistical guarantees, detecting rare contamination in controlled tests while finding little pervasive contamination in public models.

  • Problem

    Benchmark contamination complicates interpretation of language-model performance, while provider filtering and heuristic methods do not provide reliable proof without access to training data.

  • Method

    The paper compares model log probabilities for canonical and shuffled orderings of exchangeable benchmarks, using sharded statistical tests that require neither training data nor model weights.

  • Results

    The test detected contamination in a 1.4-billion-parameter model with 1000-example datasets and sometimes two pretraining occurrences, while audits found little pervasive contamination in public models.

  • Takeaways & Limitations

    Black-box identification of test-set contamination is practical and may support third-party auditing of language models.

  • Takeaways & Limitations

    The test targets verbatim contamination and cannot rule out more complex forms of partial contamination or contamination through benchmark source data.

Abstract

from arXiv · show

Large language models are trained on vast amounts of internet data, prompting concerns and speculation that they have memorized public benchmarks. Going from speculation to proof of contamination is challenging, as the pretraining data used by proprietary models are often not publicly accessible. We show that it is possible to provide provable guarantees of test set contamination in language models without access to pretraining data or model weights. Our approach leverages the fact that when there is no data contamination, all orderings of an exchangeable benchmark should be equally likely. In contrast, the tendency for language models to memorize example order means that a contaminated language model will find certain canonical orderings to be much more likely than others. Our test flags potential contamination whenever the likelihood of a canonically ordered benchmark dataset is significantly higher than the likelihood after shuffling the examples. We demonstrate that our procedure is sensitive enough to reliably prove test set contamination in challenging situations, including models as small as 1.4 billion parameters, on small test sets of only 1000 examples, and datasets that appear only a few times in the pretraining corpus. Using our test, we audit five popular publicly accessible language models for test set contamination and find little evidence for pervasive contamination.

1 INTRODUCTION

The paper develops black-box statistical tests that provide provable guarantees for detecting benchmark contamination without access to training data or model weights. The approach compares model likelihoods for canonical and shuffled dataset orderings, and experiments show sensitivity on small and rarely duplicated test sets while finding little pervasive contamination in public models.

  • Motivation: Benchmark contamination makes language-model performance difficult to interpret because memorized answers can be mistaken for generalization.Filtering may fail, cover only selected benchmarks, or require trusting model providers.
  • Motivation: Heuristic membership-inference methods can provide evidence of contamination but cannot turn suspected contamination into irrefutable proof.
  • Approach: The paper provides a statistical test with provable false positive rate guarantees using only log-probability queries, without training-data or weight access.
  • Approach: The test exploits exchangeability by comparing likelihoods for a benchmark’s canonical ordering and randomly shuffled orderings, flagging statistically significant differences.A preference for a public canonical ordering can indicate that the model observed the dataset during training.
  • Approach: The sharded test improves computational efficiency and power for small p-values by partitioning the dataset and performing parallel log-probability comparisons.
  • Results: A 1.4-billion-parameter model revealed contamination with as few as 1000 examples and, in some cases, datasets appearing only twice during pretraining; public-model audits found little pervasive contamination.Datasets duplicated 10 or more times produced vanishingly small p-values.

2 PROBLEM SETTING

The testing problem is to determine whether a model’s training process included a dataset using only log-probability queries. Exchangeability yields permutation invariance under no contamination, while contamination can produce higher likelihood for the canonical ordering; power nevertheless requires additional assumptions.

  • Problem setting: The goal is to determine whether a model’s training process included dataset X using only log-probability queries, without access to its data or parameters.
  • Problem setting: Provable contamination detection is formulated as a hypothesis test that controls the probability of falsely rejecting the no-contamination hypothesis.
  • Null hypothesis: Under exchangeability, every ordering of dataset examples has equal probability, including orderings generated by uniformly random permutations.This property holds under the standard assumption that examples are independently and identically distributed.
  • Null hypothesis: Under the no-contamination hypothesis, exchangeability implies that model log probabilities are invariant to shuffling the dataset examples.This permutation invariance supports standard permutation-test guarantees.
  • Power: False positive guarantees alone do not ensure high detection rates, which require further assumptions about how contamination affects the model.An encrypted copy of the dataset could create dependence while remaining nearly impossible to detect.
  • Alternative hypothesis: Benign contamination is expected to violate this invariance because training on the dataset can make the canonical sequence more likely than shuffled sequences.

3 METHODS

The methods compare a benchmark’s canonical ordering with shuffled orderings to test contamination, then improve the basic permutation test through sharding and t-test aggregation. The sharded procedure addresses the naive test’s computational and low-p-value limitations while retaining false-positive guarantees under stated assumptions.

  • 3.1 A PERMUTATION TEST FOR CONTAMINATION: The basic permutation test compares the canonical ordering’s log probability with those of randomly permuted datasets.Under the null, exchangeability makes permutation probabilities identically distributed, so the canonical ordering’s rank is uniform.
  • 3.1 A PERMUTATION TEST FOR CONTAMINATION: The permutation p-value is uniform under H0, enabling rejection at significance level α when p < α.Because exhaustive permutation averaging is intractable, the method uses a Monte Carlo estimate with a finite-sample correction.
  • 3.1 A PERMUTATION TEST FOR CONTAMINATION: O(m|X|) runtime and a minimum p-value of 1/(m + 1) limit the naive test’s power for very small α.This creates a tradeoff between statistical power and the number of log-probability computations.
  • 3.1 A PERMUTATION TEST FOR CONTAMINATION: The naive test can also fail when models prefer certain orderings, such as placing duplicate examples together, independently of training exposure.A more suitable alternative tests whether canonical log probability exceeds the average log probability under random permutations.
  • 3.2 A SHARDED LIKELIHOOD COMPARISON TEST FOR CONTAMINATION: The sharded test partitions examples into contiguous shards, permutes examples within each shard, and compares canonical likelihood with average shuffled likelihood.Each shard contains at least k = n/r examples, and shard-level statistics are aggregated by their mean.
  • 3.2 A SHARDED LIKELIHOOD COMPARISON TEST FOR CONTAMINATION: A one-sided t-test tests whether the mean shard statistic is greater than zero, providing an efficient aggregate contamination test.The procedure uses m permutations per shard and retains O(m|X|) runtime without the naive test’s 1/m minimum p-value.
  • 3.2 A SHARDED LIKELIHOOD COMPARISON TEST FOR CONTAMINATION: Under the null, the sharded test has asymptotically valid p-values when shards are independent and log probabilities have finite second moments.The guarantee follows from zero-mean shard statistics, the central limit theorem, and t-testing, with P(p < α) → α as m → ∞.
  • 3.2 A SHARDED LIKELIHOOD COMPARISON TEST FOR CONTAMINATION: The sharded guarantee is asymptotic rather than finite-sample and its proof assumes an i.i.d. dataset X.The authors note that Berry–Esseen bounds or results for sums of exchangeable variables could relax these conditions.

4 EXPERIMENTS

Experiments validate the contamination test on intentionally contaminated models, explore its sensitivity and computational design, and apply it to existing open models. The controlled experiments show strong detection at higher duplication rates, while audits of open models provide mostly non-significant results with important caveats for non-exchangeable benchmarks.

  • 4.1 PRETRAINING WITH INTENTIONAL CONTAMINATION: Near-zero p-values occurred when injected datasets appeared at least 10 times, including datasets with no more than 1000 examples.The permutation and sharded likelihood comparison tests were reported for each injected dataset.
  • 4.1 PRETRAINING WITH INTENTIONAL CONTAMINATION: 1.96e-11: matching this MNLI p-value with the permutation test would require 10^11 dataset permutations and forward passes.The sharded test achieved comparably low p-values with substantially less computation and remained effective at small duplication rates.
  • 4.1 PRETRAINING WITH INTENTIONAL CONTAMINATION: Duplication count 4 marked the approximate detection threshold: count 1 was unreliable, while counts 2 and 4 detected some test sets.The authors suggest future power improvements could enable reliable detection at lower duplication rates.
  • 4.1 PRETRAINING WITH INTENTIONAL CONTAMINATION: The paper releases a public benchmark to support further development of contamination tests, especially for duplication rate 1.The authors identify reliable detection at duplication rate 1 as an open problem.
  • 4.2 SHARDING AND PERMUTATION COUNT: Detection was maximized around 10–20 shards, whereas larger shards counts performed worse and fewer than 10 shards provided too few samples for the t-test.More permutations generally improved power, with substantial diminishing returns beyond 25 permutations.
  • 4.3 EVALUATING EXISTING MODELS FOR DATASET CONTAMINATION: Across five open models and eight benchmarks, most results were non-significant, and the authors found no evidence of pervasive verbatim test-set contamination.AI2-ARC on Mistral produced p = 0.001, but the authors caution that multiple-testing correction and garden-of-forking-paths concerns make boundary results uncertain.
  • 4.3 EVALUATING EXISTING MODELS FOR DATASET CONTAMINATION: MMLU required heuristic filtering because at least 14 constituent test sets were considered non-exchangeable, so its aggregated p-values could no longer prove contamination.The resulting values provided heuristic evidence consistent with mild MMLU contamination.

5 RELATED WORK

The paper distinguishes its provable contamination test from heuristic memorization, n-gram, and membership-inference approaches. Its key distinction is using exchangeability for exact guarantees and a sensitive shard-based test.

  • Memorization research largely examines what language models retain or expose, whereas this work specifically tests for test set contamination.
  • N-gram contamination analyses can produce high false positives and only cover datasets selected for analysis.The passage gives SQuAD containing Wikipedia as an example of a potential false positive.
  • Prompt-based heuristics identify contamination through generated examples or completions but lack provable false-positive guarantees and require detectable strong memorization.
  • The exposure statistic compares target-sequence perplexity with random sequences, an idea related to this paper’s log-probability comparisons.
  • This work differs by exploiting dataset exchangeability for an exact null distribution and developing a sensitive, efficient shard-based test.

6 LIMITATIONS

The authors identify three limitations: uncorrected multiple testing, uncertainty about benchmark exchangeability, and coverage limited to verbatim contamination. These constraints bound how test results should be interpreted.

  • The reported p-values do not apply multiple-test corrections because the total number of development hypotheses is difficult to define.
  • Practical use requires an off-the-shelf benchmark whose exchangeability is difficult to establish without knowing its data-generating process.
  • The test targets verbatim ingestion of a benchmark and cannot rule out more complex partial contamination.

7 CONCLUSION

The paper concludes that black-box contamination tests with false-positive guarantees can be built using only log probabilities. Its sharding-based tests show power on canaries and public models, while single-duplication detection remains open.

  • The proposed statistical test provides false-positive rate guarantees and requires only the ability to compute log probabilities.
  • Sharding-based tests demonstrate contamination-detection power on carefully constructed canaries and publicly available language models.
  • Reliable detection at the single-duplication-count regime remains an open problem.

APPENDICES

For sequences longer than the model context, the appendix uses a strided-window method to compute log-likelihoods. A stride equal to half the context length is used because smaller strides provide no significant gains.

  • Long-sequence log-likelihoods are computed with strided windows using a stride equal to half the model’s context length.Decreasing the stride below half the context length does not yield significant gains.

B PRETRAINING DETAILS

This section details the 1.4B language model’s architecture and training setup, including its optimization schedule and data-consumption target.

  • The training procedure was conducted from scratch on intentionally contaminated Wikitext.
  • The model uses a GPT-2 architecture with 1.4B parameters, 1536 hidden dimensions, 24 heads, 48 layers, and sequence length 2048.
  • Training used batch size 256 for 46,000 steps, consuming the mixture datasets’ tokens exactly once.
  • The model was optimized with AdamW using a 1e-4 learning rate and 0.1 weight decay.

C 10 CANARY DATASETS

The experiments inject ten canary datasets into Wikitext, generally sampling 1000 examples while using all examples for the smaller OpenbookQA test set.

  • Six canary datasets—BoolQ, HellaSwag, MNLI, Natural Questions, TruthfulQA, and PIQA—contribute random subsets of 1000 examples.
  • OpenbookQA contributes all 500 examples because its test set contains only n=500 examples.
  • MMLU subsets were selected to exclude multi-line examples and satisfy an additional selection criterion.

D FULL MMLU RESULTS

This section presents the full MMLU results used to compute omnibus contamination p-values, alongside the empirical CDF comparison in Figure 4.

  • The reported MMLU results generate omnibus p-values for contamination before filtering suspected non-exchangeable test files.
  • Figure 4 plots empirical CDFs of MMLU p-values for LLaMA2, Mistral, and Pythia against a Uniform(0,1) CDF after excluding significant BioMedLM and GPT-2 files.
  • The full MMLU results cover LLaMA2-7B, Mistral-7B, Pythia-1.4B, GPT-2XL, and BioMedLM.
Loading 2310.17623v2…