Source-linked AI summary
Investigating Data Contamination in Modern Benchmarks for Large Language Models
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, Arman Cohan
TL;DR
Benchmark contamination may inflate LLM evaluation scores, especially when training data are undisclosed. The paper combines corpus retrieval with TS-Guessing, finding that commercial models can guess masked benchmark information and raising concerns about leakage in contemporary benchmarks.
Problem
Benchmark instances may appear in large-scale training data, while existing detection methods are limited by training-corpus access, dataset-level granularity, or open-model fine-tuning requirements.
Method
The paper develops a retrieval system for benchmark–corpus overlap and TS-Guessing, which masks words or answer options and asks LLMs to reconstruct them.
Results
Commercial LLMs guessed masked benchmark information, including a 57% Exact Match rate for ChatGPT on masked MMLU choices.
Takeaways & Limitations
The findings add evidence of potential contamination in widely used benchmarks and motivate scrutiny of benchmark leakage.
Takeaways & Limitations
The retrieval system uses only BM25, requires approximately 2–3 minutes per data point, and TS-Guessing relies on concise instruction comprehension.
Abstract
from arXiv · showhide
Recent observations have underscored a disparity between the inflated benchmark scores and the actual performance of LLMs, raising concerns about potential contamination of evaluation benchmarks. This issue is especially critical for closed-source models and certain open-source models where training data transparency is lacking. In this paper we study data contamination by proposing two methods tailored for both open-source and proprietary LLMs. We first introduce a retrieval-based system to explore potential overlaps between evaluation benchmarks and pretraining corpora. We further present a novel investigation protocol named \textbf{T}estset \textbf{S}lot Guessing (\textit{TS-Guessing}), applicable to both open and proprietary models. This approach entails masking a wrong answer in a multiple-choice question and prompting the model to fill in the gap. Additionally, it involves obscuring an unlikely word in an evaluation example and asking the model to produce it. We find that certain commercial LLMs could surprisingly guess the missing option in various test sets. Specifically, in the TruthfulQA benchmark, we find that LLMs exhibit notable performance improvement when provided with additional metadata in the benchmark. Further, in the MMLU benchmark, ChatGPT and GPT-4 demonstrated an exact match rate of 52\% and 57\%, respectively, in guessing the missing options in benchmark test data. We hope these results underscore the need for more robust evaluation methodologies and benchmarks in the field.
1 Introduction
The paper addresses benchmark contamination by combining corpus retrieval with TS-Guessing, which masks test-set information for LLM completion. Its findings report overlap missed by n-gram methods and substantial missing-option guessing by commercial models.
- Motivation: Benchmark contamination can artificially inflate evaluation metrics, while crawled web-scale training data may contain benchmark instances.This concern applies to both pretraining and fine-tuning contamination.
- Motivation: Existing n-gram detection depends on full training-corpus access, limiting its use for undisclosed proprietary models.Other corpus-independent approaches may operate only at dataset level or require open-model fine-tuning.
- Approach: The paper combines a Pyserini-based retrieval system with TS-Guessing to investigate contamination in open and black-box models.The retrieval system searches the Pile and C4 for overlap with several modern benchmarks; TS-Guessing masks benchmark information for model completion.
- Findings: The analysis found contaminated instances missed after n-gram tokenization and reported potential contamination in widely used benchmarks.The reported benchmark set includes MMLU, TruthfulQA, HellaSwag, WinoGrande, GSM8K, OpenBookQA, and PIQA.
- Findings: 57% Exact Match was achieved by ChatGPT when guessing masked options in the MMLU test set.Commercial models also performed notably in TruthfulQA when benchmark metadata was provided, and fully contaminating ChatGPT made EM nearly 100%.
2 Related Work
Prior work largely retrieves or probes training data to detect contamination, using corpus matching, model-output differences, ordering probes, and masking-based methods. These approaches address contamination from complementary angles but include access, granularity, or model-adaptation constraints.
- Retrieval from Large Corpora: Prior retrieval-based work uses n-gram indexing and overlap thresholds to compare large training corpora with benchmark data.Examples include GPT-3’s 13-gram strategy and PaLM’s 8-gram approach with a 70% overlap criterion.
- Alternative Detection Methods: Other studies probe contamination through changes in prompted outputs, data generated before or after training, or canonical test-set ordering.These methods avoid relying solely on direct document retrieval.
- Alternative Detection Methods: Related work also proposes leakage-mitigation recommendations, dynamic evaluation, and masking-based tests of memorization.These directions focus on reducing or diagnosing benchmark exposure through evaluation design and model behavior.
3 Method
The method combines retrieval-based corpus overlap analysis with TS-Guessing tasks for probing possible benchmark contamination in open and black-box LLMs. It retrieves and scores corpus documents, then masks pivotal words or wrong multiple-choice answers for the model to guess.
- Retrieval-based contamination detection: Queries use the question, label, or concatenated question-label, with top-k retrieval settings of k=1, k=5, and k=10.For MMLU, concatenating the question and label is used to improve retrieval efficiency because labels lack question context.
- TS-Guessing: TruthfulQA receives hint-augmented prompts containing metadata, whereas MMLU uses no hint because it provides only questions and answers.TruthfulQA metadata includes the data type, category, and source link associated with each example.
- Retrieval-based contamination detection: The retrieval system targets The Pile and C4 using Pyserini-based indexing and BM25 ranking.These open corpora are widely used as pretraining data for several LLM families.
- Retrieval-based contamination detection: Retrieved documents are chunked with 13-gram tokenization, and the highest chunk-to-benchmark similarity score measures potential overlap.Questions and labels are first concatenated into unified retrieval queries when labels are available.
- TS-Guessing: In the Question-based TS-Guessing task, the method identifies a pivotal keyword and replaces it with [MASK] for the model to predict.The keyword is selected to capture the sentence’s core meaning, such as masking “fortune” in a question about fortune cookies.
- TS-Guessing: The Question-Multichoice task masks one wrong test-set option while retaining the correct answer and two other wrong answers.The correct option is not masked, and filtering removes cases with strong correlations between answer options.
4 Experiment
The experiments evaluate retrieval-based contamination signals and TS-Guessing across widely used benchmarks and modern closed- and open-source LLMs. Results show human-aligned semantic scoring, notable benchmark-specific guessing success, and near-perfect detection after deliberate contamination.
- Experimental setup: The study evaluates MMLU, TruthfulQA, GSM8K, PIQA, HellaSwag, WinoGrande, and OpenbookQA using retrieval and TS-Guessing across closed- and open-source models.The model set includes ChatGPT, GPT-4, Claude variants, LLaMA 2-13B, and Mistral-7B.
- Human evaluation: 23 of 100 sampled benchmark examples were judged contaminated or benchmark data, while annotator agreement reached Krippendorff’s alpha 0.8673.The sample was drawn from seven benchmarks and evaluated by 17 NLP volunteers.
- Metric analysis: GPTscore aligned more closely with human evaluation than conventional metrics, but its greater resource requirements may limit large-scale use.The comparison was based on a small-scale experiment correlating automated and human scores.
- TS-Guessing results: 16.24% of TruthfulQA cases yielded successful missing-word guesses, with performance affected by metadata hints.The authors connect this result to TruthfulQA’s publicly accessible, web-derived sources and recommend considering source-document novelty in future benchmarks.
- TS-Guessing results: ChatGPT achieved a 57% EM rate when guessing missing incorrect choices in MMLU after filtering correlated options and simple expressions.The authors report that the remaining options were disorganized and complex, making successful guesses notable.
- Contamination probing: Fine-tuning ChatGPT on the MMLU test set produced nearly 100% EM in both question-based and question-multichoice formats.This deliberate contamination experiment was designed to test whether TS-Guessing responds to known benchmark exposure.
5 Tradeoff between Retrieval and TS-Guessing
The paper contrasts retrieval with TS-Guessing as complementary contamination-detection approaches. Retrieval is scalable and cost-effective but requires full training-corpus access, whereas TS-Guessing works without training-data transparency and applies to both model-access settings.
- Retrieval: Retrieval constructs an information-retrieval system to identify contaminated data in commonly used pretraining corpora.The approach is derived from GPT-3’s methodology and extended to LLaMA.
- TS-Guessing: TS-Guessing detects potential contamination without access to training data and applies to both closed-source and open-source models.This makes it suitable when training-corpus transparency is unavailable.
- Tradeoff: Retrieval requires the full training corpus and is time-consuming, whereas TS-Guessing does not require corpus access and is less time-consuming.The comparison identifies distinct operational tradeoffs between the methods.
- Tradeoff: Retrieval is cost-effective and scalable, while TS-Guessing extends contamination testing to models lacking transparent training data.The methods therefore differ in scalability, access requirements, and model applicability.
6 Conclusion and Future Work
The paper introduces two complementary approaches for investigating data contamination in contemporary evaluation benchmarks: retrieval-based overlap detection and TS-Guessing for assessing potential benchmark leakage.
- The paper develops an information retrieval system to identify evaluation benchmarks with significant overlap with pre-training corpora.
- TS-Guessing is proposed as an investigation protocol for assessing potential data leakage in benchmark datasets evaluated with LLMs.
- Commercial LLMs, including ChatGPT, can accurately complete missing or incorrect options in test sets.
7 Limitations
The retrieval and TS-Guessing methods have practical and methodological limitations. Retrieval precision, computational cost, contamination scoring, and instruction-following behavior constrain their use and interpretation.
- The retrieval system uses only BM25, which may reduce retrieval precision.
- Retrieval takes approximately 2–3 minutes per data point, making the system impractical without a high-performance computer.
- TS-Guessing relies on LLMs comprehending instructions succinctly, while many open-source models predicted correct answers regardless of instruction framing.
8 Ethics Statement
The paper uses pretrained corpora and public-resource benchmarks to study contamination, combining retrieval with human and automatic similarity assessments.
- The first method retrieves benchmark-related data from pretrained corpora such as The Pile and C4.
- The second method examines benchmarks derived from public resources for potential contamination.
- Human annotators scored alongside automatic metrics to measure similarity.
A Query Type
The retrieval experiments compare question-only, label-only, and question-label inputs across The Pile and C4. For MMLU and TruthfulQA, concatenating the question with its label is reported as most effective.
- The three query types are question-only, label-only, and question-label inputs.
- For MMLU, labels without question context are suboptimal for retrieval, so subsequent experiments concatenate the question and label.
- Question-label concatenation is the most effective retrieval strategy for MMLU and TruthfulQA.
C Contaminated Examples
The paper presents a concrete TruthfulQA contamination example in which benchmark data significantly overlaps with documents from the C4 corpus.
- Significant overlap exists between TruthfulQA benchmark data and documents from the C4 corpus.Models pretrained on C4 are therefore likely to have encountered this benchmark data during pretraining.
D Corrleation between TS-Guessing and Task Accuracy
The authors examine whether TS-Guessing performance relates to task accuracy using Spearman correlation and Rouge-L F1, while discussing limitations of contamination detection. They report different correlation patterns across TruthfulQA and MMLU and caution that n-gram methods may be insufficient for rigorous detection.
- Spearman correlation between task performance and Rouge-L F1 evaluates the relationship between TS-Guessing and question-answering performance.The Question-Multichoice task is used for this analysis, with scores standardized to a 0-1 scale.
- TruthfulQA shows negative correlations of −0.158 for GPT-4 and −0.128 for ChatGPT between task performance and TS-Guessing.
- MMLU shows a positive correlation of 0.279 for GPT-4 between task performance and TS-Guessing.
- N-gram-based algorithms offer convenience but may not rigorously detect data contamination in LLMs.Despite this limitation, the method is widely used for decontaminating models such as GPT-3, Llama, and Llama 2.
- Unknown training techniques and training data in closed-source LLMs make contamination analysis challenging.Different techniques, including SFT, RLHF, and MoE, may produce varying results under the same evaluation methods.