Source-linked AI summary
Time Travel in LLMs: Tracing Data Contamination in Large Language Models
Shahriar Golchin, Mihai Surdeanu
TL;DR
Data contamination may inflate LLM evaluations, so the paper proposes detecting it without access to pre-training data. Guided instruction identifies suspicious instances, while partition-level rules assess broader contamination; GPT-4 few-shot evaluation achieved the strongest agreement with human judgments, and GPT-4 showed dataset exposure.
Problem
Data contamination may inflate the apparent validity of LLM evaluations and benchmarks.
Method
The method uses guided instruction to detect instance-level contamination, then extrapolates to partitions using overlap comparisons or GPT-4 few-shot evaluation.
Results
GPT-4 few-shot evaluation classified 14/14 partitions correctly, while GPT-3.5 achieved 13/14; human evaluation found GPT-4 exposure to AG News, WNLI, and XSum.
Takeaways & Limitations
LLMs have ingested NLP datasets, motivating contamination-aware scientific experiments and benchmark evaluation.
Takeaways & Limitations
The best method does not distinguish contamination sources or forms, treating exact and near-exact replicas alike.
Abstract
from arXiv · showhide
Data contamination, i.e., the presence of test data from downstream tasks in the training data of large language models (LLMs), is a potential major issue in measuring LLMs' real effectiveness on other tasks. We propose a straightforward yet effective method for identifying data contamination within LLMs. At its core, our approach starts by identifying potential contamination at the instance level; using this information, our approach then assesses wider contamination at the partition level. To estimate contamination of individual instances, we employ "guided instruction:" a prompt consisting of the dataset name, partition type, and the random-length initial segment of a reference instance, asking the LLM to complete it. An instance is flagged as contaminated if the LLM's output either exactly or nearly matches the latter segment of the reference. To understand if an entire partition is contaminated, we propose two ideas. The first idea marks a dataset partition as contaminated if the average overlap score with the reference instances (as measured by ROUGE-L or BLEURT) is statistically significantly better with the completions from guided instruction compared to a "general instruction" that does not include the dataset and partition name. The second idea marks a dataset partition as contaminated if a classifier based on GPT-4 with few-shot in-context learning prompt marks multiple generated completions as exact/near-exact matches of the corresponding reference instances. Our best method achieves an accuracy between 92% and 100% in detecting if an LLM is contaminated with seven datasets, containing train and test/validation partitions, when contrasted with manual evaluation by human experts. Further, our findings indicate that GPT-4 is contaminated with AG News, WNLI, and XSum datasets.
1 INTRODUCTION
The paper presents an inexpensive, robust approach for detecting LLM data contamination without pre-training-data access, then evaluates it across datasets and models. It reports high accuracy and evidence of GPT-4 contamination in several dataset test partitions.
- Data contamination may inflate LLM performance on downstream tasks and raise questions about evaluation validity.
- The proposed approach samples individual instances, uses their contamination signals, and generalizes decisions to entire dataset partitions.It assumes no direct access to pre-training data and limited computational resources.
- Guided instruction prompts an LLM with a dataset name, partition type, and random initial instance segment to complete the reference instance.
- The evaluation covers 28 scenarios spanning seven datasets, GPT-3.5 and GPT-4, and classification, summarization, and NLI tasks.Train and test or validation partitions are assessed separately.
- The best method achieves 92%–100% accuracy against human contamination labels for dataset partitions.It combines guided instruction with GPT-4 few-shot in-context evaluation of generated completions.
- GPT-4 shows evidence of contamination in the test partitions of AG News, WNLI, and XSum.The paper concludes that contamination should be considered in downstream LLM evaluations.
2 RELATED WORK
Prior contamination-detection methods have often required pre-training-data access, substantial computation, or extensive manual work. The paper positions its approach alongside a related generation-based method that can fail because of sparsity and safety filters.
- Earlier studies detected overlap between pre-training and evaluation data using high-order n-grams such as 13-grams.
- Existing approaches can require access to pre-training data, substantial computational resources, or extensive manual labor.
- Sainz et al. prompted ChatGPT to generate the first instances from dataset partitions without access to pre-training data.
- The paper reports that this generation-based method can fail because reproducing first instances is sparse and safety filters may block copyrighted dataset content.
3 APPROACH
The approach detects contamination by prompting LLMs to reproduce sampled dataset instances, then extrapolating instance-level evidence to partition-level decisions.
- The method examines sampled instances for contamination and uses their signals to decide whether an entire dataset partition is contaminated.The approach operates without direct access to pre-training data and with limited computational resources.
- Components to measure instance-level contamination: An instance is flagged when the guided completion exactly or nearly replicates the reference continuation, although probabilistic generation can prevent perfect replication.General instruction omits dataset and partition identifiers to measure the effect of this guidance.
- Components to measure instance-level contamination: Guided instruction combines the dataset name, partition, and a random initial segment of a reference instance to direct the LLM's completion.Paired-instance prompts include the initial sentence and label; single-instance texts are cut at sentence or fragment boundaries.
- Measuring instance-level contamination: ROUGE-L and BLEURT compare guided and general completions with references, measuring lexical similarity and semantic relevance, respectively.The first partition-level method uses statistically significant guided-over-general overlap rather than an arbitrary score threshold.
- Detecting partition-level contamination: GPT-4 few-shot in-context evaluation identifies exact or near-exact matches, marking a partition contaminated with at least one exact or two near-exact matches.This second algorithm addresses the limitation that higher overlap scores may not correspond to human-judged exact or near-exact matches.
4 EXPERIMENTAL SETUP
The experiments evaluate contamination detection across seven datasets, two LLMs, and training and test/validation partitions, using human judgments and a comparative method.
- Data: The evaluation covers seven datasets spanning classification, summarization, and natural language inference, with experiments on training and test/validation splits.The datasets are IMDB, AG News, Yelp Full Reviews, SAMSum, XSum, WNLI, and RTE.
- Setting: GPT-3.5 and GPT-4 snapshots from June 13, 2023 are accessed through the OpenAI API with temperature zero and a 500-token completion cap.The comparative ChatGPT-Cheat? method uses the chat user interface.
- Human Evaluation: Two domain experts identify exact and near-exact instance matches and assess contamination at the partition level.Near-exact matches substantially overlap the reference while preserving significant semantic and structural similarity.
- Evaluation Metrics: BLEURT scoring differs by dataset structure: single-instance continuations are joined to severed references, while other instances use their generated sequences and references according to structure.
- Comparative Framework: The proposed binary contamination decisions are compared with ChatGPT-Cheat?, which additionally allows a suspicious category for dataset-specific outputs that are not actual instances.
5 RESULTS AND DISCUSSION
Across 28 settings, the GPT-4 few-shot evaluation method most consistently matched human judgments, while the overlap-based method performed well but varied by model and metric. Human evaluation also identified contamination patterns across several datasets and partitions.
- 13/14 for GPT-4 and 9/14 for GPT-3.5 were Algorithm 1’s best success rates across the evaluated settings.Algorithm 1 compares average overlap scores from guided and general instruction outputs.
- 7/14 was Algorithm 1’s success rate with ROUGE-L on GPT-3.5 outputs, indicating chance-level performance in that setting.Its success rate also varied between BLEURT and ROUGE-L.
- 14/14 and 13/14 success rates were achieved by Algorithm 2 for GPT-4 and GPT-3.5, respectively, matching human evaluations across settings.These accuracies exceeded all results from Algorithm 1 and remained consistent across both LLMs.
- 0/14 and 11/14 partitions were correctly classified under strict ChatGPT-Cheat? evaluation for GPT-4 and GPT-3.5, respectively.Under lenient evaluation, the method reached 9/14 and 13/14, but that interpretation was considered unrealistic because it overfit the suspicious label.
- GPT-4 contained both train and test/validation partitions of AG News and WNLI, while contamination affected selected partitions of IMDB, RTE, and XSum.For GPT-3.5, human evaluation found exposure only for the XSum test partition.
6 CONCLUSION
The paper presents a contamination-detection method that operates without pre-training-data access and combines instance-level guided completion with partition-level rules. Across broad evaluations, GPT-4 few-shot ICL most closely matched human judgments, while the method cannot distinguish contamination sources or forms.
- Conclusion: The method detects LLM data contamination without access to pre-training data by extending instance-level guided completions into partition-level decisions.Partition decisions use BLEURT and ROUGE-L comparisons or GPT-4 few-shot ICL judgments.
- Conclusion: 28 settings covered seven datasets, their train and test/validation partitions, and GPT-4 and GPT-3.5.The evaluations used 10 randomly drawn instances from each split.
- Conclusion: 14/14 scenarios for GPT-4 and 13/14 for GPT-3.5 were correctly identified by the GPT-4 few-shot ICL approach against human judgments.This was the most accurate approach among those evaluated.
- Limitations: The best-performing guided-instruction GPT-4 ICL method treats exact and near-exact replicas alike and does not identify their contamination sources.The authors call for methods that detect contamination while identifying its sources and forms.
Appendices
The appendix lists the guided and general instructions used in the experiments, including placeholders for dataset, partition, instance prefix, and label information.
- A LIST OF ALL GUIDED AND GENERAL INSTRUCTIONS: Table 5 collects the guided and general instructions used throughout the study.The appendix presents these instructions as a comprehensive list.
- A LIST OF ALL GUIDED AND GENERAL INSTRUCTIONS: The instruction templates include a placeholder for the dataset name.The extracted template identifies this as {dataset name}.
- A LIST OF ALL GUIDED AND GENERAL INSTRUCTIONS: Together, these placeholders specify the dataset, split, partial instance, and label components used to construct the prompts.The appendix thereby documents the inputs used for guided and general instruction variants.
- A LIST OF ALL GUIDED AND GENERAL INSTRUCTIONS: The templates include a placeholder for the partition or split name.This identifies whether the prompt refers to a train, test, or validation partition.
- A LIST OF ALL GUIDED AND GENERAL INSTRUCTIONS: The templates include a placeholder for the first part of the dataset instance.The instance prefix is cut at a random tail position or uses the whole first sentence for NLI datasets.
- A LIST OF ALL GUIDED AND GENERAL INSTRUCTIONS: The templates include a placeholder for the corresponding label of an incomplete input instance.Labels are included when relevant to the dataset and task.
B IMPACT OF LABEL INTEGRATION ON DOWNSTREAM COMPLETION
The appendix examines how labels affect guided completions by comparing correct and incorrect labels for the same reference prefix. Correct labels yield exact continuation matches in the reported examples, whereas incorrect labels yield inexact matches.
- B IMPACT OF LABEL INTEGRATION ON DOWNSTREAM COMPLETION: Incorrect labels produce inexact matches under the same initial segment.This contrast is described as reducing false positives in downstream completions.
- B IMPACT OF LABEL INTEGRATION ON DOWNSTREAM COMPLETION: Table 6 compares guided completions produced from the same reference prefix paired with correct versus incorrect labels.All reported completions use GPT-4 as the underlying model.
- B IMPACT OF LABEL INTEGRATION ON DOWNSTREAM COMPLETION: Correct labels produce exact matches for the subsequent reference segment in every reported example.The comparison is presented as evidence for the effect of label integration on downstream completion.
- B IMPACT OF LABEL INTEGRATION ON DOWNSTREAM COMPLETION: GPT-4 few-shot ICL uses one exact-match example and three human-labeled near-exact examples to classify candidate-reference matches.The prompt helps GPT-4 distinguish near-exact from inexact matches in line with human assessment.
- B IMPACT OF LABEL INTEGRATION ON DOWNSTREAM COMPLETION: Tables 7 and 8 pair exactness judgments and ROUGE-L/BLEURT metrics for guided completions with corresponding general-instruction outputs.The tables include human and GPT-4 few-shot ICL evaluations.
E STATISTICAL ANALYSIS: BOOTSTRAP RESAMPLING
The statistical analysis tests whether guided instruction produces completions closer to reference instances than general instruction. It uses bootstrap resampling and evaluates differences with BLEURT and ROUGE-L.
- E STATISTICAL ANALYSIS: BOOTSTRAP RESAMPLING: Bootstrap resampling with 10,000 samples tests differences between guided and general instruction results.The analysis follows the alternative hypothesis that guided completions are closer to reference instances.
- E STATISTICAL ANALYSIS: BOOTSTRAP RESAMPLING: The alternative hypothesis is that guided instruction yields outcomes closer to reference instances than general instruction.Closeness is evaluated through fluency, quality, and similarity.
- E STATISTICAL ANALYSIS: BOOTSTRAP RESAMPLING: BLEURT and ROUGE-L are the performance metrics used to compare guided and general completions.These metrics quantify the comparison underlying the statistical analysis.
- E STATISTICAL ANALYSIS: BOOTSTRAP RESAMPLING: Table 7 reports exact, near-exact, and inexact examples with BLEURT, ROUGE-L, GPT-4 few-shot ICL, and human judgments, while Table 8 reports general-instruction completions.The paired tables support comparison of guided and general outputs.
- E STATISTICAL ANALYSIS: BOOTSTRAP RESAMPLING: p-values ≤ 0.05 indicate statistically significant differences between guided and general BLEURT and ROUGE-L scores.Table 9 presents the p-values computed from the bootstrap procedure.
F CONTINUED TRAINING OF GPT-3.5 BASE MODEL FOR INTENTIONAL CONTAMINATION
The validation study intentionally continued training GPT-3.5 on selected dataset samples to reproduce potential pre-training exposure under constrained resources. It used balanced random samples from AG News, RTE, and XSum while retaining OpenAI’s default training hyperparameters.
- Training setup: The procedure used GPT-3.5 base model snapshot gpt-3.5-turbo-0613 and submitted continued-training jobs through the OpenAI API.The objective was to reproduce potential exposure during pre-training rather than conventional fine-tuning.
- Data formatting: Dataset instances were formatted with dataset names and split identifiers as metadata, without adding specific instructions to the training data.The authors distinguish this procedure from instruction tuning.
- Controlled contamination study: GPT-3.5 was intentionally continued-trained on 100 balanced random samples from each selected training partition.The study used AG News, RTE, and XSum, with samples evenly distributed by label.
- Dataset selection: The validation study selected one dataset per task: AG News, RTE, and XSum.This selection reflected the project’s budget limitations and focus on a manageable number of training samples.
- Training setup: OpenAI’s default hyperparameters were retained during the continued-training phase.This kept the validation setup aligned with the provider’s default configuration.
G EXAMPLES OF REPLICAS GENERATED PRE AND POST CONTAMINATION OF GPT-3.5
The paper illustrates intentional contamination through exact replicas from GPT-3.5 and a near-exact GSM8k reproduction from GPT-4. These examples are generated using guided instructions and contrast contaminated outputs with inexact base-model completions.
- Replica examples: Exact replicas were produced by contaminated GPT-3.5 checkpoints after training on AG News and XSum subsets.The examples come from contaminated versions of the GPT-3.5 base model and are generated with guided instructions.
- Replica examples: A GPT-4 completion nearly exactly reproduced an instance from the GSM8k training set.The example demonstrates that near-exact matches can occur alongside exact replicas in controlled contamination analyses.
- Contamination formats: Table 10 lists the data formats used to contaminate GPT-3.5, with separate templates applied to instances according to dataset task.The templates include placeholders for split names, dataset names, instances, text fields, and labels where applicable.
H DETAILED DESCRIPTION OF DATASETS
The paper evaluates contamination across classification, entailment, summarization, and mathematical reasoning datasets with varied sizes, formats, and partition structures. The described datasets include IMDB, AG News, Yelp, RTE, WNLI, SAMSum, XSum, and GSM8k.
- IMDB Movie Reviews Dataset: The IMDB Movie Reviews dataset contains 50,000 balanced reviews split evenly between training and testing for sentiment analysis.Positive reviews score at least 7 out of 10, while negative reviews score at most 4.
- AG News Dataset: AG News contains four news categories and provides 30,000 training and 1,900 test samples per class.Its categories are world, sports, business, and science/technology.
- Yelp Dataset: Yelp supports star-rating and polarity classification, with separate training and testing sample counts for each task.The polarity version contains 280,000 training samples and 19,000 test samples per polarity category.
- Recognizing Textual Entailment Dataset: RTE combines four textual-entailment challenge editions into a two-class dataset with 2,490 training, 277 validation, and 3,000 testing examples.Neutral and contradiction examples are combined into the not entailment class.
- Additional datasets: The paper also includes WNLI, SAMSum, XSum, and GSM8k, spanning coreference inference, dialogue summarization, abstractive summarization, and grade-school mathematics.Table 11 specifically presents replica examples involving GSM8k alongside contaminated AG News and XSum examples.