Source-linked AI summary
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, Eneko Agirre
TL;DR
NLP benchmark evaluation is threatened when LLMs have encountered evaluation data, because contamination can overestimate performance and compromise scientific conclusions. The paper defines contamination levels and proposes measuring exposure, maintaining a case registry, and addressing affected work through research practice and peer review. It also presents memorization-based demonstrations and an extractability-based contamination measure, while acknowledging that the solutions are early-stage and that non-memorization may not establish non-exposure.
Problem
Web-scale and third-party LLM training make it difficult to determine whether evaluation benchmarks were exposed, threatening strict test-data separation.
Method
The paper defines contamination levels and proposes detection measures, a contamination registry, collaborative auditing, and mechanisms for handling affected publications.
Results
The paper reports that contamination can overestimate benchmark performance and compromise scientific claims, and demonstrates memorization of CoNLL03 instances by three LLMs.
Takeaways & Limitations
NLP evaluations should measure and document benchmark exposure and address contamination during authorship, peer review, and post-publication review.
Takeaways & Limitations
The proposed solutions are early-stage and challenging, and lack of memorization does not yet establish that an LLM was not trained on a benchmark.
Abstract
from arXiv · showhide
In this position paper, we argue that the classical evaluation on Natural Language Processing (NLP) tasks using annotated benchmarks is in trouble. The worst kind of data contamination happens when a Large Language Model (LLM) is trained on the test split of a benchmark, and then evaluated in the same benchmark. The extent of the problem is unknown, as it is not straightforward to measure. Contamination causes an overestimation of the performance of a contaminated model in a target benchmark and associated task with respect to their non-contaminated counterparts. The consequences can be very harmful, with wrong scientific conclusions being published while other correct ones are discarded. This position paper defines different levels of data contamination and argues for a community effort, including the development of automatic and semi-automatic measures to detect when data from a benchmark was exposed to a model, and suggestions for flagging papers with conclusions that are compromised by data contamination.
1 Introduction
NLP evaluation depends on strict separation of test data, but web-scale pre-training makes benchmark exposure difficult to determine. The paper proposes measuring and documenting contamination through community tools and review mechanisms.
- Strictly unseen test data is essential for assessing models in zero-shot, few-shot, and supervised settings.
- Web-scale crawling makes it difficult to know whether a benchmark entered an LLM’s training data, especially for closed models.
- Contamination overestimates performance on an already processed benchmark and can distort comparisons among LLMs.
- Avoiding datasets with published test labels would reduce contamination but severely shrink the set of available NLP benchmarks.
- The paper proposes a registry documenting contaminated LLMs, benchmarks, and supporting evidence, alongside collaborative and automated detection.
- The paper also calls for conference mechanisms to prevent compromised conclusions and flag previously published work affected by contamination.
2 Background
Benchmark contamination has become harder to detect as pre-training corpora scale and leading LLMs keep their training data secret. Existing reports document contamination in widely used corpora and benchmarks, raising doubts about affected evaluations.
- Direct analysis of training data is increasingly difficult because pre-training corpora are large and closed-model training sources are secret.
- C4 contained test splits from several benchmarks crawled from GitHub, exposing contamination in a corpus used to pre-train models such as T5.
- ChatGPT was reported to contain several compromised benchmarks, including the popular CoNLL2003 dataset.
- Evidence of contamination casts doubt on papers evaluating ChatGPT, GPT-3, and Codex on CoNLL2003.
3 Defining data contamination
The paper defines contamination broadly as a breach of dataset-control protocols and distinguishes guideline, raw-text, and annotation contamination. These levels differ in how benchmark exposure can affect evaluation validity.
- Data contamination is any breach in the strict control of datasets required by the experimental protocol.
- Guideline contamination: Guideline contamination occurs when annotation instructions and examples for a dataset are seen by the model.
- Raw text contamination: Raw text contamination occurs when the original unannotated text is exposed, potentially helping models infer annotations such as entity boundaries.
- Raw text contamination: Web-derived datasets including MultiCoNER, IMDB, and CNN/DailyMail can be affected when their source text appears in pre-training data.
- Annotation contamination: Annotation contamination occurs when benchmark labels are exposed during training.
- Annotation contamination: Exposure of the evaluation split completely invalidates the experiment, while train or development exposure invalidates zero-shot and few-shot claims.
4 Contamination on different steps
Contamination can enter during pre-training, supervised fine-tuning, or post-deployment improvement. The paper emphasizes that large-scale web data is difficult to filter, while machine-generated fine-tuning data and service-based exposure create additional risks.
- The LLM training pipeline includes pre-training, instruction or alignment fine-tuning, and iterative post-deployment improvement, all of which can expose models to contamination.
- Pre-training: Pre-training web corpora are difficult to filter and deduplicate completely, making total avoidance of contamination unrealistic.
- Pre-training: Standard benchmarks such as GLUE and SuperGLUE should at least be excluded from the training data of LLM developers.
- Supervised fine-tuning: Supervised or instruction fine-tuning can introduce contamination, although documentation requirements make it less frequent in research practice.
- Supervised fine-tuning: Machine-generated datasets used for fine-tuning can distill contamination from closed teacher models into public white-box models.
- Post-deployment improvement: LLM service products can become aware of testing data and recreate benchmark datasets after deployment.
5 Measuring data contamination
The paper proposes measuring and documenting contamination differently for open and closed models, combining data auditing, memorization-based testing, and automatic extractability metrics. It also identifies unresolved uncertainty about whether non-memorization proves that a benchmark was not used for training.
- For open models, contamination can be documented by examining publicly available pre-training, instruction-tuning, and deployment data.
- String matching and benchmark-data overlap can measure how much of a benchmark appears in accessible pre-training data.The paper notes that large pre-training datasets make string matching difficult and that no methodology is currently agreed upon.
- For closed models, the paper proposes using memorization to uncover contamination when training data are unavailable.Auditors may need to judge manually whether generated outputs support contamination.
- An example is extractability: an example is extractable when a model generates it after being prompted with k immediately preceding dataset examples.The degree of contamination is defined as the ratio of extractable examples to all examples in the dataset.
- Whether a benchmark’s lack of memorization ensures that it was not used for training remains unresolved and requires further research.The paper presents the possibility that non-memorization may still correlate with performance after training exposure as speculation.
6 Call for action
The paper calls for coordinated community action to detect, document, prevent, and review benchmark contamination. It presents these measures as important for improving future NLP evaluations.
- The community should develop automatic or semi-automatic measures for detecting when benchmark data were exposed to a model.
- Researchers should build a registry of contamination cases that records supporting evidence.
- Authors should use these tools to reduce contamination in experimental protocols, while peer review should address contamination issues.
- Published papers with relevant contamination evidence should have mechanisms for flagging the affected work and explaining how contamination compromised it.
- Community workshops are presented as a way to develop ideas that may influence future NLP evaluations.
7 Limitations
The paper focuses on data contamination in evaluations of LLMs on standard academic benchmarks, while acknowledging that other evaluation problems are outside its scope. Its proposed solutions are early-stage and challenging.
- The paper addresses contamination in LLM evaluations on standard academic benchmarks, not all possible problems in current evaluations.
- Its proposed solutions are early-stage, difficult to implement, and presented as topics for further discussion and research.
A Empirical demonstrations of contamination
The appendix demonstrates that three models can reproduce the beginning of the CoNLL03 training split, providing empirical examples of memorized benchmark data. Figure 1 specifically presents contamination on ChatGPT through a prompt and its completion.
- WizardCoder, ChatGPT, and GitHub Copilot can perfectly regenerate the first lines of the CoNLL03 training split.The paper relates this to the dataset having been uploaded repeatedly to GitHub, where all three models were trained.
- The demonstration prompts a model to generate the first CoNLL-2003 training instances in BIO format.
- The reproduced output begins with the CoNLL03 document marker and includes token, part-of-speech, chunk, and named-entity labels.
- Figure 1 depicts ChatGPT contamination by coloring the given prompt and showing the completion in black, with the output shortened.
A.1 Data contamination reported by other works
Prior work has documented data contamination in major language models and benchmarks, including benchmark test splits appearing in pretraining corpora and contamination during GPT-3 training. Empirical examples also show models reproducing CoNLL03 training data.
- WizardCoder, ChatGPT, and GitHub Copilot can perfectly reproduce the first lines of the CoNLL03 training split, which had been uploaded several times to GitHub.