Source-linked AI summary
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, Ondřej Dušek
TL;DR
Closed-source LLMs obscure training data, leaving contamination concerns inadequately measured and overlooking leakage through user interactions. The paper systematically reviews 255 papers using GPT-3.5 and GPT-4, estimates data exposed under OpenAI’s policy, and examines evaluation practices. It reports ~4.7M samples from 263 benchmarks exposed for potential training and identifies fairness and reproducibility malpractices.
Problem
Limited access to closed-source LLM training data leaves contamination concerns difficult to assess, while existing approaches overlook indirect leakage through user interactions.
Method
The paper systematically analyses 255 papers evaluating GPT-3.5 and GPT-4, considering OpenAI’s data usage policy and reviewing their evaluation protocols.
Results
~4.7M samples from 263 benchmarks were exposed to GPT-3.5 and GPT-4 in ways that could support further training, alongside documented fairness and reproducibility malpractices.
Takeaways & Limitations
The authors propose suggested practices for evaluating closed-source LLMs and release a collaborative repository where researchers can contribute additional known leaks.
Takeaways & Limitations
The reviewed dataset list may omit test-set exposures from experiments that were never published.
Abstract
from arXiv · showhide
Natural Language Processing (NLP) research is increasingly focusing on the use of Large Language Models (LLMs), with some of the most popular ones being either fully or partially closed-source. The lack of access to model details, especially regarding training data, has repeatedly raised concerns about data contamination among researchers. Several attempts have been made to address this issue, but they are limited to anecdotal evidence and trial and error. Additionally, they overlook the problem of \emph{indirect} data leaking, where models are iteratively improved by using data coming from users. In this work, we conduct the first systematic analysis of work using OpenAI's GPT-3.5 and GPT-4, the most prominently used LLMs today, in the context of data contamination. By analysing 255 papers and considering OpenAI's data usage policy, we extensively document the amount of data leaked to these models during the first year after the model's release. We report that these models have been globally exposed to $\sim$4.7M samples from 263 benchmarks. At the same time, we document a number of evaluation malpractices emerging in the reviewed papers, such as unfair or missing baseline comparisons and reproducibility issues. We release our results as a collaborative project on https://leak-llm.github.io/, where other researchers can contribute to our efforts.
1 Introduction
Closed-source LLMs make training-data contamination difficult to assess, including indirect leakage through user interactions. This review examines 255 papers using GPT-3.5 and GPT-4, quantifies leaked benchmark data, documents evaluation malpractices, and proposes practices for fairer, more reproducible evaluation.
- Motivation: Closed-source models conceal training-data details, undermining confidence in performance evaluations when benchmark test data may have been included during training.The problem is especially serious when these models are widely used and evaluated in research.
- Motivation: Indirect data leaking occurs when benchmark data submitted through user interactions becomes available for later model updates, even if initial training was contamination-free.Existing detection approaches often focus on extreme memorization and can miss this iterative pathway.
- Findings: ~4.7M benchmark samples across 263 benchmarks were exposed to GPT-3.5 and GPT-4 through reviewed research practices.Approximately 42% of reviewed papers leaked data in a way that could support further model training.
- Findings: The review identifies evaluation malpractices that limit fairness and reproducibility, including missing comparisons and data leakage that could have been avoided.The authors also document these practices and formulate suggested evaluation practices for closed-source LLMs.
- Dissemination: The results are released as a collaborative repository listing datasets and reported leakage, with invitations for researchers to add further known leaks.The repository is available at leak-llm.github.io.
2 Prior Work on LLM Data Contamination
Prior contamination work mainly backtracks training data or tests whether models reproduce benchmark examples. These approaches are limited because closed-source models can be updated through user interactions and may avoid verbatim overlap without being free from contamination.
- Existing approaches: Closed-source models make contamination difficult to assess, so only a few practical investigation approaches have been proposed.The related work mainly approaches contamination by backtracking models’ training data.
- Existing approaches: The LM Contamination Index estimates contamination by prompting models to generate instances from specified datasets, splits, and formats.Its premise is that models should not reproduce specific benchmark formats without prior exposure.
- Existing approaches: Other methods prompt models to complete benchmark sentences and compare generated completions with references using text-overlap metrics and statistical tests.These methods assess contamination through similarity to known benchmark material.
- Limitations: Current detection methods are limited because decoding filters may prevent models from generating text that significantly overlaps with training data.A failure to reproduce benchmark text therefore does not necessarily exclude contamination.
- Limitations: Even regular adjustment on leaked data can compromise fair comparisons without producing near-complete memorization of benchmark samples.Existing approaches primarily detect the most extreme overfitting and memorization cases.
3 The Issue of Indirect Data Leaking
Indirect data leaking occurs when user interactions supply benchmark-related data to iteratively updated LLMs, even when models have not seen the original datasets. This complicates contamination tracing and may affect fair evaluation.
- 3 The Issue of Indirect Data Leaking: Indirect data leaking occurs when repeated model updates use data supplied through user interactions.The paper treats this as distinct from backtracking scraped training data.
- 3 The Issue of Indirect Data Leaking: User-provided evaluation data may include prompts, textual alterations, or truncated benchmark samples, making contamination harder to inspect.These transformations can prevent leaked data from matching benchmark samples exactly.
- 3 The Issue of Indirect Data Leaking: User instructions can function as gold-standard training data for continued model improvement, even without explicit target labels.The paper states that such updates are likely more effective than training on plain in-domain text.
- 3 The Issue of Indirect Data Leaking: Researchers may leak test-set samples by feeding them to closed-source LLMs during evaluation, with or without labels and surrounded by instructions.Robustness evaluations may additionally perturb the samples.
- 3 The Issue of Indirect Data Leaking: ChatGPT performance improved after model updates on benchmarks to which it had previously been exposed.This observation motivates quantifying data obtained through model use.
4 Methodology
The study systematically reviews evaluations of GPT-3.5 and GPT-4, using OpenAI’s data-usage policy to identify potentially trainable leaks and tracking practices relevant to fairness and reproducibility. It combines literature searches, paper-level inspection, quantitative leakage estimates, and qualitative analysis of evaluation practices.
- 4 Methodology: The review follows a systematic-review protocol and focuses on GPT-3.5 and GPT-4 evaluations in recent NLP research.The authors organize the work into five macro-steps.
- 4 Methodology: The study asks which datasets were leaked to GPT-3.5 and GPT-4 and whether evaluations fairly compare these models with existing baselines.The second question includes the fairness of baseline comparisons.
- 4 Methodology: The search covers online databases, major NLP conference proceedings, peer-reviewed papers, and preprints published from late November 2022 to early October 2023.Queries target terms related to ChatGPT, GPT models, evaluation, and large language models across multiple fields.
- 4 Methodology: Only web-interface interactions are counted as leaks because OpenAI’s policy distinguishes them from API and business-service data.The review found no evidence that surveyed papers opted out of model-improvement data use.
- 4 Methodology: The authors inspect datasets, splits, sample counts, prompts, repositories, peer review, baseline comparisons, evaluation scale, and reported model versions.When sampling or split information is absent, they assume the whole dataset was used.
- 4 Methodology: The review reports quantitative leakage estimates and qualitative evaluation problems concerning objectivity and reproducibility, alongside proposed best practices.GPT-3.5 and GPT-4 leakage is treated as effectively shared and referred to collectively as ChatGPT.
5 Results
The review found substantial indirect data leakage to ChatGPT alongside recurring reproducibility and fairness problems in evaluation studies. These results were identified across 212 relevant papers, with many papers using the web interface and reporting incomplete experimental details or unequal comparisons.
- Review scope: 212 of 255 identified papers were relevant, including 70 peer-reviewed papers and 142 pre-prints.The relevant papers were analysed for data contamination and evaluation practices.
- Indirect data contamination: 90 papers (∼42%) accessed ChatGPT through the web interface, allowing OpenAI to potentially use the submitted data for model improvement.Web-interface access declined after the March 2023 API release, but remained in use through September 2023 and may also have been used during underreported prompt engineering.
- Indirect data contamination: 4.7M samples from 263 unique datasets were reported as leaked to ChatGPT.The analysis quantified leakage by inspecting reported or available dataset usage; when authors did not clarify usage, the study sometimes assumed the entire split had been used.
- Indirect data contamination: ∼93.8% of leaked samples came from whole datasets, compared with ∼5.6% from test or development sets and ∼0.6% from training sets.The authors state that these samples were enriched with instructions that could function as de-facto novel gold-standard data in some cases.
- Evaluation practices: Reproducibility and fairness were frequently limited: 192 works (∼91%) reported prompts, but only 113 (∼53%) provided usable repositories, while comparisons were often missing or unequal.Missing comparisons affected 71 (∼50%) pre-prints and 30 (∼43%) published papers; unequal-sample comparisons affected 54 (∼38%) pre-prints and 34 (∼49%) peer-reviewed papers.
6 Suggested Practices in Closed-source LLM Evaluation
The paper recommends evaluation practices that reduce data leakage and improve fairness and reproducibility when assessing closed-source LLMs. These include following data policies, reporting leakage, using fair comparisons, and documenting experimental details.
- Access and leakage: Researchers should read current data policies and choose access methods accordingly to avoid leaking evaluation data.The paper gives API access instead of a web interface as an example for OpenAI models, while acknowledging budget and usability constraints.
- Interpretation: Closed-source LLM performance should be interpreted cautiously because the review documented over 4 million leaked samples across more than 200 NLP datasets.The paper links the lack of system and training details to uncertainty around impressive zero-shot performance.
- Model choice: The paper strongly encourages using available open-source LLMs when possible because research conducted solely on closed models limits scientific progress.It also notes that claims of consistent proprietary-model superiority may be driven by hype and that evidence exists in the opposite direction.
- Fair comparison: Comparisons should use comparable evaluation scales, apply sampling consistently, and avoid copying state-of-the-art values from prior work.Evaluating open models on much larger sets than proprietary models is described as scientifically dubious.
- Reproducibility: Researchers should disclose prompts, model versions, experiment periods, sampled data, and other setup details needed for reproducibility.The paper emphasizes prompt sensitivity and recommends releasing sampled data when possible.
- Access and leakage: Indirect data leaking should be reported because identifying affected benchmarks supports more objective evaluation of proprietary LLMs.The authors invite researchers to contribute additional known leaks to their collaborative repository.
7 Conclusion and Future Work
The paper analyzes 255 papers evaluating GPT-3.5 and GPT-4, documenting indirect contamination and evaluation problems. It proposes practices for evaluating closed-source LLMs and identifies API experiments and other proprietary models as future research directions.
- Conclusion: 255 papers evaluating GPT-3.5 and GPT-4 were analyzed for indirect data contamination and evaluation practices.The study focuses on data exposed in ways that could permit training by OpenAI.
- Conclusion: 4.7M samples from 263 distinct datasets were exposed to GPT-3.5 and GPT-4 in ways that could permit training.The authors also report concerning reproducibility and fairness practices.
- Future work: Future work will test leaked test data through the OpenAI API and investigate indirect leakage in Anthropic and Cohere models.The proposed experiments target performance on leaked datasets and the tasks generally.
Limitations
The authors qualify their contamination list and review findings because they rely on publicly revealed reports, focus mainly on ChatGPT and GPT-4, and may have missed or misclassified work. The review snapshot and unresolved author clarifications further limit completeness.
- Coverage: The contaminated-dataset list is not fully conclusive because unpublished experiments may have leaked additional test data.The review covers information publicly revealed in articles and therefore cannot capture undisclosed experiments.
- Scope: The review focuses on works using ChatGPT or GPT-4, while earlier OpenAI API policies also allowed data to improve models.This creates a scope boundary for possible GPT-3 API contamination before March 1, 2023.
- Coverage: The authors may have missed some ChatGPT performance studies because the relevant literature was vast despite their best efforts.
- Review snapshot: Publication status was recorded in early October 2023 and could change as reviewed works were updated or published.The authors note that some relevant proceedings had not yet been released, preventing consistent update checks.
- Pre-training scope: Datasets released before 2021 may have been fully leaked through model pre-training data.
- Uncertainty: Two authors did not respond to clarification requests, so some review conclusions rely on best-judgment assumptions that may be wrong.
A Full list of the reviewed work
This appendix lists the works reviewed and classified as relevant, spanning studies that evaluate or use ChatGPT and other large language models across many NLP and application settings.
- Reviewed work: The appendix provides the full list of works reviewed and classified as relevant.
B Detail on evaluation malpractices
Because the Sankey diagrams provide limited insight into evaluation reproducibility and fairness, the paper adds concrete numerical assessments in Tables 2 and 3.
- Sankey diagrams offer limited insight into evaluation reproducibility and fairness.
- Table 2 provides concrete numbers for the reproducibility assessment.
- Table 3 provides concrete numbers for the evaluation malpractices assessment.
C Detailed List of ChatGPT Data Leak
Tables 4 and 5 list leaked datasets by task and leakage severity, while Table 6 continues the dataset listing.
- Tables 4 and 5 list datasets leaked to ChatGPT.
- Table 4 categorizes dataset names by task and leakage severity across three listed portions.
- Table 5 continues the task-based dataset listing in the second portion.
- Table 6 provides the third portion of the task-categorized dataset list.