Source-linked AI summary
Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, Ge Li
TL;DR
Open-access benchmark results may reflect memorization rather than genuine generalization when test data enters LLM training, undermining trustworthy evaluation. The paper proposes output-distribution methods and corresponding datasets, finding that CDD detects contamination and TED mitigates its evaluation impact across scenarios.
Problem
Because LLMs may have seen benchmark data during training, their apparent task performance may reflect memorization rather than genuine generalization, potentially overestimating capabilities and concealing model flaws.
Method
The paper proposes CDD, which detects contamination from the peakedness of distributions over sampled texts, and TED, alongside DETCON and COMIEVAL datasets for detection and mitigation evaluation.
Results
CDD achieves state-of-the-art detection performance and identifies implicit contamination, while TED mitigates contamination effects across various evaluation scenarios; ChatGPT is likely contaminated on HumanEval.
Takeaways & Limitations
The findings call attention to data contamination as a concern for trustworthy LLM evaluation and motivate detecting it before interpreting benchmark performance.
Takeaways & Limitations
The approaches require multiple samplings to compute the output distribution, and more samplings improve their effect.
Abstract
from arXiv · showhide
Recent statements about the impressive capabilities of large language models (LLMs) are usually supported by evaluating on open-access benchmarks. Considering the vast size and wide-ranging sources of LLMs' training data, it could explicitly or implicitly include test data, leading to LLMs being more susceptible to data contamination. However, due to the opacity of training data, the black-box access of models, and the rapid growth of synthetic training data, detecting and mitigating data contamination for LLMs faces significant challenges. In this paper, we propose CDD, which stands for Contamination Detection via output Distribution for LLMs. CDD necessitates only the sampled texts to detect data contamination, by identifying the peakedness of LLM's output distribution. To mitigate the impact of data contamination in evaluation, we also present TED: Trustworthy Evaluation via output Distribution, based on the correction of LLM's output distribution. To facilitate this study, we introduce two benchmarks, i.e., DetCon and ComiEval, for data contamination detection and contamination mitigation evaluation tasks. Extensive experimental results show that CDD achieves the average relative improvements of 21.8\%-30.2\% over other contamination detection approaches in terms of Accuracy, F1 Score, and AUC metrics, and can effectively detect implicit contamination. TED substantially mitigates performance improvements up to 66.9\% attributed to data contamination across various contamination setups. In real-world applications, we reveal that ChatGPT exhibits a high potential to suffer from data contamination on HumanEval benchmark.
1 Introduction
LLM benchmark performance may reflect memorization from data contamination rather than genuine generalization, threatening trustworthy evaluation. The paper proposes output-distribution methods and benchmarks to detect contamination and mitigate its effects.
- Data contamination occurs when test data enters training data, causing exceptionally strong performance on leaked examples.
- Existing and upcoming benchmarks are difficult to protect because training data contains reused text, code, synthetic data, and unknown evolving sources.
- Contamination can inflate performance on leaked data while stagnating or degrading performance on similar unleaked data, obscuring model flaws.
- Detection remains challenging because training data is opaque, models may be black boxes, and synthetic data can introduce implicit contamination.
- CDD detects contamination from sampled texts by identifying peaked output distributions, while TED corrects output distributions to mitigate contaminated evaluation.
- The study introduces DETCON and COMIEVAL and reports that CDD detects implicit contamination, TED mitigates contamination effects, and ChatGPT likely suffers contamination on HumanEval.
2 Motivation Example
The paper motivates contamination detection through the contrast between diverse outputs from generalized models and highly similar outputs caused by memorization. Edit-distance distributions reveal that even one exposure to leaked data sharply increases exact-output repetition.
- Models that transcend memorization should generate diverse responses because many tokens have analogous semantics.
- Consistently identical or highly similar sampled texts are presented as evidence that an LLM likely memorized the data.
- The paper models output distributions using edit distances between sampled texts to measure repetition under contamination.
- A 0.0035 density of zero edit distance without contamination rose to more than 20 times that value after one exposure to leaked data.
- The proposed analysis uses sampled texts alone, without access to output probabilities or training data.
3 Methodology
The methodology models LLM output distributions from sampled texts and uses their peakedness to detect contamination. TED then corrects these distributions by excluding peaked outputs and removing duplicates before evaluation.
- 3.1 Edit Distance Distribution: CDD models an LLM’s output distribution by computing pairwise token-level edit distances among sampled texts.Token-level edit distance measures the minimum token insertions, deletions, or substitutions needed to transform one string into another.
- 3.2 CDD for Data Contamination Detection: CDD detects contamination by identifying peakedness in the edit-distance distribution of samples for a test prompt.The method samples texts from the model, computes the distribution, and applies a thresholded peakedness criterion to infer contamination status.
- 3.2 CDD for Data Contamination Detection: CDD’s default distribution replaces the reference answer with the model’s greedy output to account for implicit contamination in variant answers.The greedy output is obtained by setting sampling temperature t = 0; the method notes that a reference-based distribution assumes explicit leakage of the original answer.
- 3.3 TED for Contamination Mitigation Evaluation: TED mitigates contamination during evaluation by excluding peaked outputs and removing duplicate samples from the evaluation distribution.The correction excludes the greedy text, which is considered likely to represent memorized leaked data, and removes duplicate sampling results.
- 3.3 TED for Contamination Mitigation Evaluation: TED applies the corrected distribution to obtain an evaluation metric for contaminated model outputs.The evaluation procedure samples outputs, excludes peakedness, removes duplicates, and computes the resulting metric.
4 Experiment
The experiments evaluate CDD for contamination detection and TED for contamination mitigation across simulated scenarios and real-world ChatGPT applications. CDD performs strongly across domains and contamination forms, while TED suppresses contamination-driven performance gains with limited effect on uncontaminated models.
- Data Contamination Detection: CDD achieves state-of-the-art performance across code generation and logic reasoning, improving over other approaches by 21.8%-30.2% on Accuracy, F1 Score, and AUC.CDD requires only sampled model texts, without the additional conditions used by some baselines.
- Data Contamination Detection: CDD outperforms competing approaches at lower explicit-contamination degrees and maintains robust performance on implicit contamination, where other methods have significant limitations.Detection becomes easier for all approaches as explicit contamination increases, but CDD remains the robust method for implicit contamination.
- Data Contamination Detection: CDD's α and ξ settings still leave room for improvement, based on empirical hyper-parameter analyses on DETCON.The experiments fix α and ξ intuitively while varying one parameter at a time.
- Contamination Mitigation Evaluation: TED steadily mitigates contamination-related performance improvements across learning rates, base models, mixing ratios, contamination forms, and occurrences, with little effect at zero contamination.Its influence becomes more apparent as contamination increases.
- Contamination Mitigation Evaluation: The exclude-peakedness rule provides TED's main function, while remove-duplicates also contributes beneficially.Both components improve TED in the ablation study.
- Contamination Mitigation Evaluation: Increasing TED's τ more strongly suppresses contamination-attributed performance improvements but marginally decreases performance on uncontaminated models.This exposes a mitigation trade-off controlled by TED's hyper-parameter.
- Real-World Applications: ChatGPT shows high Avg. Peak and Leak Ratio values on HumanEval, suggesting a high likelihood of contamination that becomes more serious in the later version.The hypothesis is further examined using HumanEval_R and CodeForces2305.
5 Related Work
Prior work detects contamination through overlap, likelihood, similarity, or newly constructed evaluations, but these approaches face limitations from opaque training data and semantically equivalent rephrasing. This paper instead uses sampled-text output distributions for detection and contamination mitigation evaluation.
- Data Contamination Detection: GPT-3 used 13-gram overlap detection to remove training data conflicting with benchmark test sets.This approach addressed overlap between the pre-training corpus and evaluation benchmarks.
- Data Contamination Detection: Because most LLMs do not disclose their pretraining corpora, recent methods detect contamination without direct access to training data.Examples include Min-k% Prob, perplexity-based detection, and embedding or LLM-based similarity methods.
- Data Contamination Detection: N-gram filtering may miss semantically equivalent rephrasing, motivating LLM Decontaminator's use of another advanced LLM to measure test-training similarity.The cited investigations specifically question the effectiveness of n-gram filtering for rephrased data.
- Position of This Work: CDD detects contamination from sampled texts by analyzing output distributions and explicitly considers implicit contamination.This differs from methods requiring training-corpus access or additional information about model internals.
- Contamination Mitigation Evaluation: Prior mitigation strategies construct new benchmarks, paraphrase contaminated data, or use temporal information, but each faces practical or contamination-related concerns.The paper positions its output-distribution approach as orthogonal to these strategies.
6 Conclusion
The paper proposes CDD and TED for detecting and mitigating LLM data contamination through output distributions, supported by the DETCON and COMIEVAL datasets. Experiments indicate strong performance for both approaches and suggest ChatGPT may suffer contamination on HumanEval.
- CDD and TED address data contamination detection and mitigation evaluation by considering LLM output distributions.The paper constructs DETCON and COMIEVAL for these two tasks.
- DETCON and COMIEVAL are introduced as corresponding datasets for contamination detection and mitigation evaluation.
- CDD and TED show superiority and versatility across the reported experiments.
- ChatGPT is likely to suffer from data contamination on the HumanEval dataset.
7 Limitations
The paper identifies limitations involving benchmark coverage, sampling cost, fine-tuning choices, and assumptions about uncontaminated base models. It proposes broader validation, faster sampling, full-parameter fine-tuning, and cleaner pretraining data as future directions.
- Validation focuses mainly on code-generation and logical-reasoning benchmarks, leaving other benchmark types for future evaluation.These benchmarks are described as representative and widely adopted.
- Multiple samplings are required to compute the output distribution, and more samplings improve the effect.Parallel sampling is proposed to reduce the resulting time overhead.
- LoRA is used instead of full-parameter fine-tuning to simulate contamination because of limited computational resources.The authors plan to attempt full-parameter fine-tuning in future work.
- The dataset construction assumes the four base LLMs are uncontaminated on selected benchmarks, although slight contamination may exist in reality.Avoiding this issue completely might require retraining from scratch on training data known to exclude test data, which would be prohibitively costly.
A Details of Dataset Construction
The datasets simulate diverse contamination conditions across code-generation and logical-reasoning benchmarks, varying contamination form, mixing proportion, learning rate, and training exposure. DETCON uses contaminated and uncontaminated outputs for detection, while COMIEVAL pairs contaminated models with uncontaminated performance targets.
- The preparation uses HumanEval for code generation, GSM8K for logical reasoning, four corresponding base LLMs, and domain-specific training corpora.Code tasks use CodeLlama-7B and CodeGen-6.7B with StarCoder data; reasoning uses Llama2-7B and Bloom-7B with RedPajama.
- DETCON represents direct LLM outputs on test data as uncontaminated samples and simulates contaminated samples under different scenarios.Explicit contamination directly uses test data, whereas implicit contamination trains on test-data variants.
- Contamination settings vary the test-data proportion across 1:0, 1:0.1k, 1:1k, and 1:10k, and use learning rates 1e-3, 2e-4, and 4e-8.
- Training epochs range from 0 to 20, with zero epochs reserved for uncontaminated samples.
- COMIEVAL contains 560 contaminated-model outputs and uses corresponding uncontaminated-model performance as the evaluation target.The construction combines contamination scenarios and records greedy and sampled outputs.
B Case Study
The case study presents generated code samples for uncontaminated, explicit-contamination, and implicit-contamination scenarios. The examples include repeated or variant solutions to the same array-summing prompt, illustrating how training exposure can produce closely related outputs.
- The case study displays the first 10 samples and marks the greedy-search sample in bold.
- The examples include repeated leaked-data exposure, labeled as occurrence 2.
- The prompt asks for the sum of elements with at most two digits among the first k array elements, with output 24 for a representative input.
- Explicit-contamination examples use the original leaked data and include implementations that directly sum qualifying elements.
- Implicit-contamination examples use variant leaked data and show multiple closely related implementations of the same task.