Source-linked AI summary
Don't Make Your LLM an Evaluation Benchmark Cheater
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, Jiawei Han
TL;DR
Benchmark leakage threatens fair and reliable LLM evaluation because training data may overlap with benchmark test data and is difficult to inspect. The paper empirically studies this risk, finds leakage can substantially inflate benchmark results, and proposes safeguards for developers and maintainers.
Problem
Benchmark leakage can occur when LLMs are trained on data relevant or identical to benchmark test data, while training-corpus secrecy makes contamination difficult to examine during evaluation.
Method
The paper conducts empirical studies by incorporating selected benchmark training data into LLM training and evaluating performance on leaked benchmarks and unrelated zero-shot tasks.
Results
Benchmark leakage can largely boost evaluation results, allowing smaller models to outperform models ten times larger on certain tasks and potentially reducing performance on unrelated tasks.
Takeaways & Limitations
LLM developers and benchmark maintainers should avoid leakage, report contamination analyses, use multiple prompts, and interpret leaderboard results cautiously.
Takeaways & Limitations
The study does not fully investigate leakage introduced during pre-training, where the much larger pre-training dataset may yield different findings.
Abstract
from arXiv · showhide
Large language models~(LLMs) have greatly advanced the frontiers of artificial intelligence, attaining remarkable improvement in model capacity. To assess the model performance, a typical approach is to construct evaluation benchmarks for measuring the ability level of LLMs in different aspects. Despite that a number of high-quality benchmarks have been released, the concerns about the appropriate use of these benchmarks and the fair comparison of different models are increasingly growing. Considering these concerns, in this paper, we discuss the potential risk and impact of inappropriately using evaluation benchmarks and misleadingly interpreting the evaluation results. Specially, we focus on a special issue that would lead to inappropriate evaluation, \ie \emph{benchmark leakage}, referring that the data related to evaluation sets is occasionally used for model training. This phenomenon now becomes more common since pre-training data is often prepared ahead of model test. We conduct extensive experiments to study the effect of benchmark leverage, and find that it can dramatically boost the evaluation results, which would finally lead to an unreliable assessment of model performance. To improve the use of existing evaluation benchmarks, we finally present several guidelines for both LLM developers and benchmark maintainers. We hope this work can draw attention to appropriate training and evaluation of LLMs.
1 Introduction
LLM benchmarks and leaderboards support capability evaluation, but benchmark leakage can make comparisons unfair and results unreliable. The paper examines this risk and proposes evaluation practices for developers and benchmark maintainers.
- Benchmark leakage: Benchmark leakage is difficult to detect because evaluations commonly run through local servers or APIs without strict contamination checks, while training-corpus composition is often undisclosed.These conditions constrain benchmark maintainers’ ability to directly examine contamination.
- Benchmark leakage: Benchmark leakage occurs when test or related benchmark data is included in an LLM’s pre-training corpus, potentially creating an unfair performance advantage.The issue can arise unintentionally because pre-training data may be prepared before future evaluation datasets are known.
- Experimental findings: The paper continually trains four language models ranging from 1.3B to 7B under leakage settings involving benchmark training sets, test prompts, and test sets.It evaluates these models on existing benchmarks and examines effects on other abilities.
- Experimental findings: Benchmark leakage can unfairly boost evaluation performance, allowing a 1.3B model to outperform 10× larger models on certain tasks.The reported boost can therefore distort comparisons between models of substantially different sizes.
- Recommendations: The authors recommend diverse benchmarks, contamination checks and reporting for developers, and diverse prompts plus contamination analyses for benchmark maintainers.These guidelines aim to improve fair and appropriate LLM evaluation.
2 Empirical Study about Benchmark Leakage
The empirical study simulates increasingly severe benchmark leakage across multiple LLMs and evaluation tasks. Leakage inflates benchmark scores, can reverse apparent model-size comparisons, and may undermine fair capability assessment.
- Experimental Setup: The evaluation covers MMLU, open-domain question answering, reasoning, and reading comprehension benchmarks.The study includes 57-task MMLU, seven open-domain QA datasets, commonsense and mathematical reasoning datasets, and English and Chinese reading-comprehension datasets.
- Results and Analysis: Training on MMLU data greatly improves MMLU scores but decreases performance on unrelated tasks such as HellaSwag and GSM8k.Using all benchmark training sets instead produces notable performance increases across almost all evaluated tasks.
- Results and Analysis: Evaluation scores rise as leakage becomes more severe, and leaked test prompts let smaller models surpass much larger nonleaked models.For example, phi-1.5-1.3B+All Train S+Test P outperforms LLaMA-65B on RACE-M and RACE-H, 55.80 versus 53.00 and 52.82 versus 48.00.
- Results and Analysis: In the extreme test-set leakage case, 1.3B models outperform 65B models across most tasks without implying improved capacity.The paper characterizes this increase as benchmark cheating and concludes that benchmark leakage should be strictly avoided.
3 Potential Risk of Benchmark Leakage
The paper investigates how training on leaked benchmark data affects performance on unrelated tasks and later instruction-tuning adaptation. Experiments find degradation beyond inflated benchmark scores, especially when models are trained only on leaked data.
- Experimental design: The study continually pre-trains 1.3B–7B language models on selected benchmark training sets to examine leakage side effects.This extreme setup isolates the potential impact of benchmark leakage and is not intended for real training or evaluation.
- Effect on the Performance of Other Tasks: The evaluation tests zero-shot performance on LAMBADA, XSum, and HumanEval, covering language modeling, summarization, and code synthesis.Metrics are accuracy for LAMBADA, ROUGE-L for XSum, and pass@10 for HumanEval.
- Effect on the Performance of Other Tasks: After leaked-data training, all models degrade on the two text-generation datasets and primarily degrade on HumanEval code synthesis.OpenLLaMA-3B and LLaMA-2-7B obtain ROUGE-L scores of 0.19 and 0.25 on XSum, respectively.
- Effect on Model Adaptation: Models trained with leaked data remain below nonleaked counterparts after Alpaca or CodeAlpaca instruction tuning.The comparison evaluates whether subsequent adaptation can recover performance after leakage-focused training.
4 Discussion
The discussion explains why benchmark leakage undermines zero- and few-shot evaluation and proposes safeguards for benchmark maintainers and LLM developers. Recommended practices include broader evaluation, contamination analysis, prompt diversity, and greater reporting transparency.
- 4.1 Fairness in Evaluating Zero/Few-shot Generalization Ability: Leakage can dramatically boost benchmark results by turning zero- or few-shot generalization into easier in-domain evaluation.This violates the requirement that test-time samples are unobserved during training and can make a benchmark ineffective for comparison.
- 4.1 Fairness in Evaluating Zero/Few-shot Generalization Ability: Fully eliminating leakage is difficult because benchmarks draw on public sources whose dependencies may already appear in pre-training corpora.The paper notes that identifying all dependencies between evaluation benchmarks and pre-training data remains challenging.
- 4.2 Suggestion for LLM Evaluation: Capability evaluation should include basic abilities such as text generation alongside advanced reasoning and factual-knowledge benchmarks.The paper presents this broader coverage as necessary for a thorough estimate of model capabilities.
- Suggestions for LLM developers: LLM developers should decontaminate pre-training data, exclude mainstream benchmark training sets when possible, and disclose contamination risks and pre-training composition.Suggested checks include n-gram hashing and reporting datasets related to mainstream evaluation benchmarks.
- 4.2 Suggestion for LLM Evaluation: Benchmark maintainers should disclose data sources, analyze contamination against mainstream pre-training corpora, and report overlap statistics.Submissions are also encouraged to include a contamination-analysis report from the result provider.
- 4.2 Suggestion for LLM Evaluation: Evaluations should average results across diverse prompts and use broader benchmark sources, with manual or qualitative evaluation when feasible.These practices target prompt sensitivity and inflated results associated with contamination.
5 Conclusion
The conclusion reports that benchmark leakage can make LLM evaluation unfair and untrustworthy, including for small models. It calls for avoiding leakage and applying practical safeguards when using benchmarks and leaderboards.
- Conclusion: Data leakage can largely boost benchmark results, even for small models, making evaluation unfair and untrustworthy.The conclusion frames this as the central empirical finding of the paper.
- Conclusion: Developers and benchmark maintainers should account for contamination when interpreting and using leaderboard results.The paper mentions token-overlap calculations as one heuristic for detecting potential contamination.
- Conclusion: The work aims to focus research attention on appropriate use of existing LLM evaluation benchmarks and further identification of potentially contaminated datasets.The conclusion identifies contamination-dataset alerting as a possible direction for follow-up work.
Limitation
The study has several scope limitations: it uses continual training rather than direct pre-training leakage, omits finer-grained leakage scenarios, and does not quantify benchmark contamination risks.
- The study is described as preliminary and has several limitations.
- The experiments continually trained existing pre-trained LLMs with leaked data rather than incorporating leakage directly during pre-training.Because pre-training datasets are much larger than benchmark data, direct pre-training leakage might produce different findings.
- The study did not examine finer-grained leakage scenarios, including leaking unlabeled training examples or varying the leaked-dataset proportion.
- The authors did not calculate contamination degrees between mainstream benchmarks and commonly used pre-training datasets.Such estimates could help developers adjust evaluation settings, but accurately and efficiently assessing example-level contamination remains challenging.