Source-linked AI summary
Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation
Qin Zhu, Qingyuan Cheng, Runyu Peng, Xiaonan Li, Tengxiao Liu, Ru Peng, Xipeng Qiu, Xuanjing Huang
TL;DR
Benchmark contamination can make benchmark results diverge from practical performance, while new benchmarks are costly and remain vulnerable to leakage. ITD detects and rewrites leaked samples; proof-of-concept experiments report lower accuracy on both GSM8K and MMLU.
Problem
Benchmark leakage can compromise accurate evaluation, while constructing replacement benchmarks is costly and new public benchmarks remain at risk of leakage.
Source: §1 · p. 1Method
ITD detects potentially leaked samples and rewrites them while aiming to preserve their difficulty and the knowledge or calculations being tested.
Source: §1 · p. 2Results
22.9% on GSM8K and 19.0% on MMLU accuracy reductions followed ITD in proof-of-concept experiments; Phi3 and Mistral also declined across both benchmarks in real evaluations.
Source: §1 · p. 2Takeaways & Limitations
Experiments on GSM8K and MMLU suggest ITD can mitigate contamination effects and yield more reliable evaluation results.
Source: §5 · p. 8The rewrite methods have limited effectiveness for models intentionally trained on extensive in-domain data or subjected to extensive rewrites.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
The training process of large language models (LLMs) often involves varying degrees of test data contamination. Although current LLMs are achieving increasingly better performance on various benchmarks, their performance in practical applications does not always match their benchmark results. Leakage of benchmarks can prevent the accurate assessment of LLMs' true performance. However, constructing new benchmarks is costly, labor-intensive and still carries the risk of leakage. Therefore, in this paper, we ask the question, Can we reuse these leaked benchmarks for LLM evaluation? We propose Inference-Time Decontamination (ITD) to address this issue by detecting and rewriting leaked samples without altering their difficulties. ITD can mitigate performance inflation caused by memorizing leaked benchmarks. Our proof-of-concept experiments demonstrate that ITD reduces inflated accuracy by 22.9% on GSM8K and 19.0% on MMLU. On MMLU, using Inference-time Decontamination can lead to a decrease in the results of Phi3 and Mistral by 6.7% and 3.6% respectively. We hope that ITD can provide more truthful evaluation results for large language models.
1 Introduction
Benchmark contamination can undermine evaluation, while replacing leaked benchmarks is costly and leakage-prone. ITD detects and rewrites samples; experiments report reduced performance across GSM8K and MMLU.
1 Introduction
- Public benchmarks face leakage that can undermine evaluation fairness, while constructing replacements is labor-intensive and costly.Source: §1 · pp. 1–2
- Altering a question’s background may expose answer memorization, while models that learned relevant skills can still answer; only the memorization case might be reusable.Source: §1 · p. 2
- ITD screens potentially leaked samples, then rewrites knowledge questions’ phrasing or math-question backgrounds while preserving tested content and intended difficulty.For GSM8K, the original numbers and calculations remain unchanged; for MMLU, the tested knowledge points remain unchanged.Source: §1 · p. 2
- 22.9% on GSM8K and 19.0% on MMLU accuracy reductions followed ITD in proof-of-concept tests; Phi3 and Mistral also declined across both benchmarks in real evaluations.The real-evaluation reductions varied by model and benchmark.Source: §1 · p. 2
- The authors release a rewritten GSM8K dataset and a category-sampled rewritten MMLU dataset for future evaluation work.Source: §1 · p. 2
2 Related Work
Related work covers detection methods for benchmark contamination and alternatives to reusing existing benchmarks. The cited approaches include overlap detection, distributional or sample-level assessment, and dataset replacement or updating.
2 Related Work
- Overlap-based detection uses n-gram analysis or BM25, but large training corpora make n-gram statistics resource-intensive, and mainstream LLM training corpora are mostly inaccessible.Source: §2 · p. 2
- Other detection approaches compare training and test distributions, use benchmark text with black-box model access, or assess generated content and coherent continuations.Source: §2 · pp. 2–3
- Alternative benchmark strategies include temporal datasets, unreleased datasets, and monthly question releases, but they entail leakage exposure or substantial overhead.LatestEval requires continual updates as released datasets become exposed; the passage also notes substantial overhead for these approaches.Source: §2 · p. 3
3 Method
ITD estimates contamination in an evaluation set and uses that estimate to adjust the set before computing model performance. Its three-stage workflow detects potentially contaminated samples, rewrites them, and checks the rewrites again.
3.1 Problem Formulation
- ITD formulates evaluation as estimating an unknown contamination indicator and using it to adjust a potentially contaminated out-of-distribution dataset.The adjusted dataset E′ is used to compute the final evaluation result.Source: §3.1 · p. 3
3.2 Inference-Time Decontamination
- The framework has three stages: Detection, Rewrite, and Assurance, with Assurance repeating detection and rewriting until content passes or reaches the iteration limit.Human evaluations are also used to assess decontaminated data quality.Source: §3.2 · pp. 3–4, Figure 2 · p. 3
- MinKProb approximates the contamination indicator by detecting training-data presence, splitting samples into uncontaminated and potentially contaminated groups.It averages probabilities for the K lowest-probability tokens and applies threshold ϵ to assess whether text appears in pretraining data.Source: §3.2 · pp. 3–4
- Automated rewriting targets detected samples while aiming to preserve challenge level, addressing the cost of skilled human rewriting.The paper explores mathematical-reasoning and knowledge-based tasks.Source: §3.2 · p. 4
- For GSM8K-style math problems, rewrites change scenarios and characters while keeping numbers, computational steps, answers, and difficulty consistent.The authors report that step distributions and numbers are identical to GSM8K in their verification.Source: §3.2 · p. 4
- For MMLU-style knowledge questions, rewrites use synonyms while retaining proper nouns and numbers, avoiding uncommon words to preserve knowledge points and difficulty.The authors note that changing question backgrounds would require extensive knowledge and correctness verification.Source: §3.2 · p. 4
4 Experiment
Across controlled-leakage and real-model evaluations, ITD lowered benchmark accuracy where memorization inflated it; analyses also examined whether rewriting preserved difficulty and answer correctness.
4.1 Setup
- Experiments used GSM8K and MMLU, three models, MinKProb or an all-inputs detector, and up to three rewriting rounds.Source: §4.1 · pp. 4–5
4.2 Proof-of-concept Experiment
- 22.9% on GSM8K and 19.0% on MMLU were the accuracy reductions after ITD in the controlled-leakage experiment.Source: §4.2 · p. 5, Table 2 · p. 6
4.3 Real Model Experiment
- In real-model evaluations, ITD lowered scores in most cases, but unknown training data prevented accurate assessment of contamination, overhead, and accuracy.Source: §4.3 · p. 6, Table 3 · p. 7
4.4 Analysis
- Llama2 showed minimal variation on the same rewritten data, suggesting question difficulty remained unchanged within the margin of error.Source: §4.4 · p. 7, Table 4 · p. 7
- The passage describes more common, straightforward rewrites in MMLU*, and reports that 8.03% of GSM8K rewrites had no actual difficulty change.Source: §4.4 · p. 7
- Answer shifts occurred in 3.9% of GSM8K samples during rewriting, yet the reference answers remained correct.Source: §4.4 · p. 8
- After the first rewrite, accuracy and contamination rate continued to fall through later rounds before leveling off.Source: §4.4 · p. 8, Figure 3 · p. 8
- 51.3% contaminated-data accuracy returned to 30.9% after several rewrites, matching the uncontaminated-data level on GSM8K.Source: Figure 4 · p. 8, §4.4 · p. 8
5 Conclusion
The paper presents inference-time decontamination as a way to reduce contamination’s effects and obtain more realistic model-capability assessments. Experiments on GSM8K and MMLU suggest the method can mitigate contamination without requiring entirely new datasets.
5 Conclusion
- ITD uses detection and iterative rewriting of contaminated data to mitigate contamination effects in model evaluation.The conclusion describes the resulting assessments as more accurate and reliable.Source: §5 · p. 8
- The framework’s detection, rewrite, and assurance stages support consistent, fair assessments without entirely new datasets.The paper describes the approach as providing a realistic view of model capabilities.Source: §5 · pp. 8–9
- The paper presents this work as a foundation for future research on improving language-model evaluations.It identifies advanced detection and rewriting techniques as areas for further exploration.Source: §5 · p. 9
Limitations
The discussion identifies limits in real-model evaluation, difficult contamination scenarios, and potential evaluation biases. It also distinguishes limitations of current detection and rewriting methods from limitations inherent to the ITD setting.
Limitations
- For real models, the correction magnitude is not substantial, and effective evaluation depends on a reliable detector because model–benchmark contamination is unknown.The paper says the specific contamination relationship between a model and benchmark is not known.Source: Limitations · p. 9
- The rewrite methods have limited effectiveness for models trained on substantial in-domain data or exposed to extensive rewrites.The detection stage can still distinguish these models from models without contamination.Source: Limitations · p. 9
- Different contamination levels can produce different evaluation questions, and model-generated rewrites may introduce bias.The paper says rewrite-generation bias is relatively small compared with false score increases from memorization; cached rewrites reduce cost and randomness and increase relative comparability.Source: Limitations · p. 9
- The paper characterizes the detection and rewriting concerns as limitations of current methods, not inherent limitations of the ITD setting.It says better methods could mitigate them.Source: Limitations · p. 9
A Experiment on Hyper Parameter
The detector uses two hyperparameters, K and ϵ. The paper selects them using rewriting sensitivity and classification accuracy on artificially constructed seen and unseen sets.
A Experiment on Hyper Parameter
- The detector implementation requires choosing K and ϵ.K is selected using the MinKprob change after rewriting; ϵ uses classification accuracy on constructed seen and unseen data.Source: A Experiment on Hyper Parameter · pp. 11–12
- The ϵ threshold was selected by searching for the highest classification accuracy on constructed seen and unseen sets.The paper constructs seen data through training and approximates unseen data with multiple rewrites of sampled test items.Source: A Experiment on Hyper Parameter · p. 12
- K = 20 was selected by searching for the maximum difference in MinKprob before and after rewriting.The difference was used as the selection criterion for sensitivity to rewriting’s impact.Source: A Experiment on Hyper Parameter · p. 12
B Rewrite details
The rewrite details tailor question changes to benchmark type while aiming to preserve the original questions’ integrity and difficulty. The paper describes rephrasing math-question stems with GPT-4.
B Rewrite details
- For MMLU, rewriting changes question phrasing while keeping the original sample’s tested knowledge points unchanged.Source: §B.1 · p. 12
- For GSM8K, rewriting changes question backgrounds while preserving the original numbers and calculations.Source: §B.1 · p. 12
- The prompts aim to preserve question integrity and difficulty while diversifying context and entities.The described prompt process rephrases fill-in-the-blank math-question stems using GPT-4.Source: §B.1 · p. 12
Mathematical Reasoning Problems
The math-question rewrite guidance aims to preserve the original problem’s meaning, answer, and difficulty while changing its surface wording. It also emphasizes clear, unbiased rewrites that retain the steps needed to solve the problem.
Mathematical Reasoning Problems
- Figures 5 and 6 document hyperparameter searches over ϵ for GSM8K and MMLU, respectively.Source: Figure 5 · p. 12, Figure 6 · p. 13
- Rewrites may change scenes and entities, but must preserve the numbers, operational logic, and final answer rather than merely substitute synonyms.The guidance gives replacing eggs with candies as an example of an allowed change.Source: Mathematical Reasoning Problems · p. 12
- Rewrites should keep the stem’s semantics consistent and avoid bias or clues that could make the question unfairly easier.Source: Mathematical Reasoning Problems · p. 12
- The revised question should remain clear, concise, and comparably complex while supporting step-by-step solving without changing the expected mathematical solution.The guidance also stresses retaining precision for the correct fill-in response.Source: Mathematical Reasoning Problems · pp. 12–13
- Names may change as long as the characters’ identities remain unambiguous.Source: Mathematical Reasoning Problems · p. 12
Knowledge-based Problem
The multiple-choice rewrite guidance calls for preserving meaning, accuracy, and difficulty while varying expression without changing mathematical content. Examples illustrate reworded questions and answer options.
Knowledge-based Problem
- Revised questions should retain the original meaning and accuracy, avoid bias or answer hints, and keep the difficulty of understanding and solving consistent.Source: Knowledge-based Problem · p. 13
- The guidance recommends varied but familiar wording and says not to alter mathematical expressions.It describes the structured prompt as maintaining question quality and difficulty.Source: Knowledge-based Problem · p. 13
- Figure 7 pairs a robe-and-fiber quantity question with a pizza-and-ingredients rewrite; Figure 8 is identified as a GSM8K ϵ search.Source: Figure 7 · p. 13, Figure 8
- A science multiple-choice example rephrases a question about boiling and freezing while restating its four answer choices.Source: Knowledge-based Problem · p. 13
B.2 Written Examples
The written examples show rewritten benchmark questions at different rewrite rounds, including examples from GSM8K and MMLU high-school psychology. The section also points to revision instructions and a released three-round dataset.
B.2 Written Examples
- The examples include single-round rewrites in Figure 8 and multi-round rewrites in Tables 7 and 8.Source: §B.2 · p. 13
- The authors say that three-round rewritten data for the entire dataset is released.Source: §B.2 · p. 13
- Tables 5 and 6 contain instructions for revising math question stems and multiple-choice questions and options.Source: Table 5 · p. 14, Table 6 · p. 15
- Tables 7 and 8 present written examples for GSM8K and MMLU high-school psychology, respectively.Source: Table 7 · p. 16, Table 8 · p. 17