PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmark Data Contamination of Large Language Models: A Survey

Cheng Xu, Shuhao Guan, Derek Greene, M-Tahar Kechadi

arXiv:2406.04244v1cs.CL

TL;DR

Benchmark data contamination can skew LLM evaluation, making performance assessments unreliable. This survey reviews detection methods and mitigation strategies, concluding that none is immune to secondary or semantic-level contamination.

  • Problem

    Benchmark data contamination occurs when models encounter evaluation data during training, potentially inflating scores and making their capabilities difficult to assess accurately.

    Source: §1 · p. 1, §2.2 · p. 4

  • Method

    The paper reviews BDC detection methods and groups mitigation strategies into data curation, data refactoring, and benchmark-free evaluation.

    Source: §6 · p. 20

  • Results

    The reviewed mitigation strategies are not immune to secondary or semantic-level contamination.

    Source: §6 · p. 20

  • Takeaways & Limitations

    Fully eliminating BDC risks is nearly impossible, so evaluation approaches must balance performance with practicality.

    Source: §6 · p. 20

    Mitigation approaches involve trade-offs: curation raises accessibility and bias concerns, while refactoring may require substantial resources.

    Source: §4.3 · p. 18

Abstract

from arXiv · show

The rapid development of Large Language Models (LLMs) like GPT-4, Claude-3, and Gemini has transformed the field of natural language processing. However, it has also resulted in a significant issue known as Benchmark Data Contamination (BDC). This occurs when language models inadvertently incorporate evaluation benchmark information from their training data, leading to inaccurate or unreliable performance during the evaluation phase of the process. This paper reviews the complex challenge of BDC in LLM evaluation and explores alternative assessment methods to mitigate the risks associated with traditional benchmarks. The paper also examines challenges and future directions in mitigating BDC risks, highlighting the complexity of the issue and the need for innovative solutions to ensure the reliability of LLM evaluation in real-world applications.

1 INTRODUCTION

The introduction defines benchmark data contamination (BDC) as a threat to reliable LLM evaluation and presents the survey as a systematic review of detection methods, mitigation strategies, and future challenges.

INTRODUCTION

  • BDC occurs when training data incorporates evaluation-benchmark information, potentially making LLM evaluations unreliable and undermining their validity and real-world trust.The introduction notes that BDC also raises privacy and security considerations [17] [18] [53] [73].Source: §1 · pp. 1–2
  • Traditional benchmarks can misrepresent LLM capabilities because training or fine-tuning data may contain benchmark metadata, label distributions, or contextual information, a problem complicated by AI-generated content.Such contamination can affect model behavior and evaluation performance, leading to misguided conclusions.Source: §1 · p. 2
  • Alternative approaches include regenerating benchmark data [158] [181] and benchmark-free evaluation [24] [87] [166] to reduce reliance on traditional benchmarks.These approaches aim for more flexible, adaptive, and reliable assessment.Source: §1 · p. 2
  • The survey addresses the lack of a comprehensive, systematic account of BDC by defining the problem and organizing prior research into detection techniques and mitigation strategies.Detection identifies BDC risks; mitigation addresses them in current LLM evaluation.Source: §1 · p. 2
  • The survey reviews detection methods in Sections 3–4, covering matching- and comparison-based detection, data curation and refactoring, and benchmark-free evaluation; Section 5 discusses challenges and future directions.Section 2 provides background and defines the BDC problem with examples.Source: §1 · p. 2

2 BACKGROUND

This background section introduces LLM capabilities and limitations, defines benchmark data contamination (BDC), and explains its sources, evaluation impacts, and vulnerabilities across common NLP tasks.

2.1 Large Language Models

  • LLMs use diverse architectures and support generation, knowledge use, reasoning, and tool interaction, but hallucinations, specialized-domain weaknesses, and stale knowledge remain challenges.Decoder-only transformers predominate among the largest models, while other recent architectures also exist.Source: §2.1 · pp. 3–4

2.2 Benchmark Data Contamination

  • BDC occurs when models encounter benchmark data during training, distorting evaluation scores so they may not reflect true capabilities.The section categorizes exposure at semantic, information, data, and label levels.Source: §2.2 · p. 4
  • BDC severity increases from semantic to label exposure, but semantic- and information-level contamination are harder to detect and mitigate than full label exposure.Label exposure can enable more straightforward detection and prevention, whereas abstract or intricate exposure is more difficult to address.Source: §2.2 · pp. 4–5

2.3 Sources and Impact

  • Diverse, large-scale pre-training datasets are the primary BDC source; fine-tuning is generally more controlled and presents less contamination risk.Pre-training draws on public sources such as news, forums, and social media, increasing the chance of unintended test-data exposure.Source: §2.3 · p. 5
  • BDC can inflate traditional benchmark scores and skew automatic and human evaluations; self-supervised automatic methods may mitigate but cannot eliminate risk, and human judgments remain bias-prone.Contamination can distort accuracy, calibration, fairness, and robustness measures; human evaluators may also be influenced by familiarity with data.Source: §2.3 · pp. 5–6

2.4 Related Tasks

  • Code-generation benchmarks risk semantic leakage from related tutorials, while translation benchmarks can inherit topic or narrative biases from shared source texts.Filtering exact test answers may not remove related problem-solving content in code tasks.Source: §2.4 · pp. 6–7
  • Question-answering data can leak through related discussions, and topic context in sentiment benchmarks can bias predictions and distort evaluation.The passage gives COVID sentiment as an example in which prior topic knowledge may predispose the model toward negative labels.Source: §2.4 · p. 7
  • Entity-related pre-training can distort named-entity recognition; news-event coverage can expose fake-news benchmarks at multiple contamination levels, and complete texts can undermine reconstruction tests.Text reconstruction is vulnerable when the complete benchmark text already appears in pre-training data.Source: §2.4 · p. 7

3 BDC DETECTION TECHNIQUES

The survey organizes BDC detection into matching-based methods that seek direct evidence of overlap and comparison-based methods that examine model outputs or performance. Both offer ways to investigate contamination, but their access, computational, and scope constraints complicate practical use.

3.1 Matching-based Methods

  • Matching-based methods seek overlap through dataset inspection, membership inference, generated examples, retrieval, and masked-answer guessing [30].Li and Flanigan [85] also use chronological analysis; Deng et al. [30] pair retrieval with Testset Slot Guessing, where models fill masked multiple-choice answers.Source: §3.1 · pp. 8–9, Figure 1 · p. 9
  • DCQ found higher contamination levels than other methods on GPT-3.5/4, while a combined instance- and partition-level technique reached 92%–100% accuracy across seven datasets [46] [47].The latter study identified contamination in AG News, WNLI, and XSum when tested with GPT-4.Source: §3.1 · pp. 9–10
  • Contamination ranged from 1% to 45% across over 15 LLMs and six QA benchmarks [91], while book memorization tracked web frequency and affected downstream cultural-analytics assessments [20].Li et al. [91] also found contamination did not always correspond to improved performance, though larger models could benefit more on some benchmarks.Source: §3.1 · p. 10
  • Matching can be impractical without proprietary training data and computational resources, while paraphrasing and Evasive Augmentation Learning can evade string-matching detection [107] [162] [29].EAL rephrases benchmark samples during fine-tuning and improved benchmark performance by up to 15% while remaining undetected by existing methods.Source: §3.1 · p. 10

3.2 Comparison-based Methods

  • Comparison-based methods include content similarity, output distributions, perplexity, generation order, and performance changes across dataset release periods [82] [96] [34] [89] [61] [121].CDD improved contamination detection and TED reduced contamination-related performance inflation; Dong et al. [34] also introduced the DetCon and ComiEval benchmarks.Source: §3.2 · p. 11
  • Seen-versus-unseen performance can reveal exploitation that varies with model size and duplication [96], while perplexity audits found reading-comprehension and summarization benchmarks more affected than multiple-choice ones [89].Canonical-versus-shuffled ordering tests can provide provable contamination guarantees without access to pre-training data or model weights, though audits found minimal evidence of widespread contamination.Source: §3.2 · p. 11
  • GPT-4 performance declined significantly on Codeforces problems released after September 2021 [61], and code/math pass rates correlated with GitHub popularity and benchmark release dates [121].Fine-tuning and Chain-of-Thought prompting did not consistently mitigate the decline; Roberts et al. [121] contributed an open-source dataset, raw results, and evaluation framework.Source: §3.2 · pp. 11–12
  • Comparison-based methods require substantial testing resources and may generalize poorly when focused on specific contamination types or datasets, including when chronological data is unavailable.Together with training-data access and computational barriers for matching approaches, these limits make practical detection a balance between effectiveness and feasibility.Source: §3.2 · pp. 12–13

4 BDC MITIGATION STRATEGIES

The survey groups BDC mitigation into data curation, refactoring existing data, and benchmark-free evaluation. These approaches reduce contamination risks in different ways, but each has limitations and may remain vulnerable to secondary or semantic contamination.

4.1 Curating New Data

  • Private benchmarks isolate test data from public exposure, while encryption and exclusion controls help protect integrity; limited access raises trust and ethical concerns.Chandran et al. [19] describe confidential computing and cryptography, while Jacovi et al. [65] propose encryption, training-exclusion controls, and avoiding online data.Source: §4.1 · pp. 13–14
  • Dynamic benchmarks refresh evaluation data; LatestEval showed minimal memorization, while LiveCodeBench measures coding generalization and highlights potential overfitting in existing benchmarks.Dynaboard [95] evaluates models in the cloud with customizable Dynascore metrics, and Fan et al. [41] propose monthly reasoning-benchmark updates; dynamic approaches can introduce bias, inconsistent results, and resource costs.Source: §4.1 · pp. 13–14

4.2 Refactoring Existing Data

  • On 51 LLMs, EvoEval’s five prompt transformations yielded an average 39.4% performance reduction on HumanEval [23].The prompts probe difficult, creative, subtle, combined, and tool-use dimensions of existing coding questions.Source: §4.2 · p. 15
  • DyVal 2’s Meta Probing Agents dynamically assess language understanding, problem-solving, and domain knowledge; related methods regenerate samples or decontaminate training datasets.MPA evaluations found a correlation between model size and basic abilities, and MPA can augment training data; Ying et al. [164] use mimicking and difficulty extension, while Yang et al. [162] advocate stronger decontamination and fresh one-time exams.Source: §4.2 · pp. 15–16

4.3 Benchmark-free Evaluation

  • TreeEval evaluates LLMs without fixed benchmarks through adaptive question trees and achieved high correlation with AlpacaEval2.0 [37] using approximately 45 questions.Its irreproducible evaluation sessions are intended to prevent data leakage, though LLM judges may themselves rely on contaminated or biased training data.Source: §4.3 · pp. 16–17
  • Chatbot Arena [24] collects over 240K crowdsourced pairwise preference votes, while FreeEval [166] unifies traditional, regenerated, LLM-judged, and human evaluation methods.Human evaluation can reflect real-world preferences but introduces subjectivity and requires time and resources; benchmark-free approaches also face uncertainty about external evaluators.Source: §4.3 · pp. 17–18
  • Mitigation methods remain vulnerable to secondary and semantic contamination, and filtering alone may miss contamination while evaluator LLMs draw on their training data.Curation, refactoring, and benchmark-free approaches also face resource demands and risks involving accessibility, bias, subjectivity, or reliability.Source: §4.3 · pp. 17–18

5 CHALLENGES AND FUTURE DIRECTIONS

The paper argues that BDC risks are impracticable to eliminate fully because large-scale pre-training and recursive AI-generated content make contamination difficult and costly to detect or prevent. It proposes several evaluation and transparency directions while emphasizing that each requires attention to practical limitations and data integrity.

Challenges

  • Large-scale pre-training inevitably includes benchmark-relevant information, while recursive AIGC makes semantic contamination harder to identify and comprehensive mitigation potentially prohibitively costly.Data filtration and regeneration mitigate but do not address semantic and informational dimensions of BDC; the paper calls for solutions that balance performance with practicality.Source: §5 · pp. 18–19

Human Evaluation

  • Human evaluation may offer an ideal approach, but it is resource-intensive and vulnerable to subjective biases shaped by evaluators’ backgrounds.The paper cites Chiang et al. [24] and identifies political, cultural, personal, and educational influences as potential sources of bias.Source: §5 · p. 19

Dynamic System

  • Dynamic evaluation can move beyond fixed benchmark paradigms and mitigate BDC, but static or AIGC evaluation data leave residual risks that require scrutiny of data origin and composition.Adaptive scheduling is exemplified by Li et al. [87]; Li et al. [90] introduces fresh data streams, yet evaluation data may remain inherently static.Source: §5 · p. 19

Benchmark Content Tags

  • Standardized benchmark-content tags could identify benchmark-related content and reduce the burden of filtering pre-training data.The proposed protocol would attach fixed tags to content relevant to benchmarks to indicate that model evaluation is implicated.Source: §5 · p. 19

Comprehensive Evaluation Systems

  • Comprehensive evaluation systems can integrate multiple BDC-mitigation options, including LLM-as-judge and human participation, to address risks that single-perspective strategies may miss.The paper cites Yu et al. [166] as a framework that complicates LLM performance assessment, while arguing integrated tools can minimize BDC risks.Source: §5 · p. 20

6 CONCLUSION

The paper surveys BDC detection and mitigation strategies, while emphasizing that none fully avoids contamination and that large-scale pre-training and AIGC make eliminating BDC risks nearly impossible. It identifies several future directions and calls for continued work on robust LLM evaluation.

6 CONCLUSION

  • BDC detection methods fall into matching-based and comparison-based classes, while mitigation strategies center on data curation, data refactoring, and benchmark-free evaluation, none immune to secondary or semantic-level contamination.The paper notes that each approach has its own challenges.Source: §6 · p. 20
  • Large-scale pre-training and AIGC make fully eliminating BDC risks nearly impossible, motivating human evaluation, dynamic systems, benchmark content tags, adversarial evaluation, and comprehensive evaluation systems.Each proposed direction also has its own challenges and considerations.Source: §6 · p. 20
  • BDC requires multifaceted solutions, and further work on robust, reliable LLM evaluation will become increasingly important as LLMs evolve and enter daily life.The paper presents its review as a resource for researchers and practitioners navigating LLM evaluation under BDC.Source: §6 · p. 20

7 AUTHOR CONTRIBUTIONS

All authors contributed to the paper’s conception, design, and execution and approved its final version.

7 AUTHOR CONTRIBUTIONS

  • CX shaped the core ideas and framework, led the research, and contributed substantially to most sections; SG contributed to Section 2, while DG and MTK provided supervision and guidance.All authors read and approved the final manuscript.Source: §7 · p. 20

Related papers

Loading 2406.04244v1…