PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Benchmark Leakage in Large Language Models

Ruijie Xu, Zengzhi Wang, Run-Ze Fan, Pengfei Liu

arXiv:2404.18824v1cs.CLcs.AIcs.LG

TL;DR

Opaque training data and undisclosed benchmark use make it difficult to assess LLM evaluation results fairly. The paper proposes a scalable leakage-detection pipeline and applies it to 31 open-source models in mathematical reasoning, finding that many may have used benchmark data and recommending clearer documentation.

  • Problem

    Opaque training processes and benchmark use complicate objective assessment of LLM results and can undermine benchmark integrity and fair comparisons.

    Source: §1 · p. 2

  • Method

    The authors use Perplexity and N-gram Accuracy in a simple, scalable pipeline to detect potential leakage, including across paraphrased benchmark versions.

    Source: §1 · p. 3

  • Results

    Approximately half of 31 open-source LLMs may have leveraged benchmark data to improve mathematical-reasoning performance, with potential unfair advantages and possible test-set leaks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors recommend model documentation, benchmark setup, and evaluation changes, including a Benchmark Transparency Card that records benchmark use and data augmentation.

    Source: §1 · p. 3

    The analysis covers 31 open-source LLMs and mathematical-reasoning tasks.

    Source: §8 · p. 13

Abstract

from arXiv · show

Amid the expanding use of pre-training data, the phenomenon of benchmark dataset leakage has become increasingly prominent, exacerbated by opaque training processes and the often undisclosed inclusion of supervised data in contemporary Large Language Models (LLMs). This issue skews benchmark effectiveness and fosters potentially unfair comparisons, impeding the field's healthy development. To address this, we introduce a detection pipeline utilizing Perplexity and N-gram accuracy, two simple and scalable metrics that gauge a model's prediction precision on benchmark, to identify potential data leakages. By analyzing 31 LLMs under the context of mathematical reasoning, we reveal substantial instances of training even test set misuse, resulting in potentially unfair comparisons. These findings prompt us to offer several recommendations regarding model documentation, benchmark setup, and future evaluations. Notably, we propose the "Benchmark Transparency Card" to encourage clear documentation of benchmark utilization, promoting transparency and healthy developments of LLMs. we have made our leaderboard, pipeline implementation, and model predictions publicly available, fostering future research.

1 Introduction

Opaque training practices and benchmark leakage complicate objective evaluation of LLMs, risking inflated benchmark effectiveness and unfair comparisons. The paper studies leakage in mathematical reasoning benchmarks and presents a scalable detection pipeline, analysis of existing models, and transparency recommendations.

1 Introduction

  • Opaque LLM training processes and lagging evaluation protocols make objective assessment difficult, potentially overstating benchmark effectiveness and obscuring unfair comparisons.The paper notes that supervised data is included in some models’ pretraining.Source: §1 · p. 2
  • GSM8K and MATH provide training and test splits, are challenging to improve on with limited effective datasets, and are widely used to evaluate popular models.The authors select them to investigate potential leakage in a relevant mathematical-reasoning setting.Source: §1 · p. 2
  • The authors propose a simple, computationally efficient, scalable pipeline using Perplexity and N-gram Accuracy to identify potential leakage from metric discrepancies across paraphrased benchmark versions.The paper also summarizes training behaviors and challenges that affect leakage detection.Source: §1 · pp. 2–3
  • Analysis of 31 open-source LLMs suggests approximately half may have leveraged benchmark data for mathematical reasoning, with potential unfair advantages and possible test-set leaks.The pipeline can detect instances; for example, Qwen-1.8B correctly predicts all 5-grams in 25 MATH test examples.Source: §1 · p. 3
  • The authors recommend changes to model documentation, benchmark setup, and evaluation, including a Benchmark Transparency Card registering benchmark use and data augmentation.The card is intended to support transparency in model development.Source: §1 · p. 3

2 Preliminaries

The preliminaries describe benchmark training behaviors, define benchmark leakage, and outline why detecting it is difficult. In particular, train–test metric comparisons can be inconclusive, leakage thresholds vary across models and data, and benchmark use is often unknown.

2.1 Typical Training Behaviors

  • The paper distinguishes training without benchmark data, training with its training split, and training with its test split; exposure can be inadvertent or intended to improve capabilities.Test-split exposure does not necessarily indicate generalization and may make benchmarks ineffective; training-split exposure can lead to unfair comparisons and overly optimistic generalization estimates.Source: §2.1 · pp. 3–4

2.2 Definition of Benchmark Leakage

  • Benchmark leakage occurs when an example from either benchmark split appears in pretraining data, as input alone or as an input-output pair.The latter can confer an unfair advantage.Source: §2.2 · p. 4

2.3 Challenges for Detecting Benchmark Leakage

  • Perplexity comparisons cannot reliably identify models exposed to both benchmark splits, since their scores may resemble those of models exposed to neither.The intuitive train-versus-test comparison is more informative when only the training split was seen.Source: §2.3 · p. 4
  • No universal leakage threshold is easy to set: model size, training strategies, and pretraining-data distributions affect scores, while retraining models to simulate leakage is costly and data distributions are unknown.Generalization from training data to the test set further complicates detection of exposure to both splits.Source: §2.3 · p. 4
  • Benchmark use during pretraining is often unknown, and datasets may be augmented, synthesized, or used for hyperparameter tuning.The paper gives paraphrasing and reformatting as examples of augmentation.Source: §2.3 · p. 4
  • Closed-source models may expose only API access, leaving logits inaccessible and making perplexity-related metrics difficult to calculate.Source: §2.3 · p. 4

3 Detection Methodology

The methodology combines Perplexity and N-gram Accuracy with synthesized reference benchmarks to assess potential benchmark leakage. It compares normalized metric changes across training and test splits and uses their disparity to indicate relative familiarity, while acknowledging test-set leakage may remain possible.

3.1 Atomic Detection Metrics

  • Perplexity measures next-token uncertainty, while N-gram Accuracy samples five continuations and assesses whether predicted n-grams match their targets.For N-gram Accuracy, the method combines question and answer text, samples five starting points, and uses subsequent n-grams as prediction targets.Source: §3.1 · p. 5

A Unified View

  • Perplexity captures sequence likelihood while N-gram Accuracy checks exact subsequences; N-gram Accuracy can also analyze closed-source models without access to logits.The method also considers ROUGE-L and edit-distance similarity to accommodate some paraphrasing or reformatting.Source: A Unified View · pp. 5–6, §3.1 · p. 5

3.2 Reference Benchmark Synthesis

  • The authors generate three reference-benchmark versions with ChatGPT, aiming to preserve the original format and reasoning difficulty through surface paraphrasing while acknowledging phraseology bias.Generation uses temperature 0.7 and top_p 0.9.Source: §3.2 · p. 6

3.3 Detection Pipeline

  • The pipeline averages atomic-metric scores across three synthesized datasets, normalizes the original-versus-reference decrement by the original score, and compares training and test decreases.The train-test disparity is δtrain-test = δtrain − δtest; the authors describe it as offsetting synthesis bias.Source: §3.3 · pp. 6–7
  • A significant δtrain-test suggests disproportionate familiarity with the training split, while a notably negative value points to potential test-split leakage; near zero can mean no leakage or leakage in both splits.The authors caution that test-set leakage cannot be entirely ruled out even when training-set leakage appears more likely.Source: §3.3 · p. 7

4 Meta-Experiment: the Reliability of the Detecting Pipeline

The authors validate the pipeline by training Mistral-7B-v0.1 on benchmark data using pre-training or SFT, then comparing seen and unseen sets with synthetic references. Both strategies produce larger metric gaps on seen data, while patterns across metrics and datasets support pipeline effectiveness; simultaneous train/test leakage remains a caveat.

Setup

  • The experiment trained Mistral-7B-v0.1 using either full-loss pre-training or solution-only SFT, with 1,000 seen and 1,000 unseen examples per benchmark.Both sets were paraphrased into three synthetic reference sets for comparison.Source: §4 · p. 7

Limitation

  • When both training and test splits leak simultaneously, the pipeline may not detect leakage effectively, although N-gram accuracy can greatly mitigate this issue.This caveat qualifies detection in the simultaneous-leakage case.Source: §4 · p. 7

Results

  • For both pre-training and SFT, benchmark-trained models show greater differences between original and synthetic references on seen data; Figure 3 reports GSM8K and MATH results for ∆ and δ.The caption identifies ∆ as the decrement and δ as the percentage decrease.Source: §4 · p. 7, Figure 3 · p. 8
  • Without benchmark training, metrics remain low and δseen-unseen is essentially zero, while benchmark training also increases ∆unseen and δunseen.These patterns distinguish benchmark familiarity from the no-benchmark-training condition.Source: §4 · p. 8
  • N-gram accuracy produces higher ∆ and δ for pre-training, whereas perplexity produces higher ∆ and δ for SFT; SFT also yields higher δseen-unseen.The passage attributes the metric differences to n-gram accuracy covering questions and solutions, while perplexity is calculated only on solutions.Source: §4 · p. 8
  • The δseen-unseen ordering is consistent across atomic metrics and benchmark datasets, supporting the pipeline’s effectiveness.The comparison concerns the relative scores across the tested strategies, metrics, and datasets.Source: §4 · p. 8

5 Evaluation in the Wild

The evaluation applies the pipeline to 31 LLMs on GSM8K and MATH, comparing split-level metrics and examining sample-level predictions. The results identify potential leakage patterns while showing where metric comparisons can miss or mischaracterize them.

5.1 Setup

  • 31 LLMs were evaluated on GSM8K and MATH using perplexity, 5-gram accuracy, and 10-gram accuracy.For each dataset, the study sampled up to 3,000 training and test examples; perplexity was calculated on solutions, while n-gram accuracy used concatenated question-and-solution text.Source: §5.1 · p. 8, §5.2 · p. 8

5.2 Main Results

  • Aquila2 and non-Base InternLM-2 show higher δtrain-test scores, while LLaMA models and Mistral-7B-v0.1 are near zero; PPL and 5-gram rankings largely align.Figures 4 and 5 order models by δtrain-test; higher scores indicate greater potential training-set use relative to the test set.Source: Figure 4 · p. 9, Figure 5 · p. 9, §5.2 · p. 9
  • A small absolute ∆ paired with a large percentage decrease can make δtrain-test unreliable, and low scores do not rule out leakage when both splits are contaminated.The authors suggest examining train and test ∆ and δ scores alongside n-gram predictions in such cases.Source: §5.2 · pp. 9–10
  • The two metrics can rank models differently: n-gram accuracy is sensitive to exact formatting, while perplexity can capture adaptation to rephrased or reformatted data.The authors attribute ranking discrepancies to the metrics’ different language-modeling approaches.Source: §5.2 · p. 10

5.3 N-gram Accuracy Helps Instance-level Leakage Detection

  • Qwen-1 8B correctly predicts all 5-grams in 223 GSM8K training examples, 67 MATH training examples, and 25 MATH test examples.The authors say n-gram accuracy can complement split comparisons when training and test sets are simultaneously leaked, but detects examples only in their original format and wording unless the training format is known.Source: §5.3 · p. 11

5.4 Case Study

  • A Qwen-1.8B training example has perfect 5-gram predictions, while Aquila2-34B’s test examples use “The answer is” where the ground truth has a “####” placeholder.The Aquila2 behavior helps explain why n-gram accuracy can be challenging to use for detecting leakage.Source: §5.4 · p. 11

6 Recommendation for Model Documentation and Benchmarks Setup

The paper recommends steps for fairer LLM evaluation through model documentation, benchmark design and access, and cautious interpretation of results.

6 Recommendation for Model Documentation and Benchmarks Setup

  • Model releases should disclose benchmark training use and evaluation details through a proposed Benchmark Transparency Card.The card supplements Data Cards and Model Cards and covers benchmark use, data augmentation, and evaluation details.Source: §6 · p. 12
  • Build benchmarks from recent corpora and regularly update evaluation datasets with dynamic benchmarks to reduce overlap risk and guard against overfitting to static tests.Source: §6 · p. 12
  • Avoid posting paired benchmark questions and answers online; encrypted test sets or private leaderboard evaluations are alternatives.Source: §6 · p. 12
  • Treat strong performance on specific benchmarks cautiously and use contemporary challenges for a more balanced assessment of model abilities.The authors also caution that benchmark data submitted through proprietary-model APIs could be used for further training.Source: §6 · p. 12

7 Related Work

The related work covers benchmark development, leakage detection, and model memorization; this paper focuses on mathematical reasoning benchmarks and proposes a detection approach for settings with limited training-data transparency.

7 Related Work

  • The paper uses challenging mathematical reasoning tasks such as GSM8K and MATH as its testbed for leakage detection.Source: §7 · p. 12
  • Earlier leakage studies used n-gram overlap with pre-training corpora, while data access or information about sources and time frames can also support detection.Source: §7 · p. 12
  • Prior approaches include benchmark perturbation, synthetic data, and paraphrasing, alongside studies simulating leakage and examining memorization.Source: §7 · p. 13
  • The paper's approach uses basic language-modeling metrics and data synthesis without relying on instruction-following capabilities, and can detect leakage at instance level.Source: §7 · p. 13

8 Conclusion and Limitations

The paper presents a scalable leakage-detection pipeline and analyzes 31 open-source LLMs on mathematical reasoning benchmarks. It reports possible benchmark misuse and unfair advantage, while noting detection limits for augmented or reformatted data and concurrent train-test leakage.

8 Conclusion and Limitations

  • 31 open-source LLMs analyzed for mathematical reasoning reveal that many may have exploited benchmark data for an unfair advantage.Source: §8 · p. 13
  • The pipeline combines two fundamental metrics and is described as simple, computationally efficient, scalable, and validated through meta-experiments.Source: §8 · p. 13
  • The introduced n-gram accuracy metric supports instance-level detection and can accurately identify many instances that were trained on.Source: §8 · p. 13
  • The authors recommend transparent benchmark-use documentation, including adoption of a Benchmark Transparency Card.Source: §8 · p. 13
  • The method might miss benchmark training data that was augmented or reformatted, as well as concurrent leakage in training and testing datasets.The authors state that n-gram accuracy mitigates some of these issues.Source: §8 · p. 13

A.1 Reference Benchmark Synthesis

The appendix identifies tables containing the data-synthesis prompts and synthesized examples.

A.1 Reference Benchmark Synthesis

  • Tables 1 and 2 provide full data-synthesis prompts, while Tables 3 and 4 show synthesized examples.Source: §A.1 · p. 21

A.2 The Details of Meta Experiments

The meta experiments train Mistral-7B-v0.1 on GSM8K and MATH using specified training durations, batch size, optimizer settings, and eight A100 GPUs.

A.2 The Details of Meta Experiments

  • Mistral-7B-v0.1 was trained for 20 epochs on GSM8K and 8 epochs on MATH, with a batch size of 64 on both datasets.The experiments used eight NVIDIA A100 80G GPUs.Source: §A.2 · p. 21
  • The experiments used AdamW with β1 = 0.9, β2 = 0.95, ϵ = 10^-5, and weight decay of 0.1.Source: §A.2 · p. 21
  • A cosine learning-rate schedule decayed each maximum learning rate to 10% of its value, with rates varying by dataset and loss setting.Maximum rates were specified separately for GSM8K and MATH under full-loss and answer-loss training.Source: §A.2 · p. 21

A.3 The Description of Edit Distance Similarity

This appendix describes edit-distance similarity and benchmark-documentation materials, including a transparency card that specifies what model and benchmark-use information to report.

A.3 The Description of Edit Distance Similarity

  • Edit-distance similarity is defined from string edit distance and string length, while edit distance counts insertions, deletions, or substitutions needed to transform one string into another.The distance is computed with a dynamic-programming recurrence, and the final value is the distance between the full strings.Source: §A.3 · p. 21
  • The appendix includes paraphrasing prompts for GSM8K and MATH and examples of paraphrased data for both benchmarks.Source: Table 1 · p. 22, Table 2 · p. 23, Table 3 · p. 23
  • The listed meta-experiment tables cover 5-gram accuracy and perplexity on GSM8K and MATH, while other tables order models by train-test differences and report suspicious-sample statistics.Source: Table 5 · p. 24, Table 6 · p. 24, Table 7 · p. 24

A.4 Benchmark Transparency Card

  • The Benchmark Transparency Card is divided into basic model details, a benchmark utilization statement, and benchmark evaluation details.Source: §A.4 · p. 29, Table 19 · p. 30

BENCHMARK EVALUATION DETAILS

BASIC MODEL DETAILS

BENCHMARK UTILIZATION STATEMENT (1/N)

Related papers

Loading 2404.18824v1…