Source-linked AI summary
Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards
Xingyao Xiao, Yihong Cheng
TL;DR
The paper examines whether benchmark contamination merely inflates LLM scores or also changes leaderboard rankings, using original–paraphrase contrasts to separate memorization from capability. It finds contamination is largely uniform: scores rise, but rankings rarely change, with standard and paraphrase-controlled rankings correlating at 0.997.
Problem
Existing contamination research provides limited evidence about whether leaked benchmark items reorder model rankings, beyond inflating absolute scores.
Method
The paper measures contamination as an anchor-item invariance violation by comparing model performance on original items with semantically equivalent paraphrases.
Results
Contamination among examined public models largely inflated absolute scores without reordering leaderboards; standard and paraphrase-controlled rankings correlated at 0.997.
Takeaways & Limitations
Leaderboard reliability appears more robust than contamination concerns suggest when rankings are compared with paraphrase-controlled rankings.
Takeaways & Limitations
The within-item contrast requires both an original and a corresponding paraphrase and is blind to contamination that is perfectly uniform across models.
Abstract
from arXiv · showhide
Benchmark contamination, the leakage of test items into training data, is widely described as a threat to the reliability of large language model (LLM) leaderboards. We argue that this concern conflates two distinct questions: whether contamination inflates absolute scores, and whether it reorders the ranking of models. We recast contamination as a violation of anchor-item invariance and measure it through the differential functioning of original versus semantically equivalent paraphrased items, a within-item contrast that holds the measured skill fixed and isolates memorization from capability. Using per-instance responses from 47 publicly released models and 74 models finetuned with a known dose of contamination, across four benchmarks (ARC, GSM8K, HellaSwag, MMLU), we first calibrate the measure against ground truth: it recovers injected contamination dose-responsively (a corrected effect of +0.187 accuracy points for test-set leakage) and never flags a negative-control model trained only on the legitimate training split (-0.012). We then quantify leaderboard impact: the rank correlation between a standard leaderboard and a paraphrase-controlled leaderboard is 0.997, and a sensitivity analysis shows that the observed differential contamination is far below the level needed to move rankings, with only 3 of 188 model-by-benchmark cases showing differential contamination corroborated across two references. Contamination among these public models is therefore largely uniform: it inflates absolute scores without reordering the leaderboard, and ranking distortion requires the rare case of differential contamination. We provide a calibrated invariance audit, released as a reference implementation, and recommend that leaderboards report paraphrase-controlled rankings alongside confidence intervals.
Introduction
The introduction reframes benchmark contamination as a measurement-invariance problem, distinguishing absolute score inflation from leaderboard reordering. It motivates a comparative audit of differential contamination and identifies calibration and ranking impact as unresolved gaps.
- Motivation: Benchmark contamination can inflate measured performance without a corresponding gain in ability, raising concerns about benchmark and leaderboard reliability.
- Core distinction: Absolute score inflation and leaderboard reordering are distinct outcomes: shared contamination raises scores together, whereas ranking distortion requires differential contamination.
- Research gaps: Existing approaches mainly detect whether individual models are contaminated, while leaving leaderboard effects, detection calibration, and uniform-versus-differential contamination unresolved.
- Conceptual framework: A contaminated benchmark item is framed as a differential item functioning violation because memorization makes it differentially easy for some models.
Contamination as Anchor-Item Invariance
The paper reframes contamination as a violation of anchor-item invariance and detects it with within-item contrasts between original and semantically equivalent paraphrased items. This design separates memorization from capability without relying on confounded cross-model group comparisons.
- Conceptual basis: Anchor-item invariance assumes that, conditional on ability, models have the same probability of answering an item correctly.Contamination violates this assumption for models that have seen the item.
- Conceptual basis: Training cutoff is a confounded proxy for contamination because newer, more capable models are also more exposed to benchmark items.This makes cross-model comparisons difficult to interpret.
- Within-item test: The design compares each item’s original wording with a semantically equivalent paraphrase that measures the same skill but was not the memorized string.The within-item contrast removes the cross-model group contrast.
- Within-item test: Differentially higher performance on the original than the paraphrase indicates memorization, whereas equal performance indicates possession of the underlying skill.The contrast tests whether a model responds differently to the memorized wording while holding the measured skill fixed.
- Aggregated identification: Aggregating the original–paraphrase contrast across models identifies which items function as contaminated anchors.Prior work applied related paraphrase tests at smaller scale, across a few models or a single benchmark.
The Present Study · Method · Data
The study treats benchmark contamination as anchor-item measurement non-invariance, contrasting original with semantically equivalent paraphrased items while holding measured skill fixed. It calibrates this audit with controlled contamination and evaluates uniform versus differential effects using public and controlled model groups across four benchmarks.
- The Present Study: Contamination is recast as an anchor-item invariance violation measured by contrasting original and paraphrased items within item pairs.The contrast holds measured skill fixed, isolating contamination-related differential functioning.
- The Present Study: The audit is calibrated against models contaminated with known doses to establish its detection floor.This calibration provides ground-truth labels for assessing whether the measure detects contamination.
- The Present Study: Contamination is decomposed into a uniform component that shifts absolute scores and a differential component that can reorder rankings.The study quantifies both components on 47 publicly released models.
- The Present Study: The headline result is that contamination among the studied models is largely uniform, inflating scores but rarely reordering leaderboards.Rare ranking-changing exceptions are detectable with the proposed audit.
- The Present Study: The audit is a reusable, ground-truth-calibrated procedure for treating benchmark contamination as measurement non-invariance, released as a reference implementation.This describes the study’s methodological contribution and implementation release.
- Data: Per-instance binary correctness is collected for normal, rephrase, and synthetic versions of ARC, GSM8K, HellaSwag, and MMLU.Scoring uses exact match for GSM8K, accuracy for MMLU, and length-normalized accuracy for ARC and HellaSwag; synthetic items were never published.
- Data: The release contains 47 publicly released models and 74 controlled models finetuned on individual benchmarks under known contamination conditions.Conditions include original-test leakage, paraphrased-test leakage, repetition, two learning rates, and legitimate-training-split finetuning as a negative control.
- Data: Two observational datasets—the 12-model PSN-IRT response matrix and 69 HELM Classic models—are used only for the first result’s identifiability check.These datasets are keyed to training cutoff relative to benchmark release.
Measures
The study measures contamination as the original-minus-paraphrase correctness gap, calibrated against a clean baseline, and compares rankings from original and paraphrased responses to isolate differential contamination. Model ability is estimated with a Rasch model, with robustness checks confirming the ranking conclusion under raw accuracy.
- Contamination signal: The model-level contamination signal is the mean original-minus-paraphrase correctness difference across shared items, referenced to the same gap for a clean Llama-2 base model.The item-level contaminated-anchor statistic is the benchmark-centered mean original-minus-paraphrase gap.
- Ability estimation: Model ability is estimated with a Rasch model whose correctness probability is the logistic function of θ_m−b_i, fit by penalized joint maximum likelihood.Alternating Newton updates use a ridge penalty λ = 0.1, keeping estimates finite near perfect or zero scores and centering abilities at zero for identification.
- Robustness: The Rasch model is defensible because models answer hundreds to thousands of items, and the ranking conclusion survives replacing it with raw accuracy.The reported regime makes finite-sample bias negligible.
- Leaderboard comparison: Rankings are fit separately to original and paraphrased responses: the standard leaderboard uses original items, whereas the paraphrase-controlled leaderboard uses paraphrased items.Uniform contamination shifts both equally without changing relative order, so rank discrepancies indicate differential contamination.
- Independent check: An independent comparison repeats the leaderboard analysis against rankings estimated from synthetic items, without depending on paraphrasing.
Observational identifiability check … Observational grouping by training cutoff is not identified
Observational grouping by training cutoff fails to identify contamination because clean reference models are usually absent or incomparable in ability. The paper therefore motivates a within-item design and evaluates its sensitivity through detection-rate, minimum-detectable-effect, and simulated-reordering analyses.
- Observational identifiability check: The observational alternative groups models by whether their training data could contain a benchmark, using release date relative to benchmark release as an exposure proxy.It fits item-level logistic-regression DIF models with leave-one-benchmark-out ability estimates and compares unweighted with inverse-variance-weighted pooling.
- Observational identifiability check: Pooling sign agreement serves as an identification diagnostic: disagreement indicates that the exposure contrast is confounded rather than informative.The check is applied to PSN-IRT and HELM datasets to motivate the within-item design used in the Results section.
- Sensitivity analysis: The sensitivity analyses use only public data on a laptop, with no model inference run.The sample is fixed by public release, so sensitivity is characterized directly rather than through an a priori power analysis.
- Sensitivity analysis: The sensitivity analysis characterizes detection using contamination-condition detection rates, a minimum detectable effect, and simulated differential contamination followed by reordering measurement.The simulation injects differential contamination of magnitude delta into half the models on half the items under a fixed seed.
- Observational grouping by training cutoff is not identified: 9 of 11 benchmarks in the 12-model PSN-IRT matrix have zero clean reference models, so training-cutoff grouping does not identify contamination DIF.Most models postdate most benchmarks, leaving no clean comparison for nearly all benchmarks.
- Observational grouping by training cutoff is not identified: The 69-model HELM Classic analysis still lacks common ability support: the clean group contains only a handful of old, low-ability models.This makes the clean and contaminated groups incomparable for the observational contrast.
- Observational grouping by training cutoff is not identified: +1.724 on GSM8K versus -0.777 shows sign instability between unweighted and inverse-variance-weighted pooled logistic-DIF coefficients.Both estimates use the same items, demonstrating that the observational effect is not robust to pooling choice.
- Observational grouping by training cutoff is not identified: +0.248, z = 7.10 emerged only after constructing an ability-balanced contrast, motivating the within-item, ground-truth approach.Balancing used a benchmark released in the middle of the model cutoff distribution so clean and contaminated models overlapped in ability.
The audit recovers ground-truth contamination dose-responsively
The base-referenced original-minus-paraphrase gap recovers known contamination dose while leaving legitimate training exposure indistinguishable from zero. Correcting for benchmark-level difficulty shifts is necessary to avoid false contamination flags.
- Ground-truth calibration: The base-referenced original-minus-paraphrase gap tracks known contamination dose, whereas legitimate training exposure is not mistaken for contamination.The negative-control training condition has a corrected effect of -0.012 (SD = 0.035), indistinguishable from zero.
- Ground-truth calibration: +0.187 corrected effect (SD = 0.146) is observed for test-set leakage, compared with +0.226 (SD = 0.162) under the higher learning rate.Lower-dose conditions fall in between.
- Ground-truth calibration: 62% to 86% detection power is achieved across genuinely contaminated conditions, versus 0% for the negative control.Detection requires a paired z above 1.96 and a corrected effect above 0.05.
- Limitations: Paraphrase-only memorization is a blind spot: models finetuned on paraphrased tests produce a near-zero or negative original-minus-paraphrase gap.The reported negative gap is -0.091.
- Baseline correction: +0.21 raw gap on HellaSwag appears even in clean base models because paraphrasing adversarial endings makes items harder for everyone.Referencing each model to a clean baseline removes this benchmark-level difficulty shift and prevents widespread false contamination reports.
Leaderboards are robust to contamination among public models · The null is informative, and a second reference confirms it
Among 47 public models, paraphrase-controlled scoring leaves leaderboard order nearly unchanged because observed differential contamination is generally below the distortion threshold. Controlled contamination sharply reorders deliberately contaminated models, while synthetic references and benchmark-specific checks confirm the public-model result.
- Leaderboards are robust to contamination among public models: 215 flagged anchors, including 152 HellaSwag and 60 ARC items, reflect broad paraphrase-difficulty heterogeneity rather than model-specific contamination.Removing these items therefore does not move the ranking.
- Leaderboards are robust to contamination among public models: 17 positions is the rank drop for the higher-learning-rate GSM8K contaminated model, versus 10 for the default-leakage model when scoring switches to paraphrases.Lower-dose conditions move −4 and +1 positions, while the negative control moves +2 and the paraphrase-leakage model moves +3.
- The null is informative, and a second reference confirms it: 0.16 accuracy points of differential contamination is required before leaderboard reordering becomes meaningful, defined as rank correlation falling below 0.97.The observed range remains where rank correlation is near unity.
- The null is informative, and a second reference confirms it: 0.972 rank correlation between standard and synthetic-reference leaderboards confirms the result beyond paraphrasing, compared with 0.997 for paraphrase-controlled scoring.The synthetic reference detects paraphrase leakage at +0.334, while the negative control remains near zero on both axes.
- The null is informative, and a second reference confirms it: 3 corroborated cases remain among 188 model-by-benchmark combinations when both paraphrase and synthetic references must exceed the detection floor.These model-level cases are rare and concentrated in a few large models on GSM8K and MMLU, consistent with near-invariance of the leaderboard.
Discussion
For widely used public models on standard benchmarks, contamination inflates scores but rarely reorders leaderboards because differential contamination is uncommon and too small to move rankings. The audit supports reporting paraphrase-controlled rankings alongside standard rankings, while recognizing limits in coverage, calibration, and adversarial detection.
- Leaderboard impact: 0.997 rank correlation between standard and paraphrase-controlled rankings shows that contamination generally raises scores without reordering the leaderboard.A second synthetic reference agrees at 0.972.
- Leaderboard impact: 3 of 188 model-by-benchmark cases showed differential contamination corroborated across two independent references, indicating that ranking distortion requires rare differential effects.Sensitivity analysis found the observed differential contamination several times too small to move rankings.
- Measurement and identification: The measurement-invariance reframing separates contamination from capability through within-item contrasts, avoiding the confounding and lack of common support in training-cutoff comparisons.Grouping models by training cutoff cannot identify contamination because exposure and capability are collinear.
- Practical implication: Leaderboards should report paraphrase-controlled rankings alongside standard rankings and confidence intervals, making agreement a direct certificate that contamination is not driving comparisons.The recommendation applies the audit’s ranking-integrity focus rather than asserting that benchmarks are absolutely clean.
- Boundaries: The conclusion is bounded by four benchmarks, 7-billion-parameter calibration models, possible weaker signals from incidental exposure, and under-detection of adversarial or reasoning-model contamination.The contrast also requires original and paraphrased items answered by many models, while providers can train on variants or conceal memorization through reasoning.
Declarations
The authors report no funding or relevant competing interests, and ethics and consent approvals are not applicable because the study uses publicly available non-human model-evaluation data. Derived response matrices and analysis code are available in the anchor-dif repository, with contributions led by X. Xiao and Y. Cheng responsible for writing, reviewing, and editing.
- Funding: No funding was received for conducting this study.
- Conflicts of interest / Competing interests: The authors declare no competing interests relevant to the article.
- Ethics and consent: Ethics approval and consent to participate and publication are not applicable because the study analyzes publicly available non-human model-evaluation data without human or animal participants.
- Availability of data and materials: Derived model-by-item response matrices and the analysis code are available in the anchor-dif repository at https://github.com/DoriaXiao/anchor-dif.
- Authors’ contributions: X. Xiao is the corresponding author responsible for all aspects of the work, while Y. Cheng handled writing, reviewing, and editing.
Open Practices Statement
The study reused public data without new inference or human data collection, and provides its response matrices and analysis code for reproducibility. All reported quantities are computed inline, with the leaderboard simulation using fixed seed 1; the study was not preregistered.
- Open Practices Statement: The study reused publicly available data and conducted no new model inference or human data collection.
- Open Practices Statement: The per-instance benchmark response matrices come from the ConStat release, while the derived matrices and analysis code are available at https://github.com/DoriaXiao/anchor-dif.
- Open Practices Statement: Every reported quantity is computed inline from the response matrices, and the leaderboard-distortion simulation runs under fixed seed 1.
- Open Practices Statement: The study was not preregistered.
Appendix Computational Environment
The appendix documents a reproducible computational environment using R 4.5.1 on aarch64 Apple Darwin under macOS Tahoe 26.5.1, with specified numerical libraries, locale, time zone, and packages.
- Core environment: R 4.5.1 ran on the aarch64-apple-darwin20 platform under macOS Tahoe 26.5.1.The appendix reports the R version, platform, and operating system used.
- Numerical libraries: The environment used the specified BLAS and LAPACK libraries from the R 4.5-arm64 framework.Both numerical-library paths are recorded in the computational-environment details.
- Locale and time zone: The locale was en_US/en_US/en_US/C/en_US/en_US, and the time zone was America/Los_Angeles.These settings define the reported regional and time-handling configuration.
- Loaded packages: The attached R packages included tidyr_1.3.1, ggplot2_3.5.2, dplyr_1.2.1, and readr_2.1.5.The appendix also lists the base and recommended R packages loaded in the session.