Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdouts

1. Introduction

• A statistically meaningful comparison requires an IID benchmark split untouched by model development, but benchmark holdouts are typically unavailable.Benchmark developers usually release all evaluation data, with notable exceptions.Source: §1 · p. 1
• The authors propose retro-holdout datasets, independently and retroactively created and tested for similarity to a target benchmark, to quantify evaluation-performance gaps.Figure 1 sketches construction on the left and use to quantify benchmark inflation on the right.Source: §1 · p. 1, Figure 1 · p. 2

1.1. Contributions

• The paper presents a novel process for constructing retro-holdout datasets.Source: §1.1 · p. 2
• The authors release Retro-Misconceptions, a retro-holdout dataset for TruthfulQA that can quantify model performance gaps on the original dataset.Source: §1.1 · p. 2
• The authors evaluate twenty models using Retro-Misconceptions and report measurable score inflation.The introduction states that the results indicate evaluation gaming is occurring.Source: §1 · p. 1, §1.1 · p. 2

2.1. Understanding the RETRO Holdout Framework

• Because retro-holdouts are created post-release, four tests must fail to reject a shared-distribution hypothesis; the process avoids LLMs to prevent model bias.The framework targets labeled datasets and excludes evaluations designed to probe out-of-distribution performance. Requiring the same task as the public benchmark increases the time cost.Source: §2.1 · p. 2

2.2. Creating a RETRO

• RETRO creation starts from the TARGET’s original construction process and recommends using a randomly selected, representative TARGET entry without replacement as a basis for each new entry.The authors note that reaching sufficient indistinguishability may require iteration.Source: §2.2 · p. 3, §2.2.2 · p. 4

2.2.1. SUFFICIENT INDISTINGUISHABILITY

• Four tests assess difficulty, embedding similarity, model-based classification, and human distinguishability; all must fail to reject the null at p = 5%.The difficulty test compares pre-release models on both datasets; Figure 2 shows their accuracy comparison against a 95% confidence band.Source: §2.2.1 · p. 3, Figure 2 · p. 3
• The difficulty check uses pre-release models, while the method also strengthens that comparison with top-k answers, five-shot examples, and a helpful prompt.These techniques address the practical constraint that pre-release models may be less capable than current models.Source: §2.2.1 · p. 3

2.2.2. AN ITERATIVE PROCESS

• Iteration tools combine BERT token attribution with embedding-based visualization, similarity distributions, and review of the ten most-similar entry pairs.The embedding tools support outlier detection and topic clustering, but their insights may not suffice to establish indistinguishability.Source: §2.2.2 · p. 4

2.3. Evaluating Models

• Because TruthfulQA scoring uses log-probabilities that may penalize longer answers and are unavailable from some APIs, the authors instead prompt models to select numbered answer choices.They rotate and resample the options to reduce imbalance and ensure a dominating option.Source: §2.3 · pp. 4–5

2.4. The Challenges of TruthfulQA

• The case study focuses on TruthfulQA’s non-adversarial portion because filtering bias and performance differences complicate coverage, and the original GPT-3 filter model is unavailable.The authors note that their evaluation and difficulty test use identical datasets and methods despite this departure from a full reflection of TruthfulQA.Source: §2.4 · p. 5

3.1. Retro-holdout TruthfulQA Dataset

• Retro-Misconceptions passed all four indistinguishability tests, making it the first retro-holdout reported as sufficiently indistinguishable from its target.It mirrors TruthfulQA's Misconceptions category and is guaranteed accurate only for models with training cutoffs before January 1, 2024.Source: §3.1 · p. 5, Table 2 · p. 6

3.2. The Performance Gap

• Figure 4 shows evaluation gaming in both open-release and closed-source models despite conservative confidence bands and strict retro-holdout similarity criteria.The analysis quantifies benchmark inflation as the percentage-point difference between public-benchmark and retro-holdout performance for 20 models.Source: §3.2 · p. 5, Figure 4 · p. 6

Difficulty Gap

• Exact matches indicate some benchmark inclusion in training, and the three examined models showed substantial inflation; three datapoints cannot establish a correlation between match rate and inflation.Table 3 records exact-match rates and benchmark inflation for the models discussed.Source: Difficulty Gap · p. 6, Table 3 · p. 6

3.3. Why are Retro-holdouts Necessary?

• Retro-holdout creation requires substantial expert time and computational validation, while public/private split alternatives burden dataset developers and lack an easy platform.The authors avoided LLM assistance to prevent model bias, though they note it could automate time-consuming steps.Source: §3.3 · p. 6

3.4. Limitations

• The framework was tested only on TruthfulQA's Non-Adversarial Type, and distribution shifts may undermine the same-distribution assumption while matching preserves target-dataset biases.The authors designed the process for supervised text datasets but cannot yet guarantee its generality.Source: §3.4 · p. 8

• Test-data contamination can lead to memorization or indirect overfitting, while high benchmark scores are incentivized despite the lack of a standard detection methodology [1].The cited discussion also notes that data quality remains undervalued and under-incentivized.Source: §4 · p. 8
• Sainz et al. [37] elicit reproduction of test examples, while Golchin and Surdeanu [16] compare completions with data; GSM1k reports drops up to 13% but fails the authors' indistinguishability tests.Zhang et al. [47] introduced GSM1k as a dataset extension; the authors report it is not sufficiently indistinguishable from GSM8k.Source: §4 · p. 8
• Prior work links incentives to weaker metric predictive power [22], alongside calls for data-quality reforms and rigorous evaluation frameworks [3].Source: §4 · p. 8

• Accuracy fell by up to 16 percentage points on an unpublished dataset, with benchmark inflation reported for both API and open-release models.Examples include GPT and Claude among API models, and Mistral, Gemma, and Phi-2 among open-release models.Source: §5 · p. 8
• The authors caution against taking public benchmark scores at face value after evaluation data has been publicly available and present retro-holdouts as a practical evaluation path.Source: §5 · p. 8

A. Hold-out Testing Formalization

• A retro-holdout is tested against the public dataset’s distribution; if they are statistically consistent but model accuracies differ beyond a bound, the gap must reflect direct or indirect exposure to public data.The formalization assumes the model is independent of the samples when bounding expected accuracy differences.Source: A. Hold-out Testing Formalization · p. 11

A.1. Permutation Tests

• Permutation tests assess the null hypothesis that two datasets share a distribution by comparing the observed statistic with values under permutations of the pooled data.When enumerating permutations is impractical, Monte Carlo sampling can estimate the p-value.Source: §A.1 · p. 11

B. Semantic Embeddings

• Each entry is embedded using all-mpnet-base-v2, with the question and alphabetically ordered answer choices combined; similarity is measured by cosine similarity.Source: B. Semantic Embeddings · p. 11

C.1. Evaluation Harness

• The harness normalizes generated output and records a matching option; if none matches, it retries with an index-based prompt.It shifts option ordering and resamples at least five times until one choice leads alternatives by three selections, or uses the top choice after 100 attempts.Source: §C.1 · p. 12
• If the response does not match an option or an index, the harness supplies another prompt asking the model to select by index or letter.Source: §C.1 · p. 12

C.2. Prompts

• The prompts ask models either to repeat an option exactly or to select it by numeric or letter index.Source: §C.2 · p. 12

C.3. Compute

• The experiments used API and hosted models, with the largest open-release models run on 4xT4 GPUs; total compute was under $1000, and individual model evaluations cost $1–$50.Approximately 200 experiments produced the reported values and performance gaps.Source: §C.3 · p. 13

D. Guide for Initial RETRO Creation

• Create entries by matching representative target examples and preserving their topic and syntax, while accounting for dataset patterns, release-era knowledge, and recurring mistakes.The authors recommend first testing a large subset, then continuing with individual subsets; trivia-style benchmarks benefit from gathering factual sources before writing entries.Source: D. Guide for Initial RETRO Creation · pp. 13–14

E. RETRO Iteration

• During iteration, preserve the benchmark’s intended failure mode and make sure edited entries remain natural; matching the number of answer choices may improve prediction-accuracy and difficulty-similarity tests.Source: E. RETRO Iteration · p. 14

E.1. Fine-Tuned Prediction Model Attention

• The fine-tuned BERT attention tool often highlighted terminating periods rather than intermediate tokens, prompting a shift to comparing 1- and 2-gram frequencies.Figure 6 encodes token impact through color saturation, with green contributing to and red detracting from classification.Source: §E.1 · p. 14, Figure 6 · p. 15

E.2. Datapoint Embeddings

• UMAP projects entry embeddings into two dimensions to reveal clusters, outliers, and topic-coverage gaps; Figure 7 contrasts Retro-Misconceptions with TruthfulQA’s non-adversarial misconceptions and shows Law and Sociology entries.Source: §E.2 · p. 15, Figure 7 · p. 15
• Pairwise cosine-similarity histograms assess within-dataset similarity, and an early iteration showed RETRO was less internally similar than TARGET, with fewer highly similar pairs.Source: §E.2 · p. 15
• The highest-similarity tool compares each dataset’s ten most similar entry pairs; Figure 8 gives examples of pairwise cosine similarities from TruthfulQA.Source: §E.2 · pp. 15–16, Figure 8 · p. 16

F. Human Indistinguishability

• The human test presents annotators with labeled examples followed by tasks to identify the entry from the other dataset; chance-level performance is the criterion for indistinguishability.Twenty-three approved participants completed 230 tests.Source: F. Human Indistinguishability · p. 16

G. Similarity of Difficulty

• The Similarity of Difficulty test asks whether models uninfluenced by the target perform similarly on the target and retro-holdout.It therefore uses only models developed before the target’s release.Source: G. Similarity of Difficulty · p. 17
• Older closed-source models may become unavailable as developers discontinue access to reduce maintenance costs.This can constrain researchers who rely on older models for baselines or improvement analysis.Source: G. Similarity of Difficulty · p. 17
• Less capable pre-release models may perform similarly on datasets with different difficulty profiles once both exceed their capability threshold.The authors use techniques to enhance these models’ capabilities and expect performance to remain similar across sufficiently indistinguishable datasets, regardless of the boost technique.Source: G. Similarity of Difficulty · p. 17

H. Contemporaneous Work

• Zhang et al. [47] use human tests to assess similarity between GSM1k and GSM8k, and report many models overperforming on the target evaluation.Their GSM1k dataset was introduced for assessing mathematical reasoning.Source: H. Contemporaneous Work · p. 17
• GSM1k contains over 1,000 entries, but only 50 had been publicly released; its authors delayed full release to limit renewed benchmark leakage.They set release conditions of top open-source models scoring above 95% or the end of 2025.Source: H. Contemporaneous Work · p. 17
• The authors evaluate the 50 published GSM1k questions as GSM1k50 against GSM8k’s 1,319-question test split.They apply the same methods used for Retro-Misconceptions.Source: H. Contemporaneous Work · p. 17
• GSM1k50 receives TARGET and RETRO random-permutation p-values of 3.02 ± 0.05% and 98.7 ± 0.02%, respectively, while Prediction Accuracy detects a small statistically significant distinction from GSM8k.The authors present these findings as evidence for the rigor of their standard of sufficient indistinguishability.Source: H. Contemporaneous Work · p. 17
• The two independently developed projects use different methodologies but both emphasize comprehensive dataset validation for accurate model evaluation.This is the authors’ stated connection between the studies.Source: H. Contemporaneous Work · p. 17