Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

Ruijie Hou; Yueyang Jiao; Zhao Wang; Yingming Li

Source-linked AI summary

Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

Ruijie Hou, Yueyang Jiao, Zhao Wang, Yingming Li

arXiv:2608.07341v1Aug 7, 2026cs.CLcs.AI

TL;DR

G-AP cannot reliably measure contamination restoration because discrete readouts and aggregate-first differencing obscure per-question behavior. The paper introduces SA-PPG and RailCap; across models and benchmarks, prior restoration is substantially overestimated, while RailCap achieves state-of-the-art restoration.

Problem
G-AP inadequately evaluates restoration because discrete readouts miss per-question performance and aggregate-first differencing allows over- and under-suppression to cancel.
Method
The paper estimates per-question solve probabilities, stratifies probability gaps by clean-model solve probability in SA-PPG, and proposes RailCap for generation-time contamination suppression.
Results
Across models and benchmarks, SA-PPG shows prior strategies’ restoration was substantially overestimated by G-AP, while RailCap achieves state-of-the-art restoration.
Takeaways & Limitations
A zero G-AP does not certify restoration, so per-question restoration should instead be judged with A-PPG or the stratified SA-PPG.
Takeaways & Limitations
G-AP is not a reliable restoration metric because its aggregate-first procedure permits over- and under-suppression to cancel, with additional sampling noise.

Abstract

from arXiv · show

Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. \textbf{Contamination mitigation evaluation} intervenes in the decoding process to suppress memorization and restore a contaminated model's genuine capability, but its prevailing metric, the \textbf{G-AP} (\textbf{G}ap of \textbf{A}ggregate \textbf{P}erformance), is flawed. Discrete correct/incorrect readouts cannot characterize per-question performance, averaging before differencing lets over- and under-suppression cancel out, and uniform per-question weighting invites strategies to push solve probabilities onto the clean model's high-frequency values. We propose \textbf{SA-PPG} (\textbf{S}tratified \textbf{A}ggregate of \textbf{P}er-question \textbf{P}robability \textbf{G}aps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate. \textbf{RailCap} instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution becomes sufficiently dispersed. Across multiple contaminated models and benchmarks, SA-PPG reveals that prior strategies' restoration is substantially overestimated, while RailCap attains the lowest SA-PPG.

1 Introduction

Public-benchmark contamination inflates scores, motivating decoding-based restoration rather than repeatedly rebuilding leaked datasets. The paper introduces SA-PPG to expose restoration errors and RailCap to mitigate contamination online during generation.

Memorized benchmark items inflate evaluation scores and weaken benchmarks’ ability to measure genuine capability.
Contamination mitigation intervenes during decoding to suppress memorization without constructing a new dataset.
G-AP is flawed because discrete 0/1 readouts and averaging before differencing obscure per-question performance and allow suppression errors to cancel.
SA-PPG estimates each question’s solve probability, computes its gap from the clean model, and aggregates gaps within clean-probability groups.
RailCap caps the next greedy-trajectory token to the runner-up whenever sampling returns to that trajectory, allocating intervention online through step-wise supervision.
SA-PPG reveals that prior strategies’ restoration is systematically overestimated by G-AP, while RailCap achieves state-of-the-art restoration across models and benchmarks.

2 Related Work

Prior contamination-mitigation work intervenes during decoding to suppress memorization, but evaluates restoration with inconsistent gap-of-aggregate-performance metrics. This paper argues for probabilistic per-question evaluation with repaired aggregation because existing readouts and two-step mitigation strategies can misrepresent restoration.

Contamination and mitigation: Public benchmark test data commonly enters pretraining, allowing memorization to inflate scores and obscure genuine model capability.This contamination has motivated calls to protect benchmark test data.
Contamination and mitigation: Contamination mitigation evaluation suppresses memorization during decoding to restore genuine capability without constructing a new dataset.Reliable evaluation is needed to determine whether claimed restoration actually holds.
Evaluation metrics: Prior mitigation metrics are all G-AP measures that average per-question performance and compare it with the clean model’s average, differing mainly in how readouts are obtained.LNE-blocking uses a 0/1 greedy-response readout, whereas TED estimates pass@1 after filtering near-greedy sampled responses.
Evaluation metrics: Probabilistic readouts are preferable because a single-sample 0/1 result cannot reproduce even the same model’s performance across evaluations, while SA-PPG also repairs aggregation.No prior work had compared discrete and probabilistic representations or conducted cross-method comparison under one metric.
Mitigation strategies: Existing mitigation strategies first estimate contamination and then operate on the estimated portion, with granularity varying from contaminated responses to contaminated questions.TED filters suspected memorized samples using edit distance to the greedy decode, while LNE-blocking estimates each question’s contamination degree with length-normalized entropy.

3 SA-PPG: A Stratified Per-Question Restoration Metric

SA-PPG addresses G-AP’s failure to measure per-question restoration by using sampled solve probabilities, taking absolute gaps before aggregation, and stratifying questions by clean-model solve probability. This prevents cancellation and frequency-based shortcuts: zero SA-PPG certifies perfect restoration on every question.

Per-question probability gaps: Discrete correctness samples are replaced by estimated solve probabilities, obtained as the fraction of correct responses across m independent samples.The binary readout is a single Bernoulli draw, whereas the solve probability is its expectation.
Per-question probability gaps: A-PPG averages absolute per-question probability gaps, unlike G-AP, which averages first and can cancel over- and under-suppression.A-PPG equals zero if and only if every question’s solve probability matches the clean model’s.
Aggregation-order failure: G-APP can equal zero despite no restoration: with half the questions at −δ and half at +δ, G-APP = 0 while A-PPG = δ.A-PPG adds over- and under-suppression, whereas G-APP nets them against each other.
Stratification: Equal-weight A-PPG can reward driving solve probabilities toward frequent clean-model values, because most questions may cluster near zero.A strategy with zero solve probability attains zero gap on the zero-probability majority, diluting gaps on the minority.
Stratification: SA-PPG groups questions into equal-width clean-model solve-probability intervals, aggregates gaps within non-empty groups, and then averages across groups.The zero-probability majority receives group weight 1/|B| regardless of its size, so low SA-PPG requires restoration at every probability level.

4 RailCap: Step-wise Supervision during Generation

RailCap detects memorization online during generation by identifying returns to the contaminated model’s greedy trajectory and capping its next trajectory token to the runner-up. Repeated step-wise suppression accumulates until responses become sufficiently dispersed, without relying on a prior contamination estimate.

Motivation: Existing mitigation strategies estimate contamination before intervention, so restoration quality depends entirely on the estimate’s accuracy.SA-PPG requires per-question adjustment of solve probabilities, whereas prior strategies operate on an estimated contaminated part.
Online contamination signal: On leaked questions, contaminated-model samples collapse onto the greedy trajectory, while unleaked-question samples disperse across many paths.This makes fallback onto the greedy trajectory an online memorization signal.
Per-step operation: When sampled tokens match a greedy-trajectory window, RailCap caps the trajectory successor to the current second-largest logit before sampling.Otherwise logits remain unchanged, and the next token is sampled from Softmax(ℓ_t/τ).
Step-wise supervision: The cap makes samples likely to leave the greedy trajectory, and repeated interventions accumulate suppression until the response distribution becomes sufficiently dispersed.The same rule applies at every decoding step and every prompt, with intervention determined online by observed responses.

5 Experiments

Experiments across three model families and two GSM8K-based contamination settings show that SA-PPG overturns G-AP’s near-perfect restoration verdict and consistently favors RailCap. Probability readouts, stratified aggregation, and RailCap’s adaptive suppression expose and reduce residual contamination without the same degree of collateral damage.

Experimental setup: The evaluation uses GSM8K and paraphrased PQ questions across Llama-2-7B, Gemma-4-E2B, and Pythia-12B, with 50 samples per question at T=0.7.PQ rewrites question wording while preserving numbers and answers, creating a harder contamination setting than verbatim memorization.
Metric comparison: Under G-AP, LNE-blocking scores 0.0235 versus 0.3192 for Identity, but SA-PPG reverses the ranking: RailCap scores 0.1914 versus 0.2932 for LNE-blocking.Shortcut scores 0.2476 under SA-PPG, while Identity scores 0.3261.
Readout stability: The solve-probability readout reduces the mean per-question gap between independent batches from 0.190 for r0/1 to 0.041.This shows that probability estimates are substantially more stable per question than single-sample discrete readouts.
Aggregation failure: LNE-blocking’s over-suppression is 0.0836 versus 0.0420 for RailCap, yet cancellation makes its G-APP 0.0207 versus 0.0794 for RailCap.The lower G-APP therefore masks collateral damage and residual-contamination tradeoffs rather than indicating genuine per-question restoration.
Cross-setting results: RailCap attains the lowest SA-PPG in all six settings, including 0.1914 on Llama-2 × GSM8K versus 0.2476 for Shortcut.Shortcut is runner-up in five settings, while LNE-blocking trails both Shortcut and RailCap in all six.
RailCap ablation: With n=4, RailCap balances stratified residual contamination and collateral damage at 0.0942 versus 0.0972 and achieves the best SA-PPG of 0.1914.Thresholds n=3 through 7 remain within 0.008 of the best, while hard banning the rail token raises collateral damage to 0.14.

6 Conclusion

The paper addresses how to measure restoration from contamination mitigation and how to design better mitigation strategies. It replaces G-AP’s flawed aggregate evaluation with per-question solve probabilities and grouped aggregation.

The conclusion frames contamination mitigation around two coupled problems: evaluating restoration and designing better mitigation strategies.
G-AP’s discrete readout cannot represent per-question performance, while averaging before differencing allows over- and under-suppression to cancel.
The proposed correction operates on per-question solve probabilities and aggregates within groups defined by the clean model’s solve probability.Grouping prevents strategies from pushing solve probabilities toward the clean model’s frequent values under equal weighting.

A PQ Construction and Evaluation Details · A.1 PQ construction.

The PQ construction rewrites each leaked GSM8K question while preserving its numbers and final answer, forbidding solutions, and retaining the original reasoning chain and gold answer. The rewritten questions are generated in a strict one-line format by DeepSeek-V4-Flash.

A.1 PQ construction.: Each leaked question is substantially rephrased before evaluation.The rewriting is performed with DeepSeek-V4-Flash.
A.1 PQ construction.: The final answer remains identical after question rewriting.The construction preserves the original final answer rather than generating a new target.
A.1 PQ construction.: Each rewritten question is output as exactly one line.The required format is “New Question: <rephrased question>” and nothing else.
A.1 PQ construction.: All numerical values in the original GSM8K question are preserved.The rewriting instruction explicitly requires keeping all numbers identical.
A.1 PQ construction.: The rewritten output must contain no solution or explanatory reasoning.The instruction forbids including any solution in the rewritten question.
A.1 PQ construction.: The reasoning chain and gold answer remain the original GSM8K versions.Only the question text is rewritten; the reasoning chain and answer are retained.