PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models

Yongding Tao, Tian Wang, Yihong Dong, Huanyu Liu, Kechi Zhang, Xiaolong Hu, Ge Li

arXiv:2510.09259v2cs.CLcs.AIcs.LG

TL;DR

Data contamination threatens reliable LLM evaluation, but existing detectors largely target pre-training and SFT rather than RL post-training. The paper introduces Self-Critique, which probes RL-induced policy collapse, and RL-MIA, a benchmark for simulated RL contamination. Self-Critique outperforms baselines by up to 30% AUC, while existing methods operate near random guess.

  • Problem

    Existing contamination detectors largely target pre-training and SFT, leaving RL post-training insufficiently addressed despite its importance for improving LLM reasoning.

  • Method

    Self-Critique generates two responses and detects contamination from their entropy-space similarity, while RL-MIA simulates controllable RL-phase contamination for evaluation.

  • Results

    Self-Critique consistently outperforms baselines, improving average AUC by up to 30%, with further gains of up to 55% in dual-contamination scenarios.

  • Takeaways & Limitations

    The results make RL-phase contamination detection possible where existing methods perform near random guess and motivate detectors tailored to reward-driven training.

  • Takeaways & Limitations

    The study is scoped to public, non-sensitive benchmarks to mitigate acknowledged dual-use and data-privacy concerns.

Abstract

from arXiv · show

Data contamination poses a significant threat to the reliable evaluation of Large Language Models (LLMs). This issue arises when benchmark samples may inadvertently appear in training sets, compromising the validity of reported performance. While detection methods have been developed for the pre-training and Supervised Fine-Tuning stages, a critical research gap exists for the increasingly significant phase of Reinforcement Learning (RL) post-training. As RL post-training becomes pivotal for advancing LLM reasoning, the absence of specialized contamination detection methods in this paradigm presents a critical vulnerability. To address this, we conduct the first systematic study of data detection within RL post-training scenario and propose Self-Critique. Our method is motivated by a key observation: after RL phase, the output entropy distribution of LLMs tends to collapse into highly specific and sparse modes. Self-Critique probes for the underlying policy collapse, i.e., the model's convergence to a narrow reasoning path, which causes this entropy reduction. To facilitate this research, we also introduce RL-MIA, a benchmark constructed to simulate this specific contamination scenario. Extensive experiments show that Self-Critique significantly outperforms baseline methods across multiple models and contamination tasks, achieving an AUC improvement of up to 30%. Whereas existing methods are close to a random guess for RL-phase contamination, our method makes detection possible.

1 INTRODUCTION

RL-phase contamination threatens LLM evaluation because likelihood-based detectors target pre-training and SFT, while RL uses sparse rewards and erases those signals. The paper introduces Self-Critique and RL-MIA to detect this overlooked contamination through policy-collapse probing.

  • RL post-training creates a major contamination-detection gap because reward-driven optimization does not preserve the likelihood signals used by existing methods.RL uses sparse rewards to find successful reasoning paths rather than maximizing the probability of observed data.
  • Direct entropy inspection is unreliable because policy collapse can produce sparse token-level entropy for both contaminated and clean samples.The method therefore uses an active probing strategy rather than a passive entropy check.
  • Self-Critique generates two responses for the same problem and flags samples whose responses have highly similar entropy patterns.This probes whether RL-induced policy collapse makes contaminated samples converge to a narrow reasoning path.
  • The paper presents the first systematic study of contamination detection during the RL post-training phase of LLMs.This contribution targets a stage increasingly used to improve LLM reasoning.
  • Self-Critique significantly outperforms existing detectors, which perform near random guess, achieving an AUC improvement of up to 30%.The method is evaluated across multiple models and contamination tasks.
  • RL-MIA is a benchmark that simulates RL-specific contamination across math and logic tasks for systematic detector evaluation.Its construction addresses the absence of an existing benchmark isolating contamination from the RL phase.

2 RELATED WORKS

Related work frames LLM data-contamination detection as membership inference and emphasizes its importance for benchmark validity. It also situates entropy as a central factor in RL post-training, balancing exploration and exploitation.

  • Data-contamination detection is treated as a membership-inference problem connected to memorization, privacy risks, and benchmark validity.Prior LLM detection work has mainly focused on pre-training and Supervised Fine-Tuning.
  • RL post-training uses entropy to balance stochastic exploration against deterministic exploitation in large language models.The passage identifies entropy as a key factor in RL post-training.

3 THE CHALLENGE OF CONTAMINATION DETECTION IN RL

RL-phase contamination is difficult to detect because reinforcement learning removes the likelihood-based signals used by existing detectors. Policy collapse and sparse token-level entropy patterns provide an alternative signal for identifying contaminated samples.

  • Problem Definition: RL post-training detection is formalized as a black-box membership inference task over whether a sample belongs to the RL training dataset.The detector queries model outputs only and returns membership or non-membership predictions.
  • Why RL Post-training Is a Unique Case: Existing detectors target pre-training and SFT because both maximize likelihood on observed training sequences, creating signals such as low perplexity.Perplexity and Min-K% Prob rely on this shared likelihood principle.
  • Why RL Post-training Is a Unique Case: RL instead maximizes reward for generated outputs rather than token-level likelihood of ground-truth responses, weakening likelihood-based contamination signals.Its external reward may be sparse, such as assigning 1 for a correct answer and 0 otherwise.
  • Entropy as a New Signal for RL Detection: RL frequently induces policy collapse, where consistently rewarded samples converge to narrow reasoning paths and produce overly stable outputs.This collapse is reflected in token-level entropy and can concentrate probability mass on particular tokens.
  • Entropy as a New Signal for RL Detection: Contaminated samples show stronger entropy collapse than clean samples, motivating entropy-based comparison of alternative reasoning paths instead of likelihood measurement.Self-Critique probes this difference by regenerating a path and comparing token-level entropy sequences.

4 DETECTION VIA SELF-CRITIQUE

Self-Critique probes RL-induced path dependence by eliciting an initial response and a self-critique response, then comparing their token-level entropy sequences. Higher length-aware similarity indicates that the model stayed on the same reasoning path, suggesting contamination.

  • Detection via Self-Critique: For RL-seen problems, high-reward path dependence makes the model more likely to reproduce a similar trajectory, whereas unseen problems permit alternative reasoning paths.This contrast supplies the method’s contamination-detection hypothesis.
  • Detection via Self-Critique: Self-Critique elicits a deterministic initial response, asks for a different solution conditioned on it, and compares both responses’ entropy sequences.The workflow is illustrated in Figure 2.
  • Detection via Self-Critique: The initial response is generated with deterministic decoding and converted into an entropy sequence representing the baseline reasoning trajectory.This sequence is then used to condition the self-critique prompt.
  • Detection via Self-Critique: The self-critique response is generated by appending an instructional meta-prompt and the initial response to the original problem.The second response also yields a token-level entropy sequence for comparison.
  • Detection via Self-Critique: Higher entropy-sequence similarity indicates that the model remains on the same reasoning path despite being instructed to change it, suggesting memorization.The score uses length-aware penalized cosine similarity, including zero-padding and a response-length penalty.

5 EXPERIMENTS

The experiments evaluate Self-Critique on RL-MIA and related contamination settings, comparing it with likelihood-based and probing baselines across models, datasets, and RL algorithms. Self-Critique generally provides the strongest RL-phase contamination detection, including when pretraining and RL contamination coexist.

  • 5.1 RL-MIA BENCHMARK: The experiments construct RL-MIA to simulate controllable RL-phase contamination across diverse mathematical and logic benchmarks, including settings with possible prior pretraining exposure.The benchmark selectively includes items in RL post-training data and tests whether detectors identify used samples.
  • 5.3 MAIN RESULTS: Self-Critique achieves the best average AUC on both evaluated models: 0.70 on Qwen2.5-7B-Instruct and 0.64 on DeepSeek-Math-7B-Instruct, each 19% above the best non-ours baseline.It also leads on most per-dataset AUCs in the RL-MIA benchmark.
  • 5.3 MAIN RESULTS: Self-Critique improves over other entropy-based methods by +13% on Qwen2.5-7B-Instruct and +16% on DeepSeek-Math-7B-Instruct.The comparison uses the same probing-mechanism family, highlighting the contribution of self-critique probing.
  • 5.4 DUAL-STAGE CONTAMINATION IN PRE-TRAINING & RL: On dual-contamination data, Self-Critique improves as pretraining contamination decreases, whereas the random-control subset does not improve and PPL approaches random guessing.This rules out a pure sample-size explanation and supports clearer identification of RL-phase memorization or path dependence under weaker pretraining signals.
  • 5.5 ABLATION STUDIES: Across PPO, GRPO, and DAPO, Self-Critique has the best AUC for every algorithm and the highest average AUC of 0.60.Entropy-Temp averages 0.58, while likelihood baselines remain around 0.46–0.51.
  • 5.5 ABLATION STUDIES: Reducing Top-K for entropy approximation does not materially harm detection, with only a slight AUC drop even at K = 3.Top-K approximation is motivated by the concentration of most next-token probability mass among a small set of tokens.

6 CONCLUSION

The paper presents Self-Critique as a detector for RL-induced policy collapse and introduces RL-MIA for controlled evaluation of RL-phase contamination. Across experiments, Self-Critique outperforms baselines and improves average AUC by up to 30%, with stronger isolation of RL-specific signals in dual-contamination settings.

  • 6 CONCLUSION: Self-Critique identifies RL-induced policy collapse by probing reasoning-path dependencies through token-level entropy.RL-MIA provides a controlled benchmark for validating this RL-phase contamination detector.
  • 6 CONCLUSION: Self-Critique consistently outperforms baselines, improving average AUC by up to 30%.In dual-contamination scenarios, its performance improves by up to 55%, highlighting its ability to isolate RL-specific signals.

ETHICS STATEMENT

The ethics statement frames Self-Critique as a defensive validation tool for improving LLM evaluation integrity while acknowledging potential dual-use concerns. The study is scoped to public, non-sensitive benchmarks without personal data.

  • ETHICS STATEMENT: The study targets reliable LLM evaluation using a defensive membership-inference tool applied exclusively to public, non-sensitive benchmarks.The authors state that the evaluated AIME, K&K, SAT, and GSM8K benchmarks contain no personal data.

REPRODUCIBILITY STATEMENT

The paper provides reproducibility materials, formal algorithm details, and additional evaluation results for Self-Critique under RL-MIA.

  • The reproducibility release includes code, RL-MIA, baseline implementations, experiment scripts, and a formal algorithm description.
  • The algorithm requires a black-box model with per-token log-probability or top-k probability access and uses deterministic greedy decoding.
  • Self-Critique generates two responses, computes token-level entropy sequences, pads them, and scores cosine similarity between the sequences.
  • Self-Critique achieves the best AUC on Qwen2.5-7B-Math for AIME24 (0.76), AIME25 (0.72), and their average (0.74).
  • As pretraining contamination decreases, Self-Critique AUC rises from 0.59 to 0.88, isolating an RL-specific detection signal.

B.3 ADDITIONAL RESULTS ON RLHF PARADIGM

Additional RLHF experiments test whether Self-Critique generalizes beyond the primary RLVR setting. Across four alignment paradigms, it remains the strongest detector while several baselines approach chance.

  • Across four RLHF alignment paradigms, Self-Critique achieves the highest AUC scores, ranging from 0.62 to 0.70.
  • PPL and CDD often perform near random guessing at AUC ≈0.50 or show instability in the RLHF experiments.
  • The results suggest policy collapse is a fundamental characteristic of alignment training across sparse, implicit, and token-level reward signals.

C ADITIONAL ABLATIONS

Ablations show that Self-Critique depends on anchoring the alternative response to the initial response, is affected by sampling choices, and remains robust to prompt paraphrases.

  • Self-critique probing: Removing the initial response anchor causes the unconventional-technique variant’s performance to collapse toward random guessing.
  • Sampling strategy: The sampling ablation compares greedy and temperature sampling for the initial and self-critique responses across multiple temperatures.
  • Prompt sensitivity: A five-template ablation evaluates whether Self-Critique depends on the wording of its meta-instruction.
  • Prompt sensitivity: AUC standard deviations remain low across prompt variants: 0.0251 on AIME25 and 0.0254 on K&K.

D VISUALIZATION OF CONTAMINATION SCORE DISTRIBUTION

The paper visualizes Self-Critique similarity scores and describes the benchmark construction and implementation details used to analyze contamination detection.

  • Visualization: Contaminated and uncontaminated samples show clearly different Self-Critique similarity-score distributions in histograms and KDE plots.
  • Prompt design: The self-critique prompt asks the model to generate a second response following a different reasoning path from the initial response.
  • Prompt design: Prompt variations preserve the same request for an alternative reasoning path while changing its wording.
  • Benchmark construction: RL-MIA injects known test samples into an RL corpus and evaluates whether detectors distinguish them from clean, unseen samples.
  • Experimental configuration: The appendix reports data splits, shared training hyperparameters, and model-specific context-length settings for reproducibility.

G.1 DISCUSSION ON RESULT VARIABILITY AND ERROR BARS

Self-Critique provides more stable RL-contamination detection than baselines across model responses, while retaining a modest inference budget. Bootstrap analysis confirms performance above likelihood-based methods and random guessing, but broader domain and scale generalization remain open.

  • Result variability and error bars: Self-Critique’s semantic probing is more stable than Entropy-Noise because random prefixes can trigger model-specific instability or hallucination.Self-Critique guides models into a more predictable state, producing more consistent detection behavior.
  • Result variability and error bars: Self-Critique shows stable SAT detection performance significantly above random guessing in bootstrap analysis.Table 12 reports 1,000 resamples with mean ± standard deviation and 95% confidence intervals.
  • Result variability and error bars: Baseline AUC fluctuates across models, while Self-Critique confidence intervals consistently surpass likelihood-based methods.For example, Min-K%++ varies from 0.21 to 0.42 on Qwen.
  • Inference cost: Active methods are necessary for valid RL-contamination detection, whereas passive likelihood-based methods are largely ineffective in this setting.RL’s reward-driven objective decouples contamination from the likelihood signals used by passive detectors.
  • Inference cost: Self-Critique offers the best active-method balance with two generations, while CDD requires many samples and Recall generally achieves lower AUC.With only two samples, CDD’s edit-distance estimate becomes noisy and unreliable.
  • Inference cost: Top-K entropy approximation with K = 5 makes entropy computation negligible relative to generation cost.It avoids materializing the full vocabulary distribution.
  • Limitations and future work: Evaluation primarily covers mathematical and logical reasoning tasks, leaving domain-specific behavior in code generation and other domains unresolved.The authors also note that policy-collapse dynamics at models larger than 7B parameters remain insufficiently understood.
Loading 2510.09259v2…