PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting RLVR Training Data via Structural Convergence of Reasoning

Hongbo Zhang, Yue Yang, Jianhao Yan, Guangsheng Bao, Yue Zhang, Yue Zhang

arXiv:2602.11792v1cs.AIcs.CL

TL;DR

The paper addresses how to detect whether examples appeared in RLVR training when conventional likelihood-based signals are ineffective. It measures structural convergence by sampling multiple completions and computing Min-kNN Distance, which reliably separates RL-seen from unseen examples and outperforms baselines. The method requires only black-box sampling but assumes structural collapse is observable and incurs multi-completion generation costs.

  • Problem

    Detecting RLVR training-data exposure is difficult because reward feedback on self-generated reasoning trajectories makes conventional token-level and likelihood-based signals ineffective.

  • Method

    Min-kNN Distance samples multiple completions and quantifies their structural diversity through nearest-neighbor edit distances without token log probabilities or reference models.

  • Results

    Min-kNN Distance reliably distinguishes RL-seen from unseen examples and achieves the highest AUC across evaluated models, averaging 0.70 and improving 17% relatively over the strongest baseline.

  • Takeaways & Limitations

    RLVR exposure leaves a structural behavioral signature that can support black-box contamination detection across models and RL algorithms.

  • Takeaways & Limitations

    The method assumes RLVR-induced structural collapse is observable across models and tasks, requires multiple completions per prompt, and does not directly address training-data bias.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) is central to training modern reasoning models, but the undisclosed training data raises concerns about benchmark contamination. Unlike pretraining methods, which optimize models using token-level probabilities, RLVR fine-tunes models based on reward feedback from self-generated reasoning trajectories, making conventional likelihood-based detection methods less effective. We show that RLVR induces a distinctive behavioral signature: prompts encountered during RLVR training result in more rigid and similar generations, while unseen prompts retain greater diversity. We introduce Min-$k$NN Distance, a simple black-box detector that quantifies this collapse by sampling multiple completions for a given prompt and computing the average of the $k$ smallest nearest-neighbor edit distances. Min-$k$NN Distance requires no access to the reference model or token probabilities. Experiments across multiple RLVR-trained reasoning models show that Min-$k$NN Distance reliably distinguishes RL-seen examples from unseen ones and outperforms existing membership inference and RL contamination detection baselines.

1. Introduction

RLVR improves reasoning performance but creates exposure-detection challenges because it trains on self-generated trajectories rather than token-level likelihood targets. The paper identifies structural convergence in generations and introduces Min-kNN Distance as a black-box detector for RLVR-seen examples.

  • Motivation: RLVR trains models with verifiable rewards on reasoning-intensive mathematics, coding, and symbolic problem-solving tasks.The training process reinforces trajectories that produce correct final answers.
  • Motivation: Benchmark performance may not reliably generalize to newer, less exposed, expert-level, or reformulated problems.These patterns raise concerns that some post-training gains reflect over-specialization rather than robust reasoning ability.
  • Detection gap: Conventional perplexity and token-level statistics fail to capture reasoning-pattern changes induced by RLVR.RLVR optimizes self-generated reasoning trajectories through reward feedback rather than golden trajectories.
  • Structural signature: RLVR-trained models produce increasingly similar generations for seen prompts, whereas unseen prompts retain higher variability across recurring structural modes.The convergence particularly affects symbolic and algebraic reasoning components.
  • Proposed detector: Min-kNN Distance samples multiple completions and computes edit-distance-based structural diversity without token log probabilities or reference models.Seen prompts yield smaller values than unseen prompts because RLVR-induced convergence makes their completions more similar.
  • Evaluation: Experiments across diverse reasoning models, RL algorithms, and training setups show reliable separation of RL-seen and unseen examples.The evaluation includes paraphrasing, distillation, and varying decoding configurations.

2. Preliminary

The paper formulates RLVR exposure detection as membership inference for an unknown training dataset and describes the reward-driven generation process. RLVR updates the distribution of self-generated chains of thought, motivating diversity-based analysis rather than likelihood comparison.

  • Problem definition: The task is to determine whether a query input belongs to an unknown RLVR training dataset.The detector maps a query and RLVR-tuned model to a binary membership prediction.
  • Problem definition: This setting differs from standard membership inference because RLVR is reward-driven and self-generative.The distinction applies relative to pretraining and supervised fine-tuning scenarios.
  • RLVR training process: For each prompt, the model samples a chain of thought and final answer from its policy, producing an output o = (c, ˆa).The output combines the generated reasoning and answer.
  • RLVR training process: A scalar reward based on answer correctness guides parameter updates that increase expected reward.The optimization is defined over sampled reasoning-and-answer trajectories.
  • Implication: Because RLVR adjusts self-generated reasoning trajectories rather than golden chains of thought, likelihood-based comparison is unavailable or ineffective.This motivates analyzing changes in the distribution and diversity of generated reasoning.

3. Analyzing Reasoning Pattern under RLVR

RLVR narrows reasoning trajectories into recurring structural modes, especially in symbolic and algebraic segments, with stronger rigidity for seen prompts than unseen prompts.

  • Generation diversity: RLVR training reduces lexical, logical, and semantic diversity across completions for fixed prompts under both DAPO and GRPO.Across 300 training prompts, 32 completions were sampled at each checkpoint, and all three diversity metrics steadily declined.
  • Generation diversity: The global output distribution also becomes less diverse, concentrating reasoning into recurring patterns as training progresses.This reduction reflects collapse of the reasoning space into more rigid outputs.
  • Symbolic reasoning: Symbolic logic fragments increase rapidly during training, while restatement and boilerplate patterns grow more slowly.The analysis extracted high-frequency 3-grams from 50 training prompts at each checkpoint and classified them with GPT-4o.
  • Symbolic reasoning: RLVR primarily collapses symbolic reasoning segments, including algebraic manipulations, standardized transformations, and function definitions.Repeated 3-grams were categorized as problem restatements, boilerplate connectors, or symbolic and algebraic logic steps.
  • Structural modes: Reasoning does not become a single deterministic path; instead, completions converge into a small set of rigid structural patterns.Most prompts exhibit two to four stable reasoning clusters, with some prompts showing more.
  • Seen versus unseen data: Seen prompts show stronger rigidity than unseen prompts, with fewer reasoning structures and more symbolic reasoning steps.Both groups converge, but training prompts have a higher proportion of clusters containing fewer reasoning structures, whereas unseen prompts retain more variability.

4. Min-kNN Distance: A Simple Black-Box Detector for RLVR Exposure

Min-kNN Distance detects RLVR training exposure by measuring structural similarity among multiple sampled completions. It uses the smallest nearest-neighbor edit distances because seen prompts are expected to produce more compact structural modes.

  • Detector: Min-kNN Distance quantifies completion clustering for membership inference using a simple black-box statistic.It is motivated by the tendency of RLVR-seen prompts to produce completions that converge into a few tight clusters.
  • Computation: For a prompt x, the method samples m completions from the RLVR-tuned model and computes pairwise normalized edit distances.The completion set is O(x) = {o_1, o_2, ..., o_m}, with each completion sampled from the model conditioned on x.
  • Computation: Each completion receives a nearest-neighbor distance based on its smallest normalized Levenshtein distance to another completion.Normalized Levenshtein distance counts insertions, deletions, and substitutions relative to the longer completion.
  • Detection score: The detection score averages the k smallest values in the sorted list of completion nearest-neighbor distances.This focuses the statistic on the most tightly clustered sampled completions.
  • Detection score: Seen prompts are expected to have smaller Min-kNN Distance values than unseen prompts because RLVR collapse creates compact structural modes.A detector is obtained by thresholding the score.
  • Access requirements: The method requires only black-box sampling access and does not use token log probabilities or a reference model.This design avoids reliance on likelihood-based information.

5. Experiments

Experiments evaluate Min-kNN Distance across RLVR-trained models, controlled contamination settings, baselines, decoding choices, paraphrased prompts, distillation prompts, and code/math data. The method consistently outperforms baselines, remains robust across algorithms and scales, and benefits from suitable sampling configurations.

  • Main results: 0.70 average AUC represents a 17% relative improvement over the strongest baseline, with Min-kNN Distance achieving the highest AUC across evaluated models.Probability-based and consistency-based baselines often show unstable or near-random performance.
  • Main results: Min-kNN Distance remains effective across GRPO, DAPO, and PPO models and from 1.5B to 32B parameters.The reported stability suggests the signal is not dependent on a particular RL algorithm or model scale.
  • Ablation: Performance improves rapidly with k before saturating, increases with more sampled completions before plateauing, and strengthens at higher decoding temperatures.Lower temperatures reduce output variability and can obscure structural collapse; lower temperatures also make the score more sensitive to k.
  • Robustness analyses: Paraphrasing reduces AUC only slightly, from 0.72 to 0.71, indicating that detection is not tied to surface-level prompt forms.The paraphrased-prompt evaluation uses 300 RLVR training prompts under the DAPO-Qwen-32B setting.
  • Robustness analyses: Min-kNN Distance achieves AUC 0.80 on math and AUC 0.69 on coding distillation prompts, while code contamination detection is more challenging.The passage attributes the difference likely to coding’s greater diversity and flexibility compared with math’s more structured nature.
  • Contamination analysis: In the dual-stage contamination analysis, Min-kNN Distance performs significantly better on the lower-pretraining-contamination subset than on the random-control subset.The experiment injects test items into PPO training and compares subsets matched for sample size.

6. Related Work

Prior work finds that reinforcement learning can compress generation diversity, while existing exposure detectors mainly target likelihood traces from pretraining or supervised fine-tuning. These findings motivate structural analysis for RLVR exposure.

  • Data exposure detection: Existing data-exposure methods mainly target pretraining or supervised fine-tuning, where exposure can produce low perplexity or probability outliers.The related work frames these statistical signals as distinct from RLVR’s reward-based training process.
  • Generation diversity: Preference and alignment reinforcement learning has been associated with reduced lexical, syntactic, semantic, and conceptual generation diversity.This literature describes compression of the model’s output space relative to supervised fine-tuning or non-preference methods.

7. Conclusion

The paper identifies structural collapse in RLVR reasoning trajectories and proposes Min-kNN Distance as a black-box exposure detector. Across models and RL algorithms, it reports consistent baseline superiority without requiring training data or model internals.

  • Conclusion: The work identifies structural collapse in reasoning trajectories during RLVR and proposes Min-kNN Distance to leverage that signal.The method is presented as a black-box detection approach for training-data exposure.
  • Conclusion: Min-kNN Distance requires no access to training data or model internals and consistently outperforms baseline methods across models and RL algorithms.The paper states that limitations are outlined in Appendix D.

Impact Statement

The work targets RLVR exposure detection as a step toward greater transparency in machine learning. It frames contamination detection as relevant to the reliability and fairness of model evaluations.

  • Impact: The proposed detection focus is intended to improve transparency by enabling better identification of data contamination in AI models.The impact statement connects this goal with more reliable and fair evaluations.
  • Impact: The statement highlights understanding and mitigating potential biases as important as AI expands into critical applications.This claim is presented as an ethical consideration alongside the paper’s detection focus.

A.1. Training Details

The analysis trains Qwen-2.5-7B-Base with GRPO and DAPO, then examines generation diversity and repeated n-grams across model answers. Additional experiments use instructed reasoning models and standardized sampling configurations.

  • Training setup: Qwen-2.5-7B-Base is trained with GRPO and DAPO on DAPO training data for 200 rollout steps, approximately six epochs.The corresponding training hyperparameters are reported in Table 6.
  • Diversity analysis: Generation diversity is evaluated across lexical, logical, and semantic dimensions using EAD, NLI, and embedding metrics.Figure 8 measures diversity in a cross-input setting during RLVR training under DAPO and GRPO.
  • Repeated n-gram analysis: Repeated n-grams in single-prompt completions are categorized as restatement, logic, boilerplate, or other tokens, with darker shading indicating higher frequency.Figure 9 shows three GRPO answers and highlights n-grams appearing more than eight times at the final checkpoint.
  • RL-MIA setup: RL-MIA experiments use Qwen2.5-7B-Instruct and Deepseek-Math-7B-Instruct, with data splits and hyperparameters reported in Tables 7 and 8.These models and settings follow Tao et al. (2025).
  • Sampling configuration: Sampling uses vLLM on eight H800 GPUs, with temperature 0.7, top p 0.95, and a maximum output length of 1024 tokens.The stated token cap provides sufficient detection performance while reducing detection time.

C. Cost Analysis

Min-kNN Distance has measurable sampling overhead because it generates multiple completions, but the reported latency remains comparable to existing detection methods. Its average cost is reported for 32 completions per sample.

  • Detection latency: 6.65 seconds per item is the average detection time for Min-kNN Distance using 32 completions on Open-Reasoner-Zero-7B.The latency is higher than methods such as PPL and DIME because structural diversity analysis requires multiple sampled completions.
  • Practical trade-off: The reported Min-kNN Distance latency is considered reasonable and comparable to existing methods for RLVR data-exposure detection.The paper frames the additional time as an expected cost of sampling multiple completions.

D. Limitations

The method’s scope depends on structural collapse being observable across models and tasks, and its multi-completion procedure increases computational cost. It also targets structural collapse rather than broader training-data concerns such as bias.

  • Scope: Min-kNN Distance assumes that RLVR-induced structural collapse is observable across various models and tasks.This assumption limits how broadly the method’s effectiveness can be generalized.
  • Computational and analytical limits: Generating multiple completions per prompt can increase computational cost, while the method does not directly address potential biases in training data.The latter boundary distinguishes structural-collapse detection from broader data-quality analysis.
Loading 2602.11792v1…