Source-linked AI summary
Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, Yanwei Fu, Qin Liu, Songyang Zhang, Qi Zhang
TL;DR
The paper asks whether reported gains from spurious rewards reflect reasoning or benchmark memorization. It audits contamination, introduces the clean RandomCalculation benchmark, and evaluates RL under different rewards. Correct rewards alone yield consistent improvements on clean problems, motivating uncontaminated and more diverse evaluations.
Problem
Random or incorrect rewards reportedly improve Qwen on standard math benchmarks but not Llama, leaving unclear whether the effect reflects data contamination or stronger mathematical capacity.
Method
The authors audit benchmark memorization, construct the clean RandomCalculation arithmetic benchmark, and train Qwen with RLVR under correct, random, and incorrect rewards.
Results
On clean RandomCalculation problems, correct rewards improve performance steadily, whereas random or incorrect rewards provide no reliable benefit and inverted rewards rapidly collapse performance.
Takeaways & Limitations
RL conclusions should be tested on uncontaminated benchmarks and, when feasible, across diverse model series because contaminated Qwen results can be unreliable.
Abstract
from arXiv · showhide
Reasoning in large language models has long been a central research focus, and recent studies employing reinforcement learning (RL) have introduced diverse methods that yield substantial performance gains with minimal or even no external supervision. Surprisingly, some studies even suggest that random or incorrect reward signals can enhance performance. However, these breakthroughs are predominantly observed for the mathematically strong Qwen2.5 series on benchmarks such as MATH-500, AMC, and AIME, and seldom transfer to models like Llama, which warrants a more in-depth investigation. In this work, our empirical analysis reveals that pre-training on massive web-scale corpora leaves Qwen2.5 susceptible to data contamination in widely used benchmarks. Consequently, conclusions derived from contaminated benchmarks on Qwen2.5 series may be unreliable. To obtain trustworthy evaluation results, we introduce a generator that creates fully clean arithmetic problems of arbitrary length and difficulty, dubbed RandomCalculation. Using this leakage-free dataset, we show that only accurate reward signals yield steady improvements that surpass the base model's performance boundary in mathematical reasoning, whereas random or incorrect rewards do not. Moreover, we conduct more fine-grained analyses to elucidate the factors underlying the different performance observed on the MATH-500 and RandomCalculation benchmarks. Consequently, we recommend that future studies evaluate models on uncontaminated benchmarks and, when feasible, test various model series to ensure trustworthy conclusions about RL and related methods.
1 Introduction
The paper investigates why spurious rewards appear to improve Qwen’s mathematical reasoning, finding that benchmark contamination can make memorization look like reasoning. A clean evaluation shows that only correct rewards produce reliable gains.
- Recent studies report that random or incorrect rewards improve Qwen on standard math benchmarks but provide little or no benefit to Llama3.1-8B.
- The authors test data contamination against the alternative hypothesis that Qwen’s stronger mathematical capacity makes noisy rewards effective.
- Qwen2.5-Math-7B regenerates 54.60% of truncated MATH-500 problems exactly and answers 53.6% of these incomplete prompts correctly.
- The authors caution that results from contaminated Qwen benchmarks may be unreliable and recommend uncontaminated benchmarks or more diverse model series.
- RandomCalculation generates clean arithmetic problems of arbitrary length from random operands and operators, ensuring instances post-date Qwen’s public release.
- On RandomCalculation, correct rewards steadily improve performance, while random rewards are unstable and inverse rewards rapidly degrade mathematical reasoning.
- Higher lexical overlap and lower token-level KL divergence after RL on MATH-500 indicate memory retrieval rather than newly acquired reasoning skills.
2 Related Works
Related work frames mathematical reasoning as a major application of RL and highlights the field’s concentration on Qwen-2.5. The cited literature spans verifiable, data-efficient, intrinsic, and pretraining-based approaches.
- Recent work uses verifiable rewards, test-time RL, and few-shot or unlabeled data to improve mathematical reasoning in language models.
- Some studies replace external supervision with intrinsic signals such as low-entropy output distributions.
- Math-specific pretraining and mid-training corpora have been reported to improve models’ mathematical capacity and benchmark performance.
3 Experimental Setup
The experiments compare Qwen and Llama checkpoints using contamination metrics, GRPO-based RLVR, and multiple reward signals. Evaluation includes partial-prompt memorization tests and benchmark accuracy under controlled configurations.
- The study evaluates four Qwen2.5 checkpoints and two comparably sized Llama3.1 checkpoints to isolate model-specific differences.
- Memorization is assessed with Partial-Prompt Completion Rate and Partial-Prompt Answer Accuracy.
- Partial-prompt completion measures ROUGE-L and Exact Match overlap between generated continuations and reference problem text.
- Partial-Prompt Answer Accuracy measures whether an unconstrained continuation from a truncated problem contains the ground-truth answer.
- GRPO samples groups of outputs from the old policy and maximizes a clipped policy objective with a KL-divergence penalty.
- The reward conditions include correct, random, inverted, and majority-voted incorrect signals, with greedy and pass@16 reference scores reported without templates.
4 Results & Analysis
Experiments show that apparent gains from spurious rewards on MATH-500 are confounded by template adaptation and benchmark memorization, whereas clean RandomCalculation problems improve steadily only with correct rewards.
- Spurious Rewards on MATH-500: Random and mv-incorrect rewards boost Qwen2.5-Math-7B on MATH-500 but have little or adverse effects on Llama3.1-8B-Instruct.The comparison uses the replicated RLVR setup and accuracy curves reported in Fig. 3.
- Spurious Rewards on MATH-500: Qwen2.5-Math-7B’s apparent RL gains largely converge to the Greedy (w/o Template) baseline, indicating template adaptation rather than new mathematical generalization.The base model’s initial accuracy is likely underestimated because it was not trained with dialogue-specific templates.
- Analysis of Memorization Capability: Qwen2.5-Math-7B reconstructs more than half of the remaining MATH-500 problem after receiving only the first 60% of its question, while similar contamination appears on AMC and AIME2024.With only 40% of the question shown, it still recovers 39.2% of MATH-500 problems.
- Analysis of Memorization Capability: RandomCalculation contains automatically generated arithmetic expressions with 1–20 steps, providing challenging, verifiable problems designed to avoid benchmark contamination.The expressions use integers, fractions, squares, and cubes derived from integers 0 to 100.
- Spurious Rewards on RandomCalculation: On 5-step and 10-step RandomCalculation, correct rewards improve performance steadily, random or incorrect rewards are unstable, and inverted rewards cause rapid collapse.Each sub-dataset contains 1,000 problems, with 700 for training and 300 for validation; Llama3.1-8B-Instruct shows similar findings.
- Spurious Rewards on RandomCalculation: On RandomCalculation, Qwen2.5-Math-7B surpasses Max@16 with correct rewards, while spurious rewards remain marginal or unstable and their improvements disappear as steps increase from 5 to 10.Llama3.1-8B-Instruct does not surpass Max@16 even with correct rewards and falls below greedy decoding with spurious rewards.
- More Evidence for Memorization: MATH-500 responses have substantially higher pre/post-RL similarity than RandomCalculation responses, and spurious rewards produce even higher ROUGE-L than correct rewards after RL.The analysis attributes the contaminated-benchmark surge to GRPO-driven retrieval of memorized answers rather than new reasoning paths.
5 Conclusion
The paper attributes Qwen’s apparent gains from spurious rewards primarily to benchmark contamination and memorization rather than genuine mathematical reasoning. Clean evaluation shows that only correctly aligned rewards produce consistent improvement, motivating uncontaminated benchmarks and rigorous evaluation.
- Qwen’s gains from spurious rewards were primarily driven by data contamination and memorization of benchmark problems.
- Only correctly aligned rewards yielded consistent performance improvements, whereas spurious rewards provided no meaningful benefit.
- The findings support using uncontaminated benchmarks and caution when interpreting results from datasets that may suffer from data leakage.
A Discussion and Limitation
The study acknowledges restricted experimental coverage and describes RandomCalculation as an automatically generated clean benchmark for evaluating reinforcement learning methods.
- A Discussion and Limitation: Experiments were restricted to a subset of commonly used Qwen2.5 and Qwen3 models because of limited computational resources.
- A Discussion and Limitation: Future work targets broader evaluation across diverse benchmarks, reinforcement-learning methods, and model families.
C Quantitative Analysis of Answer-Related Numeric Token Probabilities
Probability analyses show that Qwen retains answer-related information on MATH-500 after random-reward RL, whereas probabilities are more dispersed on clean RandomCalculation.
- MATH-500 answer-token probabilities remain consistently high before and after random-reward RL, suggesting retained answer memory.
- Answer-token probabilities are more dispersed on the clean 10-step RandomCalculation dataset than on MATH-500.
D Quantitative Results on Partial-Prompt Answer Accuracy
Partial-prompt evaluation compares Qwen2.5 and Llama3.1 across mathematics datasets and generation configurations, with expanded results reported in the appendix.
- Tab. 4 compares partial-prompt answer accuracy between Qwen-2.5 and Llama-3.1 on MATH-500.
- Additional tables report analogous results across several mathematics datasets.
- The evaluation uses Greedy and Avg@16 decoding, each with and without templates.
E Quantitative Results for Qwen3
Qwen3 exhibits pronounced benchmark memorization similar to Qwen2.5, despite its increased capacity. On clean LiveMathBench, correct rewards provide limited improvement, whereas random rewards produce unstable and declining performance.
- Qwen3 still exhibits pronounced evidence of data contamination despite increased model capacity.The reported pattern mirrors the memorization diagnostics observed for Qwen2.5.
- Correct rewards yield limited improvement on clean LiveMathBench, mainly because the training dataset is relatively small.
- Random rewards fail to produce stable gains on clean LiveMathBench and eventually lead to declining performance.
G Other Reasoning Domains
The paper extends its contamination analysis beyond mathematics to code and provides representative examples of memorization and RLVR outputs. Qwen2.5-Math-7B reproduces substantially more LiveCodeBench problems than Llama3.1-8B under the reported diagnostic.
- Other Reasoning Domains: 56.59% of LiveCodeBench problems are accurately reproduced by Qwen2.5-Math-7B when 80% of each problem is provided as the prompt.Under the same condition, Llama3.1-8B reproduces 4.40% of problems.
- Other Reasoning Domains: 4.40% of LiveCodeBench problems are reproduced by Llama3.1-8B when 80% of each problem is provided as the prompt.This is contrasted with Qwen2.5-Math-7B's 56.59% reproduction rate.
- Other Reasoning Domains: The paper presents memorization examples for Qwen2.5-7B and Qwen2.5-Math-7B across AMC, AIME2024, and MATH-500.It also shows a representative correct RLVR-fine-tuned Qwen2.5-Math-7B response on the 5-step RandomCalculation benchmark.
- Other Reasoning Domains: The experiments use NVIDIA A800 80G GPUs, with RL training performed on 8 GPUs and 512 GB of RAM.The reported environment uses Ubuntu 20.04.6 LTS and CUDA 12.4.
J RLVR Training Details
The paper reports RLVR configuration details, token-level probability comparisons, benchmark setups, memorization diagnostics, and representative outputs. These materials cover reward-signal experiments, prompt-prefix evaluations, and examples from mathematics and code tasks.
- RLVR Training Details: RLVR training uses a learning rate of 5e-7, temperature 1.0, 16 samples per prompt, and a batch size of 128.FlashAttention is enabled to improve efficiency and reduce memory usage.
- RLVR Training Details: Figure 9 compares pre- and post-RL probabilities for answer-relevant numeric tokens on MATH-500 and 10-step RandomCalculation under random rewards.Each benchmark contributes 100 randomly selected problems, and probabilities are evaluated on numeric tokens appearing in generated answers.
- RLVR Training Details: Figure 10 evaluates Qwen2.5-Math-7B on LiveMathBench under correct and random reward signals.The dataset combines LiveMathBench versions v202412 and v202505.
- RLVR Training Details: Table 14 reports Exact Match and ROUGE-L scores for LiveCodeBench test generation under different prompt-prefix ratios in greedy decoding.
- RLVR Training Details: Figures 11–15 provide partial-prompt memorization examples on AMC, AIME2024, and MATH-500 using the first 40% of each original problem.The generations use the Greedy (w/o Template) configuration.
- RLVR Training Details: The diagnostic examples include arithmetic, complex-number, divisor-counting, triple-counting, taxicab-distance, and LiveCodeBench problems.Several passages present prompts, answers, or solution attempts for these examples.
- RLVR Training Details: Figure 16 shows a correct output from Qwen2.5-Math-7B after 300 RLVR training steps with correct rewards on the 5-step RandomCalculation dataset.