PaperlayerView exact revision on arXiv

Source-linked AI summary

Spurious Rewards: Rethinking Training Signals in RLVR

Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, Luke Zettlemoyer

arXiv:2506.10947v2cs.AIcs.LG

TL;DR

The paper asks why RLVR can improve mathematical reasoning despite rewards with little, no, or negative correlation with correctness, and investigates this using varied reward signals and GRPO analyses. It finds that clipping bias can amplify high-prior behaviors, producing large Qwen gains from random rewards, but these effects are model-dependent and do not reliably generalize.

  • Problem

    The mechanisms behind RLVR’s reasoning gains remain poorly understood, particularly when rewards carry little to no task-relevant signal.

  • Method

    The paper trains Qwen2.5-Math models with GRPO under ground-truth, weak, random, and incorrect rewards, then analyzes GRPO clipping and emergent reasoning behaviors.

  • Results

    Spurious rewards significantly improve Qwen2.5-Math performance, including a 21.4% MATH-500 gain from random rewards versus 29.1% from ground-truth rewards.

  • Takeaways & Limitations

    RLVR evaluations should use diverse models and dummy baselines because gains from spurious rewards can reflect amplified pretrained behaviors rather than genuine capability improvements.

  • Takeaways & Limitations

    Spurious rewards are introduced for controlled analysis and are not advocated as strategies for developing model capabilities.

Abstract

from arXiv · show

We show that reinforcement learning with verifiable rewards (RLVR) can elicit strong mathematical reasoning in certain language models even with spurious rewards that have little, no, or even negative correlation with the correct answer. For example, RLVR training with GRPO improves MATH-500 performance for Qwen2.5-Math-7B by 21.4 percentage points using randomly assigned rewards, nearly matching the 29.1-point gain from ground-truth rewards. To explain this counterintuitive observation, we show that GRPO exhibits a clipping bias from the clip term, which can amplify high-prior behaviors learned during pretraining even without informative rewards. As a case study, we identify one such behavior in Qwen2.5-Math models, which we call code reasoning -- reasoning in code without actual code execution; code-reasoning frequency increases from 65 percent to over 90 percent with spurious rewards. However, the presence of such amplifiable behaviors is highly model-dependent. In practice, spurious rewards that are effective for Qwen models often fail to produce gains for other model families, such as Llama3 or OLMo2. Our results highlight the importance of validating RL methods across diverse models rather than relying on a single de facto choice: large gains can arise on Qwen models even from random rewards that do not reflect genuine capability improvements.

1. Introduction

RLVR can substantially improve Qwen2.5-Math reasoning even with rewards carrying little or no task-relevant signal, while the effect depends on the model. The paper attributes these gains to GRPO amplifying high-prior behaviors and highlights code reasoning as one such behavior.

  • Motivation: 21.4% absolute MATH-500 accuracy gain from randomly assigned rewards nearly matches the 29.1% gain from ground-truth rewards on Qwen2.5-Math models.The paper presents this as evidence that substantial RLVR gains can arise from rewards with little to no task-relevant signal.
  • Mechanism: GRPO’s clipping function systematically amplifies tokens with high prior probabilities in the base model.This clipping bias is proposed as a mechanism for spurious improvements without informative rewards.
  • Case study: Code reasoning—reasoning chains expressed in Python without code execution—rises to roughly 90% or higher under spurious rewards.Code reasoning is highly predictive of performance, with 60.9% accuracy when present versus 28.0% without it.
  • Model dependence: Spurious rewards that improve Qwen models do not work reliably for Llama3.1-8B-Instruct and OLMo2-7B, which have different reasoning priors.The paper therefore treats model dependence as central to interpreting these gains.
  • Implications: The authors recommend confirming RLVR findings on other models and using spurious rewards as dummy baselines, not as capability-development strategies.They state that spurious rewards are introduced for analytical purposes and should not be treated as recommended training methods.

2. Spurious Rewards Yield Significant RLVR Gains

The study replaces ground-truth RLVR rewards with weaker, random, and deliberately incorrect signals and finds significant math gains for Qwen2.5-Math. These gains often approach ground-truth results, although convergence and performance vary by reward and benchmark.

  • Experimental setup: GRPO training replaces verifiable ground-truth rewards with weak or spurious binary rewards to test whether task-relevant supervision is necessary.The reward family includes majority-vote, format, random, and incorrect-label conditions.
  • Results: Qwen2.5-Math models significantly improve on MATH and AMC across meaningful and spurious reward signals.Figure 2 reports pass@1 for MATH and average@8 for AMC, with curves smoothed over a window of 10.
  • Conditions: Random rewards converge more slowly, while Qwen2.5-Math-1.5B gains from them are slower and reach only 4.9% on AMC.Other spurious rewards often improve performance within a few points of ground-truth rewards.
  • Results: 24.1% MATH-500 gain from incorrect-label rewards on Qwen2.5-Math-7B approaches the gain from ground-truth RLVR.The reported ground-truth comparison is a 29.1% gain.
  • Random rewards: Random rewards produce a 21.4% MATH-500 boost and AMC gains of 21.4%, despite providing no guidance in the rewarding process.The random reward assigns 1 with probability γ and 0 otherwise; γ = 0 yields no change analytically.
  • Interpretation: The findings support the hypothesis that RLVR at open-source post-training scales may trigger latent reasoning capabilities rather than teach entirely new ones.The paper presents the simple-reward results as additional evidence for this emerging hypothesis.

3. (Lack of) Generalization to Other Models

Spurious-reward gains are strongly model-family dependent: they recur across Qwen2.5 models but rarely help non-Qwen families. The authors therefore evaluate RLVR methods across diverse models rather than relying on Qwen results alone.

  • Cross-family results: Spurious rewards benefit Qwen2.5 models but rarely help non-Qwen models.Across Qwen2.5, non-random rewards improve MATH-500, whereas OLMo remains flat under spurious rewards and gains mainly from ground-truth rewards.
  • Family and scale effects: Models within each family behave similarly, while smaller models benefit less from spurious rewards such as random rewards.The paper conjectures that shared pretraining distributions and stronger retained priors contribute to these patterns.
  • Generalization: Each weak or spurious reward fails to help at least one other model and can remain flat or become harmful.This indicates that reward effectiveness does not reliably transfer across model families.
  • Implications: The authors argue that proposed RLVR reward signals should be tested on diverse models.They motivate this recommendation by noting that many recent reasoning studies draw conclusions primarily from Qwen gains.

4. Understanding Training Signals from Random Rewards in GRPO

Random rewards can improve Qwen2.5-Math performance because GRPO’s clipping term creates a biased signal that favors already-likely token behaviors, even when rewards are uninformative. Removing clipping eliminates these consistent gains.

  • Spurious rewards generally benefit Qwen2.5 models but yield substantially smaller or no gains for other model families.Similar reward-dependent differences appear on AMC, while Qwen models show the strongest improvements.
  • Random rewards alone do not yield effective training when GRPO’s clipping term is removed.This holds across three no-clipping variants, including implementation ablation, matched mini-batch and rollout sizes, and single-update rollouts.
  • With clipping enabled, random rewards produce stable and consistent performance improvements across random seeds.The effect persists across different random reward distributions, indicating stability.
  • Clipping introduces a reward-independent gradient bias based on deviations between current-policy and behavior-policy token probabilities.The expected gradient depends on whether the importance ratio is below, within, or above the clipping interval.
  • High-prior tokens can be amplified because upper clipping may be unreachable for tokens whose behavior-policy probability is already high.With ϵc = 0.2 and prior probability 0.85, the upper threshold is 1.02, exceeding the maximum possible probability.

5. Case Study: Code Reasoning as a Representative Pre-Existing Behavior

The paper uses code reasoning as a representative pre-existing strategy to explain why spurious RLVR rewards affect model families differently. In Qwen2.5-Math, spurious-reward training and direct interventions increase code reasoning alongside performance, whereas other models often do not benefit.

  • 5.1. Different Models Exhibit Pre-existing Reasoning Strategy Discrepancies: Code reasoning is treated as a controlled, observable example of how RLVR interacts with pre-existing reasoning strategies, not as a complete explanation.The paper also identifies lexical repetition and other behaviors as potentially relevant.
  • 5.1. Different Models Exhibit Pre-existing Reasoning Strategy Discrepancies: Qwen2.5-Math-7B uses code reasoning in 65.0% of MATH-500 responses before RLVR, and code-containing responses are more accurate than natural-language-only responses.Other models either do not generate code or generate it without the same performance benefit.
  • 5.2. RLVR with Spurious Rewards Upweight Pre-existing Reasoning Strategies: Under weak or spurious rewards, code-reasoning frequency rises from 65.0% to about 90% within 15 steps and reaches 95.6% with random rewards, tracking accuracy gains.With ground-truth rewards, code frequency later declines as natural-language reasoning improves; Bad-Code models show the opposite relationship.
  • 5.3. Intervening Explicitly on Code Reasoning Frequency: Prompting Qwen2.5-Math models to begin with Python reasoning improves MATH-500 accuracy by 24.2%, 15.0%, and 10.0% for the 1.5B, 7B, and 1.5B variants, respectively.The same intervention degrades performance for models such as Llama and OLMo2.
  • 5.3. Intervening Explicitly on Code Reasoning Frequency: A Python reward raises Qwen2.5-Math-7B code reasoning above 99% within 20 training steps and primarily improves Qwen2.5-Math models.RLVR-based induction matches or exceeds prompting gains for these models, while other models show limited improvement.

6. Related Work

Prior work has applied reinforcement learning to language-model alignment and mathematical reasoning, including unsupervised or weakly supervised approaches that construct pseudo-rewards without ground-truth labels.

  • Unsupervised approaches include preference optimization for consistent answers and test-time reinforcement learning using majority voting to estimate pseudo-rewards.
  • RLHF is widely used to align language models with human preferences, while RLVR targets tasks with deterministic answers.

7. Conclusion

The paper finds that RLVR outcomes depend strongly on base-model pretraining and that spurious supervision can enhance existing reasoning behaviors. It therefore emphasizes broader evaluation across models and reward baselines, while framing its impact as improving research rigor rather than enabling applications.

  • Base-model pretraining significantly affects RLVR outcomes, and spurious supervision can enhance useful existing behaviors.
  • Effects observed in one model family may not generalize to others, motivating evaluation across models with differing pretraining distributions.
  • The study recommends testing multiple baselines, including format and random rewards, when evaluating reinforcement learning techniques.
  • The work does not advocate spurious rewards for capability improvement because misuse in deployed systems could produce unreliable or undesirable behavior.
  • The anticipated impact is improving the rigor and robustness of reinforcement learning research rather than enabling new applications.

A.7. Experimental Setups for TTRL and One-Shot RL

This setup section describes evaluations of weak and random supervision, including GRPO with Bernoulli random rewards. Random rewards are theoretically mean-zero in normalized advantage, yet non-zero reward probabilities still improve Qwen2.5-Math-7B after exploration.

  • Experimental setups: TTRL and one-shot RL use weak supervision, but their reported accuracies and setups have important comparability constraints.TTRL trains on unlabeled test prompts, whereas the paper trains on distinct training prompts; compute constraints also limit hyperparameter sweeps.
  • Random rewards: Random rewards are assigned independently of model rollouts, and the analysis investigates clipping bias as a possible training signal.
  • Random rewards: All non-zero Bernoulli reward probabilities produce significant gains for Qwen2.5-Math-7B after an initial exploration period, with performance comparable to ground-truth rewards.The tested probabilities are γ ∈ {0.7, 0.5, 0.3, 0.001, 0}; γ = 0 produces no learning.
  • Random rewards: For independent random rewards, normalized group-relative advantages have expectation zero.The derivation assumes zero advantage when the group reward standard deviation is zero.
  • Random rewards: GRPO clipping can introduce bias even when the random-reward advantage is independent of the rollout.The clipping term is analyzed as a source of training bias under random rewards.

B.1.2. GRADIENT DERIVATION OF CLIPPING BIAS UNDER RANDOM REWARDS

The derivation defines clipping bias as the expected-gradient difference caused by adding GRPO clipping and shows how clipping selectively preserves updates based on advantage sign and importance-ratio thresholds. This mechanism favors high-probability behaviors, explaining gains for Qwen models but not generally for other families.

  • Clipping bias: Clipping bias is defined as the difference between expected gradients with and without clipping in the GRPO objective.
  • Gradient derivation: Under random rewards, unclipped policy gradients have zero expectation because normalized advantages are independent of other variables and mean-zero.
  • Gradient derivation: The clipped gradient is retained or zeroed according to advantage sign and whether the token importance ratio crosses 1 − ϵc or 1 + ϵc.
  • Clipping bias: The clipping bias produces a positive gradient when Rθ > 1 + ϵc and discourages the model from leaving the clipping region.
  • Model-prior amplification: Because higher-probability tokens have wider clipping ranges and fewer penalties, clipping concentrates probability mass on the existing distribution.
  • Empirical mechanism: With clipping, average token probability mass increases and code-reasoning frequency rises, while disabling clipping keeps average token probability relatively constant.
  • Model dependence: Clipping benefits Qwen2.5-Math because it amplifies pre-existing code reasoning correlated with correctness, whereas no-code and bad-code models lack an effective pattern to amplify.Qwen code reasoning occurs in 65% of responses at 64% accuracy versus 29% without code, and rises above 90% after training.
  • Clipping ablation: Disabling clipping through batch-size adjustment keeps performance within a consistent range, whereas removing clipping directly produces extreme stochasticity across runs.

F. Compound Rewards that Inhibit Code Reasoning

Compound rewards that inhibit code reasoning reveal that the usefulness of elicited behaviors depends on the model: suppressing code can reduce Qwen gains but improve results for models where code reasoning is suboptimal. No-repetition rewards provide a second example of a behavior that can be elicited without informative rewards.

  • Code inhibition: Penalizing code reasoning can eliminate format-reward gains for Qwen2.5-Math-7B, while some incorrect-reward gains persist on harder benchmarks but are reduced.
  • Code inhibition: Compound rewards improve Bad-Code models by downweighting code reasoning that is suboptimal for those models.OLMo2-7B-SFT gains +8.9 and +5.5 points when no-code reward is added to standalone format and incorrect rewards, respectively.
  • Cross-model results: Qwen2.5-Math-1.5B follows the same overall compound-reward trends as Qwen2.5-Math-7B, while gains on Qwen2.5-1.5B are comparable to its original reward.
  • No-repetition reward: Qwen2.5-Math models show higher repetition tendencies than Llama3 and OLMo2, motivating a reward that penalizes strings repeated more than 10 times.
  • No-repetition reward: No-repetition rewards improve Qwen2.5-Math-7B and Qwen2.5-Math-1.5B on MATH and AMC but yield minimal or negative improvement on other models.

H. Switch of Reasoning Strategies during RLVR

RLVR changes which reasoning strategy Qwen models use, with spurious-reward gains largely associated with switching from natural-language to code reasoning. Prompt choice also affects performance and training behavior, including when prompts contain little task-relevant information.

  • Performance decomposition: Qwen2.5-Math-7B’s accuracy increases by an average of 23.5 absolute points across training signals.The analysis separates Code→Code, Code→Lang, Lang→Code, and Lang→Lang subsets and measures each subset’s frequency and accuracy.
  • Reasoning-strategy switches: Most originally language-reasoning problems switch to code reasoning after weak or spurious-reward RLVR.Ground-truth reward does not follow this pattern, while meaningful rewards can steer Bad-Code models away from code reasoning.
  • Partial contributions: 58.3% of Qwen2.5-Math-7B’s performance gain comes from Lang→Code problems, rising to 78.7% for Qwen2.5-Math-1.5B.For Qwen2.5-7B, Code→Lang contributes 93.9% of the performance gain.
  • Partial contributions: When models do not use code reasoning before or after RLVR, performance gains or losses come from the Lang→Lang subset.These results support interpreting spurious-reward gains in Qwen models as elicitation of a reasoning strategy.
  • Prompt sensitivity: Prompt choice affects both initial performance and RLVR trajectories, and a spurious prompt produces the highest MATH-500 accuracy despite lacking task-related information.The spurious prompt also yields the highest initial performance, while Sober produces the highest parsable-answer rate.
  • Prompt sensitivity: Models trained with several task-specific prompts converge to similar performance after RLVR, suggesting prompting elicits only a subset of behaviors available to RLVR.The authors report this as a conjecture based on the observed training trajectories.

J. Additional Results on Models That Have Undergone RL Training

Additional experiments show that prior RL training changes how models respond to further RLVR. Qwen Instruct models show minimal improvement in this setup, while Llama-3.1-Tulu-3-8B retains limited gains.

  • Qwen Instruct models: Qwen Instruct models show minimal improvement from RLVR across reward types, even with ground-truth rewards.The authors conjecture that prior RL training may have already brought these models to saturation.
  • Llama-3.1-Tulu-3-8B: Llama-3.1-Tulu-3-8B gains only marginally on MATH and limitedly on ACM and AIME with ground-truth rewards.Its trends resemble those of the base Llama-3.1-8B model, although it still shows gains on MATH.

K. Qualitative Analysis on Qwen2.5-Math-7B’s Coding Reasoning Behaviors

Qwen2.5-Math-7B can solve sampled math problems by generating code and predicting its execution outputs without an external interpreter. This behavior is robust to numerical perturbations but may disappear after narrative rephrasing.

  • Code reasoning: Qwen2.5-Math-7B conducts code reasoning by writing code and predicting execution outputs without external code execution.The qualitative examples report relatively high accuracy for predicted execution outputs.
  • Robustness to perturbation: The model generates similar code snippets for numerically perturbed versions of a question.One reported perturbed example derives the correct answer 4344.
  • Narrative sensitivity: After alternative narrative rephrasing, Qwen2.5-Math-7B uses natural-language solutions rather than code reasoning.The model first identifies the same core question in its answer before switching away from code.
  • Additional reward analyses: Compound rewards combine the original rewards with a no-Python reward that favors responses without Python code.Additional results cover smaller 1.5B Qwen models and AMC and AIME benchmarks.
  • Additional reward analyses: A no-repetition reward assigns 1 to responses without obvious repetition and 0 to responses with obvious string repetition, improving Qwen2.5-Math but not other models.The figure caption reports this model-family-specific effect.
Loading 2506.10947v2…