Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward

Peter Chen; Xiaopeng Li; Ziniu Li; Wotao Yin; Xi Chen; Tianyi Lin

Source-linked AI summary

Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward

Peter Chen, Xiaopeng Li, Ziniu Li, Wotao Yin, Xi Chen, Tianyi Lin

arXiv:2512.16912v3Dec 18, 2025cs.LGcs.AIcs.CL

TL;DR

RLVR exhibits puzzling gains from both spurious rewards and entropy reduction, despite their opposing implications for exploration and exploitation. This paper analyzes clipping, entropy, and reward misalignment, finding that clipping reduces entropy but does not itself provide a useful performance signal, while random-reward gains extend beyond contaminated Qwen-Math settings.

Problem
It remains unclear whether policy entropy causally affects RLVR performance and whether spurious-reward gains arise from clipping bias, model contamination, or their interaction.
Method
The paper combines theoretical bounds on clipping bias and a policy-entropy shift analysis with experiments across multiple model families, sizes, and training variants.
Results
Random rewards can improve performance for stronger models, while clipping reduces entropy without supplying a meaningful learning signal and entropy reduction alone does not ensure gains.
Takeaways & Limitations
Spurious-reward benefits reflect a broader RLVR exploration–exploitation dynamic rather than clipping bias or contamination alone.
Takeaways & Limitations
The standard entropy approximation is limited because it ignores higher-order terms and assumes an unclipped formulation.

Abstract

from arXiv · show

This paper examines the exploration-exploitation trade-off in reinforcement learning with verifiable rewards (RLVR), a framework for improving the reasoning of Large Language Models (LLMs). Recent studies suggest that RLVR can elicit strong mathematical reasoning in LLMs through two seemingly paradoxical mechanisms: spurious rewards, which suppress exploitation by rewarding outcomes unrelated to the ground truth, and entropy minimization, which suppresses exploration by pushing the model toward more confident and deterministic outputs, highlighting a puzzling dynamic: both discouraging exploitation and discouraging exploration improve reasoning performance, yet the underlying principles that reconcile these effects remain poorly understood. We focus on two fundamental questions: (i) how policy entropy relates to performance, and (ii) whether spurious rewards yield gains, potentially through the interplay of clipping bias and model contamination. Our results show that clipping bias under spurious rewards reduces policy entropy, leading to more confident and deterministic outputs, while entropy minimization alone is insufficient for improvement. We further propose a reward-misalignment model explaining why spurious rewards can enhance performance beyond contaminated settings. Our findings clarify the mechanisms behind spurious-reward benefits and provide principles for more effective RLVR training.

1 INTRODUCTION

The paper investigates why both spurious rewards and entropy reduction can improve RLVR reasoning despite discouraging different sides of exploration–exploitation. It analyzes clipping, policy entropy, and reward misalignment to reconcile these effects.

RLVR differs from classical reinforcement learning because rewards are sparse, outcome-level, and verifiable only after long language-model rollouts.Intermediate token actions are reward-equivalent, making the exploration–exploitation regime distinct from standard per-step-reward settings.
Prior work reports both gains from spurious rewards and gains from entropy minimization, creating a tension between suppressing exploitation and suppressing exploration.The paper frames this as an unresolved mechanism-level puzzle rather than assuming either effect is universally beneficial.
Clipping under random rewards reduces policy entropy and drives more deterministic, higher-confidence outputs, but reduced entropy alone does not guarantee improvement.The paper establishes this connection theoretically and empirically while separating entropy effects from performance effects.
Random rewards can improve RLVR performance beyond potentially contaminated Qwen-Math settings, with gains also appearing in Llama and QwQ families.The experiments span Qwen-Math, Llama, and QwQ models, including multiple sizes and base or distilled variants.
The paper asks whether policy entropy causally relates to performance and whether spurious-reward gains arise through clipping bias, contamination, or their interaction.These questions address conflicting observations about entropy minimization and random-reward fine-tuning in RLVR.

2 PRELIMINARIES AND TECHNICAL BACKGROUND

This section defines RLVR and its group-based policy optimization setup, then introduces spurious rewards, clipping bias, and policy entropy as the paper’s analytical objects. It reviews conflicting evidence about whether random rewards and lower entropy improve reasoning.

RLVR uses binary outcome rewards by comparing sampled responses with ground-truth answers, while policy-gradient training estimates updates from rollout groups.Under GRPO, the token-level advantage equals the response-level group advantage and is independent of token position.
The clipped surrogate objective limits token-level importance ratios to stabilize updates when the current policy drifts from the rollout policy.The ratio is clipped using ε, producing a raw surrogate and a clipping-correction term.
2.2 SPURIOUS REWARD FOR RLVR: A random reward is a spurious feedback signal independent of the prompt and sampled response, with binary outcomes drawn symmetrically.The resulting GRPO advantages are symmetrically distributed around zero, forming the basis for later analysis.
2.2 SPURIOUS REWARD FOR RLVR: Upper clipping permits larger increases for already high-probability tokens, potentially amplifying high-prior responses rather than creating new reasoning ability.This interpretation motivates the paper’s examination of whether clipping bias is a useful learning signal.
2.3 LLM POLICY ENTROPY: Policy entropy measures response-distribution diversity: high entropy yields varied samples, whereas low entropy concentrates probability and produces more deterministic behavior.Prior studies disagree on whether avoiding entropy collapse or reducing entropy is better for reasoning.
2.3 LLM POLICY ENTROPY: The standard first-order entropy approximation predicts no entropy change under random rewards, but this conflicts with observations because it ignores higher-order terms and assumes unclipped training.The discrepancy motivates a clipped analysis of entropy dynamics.

3 CLIPPING AND MODEL PERFORMANCE

The paper’s clipping analysis shows that clipping correction is small relative to the raw surrogate and does not provide a meaningful learning signal under random rewards. Empirically, clipping can hurt validation performance even though it affects policy entropy.

3.1 THEORETICAL RESULTS: The clipping-correction bound depends on the empirical token-level activation rate, so models with higher activation rates receive larger clipping corrections.This rate can be monitored directly during training across model families.
3.1 THEORETICAL RESULTS: Under practical settings, the expected raw surrogate magnitude is much larger than the total clipping-correction magnitude.The theoretical bounds are expressed using observable token-level clipping activation rates and support a model-agnostic analysis.
3.2 MODEL-SPECIFIC EVALUATION: The experiments use Qwen2.5-Math-7B, random Bernoulli rewards, group size 16, clipping ratio 0.2, and learning rate 5 × 10^-7.The setup follows the cited prior configuration and evaluates trajectories on the MATH500 validation set.
3.2 MODEL-SPECIFIC EVALUATION: Clipping can decrease validation performance, whereas disabling clipping often improves it on Qwen2.5-Math-7B trained with random rewards.Figure 1 compares independent clipped and unclipped trajectories on MATH500 and also tracks clipping activation during training.
3.2 MODEL-SPECIFIC EVALUATION: Because clipping bias lacks a meaningful gradient signal even under contaminated settings, its role is to shape policy entropy rather than directly improve learning.The paper distinguishes this entropy-regulation effect from performance enhancement.

4 CLIPPING AND POLICY ENTROPY

The paper characterizes how clipping and initial policy skewness shape entropy under random rewards, then tests whether entropy changes reliably predict performance. Clipping reduces entropy and stabilizes updates, but entropy minimization improves performance only in some model–data regimes.

4.1 ONE-STEP POLICY ENTROPY CHANGE UNDER RANDOM REWARDS: The one-step entropy change under unclipped training depends on policy skewness, with the theoretical remainder scaling as O(η4) for small η.The formulation uses Φ(πold) to measure skewness and identifies conditions permitting entropy growth.
4.1 ONE-STEP POLICY ENTROPY CHANGE UNDER RANDOM REWARDS: Clipping systematically reduces policy entropy under random rewards, whereas unclipped training can increase entropy depending on the initial policy distribution.For two-armed policies, entropy decreases for β ∈ [0.176, 0.824] and increases for sufficiently skewed initializations; this behavior is also observed empirically.
4.2 EMPIRICAL EVALUATION: Under random rewards, clipping acts as implicit entropy minimization by concentrating probability on fewer trajectories while also enforcing a local trust region.By capping per-token likelihood ratios, clipping reduces update sizes, prevents excessive policy drift, and helps prevent gradient explosion.
4.2 EMPIRICAL EVALUATION: Unclipped training can initially improve MATH500 validation accuracy from 65.6% to 76.6% before gradient explosion around step 150 causes a sharp performance drop.The clipped counterpart is presented as the comparison condition for R1-Distill-Llama-8B.
4.3 POLICY ENTROPY AND MODEL PERFORMANCE: Entropy reduction is not reliably associated with better performance: Qwen2.5-Math-7B loses validation performance on AIME despite entropy collapse under clipped training.The paper contrasts this with stronger models, which show steady early-epoch gains on the same dataset and otherwise identical settings.
4.3 POLICY ENTROPY AND MODEL PERFORMANCE: Entropy minimization is regime-dependent: it may reinforce correct trajectories for strong models on easier data but incorrect modes for weaker models or harder data.The latter regime can produce noisy rollouts, unstable updates, and stalled or degraded performance.

5 REWARD MISALIGNMENT: WHO CAN BENEFIT FROM RANDOM REWARDS?

The reward-misalignment analysis explains why stronger models are more likely to benefit from random rewards: increasing correctness reduces total damage and especially false-positive misallocations. Experiments support this pattern beyond contaminated Qwen-Math settings, including a stronger distilled Llama model.

Empirical validation: Random reward training becomes less stable when baseline accuracy is lower, while higher baseline accuracy produces smoother validation dynamics.The reward-misalignment framework attributes these differences to the number of correct rollouts and associated damage fluctuations.
Reward-misalignment model: The analysis models random rewards through false positives on incorrect rollouts and false negatives on correct rollouts.Correct-response advantage loss measures reward mass diverted from correct rollouts by these random mislabels.
Theoretical analysis: Theorem 5.3 states that as correct-rollout count increases, false-negative regions contribute a larger share of the shrinking damage while false-positive contributions decrease monotonically.This decomposition explains why stronger models are more likely to improve under random rewards.
Theoretical analysis: Stronger models incur less total misalignment damage and a smaller false-positive contribution when correct rollouts outnumber incorrect ones.The analysis predicts that this makes improvements under random rewards more likely, including beyond contaminated-reward settings.
Empirical validation: A stronger R1-Distill-Llama-8B model improves under random rewards, whereas the weaker Qwen-Math model does not achieve similar gains.The comparison uses matched training and validation hyperparameters, with 8192-token rollouts for the distilled Llama experiment.

6 CONCLUSION

The paper concludes that random rewards can help stronger models, while clipping primarily regulates entropy rather than supplying a useful learning signal. It also finds that entropy and performance have no deterministic causal relationship.

Conclusion: Random rewards can improve performance, but stronger models benefit more and weaker models may become unstable on harder datasets.This conclusion is supported by the paper’s theoretical and empirical analyses of reward misalignment.
Conclusion: Clipping bias does not provide a useful learning signal; instead, clipping regulates policy entropy when training signals are spurious.The paper frames clipping as an entropy-regulation mechanism rather than a direct source of performance improvement.
Conclusion: Entropy and performance lack a deterministic causal relationship: entropy decreases can accompany collapse, while entropy increases can coincide with improvement.The conclusion rejects treating lower entropy alone as a reliable explanation for better performance.

A RELATED WORKS

Related work spans RLVR, entropy-based reasoning, classical spurious rewards, and alternative alignment methods. The paper emphasizes that reported random-reward gains depend strongly on experimental configuration and may reflect different mechanisms.

Spurious reward in classical RL: Classical RL literature studies spurious rewards in connection with generalization, reward misspecification, and reward hacking.These settings provide broader context but differ from the outcome-level, sparse rewards used in RLVR.
Spurious reward in RLVR: Prior RLVR studies report gains from random rewards, but competing work finds inconsistent improvements or degradation under different configurations.Differences include prompt formatting, rollout length, dataset, learning rate, and batch size.
Contamination: Contamination has been reported for Qwen-Math validation sets, especially MATH500, motivating caution when interpreting apparent random-reward gains.The paper notes possible similarity between training and validation data as an additional contamination concern.
LLM entropy: Entropy-minimization methods can improve reasoning, but entropy minimization alone may produce low-entropy yet suboptimal solutions.The paper therefore treats entropy as a stabilizing regularizer rather than a replacement for genuine RLVR signals.
Alignment methods: Alternative alignment work includes PPO-based reinforcement learning, offline preference optimization, and methods that generate or expand preference data.These approaches address computational cost, reference-model dependence, or limited human preference data.

B ADDITIONAL EXPERIMENTAL RESULTS

Additional experiments test robustness across clipping thresholds, group sizes, unclipped training, and policy skewness. They show that stricter clipping preserves the main pattern, smaller groups increase variance, and entropy changes do not uniformly predict performance.

Experimental scope: The additional experiments evaluate clipping, entropy, and performance on Qwen-Math-7B while extending the comparison to harder AIME training and broader model families.The setup is organized around characterizing spurious-reward dynamics and testing whether their benefits generalize beyond Qwen-Math.
Group-size ablation: Reducing the group size to G = 8 lets most runs improve but increases variance and destabilizes learning dynamics.Smaller groups make extreme reward-misalignment events more likely.
Clipping-threshold ablation: Across clipping thresholds ε = 0.15 and ε = 0.1, successful runs converge to roughly 70% validation accuracy, supporting robustness to ε.Some runs still fail to improve, consistent with the reward-misalignment framework.
Policy skewness: Figure 8 compares sorted response log-probabilities for 64 samples across 12 prompts, separating positive and negative policy-skewness values.The caption states that entropy increases under unclipped training can occur only for the skewed prompts with Φ(π) < 0.
Unclipped training: On the hard AIME dataset, clipped training can reduce entropy while performance degrades, showing that lower entropy is not reliably associated with better performance.Unclipped experiments examine the complementary regime in which entropy can increase under random rewards.

C THEORETICAL ANALYSIS

The section formalizes GRPO and reduces its clipped inner-loop updates to an analytically tractable policy-gradient framework. It highlights that clipping analysis depends on empirically measured activation rates because repeated micro-updates make exact analysis intractable.

Setup: The setup models next-token generation with softmax policies over histories and interprets GRPO as a natural policy-gradient step under softmax tabular parametrization.This reduction provides the theoretical framework for analyzing clipping and entropy dynamics.
GRPO analysis: GRPO samples grouped outputs, computes group-relative token advantages, and updates the policy with a clipped surrogate objective.The algorithm uses a reference policy, repeated batches, reward computation, advantage estimation, and policy updates.
GRPO analysis: A KL penalty can constrain policy drift from the reference policy, but zero-RL settings remove this explicit term.The analysis therefore treats clipping as the main mechanism limiting movement from the old policy.
GRPO analysis: The inner loop performs µ optimizer steps on the same mini-batch while clipping limits drift from πold.Importance ratios are recomputed across the micro-updates as the policy changes.
GRPO analysis: Exact clipping analysis is intractable without the per-step clip-activation rate, so the theory conditions on its empirical model- and dataset-dependent value.The resulting single NPG-style update preserves the first-order clipping effect and enables tractable bounds.

C.1 MISSING PROOFS IN §2

The proofs show that several apparent entropy signals vanish under random rewards because advantages are independent of sampled responses. Consequently, covariance-based entropy approximations cannot directly characterize random-reward training.

Unclipped update: The unclipped update is written as a normalized exponential tilt of πold by the scaled advantage.The normalization term ensures the updated distribution remains a policy.
Random-reward entropy analysis: The paper therefore identifies the approximation in Eq. (5) as invalid for random-reward RLVR and calls for a more accurate entropy-change estimate.This failure motivates the subsequent clipping and entropy analysis.
Random-reward entropy analysis: Under random rewards, the covariance between the advantage and old-policy probability is zero because the two quantities are independent.The displayed expectation reduces to zero covariance terms.

C.2 MISSING PROOFS IN §3

The section derives bounds for unclipped policy updates and their entropy changes, including dependence on the minimum policy probability and bounded advantages. These bounds support control of update ratios and higher-order remainder terms.

Update-ratio bounds: The update ratio is bounded by Rmax = e^(η/πmin) under the bounded-advantage assumption.The bound follows from controlling both the numerator and normalization factor of the exponential update.
Unclipped entropy dynamics: The proof bounds exponential moments using convexity and cosh inequalities, then sums token-level terms to obtain the sequence-level estimate.The resulting expectation bound is combined with the earlier theorem.

C.3 MISSING PROOFS IN §4

The section develops proofs for the clipped-policy entropy dynamics and illustrates how upper clipping creates asymmetric updates. The accompanying analysis shows that unclipped random-reward training may preserve or restore entropy in sufficiently skewed regimes.

Unclipped entropy dynamics: The unclipped two-armed analysis predicts entropy growth only from sufficiently skewed initial policies, as illustrated by Figure 9.The figure compares flat and skewed initializations across 10 independent trails per panel.
Upper clipping: The clipped surrogate is analyzed through upper-clipping KKT conditions that characterize the constrained policy ratios.The proof distinguishes ratios below, above, and at the clipping threshold 1 + ε.
Upper clipping: Upper clipping permits larger absolute increases for already high-probability tokens, while low-probability tokens reach the clipping threshold earlier.This asymmetry can amplify high-prior responses rather than foster new reasoning ability.
Upper clipping: Theorem 4.3 bounds the clipped entropy change using the probability and excess magnitude of upper-clipping events.The quantities ρ and δ summarize how often clipping occurs and how far the unclipped ratio exceeds the threshold.
Upper clipping: In the practical parameter example, the derived bound gives E[∆H] < −1.49 × 10^-7, establishing negative expected entropy change.The calculation uses G = 16, η = 5 × 10^-7, |V| ≈ 150000, πmin = 10^-7, and ε = 0.2.

C.4 MISSING PROOFS IN §5

The missing proofs analyze conditional variances by representing binomial counts through an auxiliary variable Z. They establish that when n_c > n_i, the variance conditional on f > g is smaller than the variance conditional on g > f.

Theorem 5.3: The proof sets X = f and Y = n_c − g, yielding Z = X + Y ∼ Binomial(G, 1/2), with f > g equivalent to Z > n_c.The binomial representation converts the comparison of f and g into conditioning on Z.
Variance comparison: The conditional variances satisfy Var(∆| f > g) = C E[h(Z) | Z > n_c] and Var(∆| g > f) = C E[h(Z) | Z < n_c], where h(z) = z(G − z).The event comparisons follow from f > g ⇔ Z > n_c and g > f ⇔ Z < n_c.
Theorem 5.3: When n_c > n_i, Var(∆| f > g) < Var(∆| g > f).This is the stated result of the proof.
Variance comparison: Conditional on Z = z, positive labels are uniformly scattered, allowing the proof to characterize the conditional behavior of X within the indices of I.The proof uses this uniform-scattering property to derive conditional expectations and variances.
Variance comparison: Because h is symmetric around G/2 and increasing up to that point, the proof derives E[h(Z) | Z < G − n_c] < E[h(Z) | Z < n_c].Multiplication by C > 0 then preserves the inequality and completes the variance comparison.