PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Sampled Token: Preserving Candidate Support in RLVR

Ruotian Peng, Yi Ren, Zhouliang Yu, Weiyang Liu, Yandong Wen

arXiv:2510.14807v3cs.AI

TL;DR

RLVR can suppress exploration as probability concentrates on the top-1 candidate and pass@K degrades. CaSP preserves support among top-N candidates through complementary gradient updates, and improves pass@1 and pass@K across benchmarks and model scales.

  • Problem

    In RLVR, the coupled rewarding of correct responses and penalizing of incorrect ones is associated with exploitation over exploration, while top-1 concentration accompanies degraded pass@K.

    Source: §1 · pp. 1–2

  • Method

    CaSP redistributes positive gradients among top-N candidates for correct responses and applies a stronger penalty to the top-1 candidate for incorrect responses.

    Source: §1 · p. 2

  • Results

    CaSP improves pass@1 and pass@K over GRPO and other strong baselines across math, logical-reasoning, and coding benchmarks, with gains persisting up to 32B parameters and K = 1024.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Preserving non-negligible probability mass on top-N candidates is the paper’s proposed desideratum for effective RLVR exploration.

    Source: §1 · p. 2

    CaSP depends on the model’s current candidate ranking and cannot directly recover useful reasoning branches that fall outside the top-N set.

    Source: §7 · p. 9

Abstract

from arXiv · show

We revisit exploration collapse in reinforcement learning with verifiable rewards (RLVR), from the perspective of the \emph{candidate distribution} for next-token prediction. We formally show that as probability concentrates on the top-$1$ candidate, the expected number of distinct responses collapses to one regardless of the sampling budget $K$. This theoretical implication is further verified by our empirical tracking of top-$N$ candidate probabilities during training, where the top-$1$ candidate progressively dominates while plausible alternatives are suppressed. These findings suggest a key desideratum for effective exploration: \emph{preserving non-negligible probability mass on the top-$N$ candidates}. To this end, we propose Candidate-aware Support Preservation (CaSP), with two complementary designs. Specifically, CaSP redistributes positive gradients among top-$N$ candidates for correct responses, and applies a stronger penalty to the top-$1$ candidate for incorrect responses. Unlike many exploration-oriented methods that improve pass@$K$ at the cost of pass@1, CaSP improves pass@$K$ across the full $K$ spectrum. These gains generalize to 6 math, 2 logical-reasoning, and 2 coding benchmarks, and scales to 32B-parameter models and sampling budgets up to $K=1024$, positioning it as a principled, candidate-level approach for RLVR exploration.

1 Introduction

RLVR can favor exploitation over exploration, with top-1 candidate concentration associated with weaker performance across multiple attempts. CaSP targets that concentration by preserving support among plausible candidates, and is reported to improve pass@K across benchmarks and scales.

1 Introduction

  • RLVR rewards correct responses and penalizes incorrect ones, a coupled process associated with exploitation over exploration, improved pass@1, and degraded pass@K for K>1.The paper links insufficient exploration to difficulty finding correct solutions with many attempts.Source: §1 · p. 1
  • Existing approaches intervene at response, token, segment, or hybrid levels, whereas CaSP acts at the candidate level to preserve probability among plausible next tokens.The paper argues response-level methods do not explicitly encourage new responses, while entropy-based token methods may assign mass to unrelated candidates.Source: §1 · p. 1, Figure 1 · p. 2
  • During RLVR fine-tuning, probability shifts toward the top-1 candidate as plausible alternatives are suppressed, matching the reported pass@K degradation and motivating top-N support preservation.The paper frames next-token candidate probabilities as a view of exploration, while noting that the full vocabulary distribution is computationally costly to capture.Source: §1 · p. 2
  • For correct responses, CaSP redistributes positive gradients among top-N candidates; for incorrect responses, it penalizes the top-1 candidate more strongly.The updates are especially effective on high-entropy reasoning tokens.Source: §1 · p. 2
  • Across math, logical-reasoning, and coding benchmarks, CaSP improves pass@K across the full K spectrum while also improving pass@1, with gains reported up to 32B parameters and K = 1024.The comparison contrasts CaSP with GRPO and exploration-oriented baselines that trade pass@1 against pass@K.Source: §1 · p. 2

2 Background and Preliminaries

The background describes RLVR and GRPO, including the group-normalized reward advantage and the gradient perspective used to motivate CaSP. Figure 2 also compares CaSP and GRPO on pass@K and candidate-probability concentration.

2 Background and Preliminaries

  • Figure 2 reports that CaSP improves pass@K across the K spectrum on math and logic benchmarks, with a less concentrated rank-wise candidate distribution than GRPO.The benchmark set shown includes AIME24/25, AMC, MATH500, Minerva, OlympiadBench, SynLogic, and BBH.Source: Figure 2 · p. 3
  • The clipped likelihood ratio γi,t compares current and previous policy probabilities for the sampled token at decoding step t.The decoding state is si,t = (x, yi,<t).Source: §2 · p. 3
  • GRPO computes each response’s advantage by normalizing its verifier reward across a group of G rollouts.The verifier reward is binary, ri ∈ {0, 1}.Source: §2 · pp. 2–3
  • The GRPO update is analyzed through the dominant gradient contribution ∇θAiγi,t, with the advantage acting as a response-level adaptive learning rate.The discussion sets aside KL and clipping terms as primarily introduced for stability, and CaSP builds on this gradient form to redistribute candidate mass.Source: §2 · p. 3

3 A Candidate-Level Analysis of Why RLVR Concentrates Probability

RLVR updates concentrate next-token probability on the rank-1 candidate, narrowing rollout diversity. The analysis motivates preserving non-negligible probability mass across top-N candidates.

3 A Candidate-Level Analysis of Why RLVR Concentrates Probability

  • The candidate-level analysis links positive and negative RLVR updates to over-concentration and identifies top-N support preservation as the exploration desideratum.The section frames candidate distributions as the basis for understanding why RLVR concentrates probability.Source: §3 · p. 3

3.1 RLVR Updates Reinforce Concentration

  • Positive advantages reinforce candidates in proportion to their current probability, while negative updates can squeeze probability toward rank-1 when another candidate is penalized.The squeeze effect occurs because alternative-token probability increases in proportion to the sampled token’s probability and the alternative’s probability.Source: §3.1 · p. 3

3.2 From Candidate to Response Collapse

  • As the rank-1 probability approaches 1, the expected number of distinct next tokens converges to 1 for any fixed sampling budget K.At branching states, this local collapse yields more homogeneous trajectories and reduces the marginal benefit of increasing K.Source: §3.2 · p. 4

3.3 Empirical Verification

  • Rank-wise tracking shows RLVR variants widen the gap between rank-1 and lower-ranked candidates; PSR amplifies this shift, while NSR slows but does not eliminate it.These patterns motivate preserving non-negligible probability mass on top-N candidates.Source: §3.3 · p. 4, Figure 3 · p. 4

4 Candidate-aware Support Preservation

CaSP preserves candidate support through complementary positive- and negative-update designs applied at high-entropy tokens. In a toy tree MDP, it preserves all rewarding modes without the accuracy loss shown by entropy regularization.

4 Candidate-aware Support Preservation

  • CaSP redistributes positive gradients across top-N candidates and applies stronger negative penalties only to rank-1 sampled tokens, with both designs used at high-entropy tokens.High-entropy tokens are selected when their entropy exceeds the response-level q-quantile threshold.Source: §4 · p. 4

4.1 Positive Updates on Top-N Candidates

  • For positive samples, CaSP uses localized label smoothing over the model’s top-N candidates to spread reinforcement across plausible alternatives rather than the full vocabulary.This forms a probability plateau instead of a sharp peak.Source: §4.1 · p. 5

4.2 Negative Updates on Top-1 Candidate

  • For negative samples, CaSP strengthens the penalty only when the sampled token is rank-1, avoiding stronger penalties on lower-ranked tokens that can sharpen the distribution through the squeeze effect.The parameter λtop1 controls the strength of the rank-1 penalty.Source: §4.2 · p. 5

4.3 A Didactic Toy Example

  • In the toy tree MDP, CaSP preserves all six rewarding modes at accuracy 1.00, while GRPO+Entropy’s recovery of all modes leaks mass to non-rewarding actions and reaches accuracy 0.48.GRPO collapses to one mode; CaSP confines exploration to plausible continuations.Source: §4.3 · p. 6, Figure 5 · p. 6

5 Experiments

CaSP improves pass@1 and pass@K across math, logical-reasoning, and coding evaluations, while ablations show that its performance depends on tuning the update strength, targeting high-entropy tokens, and choosing a small candidate set.

5.1 Setup

  • The experiments use three math backbones, a Qwen2.5-7B model for logical reasoning, and DeepSeek-R1-Distill-Qwen-7B for code reasoning across varied reasoning benchmarks.The math training datasets are MATH (Levels 3–5) for Qwen models and GSM8K plus MATH (Level 1) for Llama.Source: §5.1 · p. 6

5.2 Math Reasoning

  • CaSP improves pass@1 by 17.6% on Qwen2.5-Math-7B and 9.8% on Llama3.2-3B-Instruct, and raises pass@256 by 4.1% and 1.9%, respectively.On Qwen2.5-32B, it improves pass@1 by 19.6% and pass@128 by 0.6%.Source: §5.2 · p. 6
  • CaSP consistently outperforms baselines across math backbones and K, including higher pass@256 than GRPO by 4.4% on Qwen2.5-Math-7B and 1.3% on Llama3.2-3B-Instruct.On Qwen2.5-32B, it improves pass@128 by 4.9% over GRPO while achieving higher pass@1 across all three backbones.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 Logical and Coding Reasoning

  • On Synlogic, CaSP gains +31.5% in pass@1 and +26.4% at pass@128; on BBH, pass@1 reaches 58.4% (+16.0%) while pass@128 decreases 2.1%.On LiveCodeBench, it gains +5.17% in pass@1 and +7.17% in pass@16; on HumanEval+, it achieves 83.90% Avg@16.Source: Table 2 · p. 8, §5.3 · p. 8, Table 3 · p. 8

5.4 Ablations

  • Increasing α from 0 to 0.1 monotonically raises pass@256 by 3.3%–4.4% over GRPO, while pass@1 peaks at α = 0.01 and then slightly declines but remains above GRPO.Figure 6 presents the α, τq, and N ablations on Qwen2.5-Math-7B math benchmarks.Source: §5.4 · p. 8, Figure 6 · p. 8
  • Applying CaSP to all tokens drops pass@1 by 9.3%, although pass@256 beats GRPO across most τq values, supporting updates focused on high-entropy tokens.Figure 6 presents the τq ablation on Qwen2.5-Math-7B math benchmarks.Source: §5.4 · p. 8, Figure 6 · p. 8
  • For N from 1 to 5, pass@256 ranges from 79.1% to 80.5% and pass@1 from 42.6% to 43.4%, with both metrics outperforming GRPO.Both metrics initially rise and then fall as N increases.Source: §5.4 · p. 8, Figure 6 · p. 8

6 Conclusion

The conclusion frames RLVR exploration collapse as concentration of the candidate distribution on rank 1, reducing response diversity regardless of sampling budget. CaSP redistributes positive gradients over top-N candidates and strengthens penalties on rank 1, improving pass@1 and pass@K across tasks and scales.

6 Conclusion

  • CaSP preserves top-N candidate support through positive-gradient redistribution and stronger rank-1 penalties, improving pass@1 and pass@K over GRPO and other baselines across math, logical, and coding benchmarks.The reported gains persist up to 32B parameters and K = 1024.Source: §6 · p. 8

7 Limitations

CaSP operates within the model’s current top-N candidate ranking, so useful branches outside that set cannot be directly recovered.

7 Limitations

  • CaSP cannot directly recover useful reasoning branches that the model’s current ranking leaves outside top-N.The authors identify this as an efficiency trade-off against costly full-vocabulary updates and suggest adaptively expanding or refreshing the candidate set.Source: §7 · p. 9

Ethical Considerations

The material pairs an ethics statement with analyses of how RLVR concentrates next-token probability, limits response diversity, and motivates finer-grained exploration methods.

Ethical Considerations

  • The work reports no human subjects, personal or sensitive data, or fairness, privacy, security, or misuse concerns.It states that datasets and code follow their respective licenses and terms of use.Source: Ethical Considerations · p. 9

B.1 Squeeze Effect of Negative Updates

  • Negative updates increase the logit of the highest-probability alternative token, rather than distributing that increase uniformly across alternatives.The increase among alternatives is proportional to their current probabilities.Source: §B.1 · p. 14

B.2 Local Diversity under Concentration

  • As top-1 probability approaches one, the expected number of distinct sampled tokens converges to one for fixed K.The proof bounds the remaining candidates' contribution using their total probability mass.Source: §B.2 · p. 14

C.1 Reinforcement Learning with Verifiable Rewards in LLMs

  • Prior RLVR modifications such as DAPO, Dr.GRPO, GSPO, and Lopti improve pass@1, but do not specifically address pass@K exploration.The cited methods modify clipping, advantage normalization, importance sampling, or token-probability weighting.Source: §C.1 · p. 14

C.2 Effective Exploration for RLVR in LLMs

  • Exploration methods intervene at response, token, or data level, while prior approaches leave limitations in candidate-level control or incur added data and computation costs.Negative-only reinforcement can improve pass@K while lowering pass@1, and may reduce both metrics when applied strongly early.Source: §C.2 · pp. 14–15

C.3 Analysis of Learning Dynamics in LLMs

  • The proposed top-N probability-dynamics framework tracks how on-policy RL redistributes mass among likely candidates throughout training.It addresses entropy metrics' indirectness and individual-update analyses' inability to capture cumulative policy evolution.Source: §C.3 · p. 15

D.1 Effects of CaSP on Training Dynamics

Training-dynamics comparisons show CaSP deconcentrating top-candidate probabilities and retaining entropy at forking points; top-3 probability distributions are highly rank-skewed.

D.1 Effects of CaSP on Training Dynamics

  • CaSP achieves the strongest deconcentration among the compared methods, while GRPO, Entropy-Adv, and forking-token training exhibit substantial concentration.KL-Cov has a moderate concentration effect, and W-REINFORCE alleviates some concentration but remains prone to squeezing.Source: Figure 7 · p. 16, §D.1 · p. 16
  • CaSP preserves token-level entropy particularly at forking points, unlike GRPO, which drives most tokens toward near-zero entropy.Figure 8 visualizes the entropy distributions.Source: §D.1 · p. 16

D.2 Top-3 Candidates Probabilities Distribution

  • The rank-3 candidate falls below 0.05 in more than 95% of cases across methods, while rank-1 candidates occupy higher-probability bins.The sharp rank decay supports using top-N candidate statistics as a tractable proxy for monitoring the full next-token distribution.Source: §D.2 · p. 16

E.1 Training Details

The appendix reports temperature robustness and ablations alongside task-specific training and evaluation settings, implementation details, and artifact-use considerations.

E.1 Training Details

  • Training uses task-specific settings, with N=3 for math, logic, and coding and different batch sizes, response sampling counts, and CaSP coefficients.The reported training budgets are approximately 120 H100 GPU hours for 7B math and logic, 192 H200 GPU hours for Qwen2.5-32B, and 160 H100 GPU hours for coding.Source: §E.1 · pp. 16–17

E.2 Evaluation Details

  • Evaluation reports pass@1 and pass@K across math, logic, and coding; math uses n=300 for small datasets and n=256 for other benchmarks, except Qwen2.5-32B uses n=128; logic uses n=128.Source: §E.2 · p. 17

E.3 PseudoCode

  • The appendix presents CaSP pseudocode in Figure 10, whose caption says it modifies only a few lines of standard policy-gradient code.The cited text identifies the figure as the policy-loss computation pseudocode.Source: §E.3 · p. 17, Figure 10 · p. 18

E.4 Artifacts and Licenses

  • The work uses publicly available pretrained models and datasets under their providers' licenses and benchmark usage terms.Listed datasets include MATH, GSM8K, SynLogic, BBH, LiveCodeBench, HumanEval+, AIME, AMC, Minerva Math, and OlympiadBench.Source: §E.4 · p. 17

E.5 Artifact Use and Data Considerations

  • The artifacts are used for research and evaluation, with public benchmarks and no newly collected user data or personally identifying information.The paper also states that it does not use personal or sensitive data.Source: §E.5 · p. 17

F.1 Detailed Ablation Results

  • Applying CaSP only to correct or only to incorrect responses lowers pass@K relative to applying both designs.The passage identifies the combined designs as yielding the best results.Source: §F.1 · p. 18

F.2 The Influence of Sampling Temperature

  • At temperature 0.2, CaSP achieves 77.2% in pass@256, compared with 63.8% for Qwen2.5-Math-7B and 69.7% for GRPO.Across temperatures from 0.2 to 1.0, CaSP outperforms GRPO on both pass@1 and pass@K.Source: §F.2 · p. 18

F.3 Effect of Non-Uniform Top-N Weight Allocation

  • CaSP (softmax) slightly improves pass@K but decreases pass@1 compared with uniform smoothing in the main CaSP method.The variant allocates top-N smoothing weights proportionally using softmax.Source: §F.3 · p. 18

F.4 Scaling Sample K to 1024

At sampling budget K=1024, the paper reports that CaSP still improves over the base model on AIME24/25, including cases where the base model surpasses GRPO.

F.4 Scaling Sample K to 1024

  • At K=1024 on AIME24/25, CaSP delivers large improvements over the base model even when the base model surpasses GRPO.Source: §F.4 · p. 18

G Qualitative Analysis of Local Candidate Support

The grouped candidate matrices show that GRPO concentrates probability on fewer candidates, while CaSP retains plausible alternatives around the sampled path. The visual encoding makes local support comparable across token positions.

G Qualitative Analysis of Local Candidate Support

Related papers

Loading 2510.14807v3…