PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward

Guanhua Huang, Tingqiang Xu, Mingze Wang, Qi Yi, Xue Gong, Siheng Li, Ruibin Xiong, Kejiao Li, Yuhao Jiang, Bo Zhou

arXiv:2510.03222v2cs.LGcs.CL

TL;DR

RLVR reasoning training can plateau as entropy collapses, while focusing on overall entropy may amplify irrelevant tokens and leave meaningful exploration underexplored. The paper identifies valuable low-probability reasoning sparks, introduces Lp-Reg to selectively preserve them through a filtered proxy distribution, and reports sustained scaling with improved math-benchmark accuracy.

  • Problem

    RLVR scalability is limited by entropy-collapse plateaus, and the mechanisms governing meaningful exploration beyond overall entropy remain underexplored.

  • Method

    Lp-Reg filters presumed noise tokens from the policy to form a less-noisy proxy distribution that softly protects the remaining low-probability reasoning sparks through KL regularization.

  • Results

    60.17% average accuracy on five math benchmarks improves upon prior methods by 2.66%, while Lp-Reg sustains on-policy scaling for 3,000 steps and 81,204 GPU-hours.

  • Takeaways & Limitations

    Preserving selected low-probability tokens rather than maximizing entropy indiscriminately supports sustained exploration and state-of-the-art performance in the reported RLVR setting.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has propelled Large Language Models in complex reasoning, yet its scalability is often hindered by a training bottleneck where performance plateaus as policy entropy collapses, signaling a loss of exploration. Previous methods typically address this by maintaining high policy entropy, yet the precise mechanisms that govern meaningful exploration have remained underexplored. Our analysis suggests that an unselective focus on entropy risks amplifying irrelevant tokens and destabilizing training. This paper investigates the exploration dynamics within RLVR and identifies a key issue: the gradual elimination of valuable low-probability exploratory tokens, which we term \textbf{\textit{reasoning sparks}}. We find that while abundant in pre-trained models, these sparks are systematically extinguished during RLVR due to over-penalization, leading to a degeneracy in exploration. To address this, we introduce Low-probability Regularization (Lp-Reg). Its core mechanism regularizes the policy towards a heuristic proxy distribution. This proxy is constructed by filtering out presumed noise tokens and re-normalizing the distribution over the remaining candidates. The result is a less-noisy proxy where the probability of \textit{reasoning sparks} is amplified, which then serves as a soft regularization target to shield these valuable tokens from elimination via KL divergence. Experiments show that Lp-Reg enables stable on-policy RL, sustaining continuous scaling across $3,000$ training steps and $81,204$ GPU-hours, where baseline entropy-control methods collapse. This sustained exploration leads to state-of-the-art performance, achieving a $60.17\%$ average accuracy on five math benchmarks, an improvement of $2.66\%$ over prior methods. Code is available at https://github.com/CarlanLark/Lp-Reg.

1 Introduction

RLVR training can plateau as entropy collapses, but indiscriminate entropy control may amplify noise rather than preserve meaningful exploration. The paper identifies valuable low-probability reasoning sparks and introduces Lp-Reg to protect them, enabling sustained scaling and stronger benchmark performance.

  • 1 Introduction: RLVR performance often plateaus and collapses as policy entropy rapidly decays, signaling reduced exploration.Existing methods attempt to maintain higher entropy, but overall entropy is an indirect exploration proxy.
  • 1 Introduction: Reasoning sparks are valuable low-probability exploratory tokens whose suppression can eliminate diverse reasoning pathways.Examples include “wait”, “however”, and “perhaps”; standard GRPO suppresses their low-probability sampling.
  • 1 Introduction: Indiscriminate entropy bonuses amplify irrelevant low-probability tokens, adding noise and accelerating performance collapse.The paper contrasts meaningful tokens such as “wait” with irrelevant tokens such as “cost” and “fine”.
  • 1 Introduction: 60.17% average accuracy on five math benchmarks improves upon prior methods by 2.66% with Lp-Reg on Qwen3-14B.Lp-Reg also enables stable on-policy training for around 3,000 steps over 81,204 GPU-hours, where many entropy-control methods collapse.
  • 1 Introduction: Lp-Reg filters presumed meaningless noise to preserve the remaining low-probability tokens and create a more stable exploratory environment.Its proxy-distribution mechanism is introduced to protect valuable reasoning sparks without broadly increasing randomness.
  • 1 Introduction: Experiments show that Lp-Reg achieves state-of-the-art performance and stable extended on-policy scaling where baselines collapse.The contribution is supported by comparisons against indiscriminate entropy-control methods and extended training results.

2 Related Work

Prior RL methods address entropy collapse through entropy monitoring, selective updates, clipping, or related controls. This paper instead examines individual next-token distributions to study exploration more directly and semantically.

  • 2 Related Work: RLVR improves LLM reasoning by combining reinforcement learning with automatic checkers or symbolic verification.The related work situates methods such as GRPO, DAPO, and VAPO within this broader framework.
  • 2 Related Work: RL training for reasoning models commonly suffers rapid early policy-entropy collapse, reflecting excessive exploitation and insufficient exploration.Prior mitigation directions include selective regularization, exploratory-position advantage amplification, clipping, and weight clipping.
  • 2 Related Work: Existing methods often monitor overall entropy, which the paper characterizes as correlational rather than causal to exploration.The paper motivates analyzing candidate-token probabilities to obtain a more semantically grounded and causally informed view.

3 Preliminaries

RLVR optimizes language-model reasoning with verifiable rewards, while GRPO estimates token-level advantages from multiple outputs sampled for each prompt. Policy optimization uses probability-ratio clipping and KL regularization to constrain updates.

  • 3.1 Reinforcement Learning with Verifiable Rewards: RLVR assigns rewards to outputs through rule-based verification and optimizes the resulting objective with policy-gradient methods such as PPO and GRPO.The reward r(o, a) is assigned to output o given reference answer a; in RLVR, rule-based functions compute it.
  • 3.2 Group-Relative Policy Optimization: GRPO estimates each token’s advantage from multiple independent outputs sampled from the same prompt and their associated rewards.For output oi, Ai,t represents the advantage of its t-th token, based on group-relative rewards.
  • 3.2 Group-Relative Policy Optimization: KL regularization between the current policy and a reference policy is controlled by β.The coefficient β determines the strength of this regularization.
  • 3.2 Group-Relative Policy Optimization: The policy objective uses a probability ratio for importance weighting, while clipping limits deviation from the behavior policy during optimization.The behavior policy is denoted πθold, and ε specifies the clipping ratio.

4 Low-probability Regularization

Lp-Reg constructs a filtered proxy distribution that removes presumed noise and renormalizes the remaining probabilities. It selectively protects low-probability reasoning sparks through KL regularization and modified policy-gradient clipping.

  • 4 Low-probability Regularization: Lp-Reg constructs a proxy distribution by filtering presumed noise tokens and renormalizing probability mass over the remaining candidates.The proxy assigns zero probability to filtered tokens and preserves the others as a high-confidence reference.
  • 4.1 Proxy Distribution πproxy: The method filters tokens below a threshold τ, using either a fixed threshold or an adaptive min-p threshold based on the distribution’s peak probability.The primary experiments use the adaptive min-p strategy, while fixed thresholds are evaluated in ablations.
  • 4.2 Low-probability Regularization Objective: Lp-Reg removes the lower clipping bound to avoid clipping low-probability exploratory actions and adds a large upper bound U for numerical stability.The modified clipping is integrated into the GRPO policy-gradient objective.
  • 4.2 Low-probability Regularization Objective: The regularization activates for low-probability, non-noise tokens receiving negative advantage signals, thereby protecting reasoning sparks from over-penalization.The indicator requires all three conditions simultaneously: low sampling probability, non-zero proxy probability, and Ai,t < 0.
  • 4.2 Low-probability Regularization Objective: Lp-Reg uses forward KL divergence to penalize driving a token’s policy probability toward zero when that token retains non-zero proxy probability.This targets token elimination without requiring the policy to exactly match the heuristic proxy distribution.

5 Experiments

Experiments show that Lp-Reg supports stable, extended on-policy training and strong mathematical reasoning performance, while ablations identify low-probability targeting, noise filtering, and forward KL as important design choices.

  • 5.2 Results: 60.17% average accuracy on five mathematical reasoning benchmarks makes on-policy Lp-Reg 2.66% better than the next-best 80/20 method on Qwen3-14B.Lp-Reg achieves state-of-the-art performance across five benchmarks and both 14B and 32B model scales.
  • 5.2 Results: 3,000 training steps and 81,204 GPU-hours are sustained by Lp-Reg on Qwen2.5-32B-Base, extending on-policy reinforcement learning scaling.This extended run evaluates training stability beyond the shorter comparison runs.
  • 5.2 Results: On-policy training is more stable and performs better than off-policy training across 14B and 32B scales, while Clip-Higher is fragile across model generations.The comparison attributes on-policy stability to avoiding distribution shifts from mismatched sampling and training policies.
  • 5.2 Results: Lp-Reg’s entropy first decreases, then increases, and finally stabilizes, unlike methods that induce a continuous artificial entropy increase.This trajectory is linked to selectively protecting reasoning sparks without amplifying irrelevant low-probability noise.
  • 5.3 Ablation Study: High-entropy token regularization fails to improve performance and cannot sustain high entropy, which collapses after an initial spike.The ablation supports low-probability targeting over overall entropy as a proxy for valuable exploration.
  • 5.3 Ablation Study: Removing noise filtering causes catastrophic performance collapse and entropy explosion, while dynamic min-p filtering outperforms a fixed threshold.The results identify the threshold τ and its adaptive min-p formulation as critical for stable performance.

6 Analysis

Lp-Reg distinguishes useful low-probability reasoning sparks from noisy high-entropy tokens and preserves the former during training. This produces broader, more sustained exploratory-token use than GRPO or indiscriminate entropy regularization.

  • Low-probability tokens often contain meaningful reasoning markers, whereas high-entropy tokens are frequently dominated by noise.Examples of reasoning sparks include “But”, “Wait”, “Perhaps”, and “Alternatively”.
  • GRPO concentrates explorative tokens in low-entropy, high-probability regions, making them deterministic rather than exploratory.Tokens such as “wait” tend to appear only when the model is already confident.
  • Entropy loss scatters explorative tokens into extremely high-entropy states, sometimes exceeding 10, without producing useful exploratory signals.These occurrences do not integrate meaningfully into the reasoning process.
  • Lp-Reg sustains a higher fraction of explorative tokens throughout training than GRPO.It broadens their probability–entropy distribution while maintaining their practical use.
  • Lp-Reg filters noisy low-probability artifacts because meaningful exploratory tokens consistently have higher average probabilities than irrelevant tokens.This persistent probabilistic gap motivates using a filtered proxy distribution for regularization.

7 Conclusion

The paper attributes RLVR exploration collapse to eliminating valuable low-probability reasoning sparks and introduces Lp-Reg to selectively preserve them. This supports longer on-policy scaling and improves benchmark accuracy over baselines.

  • Figure 14 summarizes the probabilistic distinction between exploratory and irrelevant tokens across standard GRPO training steps.
  • Lp-Reg preserves reasoning sparks by filtering irrelevant noise and regularizing toward the remaining low-probability candidates.The method focuses on exploration quality rather than increasing randomness indiscriminately.
  • 2.66% absolute test-accuracy improvement over baselines accompanies continuous on-policy scaling for around 3,000 steps.

A.1 Hyperparameter Sensitivity Analysis

Lp-Reg remains stable across the tested settings of its low-probability percentile and min-p filtering ratio. Both hyperparameters produce tightly comparable training trajectories and final performance.

  • Figure 15 presents sensitivity analyses for the low-probability percentile ρ and min-p ratio κ.
  • ρ values of 0.005, 0.010, and 0.015 yield comparable training trajectories and highly comparable final performance.Higher ρ protects a wider range of candidate tokens.
  • κ values of 0.01, 0.02, and 0.03 maintain consistently high and tightly clustered training curves and final performance.Smaller κ produces more conservative filtering by removing fewer tokens.

B.1 Details of Sampling Probability Density

Token-level analyses show that Lp-Reg preserves the low-probability tails of reasoning sparks while suppressing irrelevant noise. These patterns remain visible across individual tokens rather than arising only from aggregation.

  • Figure 17 details observed sampling-probability distributions for individual reasoning sparks.
  • Standard GRPO eliminates low-probability sampling of reasoning sparks, while Lp-Reg consistently maintains their low-probability tail.The analyzed sparks include “but”, “wait”, “perhaps”, “alternatively”, and “however”.
  • Entropy loss amplifies irrelevant noise for nearly every examined token, whereas Lp-Reg keeps those tokens suppressed.The noise class includes tokens such as “cost”, “fine”, “balanced”, “ere”, and “trans”.
  • The token-level visualizations confirm that reasoning-spark elimination and irrelevant-noise amplification are consistent individual-token patterns, not aggregation artifacts.

B.3 Training Dynamics of Regularized Token

During training, the probability threshold and the share of low-probability tokens regularized both decline, indicating that the extreme low-probability range increasingly contains irrelevant noise. The filtering case study shows that tokens above the threshold are retained, including meaningful exploratory markers such as “Then” and “Wait.”

  • Training dynamics: The probability threshold ρ and regularization ratio both decline steadily with training, as the extreme low-probability range becomes increasingly dominated by irrelevant noise.This dynamic explains why a fixed threshold τ = 0.02 performs worse than the dynamic min-p threshold: it fails to regularize the lowest 1% of tokens.
  • Training dynamics: Figure 16 tracks the training dynamics of the probability threshold and regularization ratio.These are the two quantities whose decline is described in the accompanying analysis.
  • Token filtering: Tokens with probability greater than τ are retained by the filter, while tokens with probability smaller than τ are suppressed.Figures 20–22 illustrate this filtering behavior in a model-generated mathematical reasoning response.
  • Token filtering: The preserved low-probability tokens include meaningful exploratory markers such as “Then” and “Wait,” which guide the reasoning process.The paper contrasts these preserved meaningful tokens with filtered low-probability tokens in the generated-response case study.
Loading 2510.03222v2…