PaperlayerView exact revision on arXiv

Source-linked AI summary

Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities

Pengyi Li, Elizaveta Goncharova, Andrey Kuznetsov, Ivan Oseledets

arXiv:2602.05281v2cs.LGcs.CL

TL;DR

RLVR-based policy optimization can collapse onto a few high-probability reasoning paths, reducing diversity. ProGRPO addresses this by reweighting advantages with probability-based confidence signals and reports stronger multi-sample performance, including gains over GRPO on Qwen2.5-7B.

  • Problem

    RLVR training can produce entropy and mode collapse because reward-weighted optimization concentrates probability on dominant paths, suppressing equally valid alternatives.

  • Method

    ProGRPO combines prompt and answer probability signals with verifiable rewards through an Advantage Re-weighting Mechanism.

  • Results

    13.9%: ProGRPO improves Qwen2.5-7B Pass@32 over GRPO while also improving Pass@1 by 5.7%.

  • Takeaways & Limitations

    ProGRPO maintains competitive standard-metric performance while improving multi-sample generation of diverse and correct solutions.

  • Takeaways & Limitations

    The paper describes its techniques as methodological and does not anticipate significant additional societal or ethical consequences beyond common LLM risks.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an indispensable paradigm for enhancing reasoning in Large Language Models (LLMs). However, standard policy optimization methods, such as Group Relative Policy Optimization (GRPO), often converge to low-entropy policies, leading to severe mode collapse and limited output diversity. We analyze this issue from the perspective of sampling probability dynamics, identifying that the standard objective disproportionately reinforces the highest-likelihood paths, thereby suppressing valid alternative reasoning chains. To address this, we propose a novel Advantage Re-weighting Mechanism (ARM) designed to equilibrate the confidence levels across all correct responses. By incorporating Prompt Perplexity and Answer Confidence into the advantage estimation, our method dynamically reshapes the reward signal to attenuate the gradient updates of over-confident reasoning paths, while redistributing probability mass toward under-explored correct solutions. Empirical results demonstrate that our approach significantly enhances generative diversity and response entropy while maintaining competitive accuracy, effectively achieving a superior trade-off between exploration and exploitation in reasoning tasks. Empirical results on Qwen2.5 and DeepSeek models across mathematical and coding benchmarks show that ProGRPO significantly mitigates entropy collapse. Specifically, on Qwen2.5-7B, our method outperforms GRPO by 5.7% in Pass@1 and, notably, by 13.9% in Pass@32, highlighting its superior capability in generating diverse correct reasoning paths.

1. Introduction

RLVR improves reasoning but can cause entropy and mode collapse by concentrating probability on dominant solutions. ProGRPO reshapes advantage estimates with probability signals to promote diverse reasoning while preserving performance.

  • Motivation: RLVR can improve task success rates, but its reward-weighted objective concentrates probability on dominant reasoning trajectories.This compresses probability mass assigned to low-frequency reasoning paths that are also valid.
  • Motivation: Existing entropy, clipping, and token-promotion methods mainly make local reward-optimization changes, limiting their ability to model diverse reasoning paths.
  • ProGRPO: ProGRPO re-examines advantage construction from the perspective of token-by-token probabilistic sampling.
  • ProGRPO: ProGRPO combines prompt and answer probability signals with verifiable rewards to reshape advantage and the effective trajectory distribution.The method is designed to alleviate entropy collapse and improve reasoning-path diversity and training stability.
  • Results: ProGRPO is evaluated across reasoning and code-generation benchmarks using Qwen2.5 and DeepSeek models, with reported OOD adaptability.
  • Results: 5.7% and 13.9%: on Qwen2.5-7B, ProGRPO improves Pass@1 and Pass@32 over GRPO, respectively.The reported gains indicate stronger exploration efficiency and greater output diversity.

2. Preliminaries

The preliminaries move from high-variance REINFORCE to PPO’s clipped trust-region updates and then GRPO’s value-free group-relative formulation. GRPO samples multiple outputs and normalizes rewards against their group mean.

  • REINFORCE: REINFORCE maximizes expected cumulative reward, but its gradient estimates have high variance and can make training unstable.
  • PPO: PPO stabilizes policy updates through clipping that enforces a trust-region constraint.
  • PPO: PPO requires a critic to estimate advantages, creating significant computational overhead.
  • GRPO: GRPO removes the value function by using verifiable rewards and group-based relative advantages.
  • GRPO: GRPO samples a group of outputs for each query and uses the group mean as the baseline.
  • GRPO: GRPO computes each advantage by normalizing its reward relative to the rewards within the sampled group.

3. Methodology

ProGRPO reweights group advantages using prompt and answer confidence, with confidence estimated from low-probability tokens. Selective normalization focuses the signal on uncertain positions rather than diluting it across high-confidence tokens.

  • Advantage Re-weighting Mechanism: ProGRPO redefines the advantage using sample-level signals derived from the model’s low-probability-token normalized likelihood.
  • Advantage Re-weighting Mechanism: Prompt confidence c(q_i) is used as a dynamic control term that regulates training according to the model’s familiarity with the prompt.
  • Advantage Re-weighting Mechanism: Answer confidence c_θ(o_j | q_i) represents the model’s confidence when generating answer o_j for prompt q_i.
  • Advantage Re-weighting Mechanism: Advantage reweighting reshapes the effective reward distribution while preserving meaningful gradient signals when group answers may share the same correctness status.
  • Low-Probability Token Length Normalization: Approximately 20% of response positions have the highest predictive uncertainty, while remaining positions often assign above 0.9 probability to the top token.
  • Low-Probability Token Length Normalization: Full-sequence normalization can dilute the reward signal by including trivial high-confidence tokens.ProGRPO instead applies selective normalization to the approximately 20% most uncertain positions.
  • Low-Probability Token Length Normalization: Focusing on informative tokens preserves confidence variations that more directly reflect reasoning quality for policy optimization.

4. Experiments

Experiments evaluate ProGRPO across mathematical and code-generation tasks, model scales, OOD settings, and training analyses of entropy, diversity, and confidence re-weighting.

  • Experimental Settings: ProGRPO is evaluated under GRPO using DAPO mathematics data, DeepCoder code data, and Qwen2.5 and DeepSeek base models.The experiments use Qwen2.5-7B, Qwen2.5-32B, and DeepSeek-R1-Distill-Qwen models across the two domains.
  • Main Results: ProGRPO consistently outperforms GRPO and FlowRL across mathematical reasoning and code-generation domains.The primary results cover Tables 2 and 3 and compare ProGRPO with direct reward maximization and reward matching baselines.
  • Code Generation: 1422.49 CodeForces rating exceeds GRPO by nearly +180 rating and FlowRL by +293, while HumanEval+ reaches an Avg@16 score of 84.01%.On LiveCodeBench, ProGRPO also achieves an Avg@16 score of 36.47 and Pass@16 of 54.12.
  • Generalization: ProGRPO maintains a clear advantage over GRPO on unseen out-of-distribution data distributions.The OOD results are reported as evidence of adaptability beyond standard benchmarks.
  • Training Analysis: ProGRPO increases output entropy while maintaining comparable Pass@1 and stable improvements in Pass@k at higher entropy levels.The analysis attributes this pattern to generating more diverse valid reasoning paths rather than randomization.
  • Training Analysis: α = 0.3 achieves the best balance between performance and stability, whereas larger values can weaken supervision from the original reward.At α = 0, the method reduces to pure GRPO; increasing α to 0.3 improves Pass@1 and Pass@32, while α = 0.7 or 1 degrades performance.

5. Related Work

Related work studies exploration and entropy collapse in reinforcement-learning-driven reasoning models, including confidence-based approaches that seek a better balance between exploration and stability.

  • Reasoning Models: Reinforcement-learning-driven reasoning models generate explicit, lengthy chains of thought before final answers.This paradigm includes models such as OpenAI o1, DeepSeek, Kimi, and Qwen.
  • Exploration in Reinforcement Learning: Entropy-based signals and entropy regularization have been explored to improve exploration, but reported empirical gains, stability, and effectiveness remain limited.These approaches are presented as prior efforts to address exploration and entropy collapse.
  • Exploration in Reinforcement Learning: Overconfidence can make the policy increasingly deterministic, leading to entropy collapse.The paper connects model confidence with policy determinism in policy learning.
  • Exploration in Reinforcement Learning: The paper asks what happens when the model is insufficiently confident and proposes confidence balancing based on answer correctness.The stated goal is a better trade-off between exploration and stability.

6. Conclusion

The conclusion presents ProGRPO as a generative-probability-based approach to severe entropy collapse in RLVR, combining probability-aware mechanisms with competitive performance and stronger multi-sample diversity.

  • 6. Conclusion: ProGRPO addresses severe entropy collapse during RLVR training through Low-Probability Token Length Normalization and confidence-aware Advantage Reweighting.The method is designed to mitigate mode collapse while preserving reasoning capabilities.
  • 6. Conclusion: ProGRPO maintains competitive standard-metric performance while showing a significant advantage in multi-sample settings such as Pass@k.The reported pattern indicates generation of diverse and correct solutions.

Impact Statement

The impact statement frames the work as a methodological enhancement for existing reasoning and code-generation training paradigms, with formal analysis describing how AMR redistributes reinforcement among correct paths.

  • Impact Statement: The work targets stability and diversity in RLVR by mitigating mode collapse during policy optimization without introducing new model capabilities or application domains.Its stated focus is encouraging more diverse and robust reasoning behaviors.
  • Impact Statement: The techniques are primarily methodological and intended to enhance existing training paradigms for reasoning and code generation tasks.The statement identifies this as the main scope of the contribution.
  • A.1. Preliminaries and the Homogeneity Limitation: In standard GRPO, all correct responses receive identical advantages, allowing initially higher-probability correct paths to suppress other valid paths.The formal analysis describes this as a positive feedback loop leading to entropy collapse.
  • A.2. Theorem 1: Convergence to Confidence Equilibrium and Difficulty Calibration: AMR uses differential advantages so over-optimized correct paths receive less reinforcement than under-explored paths, driving equilibrium toward equal confidence among correct responses.The theorem describes this negative feedback as inducing a maximum-entropy state over the success manifold.
  • A.2. Theorem 1: Convergence to Confidence Equilibrium and Difficulty Calibration: Prompt-specific group means remove prompt-difficulty bias, isolating within-prompt path differences so diversity pressure applies across easy and hard prompts.A fixed global threshold would penalize correct answers on easy prompts while ignoring diversity on hard prompts.

A.3. Theorem 2: Semantic Diversity vs. Syntactic Fluency

Theorem 2 claims that AMR promotes semantic diversity in reasoning tokens while preserving syntactic certainty in functional segments by targeting low-probability branching points.

  • A.3. Theorem 2: Semantic Diversity vs. Syntactic Fluency: Unlike standard entropy maximization, AMR targets branching points in reasoning tokens while protecting linguistic fluency.This design is presented as the mechanism for semantic-level diversity with syntactic certainty preserved.
  • A.3. Theorem 2: Semantic Diversity vs. Syntactic Fluency: AMR separates functional tokens from reasoning tokens when analyzing confidence over the low-probability token set.The low-probability set contains positions whose token probability falls below the bottom-20% threshold.
  • A.3. Theorem 2: Semantic Diversity vs. Syntactic Fluency: Functional tokens are excluded from the low-probability set because grammatical determinism makes their probabilities approximately one.The paper gives “The answer is” as an example of a functional segment.

A.4. Theorem 3: Preservation of Correctness

The AMR mechanism preserves suppression of incorrect responses while modulating penalty magnitude, so exploration remains confined to valid reward paths.

  • AMR strictly bounds exploration within the valid reward landscape and does not encourage incorrect responses.
  • Incorrect responses receive a negative base advantage when their verifiable reward is zero.
  • For small α, AMR preserves the negative gradient sign on incorrect paths rather than reversing the objective.The paper gives α = 0.1 as an example.

A.5. Theorem 4: Implicit Entropy Regularization on the Success Manifold

Theorem 4 characterizes AMR as an implicit entropy-maximizing mechanism over correct solutions: it down-weights over-confident paths, amplifies under-confident ones, and equalizes their probability mass.

  • Entropy collapse in standard GRPO: Standard GRPO amplifies initially more likely correct paths, causing probability concentration and entropy collapse.
  • AMR as an entropy-promoting mechanism: AMR uses confidence relative to the group average to down-weight higher-confidence correct paths and amplify lower-confidence paths.The confidence score is described as a monotonic proxy for log πθ(oi | q).
  • AMR as an entropy-promoting mechanism: The AMR update direction aligns with minimizing KL divergence between the correct-solution policy and the uniform distribution.This alignment implicitly encourages higher entropy over correct solutions.
  • Stationary points: At stationary points, AMR equalizes confidence and probability mass across all correct solutions, yielding a maximum-entropy configuration.
  • ProGRPO preserves correctness while steering the policy toward a high-entropy distribution over the success manifold.The paper presents this as preventing or mitigating mode collapse.

B. Additional Results

Additional analyses report more balanced and higher-entropy rollout probabilities, while reproduction results expose instability in a prior entropy-based method and ablations emphasize relative confidence reweighting.

  • The reproduced method from Cui et al. can match reported performance in some cases but is extremely unstable and prone to collapse.
  • Math500 entropy results are consistent with the analysis on AIME2024, which has only 30 samples.
  • Higher token-level entropy and more balanced probabilities across 32 rollouts are reported for the proposed model on a single AIME2024 sample.The analysis is described as outperforming GRPO in reliability and generation diversity.
  • Relative confidence reweighting selectively attenuates dominant paths while promoting under-explored correct reasoning trajectories.
Loading 2602.05281v2…