PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification

Youssef Mroueh

arXiv:2503.06639v4cs.LGstat.ML

TL;DR

The paper studies how GRPO trains LLMs with binary verifiable rewards and what its reward calibration and KL choices imply for policy dynamics. It derives weighted contrastive and closed-form policy updates across GRPO variants, then proves that success probabilities converge to amplified fixed points under supported conditions.

  • Problem

    GRPO’s effects under binary verifiable rewards, alternative reward normalizations, and previous- versus reference-policy KL regularization require a unified theoretical characterization.

  • Method

    The paper derives an adaptive weighted contrastive-loss formulation, explicit optimal policies, and probability-of-success recursions for multiple GRPO variants.

  • Results

    The induced probability-of-success sequence converges to a fixed point at least as large as the reference policy’s success probability, demonstrating success amplification.

  • Takeaways & Limitations

    GRPO’s reward normalization and KL anchor determine its weighting and success-probability dynamics, with mirror-style updates receiving stronger theoretical and practical guarantees.

  • Takeaways & Limitations

    The fixed-point guarantees require smoothing with ε > 0; when ε = 0, fixed-point existence on [0, 1] is no longer guaranteed.

Abstract

from arXiv · show

Group Relative Policy Optimization (GRPO) was introduced and used recently for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean + variance calibration of these rewards induces a weighted contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs mean + variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (mirror), penalizing divergence from a fixed reference model $π_{\mathrm{ref}}$, or combining both forms of regularization. For each, the optimal policy $π_n$ admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under $π_{n-1}$, as well as the policies $π_{n-1}$ and $π_{\mathrm{ref}}$. Iterating results in a sequence $\{π_n\}$ whose probability of success (PoS) obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy's probability of success.

1. Introduction

This paper analyzes GRPO for LLM reinforcement learning with verifiable rewards, showing that reward calibration yields an adaptive weighted contrastive loss and that policy success probabilities follow convergent recursions with amplification.

  • 1. Introduction: GRPO uses Monte Carlo rollouts to estimate advantages, avoiding a separately learned critic while training LLMs with verifiable rewards.The rewards include correctness, execution-based, and formatting or refusal checks that provide scalable binary signals.
  • 1. Introduction: The analyzed variants differ in reward normalization and whether KL regularization targets the previous iterate, a fixed reference, or both.The paper compares mean-plus-variance and mean-only calibration alongside reference and mirror-style regularization.
  • 1. Introduction: Mean-and-variance reward calibration makes GRPO equivalent to an adaptive, weighted contrastive loss on samples from the previous policy.The paper derives this equivalence as a central contribution and uses it to analyze multiple GRPO variants.
  • 1. Introduction: GRPO induces a probability-of-success recursion that converges to a fixed point under mild assumptions.The fixed point is characterized using the reference policy, previous policy, reward statistics, and regularization choices.
  • 1. Introduction: The limiting success probability satisfies p* ≥ p_ref, establishing success amplification for GRPO.The sequence’s stepwise monotonicity depends on the particular GRPO variant, and the dynamics are also verified empirically.

2. GRPO With verifiable Rewards as an Adaptive Weighted Contrastive Loss

For binary verifiable rewards, GRPO’s whitening transforms the objective into a success-probability-dependent weighted contrastive loss. The resulting unclipped policy iterations depend on the previous policy’s success probability and reference-policy regularization.

  • 2. GRPO With verifiable Rewards as an Adaptive Weighted Contrastive Loss: GRPO whitens binary verifiable rewards using their mean and variance under the old policy, then optimizes the resulting advantage with importance sampling.The original objective also includes reference-policy KL regularization and PPO-style clipping of the likelihood ratio.
  • 2.1. Whitening the Rewards in GRPO As means of Calibration: Whitening assigns more positive credit to rare successes and makes wrong outcomes more penalized when success is already likely.Thus, calibration depends nonlinearly on the old policy’s conditional probability of success.
  • 2.3. Stabilized GRPO: A smoothing factor ε > 0 stabilizes the whitened reward when the old policy’s success probability is at a boundary.The stabilized advantage is substituted into the clipped objective before analyzing the no-clipping limit.
  • 2. GRPO With verifiable Rewards as an Adaptive Weighted Contrastive Loss: The unclipped GRPO objective is equivalent to a weighted contrastive loss whose weights depend on the old policy’s probability of success.When p > 0.5, failures receive greater weight; when p < 0.5, successes receive greater weight.
  • 2.4. GRPO Iterations: GRPO iterations optimize over policy space, producing π_n from π_n−1 through its success probability and from π_ref through KL regularization.The analysis assumes π_0 = π_ref and uses importance sampling from π_n−1 in practical iterations.

3. GRPO Dynamics: Fixed Point iteration for Probability of Success

GRPO’s probability of success follows a fixed-point recurrence governed by the reference success probability and regularization. Its fixed points amplify the reference probability for prompts with nonzero, nonperfect reference success.

  • Fixed-point iteration: GRPO’s optimal policy induces a probability-of-success fixed-point iteration initialized at the reference probability.The recurrence applies for n ≥ 1, with p0(q) = pref(q).
  • Fixed-point properties: The update map hε,pref is continuous on [0, 1], ensuring at least one fixed point, but it need not be unique.Its monotonicity changes at p = 1/2: it decreases below 1/2 and increases above 1/2.
  • Fixed-point properties: When the iteration converges, its limiting probability of success is a fixed point of hε,pref and depends on the prompt through pref(q).The fixed-point condition follows by continuity of the update map.
  • Fixed-point visualization: Figure 2 visualizes fixed points as intersections between hε,pref and the line y = p across β and reference initializations.The figure description states that p* = 1 is almost always a fixed point, while fixed points are not unique in general.
  • Success amplification: For 0 < pref(q) < 1, every fixed point p*(q) exceeds the reference probability pref(q), demonstrating success amplification.The boundary cases pref(q) = 0 and pref(q) = 1 lead to fixed points 0 and 1, respectively, with stability requiring a condition on β at the boundaries.

4. Mirror GRPO: Mirror Descent with GRPO Calibrated Reward

Mirror GRPO regularizes against the previous iterate and produces monotone probability-of-success improvement for nonzero reference success. Unlike reference-only GRPO, it converges to the stable fixed point 1, while zero reference success remains absorbing.

  • Mirror regularization: Mirror GRPO uses regularization to the previous policy iterate, with the general mixed reference-and-previous regularization case treated separately.The α = 0 case removes reference regularization, while α = 1 corresponds to reference-only regularization.
  • Probability-of-success recurrence: Mirror-GRPO’s PoS recurrence replaces the reference probability inside the GRPO logit update with the previous iteration’s probability.This makes the update depend adaptively on pn−1(q).
  • Fixed-point structure: For any interior pn−1, the positive calibrated update makes pn > pn−1, so Mirror-GRPO has no interior fixed points.Its only fixed points are the boundary values 0 and 1.
  • Convergence and stability: For pref(q) ∈ (0, 1), Mirror-GRPO probabilities strictly increase and converge globally to the stable fixed point 1.The sequence is bounded by 1, and p* = 1 is globally stable.
  • Boundary behavior: When pref(q) = 0, both reference-regularized GRPO and Mirror-GRPO preserve zero success rather than creating successes.For Mirror-GRPO, pn(q) = 0 for all n.
  • Mixed regularization: Mixed KL regularization introduces a Rényi correction and loses monotone improvement, leaving no guarantee of global stability.The resulting dynamics can return to an interior fixed point in (0, 1).

5. Dr. GRPO and mean-only Normalization

Mean-only normalization changes GRPO’s probability-of-success dynamics by removing variance calibration. The resulting updates have distinct fixed-point behavior for ordinary and mirror variants.

  • Mean-only normalization: Dr. GRPO uses mean-only reward normalization, as proposed by Liu et al. (2025).The paper analyzes the corresponding no-clipping and Mirror Dr. GRPO iterations.
  • Reward weighting: The mean-only calibration assigns positive weight 1 − p to successes and negative weight p to failures.Here p denotes the policy’s probability of success.
  • No-clipping dynamics: For no-clipping Dr. GRPO, the fixed point is constant and given by p*(q) = σ(logit pref(q) + 1/β).This is contrasted with the adaptive progression obtained for Mirror Dr. GRPO.
  • Mirror dynamics: For Mirror Dr. GRPO, the logit follows an arithmetic progression and the probability of success approaches 1 for nondegenerate reference probabilities.The progression adds an increment proportional to 1/β at each iteration.
  • Variance comparison: Variance normalization can be interpreted as mean-only normalization with an adaptive effective regularization strength βeff.The paper states that low variance produces larger PoS-logit increments, whereas high variance produces smaller increments.

6. Discussion and Conclusion

The paper compares GRPO variants by reward calibration and KL anchor, deriving practical differences in success-rate dynamics, convergence, and resource requirements.

  • Variant dimensions: GRPO variants differ in reward calibration and KL regularization anchor, including mean-plus-variance versus mean-only normalization and reference, mirror, or two-KL regularization.Mean-plus-variance normalization yields nonlinear PoS weighting, whereas mean-only normalization yields linear weighting.
  • Variant behavior: Mirror GRPO guarantees monotonic improvement and convergence to PoS 1 for nondegenerate preferences, while adding reference regularization removes that guarantee.The two-KL variant has an internal fixed point and may fail to improve monotonically.
  • Practical implications: Mean-plus-variance normalization is PoS-equivalent to mean-only normalization with an adaptive effective KL coefficient βeff = βσ(q).Mean-only calibration with fixed β gives constant increments in log PoS odds, while adaptive regularization reproduces the mean-plus-variance behavior.
  • Practical implications: Reference-policy regularization increases bandwidth and latency because the reference policy must remain in memory during training.This cost can slow training for large models.
  • Practical implications: GRPO cannot create successes: when pref(q) = 0, zero is an absorbing fixed point, making success exploration important.The paper highlights maintaining coverage of successful outcomes as a practical requirement.

Appendix A. Experimental Validation

Experiments on GSM8K validate GRPO’s success-rate amplification and examine prompt-level trajectories across training iterations.

  • Setup: The experiment uses GSM8K with Qwen/Qwen2.5-0.5B-Instruct as the reference policy and trains with GRPO on eight GPUs.The reported settings include learning rate 5e−6, clipping ε = 0.2, KL regularizer β = 0.1, and µ = 10.
  • Success Rate Amplification: 21% to 37.5%: the average test-set success rate increased from the reference policy to the end of the GRPO epoch.Evaluation used 1319 GSM8K test questions with 50 samples per question.
  • Trajectory of Success rates Along GRPO Iterations: Prompt-level success-rate trajectories resemble a fixed-point algorithm, with some prompts converging quickly to p*=1 and others showing oscillatory behavior.Trajectories were estimated from 50 samples per prompt at model checkpoints.
  • Trajectory of Success rates Along GRPO Iterations: When pref = 0, the probability of success changes little across GRPO iterations, as predicted by the theory.This behavior is observed for randomly selected GSM8K prompts.

Appendix C. (No-Clipping ) GRPO : Proofs of Section 3

The no-clipping analysis characterizes GRPO’s PoS iteration through fixed points and local convergence, including conditions for monotonic improvement and stability.

  • Proofs of Section 3: GRPO’s probability of success satisfies an explicit iteration governed by the update map hε,pref.The recurrence is derived from the optimal-policy expression.
  • Fixed points: At least one fixed point exists because hε,pref is continuous on [0,1] and maps that interval into itself.The existence argument uses Brouwer’s Fixed Point Theorem.
  • Fixed points: Every fixed point p* exceeds pref, so the no-clipping GRPO update amplifies success relative to the reference probability.The theorem states hε,pref(p*) = p* and p* > pref.
  • Convergence: Local convergence follows when the iteration map satisfies a contraction condition near p*, with β > B(p*) providing a sufficient bound.The bound depends on p*, ε, and |2p*−1|; the analysis reports divergent behavior for β = 5 and pref = 0.001 in one case.
  • Convergence: The convergence condition is prompt-conditional, so uniform guarantees require taking the supremum over prompts.This scope qualification is stated explicitly in the proof discussion.

Appendix D. Mirror GRPO: Proof of Section 4

The mirror-GRPO analysis expresses the optimal update through the previous policy and proves monotonic success-rate growth, convergence, and a zero-success absorbing state.

  • Mirror GRPO: The generalized optimal policy has an exponential reweighting form determined by the anchor policy, calibrated advantage, and success-probability weights.The corresponding success probability is obtained by taking expectations over success events.
  • Mirror GRPO: The induced success probability can be written as σ(logit(pπ◦(q)) + Ω(pν(q))).Here Ω(p) is the sum of the positive- and negative-reward weights.
  • Monotonicity: For mirror updates anchored at πn−1, the logit of success probability strictly increases, implying pn > pn−1 for interior probabilities.Because the increment is strictly positive, no interior fixed point exists.
  • Convergence and stability: Starting from pref ∈ (0,1), the success probabilities increase strictly and converge to 1, while pref = 0 remains an absorbing fixed point.The fixed point is unique and stable when pref is strictly between zero and one.

Appendix E. GRPO with Two KL Regularizers: PoS Recursion, and Fixed-Point

The two-KL GRPO variant reduces to a probability-of-success recurrence involving the reference and previous policies, with a Rényi-divergence correction. Under a conditional-divergence assumption, the iteration amplifies success probability relative to the reference.

  • PoS recursion: Two KL regularizers can be rewritten as one KL regularizer to the geometric-mean anchor of the previous and reference policies.The anchor’s probability of success is expressed through the reference and previous success probabilities before deriving the recurrence.
  • PoS recursion: Theorem 9 gives a PoS recurrence for fixed α ∈ (0, 1) and β > 0 in the two-KL GRPO iteration.The recurrence follows from the single-KL formulation and the geometric-mean anchor.
  • PoS recursion: The logit of success combines the reference logit, the previous-policy logit, and a Rényi-divergence correction between conditional successes and failures.The correction is the difference between the Rényi divergences for failures and successes under the reference and previous policies.
  • Fixed-point amplification: If conditional successes are closer than failures between the reference and previous policies, the iteration’s success probability is amplified relative to the reference.The stated condition yields a nonnegative gap in logits that contracts by a factor of 1 − α across iterations.

Appendix F. Back to Parametric GRPO Iterations

The parametric GRPO analysis relates optimized policies to the non-parametric iteration through total-variation error bounds. When these errors remain small, parametric success probabilities stay close to the non-parametric fixed point, with exact convergence when the limiting error is zero.

  • Parametric approximation: The analysis assumes the parametric policy sequence starts at the reference policy and accumulates bounded total-variation deviations from non-parametric GRPO policies.The deviations are represented by per-iteration errors δ_n and a limiting bound δ* ∈ [0, 1).
  • Parametric approximation: Theorem 10 bounds the parametric policy’s success probability when the non-parametric sequence converges to the fixed point p* of hε,pref.The theorem applies under the stated total-variation assumption.
  • Convergence: When δ* = 0, the parametric policy sequence converges to the fixed point.This is the exact-convergence case identified after Theorem 10.
  • Convergence: As statistical, approximation, and optimization errors remain small, the parametric GRPO policy’s probability of success remains close to p*.The result covers policies estimated from samples and optimized, for example, with gradient descent.
Loading 2503.06639v4…