PaperlayerView exact revision on arXiv

Source-linked AI summary

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

Xixiang He, Qiyao Sun, Ao Cheng, Xingming Li, Xuanyu Ji, Hailun Lu, Runke Huang, Qingyong Hu

arXiv:2605.21125v2cs.LG

TL;DR

Homogeneous rewards within GRPO groups can eliminate the gradient signal, and standard metrics may not reveal this collapse. The paper introduces ACR to monitor it and AVSPO to inject virtual reward samples when needed. Across the evaluated model scales and mathematical benchmarks, AVSPO reduces collapse and improves accuracy while maintaining generalization on MMLU-Pro.

  • Problem

    Identical rewards within a GRPO response group make advantages collapse to zero, and this failure can remain invisible to loss curves and accuracy metrics.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    The paper introduces ACR to quantify gradient-ineffective groups and AVSPO, which uses real-time ACR monitoring to inject virtual reward samples into collapsed groups without additional LLM forward passes.

    Source: §1 · p. 2

  • Results

    AVSPO reduces advantage collapse by 58–63% and improves accuracy by 4–6 percentage points over GRPO across 0.5B–14B models on mathematical reasoning benchmarks, while maintaining MMLU-Pro generalization.

    Source: §1 · pp. 2–3, §6 · p. 9

  • Takeaways & Limitations

    The authors propose ACR as a diagnostic for binary-reward RLVR pipelines with deterministic verifiers.

    Source: §6 · p. 9

Abstract

from arXiv · show

Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs). However, GRPO is prone to advantage collapse, a failure mode where homogeneous rewards within a group (e.g., all correct or all incorrect answers) yield near-zero advantages and vanishing gradients. To address this, we introduce the Advantage Collapse Rate (ACR), the first diagnostic metric quantifying the proportion of training batches with ineffective gradients. Across models from 0.5B to 14B parameters on mathematical reasoning benchmarks, we show that ACR strongly predicts training stagnation and final performance. We then propose Adaptive Virtual Sample Policy Optimization (AVSPO), a lightweight extension of GRPO that injects virtual reward samples, guided by real-time ACR monitoring, to enable learning from homogeneous groups without additional model rollouts. AVSPO reduces advantage collapse by 58-63% relative to GRPO and yields consistent accuracy gains of 4-6 percentage points across all model scales, while maintaining generalization on the evaluated out-of-domain task. Code and datasets are available at https://github.com/hexixiang/Advantage-Collapse-Rate.

1. Introduction

GRPO can lose its learning signal when binary rewards are homogeneous within a response group, a failure that standard training metrics may not reveal. The paper introduces ACR to diagnose collapse and AVSPO to intervene, reporting lower collapse and higher accuracy across model scales while maintaining MMLU-Pro generalization.

  • Homogeneous within-group rewards make advantages collapse to zero, and this occurs in mathematical reasoning with binary verification.Across five model scales, 28–45% of training batches exhibit complete collapse, which loss curves and accuracy do not reveal.Source: §1 · pp. 1–2
  • Existing approaches address gradient ineffectiveness through critic-based variance reduction, costly step-level annotations, uniform entropy pressure, or system-level GRPO changes, but do not quantify collapse severity beforehand.Source: §1 · p. 2
  • Early-stage ACR explains 62% of variance in final performance, enabling practitioners to identify problematic configurations before accuracy degradation becomes visible.Source: §1 · p. 2
  • AVSPO injects virtual reward samples when collapse is detected, creating non-zero advantages without additional LLM forward passes.The overview describes this as restoring reward variance before recomputing advantages.Source: §1 · p. 2, Figure 2 · p. 2
  • AVSPO reduces advantage collapse from 28–45% to 11–18% (58–63% relative reduction) and yields 4–6 percentage-point accuracy gains over GRPO across 0.5B–14B models on six mathematical reasoning benchmarks.The paper also reports maintained generalization on MMLU-Pro.Source: §1 · pp. 2–3

2. Related Work

Prior work addresses policy optimization and training diagnostics through several approaches, but the section identifies batch-level advantage collapse in GRPO as an unmet diagnostic target.

  • GAE requires a critic, PRMs require expensive annotations, and entropy regularization ignores reward structure; these approaches do not directly diagnose GRPO collapse.Source: §2 · p. 3
  • Prior GRPO variants address optimization, clipping, system recipes, value estimation, token weighting, reward shaping, exploration, and entropy control; AVSPO instead intervenes at the reward-statistics level.Scaf-GRPO addresses collapse through progressive in-prompt scaffolding, making it complementary to AVSPO.Source: §2 · p. 3
  • Standard diagnostics include entropy, KL divergence, and gradient norms, while prior diagnostics target reward hacking, off-policy entropy instability, or token-level imbalance rather than batch-level collapse in GRPO.ACR is presented as filling this diagnostic gap.Source: §2 · p. 3

3. Preliminaries

The paper frames verifiable mathematical reasoning as learning from sparse binary outcome feedback and describes GRPO’s group-based advantage calculation. Homogeneous rewards make that calculation ineffective, motivating diagnosis and intervention.

3. Preliminaries

  • Mathematical reasoning is framed as a contextual bandit: a model generates a solution trajectory and an automated verifier assigns a binary correctness reward.The outcome-only supervision leaves the challenge of extracting effective learning signals from binary feedback.Source: §3 · p. 3

3.1. Group Relative Policy Optimization

  • GRPO samples multiple responses per question and computes advantages through intra-group reward comparisons, avoiding the critic network used by actor-critic methods.Source: §3.1 · p. 3

3.2. The Advantage Collapse Phenomenon

  • When all responses in a group receive the same binary reward, reward variance vanishes and GRPO’s advantages and gradient contribution collapse to zero.This occurs for both uniformly incorrect responses to hard problems and uniformly correct responses to easy ones; standard loss and accuracy metrics can appear stable.Source: §3.2 · pp. 3–4

4. Methodology

The methodology combines real-time collapse diagnosis with conditional virtual-sample augmentation, aiming to recover learning signals while retaining GRPO’s computational efficiency. Its analysis connects reward variance to gradient strength and notes that augmentation introduces bias.

4.1. Advantage Collapse Rate (ACR)

  • ACR measures the proportion of groups in a batch with negligible gradient signals, using reward statistics already computed during GRPO training.ACR near zero indicates reward-diverse groups, near one indicates widespread homogeneity, and intermediate values indicate the fraction of computation spent on ineffective updates.Source: §4.1 · p. 4

4.2. Adaptive Virtual Sample Policy Optimization

  • AVSPO adds synthetic reward values only to normalization statistics, not to the policy gradient, so collapsed groups can yield non-zero advantages for real samples.The number of virtual samples is adaptive, with stronger support as ACR rises; stratified rewards create variance for both all-correct and all-incorrect groups.Source: §4.2 · pp. 4–5
  • AVSPO triggers augmentation conditionally and adapts its threshold to training progress, reducing unnecessary intervention during improvement and strengthening correction during stagnation under high ACR.The recomputed advantages apply to real samples, and the procedure adds no LLM forward passes while retaining O(NG) per-iteration complexity.Source: §4.2 · p. 5, §4.3 · p. 6

4.3. Theoretical Analysis

  • As reward variance approaches zero, the sum of squared advantages and the group’s policy-gradient contribution vanish, supporting ACR as a diagnostic of gradient ineffectiveness.Source: §4.3 · p. 5
  • For collapsed groups, virtual augmentation enables policy updates but introduces bias relative to standard GRPO gradients, a tradeoff the paper describes as favorable to gradient stagnation.Source: §4.3 · p. 5

5. Experiments

Across six mathematical reasoning benchmarks, AVSPO reduces advantage collapse and improves accuracy over GRPO. Experiments also validate ACR as a diagnostic and examine its sensitivity, AVSPO design choices, and filtering versus augmentation.

5.1. Experimental Setup

  • Experiments use six Qwen2.5 models from 0.5B to 14B, Level3-500 training data, six math benchmarks, and MMLU-Pro for out-of-domain evaluation.Models train for 500 steps; representative multi-seed experiments report p < 0.05 for all AVSPO improvements.Source: §5.1 · p. 6

5.2. Main Results

  • 58–63% relative ACR reduction and +4 to +6 accuracy points over GRPO; Qwen2.5-Math-1.5B shows the largest gain (+6.1%) at baseline ACR=0.40.Table 1 presents results across configurations; higher-baseline-ACR models benefit more.Source: §5.2 · p. 6
  • AVSPO outperforms DCPO by 2.9%, INTUITOR by 6.8%, and RENT by 8.9%; the authors suggest restoring batch-level gradient signal is more effective than reward engineering given verification.The compared methods target policy divergence or reward signals rather than advantage diversity.Source: §5.2 · p. 6
  • Gains are largest on GSM8K and MATH-500, more modest on AMC and AIME, and smaller but consistent for larger models, whose baseline ACR is 0.21 at 14B versus 0.45 at 0.5B.The authors suggest model capacity becomes limiting on competition-level problems.Source: §5.2 · p. 6

5.3.1. ACR PREDICTS FINAL PERFORMANCE

  • Across 245 setups, ACR100 is negatively correlated with final MATH-500 accuracy (r = −0.785, p < 10−8); the fit explains 62% of performance variance and supports early detection of poor configurations.The analysis uses mean ACR over the first 100 training steps.Source: §5.3.1 · pp. 6–7

5.3.2. SENSITIVITY TO TRAINING CONDITIONS

  • ACR responds to capacity, temperature, group size, and difficulty: G ∈[6, 8] balances gradient effectiveness and cost, while both easy and hard problems produce uniform rewards.Moderate temperatures T = 0.3–0.5 balance exploration and exploitation; intermediate difficulty supports learning.Source: §5.3.2 · p. 7, Figure 3 · p. 7

5.4. Ablation Studies

  • AVSPO’s stratified reward assignment achieves ACR 0.15 and +8.6% accuracy; Error-Only mainly suppresses all-wrong collapse, Correct-Only all-correct collapse, and the full method combines both effects.Random sampling yields +3.5% with variance, while fixed partial credit yields +4.9% but does not span the reward spectrum.Source: §5.4 · pp. 7–8
  • Adaptive thresholding reaches 60% accuracy in 295 steps versus 380 for the optimal fixed threshold, with 1.8% higher final accuracy; α = 0.5 is optimal, and tested τ_adapt and η ranges vary accuracy by less than 1%.Smaller α leaves residual collapse, while larger α over-augments and drops accuracy by 2.1%.Source: §5.4 · pp. 8–9

6. Conclusion

The paper presents ACR as a zero-overhead diagnostic and AVSPO as an ACR-guided intervention for GRPO advantage collapse. Across 0.5B–14B models on mathematical reasoning benchmarks, AVSPO reduces collapse and improves accuracy.

6. Conclusion

  • AVSPO reduces collapse rates by 58–63% and improves accuracy across 0.5B–14B models on mathematical reasoning benchmarks.ACR is presented as a zero-overhead metric for quantifying gradient ineffectiveness and predicting performance.Source: §6 · p. 9

Impact Statement

The impact statement frames ACR and the paper’s algorithmic improvements as tools for more stable and efficient RLVR training. It also connects real-time diagnosis with potentially reduced computational waste and greater transparency.

Impact Statement

  • ACR may help practitioners identify training inefficiencies in real time, potentially reducing computational waste and its associated energy footprint.The impact statement also links the diagnostic tools and algorithmic improvements to transparency in LLM alignment pipelines.Source: Impact Statement · p. 9
  • The authors hope these contributions facilitate more efficient development of language-model reasoning capabilities and further research on policy-optimization diagnostics.Source: Impact Statement · p. 9

A. Implementation Details

The appendix describes experiment settings and training-data construction, then begins a proof using the normalized-advantage definition.

A. Implementation Details

A.1. Training Data Construction

  • Level3-500 contains 500 randomly selected Level 3 problems from the MATH training split, while MATH-500 uses the official test split to avoid data leakage.Difficulty levels are assigned using Qwen2.5-Math-1.5B’s success rate; Level 3 corresponds to 30–50% success.Source: §A.1 · p. 12
  • Training runs use one epoch of 500 steps; the full experimental matrix reports single-run results, with five-seed checks on four models across MATH-500 and GSM8K.The authors justify 500 steps by learning curves plateauing within 400 steps and report that ACR and accuracy stabilize before training ends.Source: §A.1 · pp. 12–13

B. Proofs

  • The proof of Proposition 4.2 begins from the normalized advantage definition, using reward, mean reward, and reward standard deviation.Source: B. Proofs · p. 13

B.1. Virtual Samples in Homogeneous Groups: Directionality and Stability

The appendix analyzes how virtual rewards alter advantages and gradients for homogeneous groups, then bounds the resulting bias and states computational costs. Its results describe update direction and convergence under stated assumptions.

B.1. Virtual Samples in Homogeneous Groups: Directionality and Stability

  • AVSPO recomputes advantages for collapsed binary-reward groups after virtual-reward augmentation, producing uniform advantages within each such group.The analysis distinguishes all-correct from all-incorrect groups; success and failure sets are defined by reward values 1 and 0.Source: §B.1 · p. 13
  • In homogeneous groups, AVSPO’s expected update decreases failure-set probability for all-incorrect groups and increases success probability for all-correct groups; clipping limits over-sharpening.The all-correct update magnitude is bounded, and PPO clipping makes the gradient flat once token-level ratios exceed 1 + ε.Source: §B.1 · pp. 14–15

B.2. Bias Upper Bound and Convergence Implication

  • The appendix bounds AVSPO’s per-iteration gradient discrepancy relative to GRPO using the batch collapse rate and the number of virtual samples.When augmentation is not triggered, the estimators are equal; the stated bias bound is for binary rewards.Source: §B.2 · pp. 15–16
  • Under the stated smoothness, boundedness, and bias assumptions, AVSPO’s iterates reach an O(b)-stationary region when the bias is uniformly bounded by b.If the bias tends to zero, the lim inf expected gradient norm is zero; if expected ACR tends to zero, the stated corollary gives convergence to GRPO’s first-order stationary set under standard assumptions.Source: §B.2 · pp. 16–17

C. Algorithm and Computational Complexity

  • AVSPO has O(NG) per-iteration complexity, matching GRPO, and requires no additional LLM forward passes.Virtual-sample generation adds at most O(NG) operations, with negligible memory overhead.Source: C. Algorithm and Computational… · p. 17

C.1. Empirical Computational Overhead

AVSPO adds negligible training overhead and improves deterministic-verifier code-generation performance over GRPO. The appendix also specifies the math-task prompt and binary answer-checking reward.

C.1. Empirical Computational Overhead

  • AVSPO adds approximately 3ms per step (<0.01% overhead), or ∼1.5 seconds over a 500-step run, with no additional LLM calls or meaningful GPU memory.LLM generation and input preparation account for over 96% of computational cost; reward and advantage computation account for 0.3%.Source: §C.1 · pp. 18–19

D. Prompt Template and Reward Function

  • Math-task training prompts request step-by-step solutions ending in \boxed{}, and the reward checks symbolic equivalence of the extracted final answer against ground truth.Base-model evaluation uses the better accuracy from the study prompt or the default Qwen system prompt.Source: Figure 5 · p. 19, D. Prompt Template and Reward Function · p. 19

E. Code Generation with Deterministic Verifiers

F. Sensitivity Analysis Details

The analysis shows that ACR and accuracy vary with model scale, sampling temperature, group size, and problem difficulty. Case studies illustrate collapse under uniform outcomes and compare reasoning outputs from GRPO- and AVSPO-trained models.

F. Sensitivity Analysis Details

  • Larger models generally show lower ACR and higher accuracy, but the 7B model’s ACR (0.15) is below the 14B model’s (0.36), indicating task-model matching also matters.The 7B result is not strictly monotonic with scale; when capability exceeds task difficulty, uniformly correct responses can increase ACR.Source: F. Sensitivity Analysis Details · p. 20
  • ACR falls from 0.46 at T = 0.1 to 0.08 at T = 1.0, while accuracy peaks at T = 0.3–0.5, so lower collapse alone does not ensure better performance.Low temperatures yield homogeneous outputs; high temperatures increase reward diversity but can introduce excessive randomness.Source: F. Sensitivity Analysis Details · p. 20
  • Increasing group size G from 2 to 8 reduces ACR from 0.52 to 0.20, with diminishing gains beyond G = 6 as computational cost scales linearly.The analysis identifies G ∈[6, 8] as a practical balance between gradient effectiveness and resource constraints.Source: F. Sensitivity Analysis Details · p. 20
  • Both easy (Level 0–1) and hard (Level 5–6) problems have ACR >0.45, whereas intermediate Level 3–4 reaches ACR ≈0.30 and accuracy >55%.The appendix links the U-shaped pattern to uniform success on easy tasks and uniform failure on hard tasks; it also suggests intermediate-difficulty curricula could mitigate collapse.Source: F. Sensitivity Analysis Details · p. 20, §G.4 · p. 22

G.1. ACR Dynamics Across Model Scales

  • ACR trajectories separate by model scale within the first 50 steps; 7B and 14B models converge below 0.35 within 100 steps, while smaller models remain high.Larger models also show more stable ACR trajectories with less variance.Source: §G.1 · p. 20

G.2. ACR Dynamics Across Sampling Temperatures

  • Temperature curves show high ACR at T = 0.1 and 0.3, low ACR at T = 0.9 and 1.0, and an accuracy-favored balance around T ≈0.3–0.5.Accuracy does not monotonically increase at higher temperatures despite lower ACR.Source: §G.2 · p. 21, Figure 7 · p. 21, Figure 6 · p. 21

G.3. ACR Dynamics Across Group Sizes

  • Larger groups smooth ACR trajectories, while increasing G from 2 to 4 brings substantial ACR reduction and subsequent increases yield diminishing benefits.The figure summary reports accuracy returns diminishing beyond G = 6.Source: §G.3 · p. 22, Figure 8 · p. 22

H. Case Studies

  • In examples with G = 8, all-incorrect and all-correct groups both have σR = 0 and vanishing GRPO gradients; AVSPO examples show more consistent reasoning than GRPO.The output comparison describes similar initial problem decomposition, followed by reasoning leaps or arithmetic errors in GRPO outputs while AVSPO maintains more rigorous progression.Source: §H.1 · p. 23, §H.2 · p. 23, Figure 10 · p. 24

I. Statistical Significance Analysis

Across four models and two benchmarks, AVSPO improves on GRPO across all evaluated seed-model-benchmark combinations. The gains are statistically significant and accompanied by lower outcome variance.

I. Statistical Significance Analysis

  • AVSPO gains range from +5.6 to +9.6 percentage points over GRPO across all seed-model-benchmark combinations, with non-overlapping distributions.The evaluation covers four models from 0.5B to 14B parameters and MATH-500 and GSM8K; the multi-seed means closely match the single-run results.Source: §I · p. 25, Table 9 · p. 25, Figure 12 · p. 26
  • All eight AVSPO-versus-GRPO comparisons yield p < 0.05 under paired t-tests.Source: §I · p. 25, Table 9 · p. 25
  • AVSPO has lower mean standard deviation than GRPO (1.40 vs. 1.70), indicating less variable results across the reported comparisons.The reported distributions are non-overlapping for the compared combinations.Source: §I · p. 25, Figure 12 · p. 26
Loading 2605.21125v2…