Source-linked AI summary
Your Group-Relative Advantage Is Biased
Fengkai Yang, Zherui Chen, Xiaohan Wang, Xiaodong Lu, Jiajun Chai, Guojun Yin, Wei Lin, Shuai Ma, Fuzhen Zhuang, Deqing Wang, Yaodong Yang, Jianxin Li, Yikun Ban
TL;DR
Group-relative advantage estimation in RLVR lacks a complete theoretical characterization and is biased relative to true advantages. The paper analyzes this bias and introduces HA-DW, which uses an evolving difficulty anchor and training history to adapt advantage weights. Across five mathematical reasoning benchmarks, HA-DW consistently improves GRPO and its variants, while the study’s main scope remains group-relative methods.
Problem
Group-relative RL uses finite within-group rewards to avoid learned critics, but the estimator’s theoretical bias and its effect on prompt-difficulty balancing were poorly characterized.
Method
HA-DW dynamically adjusts advantage weights using an evolving difficulty anchor that integrates long-term reward trends and historical training information.
Results
HA-DW consistently improves GRPO and its variants across five mathematical reasoning benchmarks and different model scales.
Takeaways & Limitations
Correcting biased group-relative advantage estimates can help RLVR better leverage overlooked prompts and improve reasoning performance.
Takeaways & Limitations
The analysis and HA-DW application primarily focus on group-relative methods, with broader extension left for future work.
Abstract
from arXiv · showhide
Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we uncover a fundamental issue of group-based RL: the group-relative advantage estimator is inherently biased relative to the true (expected) advantage. We provide the first theoretical analysis showing that it systematically underestimates advantages for hard prompts and overestimates them for easy prompts, leading to imbalanced exploration and exploitation. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.
1 Introduction
Group-relative RL avoids a learned critic by estimating advantages from sampled groups, but this estimator is biased: it underestimates hard prompts and overestimates easy ones. HA-DW addresses this bias through adaptive reweighting and improves results across mathematical reasoning benchmarks.
- Motivation: Group-relative methods use within-group average rewards to estimate advantages without a separate critic model.This design is central to GRPO and its variants but has lacked detailed theoretical characterization.
- Problem: The group-relative advantage estimator underestimates expected advantages for hard prompts and overestimates them for easy prompts.The resulting imbalance can cause under-learning on hard questions and over-exploitation of easy ones.
- Approach: HA-DW dynamically reweights advantage estimates using an evolving difficulty anchor that incorporates long-term reward trends and historical training information.The method is designed to compensate for group-relative estimation bias and balance exploration with exploitation.
- Results: HA-DW consistently improves GRPO and its variants across five mathematical reasoning benchmarks and different model scales.The paper presents these comparisons in Figure 1(a).
- Scope: The paper focuses on exposing statistical bias in group-relative algorithms rather than modeling every RLVR setting.It argues that lightweight corrections can nevertheless yield consistent gains.
2 Why Your Advantage Estimation is Biased?
The analysis models binary-reward group-relative RL under non-degenerate sampling and shows that finite-group advantage estimates are systematically biased by prompt difficulty. The bias is strongest for extreme prompts and smaller groups, motivating precise probabilistic characterization.
- Setup: Group-relative RL samples G responses, assigns binary rewards, and computes an empirical group baseline from total reward divided by G.The resulting advantage estimate is used for policy updates by GRPO-style methods.
- Definitions: The expected reward pt is estimated by the finite-group baseline ˆpt, making ˆAt,i an empirical estimator of the expected advantage At,i.Different group-relative variants primarily differ in how this estimate is transformed within their objectives.
- Effective update regime: The analysis conditions on non-degenerate groups because all-correct or all-incorrect groups produce zero advantages and therefore no parameter updates.This isolates samples that actively drive learning without changing the optimization trajectory.
- Expectation-level bias: The estimator is unbiased only at pt = 0.5; it underestimates for pt < 0.5 and overestimates for pt > 0.5.The bias increases as pt moves farther from 0.5 and as group size G decreases.
- Probabilistic bias: Theorem 2 characterizes the probability of large underestimation on hard prompts and overestimation on easy prompts under finite group sizes.Corollary 1 further reports that these directional errors occur with high probability for practical group sizes.
- Extreme regimes: For extremely difficult prompts, the estimator necessarily underestimates, while for extremely easy prompts it necessarily overestimates.These deterministic extreme-regime behaviors are summarized in the paper’s discovery statement.
- Reward assumptions: The binary-reward analysis is extended to continuous bounded rewards, where the results suggest the bias persists beyond Bernoulli rewards.The authors state that this extension indicates the phenomenon is not an artifact of the binary assumption.
3 Proposed Solution
HA-DW combines a history-aware difficulty anchor with adaptive advantage reweighting to correct bias in group-relative estimation. The anchor tracks evolving model capability from cross-batch information, while reweighting adjusts prompt contributions according to difficulty.
- 3 Proposed Solution: HA-DW uses two collaborative phases: an evolving difficulty anchor followed by adaptive prompt-weight adjustment to compensate for biased advantage estimates.The anchor incorporates historical information, and the second phase reweights prompts according to estimated difficulty.
- 3.1 Evolving Difficulty Anchor: The cross-batch anchor updates the model’s latent solving-capability belief using current prompt accuracy and historical training information.The framework treats model capability as an evolving belief state and aggregates observations across batches.
- 3.1 Evolving Difficulty Anchor: The adaptive forgetting factor gives more weight to rapid capability changes early in training and preserves historical information during later stable stages.Its influence is dynamically modulated by model stability.
- 3.2 History Aware Adaptive Difficulty Weighting (HA-DW): HA-DW defines prompt difficulty relative to the evolving model belief, capturing both the direction and magnitude of each prompt’s deviation from current capability.The method uses a signed difficulty measure and an absolute difficulty magnitude before constructing the reweighting factor.
- 3.2 History Aware Adaptive Difficulty Weighting (HA-DW): The resulting exponential reweighting factor smoothly and multiplicatively adjusts advantage weights, while remaining compatible with group-relative RL variants.HA-DW is described as a plug-and-play module for GRPO and related algorithms.
4 Theoretical Analysis
The theoretical analysis characterizes bias in group-relative advantage estimation and studies rectification through baseline adjustment and HA-DW scaling. It shows that suitable reweighting can bring expected estimates closer to true advantages.
- 4 Theoretical Analysis: The analysis examines how reweighting the empirical group baseline affects expected estimation bias.It introduces baseline rectification before presenting the main theoretical result.
- 4 Theoretical Analysis: An appropriate rectification factor can reduce the estimation bias of the empirical group baseline under the stated bounded-probability condition.Lemma 1 assumes the prompt success probability remains within [∆, 1 −∆].
- 4 Theoretical Analysis: Table 1 compares base and HA-DW-equipped GRPO, GSPO, and DAPO across Qwen and LLaMA models and different model scales and families.The caption specifies the algorithms, model families, and accuracy comparisons reported.
- 4 Theoretical Analysis: Figure 4 tracks average accuracy across five benchmarks, training reward, and response length for Qwen3-4B-Base and Qwen3-8B-Base under different training strategies.The figure focuses on temporal training dynamics rather than only final performance.
- 4 Theoretical Analysis: Theorem 3 states that, with an appropriate scaling parameter λscale, HA-DW produces advantage estimates closer in expectation to the true advantage.The theorem provides guidance for selecting λscale in practice.
5 Experiments
Experiments across five RLVR benchmarks test HA-DW on multiple models and group-relative algorithms, showing consistent gains, stronger performance on hard prompts, and benefits from dynamic thresholding.
- Main Results: HA-DW improves GRPO, GSPO, and DAPO across five benchmarks, models of different scales, and model families.The experiments compare HA-DW-equipped algorithms with their original methods.
- Difficulty-Level Analysis: 3.4% higher performance on Hard prompts was achieved by GRPO+HA-DW than GRPO on Qwen3-4B-Base evaluated on MATH500.Performance on Easy and Mid levels was comparable between the two methods.
- Training Dynamics: HA-DW reached higher accuracy plateaus and training rewards than original RL algorithms across five benchmarks during training.The dynamics were measured for Qwen3-4B-Base and Qwen3-8B-Base.
- Ablation Study: Dynamic adjustment of Ct achieved the best performance across five benchmarks, while removing Ct degraded performance and fixing it retained partial gains.The ablation used Qwen3-4B-Base with GRPO-based training.
- Supplementary Experiments: The supplementary experiments include evaluations of advantage-estimation bias and ablations on rollout group size and the scaling parameter λscale.The listed experiments appear in Appendix E.
6 Related Work
Related work describes GRPO variants developed to improve stability, efficiency, sampling, or bias characteristics in group-based reinforcement learning.
- GRPO and GRPO Variants: GRPO variants include Dr.GRPO, DAPO, and GSPO, which target stable updates, dynamic sampling, decoupled clipping, or sequence-level optimization.Dr.GRPO removes heuristic normalizations, DAPO uses decoupled clipping and dynamic sampling, and GSPO uses sequence-level ratios and clipping.
7 Conclusion
The paper identifies biased advantage estimation as a limitation of group-relative RL and proposes HA-DW to mitigate it, with experiments reporting improved reasoning performance.
- Conclusion: HA-DW dynamically adjusts advantage weights based on the model’s evolving state to mitigate biased advantage estimation.The conclusion presents this as the paper’s proposed response to the identified limitation.
- Conclusion: Experiments demonstrate that HA-DW improves reasoning performance by mitigating biased advantage estimation.
Limitations
The paper’s scope is centered on group-relative methods and includes methodological descriptions of GRPO, GSPO, and DAPO, while broader extensions remain future work.
- Scope Boundary: HA-DW is restricted to group-relative methods because the study primarily focuses on group-wise estimation bias.The authors state that extending the concept to a broader scope is future work.
- GRPO: GRPO derives group-relative advantages without a value network and assigns the normalized group advantage uniformly across response tokens.
- GSPO: GSPO computes advantages from normalized relative group rewards and applies them at the sequence level without a critic model.
- DAPO: DAPO operates at the token level and combines decoupled clipping with dynamic sampling to stabilize training and adapt sample weighting.
C Setup Details
The experiments evaluate group-relative methods across three models and five mathematical reasoning benchmarks, while the analysis characterizes bias from truncated group sampling. The resulting estimator is biased differently depending on prompt difficulty.
- Models & Datasets: Experiments use Qwen3-4B-Base, Qwen3-8B-Base, and LLaMA-3.2-3B-Instruct, evaluating MATH500, AMC23, AIME25, Minerva, and OlympiadBench.Training uses 7.5k MATH questions.
- Baseline: HA-DW is evaluated on GRPO, GSPO, and DAPO to assess its effectiveness and scalability across group-relative algorithms.The comparisons apply HA-DW on top of the original algorithms.
- Advantage Estimation: Under binary rewards, truncation discards groups with all-correct or all-incorrect responses, retaining only groups with non-degenerate total reward.The retained groups exclude R=0 and R=G.
- Advantage Estimation: Conditioning on retained groups underestimates the empirical reward baseline when pt < 1/2 and overestimates it when pt > 1/2.The baseline bias transfers to the group-relative advantage estimator.
- Advantage Estimation: The resulting group-relative advantage is underestimated for pt < 0.5 and overestimated for pt > 0.5.This follows from the direction of baseline estimation error.
D.2 Proof of Theorem 2 and Corollary 1
The proof analyzes conditional sampling after excluding degenerate groups and derives how group size and prompt difficulty determine advantage-estimation bias. It also motivates an adjustment coefficient for correcting the empirical baseline and advantage.
- Conditional Analysis: Conditioning on the non-degenerate event S excludes R=0 and R=G, so only response counts R∈{1,...,G−1} remain.The proof expresses deviation probabilities using conditional probability over this restricted sample space.
- Bias by Difficulty: For hard prompts with pt∈(0,0.25) and 2≤G≤8, P(G,0,0.25)>0.78; by symmetry, easy prompts with pt∈(0.75,1) show the opposite baseline bias.The hard-prompt baseline is likely overestimated, while the easy-prompt baseline is underestimated.
- Large-Group Analysis: For sufficiently large groups, the proof uses the change of variable xt=Gpt and a Poisson approximation to analyze the limiting bias probability.The binomial distribution converges to a Poisson distribution with fixed xt as G→∞.
- Large-Group Analysis: Under a uniform distribution for pt, numerical evaluation gives P(0,2)≈0.7818 for sufficiently large G.The calculation evaluates the limiting integral associated with the hard-prompt interval.
- Bias Correction: An adjustment coefficient c is introduced so that, over a specified range, |˜pt−pt|<|ˆpt−pt|, reducing baseline estimation error.The analysis derives a feasible range for c and connects it to adjusted advantages.
- Bias Correction: When Equation (101) holds, HA-DW is efficient in compensating for biased advantage estimation.This conclusion is stated for the proposed adjustment during training.
D.5 Non-binary Reward Analysis
The analysis extends group-relative advantage bias beyond binary rewards to continuous bounded rewards, including truncated Gaussian distributions. Bias grows as expected reward moves farther from 0.5, and experiments support the predicted pattern.
- General Reward Distributions: The continuous-reward analysis shows that group-relative advantage estimation remains systematically biased under bounded reward distributions.The extension targets soft verifiers and learned reward models.
- General Reward Distributions: Theorem 4 formulates the conditional bias for G≥2 i.i.d. rewards drawn from a distribution with CDF F and PDF f.The result defines the update event and conditional quantities for continuous rewards.
- Truncated Gaussian Rewards: For truncated Gaussian rewards, the theoretical and numerical bias magnitude increases as pt deviates farther from 1/2.Reduced dispersion near extreme prompt difficulties makes retained groups atypically variable, shifting conditional expected advantages.
- Truncated Gaussian Rewards: For group sizes G=4 and G=8, Figure 5 shows |At,i−E[ˆAt,i|Sσ]| increasing as pt moves farther from 0.5.The figure corroborates the theoretical prediction for two representative group sizes.
- Empirical Rollout Analysis: In the rollout experiment, 24 MATH groups and 15 DAPO-Math-17k groups with one correct response at rollout=8 had fewer than 16 correct responses at rollout=128.The authors interpret this as evidence that correct-response advantages can be underestimated under limited rollouts.
- Empirical Rollout Analysis: Increasing rollout size can improve performance and stabilize the baseline, but computational constraints limit this strategy as training time grows and OOM can occur.HA-DW is presented as an alternative under limited rollout budgets.
E.3 Ablation Study on λscale
The ablation varies λscale within GRPO+HA-DW and examines how the adjustment affects training, alongside qualitative output examples. Values of 1.3 or 1.5 perform best across five benchmarks.
- λscale Ablation: λscale=1.3 or 1.5 produces the best performance across five benchmarks for Qwen3-4B-Base trained with GRPO+HA-DW.The result supports an optimal scaling value that balances adjustment across prompt difficulties.
- λscale Ablation: The ablation interprets the best λscale values as balancing adjustment across prompts with varying difficulties.This connects the scaling-factor sweep to HA-DW’s bias-compensation objective.
- Implementation Variant: A hard-update variant simplifies the evolving-belief update by synthesizing accuracy information from the preceding h batches with current observations.The simplification ignores short-term oscillations in belief updates.
- Qualitative Examples: Figures 7 and 8 provide output examples from policy models trained with GRPO and GRPO+HA-DW.The appendix presents these examples as qualitative training outputs.