Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO

Zhiyuan Zeng; Jiameng Huang; Zhangyue Yin; Jiashuo Liu; Ziniu Li; Bingrui Li; Yuhao Wu; Yining Zheng; Ge Zhang; Wenhao Huang; Xipeng Qiu

Source-linked AI summary

Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO

Zhiyuan Zeng, Jiameng Huang, Zhangyue Yin, Jiashuo Liu, Ziniu Li, Bingrui Li, Yuhao Wu, Yining Zheng, Ge Zhang, Wenhao Huang, Xipeng Qiu

arXiv:2605.04077v1Apr 14, 2026cs.LGcs.AIcs.CL

TL;DR

Aggregation in GRPO-style RLVR is underexplored despite inducing different weighting biases across responses and tokens. The paper proposes Balanced Aggregation, which separates positive and negative token means before sequence-count weighting, and reports more robust training and strong final performance across models, datasets, and benchmarks.

Problem
Aggregation in GRPO-style RLVR remains underexplored, although token and sequence rules impose different weighting biases that affect optimization.
Method
Balanced Aggregation computes token-level means separately for positive and negative responses, then combines them using weights proportional to subset sequence counts.
Results
Across models, datasets, and benchmarks, BA delivers more robust training and strong final performance than standard aggregation rules.
Takeaways & Limitations
Aggregation is a first-class GRPO-style RLVR design choice, with balancing inter-sign weighting while retaining within-sign token information supporting more stable optimization.
Takeaways & Limitations
The empirical comparisons keep advantage normalization, PPO clipping, and sampling identical across methods, limiting conclusions to aggregation differences under this setup.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become a central paradigm for improving reasoning and code generation in large language models, and GRPO-style training is widely adopted for its simplicity and effectiveness. However, an important design choice remains underexplored: how token-level policy gradient terms are aggregated within each sampled group. Standard GRPO uses sequence aggregation, while recent work has advocated token aggregation as a better alternative. We show that these two rules induce different optimization biases: token aggregation introduces sign-length coupling, while sequence aggregation implicitly downweights longer responses through sequence-level equal weighting. To address this tension, we propose \textbf{Balanced Aggregation (BA)}, a simple drop-in replacement that computes token-level means separately within the positive and negative subsets and then combines them with sequence-count-based weights. Experiments with Qwen2.5-Math-7B and Qwen3-1.7B on DAPO-17k and Polaris, evaluated on six reasoning and coding benchmarks, show that BA consistently improves training stability and final performance over standard token and sequence aggregation. Our analysis further shows that the relative effectiveness of token and sequence aggregation is largely governed by response-length variation and the positive-negative length gap, highlighting aggregation as a critical design dimension in GRPO-style RLVR.

1 Introduction

The paper identifies distinct aggregation biases in GRPO-style RLVR and proposes Balanced Aggregation (BA) to combine sign-balanced updates with token-level averaging. Across models, datasets, and benchmarks, BA improves training stability and final performance over standard token and sequence aggregation.

1 Introduction: GRPO-style RLVR is widely adopted because it uses verifiable rewards and group-wise advantages without requiring a separate critic.It samples multiple responses per prompt and optimizes a PPO-style objective using relative learning signals within each group.
1 Introduction: Token aggregation can benefit settings with large response-length variance but is more sensitive to positive-negative length imbalance and often produces less stable optimization.This motivates preserving sequence aggregation’s sign balance while retaining token-level averaging.
1 Introduction: Balanced Aggregation (BA) separately computes token-level means for positive and negative responses, then combines them using weights proportional to subset sequence counts.This removes token aggregation’s positive-negative length coupling while avoiding sequence aggregation’s strong sequence-level equal-weighting effect.
1 Introduction: BA consistently improves final performance and training stability over standard token and sequence aggregation across Qwen2.5-Math-7B and Qwen3-1.7B, DAPO and Polaris, and six evaluation benchmarks.The benchmarks include Math-500, AIME 2024, AIME 2025, OlympicBench, Minerva-MATH, and LiveCodeBench.
1 Introduction: Aggregation is an underexplored GRPO design choice: token aggregation couples update contributions to sign and response length, whereas sequence aggregation equal-weights responses and downweights longer ones.Token aggregation can amplify positive or negative updates when their length distributions differ; sequence aggregation assigns each response equal weight regardless of token count.

2 Related Work

RLVR has emerged as an attractive paradigm for post-training on tasks with programmatically verifiable outcomes, while GRPO provides a practical critic-free optimization backbone. Related work studies training stability, clipping, importance sampling, advantage normalization, and the underexplored aggregation of token-level policy-gradient terms.

RLVR and GRPO-Style Post-Training: RLVR is attractive for verifiable tasks because outcomes can be checked programmatically, avoiding learned reward modeling.
RLVR and GRPO-Style Post-Training: PPO has been a standard optimization backbone, while GRPO reduces training cost by replacing the critic with group-relative reward normalization.GRPO’s critic-free formulation supports large-scale RLVR training.
RLVR Training Tricks: RLVR training methods target train-infer mismatch, training stability, clipping, trust-region design, importance sampling, advantage estimation, and normalization.Examples include asymmetric and soft clipping, GSPO, and ASPO.
Aggregation in GRPO-Style RL: Aggregation of token-level policy-gradient terms has received less systematic attention than other GRPO-style design choices.Standard GRPO averages token contributions within each response before averaging across responses.
Aggregation in GRPO-Style RL: DAPO and Dr.GRPO identified limitations of sequence aggregation in long-form reasoning and advocated token-level alternatives.GMPO is also described as improving optimization stability, though the supplied passage is truncated.

3 Method

The method frames GRPO-style RLVR as token-level PPO optimization with sequence-level advantages, making the group aggregation rule determine response and token weighting. It introduces Balanced Aggregation (BA), which preserves positive-negative balance while avoiding the competing length biases of token and sequence aggregation.

GRPO objective: GRPO assigns each response a sequence-level advantage shared by all generated tokens, then aggregates token-level PPO contributions across the sampled group.The aggregation rule therefore determines how response length affects the relative weight of samples.
Aggregation biases: Token aggregation introduces sign-length coupling because positive and negative contributions are weighted by their average response lengths.When the positive and negative subsets have different average lengths, token aggregation changes the effective balance of policy gradients.
Aggregation biases: Sequence aggregation removes sign-length coupling but assigns equal weight to every response regardless of token count, creating sequence equal-weighting bias.Thus, neither standard aggregation rule fully resolves the tension between sign balance and length-sensitive weighting.
Balanced Aggregation: Balanced Aggregation (BA) averages token-level losses separately within positive and negative subsets, then combines them with sequence-count-based weights.This retains token-level averaging within each sign group while restoring sequence aggregation’s positive-negative balancing principle.
Balanced Aggregation: BA shares sequence aggregation’s inter-sign balancing but averages over tokens within each sign group rather than assigning equal weight to each response.It therefore removes token aggregation’s sign-length coupling while avoiding sequence aggregation’s strong per-sequence equal-weighting effect.

4 Experiments

Experiments across two RLVR datasets and two Qwen models show that aggregation performance depends on the model and corpus, while balanced aggregation provides the most reliable overall behavior. It improves training stability by preserving strong peak performance and reducing degradation at the final checkpoint.

Experimental setup: Balanced-agg is compared with token-agg and seq-agg on DAPO-17k and Polaris using Qwen2.5-Math-7B and Qwen3-1.7B.All methods share the same advantage normalization, PPO clipping, and sampling components.
Overall results: Balanced-agg consistently ranks as the best or highly competitive method across the evaluated models and datasets.On DAPO-17k, it establishes the highest peak metrics for Qwen2.5-Math-7B and surpasses both baselines in the reported comparisons.
Overall results: Across both datasets, token-agg is stronger for Qwen2.5-Math-7B, whereas seq-agg is more stable and accurate for Qwen3-1.7B.This model-dependent reversal shows that neither standard aggregation rule provides a consistently reliable optimization signal.
Peak versus last-step performance: At peak performance, token-agg and balanced-agg are very close and usually outperform seq-agg, but token-agg shows the largest peak-to-last drop on nearly all benchmarks.Balanced-agg preserves its gains better at the final checkpoint and achieves the best reported final behavior.
Why aggregation preferences flip: Qwen2.5-Math-7B favors token-agg because its larger response-length variation makes seq-agg’s equal-weighting effect more harmful than token-agg’s sign-length coupling.Qwen3-1.7B exhibits the opposite pattern: smaller length variation reduces seq-agg’s weakness, while its larger positive–negative length gap worsens token-agg’s coupling bias.

5 Conclusion

The paper shows that GRPO aggregation rules impose distinct optimization trade-offs: token aggregation couples sign with length, while sequence aggregation uses strong per-sequence equal weighting. Balanced Aggregation preserves sign balance with token-level averaging within sign groups and delivers robust training with strong final performance across settings.

Token aggregation introduces sign-length coupling, whereas sequence aggregation avoids it but strongly weights each sequence equally.These are the central optimization trade-offs identified for GRPO-style RLVR aggregation.
Balanced Aggregation preserves sign balance while retaining token-level averaging within each sign group.BA is presented as a simple alternative to token and sequence aggregation.
Across models, datasets, and benchmarks, BA delivers more robust training and strong final performance.

Appendix A: Why Use Sequence-Count Weights in BA?

BA uses sequence-count weights k/G and (G − k)/G because, under binary-reward GRPO, they preserve sequence aggregation’s positive-negative balancing while retaining token-level averaging within each sign group.

Weight choice: BA’s weights are k/G for positive sequences and (G − k)/G for negative sequences.The appendix introduces these weights under the binary-reward GRPO setting.
Weight choice: Substituting these weights into BA yields the same inter-sign prefactor as sequence aggregation.This equivalence follows directly from the BA objective after using the normalized advantages.
Weight choice: Thus, sequence-count weights preserve sequence aggregation’s positive-negative balancing while enabling token-level averaging within each sign group.The weights are therefore principled rather than arbitrary.

Appendix B: Extension to Non-Binary Rewards

The appendix extends Balanced Aggregation to non-binary rewards, where advantage magnitudes make sequence counts insufficient for balancing positive and negative subsets. The generalized objective normalizes each sign subset separately by advantage mass, preserves policy-gradient weighting, removes cross-sign sign-length coupling, and reduces exactly to the original BA under binary rewards.

Motivation: For real-valued rewards, advantage magnitudes matter alongside response counts and lengths, so sequence counts alone cannot characterize positive and negative subset contributions.The token-level contribution is ϕ_i,t(θ) = ˆA_iδ_i,t(θ), making subset contributions depend on advantage magnitudes.
Construction: Generalized BA defines positive and negative subsets using the signs of normalized advantages and constructs sign-wise advantage masses and advantage-weighted token masses.The subsets are S+ = {i | ˆA_i > 0} and S− = {i | ˆA_i < 0}.
Generalized objective: The generalized objective preserves the original ˆA_i policy-gradient weighting while normalizing positive and negative samples separately and recombining them in a sign-balanced manner.Unlike the binary formulation, sign balance is determined by advantage mass rather than sequence count.
Property: Under the usual group-normalization condition, generalized BA keeps positive and negative subsets symmetric across signs and removes token aggregation’s cross-sign sign-length coupling.This preserves the core inter-sign balancing property of BA in the non-binary setting.
Reduction to binary rewards: Under binary rewards, generalized BA recovers exactly the original BA objective, making the formulation a strict extension rather than a different objective.Binary rewards make ˆA_i constant within each sign subset, represented by constants a+ > 0 and a− > 0.