PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometric-Mean Policy Optimization

Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, Fang Wan, Furu Wei

arXiv:2507.20673v3cs.CL

TL;DR

GRPO can produce unstable policy updates when outlier importance-weighted rewards create extreme importance sampling ratios. GMPO replaces GRPO’s arithmetic mean of token-level rewards with a geometric mean, and experiments report higher reasoning accuracy across mathematical and multimodal benchmarks.

  • Problem

    GRPO’s arithmetic-mean objective is sensitive to outlier importance-weighted rewards, leading to extreme importance sampling ratios and unstable policy updates.

  • Method

    GMPO stabilizes GRPO by maximizing the geometric mean of token-level rewards instead of the arithmetic mean.

  • Results

    GMPO-7B achieves 4.1% higher average Pass@1 accuracy than GRPO-7B across five mathematical reasoning benchmarks and 1.4% higher accuracy on Geometry3K.

  • Takeaways & Limitations

    GMPO provides a plug-and-play GRPO variant with improved stability, exploration, and reasoning capacity across language-only and multimodal benchmarks.

Abstract

from arXiv · show

Group Relative Policy Optimization (GRPO) has significantly enhanced the reasoning capability of large language models by optimizing the arithmetic mean of token-level rewards. Unfortunately, GRPO is observed to suffer from unstable policy updates when facing tokens with outlier importance-weighted rewards, which manifest as extreme importance sampling ratios during training. In this study, we propose Geometric-Mean Policy Optimization (GMPO), with the aim to improve the stability of GRPO through suppressing token reward outliers. Instead of optimizing the arithmetic mean, GMPO maximizes the geometric mean of token-level rewards, which is inherently less sensitive to outliers and maintains a more stable range of importance sampling ratio. GMPO is plug-and-play-simply replacing GRPO's arithmetic mean with the geometric mean of token-level rewards, as the latter is inherently less sensitive to outliers. GMPO is theoretically plausible-analysis reveals that both GMPO and GRPO are weighted forms of the policy gradient while the former enjoys more stable weights, which consequently benefits policy optimization and performance. Experiments on multiple mathematical reasoning benchmarks show that GMPO-7B improves the average Pass@1 of GRPO by up to 4.1%, outperforming many state-of-the-art approaches. Code is available at https://github.com/callsys/GMPO.

1 INTRODUCTION

GRPO improves reasoning but can become unstable when outlier importance-weighted rewards produce extreme policy ratios. GMPO replaces the arithmetic mean with a geometric mean to improve stability, exploration, and reasoning performance.

  • Motivation: GRPO’s arithmetic-mean objective is sensitive to outliers, causing expanding importance sampling ratios, unstable updates, and degraded performance.Clipping limits ratio deviations but can restrict exploration and lead to an early deterministic policy.
  • Stability and exploration: GMPO maintains stable importance sampling ratios with fewer extreme values while permitting a larger clipping range for greater exploration.The method is reported to balance stable policy optimization with expanded exploration.
  • Analysis: GMPO provides a narrower objective range, more balanced gradients, smaller KL divergence, and higher token entropy than GRPO during training.These analyses associate GMPO with reduced training variance, stability, and greater policy exploration.
  • Method: GMPO stabilizes GRPO by maximizing the geometric mean of token-level rewards.The geometric mean is less sensitive to outliers and yields lower-variance importance sampling ratio distributions.
  • Results: 4.1% higher average Pass@1 accuracy is achieved by GMPO-7B than GRPO-7B across five mathematical reasoning benchmarks.GMPO-7B also achieves 1.4% higher accuracy on Geometry3K.

2 BACKGROUND

RL methods for LLM post-training build on PPO and increasingly use relative rewards to improve efficiency. GRPO removes the need for a value model, but the supplied background identifies RL stability as an important underexplored issue.

  • Background: RL with verifiable rewards has become a key approach for post-training LLMs and improving reasoning.The approach builds on PPO and has motivated numerous optimization variants.
  • GRPO: GRPO estimates baselines from relative rewards within rollout groups, avoiding a computationally expensive value model while retaining strong benchmark results.For each question, GRPO samples multiple rollouts, computes their rewards, and optimizes the policy using relative advantages.
  • Open issue: RL stability for LLMs remains underexplored despite being essential for reliable and scalable post-training systems.Related methods address rollout selection, sampling, length bias, and gradient variance.
  • GRPO: GRPO’s objective is equivalent to the arithmetic mean of token-level rewards when the clipping term is ignored.The supplied passage also states that KL regularization is omitted for simplicity and memory savings.
  • GRPO: GRPO weights each rollout’s normalized advantage by the token-level importance sampling ratio to approximate updates from current-policy rollouts.The ratio compares current-policy and old-policy token probabilities.

3 GEOMETRIC-MEAN POLICY OPTIMIZATION

GMPO stabilizes GRPO by replacing the arithmetic mean of token-level rewards with a geometric mean that is less sensitive to outliers. Its gradient weighting and clipping designs aim to reduce extreme importance sampling ratios while preserving exploration.

  • Motivation: Extreme importance sampling ratios in GRPO can produce overly aggressive policy updates and instability.The instability is attributed to outlier importance-weighted rewards during training.
  • Geometric-mean objective: GMPO replaces GRPO’s arithmetic mean of token-level rewards with their geometric mean.The geometric mean is presented as inherently less sensitive to outlier values.
  • Stability: GMPO maintains a narrower importance sampling ratio range, indicating lower optimization-objective variance and more stable policy updates.The paper connects this stability to the geometric mean’s reduced sensitivity to outlier values.
  • Gradient interpretation: GMPO and GRPO are weighted sums of token policy gradients, but GMPO uses more balanced weights that are more robust to outliers.GRPO weights each token gradient by its individual importance sampling ratio, whereas GMPO aggregates ratios across tokens in the same sequence.
  • Clipping strategy: Token-level clipping is reported as more stable than sequence-level clipping because sequence-level clipping produces a larger importance sampling range.The comparison uses GMPO with clipping range (e−0.4, e0.4).

4 EXPERIMENT

Experiments compare GMPO with GRPO across language-only, multimodal, and Mixture-of-Experts settings, including objective and clipping ablations. GMPO consistently improves reasoning performance while maintaining more stable training dynamics and exploration.

  • Performance: GMPO improves performance across language-only, multimodal, and Mixture-of-Experts evaluations.It is evaluated on five mathematical reasoning benchmarks and Geometry3K, with additional MoE comparisons.
  • Performance: 1.4% higher Pass@1 accuracy is achieved on Geometry3K by GMPO than GRPO.The comparison uses Qwen2.5-VL-Instruct-7B for multimodal geometry reasoning.
  • Ablation Studies: Replacing the arithmetic mean with the geometric mean raises average performance from 51.2% to 52.7%, a 1.5% improvement.The comparison uses the same training settings for GRPO and GMPO.
  • Ablation Studies: Removing objective normalization lowers average performance from 52.7% to 52.0%, while removing clipping lowers it to 52.3%.These ablations indicate that both normalization and clipping contribute to the reported GMPO performance.
  • Training Analysis: GMPO maintains higher entropy, steadier gradients, lower KL divergence, and higher validation scores than GRPO during training.These patterns are reported across language-only and multimodal tasks and across dense and Mixture-of-Experts models.

5 CONCLUSION

The conclusion presents GMPO as a stabilized GRPO variant that improves exploration and reasoning capacity through geometric-mean rewards and a larger clipping range.

  • 5 CONCLUSION: GMPO stabilizes GRPO by optimizing the geometric mean of token-level rewards and enlarging the importance-sampling clipping range.The conclusion associates these changes with more stable policy updates and enhanced exploration.
  • 5 CONCLUSION: GMPO shows a narrower objective-value range, more stable gradients, lower KL divergence, and higher token entropy throughout training.These indicators are presented as evidence of improved training stability and exploration.
  • 5 CONCLUSION: GMPO outperforms GRPO on language-only and multimodal reasoning benchmarks in both stability and reasoning capacity.The conclusion frames the results as support for more reliable and scalable reinforcement-learning systems.

A GRADIENT DERIVATION

The gradient analysis derives how importance-sampling ratios and the GRPO and GMPO objectives vary with model parameters. It is designed to compare their robustness to extreme ratios.

  • A GRADIENT DERIVATION: The analysis derives the gradient of the importance-sampling ratio ρ_i,t(θ) with respect to model parameters θ.This result is stated as Lemma 1.
  • A GRADIENT DERIVATION: Subsequent lemmas derive the gradients of the GRPO and GMPO objectives with respect to θ.These are presented as Lemmas 2 and 3.
  • A GRADIENT DERIVATION: The derivation analyzes GMPO robustness to tokens with extreme importance-sampling ratios from a gradient perspective.The clipping-range term is omitted for clarity in the gradient derivation.

B PERFORMANCE ON MIXTURE-OF-EXPERTS MODELS

MoE experiments on DeepScaleR and CountDown examine GMPO’s stability against GRPO. GMPO maintains steadier training behavior and achieves stronger validation outcomes.

  • B PERFORMANCE ON MIXTURE-OF-EXPERTS MODELS: MoE experiments use the DeepScaleR and CountDown datasets to assess training stability.DeepScaleR contains approximately 40,000 mathematics problem-answer pairs, while CountDown contains arithmetic puzzles.
  • B PERFORMANCE ON MIXTURE-OF-EXPERTS MODELS: Figure 5 tracks KL divergence, entropy, gradient norm, and validation score over training steps for Mixture-of-Experts models.The figure reports GMPO’s lower KL divergence, higher entropy, steadier gradients, and higher validation scores relative to GRPO.
  • CountDown: On CountDown, GMPO maintains lower KL divergence and a steadier gradient norm than GRPO, while GRPO collapses after about 250 steps.GMPO consequently achieves stable validation scores.
  • DeepScaleR: On DeepScaleR, GMPO maintains higher entropy and a steadier gradient norm than GRPO, achieving higher validation scores.The comparison is shown in Figure 5.

C ANALYSIS OF THE NORMALIZATION FACTOR IN THE GEOMETRIC-MEAN

GMPO's power-based normalization is critical because, without it, sequence-level importance sampling ratios can become very large as response length increases, destabilizing policy optimization. The normalization supports more stable ratios and final performance.

  • Normalization factor: The additional power-based normalization term is described as critical to the GMPO objective.GMPO introduces this term in contrast to DeepSeek-R1.
  • Effect of normalization: Without normalization, sequence-level importance sampling ratios can become very large, especially as response length increases.The instability is illustrated for trajectories yielding positive rewards during GRPO training.
  • Effect of normalization: Unnormalized sequence-level importance sampling ratios lead to unstable policy optimization and degraded final performance.
Loading 2507.20673v3…