PaperlayerView exact revision on arXiv

Source-linked AI summary

GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models

Peizheng Guo, Jianqi Zhang, Xingyu Zhang, Yun Fan, Jiahuan Zhou, Changwen Zheng, Wenwen Qiang

arXiv:2608.17411v1cs.LG

TL;DR

GRPO averages query-group gradients that may conflict, without accounting for differences in their reliability. GUPO models gradients probabilistically and uses Dirichlet-derived uncertainty to calibrate aggregation; experiments across multiple benchmarks report its effectiveness.

  • Problem

    GRPO directly averages potentially conflicting query-group gradients without accounting for differences in their reliability, leaving reliable aggregation under conflicts unresolved.

    Source: Motivation Analysis · p. 4

  • Method

    GUPO models each group gradient as a Bayesian probability distribution, derives uncertainty through a Dirichlet-based evidential formulation, and uses it to calibrate aggregation.

    Source: Introduction · p. 2

  • Results

    Across multiple benchmarks and base models, GUPO demonstrates effectiveness relative to representative baselines and improves validation performance over GRPO across sampled mini-batches.

    Source: Performance Comparison · p. 7, Analysis · p. 7

  • Takeaways & Limitations

    Conflict-focused analyses suggest GUPO may mitigate degradation in update effectiveness associated with severe group-gradient conflicts.

    Source: Analysis · p. 7

Abstract

from arXiv · show

Group Relative Policy Optimization (GRPO) has become a widely used approach for post-training Large Language Models (LLMs) for reasoning. In GRPO, the group gradients induced by different queries within the same mini-batch are directly averaged to form the policy update. However, these group gradients can point in conflicting directions. Our empirical analysis suggests that group-gradient conflicts tend to be associated with less effective policy updates, motivating the need for a reliable aggregated update direction under such conflicts. Standard GRPO aggregation treats the realized group gradients as deterministic contributions and does not account for differences in their reliability during aggregation. To address this issue, we propose Gradient Uncertainty-Aware Policy Optimization (GUPO), which models each group gradient as a random variable under a Bayesian formulation and estimates its probability distribution. GUPO then derives gradient uncertainty using a Dirichlet-based formulation and uses it to calibrate the contribution of each group gradient during aggregation. Extensive experiments on multiple benchmarks demonstrate the effectiveness of GUPO.

Introduction

GUPO addresses conflicting group gradients in GRPO by estimating their uncertainty and using it to weight their contributions during aggregation. The paper reports that this approach improves GRPO-based post-training across multiple benchmarks.

  • A concentrated gradient distribution indicates lower uncertainty and higher reliability, whereas a dispersed distribution indicates higher uncertainty and lower reliability.Direct aggregation gives group gradients equal coefficients, so a more uncertain gradient may exert excessive influence on the aggregated direction.Source: Introduction · pp. 1–2
  • GUPO models each group gradient as a Bayesian probability distribution and uses a Dirichlet-based evidential formulation to estimate uncertainty for aggregation.It maps distribution precision to gradient evidence, derives dimension-wise beliefs and group-level uncertainty, then increases the relative contribution of lower-uncertainty gradients while attenuating higher-uncertainty ones.Source: Introduction · p. 2
  • Experiments across multiple benchmarks demonstrate GUPO’s effectiveness in improving GRPO-based post-training.The paper attributes the improvement to incorporating group-gradient uncertainty into policy optimization.Source: Introduction · p. 2
  • Group-gradient conflicts within a GRPO mini-batch can be associated with less effective policy updates, motivating a reliable aggregated direction.The paper identifies directional conflicts among gradients induced by different queries in the same mini-batch.Source: Introduction · p. 2

Related Work

Prior work studies GRPO gradient conflicts at several optimization levels and uses uncertainty to adjust rewards or advantages. GUPO instead models query-level group-gradient uncertainty to calibrate mini-batch aggregation.

Gradient Conflict in LLM Post-Training

  • DaGRPO, PCR, and ResRL address conflicts within rollout groups, between plasticity and stability gradients, and between positive and negative responses; GUPO studies conflicts among query-level group gradients.Prior methods use sequence-level correction, probabilistic projection, and projection residuals, respectively.Source: Gradient Conflict in LLM Post-Training · pp. 2–3

Uncertainty Estimation in LLM Post-Training

  • Uncertainty-aware reward and advantage methods assess predictions, sampled-response diversity, or advantage confidence; GUPO models each query-level group gradient as a Bayesian random variable and uses its uncertainty in aggregation.The aim is to reflect the reliability of each update signal and mitigate adverse effects of gradient conflicts.Source: Uncertainty Estimation in LLM… · p. 3

Problem Settings and Analyses

The section shows that conflicting query-group gradients can accompany less effective GRPO updates, motivating aggregation that accounts for differences in gradient reliability.

Problem Settings

  • GRPO samples candidate outputs per query, scores them with outcome rewards, and uses within-group relative advantages to encourage higher-reward outputs.For verifiable reasoning tasks, rewards are usually based on final-answer correctness.Source: Problem Settings · p. 3

Empirical Analysis

  • Using the same starting checkpoint, the analysis defines conflict by negative pairwise cosine similarity and evaluates updates with validation ΔNLL, where larger values indicate more effective updates.Conflict level is measured as the fraction of group-gradient pairs with negative cosine similarity.Source: Empirical Analysis · p. 4
  • Among 563 group-gradient pairs, 262 (46.5%) had negative cosine similarity; high-conflict batches showed lower median validation ΔNLL and negative updates, suggesting conflicts can accompany less effective updates.Figures 2a–d show example batches, pairwise similarities, and validation-update results.Source: Empirical Analysis · p. 4

Motivation Analysis

  • Under conflicts, equal-weight averaging can let uncertain gradients disproportionately steer updates; treating gradients as distributions motivates emphasizing more reliable signals.Lower uncertainty corresponds to a more concentrated distribution and higher reliability; this describes consistency, not whether a gradient direction is correct.Source: Motivation Analysis · p. 4

Methodology

GUPO models query-level group gradients as uncertain quantities and uses their uncertainty to calibrate GRPO’s mini-batch gradient aggregation. Its method estimates parameter and gradient distributions, derives Dirichlet-based uncertainty, and gives lower-uncertainty gradients greater influence.

Methodology

  • GUPO estimates group-gradient uncertainty and uses it to calibrate each gradient’s contribution to the GRPO mini-batch update.The framework estimates a Bayesian distribution over policy parameters, derives group-gradient distributions, then uses their uncertainty during aggregation.Source: Methodology · pp. 4–5

Bayesian-based Group Gradient Distribution

Posterior Distribution Estimation of Policy Parameters

  • To avoid an intractable full-model posterior and Hessian, GUPO models the final trainable layer and approximates its posterior with a Gaussian using diagonal empirical Fisher information.The Gaussian approximation is centered at the current parameters; the diagonal empirical Fisher approximation is used for the negative Hessian.Source: Posterior Distribution Estimation of… · p. 5

Group Gradient Distribution Estimation

  • GUPO estimates each group gradient’s distribution from sampled parameter values and uses element-wise variance rather than a full covariance matrix.The full covariance is avoided because the gradient space is high-dimensional.Source: Group Gradient Distribution Estimation · p. 6

Gradient Uncertainty-aware Policy Optimization

  • GUPO maps gradient precision to evidence, derives dimension-wise belief and group-level uncertainty through a Dirichlet formulation, then weights lower-uncertainty gradients more heavily.Its aggregation combines uncertainty-aware weights with GRPO’s original coefficient, with η controlling how strongly uncertainty adjusts the aggregation.Source: Gradient Uncertainty-aware Policy… · p. 7

Experiments

Experiments evaluate GUPO across reasoning benchmarks and base models, and the reported comparisons demonstrate its effectiveness when group-gradient uncertainty is incorporated into gradient aggregation.

Benchmarks and Baselines

Implementation Details

  • Experiments use DeepScaleR-1.5B-Preview, DeepSeek-R1-Distill-Qwen-1.5B, and DeepSeek-R1-Distill-Qwen-7B, with a learning rate of 1×10−6, weight decay 0.01, and global batch size 256.All experiments are conducted on an NVIDIA H100 GPU cluster.Source: Implementation Details · p. 7

Performance Comparison

  • GUPO’s comparisons with representative baselines across multiple benchmarks and base models demonstrate its effectiveness in incorporating group-gradient uncertainty into gradient aggregation.The supplied passage refers to Table 1 but does not provide its values, so no specific ranking or quantitative result can be reported.Source: Performance Comparison · p. 7

Ablation Study about Parameter Sensitivity

  • The best reported sensitivity settings are s = 0.5 and η = 0.1, selected from the tested ranges for evidence sensitivity s and aggregation coefficient η.Figure 6 is captioned as parameter sensitivity; the passage provides the settings and tested values.Source: Ablation Study about Parameter… · p. 7, Figure 6 · p. 7

Analysis

  • Across sampled mini-batches, GUPO improves validation performance over GRPO, with paired comparisons on high-conflict mini-batches supporting mitigation of conflict-related update degradation.The passage attributes less effective updates to severe group-gradient conflicts.Source: Analysis · p. 7

Conclusion

The conclusion identifies conflicting query-level group gradients as a potential source of less effective GRPO updates and presents GUPO as an uncertainty-aware aggregation method. Experiments across multiple reasoning benchmarks demonstrate GUPO’s effectiveness.

Conclusion

  • Conflicts among query-level group gradients within a GRPO mini-batch can be associated with less effective policy updates.Experiments across multiple reasoning benchmarks demonstrate GUPO’s effectiveness.Source: Conclusion · pp. 7–8
  • GUPO models each group gradient as a probability distribution, estimates its uncertainty, and uses that uncertainty to strengthen lower-uncertainty gradients and mitigate highly uncertain ones during aggregation.This is intended to make the aggregated gradient better reflect the optimization tendency supported by the mini-batch.Source: Conclusion · p. 8

Related papers

Loading 2608.17411v1…