Source-linked AI summary
GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
Peizheng Guo, Jianqi Zhang, Xingyu Zhang, Yun Fan, Jiahuan Zhou, Changwen Zheng, Wenwen Qiang
TL;DR
Conflicting group gradients in GRPO can make the averaged policy update less effective, creating a need for reliable aggregation. GUPO models gradient uncertainty with a Bayesian, Dirichlet-based formulation to calibrate group contributions, and experiments across multiple benchmarks suggest its effectiveness.
Problem
Conflicting group gradients within GRPO mini-batches motivate the question of how to obtain a reliable aggregated update direction.
Method
GUPO models each group gradient as a Bayesian probability distribution, derives uncertainty through a Dirichlet-based formulation, and uses it to calibrate gradient aggregation.
Results
Experiments across multiple benchmarks demonstrate GUPO's effectiveness.
Takeaways & Limitations
GUPO provides an uncertainty-aware approach for obtaining a more reliable update direction when GRPO group gradients conflict.
Abstract
from arXiv · showhide
Group Relative Policy Optimization (GRPO) has become a widely used approach for post-training Large Language Models (LLMs) for reasoning. In GRPO, the group gradients induced by different queries within the same mini-batch are directly averaged to form the policy update. However, these group gradients can point in conflicting directions. Our empirical analysis suggests that group-gradient conflicts tend to be associated with less effective policy updates, motivating the need for a reliable aggregated update direction under such conflicts. Standard GRPO aggregation treats the realized group gradients as deterministic contributions and does not account for differences in their reliability during aggregation. To address this issue, we propose Gradient Uncertainty-Aware Policy Optimization (GUPO), which models each group gradient as a random variable under a Bayesian formulation and estimates its probability distribution. GUPO then derives gradient uncertainty using a Dirichlet-based formulation and uses it to calibrate the contribution of each group gradient during aggregation. Extensive experiments on multiple benchmarks demonstrate the effectiveness of GUPO.
Introduction
The introduction identifies conflicting group gradients as a limitation of GRPO aggregation because severe conflicts tend to accompany limited validation gains. It motivates GUPO, which models group gradients probabilistically and uses uncertainty to obtain more reliable updates during conflicts.
- GRPO and gradient conflicts: GRPO averages query-level group gradients across a mini-batch to form the policy update.Each group gradient represents the contribution of one query group.
- GRPO and gradient conflicts: Group gradients can point in conflicting directions, and mini-batches with severe conflicts tend to exhibit limited validation update gains.The introduction reports both low-conflict alignment and clear directional conflicts across mini-batches.
- Gradient uncertainty: Gradient uncertainty measures reliability: concentrated distributions indicate lower uncertainty, whereas dispersed distributions indicate higher uncertainty.Lower uncertainty corresponds to higher reliability, while higher uncertainty corresponds to lower reliability.
- GUPO: GUPO models each group gradient as a random variable represented by a probability distribution under a Bayesian formulation.This replaces treating each group gradient solely as a deterministic vector.
- GUPO: GUPO aims to reduce the influence of highly uncertain gradients and obtain a more reliable aggregated update direction when group-gradient conflicts occur.Experiments across multiple benchmarks suggest the effectiveness of the method.
Related Work
Related work identifies gradient conflicts in GRPO-based LLM post-training and explores uncertainty estimation for evaluating model outputs and alignment signals. GUPO addresses these themes by characterizing query-level group-gradient uncertainty and calibrating aggregation contributions.
- Gradient conflicts: GRPO gradients can point in incompatible directions, interfering with effective policy updates and motivating studies of gradient conflicts in GRPO-based post-training.DaGRPO is identified as one recent study investigating such conflicts.
- Uncertainty-aware aggregation: GUPO explicitly characterizes uncertainty in query-level group gradients and calibrates their contributions during mini-batch gradient aggregation.This framework connects uncertainty modeling with gradient aggregation in the optimization process.
- Uncertainty estimation: Uncertainty estimation quantifies the reliability of model outputs or evaluation signals and is widely applied to LLMs.Uncertainty-aware Reward Model models stochastic human preferences and disagreement among reward models to quantify reward-model prediction uncertainty.
Problem Settings and Analyses
The section formulates GRPO’s group-gradient aggregation and analyzes conflicts among query groups. It finds that severe conflicts are associated with less effective updates, motivating uncertainty-aware aggregation to obtain more reliable directions.
- GRPO setting: For each query, GRPO samples G candidate outputs, evaluates outcome rewards, and uses within-group relative advantages to favor higher-reward outputs.In verifiable reasoning tasks, the outcome reward typically depends on final-answer correctness.
- Group-gradient aggregation: A mini-batch gradient decomposes into group gradients from its B query groups, which GRPO aggregates by averaging.The group gradient aggregates token-level gradients from the G responses associated with each query.
- Conflict analysis: Cosine similarities below zero identify directional conflicts, and conflict rate measures the proportion of group-gradient pairs exhibiting such conflicts.Controlled experiments compare mini-batches sampled from the same checkpoint under identical settings.
- Conflict analysis: Higher conflict rates correspond to lower-median validation ∆NLL and negative updates, indicating that severe gradient conflicts can produce less effective policy updates.Here, larger ∆NLL denotes a more effective update because it equals NLLbefore − NLLafter.
- Motivation: Because directly averaging conflicting gradients may be unreliable, the section motivates weighting group gradients by uncertainty to emphasize reliable signals and limit disproportionate conflicting influence.A group gradient is modeled as a random variable with a probability distribution, whose concentration around the mean reflects reliability.
Methodology
GUPO models group gradients as uncertain quantities by placing a Bayesian distribution over trainable policy parameters and estimating each induced gradient distribution. It converts gradient precision into Dirichlet-based uncertainty and uses that uncertainty to weight group-gradient aggregation, favoring more reliable update contributions.
- Bayesian gradient uncertainty: GUPO models uncertainty over group gradients by placing a Bayesian distribution on the final trainable policy layer and propagating it to each query-level gradient.The full-parameter posterior is avoided because it is computationally prohibitive.
- Bayesian gradient uncertainty: A Gaussian posterior approximation is constructed around current parameters, with the infeasible Hessian replaced by a diagonal empirical Fisher approximation plus δI.The approximation is q(Φ | D) = N(Φ | Φ0, H−1), while H ≈eH = Diag(FΦ) + δI, δ > 0.
- Gradient distribution estimation: Monte Carlo sampling estimates each group-gradient distribution, using element-wise variance vectors instead of full covariance matrices in the high-dimensional gradient space.Parameter samples are drawn from q(Φ|D), and each sampled parameter set produces a corresponding group gradient.
- Dirichlet uncertainty modeling: GUPO transforms gradient precision into evidence, defines Dirichlet concentration parameters, and derives dimension-wise belief together with group-level uncertainty.Precision is λb,d = 1/σ2b,d, evidence is eb,d = ϕ(λb,d), and the concentration parameter is αb,d = eb,d + 1.
- Uncertainty-aware aggregation: The final aggregation assigns greater contributions to lower-uncertainty group gradients and smaller contributions to higher-uncertainty gradients while retaining GRPO’s original aggregation structure.The interpolation parameter η ∈ [0, 1] controls how strongly uncertainty modifies the original coefficient 1/B.
Experiments
Experiments evaluate GUPO against representative post-training baselines across reasoning benchmarks, base models, and controlled settings. Results show effectiveness from uncertainty-aware gradient aggregation, including improved performance under severe group-gradient conflicts.
- Experimental comparison: GUPO is compared with GRPO, Length Penalty, ReST-MCTS, GVPO, Dr. GRPO, GCPO, and MRT across multiple reasoning benchmarks.The benchmarks include AIME 2024, AIME 2025, AMC 2023, MATH500, MinervaMATH, and GSM8K, using Pass@1 accuracy and average performance.
- Experimental setup: Experiments use DeepScaleR-1.5B-Preview, DeepSeek-R1-Distill-Qwen-1.5B, and DeepSeek-R1-Distill-Qwen-7B.Training uses a learning rate of 1×10−6, weight decay of 0.01, and global batch size of 256 on an NVIDIA H100 GPU cluster.
- Main results: Across multiple benchmarks and base models, Table 1 results demonstrate GUPO’s effectiveness by incorporating group-gradient uncertainty into gradient aggregation.The comparison is conducted against representative baselines.
- Parameter sensitivity: Sensitivity analysis finds best performance at evidence sensitivity s = 0.5 and aggregation coefficient η = 0.1.Evidence sensitivity is evaluated over [0.01, 0.05, 0.1, 0.3, 0.5, 0.7, 0.9], while η is evaluated over [0, 0.5] at intervals of 0.1.
- Conflict analysis: Under group-gradient conflicts, GUPO improves validation performance over GRPO across sampled mini-batches, including high-conflict mini-batches.The results suggest GUPO mitigates degradation in update effectiveness associated with severe group-gradient conflicts.
Conclusion
The paper identifies conflicting group gradients in GRPO mini-batches as associated with less effective policy updates and proposes GUPO to obtain more reliable aggregation directions. GUPO models gradient uncertainty Bayesianly and uses it to emphasize reliable gradients while reducing uncertain ones.
- Group-gradient conflicts among queries in the same GRPO mini-batch can be associated with less effective policy updates.
- GUPO models each group gradient as a probability distribution under a Bayesian formulation, estimates uncertainty, and incorporates it into gradient aggregation.
- GUPO enhances lower-uncertainty gradients and mitigates highly uncertain ones, helping the aggregate better reflect the mini-batch’s supported optimization tendency.Experiments across multiple reasoning benchmarks demonstrate GUPO’s effectiveness.