Source-linked AI summary
GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Yejin Choi, Jan Kautz, Pavlo Molchanov
TL;DR
Multi-reward RL often applies GRPO without establishing that its normalization preserves distinctions among reward combinations. The paper introduces GDPO, which decouples reward normalization and adds batch-wise normalization, and reports consistent cross-task gains over GRPO, while noting that reward weighting can remain unreliable when objective difficulty differs substantially.
Problem
Multi-reward RL commonly applies GRPO, but directly normalizing summed rewards can collapse distinct reward combinations into identical advantages and reduce training-signal resolution.
Method
GDPO normalizes each reward separately to preserve cross-reward differences, then applies batch-wise advantage normalization to maintain a stable numerical range.
Results
Across tool calling, math reasoning, and code reasoning, GDPO consistently outperforms GRPO across different reward counts, models, and reward functions.
Takeaways & Limitations
GDPO provides a more stable, accurate, and preference-aligned optimization method for multi-reward reinforcement learning.
Takeaways & Limitations
When objectives differ substantially in difficulty, reward weighting may still favor an easier objective despite assigning greater weight to a harder one.
Abstract
from arXiv · showhide
As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct preference, to guide models toward these desired behaviors. However, recent work has defaulted to apply Group Relative Policy Optimization (GRPO) under multi-reward setting without examining its suitability. In this paper, we demonstrate that directly applying GRPO to normalize distinct rollout reward combinations causes them to collapse into identical advantage values, reducing the resolution of the training signal and resulting in suboptimal convergence and, in some cases, early training failure. We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues by decoupling the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization, along with substantially improved training stability. We compare GDPO with GRPO across three tasks: tool calling, math reasoning, and coding reasoning, evaluating both correctness metrics (accuracy, bug ratio) and constraint adherence metrics (format, length). Across all settings, GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
NVIDIA
The paper lists its implementations and project links, and identifies NVIDIA-affiliated contributors.
- Implementations are provided for HF-TRL, verl, and Nemo-RL.
- The paper provides links to the project and lab.
- The author list notes a project lead and an HKUST affiliation, with internship work conducted at NVIDIA Research.
1. Introduction
The paper examines whether GRPO is suitable for multi-reward reinforcement learning and identifies reward-signal collapse as a central problem. It proposes GDPO and reports consistent improvements across tasks, objectives, and models.
- Language models must satisfy diverse preferences, including efficiency, safety, coherence, logic, and bias-related objectives.
- Multi-reward RL commonly relies on GRPO without fully examining whether it preserves distinctions among reward dimensions.
- Naively applying GRPO can collapse distinct reward combinations into identical advantages, reducing learning-signal resolution and causing inaccurate updates, weak convergence, or early failure.
- GDPO separately normalizes each reward, preserves cross-reward distinctions, and applies batch-wise advantage normalization to keep magnitudes stable as rewards are added.
- Across tool calling, math reasoning, and code reasoning, GDPO consistently outperforms GRPO on accuracy, formatting, length constraints, and code-related objectives.
2. GRPO’s propensity for reward signal collapse in multi-reward RL
The paper argues that standard multi-reward GRPO compresses reward information when summed rewards are normalized together. Its examples show that GDPO preserves more advantage distinctions, while alternative normalization changes provide only limited relief.
- GRPO and its variants are efficient policy-optimization methods that estimate group-relative advantages without a value model.
- Existing multi-reward approaches generally sum reward components before applying GRPO.
- In a two-binary-reward example, six order-agnostic reward combinations collapse into only two normalized advantage groups under GRPO.
- GRPO assigns the same advantages to (0, 1) and (0, 2), although the latter represents simultaneous satisfaction of two rewards and should provide a stronger signal.
- The collapse can produce training instability; in one math-reasoning setting, GRPO correctness began declining after approximately 400 steps.
- Removing standard-deviation normalization distinguishes some combinations, but yields only modest increases in distinct advantage groups as rollout or reward counts grow.
3. Method
GDPO decouples reward normalization before aggregation to preserve distinctions among multi-reward outcomes, then applies batch-wise normalization to stabilize the final advantage scale. The method also supports reward weighting and conditioning for prioritized objectives.
- Group reward-Decoupled normalization Policy Optimization: GDPO normalizes each reward separately before aggregating normalized advantages, unlike GRPO, which normalizes the aggregated reward sum.This design better preserves relative differences across reward combinations.
- Group reward-Decoupled normalization Policy Optimization: Batch-wise normalization keeps the final multi-reward advantage scale stable as additional rewards are introduced.Removing this step occasionally leads to convergence failures.
- Group reward-Decoupled normalization Policy Optimization: Separately normalized rewards preserve meaningful differences that GRPO can collapse into identical advantages across heterogeneous reward partitions.For example, GDPO yields non-zero advantages for reward dimensions that can be obscured by GRPO’s aggregation.
- Group reward-Decoupled normalization Policy Optimization: GDPO produces substantially more distinct advantage groups than GRPO and GRPO w/o std as rollout or reward counts increase.The increased granularity provides more expressive advantage estimations.
- Effective incorporation of priority variation: Reward weights control objective contributions, but substantially different objective difficulties can make the model favor an easier reward despite assigned priorities.Weight differences may need to be sufficiently large, yet the easier reward may still dominate.
- Effective incorporation of priority variation: Conditioning an easier reward on a predefined threshold for a harder reward can force prioritization of the harder objective.Models with conditioned reward functions achieve higher performance on prioritized objectives than models using larger weights alone.
4. Experiments
Experiments compare GDPO with GRPO across tool calling, mathematical reasoning, and coding reasoning under multiple reward objectives. Across these settings, GDPO generally improves convergence, accuracy, and constraint adherence, while avoiding several GRPO failure modes.
- Experiments evaluate GDPO and GRPO on tool calling, mathematical reasoning, and coding reasoning with accuracy, format, length, and code-quality objectives.
- Tool calling: GDPO converges to higher correctness and format rewards than GRPO in tool-calling training curves.GDPO also shows faster early correctness improvement and higher later-stage correctness reward.
- Tool calling: GDPO improves tool-calling accuracy by roughly 2.7% overall and format correctness by more than 4% for Qwen2.5-Instruct-1.5B versus GRPO.The 1.5B model also gains almost 5% and 3% on Live and non-Live tasks, respectively.
- Tool calling: GRPO w/o std matches correctness gains but reaches 0% correct format ratio, failing to learn the required output structure.
- Mathematical reasoning: After roughly 400 steps, GRPO correctness declines and length violations increase, whereas GDPO continues improving correctness and controlling response length.Both methods rapidly maximize the easier length reward, initially suppressing correctness.
- Mathematical reasoning: GDPO-trained models achieve up to an 80% reduction in length-exceeding ratios on AIME while delivering higher accuracy on most mathematical tasks.For DeepSeek-R1-1.5B, GDPO improves accuracy by 2.6% on MATH, 6.7% on AIME, and 2.3% on Olympiad versus GRPO.
- Mathematical reasoning: Changing reward weights has little and inconsistent impact on length-exceeding ratios, unlike GDPO’s more consistent accuracy–efficiency improvements.
- Coding reasoning: In coding reasoning, GDPO2-obj improves Codecontests pass rate by 2.6% with only a 0.1% increase in length-exceeding ratio versus GRPO2-obj.On Taco, it gains 3.3% pass rate while reducing length violations by 1%; with three rewards, it reduces both length-exceeding and bug ratios at similar pass rates.
5. Related Work
Related work extends GRPO for stability and efficiency, while multi-reward reinforcement learning addresses diverse or conflicting preferences through reward design and policy strategies.
- GRPO variants: GRPO extensions modify group normalization or policy updates to improve framework stability, effectiveness, and efficiency.
- GRPO variants: GFPO addresses length explosion by sampling larger groups and filtering responses using length and reward-per-token criteria.
- GRPO variants: DLER combines batch-wise reward normalization, higher clipping, dynamic sampling, and truncation length penalties for accuracy–efficiency trade-offs.
- Multi-reward reinforcement learning: Multi-reward RL methods model diverse human preferences using decoupled preferences, distinct policies, merged inference models, or hierarchical rewards.
6. Conclusion
The paper identifies reward collapse as a fundamental problem when GRPO is applied directly to multi-reward optimization and introduces GDPO to preserve distinctions between rewards. Across tool calling, math reasoning, and coding reasoning, GDPO consistently outperforms GRPO and supports more stable, preference-aligned optimization.
- Directly applying GRPO to summed multi-reward signals can collapse distinct reward combinations into identical advantages, weakening policy updates and causing early training failure.
- GDPO decouples normalization across individual rewards and adds batch-wise advantage normalization to preserve reward differences and stabilize advantage magnitudes.
- The study systematically examines how to incorporate human preference priorities and adjust reward functions when objectives differ substantially in difficulty.
- GDPO consistently outperforms GRPO across tool calling, math reasoning, and coding reasoning, with advantages across reward counts, models, and reward functions.
- Overall, GDPO is presented as a more stable, accurate, and preference-aligned method for multi-reward reinforcement learning.
A. Training stability issue of GDPO without batch-wise advantage normalization
Batch-wise advantage normalization affects GDPO’s training stability. Without normalization, some runs fail to converge.
- Without batch-wise advantage normalization, GDPO training occasionally fails to converge.
B. ToolRL Training Prompt Format
The ToolRL training prompt defines a structured interaction format for reasoning, tool calls, observations, user inputs, and final responses. It also specifies when and how tool calls and responses should be produced.
- The prompt frames the model as a dialogue assistant that can use listed tools to solve user tasks and provide structured responses.
- Tool calls are JSON objects containing a name and parameter dictionary, and multiple calls may be issued simultaneously.
- The model must include a <think> field and provide at least one <tool_call> or <response> field.
- The prompt requires using previous dialogue records, including earlier calls, responses, and tool feedback, when generating subsequent outputs.
- Each interaction includes explicit fields for model reasoning, tool-call or response output, observations, and user input.
C. Tool Calling Reward Functions
The tool-calling reward functions evaluate structural compliance and correctness of predicted tool calls. Correctness compares predictions with ground-truth calls using tool-name, parameter-name, parameter-content, and optimal-matching components.
- The format reward ℛformat is binary and checks whether required fields appear in the correct structure and order.
- The correctness reward ℛcorrect ranges from −3 to 3 and evaluates predicted tool calls against ground-truth calls.
- Correctness includes separate components for tool-name matching, parameter-name matching, and parameter-content matching.
- Tool-name matching compares the sets of names in ground-truth and predicted calls.
- Parameter-name and parameter-content matching compare corresponding keys and values between ground-truth and predicted calls.
- The final correctness reward uses an optimal matching between predicted and ground-truth calls to maximize total match score.
D. ToolRL Hyperparameters Setting
GDPO’s verl training configuration uses the same hyperparameter settings as ToolRL[12].
- All GDPO verl hyperparameter settings are kept identical to those used in ToolRL[12].
E. Math/Coding Reasoning Hyperparameters Setting
The math-reasoning experiments examine GRPO and GDPO training behavior across model sizes, reward configurations, and length-reward weights. GDPO maintains improving correctness and better adherence to length constraints than GRPO in the reported training curves.
- Training curves compare GRPO and GDPO on DeepSeek-R1-7B and Qwen3-4B-Instruct using correctness and length rewards.
- GDPO maintains improving correctness and better adherence to length constraints than GRPO for DeepSeek-R1-7B on math reasoning data.
- GDPO maintains improving correctness and better adherence to length constraints than GRPO for Qwen3-4B-Instruct on math reasoning data.
- The experiments compare GRPO- and GDPO-finetuned DeepSeek-R1-7B models under length reward weights of 1.0, 0.75, 0.5, and 0.25.
- The conditioned-length-reward comparison uses the same length reward weights: 1.0, 0.75, 0.5, and 0.25.