PaperlayerView exact revision on arXiv

Source-linked AI summary

GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization

Haotian Liu, Yihao Liu, Jingwei Ni, Siyuan Huang, Xinpeng Liu, Pengyu Cheng, Jiajun Song, Ruijin Ding, Junfeng Li, Zhechao Yu, Mengyu Zhou, Hongteng Xu, Xiaoxi Jiang, Guanjun Jiang

arXiv:2606.16771v1cs.LG

TL;DR

Multi-reward policy optimization can suffer destructive cancellation when reward-wise advantages disagree. GD2PO filters conflicted rollouts and reweights queries by retained consensus, consistently outperforming existing baselines across tool calling and helpfulness-safety alignment.

  • Problem

    GDPO’s late aggregation can collapse opposing reward-wise advantages into conflicting signals, limiting efficient multi-reward policy optimization.

  • Method

    GD2PO filters rollouts with inconsistent reward-wise advantages and reweights queries according to the fraction of retained, consensus rollouts.

  • Results

    GD2PO consistently outperforms multi-reward RL baselines across tool calling and helpfulness-safety alignment, with up to 1.74 percentage-point Correct Acc gains.

  • Takeaways & Limitations

    GD2PO focuses policy updates on rollouts and queries with more consistent multi-reward signals, reducing advantage cancellation during aggregation.

  • Takeaways & Limitations

    The optimal SNR filtering threshold can vary across tasks, reward scales, reward combinations, and model backbones despite robustness within a practical range.

Abstract

from arXiv · show

As LLMs advance, post-training reinforcement learning (RL) increasingly relies on multi-dimensional rewards to cultivate comprehensive capabilities. This shift demands new algorithms capable of optimizing diverse and potentially competing objectives simultaneously. To address this, existing methods such as Group reward-Decoupled Policy Optimization (GDPO) decompose the overall score into independent reward groups, then compute the RL loss separately within each group. However, this strategy still encounters multi-reward conflicts: a single rollout can yield positive advantages on certain reward dimensions but negative ones on others, causing opposing signals to cancel each other out during aggregation, further hindering RL training efficiency. Inspired by Dynamic sAmpling Policy Optimization (DAPO), which improves RL training efficiency by filtering out ineffective rollouts with near-zero advantages, we propose Group-Dynamic reward-Decoupled Policy Optimization (GD$^2$PO). Specifically, GD$^2$PO employs a conflict-aware filtering mechanism to mask out rollouts suffering from severe reward-wise disagreement. By preventing conflicting signals from canceling each other out, this masking strategy preserves and enhances the magnitude of effective RL advantages, thereby significantly accelerating learning efficiency. Furthermore, we introduce query-level reweighting to dynamically adjust the update intensity of each query based on its overall reward consensus. Experiments on various multi-reward scenarios, including tool calling and human preference alignment, demonstrate that GD$^2$PO consistently and significantly outperforms existing baselines. The code is available at https://github.com/Qwen-Applications/GD2PO.

1. Introduction

The introduction motivates GD2PO by identifying conflicts among diverse reward objectives as a central challenge in multi-reward alignment. It presents GD2PO as a dynamic, group-level filtering and query-reweighting method evaluated on tool calling and helpfulness-safety alignment.

  • LLM deployment requires alignment with diverse human preferences and task-specific requirements spanning potentially conflicting dimensions.
  • Multi-reward optimization faces severe conflicts because opposing reward signals can produce incompatible optimization directions when aggregated directly.
  • Existing approaches address conflicts through reward re-weighting, gradient coordination, and reward-wise advantage normalization, while dynamic filtering improves policy-optimization efficiency by selecting cleaner supervision.
  • GD2PO dynamically filters rollouts according to whether their reward-wise advantages support a consistent updating direction before loss aggregation.It additionally uses query-level reweighting based on the fraction of retained rollouts as a proxy for reward consensus.
  • GD2PO mitigates multi-reward conflicts at rollout and query granularities and is evaluated on tool calling and helpfulness-safety alignment.

2. Preliminary

The preliminary section introduces policy optimization and GRPO, then explains how multi-reward extensions evolve from early scalarization to GDPO’s dimension-wise advantage computation. It also motivates dynamic sampling by showing how DAPO filters uninformative groups and frames the remaining conflict problem.

  • Policy Optimization: Policy optimization trains a policy to generate responses for prompts and maximize their expected scalar reward.Each response is evaluated by a scalar reward function.
  • Policy Optimization: PPO estimates rollout-level advantages with a value model and GAE, while GRPO estimates advantages relative to groups of sampled outputs.GRPO is presented as an efficient alternative for LLM policy optimization.
  • Multi-Reward RL: Early-scalarization multi-reward GRPO combines reward dimensions before advantage normalization, discarding fine-grained dimension-wise feedback.The approach is computationally simple but merges reward signals prematurely.
  • Multi-Reward RL: GDPO computes group-relative advantages separately for each reward dimension before aggregating them into a unified scalar advantage.This preserves dimension-specific reward scales before aggregation.
  • Dynamic Sampling: DAPO retains prompt groups with non-uniform verification outcomes, filtering homogeneous groups with flat advantage landscapes to improve learning efficiency.Each retained group must contain both correct and incorrect candidate responses.

3. Methodology

GD^2PO prevents cross-reward advantage cancellation through rollout-level conflict filtering, then reweights queries according to the fraction of retained, non-conflicting rollouts. It combines these mechanisms to preserve effective update signals and improve learning efficiency.

  • Motivation: GDPO’s weighted aggregation can cancel opposing positive and negative reward-wise advantages, motivating GD^2PO’s conflict-aware intervention before scalar aggregation.The method generalizes DAPO’s dynamic sampling idea to the multi-reward advantage space.
  • Rollout-Level Conflict Filtering: GD^2PO uses Hard Filtering to mask any rollout with opposing signs across reward-wise advantages, while zero-valued advantages remain neutral.A conflict exists when two reward dimensions have opposing advantage signs; zero-valued advantages do not create conflicts.
  • Rollout-Level Conflict Filtering: SNR-Based Filtering preserves rollouts whose aggregate direction remains robust and removes those whose constructive signals are severely diluted by cross-reward cancellation.Its SNR metric approaches 1 for aligned advantages and drops toward 0 when opposing advantages largely offset one another.
  • Unified Filtering Objective: The unified filtering objective uses a binary retention indicator to exclude severely conflicted rollouts and preserve uncontaminated, effective advantages.The group-dynamic filtering strategy is presented as preventing destructive cancellation and accelerating learning efficiency.
  • Query-Level Reweighting: GD^2PO reweights each query by its retained rollout fraction, because updates supported by more non-conflicting rollouts are considered more reliable.The retained count is normalized by the rollout group size, and the resulting fraction adjusts update strength.
  • Final Objective: The final GD^2PO objective combines rollout filtering δ(y_n) with query reweighting κ̂(x), scaling down queries supported by only a small retained fraction.Such queries provide weaker evidence of reward-wise consensus.

4. Experiments

Experiments across tool-calling and helpfulness-safety alignment show that GD2PO improves multi-reward optimization by filtering conflicting rollouts and reweighting queries according to reward consensus. Conflict-aware filtering remains effective across reward settings, with hard filtering favored for two rewards and SNR-based filtering for three.

  • Experimental Setup: GD2PO is evaluated on tool-calling and helpfulness-safety alignment using multiple instruction-tuned backbones, with comparisons against GRPO and GDPO.The tool-calling setting uses correctness, format, and length rewards, while the alignment setting uses useful and harmless rewards.
  • Two-Reward Tool Calling: In the two-reward correctness+length setting, GD2PO generally achieves higher overall Correct Acc across all three backbones while methods maintain Overall Length scores close to 1.00.Both GD2PO variants outperform the baselines overall, and GD2PO-Hard generally exceeds GD2PO-SNR on overall Correct Acc.
  • Helpfulness-Safety Alignment: In helpfulness-safety alignment, the best GD2PO variant achieves higher Overall Avg on both backbones without a clear sacrifice of one reward dimension for the other.This extends the observed effectiveness beyond tool calling to the useful+harmless setting.
  • Three-Reward Tool Calling: In the three-reward correctness+format+length setting, GD2PO-SNR achieves the highest overall Correct Acc across the three backbones while retaining competitive format and length performance.SNR-based filtering is more prominent than hard filtering when three reward dimensions are involved because it preserves useful signals amid mild disagreement and suppresses severe cancellation.
  • Conflict Dynamics: Non-negligible conflict ratios occur across backbones and tasks, with higher early-stage conflict in three-reward tool calling and more persistent conflict in helpfulness-safety alignment.These dynamics motivate conflict-aware filtering as a general mechanism whose behavior depends on the backbone, reward structure, and task objectives.
  • Sensitivity and Ablation: GD2PO-SNR consistently outperforms GDPO across tested thresholds, τ= 0.5 performs best in the three-reward Qwen2.5-3B-Instruct setting, and query-level reweighting further improves filtering.Filtering alone improves overall Correct Acc over GDPO, while adding reweighting produces an additional gain.

5. Related Work

Related work addresses multi-reward policy optimization through reward-level coordination and improves RL post-training efficiency through sample selection or dynamic filtering. These approaches respectively combine rewards before RL updates and remove rollout groups that provide near-zero advantages.

  • Multi-reward policy optimization: Multi-reward policy optimization commonly scalarizes multiple rewards into a single reward before applying standard RL algorithms, often using fixed weights.This reward-level coordination appears in prior multi-objective RL and multi-preference alignment methods.
  • Sample Selection in RL Post-Training: DAPO filters prompt groups whose sampled responses are all correct or all incorrect because they yield near-zero advantages under group-based estimation.This exemplifies sample selection and dynamic filtering to improve RL post-training efficiency and signal quality.

6. Conclusion

The paper identifies destructive cancellation in direct multi-reward advantage aggregation and proposes GD2PO to mitigate it through conflict-aware rollout filtering. GD2PO improves update reliability, but its SNR-based filtering depends on a task-sensitive threshold.

  • Conclusion: Direct linear aggregation suffers destructive signal cancellation across opposing reward dimensions in multi-reward policy optimization.The paper frames this cancellation as a bottleneck for advantage aggregation.
  • Conclusion: GD2PO prunes highly conflicted rollouts using strict sign-consistency or Signal-to-Noise Ratio (SNR)-based rules before final advantage aggregation.This hierarchical design targets severe multi-reward conflicts before combining advantages.
  • Limitations and Future Work: SNR-based filtering improves the reliability of multi-reward policy updates but introduces a threshold τ controlling filtering aggressiveness.Threshold sensitivity analysis indicates reasonable robustness within a practical range, while the optimal threshold can vary across tasks and reward scales.

A. Experimental Details · A.1. Tool-Calling Task

The tool-calling experiments train on RLLA with correctness, format, and length rewards, then evaluate exact tool-call matching on API-Bank across three difficulty levels. The protocol also reports aggregated correctness, format accuracy, and length reward.

  • A.1. Tool-Calling Task: RLLA provides 4K tool-use training examples from ToolACE, Hammer (Masked), and xLAM.The dataset combines 2K ToolACE, 1K Hammer (Masked), and 1K xLAM examples.
  • A.1. Tool-Calling Task: Training responses must include structured reasoning traces and tool invocations.The required output is a structured response containing both elements.
  • A.1. Tool-Calling Task: API-Bank evaluation covers 73 API tools and 314 annotated dialogues containing 753 API calls across three tool-use levels.Responses are generated with vLLM using a maximum generation length of 4096 tokens.
  • A.1. Tool-Calling Task: The correctness reward compares predicted and ground-truth tool names, parameter names, and parameter values, with malformed outputs receiving the minimum score.Its default range is [−3, 3].
  • A.1. Tool-Calling Task: The binary format reward scores whether responses follow the required ordering of <think>, <tool_call>, and/or <response> fields.It takes values in [0, 1].
  • A.1. Tool-Calling Task: The length reward uses word count inside <think>, clips scores to [0, 1], and assigns the minimum score without a valid <think> block.The default maximum length parameter is Lmax = 512.
  • A.1. Tool-Calling Task: Evaluation extracts the <tool_call> block and computes accuracy by exact matching of tool names and parameters against ground truth.Accuracy is reported separately by difficulty, while Correct Acc aggregates examples across all levels.
  • A.1. Tool-Calling Task: Format Acc is reported as a percentage, whereas Length Reward remains on its original [0, 1] scale.The protocol reports Format Acc alongside correctness metrics.

A.2. Helpfulness-Safety Alignment Task

The helpfulness-safety alignment task trains on Alpaca prompts and evaluates on Alpaca, HH-RLHF, and PKU-SafeRLHF/PKU-Alignment. It uses separate useful and harmless reward dimensions to assess general instruction following and safety-aware alignment.

  • Dataset: The task uses Alpaca prompts for policy optimization and evaluates models on Alpaca, HH-RLHF, and PKU-SafeRLHF.These datasets cover general instruction following and helpfulness-safety alignment behavior.
  • Dataset: Alpaca supplies general instruction-following prompts for both RL training and evaluation, while HH-RLHF evaluates helpful and harmless assistant responses.PKU-SafeRLHF provides preference annotations for both helpfulness and safety.
  • Dataset: Alpaca is split into 51,490 policy-optimization examples and 512 evaluation examples with no prompt overlap.HH-RLHF yields 8,520 prompt-only inputs, and PKU-Alignment uses 8,211 official test examples in the same format.
  • Dataset: Validation evaluates all prompt-only examples and reports prompt-level mean@1 across the validation sets.HH-RLHF inputs retain pairs sharing the same dialogue history before the final assistant turn, while PKU-Alignment uses its official test split.
  • Reward models: The reward models provide two dimensions—useful and harmless—where higher scores indicate better helpfulness or safety.They are trained on PKU-SafeRLHF pairwise preference data using Align-Anything and come from the Amo project.

A.3. Implementation Details · B. Additional Details of Our Method · B.1. Overall Algorithm

The experiments use verl on eight NVIDIA A800 GPUs, with task-dependent computational budgets. The method identifies conflicting rollouts, filters them, and reweights each query according to its retained rollout fraction.

  • A.3. Implementation Details: All experiments are implemented using the verl framework.
  • A.3. Implementation Details: Training uses 8 NVIDIA A800 GPUs, each equipped with 80GB of memory.
  • A.3. Implementation Details: Tool-calling runs require approximately 10–20 hours on 8 NVIDIA A800 GPUs.
  • A.3. Implementation Details: Helpfulness-safety alignment runs require approximately 8 hours.
  • B.1. Overall Algorithm: The method identifies conflict rollouts using either sign disagreement or SNR-Based signal preservation.
  • B.1. Overall Algorithm: It computes each query’s retained rollout fraction and uses that fraction to scale the filtered training signal.

B.2. Additional Analysis of Query-level Reweighting · C. Additional Experimental Results · C.1. Validation of 𝛿SNR

The additional analysis explains query-level reweighting through retained-rollout evidence and a signal-to-noise-style reliability perspective. Experiments validate the continuous SNR-based score by showing that low-conflict rollouts provide stronger training signals than high-conflict rollouts.

  • B.2. Additional Analysis of Query-level Reweighting: For each query, GD^2PO defines a retained rollout set S(x) containing rollouts satisfying the filtering rule, with retained count κ(x).The analysis treats each retained rollout as contributing a scalar update quantity z_n.
  • B.2. Additional Analysis of Query-level Reweighting: The training procedure samples grouped responses, applies the filtering rule, estimates each query’s retained fraction, and updates the policy using query-level reweighting.These steps are specified in Algorithm 1’s overall GD2PO training procedure.
  • B.2. Additional Analysis of Query-level Reweighting: Under an independence-style simplifying assumption, the filtered query-level update is analyzed from the retained rollout contributions while ignoring covariance terms.The derivation introduces the filtered update and separately approximates its variance without covariance terms.
  • B.2. Additional Analysis of Query-level Reweighting: A signal-to-noise-style reliability ratio motivates using the retained fraction as a monotonic proxy for query-level update reliability.The analysis argues that, when retained-rollout quality is comparable, more retained rollouts tend to yield more reliable filtered update signals.
  • C.1. Validation of 𝛿SNR: Table 6 validates continuous SNR_n ranking under the correctness+length tool-calling setting by comparing training on the top and bottom halves of rollouts.The halves are labeled Low-Conflict Half and High-Conflict Half according to their SNR_n rankings.
  • C.1. Validation of 𝛿SNR: The Low-Conflict Half achieves the highest overall Correct Acc, while the High-Conflict Half performs worse than GDPO despite both half-settings using the same number of rollouts.This supports the SNR-based score’s ability to distinguish more reliable from less reliable training signals.

C.2. Ablation on Rollout Number · C.3. Threshold Sensitivity on Helpfulness-Safety Alignment · D. Case Study

The ablations show that larger rollout groups improve performance while exposing more reward conflicts, whereas GD2PO-SNR remains stable across SNR thresholds. Case studies further indicate that conflict-aware filtering improves tool-use decisions and balances helpfulness with safety-oriented redirection.

  • C.2. Ablation on Rollout Number: The rollout-number ablation evaluates GDPO and GD2PO-Hard on helpfulness-safety alignment with Qwen2.5-7B-Instruct using G=4 and G=8 rollouts per prompt.Group-based policy optimization uses multiple prompt-level rollouts for advantage estimation and conflict detection.
  • C.2. Ablation on Rollout Number: Increasing rollouts from G=4 to G=8 improves both GDPO and GD2PO-Hard, while GD2PO-Hard maintains a higher Overall Avg under both settings.Larger rollout groups provide more informative within-query comparisons, and conflict-aware filtering remains effective across rollout numbers.
  • C.2. Ablation on Rollout Number: The conflict ratio is generally higher with G=8 than with G=4, indicating that larger groups expose more diverse reward-wise comparisons and detected conflicts.This supports conflict-aware handling before advantage aggregation.
  • C.3. Threshold Sensitivity on Helpfulness-Safety Alignment: Across HH-RLHF, PKU-Alignment, and Alpaca, GD2PO-SNR achieves comparable average scores under different SNR thresholds τ.Results are generally stable in the middle-to-high threshold range, suggesting filtering suppresses unreliable conflict signals without noticeably degrading usefulness or harmlessness.
  • D. Case Study: Four case studies qualitatively compare methods on two API-Bank tool-calling examples and two helpfulness-safety alignment examples.The examples cover inferring the next tool call from multi-turn dialogue and a benign dental consultation alongside an additional alignment case.
  • D. Case Study: Conflict-aware filtering improves semantic decisions by tracking tool dependencies and unfinished dialogue states while avoiding premature or repeated tool calls.The tool-calling examples include password changes requiring authentication and tracking a remaining LikeCount call.
  • D. Case Study: In helpfulness-safety cases, conflict-aware filtering preserves helpfulness on benign requests and produces clearer safety-oriented redirections for potentially inappropriate requests.One example concerns a teeth-veneers consultation and presents a helpful explanatory response.
Loading 2606.16771v1…