PaperlayerView exact revision on arXiv

Source-linked AI summary

GPDPO: Learning from Reward Conflicts via Group Pareto-Dominance Policy Optimization

Zixuan Xu, Haoxuan Chen, Jiangxin Sun, Jian-Fang Hu

arXiv:2610.35480v1cs.LGcs.CL

TL;DR

Conflicting reward signals complicate multi-reward LLM optimization, and response-level filtering can discard useful supervision. GPDPO recovers pairwise preferences from Pareto-dominance relations among rollouts while retaining conflict-aware masking for direct updates. On API-Bank, it consistently outperforms competitive baselines and reduces both group-relative and pairwise reward conflicts.

  • Problem

    Conflict filtering removes responses with sign-disagreeing reward-wise advantages, even though such responses may still contain useful pairwise supervision.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    GPDPO preserves conflict-aware masking for direct updates and derives auxiliary pairwise preference supervision from Pareto-dominance relations involving filtered rollouts.

    Source: §1 · p. 2

  • Results

    Across all four model-scale and reward-setting combinations, GPDPO attains the highest Correct Acc. and Overall; under 2R, it improves Correct Acc. over GD2PO by 2.51 percentage points on Qwen3-1.7B.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    GPDPO reuses existing rollouts without additional sampling or reward evaluation and reduces both group-relative and pairwise reward conflicts.

    Source: §1 · p. 2, §5 · p. 9

    Evaluation is limited to two- and three-reward settings because suitable benchmarks with more reward dimensions are unavailable.

    Source: B LIMITATIONS · p. 15

Abstract

from arXiv · show

Multi-reward reinforcement learning offers a promising framework for aligning large language models (LLMs) with diverse objectives, but conflicting reward signals can induce ambiguous update directions and hinder stable optimization. Existing approaches address such conflicts through reward reweighting, gradient coordination, or rollout filtering, yet they primarily operate on per-response or group-level optimization signals and underexploit relational information among sampled responses. As a result, conflict-filtered rollouts may still contain useful supervision that is discarded during training. We propose Group Pareto-Dominance Policy Optimization (GPDPO), a multi-reward policy optimization method that recovers pairwise supervision from conflict-filtered rollouts. GPDPO identifies Pareto-dominance relations among sampled responses and converts them into auxiliary preference signals, which are jointly optimized with the original policy objective. This enables more effective reuse of existing rollouts without additional sampling or reward evaluation. Experiments on API-Bank across multiple model scales and reward configurations show that GPDPO consistently outperforms competitive multi-reward optimization baselines. Further analysis demonstrates that GPDPO reduces both group-relative and pairwise reward conflicts, supporting the effectiveness of exploiting relational supervision for multi-reward alignment. Code will be released.

1 INTRODUCTION

Multi-reward optimization can produce conflicting signals, and response-level filtering may discard all supervision from a response. GPDPO retains conflict-aware masking for direct updates while recovering reward-consistent pairwise preferences from filtered responses.

1 INTRODUCTION

  • Conflict-aware filtering removes responses with sign-disagreeing reward-wise advantages, but discards all supervision associated with each filtered response.This motivates asking whether a response excluded from a policy update can still provide useful learning information.Source: §1 · p. 1
  • A response with conflicting group-relative advantages may still dominate another consistently across reward dimensions, yielding a reliable pairwise preference.Thus, an unreliable direct optimization signal can remain informative as relational supervision.Source: §1 · pp. 1–2
  • GPDPO preserves conflict-aware masking for direct updates and converts Pareto-dominance relations involving filtered rollouts into an auxiliary pairwise preference loss.It reuses existing samples without additional sampling or reward evaluation.Source: §1 · p. 2, Figure 1 · p. 2
  • Across API-Bank experiments spanning two Qwen3 model scales and two reward configurations, GPDPO consistently outperforms competitive multi-reward baselines.The contributions summary reports the same benchmark-level result across varied model scales and reward configurations.Source: §1 · p. 2

2 RELATED WORK

Prior work tackles multi-reward learning through reward aggregation, gradient or advantage coordination, and Pareto-based comparisons. GPDPO adds pairwise supervision from relations among rollouts, including responses filtered for conflicts.

2 RELATED WORK

  • RLVR work develops group-relative advantage estimation, sampling and objective refinements, and sequence-level policy optimization; scalar reward aggregation may obscure reward-specific distinctions.The cited approaches include GRPO, sampling and objective refinement, and sequence-level optimization.Source: §2 · p. 2
  • Multi-reward methods use fixed or adaptive reward weighting, gradient balancing, and reward-wise or blockwise advantage estimation; GD2PO filters rollouts with conflicting advantages.These approaches alter how reward signals contribute to policy updates.Source: §2 · pp. 2–3
  • Prior work uses Pareto or reward-consistency criteria to assign trajectory advantages, construct preference pairs, or downweight conflicting pairs; GPDPO uses dominance relations for supplementary rollout supervision.The cited applications include NSGA-II and Pareto-based methods for LLM alignment.Source: §2 · p. 3

3 METHOD

GPDPO addresses the sensitivity and information loss of group-level conflict filtering by recovering Pareto-consistent comparisons from filtered rollouts and adding them to multi-reward policy optimization.

3.1 PRELIMINARIES

  • Multi-reward group optimization normalizes each reward separately and filters responses whose reward-wise advantages disagree in sign; Pareto dominance requires no worse rewards in every dimension and a strict gain in one.The binary gate retains responses with consistent advantage signs and filters those with conflicting directions.Source: §3.1 · p. 3

3.2 RETHINKING CONFLICT FILTERING IN MULTI-REWARD RL

  • 88.42% (2R) and 86.67% (3R) of filtered responses had consistent comparisons with at least one peer; 23.22% (2R) and 27.33% (3R) matched the ground-truth ordered tool-call structure.These findings show that filtered responses can retain useful pairwise and task-level supervision.Source: Figure 2 · p. 4, §3.2 · p. 4
  • 16.19% (2R) and 27.09% (3R) of fixed targets changed filtering decisions when the group expanded from G = 4 to G = 64.With peer replacement, inconsistency rates were 9.15% (2R) and 15.92% (3R) at G = 4, and 5.43% (2R) and 6.09% (3R) at G = 64.Source: Figure 2 · p. 4, §3.2 · p. 4
  • Group-relative filtering aggregates reward differences across peers, so its coarse conflict signals can miss consistent comparisons between individual responses.The same response can receive different filtering decisions when its reference group changes.Source: §3.2 · p. 4

3.3 GPDPO: GROUP PARETO-DOMINANCE POLICY OPTIMIZATION

  • GPDPO selects Pareto-dominance pairs involving at least one filtered rollout, stops gradients through retained responses, and adds the pairwise preference loss to the standard multi-reward objective.It reuses supervision in existing rollouts without additional sampling or reward evaluation.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Across API-Bank model scales and reward settings, GPDPO achieves the highest Correct Acc. and Overall scores, while its training dynamics show stronger reward optimization and lower conflict rates. Ablations identify benefits from allowing Pareto ties, refreshing the reference policy, and directly optimizing preference margins.

4.1 EXPERIMENTAL SETTINGS

  • Experiments use Qwen3-1.7B and Qwen3-4B on API-Bank, with correctness and length rewards in 2R and format added in 3R.Training runs for 200 rollout steps; API-Bank evaluation uses the selected step-200 checkpoint, and Correct Acc. is the primary task-performance measure.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • GPDPO attains the highest Correct Acc. and Overall score across all four model-scale and reward-setting combinations; in 2R, Correct Acc. rises from 62.48% to 64.99% on Qwen3-1.7B and from 64.66% to 66.00% on Qwen3-4B.In 3R, it achieves 63.32% and 65.33% Correct Acc. on Qwen3-1.7B and Qwen3-4B, respectively, outperforming GD2PO by 0.84 and 2.01 percentage points.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7

4.3 TRAINING AND CONFLICT DYNAMICS

  • Over steps 151–200, Qwen3-1.7B GPDPO reaches 1.6260 mean correctness reward versus 1.4242 for GRPO and 1.4035 for GD2PO, while retaining higher entropy than both.Its format and length rewards are 0.9566/0.9914 versus GD2PO’s 0.8902/0.9328; mean entropy is 0.0795 versus 0.0746 for GRPO and 0.0719 for GD2PO.Source: Figure 4 · p. 8, §4.3 · p. 8
  • GPDPO’s group-relative and pairwise conflict rates decline faster and finish lower than those of GRPO, GDPO, and GD2PO.The pairwise rate covers non-tied within-group pairs, while the GD2PO rate covers valid rollouts.Source: Figure 5 · p. 8, §4.3 · p. 8

4.4 ABLATION AND SENSITIVITY

  • Strict dominance scores 63.48% Correct Acc., 1.51 points below full GPDPO, while scalarized ordering scores 61.81%, below GD2PO’s 62.48%.Strict dominance excludes tied comparisons; standard Pareto dominance permits ties when one response is no worse on every reward and better on at least one. The scalarized result is attributed to losing reward-consistent preference directions.Source: §4.4 · p. 9
  • With a fixed reference, Correct Acc. falls to 62.81% from full GPDPO’s 64.99% under the tested configuration.The fixed-reference variant keeps the initial policy as reference throughout training; the authors suggest refreshing the reference alongside the evolving sampling policy provides a more effective optimization target.Source: §4.4 · p. 9
  • Pareto-guided advantage adjustment reaches 63.82% Correct Acc. versus GD2PO’s 62.48%, while full GPDPO reaches 64.99%.The authors suggest directly optimizing pairwise preference margins is a more effective way to exploit recovered relations.Source: §4.4 · p. 9
  • Across λ = 0.05, 0.1, 0.15, and 0.2, GPDPO scores 63.82%, 64.99%, 64.66%, and 63.15% Correct Acc., respectively, versus 62.48% for GD2PO.The authors report improvements across the tested auxiliary-supervision strengths rather than dependence on a single tuned coefficient.Source: §4.4 · p. 9

5 CONCLUSION

GPDPO recovers reward-consistent pairwise supervision from responses discarded by conflict filtering. Experiments show improved multi-reward optimization and reduced conflicts, supporting relational supervision for multi-objective LLM alignment.

  • GPDPO consistently improves multi-reward optimization performance while reducing both group-relative and pairwise reward conflicts.Source: §5 · p. 9
  • GPDPO uses Pareto-dominance relations within existing rollout groups to recover pairwise supervision without additional sampling or reward evaluation.Source: §5 · p. 9
  • The results support the value of fine-grained relational supervision for multi-objective LLM alignment.Source: §5 · p. 9
  • The authors report using generative AI tools only for manuscript writing and readability, and state that they will release the source code soon.Source: REPRODUCIBILITY STATEMENT · p. 10

A IMPLEMENTATION DETAILS

The implementation trains GPDPO on RLLA-4K with Qwen3 models and specified reward and evaluation settings. Its algorithm constructs Pareto-dominance pairs involving filtered responses and jointly optimizes the policy and pairwise losses.

A.1 DETAILED SETTINGS

  • Experiments use Qwen3-1.7B and Qwen3-4B on 3,920 RLLA-4K examples, with 512×4 generated responses per rollout step and four mini-batch updates.Overlength prompts are filtered during loading; all reward dimensions use unit aggregation weights, and GPDPO uses uniform pair weights with λ = 0.1 and β = 0.1.Source: §A.1 · p. 13
  • Correctness, length, and format rewards measure tool-call agreement, think-block length, and prescribed output-tag structure, respectively.Correctness is scaled to [−3, 3], length is capped at 1, and format is binary; a missing complete think block receives zero length reward.Source: §A.1 · p. 13
  • Each main-table result is one evaluation of a selected run; the 4,096-token sequence limit includes the prompt and can reduce generation length.CORRECT ACC. covers all 597 questions; OVERALL adds length reward to CORRECT ACC./100 in 2R and also adds FORMAT ACC./100 in 3R.Source: §A.1 · p. 13

A.2 DETAILED ALGORITHM

  • For each rollout batch, the algorithm builds dominance relations and preference pairs once, then evaluates scores and joint updates per mini-batch.It selects dominance edges involving at least one filtered response for the pairwise loss while retaining the standard conflict-filtered advantages.Source: §A.2 · p. 14
  • Pair comparisons use reward-wise normalized advantages before filtering with tolerance 10^-8; positive per-dimension normalization preserves Pareto ordering in exact arithmetic.Sequence scores average log-probability ratios over valid response tokens; retained endpoints are detached only in the pairwise term, while two filtered endpoints both receive gradients.Source: §A.2 · p. 14
  • The pair loss is softplus(−βm), normalized by the full rollout batch’s pair count, and pairs reuse existing rollout responses and rewards.MaskedWhiten whitens retained-token advantages and leaves filtered advantages zero; the clipped policy loss is GD2PO’s standard objective and updates use AdamW.Source: §A.2 · p. 14

B LIMITATIONS

The evaluation covers only two- and three-reward settings because suitable benchmarks with more reward dimensions are unavailable. Future work will examine higher-dimensional settings and scalable ways to exploit signals from conflicting responses.

  • The evaluation is limited to two- and three-reward settings because suitable benchmarks with more reward dimensions are currently unavailable.Source: B LIMITATIONS · p. 15
  • Future work will explore reward conflicts in higher-dimensional settings and scalable strategies for exploiting informative signals from conflicting responses.Source: B LIMITATIONS · p. 15
  • The stated scope boundary concerns benchmark availability, not a reported failure of GPDPO in higher-dimensional settings.Source: B LIMITATIONS · p. 15

C ADDITIONAL DETAILS FOR THE CONFLICT-FILTERING ANALYSIS

The supplementary analysis reports settings and results for Qwen3-1.7B trained with GD2PO under two- and three-reward configurations.

C.1 SENSITIVITY TO THE REFERENCE GROUP

The filtering label of a fixed response can change with the reference group’s composition. Both adding responses and replacing peers produce label changes in the 2R and 3R settings.

C.1 SENSITIVITY TO THE REFERENCE GROUP

  • The comparisons use fixed target responses and rewards, varying group size or peer composition across repeated comparisons.The analysis uses step-50 checkpoints, four target responses per prompt, and 128 additional pool responses per prompt.Source: §C.1 · p. 15
  • At G = 64, peer replacement produces 5.43% inconsistent labels under 2R and 6.09% under 3R.Expansion changes 16.19% of labels under 2R and 27.09% under 3R.Source: §C.1 · p. 15
  • Across 65,536 paired label comparisons per reward setting, group expansion changes 10,608 labels under 2R and 17,756 under 3R.These correspond to 16.19% under 2R and 27.09% under 3R.Source: §C.1 · p. 15

C.2 INFORMATION RETAINED IN FILTERED RESPONSES

Many conflict-filtered responses retain task-relevant content or consistent Pareto comparisons. The analysis finds such relations for most filtered responses in both reward settings.

C.2 INFORMATION RETAINED IN FILTERED RESPONSES

  • The analysis covers steps 1–200 of two GD2PO runs at G = 4 and compares each filtered response with its original peers using Pareto dominance.Table 7 is scoped to edge statistics under 2R, with all edge counts involving at least one filtered response.Source: §C.2 · pp. 15–16, Table 7 · p. 16
  • At least one consistent relation exists for 88.42% of filtered responses under 2R and 86.67% under 3R.The corresponding counts are 31,774/35,937 and 50,865/58,689; 4,163 and 7,824 responses, respectively, have no such relation.Source: §C.2 · p. 16
  • Ground-truth tool-call matches account for 23.22% of filtered responses under 2R and 27.33% under 3R.Matches assess ordered tool-call lists, including call order, tool names, and parameter names and values.Source: §C.2 · pp. 15–16

D EXPERIMENTS ON PARETO-DOMINANCE EDGES

The comparison examines pairwise edges involving filtered responses, contrasting Pareto dominance with strict improvement in every reward dimension. Pareto dominance admits many additional edges where a reward is tied but another improves.

D EXPERIMENTS ON PARETO-DOMINANCE EDGES

  • The comparison focuses on how allowing ties in individual reward dimensions affects the available edges for recovering pairwise supervision from filtered responses.Source: D EXPERIMENTS ON PARETO-DOMINANCE EDGES · p. 16
  • Pareto dominance requires no lower reward in any dimension and a higher reward in at least one, while strict improvement requires higher rewards in every dimension.Both criteria use candidate pairs involving at least one filtered response, excluding retained–retained pairs.Source: D EXPERIMENTS ON PARETO-DOMINANCE EDGES · p. 16
  • Under GD2PO, Pareto dominance retains 52,421 edges, versus 6,374 under all-dimension strict improvement.The additional 46,047 edges include 29,229 correctness ties and 16,818 length ties with improvement in the other reward; the GPDPO run has 46,293/52,631 (87.96%) Pareto edges involving such ties.Source: D EXPERIMENTS ON PARETO-DOMINANCE EDGES · p. 16
  • Allowing ties in some reward dimensions while requiring improvement in others retains more comparison edges and provides richer pairwise learning signals.The analysis uses historical GD2PO and GPDPO runs with Qwen3-1.7B on RLLA-4K, using correctness and length rewards and groups of size G = 4.Source: D EXPERIMENTS ON PARETO-DOMINANCE EDGES · p. 16

Related papers

Loading 2610.35480v1…