PaperlayerView exact revision on arXiv

Source-linked AI summary

Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization

Xiangchi Yuan, Xingjian Diao, Zheyuan Liu, Chunhui Zhang, Dachuan Shi, Kejing Xia, Zhenbang Du, Shenglong Yao, Soroush Vosoughi, Wenke Lee

arXiv:2610.34862v1cs.LGcs.CL

TL;DR

Multi-reward LLM post-training can use additive aggregation that allows auxiliary signals to overturn primary preferences, while hard gating loses auxiliary distinctions. PAPO fuses advantages to preserve primary priority while retaining graded auxiliary supervision, and is reported to improve primary performance across domains and optimization methods while maintaining strong auxiliary behavior.

  • Problem

    In multi-reward post-training with unequal priorities, additive aggregation can let a sufficiently large conflicting auxiliary advantage overturn the preference induced by the primary objective.

    Source: §1 · p. 1, §4 · p. 3

  • Method

    PAPO retains auxiliary contributions when objectives agree and smoothly attenuates conflicting signals to preserve the primary sign, using sequence-level fusion for GRPO and token-level fusion for PPO.

    Source: §1 · p. 2, §5.1 · p. 4

  • Results

    Across mathematical reasoning, agentic tool use, and multimodal safety alignment, PAPO consistently improves primary-objective performance while maintaining strong auxiliary behavior across models and optimization frameworks.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The results suggest that explicitly modeling reward-objective priority is an effective approach to multi-reward policy optimization.

    Source: §7 · p. 9

Abstract

from arXiv · show

LLM post-training often optimizes multiple objectives with unequal priorities, such as correctness and reasoning efficiency. However, the widely used additive reward aggregation in RL can violate these priorities: when primary and auxiliary advantages conflict, a sufficiently large auxiliary signal can overturn the preference induced by the primary objective. We term this problem \emph{advantage priority inversion}. Although reward-level gating can protect priority more directly, it still removes potentially useful auxiliary distinctions among rollout samples that fail the primary condition. To address this trade-off, we propose Primary Advantage Policy Optimization (\textbf{PAPO}), a priority-aware advantage aggregation method that asymmetrically combines a primary and an auxiliary advantage. PAPO retains auxiliary supervision when the objectives agree and smoothly attenuates conflicting auxiliary signals, preserving the primary sign during fusion for \emph{primary-consistent ordering}. This rule applies to sequence-level advantages in GRPO and token-level advantages in actor--critic PPO. Experiments on multiple domains and models show that PAPO consistently achieves the best primary objective while maintaining best or competitive auxiliary-objective performance across both GRPO- and PPO-based optimization. \href{https://xiangchi-yuan.github.io/papo-project/}{Code is available}.

1 INTRODUCTION

The introduction identifies priority inversion and information loss as problems in multi-reward optimization, then presents PAPO as a priority-aware alternative. It reports improved primary-objective performance across domains and optimization methods while maintaining strong auxiliary performance.

1 INTRODUCTION

  • The paper frames multi-reward post-training as optimizing objectives with unequal priorities, such as correctness over reasoning length or task success over tool-use efficiency.Source: §1 · p. 1
  • Additive aggregation can let a conflicting auxiliary advantage overturn the primary preference, while hard gating discards useful distinctions and response-level rules hide token conflicts in PPO.Source: §1 · pp. 1–2
  • PAPO retains auxiliary contributions when objectives agree and smoothly attenuates conflicts so fusion preserves the primary sign and induces primary-consistent ordering.Source: §1 · p. 2
  • PAPO uses sequence-level advantage fusion for GRPO and token-level fusion for actor–critic PPO, with a dual-head critic for objective-specific return estimates.Source: §1 · p. 2
  • PAPO consistently improves the primary objective across multiple domains and models while maintaining strong auxiliary performance under both GRPO- and PPO-based optimization.Source: §1 · p. 2

2 RELATED WORKS

Related work covers policy optimization for LLM reasoning and methods for multi-reward learning. Figure 1 contrasts additive aggregation, hard gating, and PAPO’s advantage-fusion behavior.

Policy Optimization for LLM Reasoning

  • PPO and VAPO are described as widely used in LLM post-training, while GRPO estimates relative advantages from rollout groups without a learned critic.Source: §2 · p. 2

Multi-Reward Reinforcement Learning

  • Multi-reward RL methods include constrained reward-and-cost optimization, conditional length rewards, and dynamically adjusted reward weights.Source: §2 · p. 2
  • Figure 1 contrasts additive sign reversal, hard-gating loss of auxiliary distinctions, and PAPO’s sign-preserving fusion with graded auxiliary supervision.Source: Figure 1 · p. 3

3 PRELIMINARIES

The preliminaries describe PPO’s clipped objective and token-level advantage estimation, alongside GRPO’s sequence-level relative-advantage calculation from response groups.

Proximal Policy Optimization

  • PPO uses an importance ratio between current and old policies in a clipped surrogate objective.Source: §3 · p. 3
  • Actor–critic PPO commonly estimates token-level advantages with GAE from TD errors and trains its value function toward return targets.Source: §3 · p. 3

Group Relative Policy Optimization

  • GRPO samples response groups, normalizes their rewards to estimate relative advantages, and uses those response-level advantages in a PPO-style clipped objective.Source: §3 · p. 3

4 REVISITING MULTI-REWARD LLM: ADVANTAGE PRIORITY INVERSION

The section explains how additive advantage aggregation can invert primary preferences and why reward gating and sequence-level aggregation leave useful distinctions or token-level conflicts unresolved.

4 REVISITING MULTI-REWARD LLM: ADVANTAGE PRIORITY INVERSION

  • When conflicting auxiliary advantages dominate the primary signal, additive aggregation can reverse the primary-induced preference, a problem termed advantage priority inversion.Source: §4 · p. 3
  • Reward-level gating blocks auxiliary rewards when the primary condition fails, but removes potentially useful auxiliary distinctions among those responses.Source: §4 · pp. 3–4
  • In PPO, primary and auxiliary advantages can agree at some tokens and conflict at others, so sequence-level aggregation can hide local conflicts affecting token updates.Source: §4 · p. 4

5 METHOD

PAPO fuses primary and auxiliary advantages so aligned signals contribute while conflicting auxiliary signals cannot overturn the primary preference. It instantiates this rule for sequence-level GRPO and token-level PPO optimization.

5.1 PRIMARY ADVANTAGE POLICY OPTIMIZATION (PAPO)

  • When the advantages agree, PAPO retains the auxiliary contribution; when they conflict, it smoothly attenuates that signal without reversing the primary sign.The resulting primary-consistent ordering lets auxiliary supervision refine preferences within each primary-preference side, but not promote a primary-disfavored unit above a favored one.Source: §5.1 · p. 4
  • Centered normalization preserves the ordering between units favored and disfavored by the primary objective.For units in the same normalization set, a positive primary advantage remains ranked above a negative primary advantage after normalization.Source: §5.1 · p. 4

5.2 GRPO-BASED PAPO: SEQUENCE-LEVEL ADVANTAGE AGGREGATION

  • GRPO independently normalizes primary and auxiliary sequence-level advantages within each rollout group, fuses them with PAPO, and then applies centered normalization.Although centering may change an individual advantage’s absolute sign, auxiliary signals cannot rank a primary-disfavored response above a primary-favored response.Source: §5.2 · p. 5

5.3 PPO-BASED PAPO: TOKEN-LEVEL ADVANTAGE AGGREGATION

  • PPO applies PAPO to token-level advantages computed separately for primary and auxiliary objectives using GAE.Token-level fusion addresses local primary–auxiliary conflicts that may differ across positions within one trajectory; centered normalization retains the primary-consistent ordering.Source: §5.3 · pp. 5–6
  • PPO’s dual-head critic estimates each objective’s returns separately, leaving PAPO fusion to the actor update.Separate unfused targets preserve each objective’s value-estimation semantics and token-level credit assignment.Source: §5.3 · p. 6

6 EXPERIMENTS

Experiments test whether PAPO improves primary objectives while preserving competitive auxiliary behavior across mathematical reasoning and tool use, then examine training dynamics, design components, and generalization.

6.1 COMPETITION-LEVEL MATH REASONSING

  • Accuracy rose from 43.6 to 56.6 under PPO and 51.0 to 59.9 under GRPO; length compliance rose from 76.9 to 80.3 and 81.2 to 87.6, respectively.PAPO achieved the best primary accuracy under both optimization methods.Source: Table 1 · p. 7, §6.1 · p. 7

6.2 AGENTIC TOOL USING

  • Average accuracy rose from 86.26 to 88.85 under PPO and 87.70 to 89.93 under GRPO; GRPO tool use stayed at 0.33 calls, while PPO rose from 0.37 to 0.56.Compared with GDPO, PAPO-GRPO raised average accuracy from 86.85 to 89.93 with calls increasing from 0.25 to 0.33.Source: Table 2 · p. 7, §6.2 · p. 8

6.3 TRAINING AND EVALUATION DYNAMICS

  • Across GRPO and PPO training, PAPO maintains stronger, more stable accuracy while progressively improving length reward; baselines often optimize length without consistent accuracy gains.The instability or degradation in correctness with aggressive length optimization is especially pronounced under PPO.Source: Figure 3 · p. 8, §6.3 · p. 8

6.4 ABLATION STUDIES

  • Across α values, PAPO maintains stronger primary accuracy with competitive auxiliary performance; under PPO, adding PAPO to MHC raises math accuracy to 61.1 and tool-use accuracy to 88.25.MHC alone raises math accuracy from 54.5 to 55.7 and tool-use accuracy from 86.25 to 86.84, with improved length compliance.Source: §6.4 · p. 8, Table 3 · p. 9, Figure 4 · p. 9

6.5 ADDITIONAL MODEL SCALE AND DOMAINS

  • On DeepSeek-R1-Distill-1.5B, PAPO raises average accuracy from 45.8 to 48.1 and length compliance from 75.8 to 80.7 versus PPO.Gains are consistent across most benchmarks.Source: Table 4 · p. 9, §6.5 · p. 9
  • In multimodal safety alignment, PAPO consistently improves safety while preserving strong helpfulness, with a favorable trade-off across Qwen2-VL-7B and LLaVA-1.6-7B.Safety is the primary objective and helpfulness the auxiliary objective in this setting.Source: Figure 5 · p. 9, §6.5 · p. 9

7 CONCLUSION

PAPO is a priority-aware advantage aggregation framework that preserves aligned auxiliary supervision while constraining conflicting signals. Across three domains, experiments report improved primary-objective performance with strong auxiliary behavior.

7 CONCLUSION

  • Across mathematical reasoning, agentic tool use, and multimodal safety alignment, PAPO improves primary-objective performance while maintaining strong auxiliary behavior.The reported gains generalize across models, scales, reward semantics, and optimization frameworks.Source: §7 · p. 9
  • PAPO preserves aligned auxiliary supervision while smoothly constraining conflicting signals.Source: §7 · p. 9
  • PAPO applies to GRPO and PPO with token-level optimization and objective-specific critic heads.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

The supplementary material documents experiment settings, implementation details, additional analyses, and evaluation protocols to support reproduction of the reported experiments.

REPRODUCIBILITY STATEMENT

  • The supplementary material provides implementations, training configurations, and evaluation scripts for the main experiments, covering GRPO- and PPO-based PAPO implementations.It includes the dual-head critic used in the PPO setting.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix C documents datasets, reward definitions, model backbones, training objectives, hyperparameters, and evaluation protocols.It also gives implementation details for mathematical reasoning and agentic tool use, including greedy and Pass@k decoding protocols.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix B reports ablations and analyses of auxiliary-reward weight, critic design, training and evaluation dynamics, and stochastic-decoding Pass@k results.Prompt templates and task-specific evaluation details are also provided.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The work studies multi-objective reinforcement learning for LLM post-training across mathematical reasoning, agentic tool use, and safety alignment. It reports no human subjects or private user data and uses controlled benchmark environments for tool-use experiments.

ETHICS STATEMENT

  • The work studies multi-objective reinforcement learning for LLM post-training across mathematical reasoning, agentic tool use, and safety alignment.Source: ETHICS STATEMENT · p. 10
  • Experiments involve no human subjects or private user data, and tool-use experiments use controlled benchmark environments.Source: ETHICS STATEMENT · p. 10
  • The authors encourage future applications to define the objective hierarchy carefully and assess whether it reflects intended safety and deployment requirements.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

Generative AI tools assisted with manuscript language and literature discovery; the authors reviewed edits, verified references, and take responsibility for the final work.

AI USE STATEMENT

  • Generative AI tools assisted with grammar correction, language polishing, manuscript clarity and concision, and discovery of potentially relevant literature.Source: AI USE STATEMENT · p. 10
  • The authors reviewed all AI-assisted edits and independently verified suggested references against original sources.Source: AI USE STATEMENT · p. 10
  • The authors take full responsibility for the final text, citations, claims, experimental results, and presentation.Source: AI USE STATEMENT · p. 10

A PROOF

PAPO preserves the primary sign during advantage aggregation and preserves primary-consistent ordering after centered normalization. Additional tool-use results show stronger accuracy with comparable tool-call efficiency, while training dynamics show that aggressive tool reduction can coincide with weaker correctness.

A PROOF

  • PAPO preserves each nonzero primary advantage’s sign during aggregation and preserves ordering across primary-favored and primary-disfavored units after centered normalization.Normalization may change an individual fused advantage’s absolute sign by shifting the common reference point, without reversing the primary preference.Source: A PROOF · pp. 14–15

B.1 ADDITIONAL MODELS’ RESULTS

  • 81.11 overall accuracy for PAPO on Qwen3-1.7B, versus 75.96 for GRPO and 76.85 for GDPO, with slightly more tool calls but comparable efficiency.PAPO also achieves the best primary safety performance and best or very competitive auxiliary helpfulness on Qwen2-VL-7B.Source: §B.1 · p. 15, Table 5 · p. 15, Figure 6 · p. 15

B.2 TRAINING AND EVALUATION DYNAMICS FOR TOOL

  • PAPO maintains stronger, more stable evaluation accuracy with moderate tool use, while more aggressive tool reduction can coincide with weaker or more volatile correctness.This trade-off is reported for both GRPO- and PPO-based methods; standard PPO shows especially aggressive tool-call reduction.Source: §B.2 · pp. 15–16, Figure 7 · p. 16

B.3 PER-DATASET EVALUATION DYNAMICS

This section examines PAPO’s per-dataset evaluation dynamics, including long-horizon tool-use tasks where correctness is primary and efficiency is auxiliary. It describes how PAPO preserves correctness while improving efficiency and remains competitive under stochastic Pass@k evaluation.

B.3 PER-DATASET EVALUATION DYNAMICS

  • Across math benchmarks, PAPO preserves stronger accuracy while progressively improving length compliance, whereas GRPO, GDPO, and especially PPO can improve length reward faster while accuracy lags or drops.Minerva is less discriminative because length compliance quickly reaches its ceiling and accuracy varies over a narrow discrete range.Source: §B.3 · p. 16, Figure 8 · p. 17
  • On easy, medium, and hard tool-use subsets, PAPO maintains strong accuracy, particularly on medium and hard tasks, while PPO can reduce tool calls earlier but show volatile correctness.PAPO retains additional tool usage when needed to prioritize successful task completion over efficiency.Source: §B.3 · pp. 16–17, Figure 9 · p. 18

B.4 ADDITIONAL PASS@K EVALUATION FOR MATH

  • Pass@k uses stochastic multiple-response decoding, so its absolute values are not directly comparable to the main results, which use greedy single-response decoding.Pass@k complements greedy evaluation by measuring whether at least one of multiple sampled generations is correct.Source: §B.4 · p. 18
  • PAPO-GRPO raises average Pass@1 from 52.49 to 59.97 over GRPO, while PAPO-PPO raises it from 50.08 to 59.08 over PPO and reaches Pass@5 67.90.PAPO-GRPO reaches Pass@5 67.63; the two variants occupy the top two positions for every Pass@k metric, and achieve length-budget compliance of 89.86% and 83.42%, respectively.Source: §B.4 · p. 18, Table 6 · p. 19

D.2.1 THREE-STEP CODE EXECUTION EXAMPLE

  • In the example, execution results 987 and 549 feed successive programs, whose final reported result is 93; the associated reward vector is (1, 0.25).The second and third programs are dynamically constructed from preceding execution observations.Source: §D.2.1 · pp. 27–29

D.2.2 THREE-HOP RETRIEVAL EXAMPLE

  • Three-hop retrieval proceeds from Herman Melville to the United States to Washington, D.C., with each query built from the previous retrieval.The agent must carry intermediate state across steps and return only the requested final entity.Source: §D.2.2 · pp. 29–32
  • The examples contrast numerical program construction with semantic query construction while using correctness as primary and tool-use efficiency as auxiliary supervision.The code task requires prior numerical outputs to instantiate later programs; retrieval requires prior entities to construct later queries.Source: §D.2.1 · p. 29, §D.2.2 · p. 32

Related papers

Loading 2610.34862v1…