Source-linked AI summary
Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning
Yu Yuan, Yaoyou Fan, Lili Zhao, Guangting Zheng, Kai Zhang, Lu Pan, Ke Zeng, Qi Liu
TL;DR
Fixed scalarization can alias distinct reward profiles and lock optimization onto easy or dense dimensions, leaving balanced, higher-performing profiles unreached. The paper proposes AMRP, which adapts projection weights using three online signals, and reports consistent improvements across multi-reward settings with compatibility across RL algorithms.
Problem
Static scalarization can alias distinct reward profiles and lock optimization into easy, dense, or fast-improving dimensions, leaving important dimensions under-optimized.
Method
AMRP adapts reward-projection weights online using relative shortfall, reward volatility, and recent progress.
Results
AMRP consistently improves reward-profile trade-offs and downstream performance across structured reasoning, grounded generation, and open-ended alignment, while remaining compatible with GRPO, GDPO, and PPO.
Takeaways & Limitations
Adaptive aggregation provides a lightweight way to prioritize lagging or unstable reward dimensions without modifying the underlying policy objective.
Takeaways & Limitations
The evaluation does not cover broader settings such as strongly conflicting or numerous reward dimensions, full-parameter training, code generation, tool use, or safety-oriented alignment.
Abstract
from arXiv · showhide
Reinforcement learning fine-tuning of large language models increasingly adopts multiple reward dimensions, including verifiable rules, task-specific evaluators, and learned reward models, to provide richer supervision across diverse capabilities. These dimensions are commonly scalarized with fixed aggregation weights. We identify a failure mode in which aggregation itself induces reward hacking: static projection aliases qualitatively different reward profiles into a single scalar, steering optimization toward whichever dimensions are easiest, densest, or systematically favored by the reward signal. Over training, this traps the policy in suboptimal profiles and prevents convergence to better-balanced ones that would yield higher task performance. To address this, we propose Adaptive Multi-Reward Projection (AMRP), a lightweight online method that reallocates aggregation weights using three signals, relative shortfall, reward volatility, and recent progress, increasing pressure on lagging, unstable, or stagnant dimensions while relieving saturated ones. Across structured reasoning, citation-grounded generation, and open-ended alignment under GRPO, AMRP consistently improves reward-profile balance and downstream performance over fixed and dynamic weighting baselines; it also remains effective with GDPO and PPO, supporting compatibility across RL algorithms. Our code is available at https://github.com/yyhappier/AMRP.git.
1 Introduction
Multi-reward LLM post-training commonly compresses heterogeneous reward dimensions into a fixed scalar, which can favor easy dimensions and induce reward-profile collapse. AMRP adapts projection weights online to rebalance learning, with reported gains across several settings and RL algorithms.
- Motivation: Fixed scalarization is simple and compatible with PPO and GRPO but can make qualitatively different reward profiles indistinguishable.This lossy projection obscures which reward dimensions drive learning.
- Motivation: Reward-profile collapse arises when easy or dense dimensions improve first, leaving harder dimensions under-optimized and trapping the policy in shortcut profiles.In mathematical reasoning, format and length rewards can improve faster than answer correctness.
- AMRP: AMRP adapts aggregation weights online using relative shortfall, reward volatility, and recent progress.It raises pressure on lagging, unstable, or stagnant dimensions while reducing weight on saturated and stable dimensions.
- Evaluation: AMRP is evaluated on mathematical reasoning, citation-grounded generation, and open-ended alignment with multiple backbones.The evaluation examines both downstream performance and reward-profile dynamics.
- Results: AMRP shows consistent downstream gains and compatibility with GRPO, GDPO, and PPO.The reported compatibility follows from adapting aggregation rather than changing the underlying policy optimization method.
2 Related Work
Prior work studies reward hacking, multidimensional rewards, and dynamic aggregation, while this paper focuses on failures caused by scalarizing heterogeneous rewards with fixed weights. Its diagnostic framing centers on collapse toward easy or dense dimensions.
- Reward Hacking and Overoptimization: Reward hacking and shortcut learning describe optimization that exploits imperfect proxies or spurious features instead of the desired solution.Related RLHF concerns include reward-model overoptimization and Goodharting.
- Multi-Dimensional Rewards: Modern LLM post-training combines deterministic feedback, automatic evaluators, and learned multi-attribute reward models.Examples include answer verification, code execution, format constraints, citation quality, fluency, and multi-objective reward models.
- Diagnostic Evidence: Static weighting in math reasoning can increase scalar reward while answer accuracy drops as format and length rewards saturate.The figure caption identifies a shortcut profile with zero accuracy and full format and length rewards.
- Multi-Reward Aggregation: Fixed weighted sums are common because response quality is multidimensional, but differences in density, variance, scale, or difficulty can discard learning-relevant information.This motivates examining aggregation itself as a source of reward hacking.
3 Aggregation-Induced Reward Hacking
Static aggregation projects multidimensional reward profiles into scalar signals, allowing distinct behaviors to receive identical training treatment and enabling shortcut lock-in. Mathematical-reasoning evidence shows scalar reward rising while accuracy falls as format and length rewards saturate.
- Static Aggregation as Projection: Multi-reward reinforcement learning evaluates each generated response across K reward dimensions representing distinct aspects of quality.The reward profile is written as r(x, y) = [r1(x, y), . . . , rK(x, y)].
- Static Aggregation as Projection: Static aggregation maps the reward profile to a scalar, so policy updates no longer directly expose distinctions among reward dimensions.The projection weights determine the scalar reward gaps that drive group-relative advantages in GRPO.
- Reward-Profile Collapse: Profile aliasing occurs when distinct profiles receive the same scalar value, such as [1, 0, 0], [0, 1, 0], and [0, 0, 1] under equal weights.These profiles can represent correctness, formatting, or length satisfaction in different combinations.
- Reward-Profile Collapse: Shortcut lock-in occurs when easy or dense dimensions improve faster, producing high scalar reward without solving the task and weakening signals from harder dimensions.Once shortcut profiles dominate rollouts, recovery becomes difficult and balanced profiles with higher task performance may remain unreachable.
- Diagnostic Evidence: In mathematical reasoning, short format-compliant responses can satisfy format and length rewards while failing answer correctness, reducing informative correctness signals in rollout groups.This motivates down-weighting saturated rewards and shifting pressure toward lagging, unstable, or stagnant dimensions.
- Diagnostic Evidence: Between steps 35 and 40, scalar reward rises from 1.445 to 1.973 while accuracy drops from 0.258 to 0.090.Format and length rewards rise near saturation, revealing the shortcut profile [0, 1, 1], which quickly dominates rollouts.
4 Adaptive Multi-Reward Projection
AMRP replaces static reward projection with online adaptation driven by reward dynamics, prioritizing dimensions that are lagging, unstable, or stagnant while reducing pressure on saturated dimensions. It estimates these signals from training batches, refreshes normalized weights periodically, and supports both scalar- and dimension-wise policy optimization methods.
- AMRP replaces the fixed projection vector with online weights that adapt at each training step according to reward dynamics.
- Relative shortfall, reward volatility, and recent progress prioritize dimensions that are lagging, unstable, or stagnant, respectively.
- The three signals combine into a priority score that is normalized into mean-one projection weights.
- Weights above 1 amplify a dimension relative to static equal weighting, whereas weights below 1 reduce its relative influence.
- AMRP can incorporate a mean-one non-uniform prior while retaining online adaptation, with the default prior assigning 1 to every dimension.
- Reward statistics are estimated from training batches, and refreshed weights are reused for N training steps to reduce sensitivity to batch-level noise.
- For GRPO and PPO, AMRP scalarizes rewards before advantage estimation; for GDPO, it aggregates normalized per-reward advantages without modifying the underlying policy objective.
5 Experiments
AMRP is evaluated across structured reasoning, grounded generation, and open-ended alignment, with additional tests of reward-update design and compatibility with GRPO, GDPO, and PPO. Across settings, it improves reward-profile trade-offs and task performance by reallocating pressure toward lagging or unstable dimensions and away from saturated ones.
- Structured Reasoning: AMRP achieves the best accuracy on MATH, AMC, and AIME for both base models, including Qwen3-4B-Instruct gains from 17.92 to 31.98 on AIME and 53.92 to 63.59 on AMC.Harder problems often require longer reasoning, so AMRP prioritizes correctness instead of short format-compliant shortcuts.
- Grounded Generation: AMRP improves grounded-generation correctness while maintaining strong citation quality, increasing overall scores from 51.03 to 59.13 for Qwen3-4B-Instruct and from 59.21 to 75.56 for Llama-3.1-8B-Instruct.The method yields the best overall trade-off for both models.
- Open-ended Alignment: AMRP consistently improves both backbones and achieves the best overall score across AlpacaEval, ArenaHard, and MT-Bench, despite less imbalance among correlated learned rewards.The setting offers less headroom because helpfulness, correctness, and coherence largely improve together.
- Cross-setting Analysis: Across all three settings, AMRP produces stronger reward-profile trade-offs by shifting pressure from saturated dimensions to low, unstable, or stagnant ones.This mitigates shortcut-dominated optimization under static scalarization.
- Projection Dynamics: Under static aggregation, format and length rewards saturate while accuracy lags; AMRP increases w_acc and reduces saturated dimensions' weights according to reward dynamics.The projection-level change indicates online reallocation rather than a fixed accuracy bias.
- Shortcut Behavior: AMRP substantially reduces shortcut ratios and maintains longer generations, making short format-compliant responses less attractive during policy optimization.Static aggregation instead produces frequent shortcut profiles accompanied by sharply shortened responses.
- Compatibility with RL Algorithms: AMRP remains effective across GRPO, GDPO, and PPO, improving average accuracy from 37.53 to 60.72, 52.54 to 57.24, and 54.34 to 57.79, respectively.Combining AMRP with GDPO provides further gains, while AMRP achieves the best average accuracy among compared aggregation methods under PPO.
- Ablations: Removing shortfall, volatility, or progress signals reduces average accuracy to 57.0, 56.2, and 57.1, while removing volatility or progress lowers AIME accuracy from 31.9 to 20.6 and 22.8.The full multiplicative design performs best overall, whereas the additive variant preserves slightly higher auxiliary rewards but lowers accuracy.
6 Conclusion
The paper identifies reward-profile collapse as aggregation-induced reward hacking and proposes AMRP, which adapts projection weights online using three reward-dynamics signals. Across three post-training settings, AMRP improves reward-profile trade-offs and downstream performance over static aggregation and DRBO-style dynamic weighting baselines.
- Reward-profile collapse aliases distinct reward profiles under static scalarization and can lock optimization into easy, dense, or fast-improving dimensions.
- AMRP adapts projection weights online using relative shortfall, reward volatility, and recent progress.
- Across structured reasoning, grounded generation, and open-ended alignment, AMRP improves reward-profile trade-offs and downstream performance over static aggregation and DRBO-style dynamic weighting baselines.
Limitations
The evaluation covers diverse reward structures and multiple RL algorithms and reward dimensionalities, but does not capture the full range of modern multi-reward pipelines. Broader evaluation across more conflicting dimensions, larger-scale training, and additional domains remains future work.
- The study evaluates rule-based rewards, automatic evaluators, and learned reward-model scores across three representative multi-reward post-training settings.
- The experiments span multiple RL algorithms and reward dimensionalities but do not capture the full range of modern multi-reward pipelines.
- Future work should examine more numerous or strongly conflicting reward dimensions, larger-scale full-parameter training, and domains including code generation, tool use, and safety-oriented alignment.
Ethical Considerations
AMRP improves reward-profile balance under the studied rewards but does not validate reward quality or guarantee safety or universal alignment. The work therefore retains the need for reward design, safety constraints, human evaluation, and monitoring.
- AMRP mitigates reward imbalance but does not validate rewards, so artifacts in reward models and automatic evaluators may still propagate through optimization.
- Adaptive weighting is not a substitute for careful reward design, safety constraints, human evaluation, or monitoring, especially in user-facing or high-stakes settings.
- The results demonstrate improved reward-profile balance under the studied rewards, not guaranteed safety or universal alignment.
A Additional Experimental Results and Analysis
Additional analyses show that AMRP counteracts reward imbalance in grounded generation and adapts to persistent gaps in open-ended alignment. A qualitative grounded-generation case illustrates how citation rewards can dominate while answer correctness fails.
- Additional Experimental Results and Analysis: Fixed scalar projection can over-emphasize reward dimensions that improve rapidly or saturate earlier, while AMRP increases pressure on lagging, unstable, or stagnant dimensions.
- Additional Experimental Results and Analysis: In grounded generation, citation support improves rapidly under static aggregation, whereas AMRP reallocates weight toward larger gaps for a more balanced correctness–citation profile.
- Additional Experimental Results and Analysis: In open-ended alignment, AMRP adjusts projection according to persistent gaps among helpfulness, correctness, and coherence, improving reward-profile balance.
- Additional Experimental Results and Analysis: A citation-dominant shortcut retrieves and cites relevant evidence but produces a fluent description instead of stating the correct answer, “Rob Davies.”
- Additional Experimental Results and Analysis: Under static aggregation, citation-dominant responses can receive strong citation rewards while correctness remains under-optimized, illustrating aggregation-induced reward hacking.
A.3 Hyperparameter Sensitivity
AMRP remains robust across the tested hyperparameter configurations, substantially outperforming static aggregation and DRBOδ in mathematical reasoning. Frequent weight updates perform best, but performance stays strong across update intervals.
- A.3 Hyperparameter Sensitivity: The sensitivity analysis varies one hyperparameter at a time from the default configuration (λ, ks, kp, N) = (1.0, 5, 5, 1).
- A.3 Hyperparameter Sensitivity: Even the weakest configuration, λ = 0.5, reaches 54.45 average accuracy, 3.30 points above DRBOδ.
- A.3 Hyperparameter Sensitivity: Across update intervals N ∈{1, 2, 4, 8}, average accuracy ranges from 58.33 to 60.72 and remains above both baselines.
- A.3 Hyperparameter Sensitivity: N = 1 achieves the highest overall accuracy, suggesting frequent updates help AMRP respond promptly to reward dynamics.
B.1 Datasets and Evaluation Protocols
The experiments cover mathematical reasoning, citation-grounded long-form QA, and open-ended alignment using distinct datasets, reward profiles, and evaluation protocols. Methods share experimental resources and optimization settings within each setting, differing only in reward-profile projection.
- Mathematical Reasoning: Mathematical reasoning trains on a 10K-prompt NuminaMath-1.5 subset and evaluates on MATH-500, AMC, and AIME 2024.MATH-500 has 500 unique problems; AMC and AIME 2024 have 83 and 30 unique problems, each replicated 32 times.
- Citation-Grounded Long-Form QA: Citation-grounded QA uses ASQA and ELI5 with three retrieved passages and one in-context demonstration per query.Training and evaluation follow ALCE-style grounded generation, with maximum lengths of 400 tokens for ASQA and 500 for ELI5.
- Open-Ended Alignment: Open-ended alignment trains on HelpSteer2 and evaluates held-out prompts from AlpacaEval, ArenaHard, and MT-Bench.ArmoRM attribute scores are reported on a [0, 100] scale rather than official benchmark win rates.
- Reward Scaling: All reward dimensions are oriented so larger values indicate better responses and are scaled to the [0, 1] range.
- Mathematical Reasoning: Mathematical reasoning uses racc, rfmt, and rlen for final-answer accuracy, format compliance, and length control.Responses use greedy decoding with temperature 0 and a maximum completion length of 3,000 tokens.
- Citation-Grounded Long-Form QA: Citation-grounded QA rewards correctness and citation quality through rcorr and rcite.ASQA uses short-answer exact-match coverage, ELI5 uses Claims-NLI, and rcite is based on AutoAIS citation recall and precision with an NLI verifier.
- Shared Experimental Protocol: Within each experimental setting, methods use identical data, prompts, reward functions, rollouts, and optimization hyperparameters.They differ only in how the reward profile is projected into a scalar training signal; AMRP updates projection weights online.