DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning

Guochao Jiang; Jingyi Song; Guofeng Quan; Chuzhan Hao; Guohua Liu; Yuewei Zhang

Source-linked AI summary

DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning

Guochao Jiang, Jingyi Song, Guofeng Quan, Chuzhan Hao, Guohua Liu, Yuewei Zhang

arXiv:2605.25604v1May 25, 2026cs.CLcs.LG

TL;DR

Multi-reward GRPO scalarization struggles with training instability and objective isolation. DVAO dynamically weights objectives by rollout-group reward variance, and experiments on math reasoning and tool-use show superior multi-objective trade-offs with robust stability.

Problem
Standard reward and advantage combination methods for multi-reward GRPO face advantage-magnitude instability, static weighting, and failure to capture cross-objective correlations.
Method
DVAO dynamically adjusts objective-combination weights using each objective’s empirical reward variance within a rollout group.
Results
Across mathematical reasoning and tool-use tasks and model scales, DVAO achieves the strongest simultaneous accuracy and length or format compliance among evaluated methods.
Takeaways & Limitations
DVAO provides a variance-adaptive approach that balances multiple objectives while maintaining bounded advantages and cross-objective regularization.
Takeaways & Limitations
The paper identifies fixed convex-combination coefficients as limiting dynamic adjustment of training intensity and explicit modeling of reward correlations.

Abstract

from arXiv · show

Reinforcement Learning has become a standard paradigm for aligning Large Language Models with human intent and task requirements. While Group Relative Policy Optimization offers an efficient, value-model-free alternative to Proximal Policy Optimization, adapting it to real-world multi-reward settings remains challenging. Standard scalarization practices, such as Reward Combination and Advantage Combination, suffer from significant drawbacks: Reward Combination frequently generates advantages with excessively large squared magnitudes that lead to training instability, while Advantage Combination relies on static hyperparameters and ignores cross-objective correlations. To address these limitations, we propose Dynamic Variance-adaptive Advantage Optimization (DVAO), which dynamically adjusts combination weights based on the empirical reward variance of each objective within a rollout group, effectively up-weighting objectives with a stronger learning signal while suppressing noisy ones. We mathematically prove that DVAO maintains bounded advantage magnitudes for stable training and introduces a self-adaptive cross-objective regularization mechanism. Extensive experiments on mathematical reasoning and tool-use benchmarks using Qwen3 and Qwen2.5 models demonstrate that DVAO significantly outperforms baseline methods, achieving a superior multi-objective Pareto frontier and robust training stability.

1 Introduction

The introduction frames multi-reward GRPO as necessary for real-world LLM deployment but difficult to scalarize reliably. It presents DVAO as a data-driven, variance-adaptive weighting scheme designed to improve stability and cross-objective synergy.

Real-world LLM applications require optimizing multiple objectives, including accuracy, length constraints, code bug rates, hallucination rates, and tool-calling format.
GRPO adaptation commonly uses Reward Combination or Advantage Combination, but both methods have theoretical and practical drawbacks.Reward Combination linearly combines raw rewards, whereas Advantage Combination normalizes rewards before combining their advantages.
Reward Combination can produce excessively large squared advantage magnitudes, causing erratic policy gradients and training instability.
DVAO dynamically weights objectives by empirical rollout-group reward variance, emphasizing stronger learning signals while suppressing noisy, low-variance objectives.The method is fully data-driven and hyperparameter-free, with bounded advantage magnitudes and implicit cross-objective regularization.

2 Preliminaries

GRPO and related variants simplify policy optimization by removing the value model and using group-relative advantages. In multi-reward settings, objectives can be combined at the reward or advantage level, with GDPO additionally applying batch-wise advantage normalization for stability.

GRPO: GRPO and variants such as DAPO and GSPO are widely used because they simplify policy optimization by eliminating the value model and using relative advantages within rollout groups.GRPO samples multiple rollouts for each input to compute relative advantages.
Multi-reward setting: Real-world tasks extend single-reward GRPO to multiple objectives, including accuracy, output length, code bug rate, hallucination rate, and correct tool-use function calls.The preliminaries define n independent reward functions for these objectives.
Reward Combination: Reward Combination forms the optimization reward by taking a convex combination of reward components, with each objective assigned a weight hyperparameter.Each reward function maps an input-output pair to a value in [0, 1].
Advantage Combination: Advantage Combination methods such as GDPO independently normalize each objective’s reward into an advantage before combining the resulting advantages with weight hyperparameters.The combined advantage is then used for policy optimization across multiple objectives.
Advantage Combination: GDPO further applies batch-wise advantage normalization to maintain training stability during multi-objective policy optimization.This normalization is applied after forming the combined advantage.

3 Method

Section 3 identifies instability and coordination limitations in standard reward and advantage combination, then introduces DVAO as a dynamic, variance-adaptive alternative. DVAO adjusts objective weights using rollout-group reward variance and couples each objective’s learning signal to overall multi-objective performance.

Limitations of Existing Methods: Reward combination produces larger squared advantage magnitudes and policy gradients, whereas advantage combination reduces magnitude but ignores cross-objective correlations.Larger advantage magnitudes can cause training instability and hinder convergence.
Limitations of Existing Methods: Advantage combination also uses fixed convex-combination weights, making it difficult to adjust training intensity across objectives during dynamic training.Its gradient contribution for each objective is scaled by that objective’s isolated advantage rather than cross-objective information.
DVAO: DVAO replaces fixed weights with dynamic variance-adaptive weights that up-weight higher-variance objectives and down-weight lower-variance objectives using rollout-group statistics.The weighting is described as fully dynamic and data-driven, with corresponding group standard deviations determining adaptation.
DVAO: DVAO’s gradient sensitivity depends on cross-objective interaction and overall multi-objective performance, creating an implicit variance-aware regularization mechanism.This context-aware scaling is intended to prevent greedy over-optimization of one easy objective and promote synergistic alignment toward a multi-objective Pareto-optimal policy.

4 Experiments

Experiments on mathematical reasoning and tool-use tasks show that DVAO delivers the strongest joint accuracy and length/format compliance across model scales. Training dynamics and Pareto-frontier analyses further indicate improved variance stability and superior accuracy–compliance trade-offs over fixed-weight baselines.

Experimental setup: Experiments evaluate DVAO on mathematical reasoning benchmarks and BFCL-v4 tool-use tasks, using Qwen3 base and Qwen2.5 instruct models.Mathematical reasoning objectives are accuracy and length constraint; tool-use evaluation follows ToolRL and GDPO on BFCL-v4.
Main results: DVAO achieves the highest average accuracy with near-perfect length or format compliance, whereas every baseline sacrifices one dimension for the other.On mathematical reasoning, RC and AC trade accuracy for length compliance, while GDPO has near-perfect length compliance but the lowest accuracy; on tool-use, DVAO leads both dimensions.
Training dynamics: Across Qwen3-4B-Base and Qwen3-8B-Base, DVAO maintains the highest accuracy reward while suppressing reward variance most effectively.Its accuracy-reward curve remains above all baselines throughout training, and its standard deviation declines more sharply.
Training dynamics: DVAO drives length reward closest to the target value of 1.0 and produces the most dramatic variance collapse on both model scales.Its adaptive normalization couples accuracy and length objectives, preventing either from overwhelming the combined advantage signal.
Training dynamics: DVAO reaches the highest final response length on both model scales, although its trajectories oscillate more visibly than those of RC and AC.The oscillation reflects more aggressive length-reward optimization and exploration of longer responses.
Pareto analysis: DVAO dominates the accuracy–length/format Pareto frontier across both tasks, while RC saturates, AC is unstable, and GDPO fluctuates incoherently.The analysis sweeps w1 across {0.1, 0.3, 0.5, 0.7, 0.9} with w2 = 1 − w1 and attributes baseline failures to uncontrolled advantage scaling.

5 Related Work

Prior work has streamlined LLM post-training through GRPO and developed reasoning models such as DeepSeek-R1, while multi-reward reinforcement learning addresses conflicting objectives in practical deployments.

Advancements in GRPO and Reasoning Models: GRPO eliminates the need for a heavily parameterized value model, streamlining the LLM post-training pipeline.This efficiency has supported the development of state-of-the-art reasoning models such as DeepSeek-R1.
Advancements in GRPO and Reasoning Models: GRPO’s efficiency has contributed to the development of reasoning models including DeepSeek-R1.
Multi-Reward Reinforcement Learning in LLMs: Multi-reward reinforcement learning integrates conflicting signals to balance human preferences, length efficiency, and strict formatting in agentic tool-use.These objectives reflect practical deployment requirements for large language models.

6 Conclusion

The paper identifies theoretical and practical limitations in Reward Combination and Advantage Combination for multi-reward GRPO and introduces DVAO to address them. DVAO dynamically adjusts combination weights using each objective’s empirical variance within a rollout group.

6 Conclusion: The work identifies magnitude explosion and objective isolation as fundamental limitations of standard scalarization techniques for multi-reward GRPO.
6 Conclusion: DVAO addresses these limitations through dynamically adjusted combination weights based on each objective’s empirical variance within a rollout group.
6 Conclusion: The proposed method explicitly up-weights learning signals from high-variance objectives.

A Proof of Proposition 1

The proof establishes the stated relationship between reward combination and advantage combination by expanding their definitions for a fixed query and rollout group. Equality holds if and only if all pairwise sample correlations between objectives equal 1.

Equality condition: Equality holds if and only if ρ̂_kl = 1 for all k ≠ l.Thus, equality requires perfect sample correlation between every pair of objectives within the group rollout.
Reward combination: The reward combination result follows directly from the normalization definition applied to the objective rewards.The argument derives the reward-combination expression by normalization of r^(i,j).
Proof setup: For a fixed query x_i, the proof analyzes objective advantages A_k within the rollout group using their definitions.The proof begins by fixing query x_i and relating the objective advantages to group-level quantities.
Advantage combination: The advantage combination result is obtained analogously from the corresponding definition.The proof separately applies the advantage-combination definition to the rollout-group quantities.

B Proof of Proposition 2

Proposition 2 proves that, for a fixed query and rollout group, reward combination produces a pointwise larger advantage magnitude than DVAO. The proof derives this result from a variance identity and the Cauchy–Schwarz inequality, with equality characterized by perfect positive correlation among reward pairs.

Proposition 2: Proposition 2 states that reward combination produces a pointwise larger advantage magnitude than DVAO for a fixed query and rollout group.The comparison is made pointwise over the rollout group.
Equality condition: Equality holds if and only if all reward pairs are perfectly positively correlated within the rollout group.The final equality follows from the definition of A^(i,j).
Proof: The proof begins by establishing a key identity from the definitions of the reward combination method.It fixes the query and rollout group before deriving the identity.
Proof: Expanding the variance of the combined reward and applying the Cauchy-Schwarz inequality bounds the covariance terms used in the comparison.The proof then takes absolute values and divides by σ_i to connect the identity to the advantage expression.

C Proof of Proposition 3

This section proves Proposition 3 by deriving the sensitivity of combined advantages to an individual objective’s raw reward. It shows that advantage combination depends on the isolated objective advantage, whereas DVAO incorporates cross-objective information through adaptive weighting.

Proposition statement: Proposition 3 compares the derivatives of advantage combination and DVAO with respect to the k-th raw reward r(i,j).The comparison is formulated for a fixed query x_i and rollout group {y_j}_G.
Advantage combination: Advantage combination’s sensitivity depends strictly on the isolated advantage of the k-th objective.The proof obtains this result from standard group normalization and the sample-mean properties.
DVAO: DVAO’s sensitivity depends adaptively on a cross-term involving the combined advantage, aggregating performance information across objectives within the rollout group.The derivation applies the quotient rule and substitutes the definition of the adaptive weight w_k.
Proof completion: Applying the chain rule to independently normalized objectives and completing the quotient-rule derivation establishes the stated comparison.The final substitution completes the proof.

D Implementation Details

The experiments use separate datasets for mathematical reasoning and tool use, with equal reward coefficients by default. Training is implemented in verl with AdamW, a constant learning rate, grouped rollouts, and a fixed 500-step schedule.

Training Data: 17k prompts from DAPO-MATH-17K5 support mathematical reasoning, with each prompt paired with an integer answer.The passage describes DAPO-MATH-17K5 as the mathematical reasoning training dataset.
Training Data: 4k ToolRL samples support tool use: 2k from ToolACE, 1k from Hammer, and 1k from xLAM, each containing a question and ground-truth tool calls.The cited datasets are listed with their respective sample counts and sources.
Optimization Setup: Unless otherwise specified, coefficients corresponding to all rewards are equal, satisfying Σ_k w_k = 1.The implementation passage states both the normalization condition and the default equal-coefficient setting.
Optimization Setup: 1×10^-6 is the constant AdamW learning rate, with prompt batch size 128, G = 16 sampled responses per prompt, and 500 training steps.DVAO experiments are conducted in the verl framework, and the 500-step schedule is intended to ensure convergence.

E Limitations and Future Work

DVAO’s dynamic weighting depends on accurate empirical variance estimation within rollout groups. Experiments found G = 16 robust, while hardware-constrained settings with G ≤4 mark an important boundary for extremely large models.

Limitations: DVAO’s dynamic weighting relies on empirical variance estimation within each rollout group.The passage identifies this dependence as a limitation of the method.
Limitations: G = 16 provided highly robust variance signals in the experiments.This was the standard group size used in the reported experiments.
Limitations: Extremely large models may face hardware memory constraints forcing very small groups, such as G ≤4.The passage flags this small-group regime as a boundary for DVAO’s variance estimation.