PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weighting

Yining Lu, Zilong Wang, Shiyang Li, Xin Liu, Changlong Yu, Qingyu Yin, Zhan Shi, Zixuan Zhang, Meng Jiang

arXiv:2509.11452v2cs.LGcs.CL

TL;DR

Fixed-weight scalarization is inadequate for non-convex multi-objective trade-offs in online LLM alignment. The paper introduces dynamic reward weighting through hypervolume-guided and gradient-based methods, which consistently outperform fixed-weight baselines with fewer training steps across diverse settings, though gains depend on model capacity to improve objectives simultaneously.

  • Problem

    Fixed weights and heuristic reward interpolation inefficiently handle objectives with different learning difficulty and provably miss non-convex Pareto-front regions in online LLM alignment.

  • Method

    The paper introduces dynamic reward weighting through hypervolume-guided weight adaptation and gradient-based weight optimization.

  • Results

    Across multiple online RL algorithms, datasets, and model families, both methods consistently outperform fixed-weight linear scalarization baselines with superior Pareto solutions and improved training efficiency.

  • Takeaways & Limitations

    Dynamic reward weighting provides a toolkit for multi-objective LLM alignment across varied online RL training settings.

  • Takeaways & Limitations

    Gains are limited when model capacity or pretraining saturation prevents simultaneous improvement of multiple objectives.

Abstract

from arXiv · show

Prior work in multi-objective reinforcement learning typically uses linear reward scalarization with fixed weights, which provably fails to capture non-convex Pareto fronts and thus yields suboptimal results. This limitation becomes especially critical in online preference alignment for large language models. Here, stochastic trajectories generated by parameterized policies create highly non-linear and non-convex mappings from parameters to objectives that no single static weighting scheme can find optimal trade-offs. We address this limitation by introducing dynamic reward weighting, which adaptively adjusts reward weights during the online reinforcement learning process. Unlike existing approaches that rely on fixed-weight interpolation, our dynamic weighting continuously balances and prioritizes objectives in training, facilitating effective exploration of Pareto fronts in objective space. We introduce two approaches of increasing sophistication and generalizability: hypervolume-guided weight adaptation and gradient-based weight optimization, offering a versatile toolkit for online multi-objective alignment. Our extensive experiments demonstrate their compatibility with commonly used online reinforcement learning algorithms, effectiveness across multiple datasets, and applicability to different model families, consistently achieving Pareto dominant solutions with fewer training steps than fixed-weight linear scalarization baselines.

1 Introduction

Online LLM reinforcement learning often optimizes accuracy while neglecting response length and clarity, and fixed weighting cannot reliably explore non-convex trade-offs. The paper introduces dynamic reward weighting through hypervolume-guided adaptation and gradient-based optimization, with experiments showing broader and more efficient Pareto optimization.

  • Motivation: Fixed-weight scalarization can miss non-convex Pareto-front regions, while differently difficult objectives may receive inefficiently allocated learning effort.Objectives that saturate early continue receiving equal gradient updates during later training.
  • Approach: Dynamic reward weighting adaptively rebalances and reprioritizes objectives during online reinforcement learning based on their learning progress.The approach redirects learning effort toward objectives with greater potential for improvement.
  • Results: Figure 1 shows gradient-based weighting producing Pareto fronts that dominate three fixed-weight baselines across accuracy, conciseness, and clarity.The comparison uses Qwen3-8B on Math500 with GRPO and includes accuracy-focused, balanced, and efficiency-focused configurations.
  • Approach: The toolkit includes hypervolume-guided weight adaptation for preference-informed training and gradient-based weight optimization for more flexible multi-objective alignment.The hypervolume method rewards checkpoints with positive hypervolume contributions, while the gradient-based method provides a progressively more flexible alternative.
  • Results: Across online RL algorithms, datasets, and model families, the methods achieve superior Pareto fronts with fewer training steps than static baselines.Experiments cover GRPO, REINFORCE, and RLOO; Math500, MATH, and SafeSQL; and Qwen3, Deepseek, Mistral, and Llama3.

2 Related Works

Related work studies steerable multi-objective policies and multi-capability preference alignment, but commonly relies on static reward combinations. Existing methods lack a unified toolkit for arbitrary objective sets, and prior LLM alignment evaluations remain limited in breadth.

  • 2.1 Steerable Multi-Objective Preference Finetuning: Steerable multi-objective methods support inference-time navigation across reward trade-offs but typically use fixed-weight scalarization or fixed-weight policy mixing.These approaches include off-policy RL, post-hoc policy mixing, and conditional training with preference weights.
  • 2.1 Steerable Multi-Objective Preference Finetuning: Linear scalarization cannot capture non-convex Pareto-front regions, resulting in suboptimal policies despite inference-time flexibility.Geometrically, fixed weights sweep objective space with a hyperplane whose normal vector is the weight vector.
  • 2.2 General Multi-Objective Preference Finetuning: General multi-objective preference alignment trains multiple capabilities in one run, but existing LLM methods still use static scalarization or human-defined interpolation rules.This limits their generalizability to new objectives.
  • 2.2 General Multi-Objective Preference Finetuning: MGDA-based methods combine objective gradients, yet prior work mainly targets simple supervised tasks and classical two-objective problems rather than LLM alignment.The paper identifies no unified toolkit for automatically guiding LLM training across arbitrary objective sets.
  • 2.2 General Multi-Objective Preference Finetuning: PAMA had been evaluated only on two-objective general preference alignment with Noon PPO, leaving its generalizability to complex reasoning tasks and other RL algorithms unclear.The cited evaluation used positive sentiment and longer responses on IMDb.

3 Background

The background formalizes weighted multi-objective reinforcement learning and defines Pareto optimality and hypervolume-based coverage. These concepts distinguish non-dominated trade-offs from dominated solutions and quantify objective-space coverage relative to a reference point.

  • 3.1 Notations: The framework uses K objectives, weights w = (w1, w2, . . . , wK), rewards r = (r1, r2, . . . , rK), policy πθ, and objective function J(θ).Superscript w denotes weighted quantities, while superscript i denotes single-objective quantities.
  • 3.2 Pareto Front and Hypervolume Contribution: A Pareto-optimal solution is one for which no feasible alternative improves every objective without leaving at least one objective unchanged and another changed.The Pareto front contains the corresponding non-dominated objective vectors.
  • 3.2 Pareto Front and Hypervolume Contribution: The hypervolume indicator measures the K-dimensional region dominated by objective vectors in A and bounded below by a reference point r.The reference point must be dominated by every vector in A, and higher hypervolume indicates broader objective-space coverage.
  • 3.2 Pareto Front and Hypervolume Contribution: Hypervolume contribution is the non-negative change in hypervolume caused by adding a point to a set.This quantity can identify whether a newly evaluated solution expands the covered objective-space region.

4 Hypervolume-Guided Weight Adaptation

Hypervolume-guided weight adaptation uses a meta-reward based on Pareto-front improvement to amplify training rewards when new non-dominated solutions emerge. Experiments across algorithms, datasets, and models show stronger Pareto fronts than fixed-weight and PAMA baselines, though meta-reward calibration affects convergence.

  • Method: The method combines human-specified scalar rewards with a meta-level reward that amplifies training when a policy discovers new Pareto fronts.It does not directly modify the specified weights; instead, hypervolume contributions provide the additional training signal.
  • Experiment Setup: Experiments span REINFORCE, RLOO, and GRPO; Math500, MATH, and SafeSQL; and Qwen3, Deepseek, Mistral, and Llama3 model families.The main experiments use Qwen3-8B on Math500, with additional mathematical and coding reasoning datasets and models.
  • Results: Table 1 reports that hypervolume-guided weighting outperforms fixed-weight baselines and PAMA across most objectives, configurations, and online RL algorithms, with some fully dominant Pareto fronts.Each table value averages performance across the entire Pareto front; full dominance means superior results on all three objectives.
  • Method: The hypervolume-guided procedure evaluates validation performance, maintains a Pareto set, scales rewards by rpareto, and updates the policy with the chosen online RL objective.Positive hypervolume contributions add new validation outcomes to the Pareto set and update the meta-weight.
  • Results: REINFORCE with hypervolume-guided weighting achieves better average scores across all three objectives under both accuracy- and efficiency-focused settings.The accompanying Pareto-front visualizations indicate domination of baselines in both settings, while average scores alone do not guarantee true Pareto optimality.
  • Results: Balanced weights produce higher meta-rewards and more aggressive exploration, but this can be suboptimal, making careful meta-reward calibration important for convergence.The comparison uses distributions of rpareto values and their averages.

5 Gradient-Based Weight Optimization

Gradient-based weight optimization dynamically updates reward weights from objective-gradient influence during online RL, with safeguards against weight collapse or explosion. Across training setups, it produces superior Pareto fronts relative to PAMA and fixed-weight baselines.

  • Gradient-based weight optimization: Weights are updated from each objective’s influence on the remaining objectives, increasing emphasis on objectives with high cross-objective influence and gradient magnitude.The update uses an influence signal derived from inner products between objective gradients, with learning rate η and regularization factor µ.
  • Algorithm 2 Online Reward Weights Optimization: The online procedure computes per-objective gradients, derives an influence signal, updates reward weights, and then trains policy parameters with scalarized rewards.Algorithm 2 repeats this process over mini-batches and sampled answers for T training steps.
  • Convergence analysis: Under Lipschitz, bounded-reward, learning-rate, and bounded-policy-gradient assumptions, objective weight ratios remain uniformly bounded.The bounded-ratio result supports numerical stability and prevents exponential updates from collapsing or exploding weights.
  • 5.3 Results: Gradient-based weighting yields superior Pareto fronts to PAMA and fixed-weight baselines across GRPO and REINFORCE, while improving conciseness and clarity under RLOO.Table 2 reports averages across entire Pareto fronts; the method also dominates baselines in corresponding GRPO and REINFORCE visualizations.
  • Implementation: The implementation initializes weights uniformly, uses a convergent polynomial learning-rate schedule, and omits token-level KL penalties to preserve reward-gradient linearity.These choices follow the convergence analysis and maintain the linear relation between overall and per-objective rewards.
  • 5.3 Results: In training, conciseness rapidly converges near 0.2 while its redistributed weight primarily shifts to accuracy, whose weight continues growing.The authors associate this pattern with faster conciseness improvement and greater ongoing learning difficulty for accuracy; accuracy and clarity also show intertwined optimization behavior.

6 Generalizability and Convergence Rate Tests

The gradient-based method reaches Pareto fronts more efficiently than its baseline, while generalization is tested across datasets and model families. Some accuracy–clarity projections collapse because clarity saturates, leaving other objective pairs to show meaningful trade-offs.

  • Convergence Rate Tests: Reducing required training steps by 6.1 on average, the gradient-based method has a higher convergence rate than baselines.The hypervolume-guided method provides only marginal efficiency gains compared with baselines.
  • Generalizability Tests: Experiments extend both methods across additional mathematical and coding datasets and three model families under matched learning-rate conditions.SafeSQL is used for coding tasks, while mathematical datasets share identical objective sets.
  • Generalizability Tests: Accuracy–clarity Pareto fronts can collapse to a single point when clarity saturates near its maximum, so accuracy–response-length plots capture more meaningful trade-offs.This occurs in examples involving Mistral-7B and Llama3-8B.

7 Conclusion and Future Work

The paper concludes that dynamic reward weighting improves multi-objective alignment across varied algorithms, datasets, and model families. Its gains are limited when model capacity cannot support simultaneous improvement of conflicting objectives.

  • Conclusion: Dynamic reward weighting consistently outperforms fixed-weight linear scalarization, producing superior Pareto-optimal solutions with improved training efficiency.The conclusion covers experiments across multiple online RL algorithms, datasets, and model families.
  • Future Work: When reducing response length degrades accuracy, dynamic weighting yields limited gains because the model cannot effectively improve both objectives.The authors associate this boundary with performance saturation or insufficient model capacity, which post-training alone may not resolve.

A.1 Proof for Gradient-Based Weight Optimization

The gradient-based method formulates reward-weight learning as bilevel optimization and derives a normalized update using a greedy approximation, Taylor expansion, and entropy regularization.

  • Bilevel Formulation: Reward weights w are optimized as a higher-level problem while policy parameters θ are updated at the lower level.The higher-level objective uses the loss of the updated policy model.
  • Greedy Approximation: At training step t, the method searches for weights w(t) that minimize the average loss across K objectives at step t + 1.This uses a greedy approximation based on the objectives evaluated after the policy update.
  • Approximation: A first-order Taylor approximation rewrites the next-step objective while treating the higher-order remainder as O(∥w∥).The approximation connects policy changes to the reward-weight update.
  • Regularization and Update: The update is regularized with factor µ through Bregman divergence using the entropy function h(w) = P_i w_i ln w_i, followed by differentiation.Normalization is included in the resulting reward-weight update rule.

A.2 Preliminary Findings

Preliminary analyses show that objectives differ in convergence rates and in how their optimization trajectories interact. Conciseness follows a distinct direction from accuracy and clarity, while individual objectives saturate at different stages.

  • Objective Interactions: Conciseness-only training diverges from accuracy- and clarity-focused models, indicating an orthogonal influence on parameter updates.The divergence is evidenced by steadily increasing pairwise KL divergence over time.
  • Convergence Rates: Different objectives reach saturation at different training stages, with conciseness and clarity attaining their best outcomes around steps 165 and 240, respectively.These examples illustrate why objectives may require different learning effort during training.

A.3 hypervolume-Guided Reward Adaptation Details

The appendix details reward construction, training reproducibility, and robustness considerations for the two dynamic weighting approaches. It also reports Pareto-front evaluations under REINFORCE and multiple RL training algorithms.

  • Reward Construction: Accuracy, conciseness, and clarity rewards respectively measure exact-match correctness, response length relative to prior rollouts, and explicit reasoning-step formatting.The clarity reward uses predefined rules checking terms such as “first,” “second,” and “third.”
  • Robustness Caveat: REINFORCE with a constant learning rate showed substantial run-to-run convergence variation for conciseness and clarity despite identical hyperparameters.All methods and baselines were repeated three times, with the fastest-converging run reported; this issue was not observed in other settings.
  • Hypervolume Reward Stabilization: A minimum rpareto = 0.5 threshold prevents zero hypervolume reward from increasing response length and degrading accuracy and formatting.The authors attribute the zero-reward behavior to the entropy penalty dominating gradient updates.
  • Training Configuration: The experiments use verl for reinforcement-learning training and provide GRPO-on-Math500 hyperparameters for both hypervolume adaptation and gradient-based optimization.Reproducibility information is organized in Tables 4 and 5.
  • Evaluation Scope: Pareto fronts for hypervolume-guided adaptation are evaluated against baselines under REINFORCE, while gradient-based optimization is evaluated under different RL training algorithms.Figures 9 and 10 define the respective evaluation scopes but do not state numerical winners in the supplied passages.
Loading 2509.11452v2…