Uncovering Cross-Objective Interference in Multi-Objective Alignment

Yining Lu; Meng Jiang

Source-linked AI summary

Uncovering Cross-Objective Interference in Multi-Objective Alignment

Yining Lu, Meng Jiang

arXiv:2602.06869v2Feb 6, 2026cs.CLcs.LG

TL;DR

Multi-objective LLM alignment can improve some objectives while degrading others, a model-dependent failure the paper formalizes as cross-objective interference. It systematically studies scalarization, derives local covariance and global PL-based conditions, and introduces CTWA, which more effectively mitigates interference and yields superior Pareto-optimal solutions.

Problem
Multi-objective alignment frequently improves easier objectives while degrading others, and the paper addresses the limited systematic understanding of this cross-objective interference.
Method
The paper evaluates scalarization algorithms, derives a covariance law extended to clipped surrogates, proposes CTWA, and analyzes global convergence under the PL condition.
Results
The study finds interference across evaluated methods on certain models, while CTWA mitigates it more effectively than existing baselines and yields superior Pareto-optimal solutions.
Takeaways & Limitations
Cross-objective interference reflects both covariance misalignment in training and unfavorable model geometry, providing actionable insights for multi-objective alignment.
Takeaways & Limitations
The global convergence analysis relies on assumptions including bounded rewards, a unique optimal completion with positive margin, non-saturated suboptimal policies, and model-dependent gradient alignment.

Abstract

from arXiv · show

We study a persistent failure mode in multi-objective alignment for large language models (LLMs): training improves performance on only a subset of objectives while causing others to degrade. We formalize this phenomenon as cross-objective interference and conduct the first systematic study across scalarization algorithms, showing that interference is pervasive and exhibits strong model dependence. To explain this phenomenon, we derive a local covariance law showing that an objective improves when its reward exhibits positive covariance with the scalarized score. We extend this analysis to clipped surrogate objectives used in modern alignment, demonstrating that the covariance law remains valid under mild conditions despite clipping. Building on this analysis, we propose Covariance Targeted Weight Adaptation (CTWA), a plug-and-play method that maintains positive covariance between objective rewards and the training signal to effectively mitigate cross-objective interference. Finally, we complement these local improvement conditions with a global convergence analysis under the Polyak--Łojasiewicz condition, establishing when non-convex scalarized optimization achieves global convergence and how cross-objective interference depends on specific model geometric properties.

1 Introduction

Multi-objective LLM alignment often improves easier objectives while degrading others, and this interference varies by model. The paper systematically evaluates scalarization methods, explains the failure through covariance and optimization geometry, and proposes CTWA to mitigate it.

Scalarized training frequently improves a subset of objectives while degrading others, a failure mode formalized as cross-objective interference.
The first systematic evaluation finds interference across scalarization algorithms and models, including cases where objectives are not traditionally gradient-conflicting.The issue also appears in larger models across model families and datasets.
An objective improves locally when its reward has positive covariance with the scalarized score; easy objectives can otherwise induce negative covariance for harder ones.
The covariance law extends to clipped surrogate objectives under mild conditions, motivating CTWA's adaptive weighting to maintain positive covariance for every objective.
A PL-based global analysis identifies model-dependent causes of interference, including insufficient optimal-trajectory probability, weak reward margins, and ill-conditioned gradient geometry.

2 Related Work

Related work approaches multi-objective learning through scalarization, gradient aggregation, adaptive weighting, and gradient modification. Existing guarantees and methods were largely developed for simplified or supervised settings, leaving LLM alignment's interference mechanisms insufficiently characterized.

MTL addresses joint optimization of multiple losses, with negative transfer linked to conflicting gradients and imbalanced loss scales.
Classic MOO scalarizes multiple objectives, but its convergence guarantees commonly rely on convexity or determinism assumptions that do not transfer cleanly to LLM alignment.
Prior multi-objective RL work targets KL or safety constraints, whereas this paper identifies covariance misalignment as a distinct source of interference.
Existing multi-objective alignment methods use static or dynamic reward weighting, Lagrangian relaxation, and Tchebycheff scalarization to combine rewards.
LLM RFT also faces instability, vanishing gradients, sensitivity to weighting and normalization, exploration difficulty, and model-size-dependent optimization performance.

3 Preliminaries

The paper formulates multi-objective RFT as contextual-bandit optimization over autoregressive completions. Vector rewards are mapped to a scalar score, whose expected value defines the scalarized training objective.

The preliminaries define prompts as fixed-length token sequences and completions as output sequences over a finite vocabulary.
Each completion receives an M-dimensional reward vector, representing the multiple objectives for a given prompt.
RFT is modeled as a horizon-one contextual bandit in which each input is a state and each generated completion is an action.
An autoregressive language model induces a probability distribution over completions through token-level logits and next-token probabilities.
A scalarization map converts the vector reward into a scalar score, and the induced expected value generalizes single-objective RFT.

4 Local Covariance Laws for Multi-Objective Policy Improvement

The section derives first-order conditions for when scalarized policy improvement helps or harms an individual objective. Positive reward–score covariance supports improvement, while negative covariance produces cross-objective interference, including under clipped surrogates.

KL-Regularized Policy Improvement: The local covariance law states that optimizing a scalar score improves objective m at first order when Cov(r_m, s) is positive.Negative covariance creates a local tradeoff in which increasing the scalar score decreases the objective.
Local Covariance Law: On-policy covariance can change sign during training, so an objective may improve early and degrade later even without global Pareto conflict.For linear scalarization, emphasizing an easy objective can flip covariance negative for a harder objective when rewards are weakly or negatively correlated on-policy.
Two-Mode Toy Example: In the two-mode example, KL-regularized improvement increases probability of the mode favored by the scalarized score.If the scalar score favors the objective-m worse mode, its probability approaches one.
Two-Mode Toy Example: When s_bad > s_good while r_good > r_bad, the expected objective decreases monotonically as training shifts probability toward the bad mode.The conditional covariance is negative in this interference configuration, predicting a first-order decrease for sufficiently small stepsizes.
Clipped Surrogate Objectives: The covariance analysis extends to clipped GRPO and PPO-style surrogates under mild conditions because clipping removes only a sufficiently controlled subset of weighted gradient terms.Learning-rate scheduling and reward normalization can keep clipping distortion small enough for the covariance law to remain valid.

5 Covariance Targeted Weight Adaptation

CTWA adapts scalarization weights using reward–training-signal covariance, increasing attention to objectives whose covariance falls below target. Experiments show improved robustness across models, algorithms, and covariance-target settings.

Method: CTWA is a plug-and-play controller that adapts scalarization weights to maintain sufficiently large covariance between each objective reward and its clipped advantage weight.The method monitors covariance under PPO-style updates and adjusts weights for the next policy update.
Method: CTWA uses an EMA covariance signal, increases a weight when covariance falls below threshold, and performs stable multiplicative updates through log-space parameterization.The update preserves λ_m > 0 while responding to covariance deficits.
Experiments: Experiments cover Math500, three objectives, multiple Qwen models, clipped GRPO, and SmolLM2-1.7B, with equal initial objective weights.The objectives are accuracy, conciseness, and clarity, evaluated with verifiable heuristic rewards.
Experiments: Accuracy receives exponentially faster-growing weights than conciseness and clarity, while interference baselines show lower covariance than CTWA across all objectives.The results suggest accuracy is harder to optimize and that covariance tracks alignment performance.
Sensitivity Analysis: Positive covariance targets consistently outperform negative targets, while CTWA remains robust across a broad positive range but declines at an excessively large target such as 0.20.Strong results use positive asymmetric settings including [0.12, 0.08, 0.08] and [0.15, 0.08, 0.08].
Efficiency: CTWA adds negligible computational overhead because covariance components are computed alongside standard reinforcement-finetuning updates.Unlike dynamic weighting or MGDA, it avoids per-objective gradients and projected gradient descent at each step.

6 Global Convergence of Multi-Objective Alignment via µ-PL Condition

The section analyzes global convergence of non-convex scalarized alignment through a µ-PL condition. It identifies model-geometry factors that determine convergence while separating convergence of the scalarized objective from interference on individual objectives.

Motivation: Classical convex analysis is unsuitable for the non-convex scalarized value function and autoregressive policy parameterization, motivating a µ-PL analysis.The PL framework accommodates non-convex objectives and provides sufficient conditions for global convergence.
Assumptions: A µ-PL condition follows under bounded scores, a unique optimal completion with a score margin, non-saturated suboptimal policies, and aligned token gradients.The aligned-gradient assumption requires token-level gradient contributions along favorable trajectories to reinforce rather than cancel.
Global Convergence: The resulting inequality links the squared scalarized-value gradient to the gap from the optimal value, establishing a sufficient global-convergence condition.The condition is expressed as 1/2∥∇_θV(x; θ)∥^2 ≥ µ(V(x; θ*) − V(x; θ)).
Model Dependence: The constant µ is larger when optimal-completion probability, optimal scalarized score, and logit-map conditioning are favorable.Its closed form incorporates Jacobian and non-saturation properties through γ and singular-value terms.
Model Dependence: A small or non-positive µ can result from weak aligned signal or a highly skewed logit map, whereas positive µ alone does not prevent cross-objective interference.Scalarized value can increase while an individual reward decreases when that objective’s covariance condition fails.

7 Conclusion

The paper formalizes cross-objective interference in multi-objective LLM alignment and proposes CTWA to mitigate it more effectively than existing baselines.

The study formalizes cross-objective interference as a common failure mode in multi-objective LLM alignment.
CTWA mitigates cross-objective interference more effectively than existing baselines and yields superior Pareto-optimal solutions.

A Experiment Results

Experiments show that cross-objective interference persists across model scales, datasets, scalarization methods, and reinforcement-learning algorithms, while CTWA improves balance across objectives.

Gradient cosine similarities remain mostly non-negative and converge toward 0, indicating weak coupling without observed gradient conflict.
Cross-objective interference appears in larger models as well as 1.5B and 1.7B models, across five models, two datasets, and two scalarization methods.
CTWA steadily improves all objectives under REINFORCE and yields the most Pareto-efficient results overall.
Under GRPO, CTWA achieves balanced accuracy, conciseness, and clarity, whereas baselines sacrifice or fail to improve some objectives.

B Limitations and Future Work

The paper limits its baseline coverage and identifies a computational barrier to turning its model-dependent geometric analysis into a practical training strategy.

The baseline suite is representative rather than exhaustive, leaving methods such as SDMGrad and FairGrad for future evaluation.
Applying Theorem 6.4 directly is challenging because its token-level Jacobian terms are prohibitively expensive for modern language models.

C Formal Analysis for Clipped Surrogate Objectives

This appendix develops sufficient first-order conditions for objective improvement under clipped surrogate objectives, using GRPO as a running example and extending the logic to PPO-style clipping.

C Formal Analysis for Clipped Surrogate Objectives: The analysis establishes lemmas and theorems for sufficient conditions ensuring multi-objective improvement under clipped surrogate objectives.
C Formal Analysis for Clipped Surrogate Objectives: GRPO samples completion groups, normalizes rewards into advantages, and uses KL and entropy regularization alongside clipped importance ratios.
C Formal Analysis for Clipped Surrogate Objectives: The clipped surrogate uses an indicator that retains gradient terms only on the active branch determined by advantage sign and the clipping window.
C Formal Analysis for Clipped Surrogate Objectives: The method forms clipped advantage weights from tokenwise logit-gradient features and the surrogate's advantage signal.
C Formal Analysis for Clipped Surrogate Objectives: Under Lipschitz-gradient assumptions, a sufficiently small update improves objective m whenever its gradient has nonnegative inner product with the update direction.
C Formal Analysis for Clipped Surrogate Objectives: For natural-gradient updates, nonnegative Fisher-covariance conditions across objectives are sufficient to prevent objective degradation, with strict improvement under a positive margin.
Then the clipped first-order margin satisfies: Clipping preserves the first-order improvement guarantee when the clipped margin remains nonnegative, while deleting weighted logit-gradient terms from the update.
Then the clipped first-order margin satisfies: The clipping distortion measures deleted gradient mass in natural-gradient geometry and can be controlled through learning-rate scheduling or reward normalization; the same logic applies to PPO-style objectives.

D.1 Proof of Lemma 4.1

The proof derives the optimizer of the KL-regularized reward-level update as an exponential tilting of the current policy, then expands this solution to obtain the first-order covariance improvement law. It also connects clipped-surrogate updates to the same objective-wise improvement condition through gradient decomposition and bounds.

Exponential-tilting solution: The optimizer is obtained by fixing the prompt, applying stationarity, exponentiating the resulting condition, and enforcing normalization.The proof introduces a normalization constant independent of y before substituting back into the optimizer.
Exponential-tilting solution: The KL-regularized update reweights the current policy exponentially according to the scalarized score.The resulting distribution is proportional to pθ;x(y)e^(ηs(x,y)), so unsupported completions remain unsupported.
Local improvement law: A Taylor expansion of the exponential tilt yields a first-order change in each objective proportional to its covariance with the scalarized score.Bounded rewards control the remainder term at O(η^2), after which expectation over prompts gives the stated expansion.
Clipped updates: For clipped surrogate objectives, clipping removes gradients from clipped terms while leaving unclipped terms unchanged, so the covariance-based guarantee persists when removed gradient mass is sufficiently small.The proof identifies the clipped and unclipped regions and derives a margin bound using a Fisher-metric decomposition and Cauchy–Schwarz.
Clipped updates: If every objective has a positive directional derivative along the natural update, a sufficiently small learning rate yields nonnegative improvement for all objectives, with strict improvement when all margins are positive.The result covers the zero-update case separately and otherwise bounds the step size using the minimum objective-wise margin.

D.7 Proof of Theorem 6.4

The proof establishes global convergence of the scalarized RFT objective under a Polyak–Łojasiewicz condition by relating policy-gradient strength to probability mass on the unique scalarized-reward maximizer. It derives the required geometric bounds from model Jacobians, token-level alignment, and probability assumptions, then gives CTWA implementation details.

PL framework: The PL condition links gradient magnitude to global suboptimality without requiring the scalarized objective to be convex.The proof applies this condition to the scalarized RFT objective under Assumptions 6.1–6.3.
Geometric conditions: The gradient lower bound depends on token uncertainty, sequence length, the alignment constant c, and the smallest singular value of the logit Jacobian.The probability parameter ϵ captures uncertainty: smaller ϵ means higher confidence but weaker gradient signals.
Optimal trajectory: When the scalarized reward has a unique maximizer y* and the policy class can realize it deterministically, the optimal scalarized value equals s(x,y*).The value gap is controlled by the probability gap on y*, with each unit of missing probability costing at most 2B.
Gradient alignment: Near the optimal policy, the directional derivative along the normalized optimal-trajectory gradient is strongly positive and pushes additional probability mass toward y*.This alignment supplies the lower bound needed to relate gradient strength to the scalarized value gap.
PL conclusion: The resulting µ-PL parameter is obtained by combining the directional-gradient bound with the probability-based value-gap bound.This yields sufficient model-dependent conditions for global convergence of the non-convex scalarized objective.
Algorithmic implementation: CTWA updates objective weights from exponentially smoothed covariance deficits, whereas GradNorm adjusts weights to match scaled gradient-norm targets on top of GRPO.Both procedures use per-objective statistics during GRPO training but encode different adaptation signals.