PaperlayerView exact revision on arXiv

Source-linked AI summary

Reward-free Alignment for Conflicting Objectives

Peter Chen, Xiaopeng Li, Xi Chen, Tianyi Lin

arXiv:2602.02495v3cs.CLcs.AIcs.LG

TL;DR

Conflicting alignment objectives make weighted preference aggregation unstable and unable to reliably improve all objectives, while existing approaches may require explicit reward models. RACO directly optimizes multiple preference losses with clipped conflict-averse gradient descent, proving weighted Pareto-critical convergence and two-objective acceleration. Across summarization and safety alignment tasks and several LLM families, it consistently achieves improved Pareto trade-offs over reward-free baselines.

  • Problem

    Conflicting objectives can make weighted preference losses miss jointly improving directions, while many existing multi-objective methods rely on explicit reward models or heuristic aggregation.

  • Method

    RACO directly optimizes objective-specific preference losses with clipped conflict-averse gradient descent that limits corrections according to user-specified objective weights.

  • Results

    RACO consistently achieves improved Pareto trade-offs over existing reward-free methods across multi-objective summarization and safety alignment on Qwen 3, Llama 3, and Gemma 3.

  • Takeaways & Limitations

    The results support explicit gradient-conflict handling as a practical approach to reward-free LLM fine-tuning under conflicting objectives.

  • Takeaways & Limitations

    Clipping changes vanilla CAGrad’s theoretical behavior, so the paper provides a separate convergence analysis for the clipped update rule.

Abstract

from arXiv · show

Direct alignment methods are increasingly used to align large language models (LLMs) with human preferences. However, many real-world alignment problems involve multiple conflicting objectives, where naive aggregation of preferences can lead to unstable training and poor trade-offs. In particular, weighted loss methods may fail to identify update directions that simultaneously improve all objectives, and existing multi-objective approaches often rely on explicit reward models, introducing additional complexity and distorting user-specified preferences. The contributions of this paper are two-fold. First, we propose a Reward-free Alignment framework for Conflicted Objectives (RACO) that directly leverages pairwise preference data and resolves gradient conflicts via a novel clipped variant of conflict-averse gradient descent. We provide convergence guarantees to Pareto-critical points that respect user-specified objective weights, and further show that clipping can strictly improve convergence rate in the two-objective setting. Second, we improve our method using some heuristics and conduct experiments to demonstrate the compatibility of the proposed framework for LLM alignment. Both qualitative and quantitative evaluations on multi-objective summarization and safety alignment tasks across multiple LLM families (Qwen 3, Llama 3, Gemma 3) show that our method consistently achieves better Pareto trade-offs compared to existing multi-objective alignment baselines.

1. Introduction

LLM preference alignment is inherently multi-objective, but conflicting criteria make naive aggregation unstable and produce poor trade-offs. The paper develops a reward-free, conflict-aware approach with convergence guarantees and improved empirical Pareto trade-offs.

  • Conflicting objectives such as helpfulness, harmlessness, faithfulness, and conciseness make single-objective optimization or naive preference aggregation unsuitable.
  • Existing multi-objective alignment methods combine models, condition policies, or modify decoding, but rely on linear aggregation, conditioning, or explicit objective-specific reward models.
  • Opposing objective gradients can leave weighted losses without an update direction that improves every objective, forcing some objectives to be privileged.
  • RACO directly applies conflict-averse gradient descent to objective-specific preference losses and clips corrections to respect user-specified weights.
  • The method provides convergence guarantees to specified Pareto-critical points, with clipping strictly improving the two-objective convergence rate, and consistently improves Pareto trade-offs experimentally.

2. Preliminaries and Technical Background

The paper formulates reward-free preference alignment as multi-objective optimization over pairwise preference losses. Pareto optimality and gradient-based methods provide the technical framework for handling conflicting objectives and user-specified trade-offs.

  • An LLM is modeled as an autoregressive policy πθ(y|x), whose generated responses require alignment with human preferences for helpfulness, safety, and reliability.
  • Reward-free alignment uses pairwise preference data containing prompts, preferred responses, and dispreferred responses, followed by DPO optimization without explicit reward modeling.
  • Multi-objective optimization compares solutions using Pareto dominance when objectives conflict, and seeks points that are not dominated by alternatives.
  • A Pareto-critical point admits no direction that strictly decreases every objective, equivalently characterized through a suitable convex combination of objective gradients.
  • CAGrad constructs conflict-aware descent directions, and the proposed method replaces averaged weights with user-specified weights while adding clipping for stability.

3. Main Results

RACO applies clipped conflict-averse gradient descent to multiple preference losses, limiting overcorrection while preserving user-specified trade-offs. Under smooth nonconvex assumptions it has convergence guarantees, and clipping can accelerate two-objective training.

  • Algorithm and setup: RACO computes objective-specific preference losses and gradients from minibatches of preference pairs, then forms a weighted gradient and conflict correction.
  • Algorithm and setup: The clipping modification is motivated by high-dimensional, noisy LLM policy spaces and vanilla CAGrad’s risk of shifting updates toward less-preferred objectives.
  • Algorithm and setup: CAGrad-Clip clips each correction coefficient to its corresponding preference weight, preventing objectives from being upweighted beyond their assigned importance.
  • Theory and scope: The convergence analysis is developed specifically for the clipped update rule because vanilla CAGrad’s original convergence guarantees no longer directly apply.
  • Convergence guarantees: Any limit point is both a critical point of the weighted objective and Pareto-critical under η ∈ (0, 1/ℓw] and c ∈ [0, 1).
  • Convergence guarantees: min 0≤t<T M(θt)^2 ≤ 2Lw(θ0) / [η(1 − c^2)T], establishing a sublinear convergence bound for the stationarity measure.
  • Acceleration: In the two-objective setting, clipping provides a strictly stronger one-step descent guarantee than unclipped CAGrad whenever clipping is active.

4. Experiment

The experiments evaluate RACO on multi-objective summarization and safety alignment tasks using offline pairwise preference data, across Qwen, Llama, and Gemma model configurations. Across these settings, RACO generally produces more favorable Pareto trade-offs and balanced objective improvements than comparison methods.

  • Experimental setup: The experiments cover quality–conciseness, quality–faithfulness, and helpfulness–harmlessness alignment tasks using multiple model families and weighted objectives.Reddit Summary evaluates quality with controllable conciseness and faithfulness; BeaverTails evaluates helpfulness and harmlessness.
  • Experimental setup: 60% of the quality–conciseness examples exhibit fully conflicting objectives, with the quality winner also losing under conciseness.The dataset contains 92,858 examples after preprocessing.
  • Reddit summary: Under unequal quality weights, RACO consistently improves both quality and conciseness, whereas AMoPO and DPO-LW generally improve the more heavily weighted objective at the other’s expense.On Llama3-8B with wqual = 0.8, only RACO also improves the less-weighted conciseness metric.
  • Reddit summary: Across Qwen3-4B and Llama3-8B, RACO attains the outermost quality–conciseness Pareto frontier over wqual ∈{0.8, 0.65, 0.5, 0.35, 0.2}.The reported frontier summarizes trade-offs across the swept input weights.
  • Reddit summary: RACO achieves a more favorable quality–faithfulness Pareto frontier on both Qwen3-4B and Llama3-8B, with the strongest advantage under highly imbalanced weights.For example, at wqual ∈{0.8, 0.2}, RACO dominates the Llama3-8B curve endpoints on both objectives.
  • Safety alignment: On BeaverTails, RACO improves the harmlessness–helpfulness trade-off without optimizing one objective at the expense of the other across model configurations.For Qwen3-4B-Base and Gemma3-4B-Instruct, RACO is the only method reported to consistently show this balanced behavior.
  • Safety alignment: At high harmlessness weights, RACO attains slightly lower harmlessness but substantially higher helpfulness than methods that maximize harmlessness.The accompanying judge win-rate analysis reports consistent outperformance against Qwen3-4B-Instruct and Gemma3-4B-Base overall.
  • Ablation analysis: With w1 = 0.8 and w2 = 0.2, clipping prevents CAGrad over-correction and improves validation margins toward a better Pareto equilibrium.CAGrad assigns a large correction weight to the less-preferred objective before clipping.

5. Conclusion

The paper proposes RACO, a reward-free conflict-aware framework that directly optimizes multiple preference losses while respecting user-specified weights. Across summarization and safety alignment experiments, it improves Pareto trade-offs over existing reward-free baselines.

  • RACO directly optimizes multiple preference losses without explicit reward models.
  • Clipped conflict-averse gradient descent resolves gradient conflicts while respecting user-specified objective weights.
  • RACO has convergence guarantees to Pareto-critical points and provably accelerates convergence in the two-objective case.
  • Experiments across multi-objective summarization and safety alignment show consistent Pareto-trade-off improvements over existing reward-free baselines.

Impact Statement

The work presents a reward-free alignment method intended to improve handling of conflicting objectives and controllability of trade-offs. Its evaluations use established benchmarks and avoid collecting additional harmful content.

  • The reward-free method is designed to better handle conflicting objectives such as helpfulness and harmlessness.
  • The method is intended to improve controllability over the resulting alignment trade-offs.
  • Experiments use official benchmarks from prior published work without collecting additional harmful content.
  • Sensitive case studies are placed in the appendix, with explicit warnings and redaction of the most explicit terms.

Implementation & Model Checkpoint

The paper releases code and model checkpoints for RACO and baseline methods, while limiting checkpoint release based on safety testing to reduce potential misuse.

  • The authors release the RACO implementation and checkpoints for RACO and baseline methods.
  • Only checkpoints found safe according to the authors’ testing are released to reduce potential misuse of jailbreaking-related checkpoints.

A. Further Related Works

The related work situates RACO among preference learning, multiobjective optimization, conflicting-objective alignment, and broader reinforcement-learning methods. It also states theoretical and statistical scope limits, including convergence only to Pareto-critical points and the absence of stochastic, sample-complexity, consistency, and generalization guarantees.

  • Further discussions on preference learning methods: Preference-learning research includes data augmentation and extensions of DPO, whose performance depends on the scale and quality of offline preference data.
  • Analysis of preference learning methods: Prior analyses study consistency, KL-regularized exploration, and online preference learning in contextual-bandit and Markov-decision-process settings.
  • Multiobjective optimization methods: Scalarization methods approximate Pareto fronts but may miss nonconvex regions, while alternative schemes improve coverage at the cost of additional parameter tuning.
  • Conflicting objectives in LLM training: Alignment objectives such as helpfulness, harmlessness, faithfulness, and verbosity can compete, producing alignment-tax effects and over-refusal.
  • Broader online RL methods: Recent work also explores actor–critic–free training, exploration–exploitation dynamics, and multi-agent LLM policy training.
  • Limitations and Further Discussions: The theory guarantees convergence to Pareto-critical points for a nonconvex weighted empirical objective, not global Pareto optimality.
  • Limitations and Further Discussions: The proofs do not provide a stochastic-gradient theorem for the minibatch implementation.
  • Limitations and Further Discussions: The analysis provides no sample-complexity bounds, statistical consistency guarantees, or generalization guarantees for the fixed offline preference dataset.

B. Missing Proofs

The two-objective subproblem is reduced to a one-dimensional optimization with a closed-form quadratic characterization. Candidate roots and endpoints are evaluated to select the minimizer.

  • Two-objective reduction: For m = 2, the subproblem is formulated using objective weights, gradient inner products, and auxiliary coefficients q0, q1, and q2.The weights are p = (λ, 1 − λ), with λ ∈ [0, 1].
  • Quadratic characterization: The resulting one-dimensional objective yields a quadratic equation in λ.The quadratic has coefficients involving δ, s, q1, q2, and q0.
  • Candidate selection: The solution procedure retains roots in [0, 1], evaluates the objective at those roots and at λ = 0 and λ = 1, then selects the minimizer.This checks both interior candidates and boundary solutions.

B.2. Proof of Theorem 3.1

The proof establishes that the clipped conflict-averse update is compatible with the weighted objective and converges to Pareto-critical points. It also derives a finite-iteration bound on the minimum stationarity measure.

  • Algorithm and objective: Algorithm RACO with CAGrad-Clip samples preference pairs, computes each objective loss and gradient, clips coefficients by the user weights, and forms a clipped mixture direction.The weighted loss is defined as Lw(θ) := Σ_i wi Li(θ).
  • Loss properties: Each objective loss is nonnegative because it is built from −log σ(z), whose value is nonnegative for sigmoid outputs.Averaging preserves nonnegativity, so the weighted loss is also nonnegative.
  • Pareto criticality: A stationary point of Lw is Pareto critical for the individual objectives when the weights lie in the simplex.If every objective could decrease along one direction, their nonnegative weighted sum would also decrease, contradicting stationarity.
  • Convergence: Any limit point is both critical for Lw and Pareto critical, under Lipschitz-gradient assumptions and stepsize η ∈ (0, 1/ℓw].The result holds for c ∈ [0, 1).
  • Convergence rate: min 0≤t<T M(θt)^2 ≤ 2Lw(θ0) / [η(1 − c^2)T], providing an O(1/T) bound on the minimum stationarity measure.The same bound also applies to min 0≤t<T ||∇Lw(θt)||^2.

B.3. Proof of Theorem 3.2

The two-objective analysis shows that clipping improves the alignment of the correction direction with the personalized gradient when clipping is active. This yields a strictly stronger one-step descent guarantee under the stated conditions.

  • Alignment comparison: For two objectives, the clipping analysis compares the alignment ratios of the unclipped and clipped correction directions.The proof parameterizes candidate directions by a coefficient ratio and studies the resulting normalized alignment function.
  • Effect of clipping: When clipping is active and the objective gradients are non-colinear, the clipped alignment ratio satisfies ˜ρt > ρt; otherwise the ratios are equal.The equality cases include degenerate gradient or coefficient configurations.
  • Two-objective geometry: The derivative of the alignment function has sign determined by δt(w1 − w2r), producing a unique maximizer at r* = w1 / w2.The function increases before r* and decreases after it.
  • Descent guarantee: The descent factor Γ is strictly increasing in ρ when c > 0 and η < 1/ℓw, so improved alignment strengthens the decrease guarantee.The proof uses Γ(˜ρ) − Γ(ρ) = c(1 − ℓwη)(˜ρ − ρ).
  • Interpretation: CAGrad-Clip yields a strictly stronger one-step descent guarantee for Lw than the unclipped method whenever clipping is active.A toy example illustrates the mechanism: clipping prevents over-correction toward a low-priority objective and improves alignment with g0.

C.1. Case Studies (Warning: harmful and sexually explicit content)

The case studies illustrate how RACO changes helpfulness–harmlessness trade-offs with user-specified weights and differs from competing methods under the same weights. The examples emphasize refusal, caution, and informative reframing as distinct trade-offs.

  • Input-weight effects: Under higher harmlessness weight, Qwen3-4B-Instruct-RACO decisively refuses a harmful and sexually explicit request, whereas equal weights produce a different response.The case study uses wharm = 0.8 as the higher-harmlessness setting.
  • Evaluation design: The section presents case studies comparing responses across input weights and methods under the same prompt and weight.These comparisons are intended to illustrate how methods respond to conflicting objectives.
  • Input-weight effects: For a benign prompt, higher harmlessness weight produces more cautious, affectively supportive, concise, and de-escalatory guidance than balanced weights.Both settings remain helpful while emphasizing different aspects of the response.
  • Cross-method comparison: With high helpfulness weight on Gemma-3-4B-it, RACO refuses an explicit slang glossary and reframes the request academically while remaining informative.The comparison uses AMoPO and DPO LW as competing methods.
  • Cross-method comparison: AMoPO largely abandons harmlessness by listing explicit slang, while DPO LW uses ethical framing but still enumerates porn-related slang with unnecessary explicitness.The passage characterizes both alternatives as providing only moderate or insufficient safety in this example.

<Redaction Note for AMoPO output>

The paper limits exposure to sensitive material by redacting explicit terms, minimizing excerpts, and placing qualitative examples in the appendix. It presents examples only when needed to illustrate otherwise-hidden failure modes and frames reporting around safer training and evaluation.

  • Ethical handling: The response discusses harmful terminology as offensive, dehumanizing, and connected to exploitation, coercion, and power imbalances.It emphasizes that such language can reinforce objectification and cause emotional distress.
  • Illustrative failure modes: The bike-sabotage examples contrast unsafe helpfulness-dominant behavior with RACO’s explicit rejection of illegal property damage and safer alternatives.DPO LW warns about trouble but does not clearly reject the harmful plan, whereas RACO does.
  • Ethical handling: Sensitive examples are placed in the appendix, accompanied by explicit content warnings and redactions of the most explicit terms.The paper avoids reproducing searchable glossaries and keeps excerpts minimal.
  • Ethical handling: The paper states that its reporting is intended to motivate safer training and evaluation practices rather than enable unsafe deployment.
Loading 2602.02495v3…