PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization

Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, Yu Qiao

arXiv:2310.03708v4cs.LGcs.AI

TL;DR

Diverse human preferences are difficult to serve with a single model, and MORLHF’s RL training is unstable and costly with conflicting objectives. MODPO extends DPO to optimize multiple objectives without RL by folding language modeling into reward modeling. Across safety alignment and long-form question answering, it matches or surpasses existing methods while producing models for diverse preferences.

  • Problem

    Human preferences vary, and multi-objective RLHF faces instability and inefficiency when optimizing usually conflicting objectives.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    MODPO extends DPO without RL, training language models as implicit collective reward models that combine objectives with specific weightings.

    Source: §1 · p. 1

  • Results

    Across safety alignment and long-form question answering, MODPO matches or surpasses existing methods and produces a Pareto front for diverse preferences.

    Source: §1 · p. 2

  • Takeaways & Limitations

    When a target preference is known, its preference vector can be tuned to customize a single language model.

    Source: §6 · p. 8

    The evaluation relied on automatic evaluators rather than human evaluations.

    Source: §7 · p. 8

Abstract

from arXiv · show

A single language model, even when aligned with labelers through reinforcement learning from human feedback (RLHF), may not suit all human preferences. Recent approaches therefore prefer customization, gathering multi-dimensional feedback, and creating distinct reward models for each dimension. Different language models are then optimized for various preferences using multi-objective RLHF (MORLHF) with varying reward weights. However, RL fine-tuning is unstable and resource-heavy, especially with diverse and usually conflicting objectives. In this paper, we present Multi-Objective Direct Preference Optimization (MODPO), an RL-free extension of Direct Preference Optimization (DPO) for multiple alignment objectives. Essentially, MODPO folds language modeling directly into reward modeling, training language models as implicit collective reward models that combine all objectives with specific weights. MODPO theoretically yields the same optimal solutions as MORLHF but is practically more stable and efficient. Empirical results in safety alignment and long-form question answering show that MODPO matches or outperforms existing methods, producing a Pareto front of language models catering to diverse preferences with three times less computational resources compared to MORLHF. Code is available at https://github.com/ZHZisZZ/modpo.

1 Introduction

Human preferences vary, while multi-objective RLHF becomes unstable and resource-intensive when objectives conflict. MODPO extends DPO to align models with multiple objectives without RL.

1 Introduction

  • Human preferences vary widely, making a single model based on average preferences hard to suit different groups.Recent customization approaches use fine-grained dimensions such as helpfulness, harmlessness, and honesty to train distinct reward models.Source: §1 · p. 1
  • MORLHF combines reward functions with linear scalarization, but conflicting objectives compound RLHF’s instability and inefficiency.MORLHF trains language models with different reward weights to create a selectable Pareto front.Source: §1 · p. 1
  • MODPO extends DPO without RL, folding language modeling into reward modeling to represent weighted combinations of objectives.The method is described as theoretically yielding the same optimal solutions as MORLHF while being more stable and computationally efficient.Source: §1 · p. 1
  • In safety alignment and long-form QA, MODPO matches or surpasses existing methods and produces a Pareto front of models for diverse preferences.Figure 1 characterizes the front as better than MORLHF’s, with minimal overhead.Source: Figure 1 · p. 2, §1 · p. 2

2 Background

The background contrasts alignment to the majority’s preferences with multi-objective alignment for diverse preferences. It reviews how DPO and MORLHF approach these settings.

2.2 Multi-Objective Preference Alignment

  • Homogeneous alignment fine-tunes one model for the majority, whereas multi-objective alignment uses separate feedback dimensions to support preference customization.Multi-dimensional feedback may compare responses or annotate them individually, producing datasets for distinct objectives.Source: §2.1 · p. 2, §2.2 · p. 3

2.1 Homogeneous Preference Alignment

  • Homogeneous preference alignment models preferences with a latent reward under the Bradley-Terry assumption and constrains reward maximization by a KL penalty.The KL constraint controls its strength to maintain generation diversity and avoid reward over-optimization that degrades quality.Source: §2.1 · p. 2
  • In homogeneous alignment, RLHF fits a reward model and then uses RL, while DPO directly optimizes the language model on preference data.DPO’s loss bypasses explicit reward modeling and reinforcement learning; the preference data contains preferred and dispreferred responses to the same prompt.Source: §2.1 · pp. 2–3

2.2 Multi-Objective Preference Alignment

  • MORLHF trains separate reward models, optimizes a language model for each preference vector, and iterates across vectors to approximate a Pareto front.Conflicting objectives and the need for multiple optimal models exacerbate RLHF’s instability and computational inefficiency.Source: §2.2 · p. 3

3 Multi-Objective Direct Preference Optimization (MODPO)

MODPO extends DPO to optimize multiple objectives without RL by directly training language models to represent weighted collective rewards. Its method uses preference data to derive an objective, and its training overhead over DPO is limited.

3 Multi-Objective Direct Preference Optimization (MODPO)

  • MODPO directly optimizes multiple objectives without RL by training language models as collective reward models for different preference weightings.The resulting empirical front spans diverse preferences.Source: §3 · pp. 3–4

3.1 MODPO Methodology

  • MODPO uses preferences from one objective dataset to cancel the partition function in its reward-to-policy mapping and derive a practical maximum-likelihood objective.The resulting loss addresses the preference-classification problem with additional weightings and a margin term, and guarantees the optimal model for a specific preference vector.Source: §3.1 · p. 4
  • The training outline fits margin reward models on the other datasets, then optimizes the MODPO loss for each preference vector to obtain a model front.The method assumes at least one preference dataset; a random dataset can serve as a dummy objective when preference data is absent.Source: §3.1 · p. 4, §3 · p. 4

3.2 MODPO Advantages

  • MODPO’s loss has similar training dynamics to DPO and needs only fitted margin reward models, whose costs can be amortized across preference vectors.Per-model training costs are thereby reduced to language modeling, comparable to DPO’s per-model costs.Source: §3.2 · p. 4
  • MODPO’s minimal overhead over DPO motivates its use in multi-objective settings, and the paper reports empirical support for that potential.The cited empirical support is presented in the following section.Source: §3.2 · p. 4

4 Experiments

The experiments assess whether MODPO can customize language-model fronts for diverse preferences and outperform baselines. Across synthetic safety, real safety, and long-form QA, its fronts match or exceed MORLHF's, with the QA results showing strong customization.

4.1 Experimental Setup

  • Models are trained across preference weights to form empirical fronts, which are assessed by objective rewards and, for MODPO and MORLHF, KL divergence.Source: §4.1 · pp. 5–6
  • The experiments cover harmlessness versus helpfulness in safety alignment and relevance, factuality, or completeness paired with overall preference in long-form QA.Source: §4.1 · p. 5

4.2 Experiment Results

  • In synthetic safety, MODPO generally performs better on helpfulness while MORLHF is slightly better on harmlessness; MODPO's low-KL advantage costs somewhat more KL budget.Source: §4.2 · p. 6
  • Real-feedback safety fronts use GPT-3.5 and GPT-4 win-rate evaluations against πsft, and MODPO interpolates real-world preferences.Source: §4.2 · p. 6, Figure 4 · p. 7
  • MODPO consistently surpasses MORLHF on long-form QA fronts and shows a better real-safety front; on synthetic safety, its fronts are at least as good overall.Source: §4.2 · pp. 6–7, Figure 3 · p. 7, Figure 4 · p. 7
  • As the preference weight changes in long-form QA, specialization in relevance, factuality, or completeness does not significantly degrade overall preference.Source: Figure 3 · p. 7

5 Related Work

Related work contrasts average-preference RLHF with multi-objective customization and RL-free alignment. MODPO addresses the remaining challenge of adapting RL-free methods to multiple objectives with theoretical guarantees and low overhead.

  • RLHF optimizes against a reward model representing average labeler preferences, risking marginalization of underrepresented groups.Source: §5 · p. 7
  • Multi-objective alignment includes training separate models and merging weights, or incorporating objectives during training; MODPO instead uses pure cross-entropy rather than compute-intensive RL fine-tuning.Source: §5 · p. 8
  • Existing RL-free methods align to homogeneous preference distributions, leaving their extension to multiple objectives with theoretical guarantees and minimal overhead unresolved.Source: §5 · p. 8

6 Conclusion

The conclusion presents MODPO as an RL-free DPO extension for multiple objectives, theoretically equivalent to MORLHF and practically more efficient. It also highlights preference-vector tuning as a way to customize a single language model.

  • MODPO extends DPO to multiple alignment objectives and is theoretically equivalent but practically more efficient than MORLHF.Source: §6 · p. 8
  • The method optimizes language models with a simple cross-entropy loss and shows empirical results across multiple tasks.Source: §6 · p. 8
  • When the target preference is known, its preference vector can be tuned to customize a single language model for diverse preferences.Source: §6 · p. 8

7 Limitations and Future Works

The authors identify limits in evaluation, objective coverage, scalarization assumptions, and task scope, while pointing to further settings for future study.

7 Limitations and Future Works

  • Human evaluations were not conducted; the study relied on automatic evaluators.The authors list this as a limitation of their evaluation.Source: §7 · p. 8
  • The three-objective scaling experiment was not extended to additional objectives.The authors state that they did not conduct further experiments with more objectives.Source: §7 · p. 8
  • The use of linear scalarization assumes human preferences are linearly composable, which may not always hold.Source: §7 · p. 8
  • MODPO was not validated on more complex tasks, including multi-turn dialogue and mathematical reasoning.Source: §7 · p. 8
  • The authors identify reward-difference supervision and applications to other generative models as questions beyond this study’s scope.Examples named include visual language models and text-to-image diffusion models.Source: §7 · p. 8

A Mathmatical Derivations

The appendix establishes reward-equivalence results underlying MODPO’s reparameterization, then derives its use for single-objective rewards and multi-objective alignment without preference data.

A.1 Preliminaries

  • Equivalent reward functions induce the same preference distribution and optimal policy, and reward equivalence classes can be represented through a policy reparameterization.The reparameterization uses a supervised fine-tuned policy and β > 0.Source: §A.1 · p. 11

A.2 Justification for the Reparametrization in Eq. 11

  • The appendix extends the reparameterization to arbitrary g(x, y), concluding that it does not reduce generality.The proof composes the reward with g(x, y) and uses equivalence of the resulting rewards.Source: §A.2 · p. 11

A.3 Apply MODPO to Broader Scenarios

  • MODPO’s broader scenarios include single-objective optimization and multi-objective alignment without a preference dataset.Source: §A.3 · p. 11

A.3.1 General Single-Objective Problem

  • For a generic reward not derived from relative preferences, randomizing preferences creates a dummy objective that does not affect the trained model.The resulting loss repurposes MODPO for single-objective optimization, with the generic reward providing the steering signal.Source: §A.3.1 · p. 12
  • The single-objective derivation transforms the preference-classification loss over the random-preference reward into a policy loss.Source: §A.3.1 · p. 12

A.3.2 Multi-Objective Problem without a Preference Dataset

  • Without a human-labeled preference dataset, MODPO can train on randomized preferences using the objective rewards as margin reward models.The described procedure first trains each reward model on its respective dataset, then trains language models with MODPO.Source: §A.3.2 · p. 12

B Scaling-Up Experiments with Three Objectives

In long-form QA with three objectives, the reported comparison finds that MODPO outperforms MORLHF, extending the paper’s scaling experiments.

B Scaling-Up Experiments with Three Objectives

C MODPO Implementation Details

The implementation presents MODPO as a small addition to DPO, with a loss computed from policy log probabilities, margin rewards, and objective weights.

C.1 Pseudocode

  • The MODPO loss requires only two extra lines of code on top of a DPO implementation.Source: §C.1 · p. 14
  • The loss takes policy and reference log probabilities, preferred and dispreferred completion indices, a KL-penalty temperature, margin rewards, and an objective-weight vector.The current preference dataset receives weight w[0], while the other objectives receive weights w[1:].Source: §C.1 · p. 14
  • It forms a weighted margin-reward difference between preferred and dispreferred completions and uses it with policy-reference log-ratio differences to compute the loss.Source: §C.1 · p. 14

D.1 Datasets & reward models

The paper uses BEAVERTAILS for safety alignment and QA-FEEDBACK for long-form question answering, with separate feedback and reward models for distinct objectives. The implementation reuses pretrained reward models where available and specifies separate MODPO and MORLHF setups for the two tasks.

D.1.1 Safety Alignment

  • Safety alignment uses a 10k BEAVERTAILS subset with separate helpfulness and harmlessness preferences, relabeled by open-source reward and cost models for synthetic experiments.The test split supports synthetic front plotting, while the BEAVERTAILS GPT evaluation set supports real-front plotting.Source: §D.1.1 · p. 14

D.1.2 Long-Form QA

  • QA-FEEDBACK contains overall-quality preferences and three fine-grained datasets targeting relevance, factuality, and completeness errors.Its Longformer reward models provide per-sub-sentence binary rewards for relevance and factuality, and a continuous sequence-level score for completeness.Source: §D.1.2 · p. 15
  • The fine-grained QA reward models approximate unknown latent human-decision rewards rather than providing ground-truth rewards.The paper refers readers to FINE-GRAINED RLHF for their detailed descriptions.Source: §D.1.2 · p. 15

D.2 Implementation Details

  • Models are trained with LoRA on eight Nvidia 80G A100 GPUs.Additional hyperparameters are listed in Table 2.Source: §D.2 · p. 15

D.2.1 Safety Alignment

  • For safety alignment, alpaca-7b-reproduced is reused as the SFT model, while MODPO uses an implicitly parameterized harmlessness model to guide helpfulness training.MORLHF instead models helpfulness and harmlessness rewards as linear projections from SFT embeddings and trains them with binary cross-entropy.Source: §D.2.1 · p. 15

D.2.2 Long-Form QA

  • For long-form QA, the paper fine-tunes llama-7b-hf on the QA-FEEDBACK SFT split and reuses public rule-based reward models as MODPO margin models.MORLHF reuses the rule-based models but learns a preference reward model because a public one is unavailable.Source: §D.2.2 · p. 16

E.1 MODPO Ablations for Safety Alignment

The appendix reports safety-alignment ablations and training curves, customization examples, and GPT evaluation samples. It also notes that the long-form QA Best-of-n comparison is not directly comparable to other methods.

E.1 MODPO Ablations for Safety Alignment

  • MODPO’s three safety-alignment design variants achieve similar performance, with none strictly dominating the others.The ablation varies which preference dataset trains the margin reward model and whether its parameterization is implicit or explicit.Source: §E.1 · p. 17

E.2 Long-Form QA with Best-of-n

  • Best-of-n is an oracle and not directly comparable to other methods when the same reward models are used for rejection sampling and evaluation.The appendix presents these results separately from the main comparison.Source: §E.2 · p. 17

E.3 Training Curves

  • Similar training accuracies for MODPO and DPO indicate that MODPO’s added margin and weighting term do not compromise training stability.Both losses address the same binary classification problem and differ in parameterization.Source: §E.3 · p. 18, Figure 10 · p. 19

E.4 Examples of LM Customization from Safety Alignment

  • The paper presents examples of language model customization for safety alignment in Tables 3, 4, and 5; each table represents its objective using a collective reward model.Source: §E.4 · p. 18, Table 3 · p. 22, Table 4 · p. 23

E.5 Examples of LM Customization from Long-Form QA

  • In long-form QA examples, smaller w is associated with fewer rule-based violations, less redundancy and fewer hallucinations, and better completeness.Some responses with smaller w copy passages as a heuristic to prevent hallucinations or information loss.Source: §E.5 · pp. 18–19

E.6 GPT Evaluation Samples from Safety Alignment

  • The evaluation asks judges to rate each assistant separately on harmlessness and helpfulness from 1 to 10, with higher scores indicating more of each; helpfulness scoring considers answering the question, not harmfulness.Source: §E.6 · pp. 19–20
Loading 2310.03708v4…