PaperlayerView exact revision on arXiv

Source-linked AI summary

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, Kaiqi Huang

arXiv:2405.12739v2cs.LG

TL;DR

Human preferences span multiple dimensions that can conflict, while existing approaches may collapse them into one ranking or rely on multiple reward models. SPO sequentially optimizes preference dimensions with constraints intended to preserve earlier alignments, and experiments report multi-dimensional alignment across datasets.

  • Problem

    Human preferences have multiple, potentially conflicting dimensions, while combining them into one ranking risks subjective inconsistency and dimension-wise RLHF requires multiple reward models.

    Source: Introduction · p. 1

  • Method

    SPO sequentially fine-tunes one preference dimension per round, constraining optimization to preserve alignment on previously optimized dimensions without explicit reward modeling.

    Source: Introduction · pp. 1–2

  • Results

    Experiments across training and evaluation datasets report that SPO aligns LLMs across multiple preference dimensions and outperforms baseline methods and open models.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    SPO’s constrained sequential optimization is reported to optimize new preference dimensions while preserving alignment from previous rounds.

    Source: Conclusion and Limitation · p. 7

    Experiments do not include extremely large open models because of computational limits.

    Source: Conclusion and Limitation · p. 8

Abstract

from arXiv · show

Human preference alignment is critical in building powerful and reliable large language models (LLMs). However, current methods either ignore the multi-dimensionality of human preferences (e.g. helpfulness and harmlessness) or struggle with the complexity of managing multiple reward models. To address these issues, we propose Sequential Preference Optimization (SPO), a method that sequentially fine-tunes LLMs to align with multiple dimensions of human preferences. SPO avoids explicit reward modeling, directly optimizing the models to align with nuanced human preferences. We theoretically derive closed-form optimal SPO policy and loss function. Gradient analysis is conducted to show how SPO manages to fine-tune the LLMs while maintaining alignment on previously optimized dimensions. Empirical results on LLMs of different size and multiple evaluation datasets demonstrate that SPO successfully aligns LLMs across multiple dimensions of human preferences and significantly outperforms the baselines.

Introduction

Human preferences span dimensions that can conflict, while combining them into one ranking risks subjective inconsistency and dimension-specific RLHF requires multiple reward models. SPO sequentially optimizes dimensions while constraining alignment on earlier ones, and experiments report multi-dimensional alignment across datasets.

Introduction

  • Preferences can conflict across dimensions, such as brevity and informativeness, so a generally preferred response may not be preferred on every dimension.Combining dimensions into one ranking can be influenced by annotator subjectivity and ranking inconsistency; dimension-wise RLHF methods require a reward model for each dimension.Source: Introduction · p. 1
  • SPO fine-tunes sequentially, optimizing one preference dimension per round while constraining the objective to preserve alignment on previously optimized dimensions.It directly optimizes preferences without explicit reward modeling, avoiding the need to manage multiple reward models in RLHF-based methods.Source: Introduction · p. 1
  • The paper derives a closed-form optimal SPO policy and loss, then uses gradient analysis to explain preservation of alignment from previous fine-tuning rounds.The loss is described as a simple classification loss that can be optimized efficiently.Source: Introduction · p. 1
  • Experiments across training and evaluation datasets report that SPO aligns LLMs across multiple preference dimensions and outperforms baseline methods and open models.The reported experiments include Llama 2 7B and 13B models and evaluation on multiple datasets.Source: Introduction · p. 2

Preliminaries

The preliminaries describe the SFT model as the starting point for preference optimization, model human preferences with response-pair rankings, and contrast explicit-reward RLHF with implicit-reward DPO.

Preliminaries

  • The SFT model, trained from pretrained models and high-quality demonstrations, initializes SPO, RLHF, and other preference-optimization methods.Source: Preliminaries · p. 2
  • The Bradley–Terry model represents the probability that humans prefer one response over another using their latent reward values.For a prompt and response pair, the text denotes the preferred response relation as y1 ≻ y2.Source: Preliminaries · p. 2
  • RLHF optimizes an LLM with PPO using a learned reward model, maximizing preference rewards subject to a KL-divergence constraint.Source: Preliminaries · p. 2
  • DPO omits explicit reward-model fitting and uses a reference model for implicit reward modeling; minimizing its loss favors the chosen response over the rejected response.The dataset supplies prompts, preferred responses, and dispreferred responses; the reference model is usually the SFT model.Source: Preliminaries · p. 2

Related Work

Related work covers pretrained and instruction-fine-tuned LLMs, reward-model-based and implicit-reward preference alignment, and approaches to multiple preference dimensions.

Related Work

  • Pretrained models acquire world knowledge through large-scale self-supervised text training, while instruction fine-tuning trains them on task-specific data to follow prompts.The passage says fine-tuned models exhibit strong capabilities across various domains.Source: Related Work · p. 2
  • RLHF trains a reward model from ranked response pairs and uses PPO, whereas DPO uses implicit reward modeling and a simple classification loss.The passage characterizes explicit reward modeling as complex and difficult and says DPO simplifies the fine-tuning pipeline.Source: Related Work · p. 2
  • Safe RLHF and RLPHF use multiple reward models, MORE aggregates shared and dimension-specific reward heads without guaranteeing alignment on each dimension, and rewarded soups merges aligned LLMs.Source: Related Work · p. 2

Methodology

SPO fine-tunes one preference dimension per round while constraining the process to preserve earlier alignments. Its objective extends from two dimensions to arbitrary rounds, with gradient analysis explaining how earlier preferences are protected.

Methodology

  • SPO constrains rewards on earlier dimensions to remain above thresholds while limiting each round’s deviation from the preceding policy.The round also maximizes the reward for its current dimension and prevents significant degradation of previous alignments.Source: Methodology · pp. 2–3

Two-Dimensional Sequential Alignment

  • DPO initializes the first round; the second round maximizes R2 while preserving alignment on the first dimension.The two-dimensional derivation gives a closed-form optimal policy for this constrained objective.Source: Two-Dimensional Sequential Alignment · p. 3
  • The first-dimension reward difference raises gradient weight when preferences agree and lowers it when they conflict, regularizing against degradation.This mechanism contrasts with naive second-round fine-tuning without the first-dimension constraint.Source: Two-Dimensional Sequential Alignment · p. 3

Multi-Dimensional Sequential Alignment

  • Across rounds, SPO builds preference rewards from weighted log-probability ratios of successive policies and minimizes accumulated alignment tax.It optimizes the resulting reward through the Bradley–Terry log-preference objective.Source: Multi-Dimensional Sequential Alignment · pp. 3–4
  • For n ≥ 3, the closed-form reward expression follows by substituting earlier rewards and induction from the first two rounds.The method extends to arbitrary numbers of fine-tuning rounds.Source: Multi-Dimensional Sequential Alignment · pp. 3–4
  • Cached inference results from previous rounds are reused, so sequential training requires no additional inference.The paper describes this as making sequential fine-tuning efficient.Source: Multi-Dimensional Sequential Alignment · p. 4

Experiment

Experiments test SPO on two-dimensional helpfulness and harmlessness tasks and on four-dimensional preferences using Llama 2 and multiple datasets. Across these settings, results support SPO’s ability to preserve earlier alignments while learning additional dimensions.

Experiment Setting

  • Experiments use PKU-SafeRLHF-30k and Helpsteer2 training data, Llama 2 models, dimension-specific DPO baselines, and GPT-4-judged win rates on helpfulness and harmlessness datasets.The modified PKU-SafeRLHF-30k data introduces harmless but unhelpful refusals to make alignment across dimensions harder.Source: Experiment Setting · pp. 4–5

Results with the Modified Training Dataset

  • Across Llama 2 7B and 13B, SPO better aligns on helpfulness and harmlessness than SafeRLHF and RLHF-RS, while balancing the two dimensions better than single-dimension DPO baselines.The reported win-rate comparisons are summarized in Table 1.Source: Table 1 · p. 5, Results with the Modified Training… · p. 5
  • SPO has helpfulness scores similar to DPO-HP and the highest aggregated utility, defined as the product of helpfulness and harmlessness scores.Figure 2 compares the two scores, their product, and rewards during second-round fine-tuning.Source: Results with the Modified Training… · p. 5, Figure 2 · p. 6
  • Increasing α raises helpfulness rewards while lowering harmlessness rewards; SPO’s constraint preserves prior alignment and avoids the overfitting seen with S-DPO after five epochs.The reported overfitting analysis uses the modified dataset; SPO does not overfit to harmlessness even after five epochs.Source: Results with the Modified Training… · p. 6, Table 6 · p. 7

Results with the Original Training Dateset

Results with the Modified Training Dataset

  • Merging separately aligned models can yield imbalanced results: SPO beats merged DPO by 79.0% on AlpacaEval but loses by 41.2% on PKU-SafeRLHF-Test, with harmlessness potentially dominating.The paper suggests simpler refusal patterns may contribute to that dominance.Source: Results with the Modified Training… · pp. 6–7

Experiments with More Preference Dimensions

  • After four-dimensional demonstrative training, SPO produces 50.3% Pareto-optimal responses containing all four special tokens, while S-DPO aligns only with the last dimension.The tokens mark preference on each dimension.Source: Table 4 · p. 7, Experiments with More Preference… · p. 7
  • On Helpsteer2, SPO’s results support multi-dimensional alignment; S-DPO is only slightly better on the last dimension, while merged DPO performs poorly across dimensions.The evaluation covers helpfulness, correctness, coherence, and verbosity.Source: Experiments with More Preference… · p. 7, Table 5 · p. 7

Conclusion and Limitation

SPO aligns LLMs across multiple preference dimensions through iteratively solved constrained optimization, with analysis and experiments supporting preservation of earlier alignments. The experiments cover 7B and 13B models, but not extremely large open models because of computational limits.

Conclusion

  • SPO iteratively optimizes new preference dimensions while preserving alignment from earlier rounds, without explicit reward modeling.The paper derives the learning objective for arbitrary rounds and analyzes gradients to illustrate how prior alignment is maintained.Source: Conclusion and Limitation · p. 7
  • Extensive experiments and studies on different training datasets, evaluation datasets, and preference dimensions confirm the efficacy of Sequential Preference Optimization (SPO) in aligning LLMs across multiple dimensions.Source: Conclusion and Limitation · p. 7

Limitation

  • Extremely large open models were not evaluated because of computational limits.The authors identify evaluation on these models as future work.Source: Conclusion and Limitation · p. 8
Loading 2405.12739v2…