PaperlayerView exact revision on arXiv

Source-linked AI summary

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing

Biqing Qi, Pengfei Li, Fangyuan Li, Junqi Gao, Kaiyan Zhang, Bowen Zhou

arXiv:2406.05534v1cs.AIcs.CLcs.LG

TL;DR

DPO's offline design and inability to use historical information make cross-domain preference alignment vulnerable to forgetting. OFS-DPO uses competing fast and slow LoRA modules, while COFS-DPO combines fast modules across domains; the paper reports stronger in-domain and cross-domain results in the stated tasks.

  • Problem

    DPO is designed for offline local data and cannot use historical information effectively, leaving cross-domain continual alignment vulnerable to forgetting and costly relearning.

    Source: §1 · p. 1

  • Method

    OFS-DPO pairs fast and slow LoRA modules under the DPO objective with a preference-gap regularizer; COFS-DPO combines optimal fast modules across domains.

    Source: §1 · p. 2

  • Results

    OFS-DPO outperforms DPO and competitive methods in-domain, while COFS-DPO significantly surpasses competitive baselines on cross-domain summarization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    COFS-DPO's linear combination of task-specific fast modules is reported to preserve strong individual-domain memory while approximating optimal parameters across the task domain.

    Source: §1 · p. 2

    The theoretical analysis assumes bounded gradients and bounded parameter distances across time, with the cross-domain setup specified for two task distributions.

    Source: §3.3 · p. 5, §3.4 · p. 6

Abstract

from arXiv · show

Direct Preference Optimization (DPO) improves the alignment of large language models (LLMs) with human values by training directly on human preference datasets, eliminating the need for reward models. However, due to the presence of cross-domain human preferences, direct continual training can lead to catastrophic forgetting, limiting DPO's performance and efficiency. Inspired by intraspecific competition driving species evolution, we propose a Online Fast-Slow chasing DPO (OFS-DPO) for preference alignment, simulating competition through fast and slow chasing among models to facilitate rapid adaptation. Specifically, we first derive the regret upper bound for online learning, validating our motivation with a min-max optimization pattern. Based on this, we introduce two identical modules using Low-rank Adaptive (LoRA) with different optimization speeds to simulate intraspecific competition, and propose a new regularization term to guide their learning. To further mitigate catastrophic forgetting in cross-domain scenarios, we extend the OFS-DPO with LoRA modules combination strategy, resulting in the Cross domain Online Fast-Slow chasing DPO (COFS-DPO). This method leverages linear combinations of fast modules parameters from different task domains, fully utilizing historical information to achive continual value alignment. Experimental results show that OFS-DPO outperforms DPO in in-domain alignment, while COFS-DPO excels in cross-domain continual learning scenarios.

1 Introduction

DPO offers resource-efficient preference alignment, but its offline design leaves it ill-suited to streaming preferences and cross-domain continual learning. The paper proposes fast-slow competition and cross-domain LoRA combinations, reporting gains in the stated in-domain and cross-domain tasks.

1 Introduction

  • DPO trains directly on preference data without reward models, but its offline design cannot use preference streams or historical information effectively, contributing to cross-domain forgetting and costly relearning.The stated costs include computation and recollecting annotated data.Source: §1 · p. 1
  • Existing online preference-alignment methods either construct resource-intensive reward models or use LLM-generated feedback without flexible modular design for efficient learning and memory retention.The proposed competitive components are intended to retain DPO's resource efficiency while handling incoming preference streams.Source: §1 · p. 2
  • Online regret bounds contain a min-max term whose two terms share an optimization objective, motivating the paper's intraspecific-competition design.The authors relate this pattern to GAN objectives and intraspecific competition.Source: §1 · p. 2
  • OFS-DPO retains the DPO objective and uses fast and slow LoRA modules with a preference-probability-gap regularizer; its analysis reports lower empirical regret, more stable gradients, and faster convergence.The DPO objective is retained to preserve alignment with human preferences and limit deviation from the reference model.Source: §1 · p. 2
  • OFS-DPO outperforms DPO and other competitive methods in in-domain tasks, while COFS-DPO significantly surpasses competitive baselines on cross-domain summarization.The reported in-domain tasks include controlled sentiment generation, summarization, and single-turn dialogue.Source: §1 · p. 2
  • COFS-DPO maintains domain-specific memories and jointly optimizes linear combinations of optimal fast modules from different tasks for cross-domain alignment.The authors report strong memory retention in individual domains and performance comparable to theoretically optimal parameters across the task domain.Source: §1 · p. 2

2 Related Works

Related work describes DPO's downstream use and variants for alignment objectives, then reviews continual-learning approaches and their challenges in preserving prior knowledge under domain shifts.

Direct Preference Optimization

  • DPO has been applied across downstream tasks, while MODPO balances multiple alignment objectives and DPOP modifies the objective for preference datasets with smaller edit distances [20, 21, 22, 23, 24, 25, 26].The passage lists multimodal generation, translation, and other applications before describing these variants.Source: §2 · p. 2

Continual Learning

  • Continual learning addresses rapid adaptation to dynamic in-domain streams and preservation of prior task features in cross-domain settings to limit catastrophic forgetting [27] [28].Reviewed approaches include regularization, replay, and domain generalization [29] [28].Source: §2 · p. 3
  • Replay methods retain past data or experiences and therefore require memory, while domain-generalization methods may struggle under substantial cross-domain distribution shifts [34] [35] [36] [37] [38] [39].Regularization methods balance the integration of new and old knowledge [6] [30] [31] [32] [33].Source: §2 · p. 3

3 Methodology

The methodology frames preference alignment as online learning and uses regret analysis to motivate fast-slow competition. It develops OFS-DPO for in-domain learning and extends it with domain-specific LoRA memories and their combination for cross-domain learning.

3.1 Preliminaries

  • Each online preference example is a prompt paired with a desired and an undesired response, drawn from a task distribution updated over T time steps.The setup distinguishes two task distributions and their time horizons for cross-domain comparisons.Source: §3.1 · p. 3

3.2 Motivation and Theoretical Analysis

  • The regret upper bound contains min-max terms, motivating fast and slow modules that pursue each other toward the best offline decision.The analysis interprets the min and max terms as sharing the goal of reducing cumulative loss.Source: §3.2 · pp. 3–4

3.3 In Domain: Online Fast-Slow Chasing DPO

  • OFS-DPO retains the DPO objective, applies it to fast and slow LoRA modules, and regularizes their preference-probability gap.The modules are optimized alternately at different frequencies, with their roles swapped at regular intervals when the fast module has higher DPO loss.Source: §3.3 · pp. 4–5
  • With fast-slow modules, OFS-DPO lowers the empirical regret bound; its regularizer also sustains gradient updates as original DPO gradients diminish.The lower-bound result assumes bounded gradients and bounded model parameters.Source: §3.3 · pp. 5–6

3.4 Cross Domain: Online Fast Slow Chasing DPO

  • COFS-DPO preserves domain-specific fast-module memories and optimizes their combination over the joint memory distribution to mitigate cross-domain forgetting.The combination is formed from fast modules learned separately for the two task domains.Source: §3.4 · p. 6, Figure 2 · p. 6

Definition 3.4.1. (Experience regret of cross-domain tasks: Dual-Task Regret)

4 Experiments

Across in-domain and cross-domain preference tasks, OFS-DPO improves alignment relative to DPO and PPO, while COFS-DPO shows stronger retention after training on a second domain. The experiments also test training settings and ablations of the regularization, fast–slow updates, and retained-sample count.

4.1 Experimental settings

  • Experiments test online in-domain sentiment, summarization, and dialogue, plus cross-domain continual learning across relationships and 28 other Reddit post categories.The cross-domain setup uses Task-1 for relationships and Task-2 for the remaining categories; evaluation tracks ROUGE, rPMS, and SFR.Source: §4.1 · pp. 7–8

4.2 Evaluation results on in-domain tasks

  • OFS-DPO improves win rates over DPO and PPO by about 8% on controlled sentiment generation and 5% on single-turn dialogue.GPT-4 served as a human proxy for evaluating generated content against actual preference data.Source: Table 1 · p. 7, §4.1 · p. 7, §4.2 · p. 8

4.3 Evaluation results on cross-domain tasks

  • COFS-DPO reaches SFR around -0.6 with GPT2-s and around -1.5 with LLaMA3, outperforming baselines on cross-domain memory retention.The LLaMA3 result is nearly twice as good as the best-performing PPO variant.Source: Table 2 · p. 8, §4.3 · p. 9

4.4 Ablation Studies

  • On IMDB, win rates remain around 50% at α = 0, OFS-DPO leads PPO by at least 10% across tested learning-rate multipliers, and larger batches and contrast periods stabilize win rates.Figure 3 also reports more sustained gradient stability for OFS-DPO than original DPO.Source: §4.4 · p. 9, Figure 3 · p. 9
  • With two-task retained-sample totals of 100, 200, 500, and 1000, COFS-DPO achieves optimal historical-preference retention at 500; larger totals add no improvement.Table 3 reports the sample-size comparison.Source: §4.4 · p. 9, Table 3 · p. 9

5 Conclusion

OFS-DPO uses competition between fast and slow modules under a shared objective for continual preference learning, with regret-bound and gradient-stability guarantees. COFS-DPO extends this approach to cross-domain alignment by combining LoRAs from different domains.

5 Conclusion

  • COFS-DPO aligns preferences across domains by replacing a task domain’s optimal LoRA with a linear combination of LoRAs from different domains.The approach is validated experimentally and theoretically.Source: §5 · p. 9
  • OFS-DPO uses competition between fast and slow modules under the same objective for continual preference learning, with theoretical guarantees for regret bounds and gradient stability.Source: §5 · p. 9
  • The authors present these intraspecific-competition methods as potential solutions for online human preference alignment with broad multi-domain applicability.Source: §5 · p. 9

A.1 The Proof of Lemma 3.2.1

The proof invokes a regret-bound lemma for Hedge on a finite hypothesis class and uses it to derive a result, noting a constant-factor difference in the regret definition.

A.1 The Proof of Lemma 3.2.1

  • Lemma A.1.1 considers an algorithm using Hedge on a finite hypothesis class H′ and defines its expected regret.Source: §A.1 · p. 13
  • The paper states that its regret R(T) differs from the referenced theorem’s REGRET(A, D) by at most a constant factor of ln 2.The passage identifies ln 2 as the maximum value of l(θ, x).Source: §A.1 · p. 13
  • The proof derives its result by leveraging Lemma A.1.1 on regret bounds.Source: §A.1 · p. 13

A.2 The Proof of Theorem 3.3.1

The proof concludes that OFS-DPO has a smaller lower bound of regret than vanilla DPO, under stated boundedness assumptions and probability bounds derived using Hoeffding’s inequality.

A.2 The Proof of Theorem 3.3.1

  • For positive integer k, assumptions 3.3.1 and 3.3.2 imply |g_{t−k}(θ_{t−k}−θ_t)| ≤ Gd and l(θ_T,x_i)−l(θ_i,x_i) ≥ −Gd.Source: §A.2 · p. 13
  • The proof introduces fast and slow modules and bounds a loss difference using the minimum of their losses.Source: §A.2 · p. 14
  • The derivation applies Hoeffding’s inequality to loss differences around their expectations and states a resulting inequality holds with probability 1 − δ.Source: §A.2 · pp. 13–14
  • OFS-DPO has a smaller lower bound of regret than vanilla DPO.The proof gives an intermediate loss-difference bound of 2Gd + 2c with probability (1 − δ0)^2.Source: §A.2 · p. 14

A.3 The Proof of Proposition 3.3.1

The proposition’s analysis contrasts the diminishing DPO gradient coefficient with OFS-DPO’s added regularization gradient, while stating that the fast and slow modules converge to the same objective.

A.3 The Proof of Proposition 3.3.1

  • The DPO gradient coefficient tends toward zero during training, with the passage bounding it by ϵ.Source: §A.3 · p. 15
  • OFS-DPO adds a regularization term that influences gradients during model iteration, alongside the original DPO-form gradient term.The passage states that the original DPO-form term also diminishes toward zero during training.Source: §A.3 · p. 15
  • During training, the fast and slow modules converge toward the same objective, with their preferred and dispreferred response probabilities approaching 1 and 0, respectively.Source: §A.3 · p. 15

A.4 The Proof of Theorem 3.4.1

The proof frames cross-domain empirical regret using an interpolated parameter vector, while the proposed algorithms use fast and slow modules and combine task-specific LoRAs for cross-domain learning.

A.4 The Proof of Theorem 3.4.1

  • Theorem A.4.1 states that a data perturbation can make target-distribution training loss equal source-distribution training loss on perturbed data.Source: §A.4 · p. 16
  • Theorem A.4.2 states that source-data perturbation can be represented by a model-weight perturbation with equal training loss.Source: §A.4 · p. 16
  • Cross-domain empirical regret is expressed around the interpolated parameter vector θ(β) = θ0 + β∆θ(1) + (1 − β)∆θ(2).Source: §A.4 · p. 16

B Proposed Algorithms

  • OFS-DPO uses fast and slow modules to simulate intraspecific competition, while COFS-DPO trains on task 2 from task 1’s fast-module parameters.Source: B Proposed Algorithms · p. 20
  • COFS-DPO combines fast-module parameters from both tasks, selecting weights using retained data for generalization on the overall distribution.Source: B Proposed Algorithms · p. 20

C.1 In-domain experiments

The in-domain experiments assess continual learning across three tasks, using one epoch per experiment and GPT-4 evaluation with 120 test samples.

C.1 In-domain experiments

  • Experiments cover three in-domain tasks, run for one epoch each on a single NVIDIA A800 80G GPU.Source: §C.1 · p. 20
  • GPT-4 evaluation collects 120 test samples, with prompts detailed in Table 5.Source: §C.1 · p. 21, Table 5 · p. 21
  • Table 4 lists hyperparameters for the different in-domain tasks.Source: Table 4 · p. 21

C.2 Cross-domain experiments

The cross-domain experiments split data into two task domains and evaluate whether methods retain old knowledge while learning new knowledge.

C.2 Cross-domain experiments

  • The setup splits data into two task domains to assess retaining old knowledge while learning new knowledge, using GPT2-s and LLaMA3.Source: §C.2 · p. 22
  • After training in both domains, COFS-DPO forms the final model by weighted fusion of the two LoRAs.Source: §C.2 · p. 22
  • Cross-domain metrics adapt the CPPO setup; Table 7 lists the metrics, including rPMs and ROUGE scores based on reference-PM alignment.Source: §C.2 · p. 22, Table 7 · p. 22
Loading 2406.05534v1…