PaperlayerView exact revision on arXiv

Source-linked AI summary

ORPO: Monolithic Preference Optimization without Reference Model

Jiwoo Hong, Noah Lee, James Thorne

arXiv:2403.07691v2cs.CLcs.AI

TL;DR

Preference alignment methods commonly rely on separate SFT and reference-model stages, while SFT itself does not directly penalize rejected responses. The paper introduces ORPO, which integrates an odds-ratio penalty into SFT without a reference model, and reports strong results from 2.7B and 7B models, including 12.20% on AlpacaEval2.0 and 7.32 in MT-Bench.

  • Problem

    Existing preference alignment methods typically require a separate SFT warm-up phase and reference model, while cross-entropy SFT provides no direct penalty for rejected responses.

  • Method

    ORPO combines conventional SFT loss with an odds-ratio penalty that contrasts chosen and rejected responses without a reference model or separate alignment phase.

  • Results

    Mistral-ORPO-α and Mistral-ORPO-β achieved 11.33% and 12.20% in AlpacaEval2.0 and 7.23 and 7.32 in MT-Bench, exceeding larger state-of-the-art instruction-following models in AlpacaEval.

  • Takeaways & Limitations

    ORPO was consistently preferred over SFT and RLHF across the evaluated scale, while its win rate against DPO increased with model size.

  • Takeaways & Limitations

    The study does not compare a comprehensive range of preference alignment algorithms, scale beyond 7B models, or verify generalizability across diverse domains and datasets.

Abstract

from arXiv · show

While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence. In this paper, we study the crucial role of SFT within the context of preference alignment, emphasizing that a minor penalty for the disfavored generation style is sufficient for preference-aligned SFT. Building on this foundation, we introduce a straightforward and innovative reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the necessity for an additional preference alignment phase. We demonstrate, both empirically and theoretically, that the odds ratio is a sensible choice for contrasting favored and disfavored styles during SFT across the diverse sizes from 125M to 7B. Specifically, fine-tuning Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) with ORPO on the UltraFeedback alone surpasses the performance of state-of-the-art language models with more than 7B and 13B parameters: achieving up to 12.20% on $\text{AlpacaEval}_{2.0}$ (Figure 1), 66.19% on IFEval (instruction-level loose, Table 6), and 7.32 in MT-Bench (Figure 12). We release code and model checkpoints for Mistral-ORPO-$α$ (7B) and Mistral-ORPO-$β$ (7B).

1 Introduction

Language models require instruction tuning and preference alignment for general-domain use, but existing alignment pipelines are typically multi-stage. ORPO combines SFT and preference alignment without a reference model, and achieves strong benchmark results with 7B models.

  • Instruction tuning improves generalization to unseen tasks, but models can still produce harmful or unethical outputs.
  • Preference alignment extends instruction tuning by training on pairwise preference data through methods such as RLHF and DPO.
  • Existing preference alignment methods typically require a separate SFT warm-up phase and a second reference model.
  • ORPO studies SFT within preference alignment and integrates an odds-ratio penalty into SFT, eliminating both the warm-up stage and reference model.

2 Related Works

Prior alignment approaches use reinforcement learning, direct preference optimization, and SFT-based procedures, with different trade-offs in stability, data requirements, and training stages. This motivates more stable and efficient preference alignment methods.

  • Alignment with Reinforcement Learning: RLHF trains a reward model and uses reinforcement learning to maximize scores for chosen responses, but PPO instability and reward-model sensitivity create challenges.
  • Alignment with Reinforcement Learning: Stable preference alignment algorithms are needed because RLHF faces extensive hyperparameter-search demands and sensitivity to reward models.
  • Alignment without Reward Model: DPO combines reward modeling and preference learning, while IPO addresses potential DPO overfitting problems.
  • Alignment without Reward Model: KTO and ULMA remove the need for pairwise preference datasets, whereas another approach merges SFT and preference alignment using reference-response information in the NLL loss.
  • Alignment with Supervised Fine-tuning: SFT supports stable policy updates and has also been used alone with filtered datasets to build human-aligned language models.

3 The Role of Supervised Fine-tuning

SFT adapts pretrained models to desired domains, but cross-entropy does not directly penalize rejected responses. Experiments show that chosen and rejected response likelihoods can rise together, motivating an added penalty for undesired styles.

  • SFT increases log probabilities of pertinent tokens and helps tailor pretrained language models to a desired domain.
  • Cross-entropy loss penalizes low probability for reference answers but provides no direct penalty for non-answer tokens or rejected responses.
  • Rejected-response token probabilities can increase alongside chosen-response probabilities during SFT, which conflicts with preference alignment.
  • Generalization over Both Response Styles: Both chosen and rejected response log probabilities increased during OPT-350M fine-tuning on chosen HH-RLHF responses alone.
  • Penalizing Undesired Generations: ORPO is motivated by appending a dynamic penalty that discourages each query’s disfavored response without manually crafting rejected-token sets.

4 Odds Ratio Preference Optimization

ORPO adds an odds-ratio preference term to conventional SFT loss, contrasting favored and disfavored responses during a single training stage. Its gradients penalize rejected responses and strengthen adaptation toward chosen responses when their likelihood is low.

  • ORPO incorporates an odds-ratio penalty into negative log-likelihood loss to differentiate favored and disfavored generation styles.
  • The odds ratio compares how much more likely the model is to generate the chosen response than the rejected response for the same input.
  • The ORPO objective combines supervised fine-tuning loss with a relative-ratio loss weighted by λ.
  • The SFT component maximizes reference-token likelihood, while the relative-ratio component increases the odds ratio between chosen and rejected responses.
  • Together, the two weighted losses adapt the model to the desired domain while disfavoring rejected-response generations.
  • The gradient contains a penalty term that accelerates updates when the model is more likely to generate rejected responses.
  • The contrastive gradient amplifies updates when corresponding response likelihoods are low, accelerating adaptation toward chosen responses.

5 Experimental Settings

The experiments compare ORPO with SFT, PPO, and DPO across OPT model sizes and two preference datasets, using reward models for PPO and generation assessment.

  • Models: OPT models from 125M to 1.3B parameters are compared across SFT, PPO, DPO, and ORPO.PPO and DPO are trained on top of SFT models trained for one epoch on chosen responses.
  • Datasets: Each training configuration and model is tested on Anthropic’s HH-RLHF and Binarized UltraFeedback datasets.Instances with identical, missing chosen, or missing disfavored responses are filtered out.
  • Reward Models: OPT-350M and OPT-1.3B are trained for one epoch as reward models on each dataset.The reward-model objective is given in Equation 11.
  • Reward Models: The OPT-350M reward model is used for PPO, while the OPT-1.3B reward model assesses generations from fine-tuned models.These models are referred to as RM-350M and RM-1.3B.
  • Evaluation: AlpacaEval1.0 and AlpacaEval2.0 compare ORPO with instruction-tuned leaderboard models, including Llama-2 Chat and Zephyr α and β.MT-Bench is also evaluated against scores for the same leaderboard models.

6 Results and Analysis

ORPO improves instruction-following and preference-alignment outcomes across model sizes and datasets, while avoiding the separate SFT warm-up and reference model required by many baselines. Results also show higher reward distributions and distinct diversity trade-offs relative to competing methods.

  • Single-turn Instruction Following: 81.26% and 9.44% in both AlpacaEvals were achieved by ORPO-trained Llama-2 (7B), exceeding chat versions of Llama-2 at 7B and 13B.UltraFeedback was used with λ=0.2.
  • Single-turn Instruction Following: 87.92% and 11.33% in AlpacaEval2.0 were achieved by Mistral-ORPO-α (7B), exceeding Zephyr α by 1.98% and Zephyr β by 0.34%.The model used UltraFeedback and ORPO with λ=0.1.
  • Single-turn Instruction Following: Over 91% and 12% on AlpacaEval were achieved by Mistral-ORPO-β (7B) after fine-tuning on cleaned UltraFeedback.The comparison uses the same configuration as Mistral-ORPO-α and highlights an effect of dataset quality.
  • Multi-turn Instruction Following: 7.23 and 7.32 in MT-Bench were scored by Mistral-ORPO-α (7B) and β (7B), respectively, without multi-turn conversation data during training.The ORPO-Mistral series achieved comparable results to larger or proprietary models, including Llama-2-Chat (70B) and Claude.
  • Reward Model Win Rate: 78.0% and 79.4% were the highest HH-RLHF win rates against SFT and PPO, while the largest model reached 70.9% against DPO.ORPO outperformed SFT and PPO across all model scales, and its win rate over DPO increased with model size.
  • Reward Model Win Rate: ORPO reached maximum UltraFeedback win rates of 80.5% against SFT and 85.8% against PPO, with its advantage over DPO increasing as model size grew.The UltraFeedback trend followed the HH-RLHF results.

7 Discussion

ORPO integrates preference alignment into SFT without a reference model, using the odds ratio to mildly discriminate disfavored responses while prioritizing favored ones. The analysis links this choice to stable training behavior and lower computational cost than RLHF and DPO.

  • ORPO compares favored and disfavored responses with an odds ratio during SFT, avoiding both a separate warm-up stage and a reference model.The method is presented as a monolithic preference-alignment approach.
  • The odds ratio is preferred because it is more sensitive to the model’s preference understanding and avoids the probability ratio’s more extreme discrimination of disfavored responses.The probability ratio can overly suppress disfavored-response logits when preference alignment is incorporated into SFT.
  • Figure 6 shows that log OR(X2|X1) has a wider range than log PR(X2|X1) for the same input probability pairs.
  • An overly extreme contrast can suppress disfavored-response logits and potentially cause degeneration, motivating caution when applying preference losses during SFT.
  • During ORPO training, the odds consistently increase while rejected-response log probabilities diminish and chosen-response log probabilities remain comparable to SFT behavior.This pattern is reported as evidence that ORPO preserves SFT’s domain-adaptation role while learning preferences.
  • ORPO is computationally more efficient than RLHF and DPO through lower memory allocation and fewer forward passes per batch.Because ORPO directly updates the SFT model, it requires half the forward passes described for reference-model-based training.

8 Conclusion

The paper introduces ORPO as a reference-free monolithic preference-alignment method that revisits SFT’s role in alignment. Across reported evaluations, Mistral-ORPO models achieve strong AlpacaEval2.0 and MT-Bench scores, with code and checkpoints released for reproducibility.

  • ORPO is a reference-free monolithic preference-alignment method built by revisiting the value of SFT.
  • Fine-tuned reward-model evaluations preferred ORPO over SFT and RLHF across model scales, while its win rate against DPO increased with model size.
  • Fine-tuning code and model checkpoints for Mistral-ORPO-α and Mistral-ORPO-β were released to aid reproducibility.

Limitations

The paper identifies incomplete algorithm coverage, limited scaling beyond 7B models, and a need to test broader datasets and domains as directions for future work.

  • The evaluation did not include a comprehensive range of preference-alignment algorithms beyond methods including DPO and RLHF.
  • Future work includes scaling ORPO beyond 7B models and testing datasets with more diverse domains and qualities to assess generalizability.
  • The authors also plan to study ORPO’s internal impact on pretrained language models and its relationship to subsequent preference-alignment algorithms.

B Ablation on Probability Ratio and Odds Ratio

The ablation compares probability-ratio and odds-ratio training and finds that the probability ratio lowers rejected-response log probabilities more aggressively. With the odds ratio, the same behavior occurs only after overfitting.

  • The probability ratio rapidly drives rejected-response log probabilities below -4, whereas the odds ratio reaches the same behavior only after overfitting.
  • The ablation supports the expectation that probability-ratio training lowers rejected-response log probabilities on a larger scale than odds-ratio training.
  • Figure 8 traces batch log probabilities for probability-ratio training on the left and odds-ratio training on the right under the same hyperparameters.

C Experimental Details

The experiments use distinct training configurations for SFT, RLHF, DPO, and ORPO, with additional evaluations on IFEval and a λ ablation using Mistral and UltraFeedback.

  • SFT uses a maximum learning rate of 1e-5 and one training epoch.
  • RLHF hyperparameter settings are provided in Table 5, while the HH-RLHF output lengths are set between 64 and 256 tokens.
  • DPO uses β = 0.1, a learning rate of 5e-6, and three training epochs, although the first or second checkpoint was usually selected.
  • ORPO uses a maximum learning rate of 8e-6 and 10 training epochs, selecting the best model by evaluation loss for several model families.
  • The study reports Mistral-ORPO-α and Mistral-ORPO-β results on IFEval and conducts a λ ablation with values 0.1, 0.5, and 1.0 using Mistral (7B) and UltraFeedback.

E.1 Log Probability

Increasing λ strengthens discrimination against rejected responses, but its downstream effects vary by MT-Bench category: λ = 1.0 favors open-ended categories while λ = 0.1 performs better on deterministic ones.

  • E.1 Log Probability: Larger λ generally strengthens discrimination of rejected responses during ORPO fine-tuning.
  • E.1 Log Probability: With λ = 0.1, chosen-response log probabilities increase while rejected-response log probabilities do not decrease.
  • E.1 Log Probability: With λ = 1.0, chosen and rejected log probabilities both diminish while their margin enlarges.
  • E.2 Downstream Impact: Compared with λ = 0.1, λ = 1.0 performs worse on extraction, math, and reasoning but better on STEM, humanities, and roleplay in MT-Bench.
  • ORPO consistently shifts the SFT reward distribution to the right on HH-RLHF across OPT2-125M, OPT2-350M, and OPT2-1.3B.
  • Mistral-ORPO-β (7B) surpasses Llama-2 Chat (13B) and Llama-2 Chat (70B) in most MT-Bench categories, while lacking coding and math skills.
Loading 2403.07691v2…