PaperlayerView exact revision on arXiv

Source-linked AI summary

Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss

Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, Jason Weston

arXiv:2312.16682v2cs.CLcs.AI

TL;DR

Pairwise preferences are a popular way to align language models, while Cringe Loss was developed for binary feedback. This paper extends Cringe Loss to preferences with a soft margin and reports that it outperforms established methods on AlpacaFarm; it also finds gains from iterative training.

  • Problem

    The paper asks whether binary-feedback algorithms can be adapted to pairwise preference data, a popular approach to aligning language models.

    Source: §1 · p. 1

  • Method

    Pairwise Cringe Loss adds a differentiable margin that gates Cringe Loss updates according to the probability distance between preferred and dispreferred responses.

    Source: §1 · p. 1

  • Results

    On AlpacaFarm, Pairwise Cringe Loss outperforms Binary Cringe, SFT, DPO, PPO, and Iterative DPO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that iterative versions of Pairwise Cringe and DPO lead to better performance.

    Source: §5 · p. 8

    The training objective omits KL regularization; although the authors observed no degradation without it, they note it could help in some settings.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Practitioners commonly align large language models using pairwise preferences, i.e., given labels of the type response A is preferred to response B for a given input. Perhaps less commonly, methods have also been developed for binary feedback, i.e. training models given labels of type response A is good or bad. We show how an existing performant binary feedback method, the Cringe Loss (Adolphs et al., 2022), can be generalized to the pairwise preference setting using a simple soft margin extension. Pairwise Cringe Loss is straightforward to implement and efficient to train, and we find it outperforms state-of-the-art preference optimization algorithms such as PPO and DPO on the AlpacaFarm benchmark. We show that iterations of training of our model are important for improved results, and that we can generalize DPO to Iterative DPO in the same way.

1. Introduction

The section asks whether binary-feedback algorithms can be adapted to pairwise preferences, and introduces Pairwise Cringe Loss as a soft-margin generalization of Cringe Loss. On AlpacaFarm, it reports that Pairwise Cringe Loss outperforms PPO, DPO, and other compared methods.

1. Introduction

  • Pairwise Cringe Loss extends binary-feedback Cringe Loss to preference pairs with a soft, differentiable margin that gates the loss according to the probability distance between responses.The loss turns off when the preferred response is much more likely, focusing capacity on closer pairs.Source: §1 · p. 1
  • The method can be iterated by generating responses from the improved model and labeling them with a reward model when one is available.The section also introduces Iterative DPO for comparison.Source: §1 · p. 1
  • On AlpacaFarm, Pairwise Cringe Loss outperforms Binary Cringe, SFT, DPO, PPO, and Iterative DPO.The authors describe the benchmark as a more realistic setup for general instruction following.Source: §1 · p. 2
  • For repetition reduction, Pairwise Cringe outperforms Binary Cringe, performs similarly to DPO, and produces slightly better-quality generations.The experiments also compare binary and pairwise preference algorithms in a task chosen for its accurate measurement and greater control.Source: §1 · pp. 1–2

2. Preference Learning with the Cringe Loss

The section reviews how standard Cringe Loss contrasts negative tokens with likely alternatives, then extends it to pairwise preferences with a soft margin. It also describes iterative training with reward-model-labeled completions.

2.1. Standard Cringe Loss

  • The standard objective combines cross-entropy on positive sequences with Cringe Loss on negative sequences, whose contribution is controlled by α.Training on positive responses increases their likelihood; α controls the impact of negative examples.Source: §2.1 · pp. 2–3
  • Standard Cringe Loss contrasts each negative token with a likely alternative, rather than simply lowering the negative token’s probability.The model samples an alternative positive token from its current top-k predictions, excluding the negative token.Source: §2.1 · p. 2

2.2. Pairwise Cringe Loss

  • Pairwise Cringe Optimization starts with original preference data, then uses reward-model-labeled model completions to create updated data for final training.The paper’s procedure generates multiple responses per prompt, selects the best and worst by scalar reward, and repeats the update once.Source: Figure 2 · p. 3, §2.2 · p. 4
  • Pairwise Cringe Loss applies binary Cringe Loss only when the preferred–losing response margin is small, using a sigmoid gate to switch it off as the margin grows.The gate’s temperature controls transition smoothness, while its bias sets the margin at which the loss switches off.Source: §2.2 · pp. 3–4
  • The loss has sequence-level and token-level gradient pathways, combining elements of DPO and PPO with Cringe and Unlikelihood for pairwise preferences.The sigmoid pathway increases the sequence-level margin; the binary Cringe pathway operates on token probabilities.Source: §2.2 · p. 4
  • The authors omit KL regularization because they observed no degradation without it, while noting that it could help in some settings.The objective is therefore not KL-regularized in the reported experiments.Source: §2.2 · p. 4

3. Experiments

The experiments test Cringe Loss variants on repetition reduction and AlpacaFarm instruction following, including preference-source and training-iteration comparisons. Pairwise Cringe achieves the strongest reported AlpacaFarm win rate, and further analyses examine its margin choice and iterative training.

3.1. Reducing Repetitions

  • Pairwise Cringe and DPO bring Repeat@3-gram close to the human baseline (0.892), while Pairwise Cringe has slightly higher F1 than DPO and Binary Cringe trails SFT on F1.The repetition task uses 49,285 preference pairs: beam-blocked generations are preferred over greedy generations with repeated 3-grams; iteration-one Pairwise Cringe has Repeat@3-gram 1.39 and F1 0.1236, with worse F1 than its final model.Source: §3.1 · p. 5

3.2. AlpacaFarm Evaluation

  • 54.7 win rate for Pairwise Cringe exceeds DPO (50.2), PPO (48.5), Binary Cringe (47.7), and the reported SFT baselines on AlpacaFarm.The main evaluation uses human preference data; Table 1 reports the AlpacaFarm results.Source: Table 1 · p. 6, §3.2 · p. 6
  • AlpacaFarm evaluation uses LLM judgments over 805 prompts and averages results across three seeds; iterative Cringe training adds preferences scored by the benchmark reward model from four generated responses per unlabeled prompt.The experiment starts from an SFT model and combines the original preference data with preferences derived from unlabeled instructions.Source: §3.2 · pp. 5–6

3.2.1. ABLATIONS AND FURTHER RESULTS

  • With the same preferences converted to binary feedback, Pairwise Cringe outperforms Binary Cringe in both one- and two-iteration training; at iteration one, their win rates are 52.0 and 45.9, respectively.The benchmark training uses 10k pairwise human-preference examples, converted to positive and negative responses for Binary Cringe.Source: §3.2 · p. 5, §3.2.1 · p. 6
  • The sigmoid-gated soft margin outperforms the hard margin for Pairwise Cringe in both iterations: win rates are 52.0 vs. 47.8 first, and 54.7 vs. 49.9 second.The authors speculate that the soft-margin advantage may come from the gradient available during training.Source: §3.2.1 · p. 6

Iterations of Cringe training improve performance

  • Pairwise Cringe's win rate rises from 52.0 to 54.7 across iterations; Binary Cringe also improves from 45.9 to 47.7, and Iterative DPO rises from 50.2 to 53.6 but remains below iteration-two Pairwise Cringe.Hard-margin and Binary Cringe also benefit from another iteration, though both lag Pairwise Cringe.Source: Iterations of Cringe training improve… · p. 6
  • With simulated preferences, Pairwise Cringe reaches a 50.6 win rate in iteration one and 54.5 in iteration two, ahead of the reported PPO, DPO, and Best-of-N results.The reported comparison gives PPO and DPO 46.8 each, Best-of-N 45.0, and Binary Cringe 45.6.Source: Iterations of Cringe training improve… · p. 7
  • For Cringe variants, α works best in the 0.005–0.01 range and τ in the 1–10 range; b is less important but can help, with first-iteration win rate 52.0 at b = −10 versus 50.9 at b = 0.The authors compare τ's scaling role to DPO's β parameter.Source: Iterations of Cringe training improve… · p. 7

4. Related Work

Alignment training depends on the feedback labels collected, spanning positive-only examples, binary feedback, and pairwise preferences. Prior work developed both preference-ranking methods and binary-feedback losses, including Cringe Loss and iterative alignment approaches.

4. Related Work

  • Feedback format dictates the training algorithm: positive-only examples support fine-tuning, while human feedback can provide binary labels or pairwise preferences.Pairwise preferences are more commonly collected for instruction tuning than binary feedback.Source: §4 · p. 7
  • Pairwise preference learning has a history of margin-based ranking, including SVM-era approaches using hard or soft margins (Herbrich et al., 1998; P´erez-Cruz et al., 2000).Related ranking methods also used web clicks for information retrieval.Source: §4 · p. 7
  • Recent language-model preference learning commonly trains a reward model and applies PPO (Schulman et al., 2017), while DPO (Rafailov et al., 2023) avoids a separate reward model.Other proposed methods include hard-margin approaches SliC, CLICK, and RRHF; one cited study reports evidence that hard margins underperform DPO (Xu et al., 2023a).Source: §4 · p. 7
  • Unlikelihood training lowers the probability of negative examples and was shown to reduce generation flaws (Welleck et al., 2020; Li et al., 2019), while Cringe Loss outperformed unlikelihood and other approaches (Adolphs et al., 2022).The authors chose Cringe Loss as the method to generalize to pairwise preferences.Source: §4 · p. 7
  • Cringe Loss first showed that iterative alignment methods can work well (Adolphs et al., 2022); later work explored ReST (Gulcehre et al., 2023) and theoretical analysis of iterative DPO (Xiong et al., 2023).Source: §4 · p. 7

5. Conclusion

The paper introduces Pairwise Cringe Loss for learning from pairwise preferences and reports that it outperforms competing methods on AlpacaFarm. It also finds benefits from iteration and describes use with either or both feedback types.

5. Conclusion

  • Pairwise Cringe Loss outperforms Binary Cringe and state-of-the-art preference optimization methods including PPO and DPO on AlpacaFarm.Source: §5 · p. 8
  • Iterative versions of Pairwise Cringe and DPO lead to better performance.Source: §5 · p. 8
  • The method is described as efficient and simple to implement, and its binary and pairwise losses can be used together when both feedback types are available.The authors expect it can apply to a wide range of problems.Source: §5 · p. 8

A.1. Algorithm Details

The code implements Cringe Loss as a cross-entropy-based loss, then extends it to pairwise training with a margin-dependent multiplier; a hard-margin variant and classifier-label masking are also shown.

A.1. Algorithm Details

  • CringeLoss subclasses CrossEntropyLoss and returns the final loss alongside its cross-entropy and Cringe components.Source: §A.1 · p. 10
  • For binary feedback, classifier labels mask the Cringe loss so negative-feedback labels receive that term.Source: §A.1 · p. 10
  • The negative-label contrastive term selects top predictions other than the target, samples predictions, and contrasts them with the negative target.Source: §A.1 · p. 10
  • PairwiseCringeLoss combines cross-entropy on preferred responses with Cringe loss on rejected responses, then gates their sum using a sigmoid of the preferred-minus-rejected log-probability margin.Source: §A.1 · p. 11
  • The hard-margin variant also multiplies the combined cross-entropy and Cringe losses by a margin-based multiplier.Source: §A.1 · p. 11

B. Model Hyperparameters

The appendix specifies training resources, optimizer and batch settings, Cringe hyperparameters by feedback type and iteration, and inference decoding parameters.

B. Model Hyperparameters

  • Models use up to 16 80GB NVIDIA A100 GPUs, AdamW, up to 1,000 updates, and batch sizes up to 512; standard decoder-only fine-tuning typically takes under 3 hours.AdamW settings are β1 = 0.9, β2 = 0.95, and ϵ = 1e −08.Source: B. Model Hyperparameters · p. 12
  • All Cringe experiments fix topk = 5, with α, b, and τ settings varying by feedback type and iteration.Human preferences: Binary Cringe α = 0.005 (iteration 1), 0.01 (iteration 2); Hard Margin Cringe α = 0.005, b = −10 (iteration 1), α = 0.01, b = 10 (iteration 2); Pairwise Cringe α = 0.01, b = −10, τ = 10 (iteration 1), α = 0.005, b = −10, τ = 1 (iteration 2). Simulated preferences: Pairwise Cringe α = 0.005, b = 0, τ = 10 (iteration 1), α = 0.01, b = −10, τ = 1 (iteration 2).Source: B. Model Hyperparameters · p. 12
  • Inference uses AlpacaFarm decoding parameters, temperature 0.7, and a 300-token maximum.Source: B. Model Hyperparameters · p. 12
Loading 2312.16682v2…