PaperlayerView exact revision on arXiv

Source-linked AI summary

LiPO: Listwise Preference Optimization through Learning-to-Rank

Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J. Liu, Xuanhui Wang

arXiv:2402.01878v3cs.CLcs.LG

TL;DR

Ranked-list feedback is common in LM alignment, but preference optimization lacks a thorough study of directly learning from lists. The paper frames alignment as Learning-to-Rank through LiPO and finds that LiPO-λ outperforms existing methods across several preference-alignment tasks. Its studies are offline, and the number of reward-model labels grows quadratically with list size.

  • Problem

    Preference optimization methods commonly use pairwise data, while ranked-list feedback is practical and has not received a thorough study as a direct basis for LM alignment.

    Source: §1 · pp. 1–2

  • Method

    LiPO formulates LM alignment as listwise ranking connected to Learning-to-Rank, and examines objectives including LiPO-λ's listwise-aware pair weighting.

    Source: §1 · p. 2, §2.3 · p. 3, §3 · p. 4

  • Results

    LiPO-λ outperforms pairwise and listwise DPO, SLiC, PRO, and other LTR-motivated baselines across Reddit TL;DR, AnthropicHH, and OpenAssistant.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The LiPO studies identify LiPO-λ as the best approach among the examined LTR losses, with competitive performance across several tasks.

    Source: §7 · p. 9

    The study is limited to offline learning, and reward-model labels grow quadratically with list size.

    Source: §8 · p. 9

Abstract

from arXiv · show

Aligning language models (LMs) with curated human feedback is critical to control their behaviors in real-world applications. Several recent policy optimization methods, such as DPO and SLiC, serve as promising alternatives to the traditional Reinforcement Learning from Human Feedback (RLHF) approach. In practice, human feedback often comes in a format of a ranked list over multiple responses to amortize the cost of reading prompt. Multiple responses can also be ranked by reward models or AI feedback. There lacks such a thorough study on directly fitting upon a list of responses. In this work, we formulate the LM alignment as a \textit{listwise} ranking problem and describe the LiPO framework, where the policy can potentially learn more effectively from a ranked list of plausible responses given the prompt. This view draws an explicit connection to Learning-to-Rank (LTR), where most existing preference optimization work can be mapped to existing ranking objectives. Following this connection, we provide an examination of ranking objectives that are not well studied for LM alignment with DPO and SLiC as special cases when list size is two. In particular, we highlight a specific method, LiPO-$\lambda$, which leverages a state-of-the-art \textit{listwise} ranking objective and weights each preference pair in a more advanced manner. We show that LiPO-$\lambda$ can outperform DPO variants and SLiC by a clear margin on several preference alignment tasks with both curated and real rankwise preference data.

1 Introduction

The paper addresses the gap between common pairwise preference optimization and practical ranked-list feedback, introducing LiPO to study LM alignment through Learning-to-Rank. It examines ranking objectives and highlights LiPO-λ, which outperforms existing methods across several preference-alignment tasks.

1 Introduction

  • Ranked-list feedback can amortize the cost of reading a prompt, but preference optimization lacks a thorough study of the listwise ranking perspective.The paper notes that prior work was often ad hoc; the DPO paper briefly considered Plackett–Luce without experiments, while PRO studied it more formally.Source: §1 · p. 2
  • LiPO formulates LM alignment as listwise ranking and connects preference optimization to Learning-to-Rank, where existing methods map to ranking objectives.This framework supports examining which objectives are effective for LM alignment.Source: §1 · p. 2
  • LiPO-λ applies a state-of-the-art ranking objective with listwise-aware weighting of sampled pairs, unlike methods using uniform or less informed weighting.The framework is presented as a way to analyze the pitfalls of existing listwise objectives.Source: §1 · p. 2
  • LiPO-λ outperforms pairwise and listwise DPO, SLiC, PRO, and other LTR-motivated baselines on Reddit TL;DR, AnthropicHH, and OpenAssistant.The method uses listwise-aware weighting of sampled pairs to optimize a ranking metric.Source: §1 · p. 2
  • The paper provides a comprehensive investigation of ranking objectives, including under-studied listwise objectives, and analyzes existing methods through the LTR perspective.DPO, RRHF, and SLiC can be mapped to pairwise ranking objectives; pairwise data corresponds to list size two.Source: §1 · p. 2

2 The LiPO Framework

LiPO frames LM alignment as ranking candidate responses for each prompt, extending preference optimization from pairs to ranked lists. It expresses alignment through ranking losses, maps familiar methods to specific objectives, and identifies information ignored by existing approaches.

2.1 Preliminary

  • DPO learns from prompt–response pairs with the SFT policy as reference and β controlling KL divergence; its implicit reward is aligned directly to preferences.The LM policy assigns a response distribution to each prompt.Source: §2.1 · pp. 2–3

2.2 The Listwise Formulation

  • LiPO represents each prompt with a response list that may come from an SFT policy or other sources, with higher real-valued labels indicating better responses.Pairwise data is the special case K = 2; labels may come from human raters or a reward model.Source: §2.2 · p. 3

2.3 LM alignment as Learning-to-Rank

  • LiPO casts prompts as Learning-to-Rank queries and responses as documents, then minimizes a ranking loss based on response labels and predicted scores.The framework defines the loss over the training distribution of prompts, response lists, and labels.Source: §2.3 · p. 3

2.4 Ranking Losses in Existing Work

2.5 Pitfalls of Existing Work

  • Pairwise methods ignore listwise permutation information, while existing methods also discard label values and force an ordering even when labels are tied.Lists with labels (0.99, 0.50, 0.01) and (0.51, 0.50, 0.49) are treated alike by methods that use only rank order.Source: §2.5 · p. 4

3 LiPO-λ

LiPO-λ instantiates the LiPO framework with LambdaLoss, using listwise-aware pair weights based on ranking and label information. The paper describes its weighting and permutation choices as addressing identified pitfalls and reports that the loss can optimize DCG.

3 LiPO-λ

  • LiPO-λ applies the state-of-the-art LambdaLoss ranking objective within LiPO to address identified pitfalls and performs well empirically.The framework allows exploration of a broader family of Learning-to-Rank objectives.Source: §3 · p. 4
  • Its Lambda weight combines differences in response gains and rank discounts, with the original configuration using gains derived from labels and logarithmic rank discounts.The discount uses the model-induced rank position, making the method listwise despite its pairwise expression.Source: §3 · pp. 4–5
  • LiPO-λ uses model-prediction-induced permutations rather than static label ranks, which prior work associates with smoother optimization of non-smooth ranking objectives.The gain function also incorporates label values, unlike virtually all existing methods discussed.Source: §3 · p. 5
  • LiPO-λ is a weighted DPOBT variant over ranked-list pairs, with each pair weighted according to its listwise permutation and ranking-metric change when swapped.The loss has been shown to optimize DCG, a ranking metric with consistent distinguishability properties.Source: §3 · p. 5

4 Other Ranking Losses

The section considers pointwise and softmax cross-entropy objectives among ranking losses studied under LiPO.

4 Other Ranking Losses

  • The pointwise Mean Squared Error objective is included among the ranking losses studied under LiPO.Source: §4 · p. 5
  • The pointwise sigmoid cross-entropy loss is another objective considered under LiPO.Source: §4 · p. 5
  • Softmax cross-entropy is considered as ListNet and maps to the recent NCE method.Source: §4 · p. 5

5 Experiments

The experiments compare ranking losses on Reddit TL;DR and AnthropicHH, with further evaluation on OpenAssistant. LiPO-λ leads across the reported comparisons, including human-ranked data.

5 Experiments

  • The experiments use Reddit TL;DR summarization and AnthropicHH dialogue data, with further verification on OpenAssistant.For each task, the setup trains an SFT policy, samples responses, and uses a reward-ranking model to construct preference labels; OpenAssistant uses direct human ranks.Source: §5 · p. 5, §5.4 · p. 8

5.1 Performance Comparison

  • LiPO-λ improves upon all baselines by leveraging listwise data and label values, while pointwise methods, PRO, and listwise Softmax are not competitive.The comparison uses T5-large policies and list size 8; Table 1 reports proxy-reward and PaLM 2-L-IT win rates against the SFT target.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Ablation Studies and Analysis

  • Most methods benefit from lists larger than two, but LiPO-λ with lambda-loss is the only method reported to benefit robustly from longer lists.This list-size ablation is on Reddit TL;DR.Source: §5.2 · p. 6, Figure 3 · p. 7
  • DCG weights are most effective on Reddit TL;DR and AnthropicHH, supporting the use of both gain and rank-discount functions.The comparison includes constant weights, constant δ gain, and constant δ discount.Source: §5.2 · p. 6, Figure 3 · p. 7
  • With T5-XXL policy models, DPOBT, PRO, and LiPO-λ all scale up well, and LiPO-λ is competitive on both tasks.Table 2 reports proxy-reward and AutoSxS win rates against the SFT target for list size 8.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 Human Evaluation Results

  • Human raters found LiPO-λ better than DPOBT and PRO on both evaluated tasks.The Reddit TL;DR evaluation involved 50 raters; Table 3 compares the three losses on two tasks.Source: §5.3 · p. 7, Table 3 · p. 8

5.4 Direct Alignment on Human Ranked Responses

  • In the reported comparison, LiPO-λ performs best using rank-only annotations directly from humans, after the best response is selected as the SFT target and followed by listwise calibration.Source: §5.4 · p. 8

6 Related Work

Related work describes preference optimization alternatives to RLHF and connects language-model alignment objectives with Learning-to-Rank. The paper focuses on optimization objectives, assuming reliable reward-model feedback.

6 Related Work

  • RLHF fits a reward function from preferences and then uses reinforcement learning to fine-tune the language model, but incurs computational costs and extensive hyperparameter tuning.Source: §6 · p. 8
  • Alternatives include pairwise hinge and logistic objectives, while PRO uses list MLE and RRHF applies pairwise loss to pairs from listwise data.DPO uses pairwise logistic loss without a reward model; the paper focuses on the optimization-objective perspective.Source: §6 · p. 8
  • Learning-to-Rank spans pointwise, pairwise, and listwise objectives; this work connects language-model alignment methods to those ranking objectives.The paper assumes reliable reward-model feedback and uses standard language-model architectures, leaving ranking architectures and biased feedback outside its scope.Source: §6 · p. 9

7 Conclusion

LiPO frames language-model alignment with response lists as a Learning-to-Rank problem and generalizes preference-optimization methods. Its studies highlight LiPO-λ, built on state-of-the-art ranking objectives, as a competitive approach across several tasks.

7 Conclusion

  • LiPO provides a framework for language-model alignment using a list of responses for each prompt, connected to LTR techniques.Source: §7 · p. 9
  • LiPO-λ is highlighted as the best approach among the studied LTR losses, with competitive performance across several tasks.It builds on state-of-the-art ranking objectives.Source: §7 · p. 9
  • The framework generalizes recent preference-optimization methods and analyzes their pitfalls from an LTR perspective.Source: §7 · p. 9

8 Limitations

The study is limited to offline learning from listwise responses. The authors identify online learning and more efficient label construction as future directions.

8 Limitations

  • The study evaluates listwise-response algorithms only in an offline setting.Source: §8 · p. 9
  • Online learning from response lists elicited by the policy being trained is proposed as a way to reduce distribution shift.Source: §8 · p. 9
  • Reward-model labels grow quadratically with list size; partial comparisons are suggested for reconstructing relevance scores more efficiently.In the offline setting, this cost is incurred during preprocessing.Source: §8 · p. 9

A Proofs

The proofs relate ranking losses to existing objectives through substitutions in their scores. For list size two, a proof reduces one objective to the DPO equation.

A Proofs

B Reproducibility

The authors report publicly available datasets and model checkpoints, use the open-source RAX library for ranking objectives, and are preparing organized code for release.

B Reproducibility

  • All datasets and the policy and reward model checkpoints are publicly available.Source: B Reproducibility · p. 12
  • The implementation uses open-source RAX ranking objectives, with Algorithm 1 showing loss computation in Python under Jax.Source: B Reproducibility · p. 12
  • The authors say they are in the process of releasing well-organized code to support reproducibility.Source: B Reproducibility · p. 12

C.0.1 Implementation details

The implementation details describe an automated pairwise evaluation protocol and provide few-shot examples for judging Reddit summaries and assistant responses.

C.0.1 Implementation details

  • AutoSxS is used to guard against reward-model reward hacking, since a policy’s higher reward-model win rate does not necessarily indicate better quality.Its few-shot judge is not trained on the reward model’s training data, and the authors say systematic study of its consistency and quality is beyond scope.Source: §C.0.1 · p. 12
  • PaLM 2-L-IT evaluates eight decoded samples with four response-order flips, assigning A, B, or tie scores and declaring a win or loss when the average score magnitude exceeds 0.35.The evaluation uses few-shot prompts; AutoSxS consistency and quality are not systematically studied in this work.Source: §C.0.1 · p. 12

C.0.2 Reddit TL;DR Few-Shot Prompts

  • Reddit TL;DR examples judge summary coverage, factual accuracy, concision, and fluency, including whether a summary introduces a major factual error.Other examples favor preserving who confronted the author, including a salient detail concisely, and avoiding redundant or unfluent wording.Source: §C.0.2 · pp. 13–14

C.0.3 AnthropicHH Few-shot Prompts

  • AnthropicHH examples prefer responses that engage with the user’s perspective, provide specific useful detail, and contextualize guidance rather than merely repeat prior information.The examples include a more expansive response to a Moon discussion, diverse cooking suggestions, and a U.S.-specific taxi-tip recommendation.Source: §C.0.3 · pp. 14–15

D HumanSxS Details

  • Figure 4 and Figure 5 show examples of the human-evaluation tasks for Reddit TL;DR and AnthropicHH, respectively.Source: D HumanSxS Details · p. 15

E Human Evaluation on XXL Policy

Human evaluation on the XXL policy favors LiPO-λ over two DPO variants; the reward-ranking model is a pairwise T5-XXL classifier evaluated on summarization and assistant tasks.

E Human Evaluation on XXL Policy

F Reward-ranking Model Details

  • The pairwise T5-XXL reward-ranking model reaches 73.23% validation accuracy on summarization and 69.75% on the AI assistant task.It is trained on human preference data to approximate P(y1 ≻ y2|x).Source: F Reward-ranking Model Details · p. 15
  • The reward model receives a context with two summaries or responses and outputs A or B, using the probability of A as the preference-probability estimate.Response pairs and their labels are randomly flipped to remove positional bias.Source: F Reward-ranking Model Details · p. 15
  • Training uses 64 TPU-v3 chips for about two hours per experiment, with batch size 128, up to 1024 input tokens, two output tokens, and Adafactor at learning rate 0.001.Source: F Reward-ranking Model Details · p. 15
  • Figure 4 and Figure 5 provide human-evaluation task examples for Reddit TL;DR and AnthropicHH, respectively.Source: Figure 4 · p. 16, Figure 5 · p. 17

Related papers

Loading 2402.01878v3…