Source-linked AI summary
Preference Ranking Optimization for Human Alignment
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, Houfeng Wang
TL;DR
LLM alignment is needed because pretrained models contain misleading and harmful content, while RLHF is complex and existing methods underuse long preference rankings. PRO directly trains LLMs in SFT using recursive contrasts over rankings of arbitrary length, and experiments report stronger alignment, including reward scores similar to ChatGPT with 7B parameters. The paper also identifies contextual misleading and reward-model/model-size effects as boundaries for current results.
Problem
LLM pretraining includes misleading, toxic, and detrimental content, while RLHF is complex and existing methods often reduce longer rankings to pair-wise contrasts.
Method
PRO directly fine-tunes the policy LLM in SFT by recursively applying one-to-N contrasts across human-ranked responses, with optional candidate augmentation and reward-model weighting.
Results
PRO surpasses competitive baselines with 2-rankings, improves as rankings lengthen, and achieves reward scores similar to ChatGPT using 7B parameters.
Takeaways & Limitations
PRO demonstrates that multi-positional and multi-dimensional preference contrasts can effectively and efficiently align LLMs to human preference.
Takeaways & Limitations
The authors report occasional failures to generate positive responses under contextual misleading and leave larger-model scaling for future work.
Abstract
from arXiv · showhide
Large language models (LLMs) often contain misleading content, emphasizing the need to align them with human values to ensure secure AI systems. Reinforcement learning from human feedback (RLHF) has been employed to achieve this alignment. However, it encompasses two main drawbacks: (1) RLHF exhibits complexity, instability, and sensitivity to hyperparameters in contrast to SFT. (2) Despite massive trial-and-error, multiple sampling is reduced to pair-wise contrast, thus lacking contrasts from a macro perspective. In this paper, we propose Preference Ranking Optimization (PRO) as an efficient SFT algorithm to directly fine-tune LLMs for human alignment. PRO extends the pair-wise contrast to accommodate preference rankings of any length. By iteratively contrasting candidates, PRO instructs the LLM to prioritize the best response while progressively ranking the rest responses. In this manner, PRO effectively transforms human alignment into aligning the probability ranking of n responses generated by LLM with the preference ranking of humans towards these responses. Experiments have shown that PRO outperforms baseline algorithms, achieving comparable results to ChatGPT and human responses through automatic-based, reward-based, GPT-4, and human evaluations.
Introduction
The paper frames human alignment as necessary because LLMs contain harmful content, while RLHF is complex and pair-wise methods underuse longer preference rankings. PRO addresses this by directly optimizing ranked responses in SFT and reports stronger alignment as rankings expand.
- Motivation: LLMs can contain misleading, toxic, and detrimental content, motivating alignment to human values for secure and manageable AI systems.
- Limitations of RLHF: RLHF is more complex than supervised learning, prone to optimization instability, and sensitive to hyperparameters.
- Research gap: Existing methods often reduce longer preference rankings to pair-wise contrasts, limiting macro-level distinction among candidates.
- PRO: PRO extends pair-wise contrasts to one-to-N contrasts over preference rankings of arbitrary length, recursively prioritizing each response against those ranked below it.
- Candidate construction: PRO augments rankings with candidates of varied quality and alignment, including self-bootstrapped responses labeled by an additional reward model.
- Results: PRO surpasses competitive baselines with 2-rankings, improves more with longer rankings, and achieves reward scores similar to ChatGPT using 7B parameters.
Methodology
PRO converts pair-wise policy optimization into recursive multi-positional contrasts over ranked candidates, optionally weighting penalties by reward-model score gaps. It remains an SFT objective while incorporating supervised language-quality training and flexible ranking sources.
- From RLHF to PRO: PRO transfers Bradley-Terry-style pair-wise reward contrast directly to the policy LLM, treating the LLM as both reward model and policy network.
- Multi-candidate contrast: Expanding the candidate set exposes the policy to more samples, replacing trial-and-error experience with one-to-N contrast over ranked responses.
- Multi-positional contrast: Recursive contrasts exploit the full ranking by treating each successive response as positive against all remaining lower-ranked candidates.
- Training objective: PRO combines its preference objective with the NLL loss of the top candidate, balancing text quality and human preference through β.
- Comparison with RLHF: Unlike RLHF, PRO learns from multiple samples assembled into long rankings within SFT settings, which can be more efficient.
- Flexible ranking sources: PRO can use human rankings or rankings generated from multiple LLMs and scored by an additional reward model.
- Differentiated contrast: Reward-model score gaps enable differentiated penalties: larger preference gaps produce stronger penalties through lower temperatures.
- Results: Dynamic temperature significantly improves performance when optimizing LPRO alone and provides gains when jointly optimizing LPRO and LSFT.
Experiments
Experiments evaluate PRO on HH-RLHF and augmented preference rankings using automatic, reward-based, GPT-4, and human assessments. PRO generally outperforms fine-tuning baselines, with stronger gains from longer, higher-quality, and more diverse rankings.
- Experimental Setup: PRO is evaluated on HH-RLHF and augmented datasets containing additional candidates generated by diverse LLMs.The unmodified dataset contains two human-rated conversations per sample; augmented variants expand ranking lengths with models such as Alpaca and ChatGPT.
- Evaluation Metrics: Automatic evaluation combines BLEU for text quality with separately trained and evaluated reward models for human preference.Human judgments and GPT-4 comparisons provide additional model-based and human-based evaluation axes.
- Main Results: 6.52 Reward improvement over SFT and 2.6 over DPO is achieved by PRO on the two-candidate HH-RLHFraw setting.PRO maintains considerable BLEU scores while outperforming or remaining competitive with SFT, RLHF, CoH, DPO, and RRHF.
- Effect of Expanding Preference Ranking Sequence: Longer rankings generally improve performance, and PRO remains the most competitive method on expanded sequences.Adding ChatGPT-generated responses produces reward scores similar to ChatGPT, while stronger augmentation models yield more pronounced improvements.
- Effect of Expanding Preference Ranking Sequence: Higher-quality added responses yield larger gains with more additions, while response diversity can outperform adding only high-quality responses.ChatGPT additions produce consistent gains, and Curie plus Alpaca surpasses Alpaca plus Alpaca at ranking length four.
- Human Evaluation and Ablation Study: GPT-4 and human evaluations both globally favor PRO, while ablations show that removing LSFT lowers BLEU and removing PRO loss lowers Reward.Disabling the full ranking objective also decreases both BLEU and Reward, and jointly removing LSFT and temperature causes a significant performance drop.
Related Work
Prior work uses RLHF to align LLMs with human preferences but faces training-efficiency and complexity limitations, motivating more direct SFT-based approaches.
- RLHF has been widely explored to align language-model outputs with human preferences across tasks including summarization, question answering, and dialogue.
- Despite its advantages, RLHF has limitations in training efficiency and complexity.
- Recent SFT methods seek to address these challenges through semantic templates or reward-model-selected samples from tuned models.
Conclusion
The paper models human alignment as matching the LLM’s probability ranking of responses to human preference rankings and proposes PRO to optimize this alignment directly.
- PRO models human alignment by matching the probability ranking of n LLM-generated responses with humans’ preference ranking.
- PRO extends pair-wise reward-model contrasts into multiple one-to-N contrasts that capture fine-grained human preferences.
- Extensive experiments compare PRO with baselines and examine multifaceted factors affecting its alignment performance.
Ethics Statement
The authors disclose that the research data contain sensitive and offensive content and state that the work is intended only for research purposes.
- The research data contain sensitive and offensive content and are intended only for research purposes.
Qualitative Analysis of PRO
PRO and RLHF share human-alignment and output-quality goals, but PRO uses differentiable ranking objectives and incorporates both preferred and negative responses without RL techniques.
- PRO and RLHF share the primary goals of human alignment and high-quality model outputs.
- PRO directly optimizes ranking scores, avoiding many drawbacks associated with reinforcement learning.
- PRO’s differentiable alignment objective supports multi-task learning.
- Unlike SFT, PRO uses negative responses in preference rankings rather than training only on responses judged desirable.
- For a ranking of length n, PRO requires n −1 contrasts and introduces more negative examples in each contrast than RLHF.
Data Preprocessing
The evaluation data combines four subsets, uses short chosen-versus-rejected rankings, and filters samples so candidates share contexts but differ in responses.
- Four subsets are combined for fine-tuning and evaluation on each corresponding test set.
- Validation uses 280 samples randomly selected from all test data.
- Each sample contains one chosen conversation and one rejected conversation, forming a relatively short ranking.
- Candidates within each sample are filtered to share identical contexts while differing in their responses.
Baselines
PRO is compared with zero-shot systems and LLaMA-7B models fine-tuned using baseline methods, while candidate-ranking methods distinguish responses through pairwise losses.
- PRO is compared with zero-shot baselines and fine-tuned LLaMA-7B models sharing its backbone.
- Baseline methods account for candidate ranking and distinguish candidates through pairwise ranking losses.
Self-bootstrapping Augmentation
Self-bootstrapping expands preference rankings by sampling a response from the current model, scoring and reranking candidates, and iteratively fine-tuning PRO. Its results are mixed across datasets, while naive bootstrapping can overfit the training reward model and requires regularization.
- Self-bootstrapping Augmentation: Self-bootstrapping samples a response from the current model, adds it to the existing set, and reranks all responses with a reward model.
- Self-bootstrapping Augmentation: The self-bootstrapping approach is presented as an abstract training procedure for producing the fine-tuned language model πPRO.
- Self-bootstrapping Augmentation: Self-bootstrapping produces conflicting results across datasets: BLEU rises while reward slightly falls on HH-RLHFraw, both metrics fall on HH-RLHFAlpaca,3, and reward rises while BLEU holds on HH-RLHFChatGPT,3.
- Self-bootstrapping Augmentation: Naive self-bootstrapping can overfit RMtrain, so the method prevents the augmented candidate from taking the original top position and reranks rewards.
- Self-bootstrapping Augmentation: On HH-RLHFChatGPT,3, the improvement may not exceed the effect of adding one more high-quality ChatGPT response.
Error Analysis
Despite effective human-preference alignment, fine-tuned LLMs sometimes fail to generate positive responses when contextual information misleads them.
- Fine-tuned LLMs sometimes fail to generate positive responses because they remain susceptible to contextual misleading.