PaperlayerView exact revision on arXiv

Source-linked AI summary

Preference Ranking Optimization for Human Alignment

Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, Houfeng Wang

arXiv:2306.17492v2cs.CLcs.AI

TL;DR

LLM alignment is needed because pretrained models contain misleading and harmful content, while RLHF is complex and existing methods underuse long preference rankings. PRO directly trains LLMs in SFT using recursive contrasts over rankings of arbitrary length, and experiments report stronger alignment, including reward scores similar to ChatGPT with 7B parameters. The paper also identifies contextual misleading and reward-model/model-size effects as boundaries for current results.

  • Problem

    LLM pretraining includes misleading, toxic, and detrimental content, while RLHF is complex and existing methods often reduce longer rankings to pair-wise contrasts.

    Source: Introduction · p. 1

  • Method

    PRO directly fine-tunes the policy LLM in SFT by recursively applying one-to-N contrasts across human-ranked responses, with optional candidate augmentation and reward-model weighting.

    Source: Introduction · p. 2, From RLHF to PRO · p. 3

  • Results

    PRO surpasses competitive baselines with 2-rankings, improves as rankings lengthen, and achieves reward scores similar to ChatGPT using 7B parameters.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    PRO demonstrates that multi-positional and multi-dimensional preference contrasts can effectively and efficiently align LLMs to human preference.

    Source: Introduction · p. 2, Conclusion · p. 7

    The authors report occasional failures to generate positive responses under contextual misleading and leave larger-model scaling for future work.

    Source: Error Analysis · p. 11, Self-bootstrapping Augmentation · p. 11

Abstract

from arXiv · show

Large language models (LLMs) often contain misleading content, emphasizing the need to align them with human values to ensure secure AI systems. Reinforcement learning from human feedback (RLHF) has been employed to achieve this alignment. However, it encompasses two main drawbacks: (1) RLHF exhibits complexity, instability, and sensitivity to hyperparameters in contrast to SFT. (2) Despite massive trial-and-error, multiple sampling is reduced to pair-wise contrast, thus lacking contrasts from a macro perspective. In this paper, we propose Preference Ranking Optimization (PRO) as an efficient SFT algorithm to directly fine-tune LLMs for human alignment. PRO extends the pair-wise contrast to accommodate preference rankings of any length. By iteratively contrasting candidates, PRO instructs the LLM to prioritize the best response while progressively ranking the rest responses. In this manner, PRO effectively transforms human alignment into aligning the probability ranking of n responses generated by LLM with the preference ranking of humans towards these responses. Experiments have shown that PRO outperforms baseline algorithms, achieving comparable results to ChatGPT and human responses through automatic-based, reward-based, GPT-4, and human evaluations.

Introduction

The paper frames human alignment as necessary because LLMs contain harmful content, while RLHF is complex and pair-wise methods underuse longer preference rankings. PRO addresses this by directly optimizing ranked responses in SFT and reports stronger alignment as rankings expand.

Motivation

  • LLMs can contain misleading, toxic, and detrimental content, motivating alignment to human values for secure and manageable AI systems.Source: Introduction · p. 1

Limitations of RLHF

Research gap

  • Existing methods often reduce longer preference rankings to pair-wise contrasts, limiting macro-level distinction among candidates.Source: Introduction · p. 1

PRO

  • PRO extends pair-wise contrasts to one-to-N contrasts over preference rankings of arbitrary length, recursively prioritizing each response against those ranked below it.Source: Introduction · p. 2

Candidate construction

  • PRO augments rankings with candidates of varied quality and alignment, including self-bootstrapped responses labeled by an additional reward model.Source: Introduction · p. 2

Results

  • PRO surpasses competitive baselines with 2-rankings, improves more with longer rankings, and achieves reward scores similar to ChatGPT using 7B parameters.Source: Introduction · p. 2

Methodology

PRO converts pair-wise policy optimization into recursive multi-positional contrasts over ranked candidates, optionally weighting penalties by reward-model score gaps. It remains an SFT objective while incorporating supervised language-quality training and flexible ranking sources.

From RLHF to PRO

  • PRO transfers Bradley-Terry-style pair-wise reward contrast directly to the policy LLM, treating the LLM as both reward model and policy network.Source: From RLHF to PRO · p. 2

Multi-candidate contrast

  • Expanding the candidate set exposes the policy to more samples, replacing trial-and-error experience with one-to-N contrast over ranked responses.Source: From RLHF to PRO · p. 2

Multi-positional contrast

  • Recursive contrasts exploit the full ranking by treating each successive response as positive against all remaining lower-ranked candidates.Source: From RLHF to PRO · pp. 2–3

Training objective

  • PRO combines its preference objective with the NLL loss of the top candidate, balancing text quality and human preference through β.Source: From RLHF to PRO · p. 3

Comparison with RLHF

Flexible ranking sources

Differentiated contrast

  • Reward-model score gaps enable differentiated penalties: larger preference gaps produce stronger penalties through lower temperatures.Source: Differentiated Contrast · p. 4

Results

  • Dynamic temperature significantly improves performance when optimizing LPRO alone and provides gains when jointly optimizing LPRO and LSFT.Source: Differentiated Contrast · p. 4

Experiments

Experiments evaluate PRO on HH-RLHF and augmented preference rankings using automatic, reward-based, GPT-4, and human assessments. PRO generally outperforms fine-tuning baselines, with stronger gains from longer, higher-quality, and more diverse rankings.

Experimental Setup

  • PRO is evaluated on HH-RLHF and augmented datasets containing additional candidates generated by diverse LLMs.The unmodified dataset contains two human-rated conversations per sample; augmented variants expand ranking lengths with models such as Alpaca and ChatGPT.Source: Data Prepration · p. 4

Evaluation Metrics

  • Automatic evaluation combines BLEU for text quality with separately trained and evaluated reward models for human preference.Human judgments and GPT-4 comparisons provide additional model-based and human-based evaluation axes.Source: Evaluation Metrics · p. 4

Main Results

  • 6.52 Reward improvement over SFT and 2.6 over DPO is achieved by PRO on the two-candidate HH-RLHFraw setting.PRO maintains considerable BLEU scores while outperforming or remaining competitive with SFT, RLHF, CoH, DPO, and RRHF.Source: Main Experiment · p. 4

Effect of Expanding Preference Ranking Sequence

  • Longer rankings generally improve performance, and PRO remains the most competitive method on expanded sequences.Adding ChatGPT-generated responses produces reward scores similar to ChatGPT, while stronger augmentation models yield more pronounced improvements.Source: Main Experiment · p. 5, Effect of Expanding Preference Ranking… · p. 6
  • Higher-quality added responses yield larger gains with more additions, while response diversity can outperform adding only high-quality responses.ChatGPT additions produce consistent gains, and Curie plus Alpaca surpasses Alpaca plus Alpaca at ranking length four.Source: Effect of Expanding Preference Ranking… · p. 6

Human Evaluation and Ablation Study

  • GPT-4 and human evaluations both globally favor PRO, while ablations show that removing LSFT lowers BLEU and removing PRO loss lowers Reward.Disabling the full ranking objective also decreases both BLEU and Reward, and jointly removing LSFT and temperature causes a significant performance drop.Source: Human and GPT-4 Evaluation · p. 6, Ablation Study · p. 6

Related Work

Prior work uses RLHF to align LLMs with human preferences but faces training-efficiency and complexity limitations, motivating more direct SFT-based approaches.

Conclusion

The paper models human alignment as matching the LLM’s probability ranking of responses to human preference rankings and proposes PRO to optimize this alignment directly.

  • PRO models human alignment by matching the probability ranking of n LLM-generated responses with humans’ preference ranking.Source: Conclusion · p. 7
  • PRO extends pair-wise reward-model contrasts into multiple one-to-N contrasts that capture fine-grained human preferences.Source: Conclusion · p. 7
  • Extensive experiments compare PRO with baselines and examine multifaceted factors affecting its alignment performance.Source: Conclusion · p. 7

Ethics Statement

The authors disclose that the research data contain sensitive and offensive content and state that the work is intended only for research purposes.

  • The research data contain sensitive and offensive content and are intended only for research purposes.Source: Ethics Statement · p. 8

Qualitative Analysis of PRO

PRO and RLHF share human-alignment and output-quality goals, but PRO uses differentiable ranking objectives and incorporates both preferred and negative responses without RL techniques.

Data Preprocessing

The evaluation data combines four subsets, uses short chosen-versus-rejected rankings, and filters samples so candidates share contexts but differ in responses.

Baselines

PRO is compared with zero-shot systems and LLaMA-7B models fine-tuned using baseline methods, while candidate-ranking methods distinguish responses through pairwise losses.

  • PRO is compared with zero-shot baselines and fine-tuned LLaMA-7B models sharing its backbone.Source: Baselines · p. 10
  • Baseline methods account for candidate ranking and distinguish candidates through pairwise ranking losses.Source: Baselines · p. 10

Self-bootstrapping Augmentation

Self-bootstrapping expands preference rankings by sampling a response from the current model, scoring and reranking candidates, and iteratively fine-tuning PRO. Its results are mixed across datasets, while naive bootstrapping can overfit the training reward model and requires regularization.

Self-bootstrapping Augmentation

Error Analysis

Despite effective human-preference alignment, fine-tuned LLMs sometimes fail to generate positive responses when contextual information misleads them.

  • Fine-tuned LLMs sometimes fail to generate positive responses because they remain susceptible to contextual misleading.Source: Error Analysis · p. 11

Related papers

Loading 2306.17492v2…