PaperlayerView exact revision on arXiv

Source-linked AI summary

MaxMin-RLHF: Alignment with Diverse Human Preferences

Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, Mengdi Wang

arXiv:2402.08925v2cs.CLcs.AIcs.LGcs.RO

TL;DR

RLHF commonly uses a single reward, while the diversity of human preferences across user populations has received limited attention. The paper derives an impossibility result for single-reward alignment and proposes MaxMin-RLHF, which learns multiple preference distributions and optimizes a MaxMin objective. Experiments report improved performance and socially fairer outcomes.

  • Problem

    The paper identifies limited attention to representing diverse human preferences and asks whether single-reward RLHF can align with them.

    Source: §1 · pp. 1–2

  • Method

    MaxMin-RLHF learns a mixture of preference distributions using EM and uses multiple sub-population-specific rewards in a MaxMin alignment objective.

    Source: §1 · p. 2

  • Results

    Experiments report that MaxMin-RLHF outperforms existing methodologies and produces socially fairer outcomes.

    Source: §1 · p. 2, §6 · pp. 8–9

  • Takeaways & Limitations

    The conclusion highlights the need for more inclusive strategies in RLHF methodologies.

    Source: §6 · p. 9

    The impossibility bound is stated under a positive diversity margin and an upper bound on feature-representation norms.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback (RLHF) aligns language models to human preferences by employing a singular reward model derived from preference data. However, such an approach overlooks the rich diversity of human preferences inherent in data collected from multiple users. In this work, we first derive an impossibility result of alignment with single reward RLHF, thereby highlighting its insufficiency in representing diverse human preferences. To provide an equitable solution to the problem, we learn a mixture of preference distributions via an expectation-maximization algorithm and propose a MaxMin alignment objective for policy learning inspired by the Egalitarian principle in social choice theory to better represent diverse human preferences. We elucidate the connection of our proposed approach to distributionally robust optimization and general utility RL, thereby highlighting the generality and robustness of our proposed solution. We present comprehensive experimental results on small-scale (GPT-2) and large-scale language models (with Tulu2-7B) and show the efficacy of the proposed approach in the presence of diversity among human preferences. Our algorithm achieves an average improvement of more than 16% in win-rates over conventional RLHF algorithms and improves the win-rate (accuracy) for minority groups by over 33% without compromising the performance of majority groups, showcasing the robustness and fairness of our approach. We remark that our findings in this work are not only limited to language models but also extend to reinforcement learning in general.

1. Introduction

The introduction identifies a gap in how RLHF represents diverse human preferences and questions whether a single reward can align with them. It presents an impossibility result and proposes MaxMin-RLHF, with experiments reporting improved performance.

1. Introduction

  • RLHF research has given minimal emphasis to representing the diversity of preferences across broad user populations.The introduction notes that approaches commonly use a single reward, while other approaches aggregate multiple rewards or personalize policies.Source: §1 · p. 1
  • The introduction asks whether a single-reward RLHF pipeline is sufficient to align with diverse human preferences.It also asks what strategies can align with those preferences.Source: §1 · pp. 1–2
  • The authors report an impossibility result for alignment with diverse human preferences using single-reward RLHF.They motivate the result by the diversity of preferences across human sub-populations.Source: §1 · p. 2
  • MaxMin-RLHF learns a mixture of preference distributions with an EM algorithm, then uses multiple sub-population-specific rewards in a MaxMin alignment objective.The objective is inspired by the Egalitarian rule and aims to maximize social utility.Source: §1 · p. 2
  • Experiments on GPT-2 and Tulu-7B are reported to show the approach's effectiveness and performance improvements over existing methodologies.The empirical study first examines single-reward alignment and then evaluates MaxMin-RLHF.Source: §1 · p. 2

2. Preliminaries

The preliminaries describe the language-model notation and the three-stage RLHF pipeline. They explain how preference comparisons support reward learning and how the learned reward is used in policy fine-tuning.

2. Preliminaries

  • A language model πθ is represented as a mapping parameterized by θ, taking a prompt x and generating a response y sampled from πθ(· | x).The vocabulary is denoted V, and prompts comprise tokens from that vocabulary.Source: §2 · p. 2
  • RLHF consists of supervised fine-tuning, reward modelling, and RL fine-tuning; supervised fine-tuning produces the reference model πref.The initial model is fine-tuned on a high-quality dataset for downstream tasks.Source: §2 · p. 2
  • Reward modelling uses human preferences over response pairs and a Bradley-Terry preference model to learn a parameterized reward by maximum likelihood.The negative log-likelihood is framed as a binary-classification loss.Source: §2 · p. 2
  • RL fine-tuning obtains an optimal policy by maximizing reward with KL regularization, where β controls deviation from the reference policy.The reward in this stage is the learned reward model.Source: §2 · p. 2

3. An Impossibility Result for Single Reward RLHF with Diverse Preferences

Human preferences vary across sub-populations, but single-reward RLHF aggregates them into one reward. The paper shows that this mismatch creates an alignment gap, especially for groups with distinctive preferences or smaller representation.

3.1. Diversity in Human Preferences

  • Human sub-populations can have distinct preference distributions, reflecting varied socio-demographic and socio-cultural backgrounds.The paper models a hidden sub-population variable over a finite set of groups and defines group-specific preference distributions.Source: §3.1 · pp. 3–4
  • In the IMDB example, the minority favors concise responses but is indifferent to sentiment, while the majority favors positive sentiment and shows little concern for conciseness.The figure contrasts chosen and rejected trajectories across sentiment and conciseness preferences.Source: Figure 2 · p. 4

3.2. Reward Mismatch Due to Diversity

  • Single-reward reward learning implicitly averages subgroup preferences, with reward suboptimality depending on preference distinctiveness and each group’s relative weight.The preference distribution is a weighted sum of subgroup-specific distributions.Source: §3.2 · pp. 4–5, §3.1 · p. 4

3.3. An Impossibility Results of Alignment

  • Greater preference diversity and smaller subgroup representation increase single-reward RLHF’s alignment gap, making subgroup-specific preferences harder to reflect.The paper identifies preference distinctiveness and relative subgroup weight as factors in reward suboptimality and alignment.Source: §3.2 · p. 5, §3.3 · p. 6

4. MaxMin-RLHF: One Possibility

The paper proposes learning subgroup-specific rewards and using a MaxMin policy objective to align with diverse preferences. Its policy update targets the subgroup with the lowest utility, following an egalitarian objective.

4. MaxMin-RLHF: One Possibility

  • The MaxMin objective maximizes the minimum utility across human sub-populations, following the Egalitarian principle in social choice theory.Each subgroup’s utility is its alignment objective under that group’s reward.Source: §4 · p. 6
  • MaxMin policy iteration selects the subgroup with the lowest current utility and updates the policy toward maximizing that subgroup’s reward objective with KL regularization.The algorithm outputs a policy aligned with the socially fair preference dataset.Source: §4 · p. 6
  • When subgroup rewards are unavailable, an expectation-maximization algorithm learns a mixture of reward models and user clusters from preference data.The algorithm alternates hard cluster assignment with reward updates using assigned users’ data.Source: §4 · p. 6

5. Experimental Results

The experiments examine single-reward alignment failures and test whether MaxMin-RLHF can serve diverse preferences. On GPT-2 it balances positive sentiment and conciseness; large-scale results report stronger minority performance than single-reward PPO on biased data.

5.1. Small Scale Experiments (with GPT-2): Sentiment and Conciseness Alignment

  • On GPT-2, single-reward RLHF favors positive sentiment preferred by the majority group while neglecting the minority group's conciseness preference.The preference data contain 80% majority-group and 20% minority-group users.Source: §5.1 · p. 7
  • MaxMin-RLHF generates responses that are both positive in sentiment and concise, catering to the majority and minority groups.The task uses positive sentiment and shorter response length as the groups' respective preferences.Source: §5.1 · p. 7, Figure 4 · p. 7
  • Figure 5 reports average sentiment and conciseness performance, with MaxMin-RLHF better catering to both criteria than single-reward RLHF.The reported comparison concerns the controlled sentiment-generation task.Source: Figure 5 · p. 7, §5.1 · p. 7

5.2.1. MAIN RESULTS

  • For P1A/P1B, the single-reward evaluation varies group ratios and tracks total, majority, and minority accuracy; low minority accuracy signals poor subsequent alignment.The tested ratios include 1, 2, 6, and 10.Source: §5.2.1 · p. 8
  • After the fourth EM iteration, all users are clustered correctly across the three datasets, indicating convergence of the mixture preference model.The experiments train two reward models on mixed user groups.Source: §5.2.1 · p. 8, §5.2 · p. 8
  • MaxMin alignment maintains a high win rate, while single-reward PPO on biased datasets performs relatively poorly on minority data.Large-scale tests use Tulu2-7B and GPT-4 pairwise evaluations on 50 Koala instances.Source: Table 2 · p. 8, §5.2.1 · p. 8, §5.2 · p. 8

6. Conclusions

The conclusion argues that a single-reward RLHF framework cannot align with diverse preferences, and presents MaxMin-RLHF as a more equitable alternative. The experiments report socially fairer outcomes from the proposed approach.

6. Conclusions

  • The paper's impossibility result establishes the insufficiency of single-reward RLHF for alignment with diverse preferences.The conclusion frames this as a limitation of the single-reward framework.Source: §6 · p. 8
  • MaxMin-RLHF learns a max-min policy over reward functions to pursue more equitable alignment across diverse preferences.The approach is framed around social utility objectives.Source: §6 · p. 8
  • Experiments demonstrate the effectiveness of MaxMin-RLHF in producing socially fairer outcomes.The conclusion links these findings to more inclusive RLHF strategies.Source: §6 · pp. 8–9

Impact Statement

The impact statement centers on limitations in representing diverse opinions and preferences. It presents the work as an initial formal and empirical contribution toward equitable alignment and calls for further research.

Impact Statement

  • The work's primary objective is to highlight how existing alignment techniques fail to represent diverse opinions and preferences.The impact statement frames representation of preference diversity as its central concern.Source: Impact Statement · p. 9
  • The authors describe the study as one of the first to demonstrate this limitation formally, using mathematical and empirical analyses.They characterize the work as a first step toward equitable alignment.Source: Impact Statement · p. 9
  • The impact statement calls for further research on alignment under diversity to help ensure models are not biased toward specific minority groups.This is presented as a direction the findings may encourage.Source: Impact Statement · p. 9

C. Preliminary Results

The preliminary results analyze preference-model properties and reward-learning loss. They relate bounded reward features to the Bradley-Terry model's Lipschitz behavior and show that cross-entropy learning implicitly fits a user-weighted mixture of preferences.

C. Preliminary Results

  • The Bradley-Terry preference probability is parameterized through reward scores, and its Lipschitz behavior follows under bounded feature representations.The stated bound uses L_p = 4D, where D bounds the feature representation norm.Source: C. Preliminary Results · pp. 13–14
  • Cross-entropy reward learning implicitly minimizes distance to a user-weighted mixture of individual preference distributions.The mixture weights reflect the implicit distribution among user groups.Source: C. Preliminary Results · pp. 14, 16
  • The implicit mixture fit can leave a significant gap between the learned reward and individual user utilities.The result follows from the reward-learning objective's implicit weighted combination of user preferences.Source: C. Preliminary Results · pp. 14, 16

D. Proof of Lemma 1

The proof develops a lower bound on reward-model suboptimality using total-variation distances between user-group preference distributions, then identifies the most diverse pair of groups.

D. Proof of Lemma 1

  • The derivation uses the Lipschitz property of preference probabilities to relate preference-distribution distances to the lower bound.The proof also uses the triangle inequality and Jensen’s inequality in deriving the bound.Source: D. Proof of Lemma 1 · pp. 16–17
  • The proof derives a lower bound on the mismatch between a user group's optimal preference distribution and the learned distribution using total-variation distances between groups.The proof assumes infinite data, under which maximum likelihood estimation converges to the optimal parameter.Source: D. Proof of Lemma 1 · pp. 16–17
  • A lower bound separated from zero for even one user-group pair establishes an impossibility result for the corresponding minority group.The proof first selects the pair of groups with maximum total-variation distance.Source: D. Proof of Lemma 1 · p. 17

E. Proof of Theorem 1

The proof connects reward-model suboptimality to policy differences under KL-regularized RLHF, using assumptions on the policy and reward parameterization.

E. Proof of Theorem 1

  • In this RLHF setting, the negative KL-regularized reward objective is strongly convex with respect to the policy, with strong convexity parameter μ = 1.This property is used to relate reward suboptimality to policy differences.Source: E. Proof of Theorem 1 · pp. 18–19
  • The proof assumes policies assign probability at least c > 0 to every response for every prompt, making the log-policy Lipschitz continuous with parameter 1.The stated condition is π(y|x) ≥ c for all x, y.Source: E. Proof of Theorem 1 · p. 18
  • For its reward-to-policy mapping, the proof uses an analytical equivalence specific to KL-constrained RLHF with the Bradley–Terry preference model.The proof also states a linearly parameterized reward function using feature vectors.Source: E. Proof of Theorem 1 · p. 18

F. Additional Details of the Experiments

The appendix provides additional experiment details, including a dataset summary and prompts used to express user-group preference differences.

F. Additional Details of the Experiments

  • Table 4 is identified as a dataset summary, but the supplied passage contains only its caption.The appendix section states that it provides additional details of the Section 5 experiments.Source: F. Additional Details of the Experiments · p. 19, Table 4
  • The listed preference prompts contrast elementary-school-level versus PhD-level explanations, and concise versus highly informative responses.The prompts also contrast friendly, humorous responses with unfriendly ones.Source: F. Additional Details of the Experiments · p. 19
  • The appendix identifies Figure 6 as reporting results on Dataset P1A/P1B.The supplied figure caption gives no result values or comparison.Source: Figure 6 · p. 19

G. Additional Experiments in Robotics Navigation Tasks

The additional navigation experiment presents GridWorld as an example of applying MaxMin alignment beyond language models, comparing it with single-reward RLHF.

G. Additional Experiments in Robotics Navigation Tasks

  • The paper presents GridWorld navigation as an example of extending its ideas to reinforcement learning beyond language models.The experiment shows MaxMin alignment alongside single-reward RLHF.Source: G. Additional Experiments in Robotics… · p. 19
  • Figure 7 depicts a vehicle distributing goods to green and orange groups, with the proposed formulation framed as maximizing social utility to satisfy user preferences.The caption describes separate trajectories for green-preference policies and the proposed formulation.Source: Figure 7 · p. 20
  • The Figure 7 caption describes both panels (b) and (c) as trajectories based on green user preferences, so it does not establish a contrast between those panels.Panel (d) is labeled as the result for the proposed formulation.Source: Figure 7 · p. 20
Loading 2402.08925v2…