PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalized Language Modeling from Personalized Human Feedback

Xinyu Li, Ruiyang Zhou, Zachary C. Lipton, Liu Leqi

arXiv:2402.05133v3cs.CLcs.AIcs.LG

TL;DR

Vanilla RLHF assumes uniform preferences, limiting personalization when users differ and making exhaustive preference specification impractical. P-RLHF jointly learns a lightweight user model and personalized LLM from explicit user information and implicit feedback, with experiments showing improved alignment to individual preferences across tasks.

  • Problem

    Vanilla RLHF assumes uniform user preferences, while predefined preference dimensions are difficult to exhaustively enumerate for personalized generation.

  • Method

    P-RLHF jointly learns a lightweight user model and personalized LLM from textual user information and personalized feedback, using explicit and implicit preference signals.

  • Results

    P-DPO models outperform vanilla DPO, achieving above 60% win-rates on a real-world preference dataset with a large user base.

  • Takeaways & Limitations

    P-RLHF generates responses better aligned with individual preferences while supporting personalization without requiring users to fully articulate those preferences.

  • Takeaways & Limitations

    Under the preference-uniformity assumption, generic implicit user embeddings are expected to behave similarly to vanilla DPO without additional textual user information.

Abstract

from arXiv · show

Personalized large language models (LLMs) are designed to tailor responses to individual user preferences. While Reinforcement Learning from Human Feedback (RLHF) is a commonly used framework for aligning LLMs with human preferences, vanilla RLHF assumes that all human preferences share the same distribution, preventing fine-tuned LLMs from generating personalized content when user preferences are diverse. In this work, we propose Personalized-RLHF (P-RLHF), an efficient framework that utilizes a lightweight user model to capture individual user preferences and jointly learns the user model and the personalized LLM from human feedback. P-RLHF exhibits the following three characteristics: (1) It enables an LLM to generate personalized content and scale efficiently with growing number of users. (2) It handles both explicit user preferences described as textual input and implicit user preferences encoded in the feedback data. (3) It eliminates the need for users to fully articulate their preferences, which are normally needed for prompting LLMs to generate personalized content yet are often impractical to obtain in real-world scenarios. Our experimental results show that personalized LLMs trained using P-RLHF generate responses that are more closely aligned with individual user preferences, outperforming vanilla, non-personalized RLHF and prompting-based personalization approaches across different tasks. We opensource our code at https://github.com/HumainLab/Personalized_RLHF.

1 Introduction

The paper identifies a personalization gap in vanilla RLHF and introduces P-RLHF, which jointly learns a lightweight user model and LLM from personalized feedback. The framework combines explicit textual user information with implicit preferences from feedback while avoiding separate models for different users or preference dimensions.

  • Motivation: Vanilla RLHF assumes human preferences share one distribution, limiting personalized responses when preferences are diverse or conflicting.Recent approaches can require separate reward models or LLMs for preference dimensions, creating computational and storage challenges as user bases grow.
  • Approach: The framework jointly learns user embeddings and a base LLM using a personalized learning objective.For seen users, generation uses user-specific embeddings; a generic embedding supports new users unseen during training.
  • Approach: P-RLHF uses a lightweight user model to capture explicit preferences from user information and implicit preferences from historical feedback.The approach allows missing preference information to be inferred from feedback rather than requiring users to fully specify every preference.
  • Scalability: P-RLHF avoids training separate reward models or LLMs, enabling personalization that scales across large numbers of users.The paper presents personalized DPO objectives as an instantiation of the framework.

2 Related Work

Related work includes vanilla RLHF, RLHF-based personalization, and prompt-based personalization. The paper distinguishes P-RLHF by learning directly from user information and personalized feedback without predefined preference dimensions or multiple reward models and LLMs.

  • RLHF-based methods: Vanilla RLHF aligns pretrained LLMs with human preferences through reward modeling and policy optimization, while DPO directly optimizes the LLM from preference data.DPO offers improved training efficiency relative to reward-model-based RLHF approaches.
  • RLHF-based personalization: Prior RLHF personalization methods use separate reward models or LLM policies for preference dimensions and customize reward weights or merge models.These approaches rely on predefined preference dimensions such as completeness or friendliness.
  • P-RLHF: P-RLHF learns personalized LLMs directly from user information and personalized feedback without predefined preference dimensions or multiple models.It instead uses a small user model to augment the LLM.
  • Prompt-based personalization: Prompt-based personalization uses historical user content as few-shot examples and may retrieve relevant user data to construct prompts.These methods rely on in-context learning to tailor generated content.

3 Vanilla RLHF

Vanilla RLHF and DPO implicitly assume uniform user preferences, so their learned reward or policy can collapse diverse feedback into majority preferences. The paper shows that this can misalign generations with minority users, especially when preference groups conflict.

  • Vanilla RLHF pipeline: Vanilla RLHF first obtains an SFT policy, learns a reward model from preference comparisons, and optimizes the LLM against that reward with a KL penalty.DPO provides a direct preference-optimization alternative to reward-model-based RLHF.
  • Preference assumptions: The vanilla reward model assumes preference probabilities follow the Bradley-Terry model, where pairwise preference depends on the difference between response rewards.The same underlying assumption can be learned explicitly through reward modeling or implicitly through DPO.
  • Preference assumptions: Vanilla reward modeling and DPO assume user preferences are uniform across users.This assumption may fit preferences common across users, such as factuality and safety, but is undesirable for diverse stylistic preferences.
  • Consequences: Vanilla reward modeling induces majority voting, treating responses preferred by the majority as preferred by all users.In personalized settings, this can silence minority preferences and produce generations misaligned with minority users when groups conflict.
  • Consequences: The minority group’s deviation from the assumed uniform preference increases as the majority group becomes larger.When the majority is at least as large as the minority, the minority’s deviation is greater than the majority’s deviation.
  • Transition to personalization: P-RLHF is proposed to capture individual user preferences and generate content tailored to each user.The framework introduces a user model and personalized objectives after identifying the limitations of vanilla reward learning and DPO.

4 Learning from Personalized Human Feedback

P-RLHF defines personalized language modeling around a user model that combines explicit textual information with implicit preferences learned from user feedback. It supports several implicit-preference structures and integrates the resulting user representation into a base LLM for generation.

  • 4.1 Personalized LLM: Problem setup: A personalized preference dataset pairs prompts and ranked generated texts with annotator information, including optional textual information and a user identifier.
  • 4.1 Personalized LLM: Problem setup: The personalized LLM conditions generation on a prompt and user information, while missing user-specific information yields a non-personalized response.
  • 4.2 General P-RLHF Framework: P-RLHF consists of a base LLM and a learnable user model that extracts a user embedding from textual information and user identity.
  • 4.3 P-RLHF User Models: The framework combines explicit preferences encoded from user text with implicit preferences captured from feedback data, rather than requiring all preferences to be articulated.
  • 4.3 P-RLHF User Models: Implicit user models can represent uniform, individualized, or cluster-based preferences; the cluster model expresses each user embedding as a weighted combination of K cluster centers.
  • 4.3 P-RLHF User Models: The cluster-based model also provides a low-rank approximation, replacing a separate embedding for every user with shared cluster centers and user-specific weights.

5 Experiments

Across controlled, semi-synthetic, and real-world evaluations, P-DPO learned user-specific behavior from personalized feedback, including implicit preferences, while retaining efficient scaling. It aligned responses more closely with individual users than non-personalized baselines and remained effective without explicit user information.

  • 5.1 Generation with Conflicting Preferences: P-DPO generated longer responses for majority workers and zero-length responses for minority workers who preferred shorter responses, matching the analytically derived optimal behavior.This behavior extrapolated beyond training data because the training set contained no empty responses.
  • 5.1 Generation with Conflicting Preferences: P-DPO with generic implicit user embeddings produced response lengths similar to vanilla DPO for unseen users, matching the expected user-agnostic behavior.The expected behavior follows from using no additional textual user information for unseen users.
  • 5.2 Instruction Following under Different Preference Profiles: Against an oracle personalized method, P-DPO achieved above 59% win-rates for five of six users and a 70.24% average win-rate.These results indicate that P-DPO captured implicit preferences encoded in feedback and aligned with individual users.
  • 5.3 Personalization on Real-World Preference Dataset with Large User Base: On PRISM, all P-DPO models exceeded 60% win-rates against vanilla DPO and outperformed the chosen responses, while the cluster-based user model performed best.PRISM contains 1,500 participants from 75 countries and 8,011 labeled conversations with contextual preferences and fine-grained feedback.
  • 5.3 Personalization on Real-World Preference Dataset with Large User Base: Using only the implicit user model on PRISM, P-DPO reached 65.38% per-sample and 72.86% per-user win-rates against vanilla DPO for 70 users.The variant also achieved 51.32% per-sample and 55% per-user win-rates on 20 users, showing effectiveness without explicit user information.
  • Computational / Memory Cost: P-RLHF uses a lightweight user model whose trainable parameter count is much smaller than the LLM, reducing costs relative to methods training multiple LLMs.For PRISM with K = 10, the user-model parameters were much smaller than one-tenth of the LLM parameters.

6 Conclusions

The paper introduces P-RLHF to personalize language models from textual user information and personalized feedback. Its P-DPO objectives and user-model designs support aligned responses without separate models for each preference dimension, while remaining compatible with other RLHF variants.

  • 6 Conclusions: P-RLHF jointly learns a lightweight user model and personalized LLM from explicit textual preferences and implicit feedback preferences.The framework is intended to scale efficiently as the number of users grows.
  • 6 Conclusions: P-DPO provides learning objectives for personalized language modeling, alongside user-model designs that capture structural assumptions about preferences.The framework can also be adapted to other preference optimization objectives such as IPO and integrated with reward modeling.
  • 6 Conclusions: Empirically, the resulting personalized LLMs generated responses better aligned with individual user preferences.The authors characterize P-RLHF as a general framework applicable to multiple RLHF variants.

B Proofs in Section 3.1

The proofs show that vanilla reward modeling aggregates preferences over matching prompt–response pairs and therefore behaves like majority voting. When majority and minority preferences conflict, the minority deviates more from the marginalized preference, increasingly so as the majority grows.

  • Lemma 3.2: Vanilla reward modeling is equivalent to majority voting, treating responses preferred by the majority as preferred by all users.The proof derives this result from the first-order condition for the vanilla reward objective.
  • Lemma 3.2: Vanilla reward modeling estimates preferences by aggregating annotations sharing the same prompt and response pair.The set C_i contains samples with matching prompts and response pairs, including reversed response orderings.
  • Lemma 3.3: When majority and minority preferences differ, the minority group's true preference deviates more from the assumed uniform preference than the majority group's preference.This inequality is stated under P(u_majority) ≥ P(u_minority).
  • Lemma 3.3: The minority deviation from the assumed uniform preference increases monotonically as the majority group's size increases.The result follows from decomposing the marginalized preference by user-group probabilities.

C.3 Additional Experiment Results

The experiments evaluate P-DPO's implicit reward on personalized preference comparisons and define metrics separating seen-user, unseen-user, and per-group accuracy. On the TL;DR task, P-DPO substantially improves seen-user alignment while retaining generic-user performance comparable to vanilla DPO.

  • Evaluation: P-DPO is evaluated by whether its implicit reward ranks preferred summaries above less-preferred summaries.In the controlled TL;DR setting, longer summaries are preferred by majority workers and shorter summaries by minority workers.
  • Metrics: Accuracy-top measures pooled accuracy for samples annotated by the top 10 workers, while Accuracy-generic measures accuracy for unseen workers.Unseen-worker evaluation uses a generic user embedding learned from seen-user data.
  • Metrics: Accuracy-average reports the mean and standard error of per-user accuracy for top workers, separately for majority and minority groups.This metric complements pooled seen-user accuracy with group-level per-user performance.
  • Results: 32% higher Accuracy-top accuracy was achieved by P-DPO than vanilla DPO on seen top-10 workers, reaching 91% versus 59%.On unseen workers, P-DPO achieved similar Accuracy-generic accuracy to vanilla DPO; its seen-worker Accuracy-average was 90% for both groups versus 25% for vanilla DPO's minority group.

C.4 Ablation Study

The ablation study examines user-token count, loss weighting, generic embeddings, and cluster count, while additional results test personalization with 40 workers. Performance depends on sufficient user expressivity and a shared generic component, yet P-DPO remains competitive as the user set grows.

  • User representation: 89% Accuracy-top accuracy with one user token remained below the 91% achieved with ten tokens, but both exceeded vanilla DPO.More user tokens provide greater expressivity, while one token still demonstrates effective personalization.
  • User representation: With ten user tokens, α = 1.0 slightly improved seen-user accuracy over α = 0.5 but produced more volatile Accuracy-generic curves.The comparison motivates the user-agnostic loss for learning a stable generic representation for new users.
  • User representation: Removing the generic embedding e0 slowed Accuracy-top growth, indicating that the common preference component facilitates learning individual preferences.The comparison is based on the training curves in Figure 5(a).
  • Cluster-based preference: Two clusters performed significantly worse than five clusters, although the two-cluster model still outperformed vanilla DPO and learned more slowly.The results indicate that larger cluster counts provide greater flexibility for preference modeling.
  • Scaling to more users: P-DPO remained competitive across all accuracy metrics when evaluated with the top 40 workers.The same two P-DPO configurations used in the top-10-worker experiment were tested in this larger-user setting.

D.2 P-DPO Experiment Details

The P-SOUPS experiments initialize all models from Tulu-7B SFT and train vanilla DPO and P-DPO under a shared two-epoch LoRA configuration. This provides a common training setup for comparing personalized and vanilla models.

  • Training setup: All P-SOUPS experiment models are initialized from the Tulu-7B SFT model.
  • Training setup: Vanilla DPO and P-DPO models use β = 0.1, batch size 32, learning rate 5e −5, cosine scheduling, 150 warm-up steps, and two epochs.
  • Training setup: Training uses LoRA with α = 32, rank 8, dropout 0.1, and a personalized DPO Trainer extending the TRL DPO Trainer.

E.1 Experiment details on PRISM dataset

The PRISM experiments use multi-turn conversations with ratings and selections from individual users, including separate evaluation on unseen users. P-DPO combines user modeling and personalized reward aggregation to produce responses that better sustain the target user’s preferences.

  • Dataset: PRISM contains multi-turn conversations where users rate several LLM responses and select one to continue the dialogue.Each turn uses responses from randomly selected LLMs prompted with the same system string.
  • Dataset split: The dataset separates seen and unseen users, with unseen-user dialogues appearing only in the test split.Seen-user dialogues are further divided into training and test data at a 6431:3982 ratio.
  • Generation examples: P-DPO responses maintain a friendly tone while focusing on alcohol drinking, unlike vanilla DPO’s topic drift and the chosen response’s preachy tone.The example indicates that vanilla DPO captures the explicit friendliness preference but does not consistently preserve it across the response.
  • Personalized reward model: P-RLHF represents each user with a learnable embedding that enters the personalized reward model either as a soft prompt or as a linear head.Soft prompts prepend user embeddings to token embeddings, while linear-head aggregation uses an inner product with the final hidden state.
  • Training objective: The reward-model objective balances user-specific preference comparisons against user-agnostic comparisons through α.The user-specific term uses explicit user identifiers, while the user-agnostic term uses the same preference data without user identifiers.
  • Generation: A personalized reward model can support Best-of-N selection or direct policy optimization for generating personalized text.Best-of-N ranks sampled texts with the personalized reward model, while policy optimization directly updates the language-model policy.

F.3 Another example of P-RLHF Objective: P-IPO

P-RLHF generalizes preference-optimization objectives by augmenting the base LLM with a user model and replacing the original loss with a personalized version. P-IPO is presented as an example of applying this framework beyond DPO.

  • General framework: P-RLHF augments a base LLM with a user model to produce a personalized LLM and adapt existing preference-optimization losses.The generalized objective combines personalized and user-agnostic components weighted by α.
  • General framework: The framework applies to any preference-optimization objective that maps paired LLM outputs to a scalar.The paper states that the loss can be replaced with any such preference-optimization objective.
  • P-IPO example: P-IPO is introduced as an example obtained by updating an existing IPO loss into its personalized variant.The passage presents the base loss and then identifies the resulting personalized objective as P-IPO.
Loading 2402.05133v3…