Source-linked AI summary
Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, Prithviraj Ammanabrolu
TL;DR
RLHF typically aligns language models with aggregate human preferences, leaving diverse and conflicting individual perspectives underrepresented. The paper formulates personalized alignment as MORL, trains policies for separate preference dimensions, and merges them post-hoc. It reports personalized alignment and linear rather than exponential scaling, while noting scope limits from predefined preferences and unresolved reward-signal normalization.
Problem
RLHF uses a single reward model for general population preferences, which does not represent diverse or conflicting individual preferences.
Method
The paper models personalized alignment as MORL and uses PERSONALIZED SOUPS to train distinct preference policies independently before merging their parameters during inference.
Results
The paper achieves personalized alignment and reports that PERSONALIZED SOUPS scales linearly rather than exponentially with the number of unique preferences while remaining competitive with PROMPTED-MORL.
Takeaways & Limitations
Parameter merging provides a modular way to combine independently trained preference policies and incorporate novel preferences without retraining the underlying policy.
Takeaways & Limitations
The study uses six predefined preferences, limiting declarative personalization, and notes that normalization across multiple reward-model signals is nontrivial.
Abstract
from arXiv · showhide
While Reinforcement Learning from Human Feedback (RLHF) aligns Large Language Models (LLMs) with general, aggregate human preferences, it is suboptimal for learning diverse, individual perspectives. In this work, we study Reinforcement Learning from Personalized Human Feedback (RLPHF) problem, wherein LLMs are aligned to multiple (sometimes conflicting) preferences by modeling alignment as a Multi-Objective Reinforcement Learning (MORL) problem. Compared to strong single-objective baselines, we show that we can achieve personalized alignment by decomposing preferences into multiple dimensions. These dimensions are defined based on personalizations that are declared as desirable by the user. In this work, we show that they can be efficiently trained independently in a distributed manner and combined effectively post-hoc through parameter merging. The code is available at https://github.com/joeljang/RLPHF.
1 INTRODUCTION
The paper reframes preference alignment as personalized, multi-objective reinforcement learning, addressing diverse and potentially conflicting user preferences. It introduces independently trained policy models whose parameters are merged during inference for efficient personalization.
- Motivation: RLPHF formulates human-preference alignment as a multi-objective problem supporting personalized combinations of potentially conflicting preferences.Unlike standard RLHF, which uses a single reward model aimed at general population preferences, RLPHF models multiple objectives.
- Motivation: The approach explicitly represents multifaceted preferences that users declare as desirable, giving users control over objective combinations.The motivation includes preferences such as verbosity and other potentially conflicting response characteristics.
- Approach: PERSONALIZED SOUPS trains separate policy models with distinct preferences using PPO, then merges selected policy parameters during inference.This avoids simultaneously optimizing all preferences in one policy-training process.
- Efficiency: The method reduces computational complexity from exponential to linear in the total number of unique preferences.Its modular training also avoids retraining the underlying policy whenever a novel preference is added.
- Results: PERSONALIZED SOUPS provides deeper adaptation to individual users than supervised fine-tuning, RLHF, and prompting, while remaining competitive with PROMPTED-MORL.The paper evaluates the method in settings where users add novel preferences to existing combinations.
2 RELATED WORK
Related work spans general human-preference alignment, multi-objective reinforcement learning, NLP personalization, and parameter merging. The paper combines these lines by merging policies trained with diverse reward models for multifaceted human feedback.
- Aligning Language Models To Human Preferences: Prior RLHF work trains language models against reward models derived from human preferences and applies the approach across several language-generation tasks.The cited applications include summarization, long-form question answering, and engaging responses.
- Aligning Language Models To Human Preferences: Standard RLHF commonly models average annotator preference, overlooking differences in users’ desired outputs and values.Prior studies report variation in default LLM opinions across demographic groups and other individual preference dimensions.
- Multi-objective Reinforcement Learning: MORL addresses limitations of optimizing a single scalar additive reward by representing multiple objectives in decision-making problems.The paper applies this formulation to language-model personalization and conflicting preferences.
- Personalization in Natural Language Processing: NLP personalization has primarily focused on dialogue agents conditioned on personas or user-history data.These systems target engaging chitchat with distinct user profiles or personas.
- Parameter Merging: Parameter merging uses weighted linear interpolation to compose abilities from models trained for different tasks or capabilities.The paper extends this direction to policies trained with diverse reward models and multifaceted human feedback.
3 REINFORCEMENT LEARNING FROM Personalized HUMAN FEEDBACK
RLPHF reformulates personalized alignment as multi-objective reinforcement learning, allowing conflicting preferences to be modeled explicitly. The section introduces prompted and post-hoc parameter-merging approaches for combining preference-specific policies.
- Standard RLHF optimizes a single reward model representing general population preferences, which cannot model conflicting objectives such as concise versus informative responses.
- RLPHF represents alignment with multiple rewards and dynamically varies objective importance to accommodate conflicting preferences during training and inference.The objectives correspond to preference dimensions, with weights serving as the importance assigned to each objective.
- PROMPTED-MORL: PROMPTED-MORL uses preference prompts as binary objective signals, averages reward values across preferences, and multitask-trains on eight preference combinations.The policy receives combinations such as t + P1 + P2 + P3, while a single reward model produces preference-specific signals through separate forward passes.
- PERSONALIZED SOUPS: PERSONALIZED SOUPS decomposes MORL into independently optimized single-objective policies and merges selected policy parameters with weighted sums during inference.This allows preference combinations not seen during training to be composed on the fly.
- PERSONALIZED SOUPS: Training individual objectives increases computational complexity linearly instead of exponentially and avoids retraining the underlying policy when novel preferences are added.
- Evaluation: The evaluation uses 50 open-ended Koala prompts, eight preference combinations, and aggregated win rates that disregard ties.Responses are compared across preference dimensions using WIN, TIE, and LOSE scores, then aggregated over the evaluation set and combinations.
4 EXPERIMENTS
Experiments compare preference-alignment baselines, MORL methods, and parameter merging using simulated and human pairwise evaluations. Personalized SOUPS performs strongly while reducing scaling costs and retaining helpfulness, though evaluation rankings differ between GPT-4 and humans.
- Baseline Methods: The experiments compare vanilla, RLHF, prompting, multitask training, PROMPTED-MORL, and PERSONALIZED SOUPS baselines.The study reports component differences across methods and evaluates pairwise win rates using GPT-4 and human judges.
- Main Results: P-SOUPS has the highest average win rate in GPT-4 evaluation, while P-MORL has the highest in human evaluation.Both P-MORL and P-SOUPS outperform other methods on average, but simulated and human evaluation rankings differ.
- Main Results: Parameter merging enables P-SOUPS to outperform multitask fine-tuning, exceeding a prior assumption that multitask fine-tuning was the upper bound for composition.The result supports parameter merging as both a distributed multitask method and a method that can surpass multitask training.
- Trade-off Between General Helpfulness: RLHF performs best on the unseen helpfulness objective, while RS, P-MORL, and P-SOUPS retain helpfulness similar to the instruction-tuned baseline.Prompting significantly underperforms the other methods on helpfulness, indicating that personalized preference performance and general helpfulness can trade off.
- Scaling to New Preferences: P-SOUPS remains competitive with P-MORL on 16 preference combinations while training only two new policies for newly added preferences.P-MORL must retrain on all 16 combinations, whereas P-SOUPS merges new policy parameters on the fly during inference.
- Scaling to New Preferences: P-MORL scales as O(2^n), whereas P-SOUPS scales as O(n) when each dimension has two unique preferences.P-SOUPS therefore avoids observing every combination and retraining previous preferences when new preferences are added.
5 CONCLUSION
The paper frames personalized alignment as multi-objective RLPHF and proposes P-SOUPS to compose single-objective models during inference. It reports linear scaling with new preferences, contrasting with exponential scaling in the MORL baseline.
- RLPHF formulates personalized alignment as a multi-objective problem for conflicting individual preferences.
- P-SOUPS composes models trained on single objectives on the fly during inference.
- P-SOUPS scales linearly rather than exponentially with the number of new preferences.
A DETAILS OF EVALUATION SETUP
The evaluation combines simulated pairwise judgments with human evaluation. Simulated judgments use preference-specific GPT-4 criteria, while human evaluation uses an interface with ties and both-bad responses treated as ties.
- Simulated pairwise evaluation: Simulated pairwise evaluation modifies a GPT-4 annotation prompt to judge responses on a single preference dimension.The setup includes demonstrations for ties, clear winners, and cases where both responses are good but one is better.
- Human evaluation: Human evaluation recruited 24 crowd workers using the interface shown in Figure 5.Both ‘Tie’ and ‘Both are bad’ options were counted as ties.
B CRITERIA-WISE EVALUATION
Criteria-wise win rates summarize performance for each preference dimension by averaging the win rates of combinations containing that dimension.
- Criteria-wise win rate (%) averages the win rates of preference combinations containing a specific preference dimension.For ‘Elementary,’ the combinations are AAA, AAB, ABA, and ABB.
C THE FULL LIST OF EVALUATION PROMPTS
The full list of evaluation prompts used in the experiments is provided in Table 5.
- Table 5 provides the full list of evaluation prompts used in the experiments.
D DETAILED RESULTS FOR HUMAN EVALUATION AND GPT-4 EVALUATION
The paper reports detailed win, loss, and tie results for preference combinations, using both GPT-4 and human evaluations.
- The main experimental results report win, loss, and tie outcomes for each preference combination.
- GPT-4 evaluation results are presented in Table 6.
- Human evaluation results are presented in Table 7.
E EXAMPLES OF P-SOUPS TEXT GENERATIONS
The paper illustrates P-SOUPS generations across 16 preference combinations using a shared set of prompts, with examples presented in Table 8.
- The examples use prompts spanning questions, stories, professional messages, explanations, rankings, and travel planning.
- The evaluation materials include a full list of 50 prompts and tables reporting GPT-4 and human pairwise win rates.
- Table 8 provides example output generations for each of the 16 preference combinations from P-SOUPS.