PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Model Personalization via Reward Factorization

Idan Shenfeld, Felix Faltings, Pulkit Agrawal, Aldo Pacchiano

arXiv:2503.06358v1cs.LG

TL;DR

Standard RLHF assumes a universal preference model, leaving individual user preferences unaccounted for. PReF models each user’s reward as a combination of shared base rewards, and reports a 67% win rate over default GPT-4o answers in its real-user evaluation.

  • Problem

    Standard RLHF assumes a universal preference model across users and does not account for individual preference differences, limiting personalization.

    Source: §1 · p. 1

  • Method

    PReF assumes user preferences share low-dimensional structure and represents each user’s reward as a linear combination of base reward functions with user-specific weights.

    Source: §1 · pp. 1–2

  • Results

    PReF outperformed standard RLHF by a large margin on synthetic data and achieved a 67% win rate over GPT-4o default answers with real users.

    Source: §1 · p. 2, §5.2 · p. 9

  • Takeaways & Limitations

    Once base features are fixed, new-user weights can be learned from a small number of preference feedbacks and used to adapt generations without retraining.

    Source: §6 · pp. 10–11

    Many experiments were limited to synthetic users because available datasets generally lack preference feedback from multiple users on the same response pairs, making learning ill-posed.

    Source: §6 · p. 11

Abstract

from arXiv · show

Modern large language models (LLMs) are optimized for human-aligned responses using Reinforcement Learning from Human Feedback (RLHF). However, existing RLHF approaches assume a universal preference model and fail to account for individual user preferences, limiting their effectiveness in personalized applications. We introduce a framework that extends RLHF to enable user personalization by leveraging the assumption that user preferences lie in a low-dimensional space. Instead of training a separate model per user, we represent user-specific rewards as a linear combination of base reward functions. Using only ~10 user responses, our method can infer user-specific rewards and align LLM outputs accordingly. We validate our approach through experiments with both synthetic and real users, demonstrating significant personalization achieved by our method. In human evaluations, our method achieves a 67% win rate over default GPT-4o responses.

1 Introduction

Standard RLHF assumes a universal preference model, while users’ preferences vary and per-user training can demand substantial data and computation. PReF represents user rewards through shared base functions and estimates user-specific weights to personalize responses.

1 Introduction

  • PReF assumes preferences share low-dimensional structure and represents each user’s reward as a linear combination of base reward functions.Under the Bradley-Terry formulation, user-specific coefficients weight the base functions using pairwise response preferences.Source: §1 · p. 1
  • PReF reduces personalization to estimating user-specific coefficients, making it simpler and more data-efficient than learning a separate reward model per user.The framework uses shared preference data to learn base functions, then infers a new user’s coefficients from their response choices.Source: §1 · pp. 1–2
  • Prior linear combinations of reward functions did not address learning the base functions and inferring each user’s combination, which PReF targets.The related-work passage identifies both as core problems for data- and compute-efficient personalization.Source: §1 · p. 2
  • Adaptive active learning selects questions using prior answers to reduce uncertainty in user coefficients, which the method can estimate with 10–20 questions.The figure caption reports up to 30× fewer answers than standard RLHF for the same performance.Source: §1 · p. 2, Figure 1 · p. 2
  • After identifying a user’s reward function, PReF uses inference-time alignment to generate reward-aligned responses without changing LLM weights.The passage describes this as enabling efficient, scalable adaptation without costly model updates.Source: §1 · p. 2
  • 67% win rate over default GPT-4o answers shows PReF’s personalization with real users; on synthetic data, five samples let its reward model outperform a standard reward model.The introduction also reports that personalization can require up to 30× fewer answers than standard RLHF for the same performance.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Related work spans several forms of personalization and reward-based alignment. The paper distinguishes its approach by learning user-specific combinations of reward functions and applying active learning to gather personalization feedback.

2 Related Work

  • Personalization includes user-specific knowledge, domain expertise, and response styles; this work focuses on aligning preferences.The passage cites examples across these categories, including (Ning et al., 2024; Wu et al., 2024; Richardson et al., 2023; Kirk et al., 2024a; King & Cook, 2020).Source: §2 · p. 2
  • RLHF trains a reward model from human-annotated response pairs and often requires thousands to hundreds of thousands of labeled examples.The passage cites (Gao et al., 2023) for the range.Source: §2 · p. 3
  • Multi-reward approaches decompose preferences into properties such as helpfulness, harmlessness, and factuality, but personalization variants typically require separate models per reward combination.The passage cites (Bai et al., 2022a; Wang et al., 2024b; Dorka, 2024) for decomposition and (Guo et al., 2024; Zhou et al., 2023; Yang et al., 2024b; Wang et al., 2024c) for personalization variants.Source: §2 · p. 3
  • Inference-time reweighting enables dynamic adaptation without retraining, while other approaches combine separately trained models in weight space.These approaches rely on predefined reward functions and explicitly specified user preferences.Source: §2 · p. 3
  • Closest related approaches model preferences as latent variables (Poddar et al., 2024) or ideal points in a shared latent space (Chen et al., 2024a), whereas this work uses linear combinations of base rewards.The paper also uses active learning to infer each user’s combination after learning the base functions.Source: §2 · p. 3
  • The paper reports that, to its knowledge, it is the first to apply active learning specifically to LLM personalization, though bandit, online, and active learning have been explored in RLHF.The passage cites (Das et al., 2024; Saha et al., 2023; Ji et al., 2024) for prior RLHF work.Source: §2 · p. 3

3 Preliminaries

The preliminaries formulate personalization as learning a reward for an individual user and generating responses that maximize it. They contrast pairwise preference modeling with standard RLHF’s single global reward.

3 Preliminaries

  • The objective is to generate responses that align with an individual user’s preferences by learning that user’s reward function and using it to guide generation.The reward assigns a scalar to a response for a given prompt.Source: §3 · p. 3
  • The paper uses pairwise comparisons and the Bradley-Terry choice model to represent a user’s preference between two responses.The model defines the probability that one response is preferred over another for a prompt and user; σ is the sigmoid function.Source: §3 · p. 3
  • Standard RLHF learns one global reward by maximizing the likelihood of pairwise comparisons across the dataset, assuming preferences are homogeneous across users.The objective treats all comparisons as arising from the same reward function.Source: §3 · pp. 3–4
  • Aggregating all users’ comparisons into a single reward model overlooks individual preference differences and can yield suboptimal personalization.Source: §3 · p. 4

4 The PReF framework

PReF represents each user’s reward as a weighted combination of shared base reward functions. It learns those functions from multi-user preference data, then estimates new users’ weights from pairwise feedback to adapt generations without retraining.

4 The PReF framework

  • PReF models each user’s reward as a linear combination of base reward functions, with user-specific weights capturing the importance of each function.The learned representation can support inference-time adaptation without retraining.Source: §4 · p. 4

4.1 Learning the Base Functions

  • SVD initialization followed by regularized maximum-likelihood refinement addresses instability from the bilinear coupling of user weights and base features.The factorization is non-unique, and L2 regularization of user vectors reduces extreme parameter values and scale ambiguity.Source: §4.1 · p. 5

4.2 Adaptation to a New User

  • For a new user, PReF iteratively collects pairwise feedback and estimates weights by regularized logistic regression once the base features are fixed.Active learning selects response pairs with the greatest potential prediction error to reduce uncertainty in those weights.Source: §4.2 · pp. 5–6
  • The theoretically tight confidence set is computationally infeasible to optimize directly, so PReF uses an expanded set with a Hessian evaluated at the current estimate.The expanded set introduces exponential dependence on dimension, though the authors report that their response-selection strategy is not explicitly affected.Source: §4.2 · p. 6
  • Figure 2 reports that PReF quickly achieves high ROC AUC and win rates, outperforming baselines on Attributes and PRISM.The figure caption does not provide specific metric values.Source: Figure 2 · p. 7

5 Experiments

The experiments assess PReF on synthetic and real-user settings, against personalization baselines, and through scaling, ablation, and feature-interpretation analyses.

5 Experiments

  • Experiments use synthetic Attributes and PRISM data, repeat experiments across 10 random seeds, and test users and prompts distinct from training.Evaluation includes reward-classification User Preference AUC-ROC and win rate after inference-time alignment.Source: §5 · pp. 7–8

5.1 The Benefits of personalization

  • For fewer than 10 user answers, PReF gains 10–15% AUC-ROC over Classic RLHF across both datasets; win rate improves around 10% on PRISM and over 25% on Attributes.The Model per User baseline performs poorly; the paper reports PReF requires 25× less data for the same performance.Source: §5.1 · p. 8

5.2 Can PReF capture the preferences of real humans?

  • 67% winrate over GPT-4o (95% CI [57.4%, 76.6%]) was achieved after learning user preferences from 15 interactions.The human evaluation used 28 volunteers, with 15 comparisons for learning and 15 for evaluation.Source: §5.2 · pp. 8–9

5.3 How PReF performs against other personalization frameworks?

  • On Attributes, PReF outperforms VPL at 10 and 20 user responses on unseen-response AUC-ROC, while PAL achieves significantly lower performance.Against non-personalized completions, system-prompt responses achieve 61.9% win rate; PReF reaches 76.7% with 5 responses and 80.2% with 10.Source: Table 1 · p. 9, §5.3 · p. 9

5.4 Scaling data and compute leads to better base reward functions

  • ROC AUC improves consistently with larger training datasets and reward networks, though model-size performance begins to converge as datasets grow.Figure 4 plots dataset size on the x-axis and reward-network size by color for PRISM.Source: §5.4 · p. 9, Figure 4 · p. 10
  • With 40 user-specific samples, increasing the number of base reward functions beyond six yields diminishing returns, indicating a trade-off between expressivity and data efficiency.The authors suggest the reward preference matrix’s eigenvalues as a possible heuristic for choosing the number of functions and reducing hyperparameter tuning.Source: §5.4 · pp. 9–10

5.5 Ablations

  • 2.7× efficiency: uncertainty-based active selection reaches the same performance with 15 samples that random selection needs over 40.Removing SVD initialization or L2 regularization increases training instability; removing SVD particularly increases variance in the new-user setting.Source: Figure 3 · p. 8, §5.5 · p. 10

5.6 Feature Interpretation

  • The learned base rewards recover interpretable preference dimensions resembling the synthetic Attributes, including Informal vs. Formal and Conciseness vs. Elaborateness.Source: §5.6 · p. 10

6 Discussion

The discussion presents PReF as an on-the-fly personalization method based on factorized user preferences, while noting that limited overlapping preference annotations constrain many experiments to synthetic data.

6 Discussion

  • PReF models each user’s preferences as a linear combination of base features and adapts generations on the fly.The discussion also notes that the method can learn base features from data.Source: §6 · pp. 10–11
  • Figure 5 recovers categories resembling the attributes used to generate the data, including formality and response length.The figure includes sorted principal components and LLM-generated descriptions.Source: Figure 5 · p. 11
  • Existing datasets often lack preference feedback from multiple users on the same response pair, making base-feature learning ill-posed and limiting many experiments to synthetic data.The authors identify collecting a large overlapping-annotation dataset as future work.Source: §6 · p. 11

A Uncertainty

The uncertainty analysis describes an ellipsoidal confidence set and a high-probability parameter-error bound, with a proof step that converts a matrix-induced norm constraint into an Euclidean one.

A Uncertainty

  • With probability at least 1 − δ, for all t ∈ N, the Euclidean parameter error ∥θ*−θ̂_t∥2 is at most (2 + 4S)γ_t(δ)/√β.The H_t(θ*)-norm error is at most (2 + 4S)γ_t(δ).Source: A Uncertainty · p. 15
  • The confidence region is an ellipsoid around θ̂, defined by a Σ-norm radius β in d dimensions.The text defines the matrix-induced norm as ∥z∥_A = √(zᵀAz).Source: A Uncertainty · p. 15
  • The proof maps the constraint ∥v′∥_Σ ≤ 1 to the Euclidean unit ball by setting z = Σ^1/2v′.It then uses Cauchy–Schwarz to identify the maximizing direction as proportional to Σ^-1/2x.Source: A Uncertainty · p. 16

B.1.1 Data Generation

The Attributes data generation simulates users with combinations of preference traits, collects their judgments on paired responses, and uses prompts to elicit responses and preferences.

B.1.1 Data Generation

  • The Attributes setup simulates 84 users by assigning each two traits drawn from seven categories with positive and negative alternatives.For example, the length category distinguishes preference for verbose versus concise responses.Source: §B.1.1 · p. 16
  • Each simulated user provides preferences on the same 100 randomly sampled AlpacaEval prompts, with response pairs designed to express contrasting traits.Preferences are gathered over pairs generated using the assigned user traits.Source: §B.1.1 · p. 16
  • Judgments are collected in both response orders, and the two preference matrices are averaged to reduce order bias.The two matrices encode whether the simulated user prefers response A or B.Source: §B.1.1 · p. 16

B.1.2 Prompts

  • GPT-4o is used with prompts that generate trait-conditioned responses and ask a judge to choose the output that best matches the instruction.The preference prompt is sourced from AlpacaEval; the response prompt supplies two attributes.Source: §B.1.2 · pp. 16–17

B.2.1 Data Generation

The PRISM-based data generation uses trait-conditioned response revisions and simulated preference judgments for a subset of user–response-pair combinations, with prompts guiding both steps.

B.2.1 Data Generation

  • The dataset draws on 1,500 PRISM users and 2,262 controversial prompts, using self-reported traits spanning communication style, values, demographics, and LLM familiarity.GPT-4o simulates user responses following the PERSONA roleplay protocol.Source: §B.2.1 · p. 17
  • For each prompt, a baseline response is revised with Qwen 2.5 7B to reflect a sampled user’s preferences, producing contrasting response pairs.The examples contrast revisions for users valuing factual fluency versus creativity and engagement.Source: §B.2.1 · pp. 17–18
  • Sampling 50 users per response pair yields about 110,000 preference data points, roughly 3% of the full preference matrix.The data are split by users and pairs for training and testing; judgments are elicited in both response orders.Source: §B.2.1 · p. 18

B.2.2 Prompts

  • Preference prompts ask a judge to select which response a given user profile would most likely prefer, rather than which is better generally.The prompt supplies the profile, user question, and both assistant answers.Source: §B.2.2 · pp. 18–19
  • Response-generation prompts first request a critique of a completion against a user profile, then ask for a revised completion based on that critique.The revision prompt asks not to refer directly to the demographic’s preferences or attributes.Source: §B.2.2 · p. 19

C Training Details

Training details specify shared evaluation settings and baseline-specific hyperparameter choices. Table 3 lists the models’ hyperparameters, with experiments run over 10 seeds and at most eight tuning trials per algorithm.

C Training Details

  • Experiments used 10 random seeds, with eight hyperparameter-tuning trials per algorithm; Table 3 lists the trained models’ hyperparameters.The authors state these settings were used unless otherwise noted and intended to ensure fair comparison.Source: C Training Details · p. 19, Table 3 · p. 20
  • The Classic RLHF baseline used the proposed method’s hyperparameters, except that it had one base function.The number of base functions was set to 1 for this baseline.Source: C Training Details · p. 19
  • For Model per User, freezing the backbone worked best with 5–40 user answers, while a 1e-6 backbone learning rate worked best with 100+ answers.The linear head’s learning rate was fixed at 1e-3.Source: C Training Details · p. 19

D Human Evaluations

The human evaluation reports varied preferences across 25 participants and describes its evaluator pool, protocol, and response-generation procedure. The accompanying feature-interpretation procedure uses response extremes to label and summarize principal components.

D Human Evaluations

  • Evaluators were Harvard or MIT STEM graduate students or postdoctoral researchers and judged 30 unique prompts, with the first 15 selected online and the next 15 at random.They could choose either response or neither, with neither counted as a tie; Figure 6 shows screenshots of the web app.Source: D Human Evaluations · p. 20, Figure 6 · p. 21
  • The uncertainty-guided procedure selects response pairs to maximize Equation (6) after an initial random pair, then updates user weights from all collected preferences using Equation (5).The algorithm outputs the estimated user weights.Source: D Human Evaluations · p. 20
  • Among 25 participants, some preferred personalized responses almost always while another group was close to indifferent; the authors suggest the limited feature set may leave some users without a winning personalization axis.Figure 8 presents the participant win-rate histogram.Source: D Human Evaluations · p. 21, Figure 8 · p. 23
  • Because GPT-4o could not be fine-tuned, personalized responses were selected from a pool generated with random attributes and produced by revising the baseline response.The revision approach was used to control for confounders.Source: D Human Evaluations · p. 21
  • The elbow in the first 50 preference-matrix eigenvalues is proposed as a heuristic for choosing the number of base reward functions, consistent with performance saturation in Figure 3.The authors describe eigenvalue analysis as a potentially efficient dimensionality-selection heuristic.Source: Figure 7 · p. 22
  • A reward model trained on one user’s answers required over 500 pairwise comparisons to reach full performance, which the authors deem infeasible at scale.Figure 9 presents this single-user training result.Source: Figure 9 · p. 23

E Feature Interpretation

  • For feature interpretation, principal components of the response feature matrix define axes whose top and bottom responses GPT-4 labels, then the labels are shortened into concise X-versus-Y phrases.The labeling prompt asks GPT-4 to identify the property separating the response extremes.Source: E Feature Interpretation · pp. 23–24
Loading 2503.06358v1…