PaperlayerView exact revision on arXiv

Source-linked AI summary

P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling

Pinyi Zhang, Ting-En Lin, Yuchuan Wu, Jingyang Chen, Zongqi Wang, Hua Yang, Ze Xu, Fei Huang, Kai Zhang, Yongbin Li

arXiv:2602.12116v1cs.CL

TL;DR

Personalized reward modeling must capture diverse, context-dependent preferences and generalize to new users from sparse feedback. P-GenRM converts preference signals into structured personas and scoring rubrics, then applies prototype- and individual-level test-time scaling. It achieves state-of-the-art personalized reward benchmark performance, with a reported additional 3% gain from test-time scaling.

  • Problem

    Personalized reward models struggle to represent within-user, scenario-dependent preference variability and to generalize to new users with sparse feedback.

  • Method

    P-GenRM generates structured evaluation chains with adaptive personas and scoring rubrics, then scales individual and prototype-level preference signals at test time.

  • Results

    P-GenRM establishes state-of-the-art performance across personalized reward benchmarks, and test-time scaling provides an additional 3% gain.

  • Takeaways & Limitations

    The framework supports personalized evaluation with explicit personas and rubrics while improving generalization to users with sparse feedback.

  • Takeaways & Limitations

    P-GenRM requires evaluation-chain generation and three historical preference selections, creating efficiency and data-collection constraints.

Abstract

from arXiv · show

Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of 2.31%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability.

1 Introduction

Personalized reward modeling must represent context-dependent user preferences and generalize from sparse feedback. P-GenRM addresses these challenges with structured evaluation chains and test-time user-based scaling, achieving strong benchmark performance and additional gains from scaling.

  • Motivation: Personalized alignment adapts LLM behavior to individual preferences, but open-ended evaluation combines sparse explicit signals with richer, noisy implicit signals.These conditions make reliable user-specific reward signals difficult to obtain.
  • Challenges: Existing personalized reward models struggle with within-user preference variability across scenarios and weak generalization to new users with sparse feedback.The latter limitation directly affects cold-start settings.
  • Approach: P-GenRM transforms hybrid preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics for variable contexts.Its three-stage framework combines Persona-guided Scoring Induction, Criteria-based Reasoning Enhancement, and hard-negative-aware curriculum learning.
  • Results: 3% additional gain is reported from test-time scaling, while P-GenRM achieves state-of-the-art performance on personalized reward benchmarks and strong generalization to new users.The benchmark and generalization claims are reported as experimental results.
  • Approach: Test-time User-based Scaling aggregates individual-level scoring schemes and incorporates similar users’ preferences to improve scoring accuracy and generalization.The mechanism is designed to mitigate noise in inferred preferences and transfer learned preferences to unseen users.

2 Related Works

Related work spans personalized alignment, user preference modeling, and generative reward models. P-GenRM combines these directions while targeting context-dependent preferences and scalable personalized evaluation.

  • Personalized alignment: Personalized alignment research seeks to tailor LLM responses to diverse user preferences through user-specific parameters or inference-time steering.Prior personalized reward models use latent user representations, prototypes, mixtures of experts, decomposed preference dimensions, or linear reward combinations.
  • P-GenRM: P-GenRM is presented as a workflow that infers scenario-specific personas, derives dynamic scoring rubrics, and evaluates candidate responses using hybrid preference signals.At test time, it can aggregate individual scoring schemes and incorporate similar users’ preferences.
  • Personalized alignment: SynthesizeMe infers synthetic personas from historical preferences, but its static design cannot adapt to context-dependent and shifting user preferences.This limitation motivates more dynamic personalization approaches.
  • User preference modeling: User preference modeling includes explicit multidimensional attributes, combinations of user-specified values in prompts, and few-shot prediction of group preferences.These methods represent preferences at different granularities for LLM alignment tasks.
  • Generative reward models: Generative reward models use generative LLM capabilities to support generalization, test-time scalability, and flexible or interpretable alignment evaluation.The cited line of work includes generative verifiers and generative judges.

3 Problem Formulation

P-GenRM formulates personalized response scoring as a structured process: infer preferences and personas, derive weighted criteria, generate evaluation chains, and extract candidate scores. The training pipeline and examples instantiate this process.

  • Problem formulation: A dialogue instance contains the current user query and historical preferred and dispreferred responses from earlier turns.The history represents the user’s prior interaction context up to the current turn.
  • Problem formulation: The history size is limited to h through random selection to avoid excessive reliance on historical data.This is an explicit formulation assumption for constructing the available preference history.
  • Training and preference analysis: Persona-guided Scoring Induction uses preference history to infer a user’s plausible persona and core preferences.The resulting analysis supplies the basis for personalized evaluation criteria.
  • Scoring scheme derivation: The derived scoring scheme expresses scenario-specific evaluation criteria with weights, such as Utility/Actionable Detail at 40%.The formulation also includes criteria such as clarity and structure, alongside preference-history scoring elements.
  • Training pipeline: The three-stage training pipeline combines supervised scoring induction, reinforcement-learning reasoning enhancement, and hard-negative-aware curriculum learning.These stages respectively provide personalized scoring abilities, improve evaluation-chain quality, and strengthen robustness on difficult negative cases.
  • Evaluation chain: The evaluation chain connects preference modeling, scoring-scheme derivation, LLM-as-a-Judge process reward, and final scoring.Its purpose is to assess how completely the chain covers the inferred preference criteria.
  • Final scoring: An example assigns weighted scores of 6.8 and 6.95 to two responses after rubric-based assessment.The example illustrates how criterion weights produce the final comparison.
  • Final scoring: The model formalizes personalized reward scoring by extracting scalar scores from a text-based evaluation process for each candidate response.Rθ denotes the reward model, s_i^t the scalar reward for candidate y_i^t, and b the number of candidates, typically two.

4 Methodology

P-GenRM builds structured, personalized scoring from hybrid preference signals and strengthens it with prototype-based test-time scaling. Its training combines supervised induction, reinforcement learning, and hard-negative curriculum learning, while scaling aggregates individual and similar-user evidence.

  • Persona-guided Scoring Induction: P-GenRM transforms implicit and explicit preference signals into scenario-specific personas, criteria, and personalized scores for candidate responses.The resulting structured evaluation chains support adaptive scoring rubrics across variable contexts.
  • Multi-stage Training Framework: The three-stage training framework uses SFT for initial personalized scoring, RL for evaluation-chain quality, and curriculum learning for difficult negative cases.
  • Persona-guided Scoring Induction: User personas inferred from interaction histories provide priors, while explicitly stated criteria sharpen evaluation precision.
  • Multi-stage Training Framework: RL combines process-level and rule-based outcome rewards, with a weighted overall reward used to compute advantages for GRPO training.The weighting factors α and β balance process and outcome signals.
  • Test-time User-based Scaling: Test-time User-based Scaling addresses noisy preference inference and sparse-feedback generalization through offline prototypes and dual-granularity scaling.
  • Test-time User-based Scaling: Individual-level scaling samples multiple scoring schemes, while prototype-level scaling incorporates preference signals from similar users.Prototype refinement uses historical records and regularization to form priors that guide query-specific discrimination.

5 Experiments

Experiments show that P-GenRM outperforms prior personalized reward-model baselines across model scales, while test-time scaling adds gains with modest latency. Prototype-based scaling, adaptive personas, and structured evaluation components support performance and sparse-feedback generalization.

  • Overall Results: P-GenRM improves over prior SOTA by 2.77% on the 8B model and 1.99% on the 70B model.P-GenRM-8B also surpasses the prior best 70B model by 1.04%.
  • Test-time User-based Scaling: 2.99% average improvement over P-GenRM comes from the Ind-16 and Pro-8 scaling setting with 24 scaling steps.This setting outperforms Ind-32 while using fewer scaling steps, although adding similar-user ratings alone does not always help.
  • Test-time User-based Scaling: P-GenRM maintains superior performance with only a limited increase in inference time and lower latency than prior SOTA methods.The comparison was conducted on the Chatbot Arena-Personalized test set with P-GenRM-8B.
  • Ablation Study: Removing any P-GenRM component causes a significant performance drop, and both process and outcome rewards are necessary.The ablation study evaluates curriculum learning, process reward, and outcome reward.
  • Adaptive vs. Static Personas: Adaptive personas consistently outperform static SynthesizeMe personas across base models in LLM-as-a-judge experiments.P-GenRM infers dynamic personas and corresponding evaluation criteria rather than treating personas as static priors.
  • Prototype Analysis: Performance improves substantially from 0 to 50 prototypes, then plateaus and slightly degrades at 100 under Ind-8, Pro-4 scaling.The method selects 50 prototypes because additional prototypes provide marginal information gains while increasing inference costs.
  • Prototype Analysis: Users cluster around prototypes with shared core preferences, individual variation, and divergent preferences across clusters.This balance between intra-group similarity and inter-group heterogeneity supports prototype-based test-time scaling.
  • Sparse-Feedback Generalization: P-GenRM-8B with Ind-8, Pro-4 outperforms all baselines on LaMP-QA under sparse feedback, including the larger Qwen3-235B-A22B.LaMP-QA rankings are evaluated using Spearman correlation against a constructed ground-truth ranking.

6 Conclusion

The conclusion presents P-GenRM as a personalized generative reward model that structures heterogeneous preferences and aggregates individual and prototype-level signals at test time. It reports state-of-the-art benchmark performance, further gains from scaling, sparse-feedback generalization, and explicit interpretability.

  • Conclusion: P-GenRM transforms heterogeneous preference signals into structured, scenario-aware evaluation chains and aggregates individual and prototype-level signals.The framework uses explicit personas and rubrics as part of its evaluation representation.
  • Conclusion: P-GenRM establishes a new state of the art across personalized reward benchmarks, while test-time scaling yields further gains with modest additional compute.The conclusion also reports improved fidelity in subjective evaluation and enhanced generalization to users with sparse feedback.

A.2 Preliminary Experiments on Effectiveness of User Persona

Preliminary LLM-as-a-judge experiments test whether explicit preference indicators improve personalized scoring. They find consistent gains from preference descriptions, with Persona, OSR, and SDim producing the largest improvements, while both reward types remain important in reinforcement learning.

  • Preference Indicators: Explicit preference descriptions consistently improve scoring accuracy over a baseline without persona information.The experiment uses PRISM data, OpenAI o3-2025-04-16, and mean accuracy averaged over five runs.
  • Preference Indicators: Persona, OSR, and SDim produce the most pronounced performance gains among the tested preference indicators.The result suggests persona can serve as a useful preference indicator when explicit preferences are difficult for LLMs to infer.
  • Reward Design: Both process-related and outcome-related rewards are essential, because removing either significantly degrades reinforcement-learning performance.Excessive emphasis on process rewards may overfit the model to specific criteria and impair overall performance.

A.5 Performance across different prototypes

Prototype-wise analysis examines user-group balance and performance across clustered personas. Despite a long-tail group distribution, P-GenRM shows stable accuracy across prototypes and achieves the highest macro accuracy among evaluated baselines.

  • Prototype Distribution: The PRISM user groups exhibit a clear long-tail distribution under the paper’s clustering method.Prototype-wise analysis uses macro accuracy so each persona group receives equal weight.
  • Performance Across Prototypes: Prototype-level macro accuracy is 0.6521 versus sample-level average accuracy of 0.6532, a difference of only 0.0011.Persona-level accuracies have median 0.6500, standard deviation 0.0401, and IQR 0.0544.
  • Baseline Comparison: P-GenRM achieves the highest macro accuracy, 65.21%, among all evaluated baselines.The comparison includes open-source and proprietary models with strong prompting strategies.
  • Scenario Variation: Figures 6 and 7 show that one user can have differing preferences across music recommendation and serious discussion scenarios.These examples illustrate scenario-dependent preference variation for an individual user.

A.7 Evaluation framework used in Lamp-Qa dataset

The LaMP-QA evaluation pipeline generates multiple candidate responses, scores them with reward models, and compares induced rankings against ground-truth rankings. It also evaluates sparse-feedback personalization and measures the inference-time cost of test-time scaling.

  • Evaluation pipeline: Six LLMs generate candidate responses for each LaMP-QA query because the dataset lacks fixed candidates and ground-truth responses.The evaluation uses Qwen3-8B, Qwen3-32B, Qwen-235B-A22B, GPT-5, GPT-4o, and Gemini-2.5-pro.
  • Sparse-feedback evaluation: Under sparse feedback, each reward model receives three historical user interactions, and scores are normalized by repeating scaling eight times and averaging.The evaluated systems include several foundation models, SynthesizeMe, and P-GenRM-scale.
  • Evaluation pipeline: Reward-model rankings are compared with ground-truth rankings using Spearman rank correlation across LaMP-QA’s three subsets.This evaluates whether reward models induce rankings consistent with the benchmark’s reference rankings.
  • Inference cost: Inference time increases from 00:14:16 to 00:23:05 with scaling, while the method remains faster than larger models.The added cost is attributed to prompt encoding and efficiently parallelized individual- and prototype-level scaling.
  • User–prototype analysis: User–prototype visualization shows shared preferences with within-cluster variation and clearly distinct tendencies across clusters.The figure presents learned prototypes and representative preference patterns.

A.10 Number of samples required for generating reasonable user preference

The appendix examines how many historical preference samples are needed for reliable personalization. Performance improves substantially at three samples, while additional samples mainly improve stability.

  • Sample requirement: Three preference samples provide the minimal structure for assessing consistency, reducing randomness, and supporting reliable personalization.One sample reflects a one-off choice, while two remain insufficient for a consistent pattern.
  • Sample requirement: Beyond three samples, additional data primarily improves evaluation stability rather than producing substantial performance gains.The authors therefore use three preference samples for both training and evaluation.
  • Prototype analysis: The appendix also reports P-GenRM-8B performance under different prototype counts with the (Ind-8, Pro-4) setting.The supplied passage identifies the comparison but does not provide the corresponding values.

A.12 P-GenRM for policy model’s training

P-GenRM is used as a reward model for training personalized policy models with both GRPO and DPO. Across five runs, the resulting 8B policies have confidence intervals above those of the 70B policies, while the method still requires evaluation-chain generation and three historical preferences.

  • GRPO and DPO training: P-GenRM scores candidate responses in GRPO, and its scores are incorporated into relative-advantage estimation for policy training.The setup uses users’ historical preference pairs to assess response alignment.
  • GRPO and DPO training: P-GenRM supplies DPO labels by treating its chosen responses as positive instances and rejected responses as negative instances.These preference pairs are then used to train the policy model through DPO.
  • Policy evaluation: Policy responses are judged by GPT-4o, Claude-Sonnet-4, and Gemini 2.5-Pro on a 1–5 personalized-quality scale.The evaluation averages the three judge scores and repeats the procedure five times.
  • Policy results: 95% confidence-interval lower bounds of 3.303 and 3.334 for the 8B policies exceed the 70B upper bounds of 3.174 and 3.228.The non-overlapping intervals are reported across five independent runs.
  • Limitations: The method requires evaluation-chain generation for reliable personalized scores and three historical preference selections for preference analysis.These are identified as current limitations affecting inference efficiency and practical data collection.

B.1 Dataset

The appendix describes the personalized-reward datasets and the prompt variants used to infer user preferences. The datasets span conversational, value-focused, and question-answering settings, while prompts vary the available user information and scoring criteria.

  • Datasets: Chatbot Arena contains open-ended conversations, anonymous model comparisons, and pairwise user preferences, with the evaluated subset covering 131 users.It provides in-the-wild prompts and feedback for realistic conversational evaluation.
  • Datasets: PRISM contains data from 720 users and maps detailed survey responses onto live multi-turn LLM conversations.Users rate candidate completions on a 1–100 scale and provide attribute-level feedback.
  • Datasets: LaMP-QA uses StackExchange questions and histories to model users’ information needs across three broad categories and more than 45 subcategories.Its hidden, human-checked evaluation aspects support fine-grained personalized assessment.
  • Related methods: GPO, VPL, PAL, and SynthesizeMe represent alternative approaches for few-shot group or user preference modeling.They respectively use a preference module, latent user context, prototypical preference points, or LLM-inferred personas and informative examples.
  • Prompt variants: Prompt variants provide explicit requirements, self-description, demographics, choice attributes, or response history as inputs to preference analysis.The choice-attribute prompt treats higher attribute scores as indicating greater importance.
  • Prompt variants: Persona-guided scoring constructs a personalized scoring model for the current scenario before scoring new responses, while criteria-based enhancement uses chosen and rejected history pairs.These prompts operationalize preference inference and response evaluation.
Loading 2602.12116v1…