Source-linked AI summary
Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao
TL;DR
Reward-model benchmarks largely assess general quality, leaving personalized preference modeling and its connection to downstream policy quality under-evaluated. The paper introduces Personalized RewardBench, constructs rubric-based personalized preference pairs, and finds that current models struggle while the benchmark correlates strongly with BoN and PPO performance.
Problem
Existing reward-model benchmarks underrepresent individual user preferences and may not predict downstream policy quality, creating a personalization and proxy gap.
Method
Personalized RewardBench uses user profiles and user-specific rubrics to construct chosen and rejected responses differing in personal alignment, then evaluates reward models and downstream policies.
Results
Current state-of-the-art reward models reached only 75.94% accuracy, while benchmark performance strongly correlated with downstream policy generation quality in BoN and PPO.
Takeaways & Limitations
Personalized RewardBench provides a predictive testbed for evaluating reward models in personalized alignment and downstream applications.
Takeaways & Limitations
The study leaves more advanced profile-aware reward-model architectures to future work.
Abstract
from arXiv · showhide
Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness, relevance and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94%. Crucially, because an effective reward model benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.
1 Introduction
Personalized RewardBench addresses the gap between general-quality alignment and individual user preferences by evaluating reward models on explicitly personalized distinctions. Human evaluation supports the benchmark’s focus on personal alignment, while experiments show substantial personalization difficulty and stronger downstream relevance.
- Personalized alignment remains difficult because generally good responses can be suboptimal when they miss a user’s knowledge, style, constraints, or needs.
- Current benchmarks lack rich user-specific preferences and often lack demonstrated correlation with downstream policy performance.
- Personalized RewardBench constructs response pairs that differ by adherence to user-specific rubrics rather than by general response quality.
- 75.94% was the best accuracy achieved by evaluated state-of-the-art reward models, revealing difficulty distinguishing personalized preferences.
- Human evaluation found chosen and rejected responses to have equivalent high general quality while differing exclusively in personal alignment.
- The benchmark showed superior correlation with downstream performance under both BoN and PPO compared with prior personal reward benchmarks.
2 Related Work
Prior reward-model benchmarks emphasize broad quality, while personalized alignment requires modeling heterogeneous user needs. Personalized RewardBench targets both this evaluation gap and the proxy gap between benchmark accuracy and downstream policy quality.
- RewardBench and RewardBench 2 broadened reward-model evaluation across chat, reasoning, and safety, but predominantly used general quality rubrics.
- Pluralistic alignment research argues that models must adapt to conflicting values and diverse user needs across global populations.
- LaMP-QA tests use of user history, whereas PersonalRewardBench filters personalization-compatible queries without explicitly integrating user profiles into reward evaluation.
- The proxy gap is the discrepancy between reward-model benchmark accuracy and the actual quality of the policy it produces.
- Existing attempts to connect reward accuracy with downstream tasks often rely on indirect proxies, unlike this study’s BoN and PPO consistency analysis.
3 Personalized RewardBench
Personalized RewardBench combines user profiles, personalized rubrics, and chosen/rejected responses to isolate personal-preference adherence. Human evaluation verifies that general quality remains high while personal-rubric performance distinguishes the pairs.
- Dataset Construction: Each benchmark instance contains a query, user profile, chosen response, and rejected response.
- Dataset Construction: The benchmark evaluates reward models and is designed to assess alignment between benchmark performance and downstream performance.
- Dataset Construction: Samples cover Arts & Entertainment, Lifestyle & Personal Development, and Society & Culture, where subjective preferences are important.
- Dataset Construction: User profiles retrieve the top k = 10 relevant historical interactions for each query.
- Dataset Construction: Preference pairs are generated to distinguish adherence to specific user preferences from generic response quality.
- Dataset Construction: The chosen response receives the alignment criteria, whereas the rejected response is deliberately generated without satisfying them.
- Human Evaluation: Human evaluation measures factuality, relevance, helpfulness, harmlessness, and personal-rubric satisfaction across the responses.
- Human Evaluation: Both response types score highly on general quality, but rejected responses perform markedly worse on personal rubrics.
4 Experiments
The experiments evaluate reward models on personalized preference pairs and show a substantial personalization gap. Profile integration is challenging: naive injection degrades performance, while planner-based rubrics recover or exceed the profile-free baseline.
- Experimental Setup: Accuracy is the primary metric, classifying an instance correctly when the chosen response receives a higher reward than the rejected response.Scalar models compare independently computed scores for the two responses.
- Benchmark Performance: Gemini-3-Flash fails to surpass 76% accuracy across any domain, indicating limited personalization despite strong general reasoning.The authors attribute above-chance profile-free performance to partial overlap between personal rubrics and universal preference heuristics.
- Benchmark Performance: Larger models do not consistently perform better: Internlm2-20b underperforms its 7b counterpart, and mR3-Qwen3-14B regresses relative to its 8B version.The findings suggest personalized alignment requires specialized objectives rather than scale alone.
- Benchmark Construction: The benchmark incorporates user profiles and personal rubric aspects, with chosen and rejected responses differing in personal alignment rather than general quality.It uses profile history and personalized rubric criteria to target individual preferences.
- User Profile Integration: Planner-based profile integration translates user history into structured rubrics, mitigating naive profile injection and consistently recovering or surpassing baseline accuracy across domains.The planner generates estimated rubric aspects that are then supplied to reward models; finetuned variants are presented as simple baselines rather than performance-maximizing methods.
5 Downstream Validation
The downstream validation tests whether benchmark rankings predict policy quality under Best-of-N sampling and PPO. Personalized RewardBench outperforms existing baselines across correlation metrics, supporting its use as a practical proxy for downstream reward-model effectiveness.
- Experimental Setup: The study compares reward-model rankings from the static benchmark with rankings from downstream policy performance under Best-of-N sampling and PPO.A lightweight fixed policy model is used to isolate the reward signal’s contribution.
- Experimental Setup: In Best-of-N sampling, the policy generates 16 candidates and the reward model selects the highest-scoring response for rubric-based evaluation.Because the policy remains fixed, downstream rankings directly reflect reward-model guidance.
- Main Results: Personalized RewardBench significantly outperforms existing baselines across all correlation metrics in both BoN and PPO settings.The comparison includes PersonalRewardBench, Chatbot Arena-Personalized, and PRISM-derived evaluation sets.
- Main Results: BoN top-tier identification reaches NDCG 0.9180 and weighted τ 0.3409, whereas Chatbot Arena-Personalized shows a negative correlation of -0.0736.These metrics emphasize ordering among high-performing models.
- Main Results: Weighted τ rises to 0.4793 during PPO, while baseline signals degrade further, supporting the benchmark’s stability as a training objective.The authors conclude that benchmark accuracy faithfully predicts how reward models steer policies in practice.
6 Conclusion
Personalized RewardBench addresses the lack of personalized assessment in pluralistic alignment and provides a testbed for evaluating reward mechanisms against user-specific needs. Its benchmark performance strongly correlates with downstream policy-generation quality in both BoN and PPO.
- Personalized RewardBench evaluates reward models on personalized assessment within pluralistic alignment.
- Human evaluation confirms that chosen and rejected responses have equivalent general quality and differ in adherence to user-specific needs.
- Current state-of-the-art reward models struggle significantly on personalized preferences.
- Benchmark performance strongly correlates with downstream policy-generation quality in both BoN and PPO settings.
A.1 Dataset Statistics
This section reports benchmark statistics and clarifies the evaluation basis for downstream correlation analysis. Accuracy uncertainty is estimated from the paired examples in each domain, while the correlation analysis uses reward-model rankings.
- Personalized RewardBench is composed exclusively of test sets.
- 95% Wilson score confidence intervals are reported for each reward model’s accuracy in each domain.The intervals are computed over 767–1,074 paired examples per domain, rather than over six scalar reward models.
- Each accuracy in the correlation analysis is estimated from approximately 103 independent pairwise observations per model.The resulting confidence-interval half-widths are 0.027–0.036, roughly ±0.03.
- Downstream correlations are calculated from reward-model rankings, with raw evaluation scores and corresponding rankings reported separately.
A.4 Personalized Reward Models
Personalized reward-model experiments compare architectures, profile conditioning, and an oracle upper bound. Results show that personalization depends on more than parameter count, while ground-truth rubric information enables near-perfect decisions.
- Personalized reward models: Personalized reward models are evaluated using Chatbot Arena-Personalized data with Llama-3.1-8B and Llama-3.2-3B backbones.The experiments address possible train-test misalignment when general reward models are applied to subjective tasks.
- Architectural and scale divergence: The smaller Llama-3.2-3B backbone matches or outperforms Llama-3.1-8B across most personalized methods.For example, Bradley-Terry without profiles reaches 71.15% in Lifestyle and 71.01% in Society on 3B, versus 57.53% and 60.89% on 8B.
- Role of user profile: On Llama-3.1-8B, user profiles improve accuracy in 12 of 15 method–domain combinations.The largest reported gains are PAL’s +7.38% in Lifestyle and +5.77% in Society.
- Role of user profile: Profile conditioning has mixed effects on Llama-3.2-3B, with some methods improving and Bradley-Terry and SynthesizeMe declining slightly.Bradley-Terry in Lifestyle falls from 71.15% to 68.55%.
- Upper-bound performance: The oracle model receives ground-truth rubric aspects and predicts choices from the user query, profile, and rubric information.Its accuracy reaches 97.78% in Art & Entertainment, 99.09% in Lifestyle & Personal Development, and 98.60% in Society & Culture.
- Upper-bound performance: State-of-the-art reward models peak at 75.94%, leaving a gap of over 20 percentage points from the near-perfect oracle scores.The paper attributes this gap to difficulty inferring and applying user-specific alignment criteria.
- Above-chance accuracy without user profiles: Above-chance profile-free accuracy reflects overlap between personal rubrics and universal preference heuristics, but this overlap is bounded.The benchmark quantifies the remaining genuinely personal preference signal beyond surface-level cues.
B.3 Artifact Analysis: Ruling Out Stylistic Shortcuts
The artifact analyses test whether reward models distinguish paired responses through superficial style rather than rubric adherence. High lexical similarity, style-only baselines, and paraphrase tests support a semantic preference signal.
- The artifact analysis targets templatic phrasing, response length, and stylistic fingerprints as possible shortcuts.
- Lexical-overlap statistics: Paired responses have high mutual lexical overlap, with ROUGE-1 between 0.39 and 0.41.The analysis compares paired responses with one another and with the rubric aspects they adhere to or violate.
- Style-classifier baselines: Style-only logistic-regression classifiers use function-word frequencies and stylometric features under a user-id-stratified 80/20 split.
- Paraphrase robustness: Paraphrasing reduces mean response overlap while leaving reward-model preferences statistically stable.ROUGE-1 overlap drops from 0.404 to 0.344, and accuracy shifts across six scalar reward models satisfy |∆| ≤0.021.
- Together, the analyses indicate that the benchmark’s preference signal is semantic, persists under paraphrasing, and reflects adherence to user-specific rubric aspects.
B.4 Agreement Between LLM-as-a-Judge and Human Evaluation
The LLM-as-a-judge protocol closely matches human preference directions across Personalized RewardBench, with disagreements concentrated in conservative ties rather than directional bias.
- The validation compares normalized judge scores with human ratings over all 2,830 pairs using the aspect–rubric evaluation template.The judge was Qwen2.5-32B-Instruct, and raw scores were normalized to [0, 1].
- 97.5% directional agreement was achieved across all 2,830 benchmark pairs.Agreement increased to 99.7% when ties were excluded.
- Direct contradictions were negligible at 0.32% of all pairs.Only 9 of 2,830 pair-level judgments contradicted human preferences.
- Residual disagreements were almost entirely judge-assigned ties on pairs humans preferred, reflecting conservative scoring rather than directional bias.
C Implementation Details
The implementation evaluates zero-shot personalized reward models on held-out Personalized RewardBench domains, using controlled persona ablations and training data derived separately from the benchmark.
- Training and Evaluation: Baseline reward models are trained on a separate Chatbot Arena-Personalized-derived corpus and evaluated across three Personalized RewardBench domains.The evaluation tuple contains the query, user context, chosen response, and rejected response.
- Arena Training Data Construction: Arena personas are derived by grouping interaction histories by user and using historical questions as evidence for user characterization.
- With- and Without-Persona Comparisons: Persona ablations hold architecture and optimization fixed while changing only whether persona embeddings are provided.This isolates the effect of personalized user representations.
- Finetuning Personalized Reward Models: Because benchmark instances are isolated single-query interactions, baselines are adapted to a zero-shot setting without persistent user identifiers or interaction logs.
- Baseline Adaptations: BT compares query-conditioned reward modeling with and without an additional persona embedding generated from user history.The comparison tests whether summarized persona information improves personalized reward modeling beyond the base setup.
- Baseline Adaptations: GPO and VPL incorporate persona information through conditioning or frozen LLM embeddings, while PAL replaces per-user parameters with a shared predictor for zero-shot generalization.
D Evaluation Metrics
The paper uses ranking metrics and controlled qualitative examples to assess reward-model alignment with downstream preferences and to isolate personalization from general response quality.
- D.1 Normalized Discounted Cumulative Gain (NDCG): NDCG evaluates predicted ranking quality by assigning relevance from ground-truth rank and normalizing predicted DCG by ideal DCG.The relevance score depends on an item’s zero-indexed position in the ground-truth list.
- D.2 Rank Biased Overlap (RBO): RBO measures top-weighted similarity between two potentially non-conjoint rankings using overlap at each depth and a persistence parameter set to p = 0.8.
- D.3 Weighted Kendall’s τ: Weighted Kendall’s τ gives greater importance to swaps near the top by assigning position-based weights before comparing weighted concordant and discordant pairs.
- D.4 Spearman’s ρ: Spearman’s ρ measures the monotonic relationship between two rankings as the Pearson correlation of their rank variables.The formulation uses covariance and the standard deviations of the rank variables.
- Personalization Case Study: A case study contrasts chosen and rejected answers to show that the preferred response better matches profile-specific rubric dimensions despite both offering high-quality academic advice.The analysis highlights communication with the supervisor, methodology comparison, and productivity impact.
- Benchmark Construction: The benchmark construction keeps both responses fluent, helpful, and generally high quality so that preference discrimination is isolated to adherence to the user’s profile and rubric.Prompt templates separately generate chosen and rejected answers, score aspects, and perform pairwise preference evaluation.