PaperlayerView exact revision on arXiv

Source-linked AI summary

One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment

Hongru Cai, Yongqi Li, Tiezheng Yu, Fengbin Zhu, Wenjie Wang, Fuli Feng, Wenjie Li

arXiv:2601.18731v2cs.CLcs.AI

TL;DR

Personalized reward modeling must work with scarce user feedback and adapt efficiently to unseen users. MRM learns adaptable weight initializations over shared base reward functions through meta-learning, with RPO emphasizing hard-to-learn users. Experiments report stronger few-shot personalization, robustness, and baseline performance, while limitations include static-preference assumptions and evaluation that excludes downstream generation quality.

  • Problem

    Personalized reward modeling must capture individual preferences despite scarce feedback from each user and the need to adapt to unseen users.

  • Method

    MRM represents each user reward as a weighted combination of base reward functions and meta-learns the weight initialization for rapid adaptation, with RPO emphasizing hard-to-learn users.

  • Results

    MRM consistently outperforms baselines while improving few-shot personalization and robustness across diverse user preferences.

  • Takeaways & Limitations

    Learning preference adaptation provides a robust and scalable path for aligning LLMs with diverse human values.

  • Takeaways & Limitations

    MRM assumes static user preferences, and the experiments do not extend to downstream policy optimization or text generation.

Abstract

from arXiv · show

Alignment of Large Language Models (LLMs) aims to align outputs with human preferences, and personalized alignment further adapts models to individual users. This relies on personalized reward models that capture user-specific preferences and automatically provide individualized feedback. However, developing these models faces two critical challenges: the scarcity of feedback from individual users and the need for efficient adaptation to unseen users. We argue that addressing these constraints requires a paradigm shift from fitting data to learn user preferences to learn the process of preference adaptation. To realize this, we propose Meta Reward Modeling (MRM), which reformulates personalized reward modeling as a meta-learning problem. Specifically, we represent each user's reward model as a weighted combination of base reward functions, and optimize the initialization of these weights using a Model-Agnostic Meta-Learning (MAML)-style framework to support fast adaptation under limited feedback. To ensure robustness, we introduce the Robust Personalization Objective (RPO), which places greater emphasis on hard-to-learn users during meta optimization. Extensive experiments on personalized preference datasets validate that MRM enhances few-shot personalization, improves user robustness, and consistently outperforms baselines. We release code at https://github.com/ModalityDance/MRM.

1 Introduction

Personalized alignment addresses diverse user preferences, but personalized reward modeling is constrained by sparse individual feedback and rapid adaptation to unseen users. MRM reframes the problem as meta-learning preference adaptation, using adaptable initializations and robustness weighting.

  • Personalized alignment adapts LLM behavior to individual preferences because a single universal preference standard cannot represent diverse human values.
  • Sparse feedback from each user makes unique intent difficult to model, while conflicting data from other users cannot be simply aggregated.
  • Unseen users require personalization from only a few demonstrations because collecting feedback from every potential user is impractical.
  • Personalized input methods depend on limited context, whereas personalized parameter methods struggle to train user-specific parameters for unseen users.
  • MRM treats each user as a meta-learning task and learns an adaptable initialization rather than fitting a separate static preference model.
  • RPO emphasizes hard-to-learn users during meta-optimization, while experiments report improved few-shot adaptation and robustness over baselines.

2 Related Work

Prior work includes general reward modeling, personalized LLMs, personalized reward models, and meta-learning personalization. Existing personalized reward approaches mainly condition shared models on user inputs or train user-specific parameters, while MRM uses optimization-based adaptation.

  • Reward Models: Reward models translate human preferences into optimization signals and differ by model type and supervision granularity.
  • Personalized Reward Models: Personalized reward models target individual preference differences rather than assuming one universal standard.
  • Personalized Reward Models: Personalized input approaches use user histories, personas, prior preferences, or embeddings but can produce coarse-grained personalization under sparse feedback.
  • Personalized Reward Models: Personalized parameter approaches offer flexible alignment but can overfit scarce feedback and cannot efficiently train user-specific parameters in few-shot settings.
  • Personalized LLMs: Personalized LLM research studies individualized generation and applications such as dialogue, healthcare, education, and robotics, whereas this work focuses on reward modeling.
  • Meta-Learning for Personalization: Meta-learning methods rapidly adapt to new tasks, and optimization-based methods learn effective initializations; MRM applies this paradigm to personalized reward modeling.

3 Preliminaries

The preliminaries describe MAML’s support-query adaptation procedure and pairwise reward-model training. These foundations motivate learning a shared initialization that can personalize efficiently without training a separate model for each user.

  • Model-Agnostic Meta-Learning: MAML learns shared initial parameters that adapt to new tasks using few examples from task-specific support sets.
  • Model-Agnostic Meta-Learning: The inner loop updates task parameters on support data, while the outer loop evaluates adapted parameters on query data and updates the shared initialization.
  • Model-Agnostic Meta-Learning: This two-level optimization supports rapid personalization and few-shot learning when each user provides limited feedback.
  • Reward Model Training: Reward models learn from pairwise comparisons by assigning scores to responses and training higher scores for preferred responses than rejected responses.
  • Model-Agnostic Meta-Learning: Algorithm 1 iterates over sampled tasks, performs inner-loop gradient updates, and applies an outer-loop update to the shared initialization.
  • Reward Model Training: Training a separate reward model per user can overfit sparse comparisons, while one shared model cannot capture individual preferences.

4 Method

MRM reformulates personalized reward modeling as meta-learning: shared base reward functions and a shared weight initialization support rapid user-specific adaptation. Its robust personalization objective emphasizes hard-to-learn users while updating the initialization and base functions.

  • Meta Weight Initialization: MRM represents each user reward as a weighted combination of shared base reward functions and learns a shared initialization for the user-specific weights.The user weights determine each base function’s contribution to the personalized reward.
  • Task Definition: Each user is treated as a meta-learning task with disjoint support and query sets for adaptation and outer-loop evaluation.The query set helps update the shared initialization while reducing overfitting to limited user feedback.
  • Few-Shot Personalization: The inner loop adapts only user-specific weights from the shared initialization using limited support feedback, producing personalized reward functions.At inference, one or a few gradient steps adapt weights that can rank responses by the user’s preferences.
  • Outer-Loop Meta Optimization: The outer loop uses query losses to jointly update the shared weight initialization and shared base reward functions.The resulting initialization is intended to provide a better starting point for future adaptation.
  • Robust Personalization Objective: RPO emphasizes hard-to-learn users through filtering or soft reweighting, while smoothing prevents easy users from being completely discarded.Soft reweighting preserves attention to difficult users, yielding smoother gradients and more stable training.

5 Experiments

The experiments compare MRM with existing personalized reward modeling methods across performance, robustness, component effects, unseen-user adaptation, and efficiency or scalability.

  • Research Questions: The experiments address five questions covering performance, user robustness, component effects, few-shot adaptation to unseen users, and efficiency or scalability.These questions define the evaluation scope for MRM against existing methods.

5.1 Experimental Settings

Experiments use user-level preference datasets, seen and unseen user splits, limited-feedback settings, baseline comparisons, user-level accuracy, and repeated random splits.

  • Datasets: The evaluation uses PRISM and Reddit TLDR, two datasets containing user-level preference annotations.PRISM includes multi-turn conversations, candidate responses, and user judgments.
  • Data Splits: Users are divided into seen and unseen groups, with unseen users reserved for few-shot adaptation and all users retaining separate test data.The split simulates personalization for users absent from training.
  • Data Splits: Reddit TLDR training is restricted to 100 or 150 pairs for seen users and 50 pairs for unseen users, while full test sets are retained.These settings reflect limited per-user feedback.
  • Baselines: MRM is compared with representative personalized input and personalized parameter methods, alongside a non-personalized Bradley–Terry baseline.Most methods use embeddings extracted from a pretrained reward model.
  • Evaluation: Performance is measured by user-level accuracy on test response pairs, averaged over users and reported across 20 random splits with mean and standard deviation.The implementation uses two base reward functions and one inner-loop adaptation step per user.

5.2 Overall Performance (RQ1)

MRM consistently achieves the strongest performance across the evaluated datasets and settings, while existing personalized methods show limited or feedback-dependent gains over BT.

  • Overall Results: Across methods, the seen–unseen performance gap is small, while most personalized methods provide limited improvement over BT in data-scarce settings.The results suggest that existing approaches extract weak user-specific signals under limited feedback.
  • Overall Results: MRM consistently outperforms the BT baseline, indicating stronger personalization from limited data.The reported comparison attributes this pattern to MRM’s meta-learning objective.
  • Dataset Patterns: On PRISM, personalized input methods slightly outperform personalized parameter methods, but neither group surpasses the non-personalized BT baseline.The passage links parameter-based brittleness to few-shot training and poor generalization to unseen users.
  • Dataset Patterns: On Reddit TLDR, personalized input methods achieve clear gains over BT with richer per-user feedback, while LoRe becomes more competitive among parameter methods.The result is consistent with feedback availability affecting personalization methods.
  • Overall Results: Around 1.5% relative improvement over the best-performing baselines is reported for MRM across datasets and settings.The paper attributes the advantage to meta weight initialization for few-shot generalization and RPO’s emphasis on hard-to-learn users.

5.3 In-depth Analysis

The in-depth analysis finds that MRM improves robustness for difficult users, while RPO performance depends on balanced hyperparameters. Ablations and adaptation experiments further examine the roles of MRM’s components and its behavior on unseen users.

  • User Robustness (RQ2): MRM consistently outperforms baselines on the worst 10%, 20%, and 50% of users across PRISM and Reddit TLDR.The evaluation uses 100 examples per user and measures average accuracy on the hardest user subsets.
  • Analysis of RPO (RQ2): A threshold ratio of 0.5 provides the best trade-off between emphasizing hard users and maintaining overall stability.Ratios 0.2 and 0.5 perform comparably on worst users, while all settings converge at the worst 50%.
  • Analysis of RPO (RQ2): Moderate meta batch sizes improve worst-user accuracy, whereas larger batches reduce stability and overall performance.Worst-10% accuracy improves from batch sizes 2 to 8 but declines beyond 8.
  • Analysis of RPO (RQ2): Smoothing parameter γ=0.5 achieves the best balance, outperforming near-zero smoothing and hard filtering.Very small γ destabilizes training, while large γ oversmooths the reweighting.
  • Ablation Study (RQ3): Removing meta-learning causes the largest performance drop, while removing RPO or basis combination also reduces performance.The ablations indicate that meta-learned initialization, robust weighting, and multiple basis reward functions each contribute to personalization.
  • Few-shot Adaptation and Efficiency: MRM improves as unseen users provide more examples and adapts using shared reward functions and a shared lightweight weight initialization.At inference, each user updates and stores only a small set of user-specific weights.

6 Discussion

The discussion identifies methodological and practical limitations of MRM, including static-preference assumptions, computational overhead, incomplete downstream evaluation, and ethical risks in personalization.

  • Methodological limitations: MRM assumes user preferences remain static, although real-world preferences may evolve with context and long-term changes in intent.Handling evolving preferences would require continual or online adaptation and time-varying feedback datasets.
  • Methodological limitations: MRM’s bi-level meta-learning framework costs more computationally than standard single-level approaches, mainly during meta-training.Shared base reward functions and lightweight user-specific weights reduce parameter and adaptation costs at inference.
  • Evaluation scope: Experiments measure preference prediction accuracy rather than downstream policy optimization, text generation, human evaluation, or task-based usefulness.Therefore, the metric may not fully capture generated-response quality or user satisfaction in real-world applications.
  • Ethical and privacy considerations: Personalized reward modeling raises privacy risks because user feedback may reveal personal values and habits through memorization or model responses.The paper recommends consent, anonymization, minimizing stored user-specific information, and limiting retention.
  • Ethical and privacy considerations: Optimizing individual preferences can reinforce biases, narrow exposure to alternatives, and amplify preference extremes through overfitting.Suggested mitigations include cross-user regularization and constraints encouraging preference diversity.
  • Ethical and privacy considerations: Personalized alignment may conflict with shared safety boundaries, which MRM does not explicitly enforce.Potential remedies include global safety constraints, regularization, or a shared safety model that filters unsafe outputs.
  • Ethical and privacy considerations: Limited and narrow feedback can make personalized reward models vulnerable to reward hacking and mis-specification.Models may exploit imperfections in the reward function or learn shortcuts that diverge from users’ true preferences.

7 Conclusion and Future Work

The conclusion presents MRM as a meta-learning solution to sparse-feedback personalization and adaptation to unseen users. It reports robust few-shot personalization and outlines extensions toward dynamic, active, richer, and more expressive alignment.

  • Conclusion: Existing personalized reward-modeling paradigms do not simultaneously provide fine-grained personalization under limited feedback and efficient adaptation to unseen users.The paper motivates shifting from fitting static user models toward learning the adaptation process.
  • Conclusion: MRM optimizes an adaptable initialization, using support sets for inner-loop personalization and query sets for outer-loop optimization.The framework also uses RPO to emphasize hard-to-learn users during meta-optimization.
  • Conclusion: MRM’s experiments demonstrate a robust and scalable path for aligning LLMs with diverse human values through few-shot personalization.This conclusion follows the reported framework-level experimental finding rather than a single benchmark metric.
  • Future work: Future work includes direct policy optimization, continual adaptation, active user queries, implicit behavioral signals, and more expressive meta-learned parameterizations.These directions must address scalability and training stability where applicable.
Loading 2601.18731v2…