PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, Natasha Jaques

arXiv:2408.10075v1cs.LGcs.AIcs.CLcs.RO

TL;DR

Existing RLHF methods typically average over divergent user preferences, limiting alignment for diverse populations. This paper introduces Variational Preference Learning, which infers latent user context and conditions reward models and policies on it; across simulated control and language tasks, VPL captures diverse preferences and outperforms existing RLHF approaches by 10-25%.

  • Problem

    Current RLHF commonly assumes shared user values, leaving it unable to represent divergent preferences across diverse populations.

  • Method

    VPL uses variational inference to infer a latent distribution over hidden user context from preference annotations and condition reward models and policies on that latent.

  • Results

    10-25%: VPL outperforms existing RLHF approaches across language datasets while accurately modeling diverse preferences and improving simulated-control personalization.

  • Takeaways & Limitations

    VPL provides a framework for pluralistic alignment that supports personalized policies, uncertainty measurement, and active learning for diverse users.

  • Takeaways & Limitations

    Realistic large-scale preference datasets from diverse users are not yet available, so the study relies on synthetic preference datasets.

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning foundation models to human values and preferences. However, current RLHF techniques cannot account for the naturally occurring differences in individual human preferences across a diverse population. When these differences arise, traditional RLHF frameworks simply average over them, leading to inaccurate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop a class of multimodal RLHF methods. Our proposed techniques are based on a latent variable formulation - inferring a novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide a way to combat underspecification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in terms of measuring uncertainty, and actively learning user preferences. This work enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning to foundation model alignment.

1 Introduction

Current RLHF commonly assumes one shared utility function, which can average away divergent preferences and disadvantage minority groups. VPL instead models hidden user context to personalize rewards and policies, improving preference modeling and downstream performance across simulated and language tasks.

  • Motivation: Pluralistic alignment is needed because diverse social, moral, and political values shape user preferences across applications.The paper motivates adaptation to user-specific preferences in both language models and assistive robotics.
  • Motivation: Current RLHF often assumes all users share one utility function, failing when preferences diverge across groups.A unimodal BTL model can treat responses favored by distinct user groups as equally likely without additional context.
  • Approach: VPL formulates RLHF as a latent-variable problem, inferring hidden user context from a few annotations and conditioning reward models on the inferred latent.The method uses variational inference and derives an evidence lower bound for latent-variable preference-based reward optimization.
  • Practical considerations: Binary comparisons reveal reward differences but not absolute scale, creating optimization challenges when latent-conditioned users learn rewards at different scales.The paper identifies reward-scale variation as a practical issue for multi-user reinforcement learning.
  • Results: 10-25%: VPL outperforms existing RLHF approaches across language datasets while improving reward prediction for diverse users and objectives.The paper also reports accurate reward modeling, improved personalization, active learning with fewer test-time queries, and simulated-control gains.

2 Related Work

Prior work studies preference uncertainty, societal pluralism, multi-objective alignment, and personalization, but existing approaches generally lack a technical method for modeling divergent preferences directly from preference data. VPL is positioned as complementary to these lines of research by learning latent user types without requiring explicit demographic information.

  • Preference models: Work on non-BTL preference models addresses irrationality, uncertainty, or intransitivity, whereas VPL focuses on divergence among potentially rational users.The distinction is between inconsistent behavior and different utilities across labeling and end users.
  • Personalized RLHF: Personalized RLHF research has highlighted societal challenges and diverse annotations, but has not proposed a technical method to model diverse preferences.Social Choice Theory and related work are discussed as sources of insight rather than complete modeling solutions.
  • Objective trade-offs: Pareto optimization and multi-objective RL trade off conflicting alignment objectives, while some personalization methods require explicit sub-rewards and separate user reward models.These approaches frame personalization through predefined objectives rather than inferred latent user context.
  • User information: Other methods model disagreement using annotator demographics, fixed user representations, or community-specific data, unlike VPL's latent approach.The cited alternatives require additional user information or train on user- or community-specific datasets.
  • Closest related work: Distributional Preference Learning models hidden-context reward uncertainty through mean and variance, while VPL aims to recover individual reward functions and scale to personalization.The paper identifies DPL as the closest related work and distinguishes its representational scope.

3 Technical Preliminaries

Preference-based RLHF learns rewards from ordinal comparisons and then trains policies to maximize those inferred rewards. The standard BTL formulation models noisy comparisons but assumes a single shared reward function, so it cannot represent hidden context or personalize policies to users.

  • RLHF framework: RLHF has two phases: infer a reward function from ordinal preference labels, then train a policy to maximize the inferred rewards.The framework applies to preferences over states and can generalize to trajectories or snippets.
  • MDP formulation: The MDP is defined by states, actions, transition dynamics, discount factor, and an initial state distribution.The paper denotes these components as S, A, T, γ, and ρ0.
  • BTL reward learning: Under BTL, a reward model assigns preference likelihoods to annotated pairs and is typically learned by maximum likelihood.The recovered reward then supports policy optimization with standard reinforcement-learning algorithms.
  • Limitation: The standard BTL formulation assumes every annotator shares one underlying reward function, which fails to represent diverse annotator preferences.Although BTL captures some IID preference noise, it does not model hidden context or personalize rewards and policies.

4 VPL: Incorporating Latent Context into Preference-Based Learning

VPL models divergent user preferences by inferring a user-specific latent from multiple annotations and conditioning both rewards and policies on that latent. The method also addresses reward-scale inconsistency and supports uncertainty-aware active preference elicitation.

  • Latent preference modeling: VPL replaces the single-reward assumption with a latent-conditional reward model that represents multimodal user preferences.The encoder qψ maps same-annotator preference annotations into a latent distribution, while rϕ(s, z) conditions rewards on latent context.
  • Latent preference modeling: The variational objective combines contextual BTL preference likelihood with KL regularization of the inferred latent posterior against a prior.The objective samples an annotator and annotations, then optimizes preference fit and a D_KL(qψ∥p) regularizer.
  • Personalized policies: Personalized policies are trained with latent-conditioned rewards and deployed using a user latent inferred from labeled preference queries.At training, policies optimize rϕ(s, z) across prior samples; at test time, posterior inference supplies the user-specific latent.
  • Scaled rewards: BTL comparisons identify reward differences but not absolute scale, causing latent-dependent reward magnitudes that complicate multi-user optimization and preference-divergence detection.Different reward functions can yield identical pairwise likelihoods, even when their numerical scales differ substantially.
  • Scaled rewards: VPL addresses scale mismatch by using expected pairwise preference likelihoods as rewards, producing values normalized to [0, 1] across latent variables.For state sA, the scaled reward is the expected probability that sA is preferred to sampled comparison states.
  • Active preference learning: Active learning selects preference queries with maximum information gain about the latent distribution before inferring the latent and conditioning deployment.The variational encoder supports mutual-information-based query selection because some preference pairs are more informative than others.

5 Scaling VPL for Reward Learning in Large Language Models (LLMs)

For LLM reward learning, VPL compresses user preference information into a probabilistic latent embedding while retaining reward-model performance. The implementation uses frozen prompt-response embeddings, attention-based latent inference, LoRA fine-tuning, and context augmentation for larger multi-user datasets.

  • LLM architecture: VPL scales to LLMs by compressing user preference information into a concise probabilistic embedding z without sacrificing reward-model performance.The architecture uses an encoder qψ and a reward model r(s, z) to represent user-conditioned reward learning.
  • LLM architecture: A pretrained LLM jointly encodes prompt-response pairs, whose embeddings are precomputed and frozen for efficient training.The preferred state representation contains both prompt and response, and the resulting embedding eA is reused during training.
  • Latent encoder: The latent encoder processes multiple same-user preference queries with a shared pair encoder and self-attention over their representations.Each encoded pair produces hi, and self-attention aggregates the set into the latent representation z.
  • Reward learning: The reward model concatenates a new-state embedding with a posterior-sampled latent and predicts rewards through an MLP fine-tuned with LoRA.The LLM reward model is trained for at least 1 epoch to fit both encoder and reward-model components.
  • Data augmentation: For larger multi-user datasets, VPL augments each labeled example with multiple context samples from the same user to learn an effective encoder.The implementation uses M ∈ {4, 8} duplicate contexts sampled from a larger user-annotated subset.

6 Experimental Evaluation on Simulated Control Tasks

The simulated-control evaluation tests whether VPL can learn multi-modal reward functions and use inferred user latents to steer policies across diverse tasks. Compared with baselines that average reward modes, VPL reconstructs user-specific preferences, improves task success and personalization, and supports more efficient preference elicitation.

  • Evaluation setup: The experiments evaluate whether VPL learns multi-modal rewards, latent-conditioned policies, and active preference queries across three simulated control tasks.The evaluation trains preference-based reward models and then policies using offline reinforcement learning.
  • Evaluation setup: The tasks include navigating to preferred goals, placing a box at preferred locations, relocating a bowl, and sorting objects according to user-specific attributes.These settings represent underspecified preferences in navigation, manipulation, assistive robotics, and household organization.
  • Baseline behavior: BTL averages distinct underlying rewards, producing inaccurate reward models and policies that select wrong, random, or majority-preferred behaviors.Across tasks, this leads to poor alignment with individual users, including wrong navigation goals and incorrect object-placement choices.
  • Reward reconstruction: VPL accurately reconstructs diverse rewards and learns latent-conditioned policies that can reach either preferred navigation goal.Figure 3 contrasts the averaged BTL reward with VPL’s user-specific reconstruction and z-conditioned behavior.
  • Policy performance: VPL outperforms all baselines in task success according to users’ underlying rewards, with navigation performance comparable to a goal-conditioned oracle.It also infers test-time user latents for manipulation and improves preferred-location placement and attribute-following behavior.
  • Active preference learning: Active querying improves policy performance across query budgets and reaches the same performance with only half the queries.The information-gain objective selects more discriminative queries and produces a more informative posterior for identifying users.

7 LLM Experiments

The LLM experiments evaluate VPL on pluralistic preference datasets and find that it learns more accurate, personalized reward models while supporting latent-space user clustering and robustness analyses.

  • Experimental setup: The experiments compare VPL with vanilla BTL and DPL on two datasets designed to represent divergent language preferences.The study uses GPT2- and Llama2-7B-based models.
  • Datasets: The Pets dataset simulates users who agree on the best and worst animals but disagree on middle rankings.
  • Datasets: UltraFeedback-P constructs users with distinct helpfulness or honesty preferences from fine-grained UltraFeedback attributes.
  • Reward modeling: VPL learns more accurate reward models across the datasets by inferring user latents from a few annotated samples, whereas BTL and DPL cannot fully model divergent preferences.Because the datasets are imbalanced, baselines can sometimes perform better on individual subsets.
  • Latent representation: The encoder forms latent clusters corresponding to user types, enabling the decoder to personalize reward models according to different preferred objectives.This compressed representation separates users from only a few preference labels.
  • Robustness: With 50% of preference labels flipped, VPL’s performance becomes equivalent to the baseline BTL model because the context queries provide no user information.

8 Conclusion

The paper concludes that VPL supports pluralistic, personalized preference learning across simulated control and LLM-based RLHF while modeling uncertainty and preference divergence. Its evaluation is limited by synthetic data and fixed survey-question assumptions for new users.

  • Conclusion: VPL uses variational inference to capture diverse preferences and support steerable personalized model learning with uncertainty estimates.
  • Conclusion: The method is evaluated across simulated control problems and LLM-based RLHF, where the paper reports significant improvements over current RLHF techniques.
  • Limitations and future work: A key limitation is the lack of realistic large-scale preference datasets containing opinions from diverse users, requiring synthetic datasets.
  • Limitations and future work: UltraFeedback experiments assume that new users can provide preferences over samples from a fixed set of survey questions.Future work would obtain preferences naturally during conversation instead.
  • Potential safety benefits: VPL’s uncertainty estimates could potentially support safety interventions when uncertainty about conflicting user preferences cannot be reduced.

A Additional Experiments

The didactic experiment compares standard BTL, DPL, and VPL on preferences generated by mixtures of Gaussian reward functions. VPL recovers individual reward-function modes rather than averaging or only representing their uncertainty.

  • Didactic setup: The experiment simulates multiple annotators whose binary preferences arise from distinct Gaussian reward functions.
  • Comparison: Standard BTL averages across the different reward modes because it represents only a single reward function.
  • Comparison: DPL captures uncertainty from multimodal rewards but cannot accurately predict the true modes.
  • Comparison: VPL infers the hidden latent and recovers the individual distribution of reward functions.

A.2 Does scaling rewards help improve performance?

Reward scaling is important for latent-conditioned reward models: biased or unscaled rewards can impair optimization, while SPO + VPL improves success rates over baseline scaling approaches.

  • VPLno-norm, VPL + SPO, VPLbatchnorm, and VPLmax-norm provide alternative treatments for reward-scale variation across latent variables.VPLbatchnorm normalizes each latent’s rewards by mean rewards over sampled states, whereas VPLmax-norm uses the maximum offline-dataset reward.
  • SPO + VPL outperforms baseline approaches in success rate by estimating normalized multimodal rewards.Unscaled or biased reward estimates lead to sub-optimal performance; VPL with max normalization is comparable in Ravens-Manipulation because its dataset lacks sub-optimal trajectories.
  • Batch normalization generates highly biased reward estimates and is catastrophic for the method.
  • Scaling comparisons are evaluated on Maze-Navigation and across tasks and user bases.

A.4 How does context length affect VPL?

Context length affects how accurately VPL infers user preferences and aligns decisions: additional queries increase informative evidence, while probabilistic modeling reconstructs divergent rewards rather than averaging them.

  • Increasing query length improves VPL’s user-identification accuracy and task performance in Habitat-TidyBot.Some queries are uninformative because users agree on them; longer contexts increase the chance of observing useful preference differences and reduce posterior variance.
  • Longer context enables lower-variance posterior inference and improved alignment during decision-making.The paper also reports that higher context length makes LLM experiments more robust to noise in annotated queries.
  • VPL reconstructs individual multimodal reward functions, whereas BTL averages rewards over user-preferred goals.In Ravens-Manipulation, VPL matches ground-truth rewards for two preference modes, while DPL predicts high-variance rewards.
  • In Habitat-Rearrange, unimodal BTL converges to the majority preference, while VPL captures preferences across annotators.
  • On the single-modal UF-P-4 setting, BTL reaches 77.04% evaluation accuracy and VPL reaches 77.16%.The reported result indicates that introducing the variational framework does not reduce performance in this aligned-preference case.
  • Personalized deployment can risk sycophancy or alignment with adversarial users, motivating questions about when more universal values should apply.The probabilistic user and reward distributions may help identify rare behavior and responses associated with disagreement.

B.1 Task Details

The evaluation uses simulated control environments and synthetic or curated preference datasets spanning navigation, manipulation, rearrangement, object sorting, pets, and language-model attributes.

  • The methods are evaluated on three simulated control environments.
  • Maze-Navigation: Maze-Navigation uses a torque-controlled point mass whose diverse users prefer two or ten maze locations.Users label preferences between states based on shortest-path considerations, while the robot does not observe a goal directly.
  • Ravens-Manipulation: Ravens-Manipulation models a robot arm placing a box at one of two user-preferred locations using noisy offline motion-planning data.The observation includes object and end-effector positions plus grasp state; the oracle reward assigns values based on goal and gripper distances.
  • Habitat-Rearrange: Habitat-Rearrange asks a mobile manipulator to place a bowl at one of five locations ranked according to each user’s preferences.At test time, the agent greedily chooses the location with maximum inferred reward.
  • Habitat-TidyBot: Habitat-TidyBot requires sorting five kitchen objects according to user preferences over material or function.The robot observes or queries locations for some objects before rearranging a misplaced object.
  • Pets: The synthetic Pets dataset represents divergent preferences over birds, cats, dogs, and rabbits using GPT-4-generated descriptions and user-group rankings.Contexts contain one to four same-user comparisons, with a divergent split focused on controversial dog-versus-cat choices.
  • UltraFeedback-P: UltraFeedback-P uses fine-grained helpfulness, honesty, instruction-following, and truthfulness scores to construct multimodal preference data.

B.3 Implementation Details

The implementation uses latent priors, embedding choices, active-query selection, and offline policy-optimization procedures for VPL experiments. Training and reproducibility details include swept hyperparameters and shared IQL settings.

  • Latent prior: A learned Gaussian prior can improve performance and training stability in control experiments compared with a fixed standard-Gaussian initialization.The prior uses learnable mean and variance under the ELBO objective.
  • LLM embeddings: The last-token embedding of the prompt-plus-response input performs best for encoder inputs and reward prediction among tested representations.The alternatives include llm2vec and weighted pooling.
  • Active learning: Active query selection samples S batches of Q annotations, requiring O(S * Q) model passes with batch sizes from 4 to 256.The active-query computation is performed once after training to identify discriminative queries for a model.
  • Experimental settings: The experiments sweep reward-model hyperparameters, use shared IQL parameters across experiments, and specify separate hyperparameters for language-model experiments.These settings are documented in Tables 2–4.
  • Policy optimization: VPL policy optimization samples z from a prior, computes latent-conditioned rewards for offline trajectories, and trains π(a|s, z) using IQL.The preference-model variant computes rewards from comparisons against a randomly sampled comparison set.
  • Reproducibility: The paper states that its algorithms, datasets, models, and hyperparameters are described in sufficient detail to reproduce the main experimental results.Code and data are not immediately available but are planned for future release.
Loading 2408.10075v1…