PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences

Keertana Chidambaram, Karthik Vinay Seetharaman, Vasilis Syrgkanis

arXiv:2405.15065v2cs.LG

TL;DR

The paper addresses the limits of alignment methods that assume uniform preferences and the identification limits of binary feedback. It develops methods for heterogeneous preferences and reports that ternary feedback identifies latent preferences under a simple linear reward model, with stronger empirical performance than binary feedback.

  • Problem

    RLHF and DPO commonly assume uniform preferences, while binary comparisons are insufficient to identify latent heterogeneous preferences under the paper's stated setting.

    Source: §1 · pp. 1–2

  • Method

    The paper combines EM-DPO, which discovers latent annotator types and trains an ensemble, with MMRA, which aggregates the ensemble using min–max regret.

    Source: §1 · pp. 1–2

  • Results

    Ternary preferences identify latent heterogeneous preferences under a simple linear reward model, and the paper reports stronger empirical performance than binary preferences.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MMRA is intended to produce a single fair policy when individual user types are unknown at inference time, without severely underserving a preference group.

    Source: §1 · p. 2

    The method assumes K discrete latent groups, and EM may be sensitive to initialization and converge to saddle points.

    Source: §6 · p. 9

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback (RLHF) has become central to aligning large language models with human values, typically by first learning a reward model from preference data which is then used to update the model with reinforcement learning. Recent alternatives such as Direct Preference Optimization (DPO) simplify this pipeline by directly optimizing on preferences. However, both approaches often assume uniform annotator preferences and rely on binary comparisons, overlooking two key limitations: the diversity of human evaluators and the limitations of pairwise feedback. In this work, we address both these issues. First, we connect preference learning in RLHF with the econometrics literature and show that binary comparisons are insufficient for identifying latent user preferences from finite user data and infinite users, while (even incomplete) rankings over three or more responses ensure identifiability. Second, we introduce methods to incorporate heterogeneous preferences into alignment algorithms. We develop an Expectation-Maximization adaptation of DPO that discovers latent annotator types and trains a mixture of LLMs accordingly. Then we propose an aggregation algorithm using a min-max regret fairness criterion to produce a single generative policy with equitable performance guarantees. Together, these contributions establish a theoretical and algorithmic framework for fairness and personalization for diverse users in generative model alignment.

1 Introduction

The paper addresses preference heterogeneity and the limits of binary feedback in alignment, proposing a pipeline for personalization and fair aggregation. It also establishes when heterogeneous preferences can be identified under a simple linear reward model.

1 Introduction

  • Single-reward RLHF and DPO risk aligning to majority preferences and neglecting heterogeneous, underrepresented groups.Multi-dimensional reward approaches can require costly ratings and objectives fixed before data collection, which may miss latent cultural, political, or geographical factors.Source: §1 · p. 1
  • EM-DPO uses expectation–maximization to uncover latent preference types while training an ensemble of LLMs tailored to those types.The paper presents this as a way to personalize LLMs for diverse user populations.Source: §1 · pp. 1–2
  • MMRA combines the EM-DPO ensemble using a min–max regret fairness criterion, supporting deployment when user types are unknown at inference time.The stated aim is to avoid severely underserving any preference group.Source: §1 · p. 2
  • Ternary preferences identify latent heterogeneous preferences under a simple linear reward model, whereas binary comparisons do not; empirical evidence also reports stronger performance for ternary preferences.The paper states that each user can report a preferred response among three options.Source: §1 · p. 2
  • The proposed pipeline is reward-free and combines heterogeneous-preference learning with aggregation, while prior approaches include multiple reward models and homogeneous-preference DPO methods.The paper also distinguishes its EM approach from prior hard-clustering methods and says EM provides better theoretical guarantees.Source: §1 · p. 2

2 Background

The background describes the preference-data and model inputs to RLHF, then contrasts its reward-model-and-reinforcement-learning stages with DPO's direct policy optimization. Figure 1 previews the paper's pipeline for learning an equitable policy.

2 Background

  • RLHF uses a supervised-fine-tuned language model and annotator preference data, where annotators choose a preferred response from a pair generated for a prompt.The preferred and non-preferred responses are denoted yw and yl.Source: §2 · pp. 2–3
  • Figure 1 lays out the proposed equitable-policy pipeline: collect heterogeneous annotator preferences, use EM-DPO to form an ensemble, then apply MMRA to obtain one fair policy.Source: Figure 1 · p. 3
  • The Bradley–Terry–Luce model links pairwise choice probabilities to differences in latent response rewards, and RLHF fits a reward model before fine-tuning the language model with reinforcement learning.The RLHF fine-tuning stage uses PPO to maximize expected reward while regularizing deviation from the supervised model.Source: §2 · p. 3
  • DPO bypasses the intermediate reward model and directly optimizes the policy from preference data.The paper presents this framework as the foundation for its extension to diverse annotators.Source: §2 · p. 3

3 DPO for Diverse Annotators

The paper addresses heterogeneous annotator preferences by modeling unobserved preference types and training type-specific policies. It then aggregates those policies using a worst-case regret criterion for settings where user types are unknown.

3.1 EM Algorithm

  • EM-DPO soft-clusters annotators from observed preference data and learns an optimal policy for each latent type.It assumes a finite set of discrete latent types but requires no observable annotator group labels.Source: §3 · pp. 3–4

3 DPO for Diverse Annotators

  • The method assumes prompts are independent of latent preference type, so prompts are equally likely across types.The paper formalizes this as uncorrelated contexts and latent preference types.Source: §3 · p. 4

3.1 EM Algorithm

  • The EM procedure alternates between computing annotator-type responsibilities and optimizing model parameters, using annotator weights in a weighted multi-item DPO objective.Policy optimization can be separate for each type when policies do not share parameters, or joint when parameters are shared.Source: §3.1 · p. 4

3.2 Fair Aggregation via Min-Max Regret

  • Min-Max Regret Aggregation seeks a single policy that minimizes the worst-case regret across preference subgroups.Regret is measured against each subgroup’s population-optimal policy.Source: §3.2 · p. 4
  • Directly optimizing the min-max objective is computationally and memory intensive, so the paper uses a lightweight approximation; affine combinations of ensemble models admit an efficient solution with provable guarantees.The direct approach would require continuous generation and access to all K policies to estimate subgroup regrets.Source: §3.2 · p. 5

4 Identifiability: The Value of Three Choices

Under a linear reward analysis, binary preferences from infinitely many users do not identify the population distribution of preferences. With at least three alternatives, sufficient feature variation, and other stated conditions, even incomplete ternary rankings can identify it.

4 Identifiability: The Value of Three Choices

  • The identifiability analysis uses a linear reward model with known features, while the paper says its algorithms can be applied to arbitrary reward models.The linear model represents prompt-response features weighted by user-specific preferences.Source: §4 · pp. 5–6

Case 1: Single Binary Preference per User, Infinite Users

  • Binary preferences do not identify the population distribution f, whereas ternary preferences can identify it under stated conditions, even when rankings are incomplete.Distinct preference distributions can yield the same aggregate binary probabilities; the ternary identification result is restated from Fox et al. (2012).Source: §4 · pp. 6–7

Case 2: Single Ternary Preference per User, Infinite Users

  • Ternary identification requires at least three alternatives, feature vectors supported on an open set containing zero, independence between β and features, and the stated moment conditions.The paper interprets feature variation as needed to disentangle preference types across dimensions such as helpfulness, harmfulness, and coherence.Source: §4 · pp. 6–7

Case 3: Many Diverse Binary Preferences for Each User

  • For one user, sufficiently diverse binary comparisons identify β when the matrix of feature differences is full rank, but the required comparisons scale with reward-model dimensionality.For high-dimensional feature maps, the paper expects this route to require a very rich set of responses per user.Source: §4 · p. 7

5 Experiments

The experiments test whether EM-DPO learns preference-aligned clusters, whether Min-Max Regret DPO yields fairer aggregate policies, and whether ternary feedback helps in an adversarial setting. Results report strong reward margins on GlobalOpinionQA and best reward margins and accuracy from ternary EM-DPO on MPI.

5 Experiments

  • The experiment questions cover EM-DPO cluster quality, fairness of Min-Max Regret DPO policies, and ternary-versus-binary preferences in the adversarial case.For MPI, both binary and ternary EM-DPO variants are compared.Source: §5 · p. 7, §5.1 · p. 8

5.1 Results

  • The evaluation uses GlobalOpinionQA and MPI, datasets with subgroups that have differing preferences and imbalanced sizes.GlobalOpinionQA covers four countries and uses 48,000 preference pairs; MPI uses synthetic personalities including an adversarial pair that is non-identifiable under binary preferences.Source: §5.1 · p. 7
  • The experiments assess cluster quality by matching each user group to its best ensemble policy and assess aggregation by worst-case regret across subpopulations.Benchmarks include Vanilla DPO, EM-DPO, Cluster DPO, True Label DPO, MMRA, and uniform sampling.Source: §5.1 · p. 8
  • On GlobalOpinionQA, EM-DPO achieves strong reward margins against baselines and comparable accuracy, outperforming large-data true-label models on reward margins.The authors suggest this indicates EM-DPO can uncover latent structure within annotated labels.Source: §5.1 · p. 8
  • MMRA yields zero positive regret for Britain, Indonesia, and Pakistan, and modest regret for Mexico, relative to uniform ensemble sampling and Vanilla DPO.Table 3 presents regret values for each latent subgroup across datasets.Source: §5.1 · p. 9, Table 3 · p. 9
  • Ternary preferences with EM-DPO achieve the best MPI performance across reward margins and accuracy; ternary Cluster-DPO also surpasses binary EM-DPO.The comparison highlights ternary feedback in the constructed adversarial setting.Source: §5.1 · p. 9

6 Limitations & Open Questions

The paper identifies discrete latent preference groups and EM optimization as limitations, while leaving broader reward models and EM stability as open questions. The related-work discussion situates these concerns alongside DPO generalizations and challenges in reward modeling.

6 Limitations & Open Questions

  • The method assumes K discrete annotator groups despite preferences likely lying on a continuous spectrum, and EM may be initialization-sensitive and converge to saddle points.The authors also identify extending identification theory beyond linear rewards and understanding EM convergence and stability for LLMs as open questions.Source: §6 · p. 9

A Additional Related Work

  • Related work generalizes DPO across multiple SFT models, objectives, token-level optimization, divergence terms, and robust optimization.The discussion also notes a hybrid approach combining DPO with RLHF.Source: A Additional Related Work · p. 13
  • Prior RLHF work identifies a feedback-richness versus efficiency tension, while applications of language-based feedback to LLMs remain underexplored.The passage reviews preference-based RL across tasks and its use in language-model alignment.Source: A Additional Related Work · p. 13
  • Reward modeling faces open questions about representing human preferences and irrationality, including a trade-off between efficiency and accuracy.The passage also describes work modeling particular forms of suboptimality, while general irrationalities remain open.Source: A Additional Related Work · p. 13

B.1 EM-DPO Derivation

The derivation parameterizes EM-DPO with policy, latent-type, and prompt-distribution parameters, then separates its M-step updates and computes latent-type posteriors in the E-step. The aggregation variants use learned policies to optimize min-max regret, including an affine-ensemble option.

B.1.1 M-Step

  • The M-step parameterizes policies, latent-type probabilities, and prompt distributions, then separates updates for type probabilities and policy parameters under stated independence assumptions.The type-probability update has a closed-form solution; the policy update drops the prompt-distribution term when its parameter has no joint constraints with the policy parameter.Source: §B.1.1 · pp. 14–15

B.1.2 E-Step

  • The E-step computes the posterior distribution of latent types given each annotator’s preferences, using Bayes’ rule and the M-step likelihood and prior.Under the stated assumption, the prompt-distribution term cancels from the posterior formula.Source: §B.1.2 · p. 16

B.1.3 Final Algorithm

  • The final EM-DPO algorithm combines the E- and M-steps using γ_i,k as the posterior type probabilities.The passage introduces this quantity before presenting the combined algorithm.Source: §B.1.3 · p. 16

C Min-Max Regret Aggregation Algorithm Variants

  • The original MMRA algorithm requires generating and scoring from the training policy and all ensemble policies at every step, consuming compute and memory.The affine-ensemble variant restricts search to mixtures of the already trained ensemble policies, avoiding retraining a new policy.Source: C Min-Max Regret Aggregation Algorithm… · pp. 16–17
  • The affine-ensemble aggregation formulates min-max regret as a finite zero-sum game and uses optimistic Hedge dynamics to find mixture weights.The resulting policy is described as an O(log(K) log(T) T^-1)-approximate solution.Source: C Min-Max Regret Aggregation Algorithm… · p. 17

D.1 Data Generation

The data-generation section constructs Global Opinion QA and MPI preference data, then describes Cluster-DPO as a k-means baseline. MPI uses synthetic personality types, including an adversarial pair that binary preferences cannot identify.

D.1 Data Generation

  • Global Opinion QA uses country-level polling responses from Britain, Indonesia, Mexico, and Pakistan, with question rephrasings and simulated users.The setup creates 1,500 training users and 400 test users from 32 sampled questions apiece.Source: §D.1 · p. 18
  • MPI contains 990 phrases scored on five OCEAN traits, and the setup assigns phrase rewards by taking the inner product of trait scores and synthetic personality vectors.P2 is the negative of P1, forming an adversarial pair non-identifiable under binary preferences; users contribute one preference pair and preferences are simulated with two or three items.Source: §D.1 · p. 18

D.2 Cluster DPO

  • Cluster-DPO replaces EM-DPO with k-means clustering of users’ average preferred-text embeddings, then trains a separate DPO policy for each cluster.The embeddings are generated with RoBERTa-Large.Source: §D.2 · p. 19

D.3.1 Hyperparameter Tuning

The tuning analysis selects k = 4 for both datasets, while practical group-count choices must also account for resource scaling.

D.3.1 Hyperparameter Tuning

  • Reward margins peak at k = 4 or increase only marginally above 4 across validation data from component groups for both datasets.Accuracy shows no trend across the tuning results.Source: §D.3.1 · p. 19
  • Memory and compute requirements scale linearly and at least linearly, respectively, with the number of latent groups.Compute scaling depends on convergence steps, which may increase as the number of groups grows.Source: §D.3.1 · p. 19
  • k = 4 emerges as optimal for this experiment on both datasets, regardless of the validation set’s composition.The passage notes that group composition is unobservable in practice.Source: Figure 2 · p. 19, §D.3.1 · p. 19

D.4 Hyperparameters

The section lists hyperparameters for EMDPO and lightweight MMRA and reports their experimental hardware and runtimes.

D.4 Hyperparameters

  • Experiments ran on 8 NVIDIA H100 GPUs with 80GB memory per GPU.Source: §D.4 · p. 20
  • One DPO run for EMDPO took about 15 minutes on average.Source: §D.4 · p. 20
  • MMRA’s first generation and scoring phase took roughly 1 hour, followed by 15 minutes per training-and-evaluation iteration.Source: §D.4 · p. 20

Related papers

Loading 2405.15065v2…