PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework

Kihyun Kim, Jiawei Zhang, Asuman Ozdaglar, Pablo A. Parrilo

arXiv:2506.05619v3cs.AIcs.LG

TL;DR

Conventional preference learning can fail to reflect the evaluator population’s distribution, and existing approaches may rely on group identities that are not available. This paper develops an axiomatic framework that uses pairwise comparisons to align policies proportionally while bounding manipulation. The framework is designed to satisfy monotonicity, Pareto efficiency, population-proportional alignment, and population-bounded manipulability.

  • Problem

    RLHF and NLHF may not reflect the evaluator population’s distribution, while approaches using evaluator groups generally assume group identities or labels are known.

    Source: §1 · pp. 1–2

  • Method

    The framework uses pairwise comparisons to infer feasible population distributions and constructs policies under monotonicity, Pareto-efficiency, proportional-alignment, and bounded-manipulability axioms.

    Source: §1 · p. 2

  • Results

    The proposed framework is designed to satisfy monotonicity, Pareto efficiency, population-proportional alignment, and population-bounded manipulability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework shifts alignment beyond win rate toward policies proportional to inferred evaluator population distributions.

    Source: §6 · p. 10

    Random dictatorship cannot be implemented from pairwise data because distinct profiles can yield identical preference functions but different population distributions.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Conventional preference learning methods often prioritize opinions held more widely when aggregating preferences from multiple evaluators. This may result in policies that are biased in favor of some types of opinions or groups and susceptible to strategic manipulation. To address this issue, we develop a novel preference learning framework capable of aligning aggregate opinions and policies proportionally with the true population distribution of evaluator preferences. Grounded in social choice theory, our approach infers the feasible set of evaluator population distributions directly from pairwise comparison data. Using these estimates, the algorithm constructs a policy that satisfies foundational axioms from social choice theory, namely monotonicity and Pareto efficiency, as well as our newly-introduced axioms of population-proportional alignment and population-bounded manipulability. Moreover, we propose a soft-max relaxation method that smoothly trades off population-proportional alignment with the selection of the Condorcet winner (which beats all other options in pairwise comparisons). Finally, we validate the effectiveness and scalability of our approach through experiments on both tabular recommendation tasks and large language model alignment.

1 INTRODUCTION

RLHF and NLHF can fail to reflect the distribution of evaluator preferences, while approaches requiring known group identities may be impractical when those identities are implicit. The paper proposes an axiomatic framework that infers feasible population distributions from pairwise comparisons and constructs policies with proportionality and manipulability guarantees.

1 INTRODUCTION

  • RLHF and NLHF may not reflect the evaluator population’s distribution, while group-based approaches often assume explicit group labels that may be unavailable.The framework targets proportional alignment without requiring additional information about evaluator profiles.Source: §1 · pp. 1–2

1.1 OUR CONTRIBUTION

  • The proposed framework combines monotonicity and Pareto efficiency with population-proportional alignment (PPA) and population-bounded manipulability (PBM).PPA requires at least weak proportionality to evaluator population shares; PBM bounds manipulation incentives as an affine function of true population share.Source: §1 · p. 2
  • The framework infers feasible evaluator population distributions directly from pairwise comparisons and uses them to align policies with the underlying population.It also proposes a softmax relaxation to trade off proportional alignment against selection of the Condorcet winner.Source: §1.1 · p. 2
  • The paper presents a scalable function-approximation algorithm and evaluates it in tabular and high-dimensional settings, including LLM alignment.The paper is organized to introduce the setting and axioms, develop and analyze the algorithm, and report empirical evaluations.Source: §1.1 · p. 2

2 PRELIMINARIES

The paper connects preference learning to probabilistic social choice: profiles induce pairwise preference functions, and algorithms map those functions to policies. It then relates RLHF and NLHF to established social-choice rules.

2.1 PROBABILISTIC SOCIAL CHOICE FUNCTION AND PREFERENCE LEARNING

  • A profile is a distribution over strict, complete rankings, and a probabilistic social choice function maps each profile to a probability distribution over alternatives.Obtaining full profiles may be costly when alternatives are numerous.Source: §2.1 · p. 3
  • Pairwise preference learning uses the probability that one alternative is preferred to another, avoiding the need to collect complete rankings.The induced preference function obeys consistency conditions including skew-symmetry, and an algorithm maps it to a policy.Source: §2.1 · p. 3

2.2 TWO STANDARD PREFERENCE LEARNING ALGORITHMS

  • Without regularization, RLHF implements maximal Borda, while DPO adds KL regularization relative to a reference policy.The BT reward ranking coincides with Borda rankings.Source: §2.2 · p. 4
  • NLHF models preference learning as a two-player constant-sum game and implements maximal lotteries, whose equilibrium policy cannot be uniformly outperformed.This approach addresses intransitive preferences, including cases without a Condorcet winner.Source: §2.2 · p. 4

3 AXIOMATIC FRAMEWORK FOR POPULATION-PROPORTIONAL ALIGNMENT

The motivating example shows that RLHF and NLHF can select a single alternative despite a nearly even split of evaluator preferences. The paper introduces proportionality and manipulation axioms, then shows why standard social-choice rules cannot meet the desired combination using pairwise data alone.

3.1 MOTIVATING EXAMPLE WITH BINARY ALTERNATIVES

  • With two groups split at (1/2 + ϵ, 1/2 − ϵ), RLHF and NLHF both choose the slightly more popular alternative deterministically.A small change in ϵ can flip the policy from (0, 1) to (1, 0), motivating policies that reflect the group-share ratio.Source: §3.1 · p. 4

3.2 PROPOSED AXIOMS FOR POPULATION-PROPORTIONAL ALIGNMENT AND ROBUSTNESS

  • α-PPA requires each top choice’s policy probability, divided by its group’s population share, to be at least α(σ).The axiom concerns top-choice probabilities, not lower-ranked preferences; higher α means stronger alignment, with α = 1 indicating perfect proportional alignment.Source: §3.2 · p. 5
  • γ-PBM bounds a group’s maximum manipulation influence by an affine function of its population share, with deterministic selection of its preferred alternative possible only when it is the entire population.The axiom limits group over-representation rather than individual incentives to misreport; γ1 = 1 limits manipulated policy value to the group’s true share.Source: §3.2 · p. 5

3.3 LIMITATIONS OF STANDARD PSCFS: AXIOM VIOLATIONS AND NON-IMPLEMENTABILITY

  • Random dictatorship satisfies the proposed axioms in their strongest forms, but cannot be implemented from pairwise preference data.Distinct profiles can induce identical pairwise preference functions while having different population distributions.Source: §3.3 · pp. 5–6
  • Maximal Borda and maximal lotteries violate both proposed axioms, whereas random dictatorship satisfies all four axioms.The table summarizes standard PSCFs, their implementability, and their satisfaction of the four axioms.Source: §3.3 · pp. 5–6

4 ALGORITHMIC FRAMEWORK AND THEORETICAL GUARANTEES

The framework estimates feasible evaluator-population distributions from pairwise preferences, then selects a policy to align proportionally with that uncertainty. It establishes monotonicity, Pareto efficiency, population-proportional alignment, and population-bounded manipulability, with a softmax relaxation that trades off PPA and Condorcet consistency.

4.1 POPULATION DISTRIBUTION RECOVERY FROM PAIRWISE PREFERENCES

  • A tractable polyhedral outer approximation bounds feasible population distributions using constraints that grow linearly with the number of groups.The bounds use each alternative’s minimum pairwise preference share against any other alternative.Source: §4.1 · p. 6
  • Because pairwise comparisons do not identify the true population distribution, perfect proportional alignment is unattainable, and no algorithm can guarantee α(σ) > 2/M uniformly.The framework therefore seeks the largest achievable α-PPA guarantee.Source: §4.1 · p. 6

4.2 PROPOSED ALGORITHMIC FRAMEWORK WITH AXIOMATIC GUARANTEES

  • The proposed policy assigns probabilities in proportion to conservative upper bounds on population shares, satisfying monotonicity and Pareto efficiency.Its construction minimizes worst-case misalignment over the feasible population set.Source: §4.2 · pp. 6–7
  • The framework guarantees α-PPA, with alignment improving as the sum of the population-share upper bounds approaches 1.The guarantee reaches perfect PPA with two non-1-dominated alternatives, and also approaches 1 when a single group dominates.Source: §4.2 · p. 7
  • A single group’s manipulation is population-bounded; when its upper bound and true share are each at most 1/2, its preferred alternative cannot gain majority probability.The result is described as a weaker form of strategyproofness.Source: §4.2 · p. 7

4.3 BALANCING PPA AND CONDORCET CONSISTENCY

  • No method can satisfy both α-PPA and Condorcet consistency, so the softmax relaxation uses β to trade off the two properties.At β = 0 it reduces to the original algorithm; as β approaches infinity, it converges to a Condorcet-consistent minimax Condorcet method.Source: §4.3 · p. 8

5 EXPERIMENTS

Experiments test the framework on movie recommendations and instruction-tuned LLMs. The tabular results show a win-rate/PPA trade-off and lower manipulation gains for the proposed method at the reported setting; LLM results show a similar trade-off, less pronounced on Alpaca-GPT4 because group estimates are noisy.

5.1 TABULAR EXPERIMENT: MOVIE RECOMMENDATION

  • RLHF and NLHF achieved win rates of 0.7784 and 0.7712, respectively, with PPA 0; F β’s win rate rose from 0.5987 to 0.7784 as PPA fell from 0.4869 to 0.Source: §5.1 · p. 9
  • 8.896 × 10^-4 average PBM gain for F β at β = 100 compares with 0.0611 for RLHF and 0.0124 for NLHF.The authors report that F β outperforms the baselines when β ≤ 101.Source: §5.1 · p. 9

5.2 LARGE-SCALE EXPERIMENT: INSTRUCTION-TUNED LLMS

  • On the synthetic LLM dataset, β controls a clear win-rate/PPA trade-off, while that trade-off is less pronounced on Alpaca-GPT4 because annotation-model group estimates introduce noise.The synthetic dataset permits direct estimates from response logits; the authors suggest it can be used to evaluate PPA and tune β.Source: §5.2 · p. 9
  • The two-phase function-approximation approach costs about as much computation as RLHF and more than DPO, and accurate LLM PPA evaluation remains open.The authors characterize the findings as initial evidence of scalability.Source: §5.2 · p. 9

6 CONCLUSION AND FUTURE DIRECTIONS

The paper presents a preference-learning framework that aligns policies proportionally with population distributions inferred from pairwise comparisons. It connects this approach to social choice theory and identifies future extensions to lower-ranked preferences and more efficient high-dimensional algorithms.

6 CONCLUSION AND FUTURE DIRECTIONS

  • The framework aligns policies with inferred population distributions and extends preference learning beyond conventional win-rate emphasis.The conclusion describes the framework as a new class of probabilistic social choice functions beyond maximal Borda and maximal lotteries.Source: §6 · p. 10

B ADDITIONAL RELATED WORK

  • Pluralistic alignment methods often rely on explicit group identification, whereas this approach infers population distributions directly from pairwise comparisons.The paper also frames its approach as a structured solution to recovering reward-function distributions from aggregate comparisons, with proportional-alignment and robustness constraints.Source: B ADDITIONAL RELATED WORK · p. 16

C EQUIVALENCE OF BT-MLE REWARDS RANKING AND BORDA RANKING

D FUNDAMENTAL AXIOMS: MONOTONICITY AND PARETO EFFICIENCY

  • Monotonicity requires that improving an alternative’s ranking cannot lower its policy probability, while Pareto efficiency requires unanimous preference to receive at least as much probability.These definitions are stated for probabilistic social choice functions.Source: D FUNDAMENTAL AXIOMS: MONOTONICITY AND… · p. 17

E PROOF OF PROPOSITION 3.5

The proof examples show that maximal Borda and maximal lotteries can fail population-proportional alignment and population-bounded manipulability. The proposed ΦRD satisfies the four stated axioms, including those two new guarantees.

E PROOF OF PROPOSITION 3.5

  • For every ε > 0 in the described near-equal-group profile, maximal Borda and maximal lotteries select y1 deterministically, violating α-PPA.The resulting policy assigns zero probability to alternatives whose population shares are positive.Source: E PROOF OF PROPOSITION 3.5 · pp. 17–18
  • As ε approaches zero, ΦML concentrates on y3 despite its population share being 1/3, so no γ1 > 0 guarantees γ-PBM for ΦML.For ε = 1/12, the proof also gives a stochastic policy example.Source: E PROOF OF PROPOSITION 3.5 · p. 18
  • In the three-group example, a 0.45-share group’s misreport changes the maximal-Borda winner from y1 to y2, so no γ1 > 0 guarantees γ-PBM.The reported Borda scores change from [1.3, 1.20, 0.5] to [0.85, 1.2, 0.95].Source: E PROOF OF PROPOSITION 3.5 · p. 18
  • ΦRD satisfies monotonicity, Pareto efficiency, α-PPA with α(σ) = 1, and γ-PBM with (γ1, γ2) = (1, 0).The proof ties monotonicity to the number of evaluators whose top choice is the improved alternative.Source: E PROOF OF PROPOSITION 3.5 · p. 18

F PROOF OF THE NON-IMPLEMENTABILITY OF ΦRD

Two distinct evaluator profiles can induce the same aggregate pairwise preference function but require different ΦRD policies. Therefore, no preference-learning algorithm using only that function can implement ΦRD.

F PROOF OF THE NON-IMPLEMENTABILITY OF ΦRD

G PROOF OF PROPOSITION 4.2

These appendix arguments characterize feasible population distributions, an outer approximation, and limits on what pairwise data can guarantee. They also connect the bound to prior work and establish monotonicity for the proposed policy.

G PROOF OF PROPOSITION 4.2

  • A population-share vector is feasible for a preference function exactly when it can be represented by group-specific preference functions whose weighted mixture equals that function.The proof constructs a profile with the specified population shares and aggregate preferences.Source: G PROOF OF PROPOSITION 4.2 · p. 19

I ADDITIONAL REMARKS ON THE TIGHTNESS OF THE OUTER APPROXIMATION

  • The outer approximation may differ from the true feasible set under the strict-and-complete-ranking assumption, but is tight in the stated extended profile setting.That setting allows group-specific pairwise preferences subject to skew-symmetry and unanimity constraints.Source: I ADDITIONAL REMARKS ON THE TIGHTNESS… · p. 19

J CONNECTION OF THEOREM 4.4 AND TATLI ET AL. (2024)

  • In a sufficiently high-dimensional spatial setting, the cited result makes full-profile recovery from aggregate pairwise comparisons impossible and yields the bound wi ≤ ui.The paper derives the bound by summing an inequality over regions associated with voters’ top choices.Source: J CONNECTION OF THEOREM 4.4 AND TATLI… · p. 20

K IMPOSSIBILITY OF A UNIFORM GUARANTEE α(σ) > 2/M

  • No preference-learning algorithm can guarantee α(σ) > 2/M uniformly, because uninformative pairwise data can be consistent with a highly non-uniform top-choice distribution.Under fully uninformative comparisons, an algorithm maximizing the worst-case guarantee must output the uniform distribution.Source: K IMPOSSIBILITY OF A UNIFORM GUARANTEE… · p. 20

L PROOF OF THEOREM 4.6

  • Improving an alternative’s ranking cannot decrease its probability under the proposed policy, establishing monotonicity.The proof shows its ui cannot decrease while the other alternatives’ uj values cannot increase.Source: L PROOF OF THEOREM 4.6 · p. 20

N ADDITIONAL BOUND ANALYSIS UNDER A RANDOM-RANKING MODEL

This appendix examines lower-bound guarantees under dispersed rankings and analyzes strategic reporting. Its random-ranking results are stronger than the uniform-policy baseline, while the weak strategyproofness bound is not tight.

N ADDITIONAL BOUND ANALYSIS UNDER A RANDOM-RANKING MODEL

  • With M = 20, the largest population share averages approximately 0.25, indicating a dispersed ranking distribution.The setup uses 1,000 evaluators whose rankings add independent N(0, 1) noise to independently sampled N(0, 1) latent rewards.Source: N ADDITIONAL BOUND ANALYSIS UNDER A… · p. 21
  • Both lower bounds substantially exceed the uniform-policy baseline of 1/M under the tested random-ranking model.The authors describe the guarantees as nontrivial in empirically relevant regimes.Source: N ADDITIONAL BOUND ANALYSIS UNDER A… · p. 21

O PROOF OF THEOREM 4.11

  • Truthfully reporting yk as their top choice is the optimal strategy for group Gk in the proof of Theorem 4.11.The proof states that raising yk in the reported ranking increases or does not decrease u′k.Source: O PROOF OF THEOREM 4.11 · p. 21

P WEAK STRATEGYPROOFNESS GUARANTEE

  • In the all-groups equilibrium, each group truthfully reports its top choice, but the resulting bound is not tight.The authors leave tighter bounds and detailed analysis of group strategy for future research.Source: P WEAK STRATEGYPROOFNESS GUARANTEE · p. 22

Q PROOF OF PROPOSITION 4.13

With two alternatives, a deterministic Condorcet-winner policy conflicts with any positive α-PPA guarantee when the other alternative has a positive population share.

Q PROOF OF PROPOSITION 4.13

  • A deterministic policy selecting y1 gives y2 zero probability despite wσ2 > 0, violating α-PPA for every α(σ) > 0.The example assumes Pσ(y1 ≻ y2) ∈ (0.5, 1), making y1 the Condorcet winner.Source: Q PROOF OF PROPOSITION 4.13 · p. 22

R PROOF OF PROPOSITION 4.14

The limiting policy Φ∞ selects a Condorcet winner deterministically, while finite-β behavior trades off that consistency with population-proportional alignment. The appendix also describes the synthetic evaluation setup and shows that finite-β policies do not satisfy PMC.

R PROOF OF PROPOSITION 4.14

  • If yi is a Condorcet winner, Φ∞ assigns it probability 1 because it has the highest ui.For any other alternative j, uj < 0.5 while ui > 0.5.Source: R PROOF OF PROPOSITION 4.14 · p. 22

S FINITE BEHAVIOR OF Φβ

  • As β approaches infinity, the α-PPA guarantee deteriorates, despite the finite-β proposition quantifying a condition for a Condorcet winner to receive probability at least αc.The supplied proposition passage does not include the condition's threshold.Source: S FINITE BEHAVIOR OF Φβ · pp. 22–23

T CONNECTION TO PAIRWISE MAJORITY CONSISTENCY (PMC)

  • Every finite-β Φβ violates PMC, whereas the limiting Φ∞ satisfies it.For a profile with a PMC ranking, the limiting policy selects its top-ranked alternative as a Condorcet winner.Source: T CONNECTION TO PAIRWISE MAJORITY… · p. 23

U.1 SYNTHETIC DATASET

  • The synthetic dataset uses 10 prompts and 10 color responses, with 1,000 rankings generated from Gaussian rewards and 104 pairwise comparisons for training.The comparisons are sampled i.i.d. from Pσ.Source: §U.1 · p. 24

U.2 ALPACA-GPT4 DATASET

The appendix describes an Alpaca-GPT4 evaluator-group dataset and an offline, two-phase procedure for learning a softmax policy from pairwise comparisons. It also details shared-model and reference-policy techniques for LLM fine-tuning.

U.2 ALPACA-GPT4 DATASET

  • The Alpaca-GPT4 setup uses expertise or style evaluator groups, a true population distribution wσ = [0.8, 0.2], and GPT-4.1-generated group-specific responses.The dataset contains 52k prompts, with pairwise comparisons drawn i.i.d. from Pσ.Source: §U.2 · p. 24

U.3 HYPERPARAMETER SETTING

  • All algorithms fine-tune Qwen2.5-3B-Instruct on the same offline dataset for the same iterations; NLHF is excluded because it does not support offline learning.Each run uses one H100 GPU and takes approximately 0.5–1 hour per epoch with about 20–40GB of memory using LoRA.Source: §U.3 · p. 25

V.1 OFFLINE PAIRWISE COMPARISON DATASET

  • The offline dataset contains contextual pairwise comparisons sampled from ρ(x) and πd(y | x), with labels from the context-dependent preference function P.The learning goal is a policy π mapping contexts to distributions over alternatives.Source: §V.1 · p. 25

V.2 TWO-PHASE OFFLINE PREFERENCE LEARNING ALGORITHM

  • The two-phase algorithm first estimates u using a selector model µ, then trains a policy model to match the normalized softmax policy constructed from estimated û.The policy model minimizes a distance to that target policy; the experiments use KL divergence.Source: §V.2 · p. 26

V.3 ADDITIONAL TECHNIQUES FOR LLM FINE-TUNING

  • KL-divergence regularization constrains drift from the reference policy, and a shared model for µ and π uses distinct input formats to reduce computational cost.The authors report comparable performance with improved memory usage and training efficiency compared with separate models.Source: §V.3 · p. 26

W DERIVATION OF THE LOSS FUNCTION IN PHASE 1

  • The Phase 1 loss is derived by expressing each u(y | x) as a pointwise minimum, aggregating the resulting linear programs, and approximating expectations with offline data.The aggregation assumes ρ and πd have full support; the empirical loss restricts µ to a parameterized function class.Source: W DERIVATION OF THE LOSS FUNCTION IN… · p. 27

Related papers

Loading 2506.05619v3…