PaperlayerView exact revision on arXiv

Source-linked AI summary

Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner

Kazusato Oko, Annie Ulichney, Nika Haghtalab, Han Bao

arXiv:2609.12651v1cs.LGcs.GT

TL;DR

Prior analysis raised concerns that RLHF distortion can grow exponentially with the Bradley–Terry temperature, despite RLHF’s use in practice. This paper analyzes distortion under reward clipping across KL-budget regimes and identifies mismatch between the preference-data distribution and KL reference policy as a governing factor. Its bounds show distortion of ˜Θ(Bβ + β) in a representative regime, with practical implications for preference-data collection and model fine-tuning.

  • Problem

    Prior work reported an exponential lower bound on RLHF distortion under heterogeneous preferences, raising the question of when distortion can be controlled and whether it is unavoidable.

    Source: §1 · pp. 1–2

  • Method

    The paper analyzes RLHF distortion with reward clipping across KL-budget regimes, isolating mismatch between the preference-data distribution µ and reference policy πref.

    Source: §7 · p. 9

  • Results

    In a representative regime, worst-case distortion scales as ˜Θ(Bβ + β), while matching µ and πref yields distortion Θ(β) in the discussion’s summary of regimes.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The results suggest using on-policy preference data or fine-tuning on samples from µ before RLHF to reduce mismatch.

    Source: §7 · p. 9

    Efficiently preconditioning off-policy preference data remains challenging, and when RLHF’s performance justifies the added optimization cost of heterogeneity-robust methods remains an open practical question.

    Source: §7 · p. 9

Abstract

from arXiv · show

While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent work by G\"olz et al. (2025) demonstrated that the \textit{distortion} -- defined as the multiplicative gap between the average user utility of the RLHF policy and the optimal average utility -- can scale exponentially with the Bradley-Terry temperature parameter $\beta$ when users have heterogeneous preferences. In this work, we present a fine-grained analysis of the distortion of RLHF with reward clipping and demonstrate that such exponential degradation is not a fundamental property of the algorithm but rather a consequence of distribution mismatch between the distribution generating preference data ($\mu$) and the KL reference policy ($\pi_{\mathrm{ref}}$). To this end, we establish tight upper and lower bounds on the distortion of RLHF across multiple regimes of the KL regularization strength. We show that in a representative regime, under the Bradley-Terry model, the distortion is $\tilde{\Theta}(\beta B + \beta)$, where $B$ is an upper bound on the log density ratio between $\mu$ and $\pi_{\mathrm{ref}}$. In particular, when there is no distribution mismatch (i.e., $\mu = \pi_{\mathrm{ref}}$), RLHF achieves the optimal distortion of $O(\beta)$ up to a constant. Our results suggest that, to reasonably maximize average utility with RLHF, it is preferable to use on-policy sampled preference data or to fine-tune before RLHF on data from a source close to $\mu$.

1. Introduction

The introduction frames distortion as a measure of average-utility loss under heterogeneous preferences and revisits whether RLHF’s previously reported exponential distortion is fundamental. The paper attributes distortion to distribution mismatch and presents bounds across KL-budget regimes.

1. Introduction

  • G¨olz et al. (2025) report an exponential lower bound for RLHF distortion, prompting questions about when distortion is controlled and whether remaining distortion is unavoidable.Distortion measures how far a mechanism’s average utility falls short of the highest achievable average utility.Source: §1 · pp. 1–2

1.1. Our Contributions

  • ˜Θ(Bβ + β) describes worst-case distortion, with B measuring mismatch between the preference-data distribution µ and the KL reference policy πref.The analysis also gives matching lower bounds up to logarithmic factors across KL-budget ranges.Source: §1.1 · p. 2, Figure 1 · p. 2
  • Without a KL constraint, RLHF distortion is O(β), improving on the O(β2) bound of G¨olz et al. (2025).The proof introduces effective utility, which retains utility components that provide informative signals to RLHF rewards.Source: §1.1 · p. 2
  • The RLHF-specific lower bound ˜Ω(Bβ) persists even when the KL budget is exponentially small, τ = e−Θ(B).The paper also reports two experiments: one on the practical impact of the Bradley–Terry nonlinear regime and one on mismatch-induced distortion.Source: §1.1 · p. 2

1.2. Related Work

  • Unlike deterministic preference orderings, stochastic Bradley–Terry comparisons allow the Borda winner to achieve finite distortion of O(β).The related-work discussion contrasts this with infinite distortion for the Borda rule under deterministic preferences.Source: §1.2 · pp. 2–3

2. Problem Setting

The problem setting models alternatives as candidate completions and seeks a distribution maximizing average user utility. RLHF estimates a single Bradley–Terry reward from pairwise comparisons, then optimizes a policy, optionally subject to a KL constraint.

2. Problem Setting

  • The objective is to choose a distribution over alternatives maximizing expected utility across users, with the KL-constrained and unconstrained cases called AI alignment and social choice, respectively.Each user has a utility vector over a finite set of alternatives, with entries between 0 and 1.Source: §2 · p. 3
  • Preference data are generated by sampling alternatives from µ and asking users with heterogeneous utilities to compare each pair under a Bradley–Terry model.The model’s temperature β controls preference sharpness.Source: §2 · p. 3
  • RLHF fits a single Bradley–Terry reward model to preference labels, which cannot exactly recover average utility when users’ utilities are stochastic.The analysis assumes enough samples for empirical loss to converge to population loss; when utilities are constant across users, the learned reward recovers βu up to an additive constant.Source: §2 · p. 3
  • Reward clipping truncates the learned reward to concentrate it on an informative range, and without clipping the matched-distribution case has an O(β2) distortion bound.The paper describes clipping as a technical device for obtaining the optimal rate.Source: §2 · p. 3
  • In the AI alignment setting, policy optimization maximizes learned reward subject to KL(π∥πref) ≤ τ, while social choice has no KL constraint and does not require reward clipping.The maximum log probability ratio between µ and πref is used to characterize distribution mismatch; its sensitivity to perturbations is noted as a drawback.Source: §2 · pp. 3–4

3. Warm-up: Social Choice Setting

The no-KL setting connects RLHF to Borda winners and bounds their distortion using effective utility and reward comparisons. The analysis also relates this result to AI leaderboards.

3. Warm-up: Social Choice Setting

  • Without a KL constraint, RLHF maximizes learned reward over alternatives equivalent to Borda winners, connecting the analysis to social choice.The section says this proof strategy also forms the basis for the AI alignment setting with a KL constraint.Source: §3 · p. 4
  • The improved bound implies that the top-ranked model on a leaderboard is at most a constant factor more distorted than the algorithm-independent lower bound.The section frames leaderboard models as alternatives and their fitted rankings as a social-choice setting.Source: §3 · p. 4

3.1.1. EFFECTIVE UTILITY

  • Effective utility retains utility components that provide informative signals to rewards, while treating degraded comparisons as zero and capping large utilities.When a high-utility alternative loses to most opponents under the data distribution, comparisons may provide little evidence of its utility; saturated win probabilities also make further utility increases difficult to distinguish.Source: §3.1.1 · pp. 4–5

3.1.2. REWARD SANDWICH

  • The reward-sandwich analysis bounds true utility using clipped learned rewards and relates maximizing those rewards to maximizing average utility.For the separate case where P[βu(x) > c] exceeds c^2, the analysis instead uses that μ itself achieves average utility Ω(β^-1).Source: §3.1.2 · p. 5
  • Distortion of the Borda policy is O(β), improving on the prior O(β^2) bound.The reward analysis introduces effective utility to isolate preference signals and supports extending the approach to the KL-constrained alignment setting.Source: §3 · p. 4, §3.1.2 · p. 5

4. AI Alignment Setting

In the KL-constrained setting, the paper bounds RLHF distortion in terms of preference-data and reference-policy mismatch. It also examines the proof mechanism, fine-tuning, and removing reward clipping.

4. AI Alignment Setting

  • With μ = πref, distortion is O(β); under mismatch, it can reach O(Bβ), tight up to logarithmic factors.The analysis attributes the two factors to Bradley–Terry nonlinearity and distribution mismatch, respectively.Source: §4 · pp. 5–6
  • Fine-tuning on samples from a source close to μ can improve the worst-case ratio relative to the maximum utility under the original base policy.The stated regime is Be^-B ≤ τ ≤ 1, where the ratio for the original RLHF policy is O(βB).Source: §4 · p. 6

4.1. Proof Overview of Theorem 4.1

  • The proof bounds maximal average utility using maximal average effective utility, accounting for signal degradation and underweighting of large utilities.The latter occurs when sigmoid saturation makes large utilities difficult to distinguish; the former occurs when an alternative loses to at least half of alternatives sampled from μ.Source: §4.1 · p. 6
  • For distribution mismatch, the proof interpolates between μ and πref to obtain a policy inside the KL ball, controlling loss with a factor tied to the density ratio.For small τ, it uses a linear total-variation bound on high-density-ratio regions because the standard O(√τ) bound is insufficient for the tight upper bound.Source: §4.1 · pp. 6–7

4.2. Removal of Reward Clipping

  • Reward clipping targets the informative reward region, where majority alternatives are preferred with probability around one-half, and avoids spending KL budget on uninformative regions.Outside that interval, reward estimation can incur substantial errors.Source: §4.2 · p. 7
  • Without reward clipping, the paper still derives a polynomial distortion bound when μ = πref.The no-clipping result is stated for the matched-distribution case.Source: §4.2 · p. 7

5. Lower Bounds

The lower bounds show that RLHF’s distortion increase from distribution mismatch is tight up to logarithmic factors, while an algorithm-independent Ω(β) distortion remains when the data and reference distributions match. The bounds cover different KL-budget regimes, including exponentially small budgets.

5. Lower Bounds

  • The RLHF-specific lower bounds make the upper bounds tight up to logarithmic factors across the covered KL-budget regimes.The paper presents RLHF-specific bounds for e^-Θ(B) ≤ τ ≤ B and an algorithm-independent bound for remaining cases.Source: §5 · p. 7, §5.1 · p. 7

5.1. Lower Bound for RLHF

  • A distortion of ~Ω(Bβ) from distribution mismatch persists even when the KL budget is exponentially small, τ = e^-Θ(B).The lower bound holds regardless of how reward-clipping thresholds rmin and rmax are chosen.Source: §5.1 · p. 7
  • Reward ordering can favor alternatives with equal estimated rewards but sharply different utilities, while the KL constraint limits mass on the highest-utility alternative.The construction yields an average-utility loss of roughly B^-1 × β^-1, producing the mismatch lower bound.Source: §5.1 · p. 8

5.2. Algorithm Independent Lower Bound

  • When µ = πref, an algorithm-independent Ω(β) distortion lower bound holds for every positive KL budget and Bradley–Terry temperature.The result applies to a collection of instances drawn uniformly at random and contrasts with a prior social-choice lower bound that extends to alignment only for sufficiently large budgets.Source: §5.2 · p. 8

6. Experiments

The experiments examine practical reward scales and illustrate distortion under different reference-policy and preference-data distributions. Large observed reward differences indicate substantial Bradley–Terry nonlinearity, while the synthetic comparison shows distortion growing with mismatch but converging to one when the distributions match.

6.1. Reward Scale in Practical Reward Models

  • The reward-scale experiment samples 5,000 preference instances from the models’ training datasets and plots absolute pairwise reward differences.It uses Skywork-Reward-V2-Llama-3.1-8B and UltraRM-13B.Source: §6.1 · pp. 8–9
  • Maximum pairwise reward differences are 108.8 for Skywork and 25.4 for UltraRM, indicating substantial nonlinearity in the practical reward models’ MLE loss.These differences do not exactly correspond to effective β because of optimization errors.Source: Figure 2 · p. 8, §6.1 · p. 9

6.2. Synthetic Experiment

  • Distortion converges to 1 when µ = πref, but grows during optimization when µ ≠ πref, alongside decreasing average utility.The synthetic setup uses β = 10 and B = 11.5 for the mismatched case; using πref for preference-data generation gives B = 1.Source: §6.2 · p. 9, Figure 3 · p. 9

7. Discussion and Practical Implications

The discussion concludes that distortion depends on the mismatch between preference-data and reference-policy distributions, rather than user heterogeneity alone. It recommends practical ways to reduce mismatch and identifies challenges and trade-offs that remain.

7. Discussion and Practical Implications

  • With matching data and reference distributions, RLHF has O(β) distortion; mismatch raises the bound to ~Θ(Bβ + β), with exponential degradation ruled out unless mismatch is extreme.The paper states that reward-estimation misspecification from using a single Bradley–Terry model does not by itself worsen distortion as a fundamental property of RLHF.Source: §7 · p. 9
  • On-policy preference data or fine-tuning on samples from µ before RLHF may reduce mismatch and improve average utility when enough such samples are available.The discussion also notes that off-policy preference data commonly arises from public datasets or earlier-generation models.Source: §7 · p. 9
  • Reducing distortion may require costly trade-offs: off-policy data preconditioning remains challenging at scale, and NLHF offers heterogeneity robustness at higher optimization cost.The discussion identifies mismatch between πref and µ as a key variable in deciding when that additional robustness is worthwhile.Source: §7 · p. 9
  • The effect of user heterogeneity depends on how base models are trained and preference data are collected, motivating evaluation of the full training pipeline.Source: §7 · p. 9

Impact Statement

The impact statement characterizes the work as a theoretical analysis of RLHF bias, while the appendix proves upper bounds using reward clipping and case analysis.

Impact Statement

  • The paper studies how distribution mismatch can bias RLHF rewards and the resulting policy, and says its theoretical focus and lack of concrete attacks limit misuse risk.It notes prior empirical demonstrations of reward-model manipulation, including bias from manipulating 1% of reward-training data.Source: Impact Statement · p. 10

A. Proof of Upper Bounds

  • The appendix proves Theorems 3.1 and 4.1 using reward clipping in the AI alignment setting.It denotes the unclipped rewards by r̄.Source: A. Proof of Upper Bounds · p. 13
  • The upper-bound proofs divide into cases based on the probability that βu(x) exceeds c, and, for Theorem 4.1, on r_min.The most technically involved cases are stated as Theorems A.11 and A.12, with proofs in Appendices A.1–A.4.Source: A. Proof of Upper Bounds · p. 13

A.1. Analysis of the Reduction Effect on u

The appendix analyzes how effective utility captures information available to RLHF, then relates that quantity to clipped rewards and true utility under specified conditions.

A.1. Analysis of the Reduction Effect on u

  • The reduction analysis bounds maximum true utility using maximum effective utility, whose purpose is to retain utility components informative to RLHF rewards.The bound’s factor depends on β and the mismatch term Bτ^-1.Source: §A.1 · pp. 13–14

A.2. Upper Bounding the Effective Utility by the Reward

  • The reward analysis lower-bounds the upper-clipped reward using expected effective utility, linking the learned reward to the reduced utility measure.The proof sets the clipping threshold R to r_max for Lemma 3.3.Source: §A.2 · p. 16

A.3. Upper Bounding the Reward by the True Utility

  • Under a tail assumption on large utilities, the reward analysis bounds r_max; a localized sigmoid linearization keeps its error to a constant factor in the final distortion bound.The stated condition for the r_max bound is P[βu(x) > c] ≤ c^2, with 0 < c ≤ 1/316.Source: §A.3 · pp. 17–18

A.4. Putting it all together

The section combines upper-bound arguments for the KL-constrained AI alignment setting and the social-choice setting. It also gives a linear distortion bound for social choice under its stated assumption.

A.4. Putting it all together

  • The social-choice setting has a linear distortion bound under the stated tail condition.Theorem A.11 gives a bound for Dist(πBorda).Source: §A.4 · p. 19
  • The social-choice result follows by viewing that setting as the limit τ → ∞ and applying the reduction and reward bounds.The proof combines the effective-utility argument, the clipped-reward bound, and the reward-to-utility bound.Source: §A.4 · pp. 19–20
  • Under the stated tail condition and Assumption 2.1, Theorem A.12 establishes a distortion upper bound for the AI alignment setting with clipped rewards.The proof combines bounds involving effective utility and the clipped reward.Source: §A.4 · p. 20

A.5. When the Sampling Distribution Has High Average Utility

This appendix part presents technical arguments related to distortion bounds for high-average-utility sampling, including support for a corollary, reward optimality, and technical lemmas.

A.5. When the Sampling Distribution Has High Average Utility

  • For AI alignment, Theorem A.16 treats the high-tail-probability case, while Theorem A.17 gives a bound when rmin > 0 under its stated low-tail condition.Theorem A.17 also assumes Assumption 2.1 and Pu∼D,x∼μ[βu(x) > c] ≤ c².Source: §A.5 · pp. 20, 24
  • Theorem A.15 gives a linear distortion bound for unclipped rewards when πref = μ and the sampling-utility tail condition holds, also supplying the social-choice upper bound.The social-choice result follows by taking the limit τ → ∞ because that setting has no KL constraint.Source: §A.5 · pp. 20, 22

A.6. Proof of Corollary 4.2

  • A mixture of the base-policy utility maximizer with μ is used to relate the optimal utilities under πbase and πref in the proof of Corollary 4.2.The mixture is shown to satisfy the πref KL constraint, allowing its utility to be compared with the optimum under πref.Source: §A.6 · p. 25

A.7.1. FIRST-ORDER OPTIMALITY

  • The first-order condition links estimated rewards to expected Borda scores, whose rankings coincide with reward rankings (Siththaranjan et al., 2023).The appendix formalizes the condition and provides its proof.Source: §A.7.1 · pp. 25–26

A.7.2. PROOF OF TECHNICAL LEMMAS

  • The technical-lemma proofs provide bounds for KL-feasible mixtures, tail-region total variation, and sigmoid differences used in the upper-bound analysis.The mixture lemma bounds KL for a mixture with a bounded density ratio; the sigmoid lemmas give inequalities under specified parameter conditions.Source: §A.7.2 · pp. 26–28
  • The proof uses the decreasing derivative of σ for nonnegative inputs and intermediate inequalities (64)–(66) to establish the left-hand inequality of (35).For s ≤ 4, it uses s/4 = 1/4 min{s, 4} in the argument.Source: §A.7.2 · p. 28
  • The proof derives the right-hand inequality of (35) by applying its left-hand inequality to σ(s − t) and using σ(t − s) = 1 − σ(s − t).Source: §A.7.2 · p. 28
  • For Lemma A.21, the proof introduces an auxiliary Bernoulli variable, constructs measures that differ only when Z = 0, and applies KL data processing to their pushforwards.The supplied excerpts do not state the lemma’s inequality.Source: §A.7.2 · pp. 28–29

B. Removal of Reward Clipping

This section analyzes distortion without reward clipping, establishing a polynomial bound when the reference policy matches the preference-data distribution and identifying why the proof does not extend directly to a general reference policy. It also proves a KL-constraint-dependent distortion lower bound that reward clipping cannot eliminate.

B.1. Non-expansiveness of Mixture of Bradley–Terry Models

  • The non-expansiveness lemma bounds the range of the fitted reward by β, enabling the unclipped analysis to control potentially unreliable rewards.Specifically, max_x r̄(x) − min_x r̄(x) ≤ β.Source: B. Removal of Reward Clipping · p. 30, §B.1 · p. 31

B. Removal of Reward Clipping

  • When πref = µ, removing clipping preserves the Theorem A.12 distortion bound, except that β is replaced by max{β, c^-1β^2}, with B = 0.Source: B. Removal of Reward Clipping · pp. 30–31
  • For a general reference policy, the proof in the low-tail case incurs an additional factor of β, and the paper gives a counterexample showing its pointwise reward bound is tight.The stated bound is r̄(x) ≲ β^2 E_{u∼D}[u(x)]; the authors note that avoiding the extra factor would require accounting for interactions among alternatives.Source: B. Removal of Reward Clipping · p. 31
  • When πref = µ, the reference policy already has optimal O(β) distortion in the low-tail case, but this property does not carry over directly to general reference policies.The condition is P_{u∼D,x∼µ}[βu(x) > c] < c^2.Source: B. Removal of Reward Clipping · p. 31

C.1. A Lower Bound Dependent on the KL Constraint

  • ˜Θ(min{Bβ, Bβ/τ}) lower-bounds distortion for e^(-Θ(B)) ≤ τ ≤ B.The bound is established in the lower-bound regime described in the passage.Source: §C.1 · p. 36
  • The lower-bound construction works regardless of how the reward-clipping bounds rmin and rmax are chosen.The construction also assumes KL budget τ > 0 and temperature β ≥ 3.Source: §C.1 · p. 36
  • Matching Borda-score orderings make the estimated rewards tie in pairs, so clipping preserves the ordering r(1) = r(2) ≤ r(3) = r(4).This reward structure underlies the construction's independence from the choice of clipping bounds.Source: §C.1 · p. 36

C.2. A Lower Bound Independent of the KL Constraint

This section establishes that RLHF can incur distortion of at least Ω(β) for any KL budget and any distribution mismatch, including B = 0. The lower bound persists because preference data can leave alternative permutations indistinguishable even when model parameters are identifiable.

C.2. A Lower Bound Independent of the KL Constraint

  • For any KL budget τ and mismatch B, an instance collection yields expected distortion at least Ω(β).The result includes the matched case B = 0.Source: §C.2 · p. 38
  • Even when model parameters are identifiable, a permutation of alternatives can remain indistinguishable and conceal average-utility differences of Θ(β).The construction uses this remaining permutation freedom to establish the lower bound.Source: §C.2 · pp. 38–39
  • The proof constructs a collection of instances by randomly permuting the first K + 1 alternatives.The construction fixes the final alternative while permuting the others.Source: §C.2 · p. 38

D.1. Details of Reward Scale Evaluation

This appendix describes the reward-scale evaluation, synthetic experiment, and log-likelihood-ratio analysis. It reports completion-level mismatch estimates while noting that semantic grouping could change how B is estimated.

D.1. Details of Reward Scale Evaluation

  • The reward-scale evaluation uses 5,000 training-dataset samples per model and compares selected completion reward differences.For UltraFeedback, the highest- and lowest-reward completions are selected from each prompt’s four completions.Source: §D.1 · p. 40

D.2. Details of the Synthetic Experiment

  • The synthetic experiment sets β = 10 and B = 11.5, computes rewards in the n → ∞ limit, and optimizes distributions with mirror-descent-style updates.The KL budget is tuned to τ = 0.143, and optimization starts from πref with step size η = 10^-3.Source: §D.2 · p. 41

D.3. Plot of the Log-likelihood Ratio

  • The completion-level maximum absolute log-likelihood difference is approximately 100, while the maximum token-normalized difference is 4.7.Figures 4 and 5 show the corresponding distributions.Source: §D.3 · p. 41, Figure 4 · p. 41, Figure 5 · p. 41
  • Treating each completion as an alternative may overstate mismatch because semantically equivalent completions can have large likelihood differences, and group-level B is difficult to estimate.The paper notes that no established methodology currently exists for grouping completions and estimating B at that level.Source: §D.3 · pp. 41–42
Loading 2609.12651v1…