Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

1 INTRODUCTION

• When annotator income is hidden, RLHF can favor high-income students’ preferences and reduce Pell Grant information relied on by lower-income students.The example illustrates how a single learned ordering can obscure differences in who depends on particular recommendations.Source: §1 · p. 2
• Preference learning with hidden context implicitly aggregates preferences using Borda count, which can yield unintuitive results and incentives for annotators to misreport.With diverse annotators, the learned reward function acts as a social welfare functional.Source: §1 · p. 2
• Distributional preference learning (DPL) estimates a distribution of utility values for each alternative rather than a single score.The introduction also describes using explained variance to detect effects of missing features.Source: §1 · pp. 2–3
• On combined RLHF data labeled for helpfulness and harmlessness, DPL identifies hidden context and uncertainty when the objectives conflict.The dataset combines subsets annotated according to distinct objectives.Source: §1 · p. 3
• Risk-aversion over DPL’s learned utility distribution can dramatically reduce preference for jailbroken responses by penalizing responses where helpfulness and harmlessness diverge.The paper connects this class of jailbreaks to conflicts between the two objectives.Source: §1 · p. 3

2 SETTING AND RELATED WORK

• Standard preference learning estimates a utility for each alternative from pairwise comparisons, commonly fitting a Bradley-Terry-Luce model by maximum likelihood.The comparisons are modeled as noisy or inconsistent; with uniformly random pairs and infinite data, the estimator minimizes a loss, with regularization introduced when a minimum may not exist.Source: §2 · pp. 3–4

2.1 HIDDEN CONTEXT

• Hidden context arises when feedback depends on information absent from the learned utility model, including irrationality, partial observability, diverse annotators, and mixed objectives.The formal model uses utilities u(a, z) over observed alternatives and hidden context, then marginalizes comparisons over a distribution of z.Source: §2.1 · pp. 4–5

2.2 RELATED WORK

• Prior work has studied annotator disagreement, irrational behavior, and multiple-objective optimization separately, while this paper analyzes them within a hidden-context framework.The section also situates preference learning research across RLHF, theory, and social choice connections.Source: §2.2 · p. 5

3 THEORETICAL ANALYSIS

• BTL preference learning ranks alternatives by Borda count, so maximizing its learned utility—as in RLHF—selects a Borda-maximizing alternative.Borda count is the average probability that an alternative is preferred to others.Source: §3 · p. 6

3.1 COMPARISON TO EXPECTED UTILITY AND LEAST-SQUARES REGRESSION

• With identically distributed residual noise across alternatives and pairwise noise differences supported around zero, preference learning preserves the expected-utility ordering.Uniform and normal noise distributions satisfy these assumptions.Source: §3.1 · p. 6
• Even when every pairwise majority favors the expected-utility ordering, preference learning can rank alternatives differently, and comparison data cannot generally identify expected utility.The paper also gives a case where a high-income majority's feedback leads preference learning to favor withholding Pell Grant information, although giving it has higher expected utility.Source: §3.1 · p. 7
• Borda count applies a utility CDF before averaging over hidden context, whereas expected utility averages utility values directly.When utility distributions are roughly normal, the sigmoidal CDF underweights extreme utilities; that may help or hurt depending on the setting.Source: §3.1 · p. 7, Figure 2 · p. 7

3.2 CONNECTIONS TO SOCIAL CHOICE THEORY

• With annotator identity hidden, preference learning aggregates individual comparisons like a voting rule, and Borda count can give participants an incentive to misreport preferences.The paper relates its Borda rule to the voting-theory version, which differs by an affine transformation.Source: §3.2 · p. 8

• DPL estimates a distribution of possible utilities for each alternative, rather than a single utility value.Its models estimate the distribution of utility as hidden context varies.Source: §4 · p. 8
• DPL targets utility variation from hidden context, not uncertainty caused by limited data, and its estimates need not collapse to point values with infinite data.Source: §4 · p. 9
• DPL's r2 measure estimates how much utility variation observed alternatives explain; higher instance-level utility variance flags alternatives more affected by hidden context.The paper defines 1 − r2 as the proportion of variance caused by hidden context.Source: §4 · p. 9
• In synthetic experiments, normal preference learning tracked Borda count, while DPL identified high-variance alternatives where hidden context affected utility.The synthetic setting introduced hidden-context effects for alternatives a ≥ 0.8.Source: §4 · p. 9, Figure 4 · p. 9
• DPL with risk-averse optimization mitigates jailbreaks without hurting accuracy on non-harmful prompts.The experiments are reported in Table 1.Source: §4 · p. 10, Table 1 · p. 10
• Risk aversion penalizes alternatives whose learned utility distributions indicate possible effects from hidden context.The paper proposes it as a way to use information provided by DPL.Source: §4 · pp. 9–10

Understanding jailbreak vulnerability

• HH-RLHF supplies separate helpfulness and harmlessness preference labels; the study tests whether the hidden labeling objective relates to jailbreaks and whether DPL can detect or reduce susceptibility.The study evaluates jailbreak rate using safe-versus-jailbroken response pairs and helpfulness accuracy on non-harmful prompts.Source: §5 · pp. 10–11
• Combining helpfulness and harmlessness labels gives the standard utility model an approximately 25% jailbreak rate, while harmlessness-only training has under 4% but around 50% helpfulness accuracy.The paper attributes the combined model’s jailbreak vulnerability in part to helpfulness favoring jailbroken responses that refuse less.Source: §5 · p. 11

Preventing jailbreaks

• 13% versus 25%: the combined categorical DPL model’s 0.01-quantile has a 13% jailbreak rate, compared with 25% for the combined standard utility model, with very similar helpfulness accuracy.The lower-quantile result is reported as mitigating jailbreaks without harming accuracy on non-harmful prompts.Source: §5 · p. 11
• On the illustrated jailbreak prompt, DPL assigns the unsafe response both a higher mean and higher variance, so its lower quantile favors the safe response.The figure contrasts this with normal preference learning, which assigns higher utility to the jailbroken response.Source: Figure 5 · p. 11, §5 · p. 12

6 CONCLUSION

• DPL can help detect hidden context and enable risk-sensitive optimization over utility distributions; the conclusion recommends considering it as an alternative that explicitly accounts for hidden context.The paper calls for further theoretical and empirical analysis of DPL and hidden context.Source: §6 · p. 12

A.1 PROOF THAT L(ˆu; u) IS CONVEX

• The preference-learning loss is strictly convex, and adding the stated positive quadratic regularizer makes it strongly convex.The appendix proof argues that the loss is a sum of weighted functions and that the regularizer is strongly convex.Source: §A.1 · p. 17

A.2 PROOF THAT LEAST-SQUARES REGRESSION CONVERGES TO EXPECTED UTILITY

• In the least-squares proof, each alternative’s estimate appears in only one summand, allowing the objective to be treated pointwise.The provided proof then considers when that pointwise objective is minimized.Source: §A.2 · p. 17

A.3 PROOF OF THEOREM 3.1

• BTL preference learning’s learned utility ranks alternatives exactly as Borda count does: one is higher precisely when its Borda count is higher.The proof notes that the regularized objective is strongly convex when λ > 0, yielding a unique minimum that satisfies the first-order condition.Source: §A.3 · p. 18
• For the forward direction, the proof assumes a higher Borda count but a no-higher learned utility, then compares increasing functions derived from the first-order condition.That comparison yields a contradiction to the assumed utility ordering.Source: §A.3 · p. 18
• The proof also treats λ = 0 as the unregularized case, where any solution must still satisfy the first-order condition.The reverse direction proceeds by contradiction as well.Source: §A.3 · p. 18

A.4 PROOF OF THEOREM 3.2

• When residual utilities are IID across alternatives and their pairwise differences have support around zero, learned utility ranks alternatives exactly as expected utility does.The support condition is stated for every positive δ.Source: §A.4 · p. 18
• The proof reduces the result to showing that Borda-count ordering and expected-utility ordering coincide, using the earlier Borda characterization.The proof first considers alternatives ordered by expected utility.Source: §A.4 · p. 19
• The proof’s comparison yields a positive Borda-count difference when one alternative has higher expected utility, completing that direction of the argument.It uses identical residual distributions in comparing terms of the summation.Source: §A.4 · p. 19

A.5 PROOF OF PROPOSITION 3.3

• The proposition allows every pairwise majority to favor the higher-expected-utility alternative even when the learned utility is not equivalent to expected utility.The proof illustrates this mismatch with alternatives a, b, and c.Source: §A.5 · p. 19
• Expected utility ranks the example’s alternatives a > b > c.The proof gives expected utilities for b and c as 2.8 and 2, respectively.Source: §A.5 · p. 19
• BC(b) = 0.6 and BC(c) = 0.33, and the proof states BC(b) > BC(a).Theorem 3.1 then implies the learned utility ranks b above a, contrary to expected utility’s a-above-b ordering.Source: §A.5 · pp. 19–20

A.6 PROOF OF THEOREM 3.4

• With unlimited comparison samples for all alternative pairs, a deterministic preference-learning algorithm can still fail to recover expected utility.The theorem states that some utility function and hidden-context distribution make the learned utility non-equivalent to expected utility.Source: §A.6 · p. 20
• The proof constructs two utility functions whose expected-utility rankings of a and b are opposite.One has expected utility a < b; the other has a > b.Source: §A.6 · p. 20
• Both constructions produce identical distributions over comparison outcomes, so the learner receives indistinguishable data in the two cases.The proof concludes that the same learned utility must fail to identify expected utility in at least one case.Source: §A.6 · p. 20

A.7 PROOF OF PROPOSITION 3.5

• For fixed hidden context, the proposition considers the CDF of utility values across uniformly selected alternatives.The stated definition averages the probabilities of a value being strictly below and less than or equal to a threshold.Source: §A.7 · p. 20

B.1 PRELIMINARIES

• A preference profile collects agents’ preferences, and a social welfare functional maps such profiles to a social preference relation.The appendix views individual feedback as revealed preference in a sufficiently rich comparison dataset.Source: §B.1 · p. 21

B.2 BTL AND BORDA COUNT

• Any solution to preference learning, including an L2-regularized solution, is equivalent to Borda count.With uniformly sampled agents, the expectation over agent identity scales Borda count.Source: §B.2 · p. 21

B.3 PROPORTION-REPRESENTABLE SWFS

• A social welfare functional is proportion-representable when its pairwise social preferences can be determined directly from comparison proportions.The section connects this property to implementing preference learning through a classifier on comparisons.Source: §B.3 · p. 21
• A social welfare functional is proportion-representable if and only if it is comparison-anonymous.Comparison-anonymity means swapping some comparisons between two individuals while preserving rational preferences does not change the outcome.Source: §B.3 · p. 22
• Comparison-based classifiers impose a representational limit: some social welfare functionals require models that consider whole rankings.Source: §B.3 · p. 22
• Among scoring rules, comparison-anonymity holds exactly for affine scoring rules; Borda count is the only proportion-representable scoring rule up to monotone transformations.The section gives Borda count’s scoring function as α(k) = k.Source: §B.3 · pp. 22–23
• The Copeland and maximum rules are proportion-representable social welfare functionals.Source: §B.3 · p. 23

C EXPERIMENT DETAILS

• The appendix’s experiment-details section describes the details of the LLM preference-learning experiments.Source: C EXPERIMENT DETAILS · p. 23

C.1 PREFERENCE DATASET

• Helpfulness and harmlessness prompts had different distributions, allowing combined-data preference models to respond according to the identifiable prompt type.Helpfulness prompts were mostly harmless questions or assistance requests, while harmlessness prompts were selected to elicit harmful behavior.Source: §C.1 · p. 23
• Half the dataset was randomly relabeled with the opposite objective so annotation objective could not be inferred from the prompt.GPT-3.5 generated the relabeled annotations.Source: §C.1 · p. 23

C.2 MODEL TRAINING

• Preference models fine-tuned LLAMA-2-7B with LoRA, using one output for standard preference learning, two for mean-and-variance DPL, or ten for categorical DPL.Combined-data models trained for one epoch; models trained on only helpfulness or harmlessness trained for two epochs to keep gradient-step counts roughly equal.Source: §C.2 · p. 25
• Mean-and-variance DPL predicts a Gaussian score distribution, while categorical DPL assigns probabilities to ten evenly spaced score values between 0 and 1.The mean-and-variance model uses softplus to ensure a positive output standard deviation; the categorical model applies softmax to its outputs.Source: §C.2 · p. 25
• Categorical DPL training adds an entropy bonus to stabilize training, with κ = 0.1 in all categorical-DPL experiments.The passage identifies κ as the entropy-bonus weight.Source: §C.2 · p. 26

C.3 JAILBROKEN RESPONSES

• The jailbreak-response dataset contains 187 pairs where one model produced a “good bot” response and the other a “bad bot” response to the same prompt.The pairs were filtered from ChatGPT and Claude responses to jailbreak prompts.Source: §C.3 · p. 26