Crowd-PrefRL: Preference-Based Reward Learning from Crowds

1 Introduction

• Most RLHF methods treat feedback as if it came from one user, despite differences in users’ backgrounds and expertise and the possibility of diverse preferences.Recognizing crowd disagreement could support minority-group detection, bias mitigation, fairness, or personalization to varied preferences and feedback noise.Source: §1 · p. 1
• Prior multi-user RLHF approaches do not use crowdsourcing methods that model annotator identities, while available diverse-preference datasets often lack annotator identifiers paired with multiply labeled data.The authors propose SML aggregation, which uses annotator identifiers and is designed to handle multiple viewpoints while supporting user-accuracy inference and clustering.Source: §1 · p. 2
• Crowd-PrefRL uses crowdsourcing techniques to aggregate feedback from users of unknown expertise and reliability into preference labels for reward learning, and can detect multiple viewpoints from binary preferences.The framework integrates online preference-based RL with unsupervised crowdsourcing.Source: §1 · p. 2
• Exploratory robot-locomotion experiments suggest Crowd-PrefRL agents outperform agents trained from majority-vote or individual-user preferences, especially when crowd user-error rates vary widely.The experiments also provide proof of concept for identifying minority viewpoints and clustering users by viewpoint.Source: §1 · p. 2

2 Related Work

• Prior RLHF often treats multi-user data as single-user feedback, and publicly available diverse-preference datasets often omit annotator identifiers (e.g. Ouyang et al. (2022); Bai et al. (2022); Nakano et al. (2022); Durmus et al. (2024)).The paper contrasts this with explicitly modeling feedback from multiple users with differing noise levels and potentially multiple reward functions.Source: §2 · pp. 2–3

Label Aggregation

• Majority vote cannot account for patterns across users, whereas SML uses crowd-label covariance to estimate worker reliability and identify minority groups (Parisi et al. (2014)).The paper notes that applying label aggregation to preference learning and RL remains understudied.Source: §2 · p. 3

RL from Multi-Human Feedback

• Several multi-human preference-learning methods use offline RL, while some infer reward distributions with modeling assumptions or higher data requirements; Crowd-PrefRL instead aggregates labels for on-policy RL and infers user clusters and accuracies.Poddar et al. (2024) is described as requiring 7,500 preference pairs per user.Source: §2 · p. 3

2 Related Work

• Pluralistic alignment methods may require specified clusters or beliefs and may constrain how user objectives are balanced, while Crowd-PrefRL enables cluster inference for downstream reward-balancing methods.The cited examples include maximizing minimum utility over individuals (Chakraborty et al. (2024); Ramesh et al. (2024)).Source: §2 · p. 3

3 Problem Formulation

• The formulation asks whether crowd preferences can yield better performance than any individual user and whether minority feedback groups can be detected and clustered by objective without ground truth.It frames these questions around users whose expertise and reliability are unknown.Source: §3 · p. 3
• Preference-based RL addresses tasks with difficult-to-construct rewards by learning a reward from human preferences between fixed-length segments of agent observations and actions.A user labels one segment as preferred over the other.Source: §3 · p. 4
• The crowd labels sequentially selected segment pairs, with the assumption that a majority share a noisy understanding of the desired behavior while minority groups may follow different reward functions.No further crowd knowledge, such as expertise or reliability, is assumed.Source: §3 · p. 4
• Using only users’ pairwise labels and no ground-truth rewards or preferences, the goal is to estimate true preference labels and investigate whether users with consistently different feedback form minority groups.Each user’s reliability and error rate are unknown to the algorithm.Source: §3 · p. 4

4.1 Simulating Diverse Crowds

• Simulated users vary in myopia, rationality, and accidental errors, with parameters chosen to produce error rates primarily between 15% and 40%.Crowds contain 7, 11, or 15 users; Figure 2 compares MAJ and SML prediction errors across 100 Walker-walk crowds at different levels of user-error variability.Source: §4.1 · p. 4, Figure 2 · p. 6

4.2 Estimating Preference Labels from a Crowd

• SML aggregates crowd labels using the leading eigenvector of users’ preference covariance, under an independent-error assumption; majority vote is the simpler baseline.The paper describes SML as approximating maximum-likelihood label estimates and as able to rank users by estimated error rates.Source: §4.2 · p. 5

4.3 Crowdsourced Preference Learning for RL

• Crowd-PrefRL trains a reward model on SML-aggregated preference labels, selects query pairs by reward-predictor disagreement, then optimizes the policy against the learned reward.The implementation uses PrefPPO; on-policy methods are used to reduce non-stationarity from simultaneous reward and policy learning.Source: §4.3 · pp. 5–6

5.1 Experiment Setup

• The experiments use PrefPPO across Walker-walk, Quadruped-walk, and Cheetah-run, comparing SML aggregation with majority-vote and oracle labels over 4M environment steps.Each comparison has 10 runs; reported means and standard errors use six runs after omitting the top and bottom two by episode return at 4M steps.Source: §5.1 · p. 6

5.2 Results

• SML labels outperform MAJ in nearly all tested crowd configurations, with the advantage increasing as user error rates become more variable.SML has lower label error than the best individual crowd member in nearly all cases; results for crowd sizes M ∈ {7, 11, 15} are shown for Walker-walk because other tested environments were similar.Source: §5.2 · pp. 6–7
• Crowd-PrefPPO-SML outperforms Crowd-PrefPPO-MAJ in trajectory return across all three tested environments, approaching Oracle performance on Walker-walk and Quadruped-walk.The comparison uses two crowd configurations: one where SML is expected to outperform MAJ and one where their label accuracy is expected to be similar.Source: §5.2 · p. 7, Figure 3 · p. 7
• In MO-Hopper, SML weights separate minority and majority users at 25K training steps, but not at 100K when the agent’s returns indicate it can satisfy both objectives.The simulated crowd has 11 users preferring forward velocity and four preferring vertical height; the two objectives’ returns change from weakly negative to highly positive correlation.Source: §5.2 · p. 8, Figure 4 · p. 8
• For a 150-user MO-Hopper crowd, a GMM fitted to SML weights identifies two groups at 25K steps but only one at 100K; inferred group labels have about 1% minority and 0% majority error.The crowd has a 110/40 majority/minority split, and the Bayesian Information Criterion selects the number of clusters.Source: §5.2 · p. 8, Figure 5 · p. 9
• The inferred user groups provide a basis for learning reward functions and corresponding behaviors specific to each cluster.The proposed downstream use is presented as a possibility based on the clustering results.Source: §5.2 · p. 9

• Crowd-PrefPPO learns rewards from crowd-aggregated labels that are more accurate than any single user’s feedback, without ground-truth user information.Source: §6 · p. 9
• SML labels outperform MAJ labels in nearly all cases across crowd sizes and tasks.Source: §6 · p. 9
• SML weighting is computationally fast and requires only a few lines of code.Source: §6 · p. 9
• On a multi-objective robotics task, Crowd-PrefPPO identifies minority groups and shows potential to learn minority reward functions.Source: §6 · p. 9
• Every crowd member must label each preference query, and the evaluation uses simulated rather than human feedback.Source: §6 · p. 10

• The algorithm initializes policy and reward-model parameters and begins with an exploration phase.Source: Full Crowd-PrefRL Algorithm · p. 13
• At feedback intervals, the algorithm samples preference pairs, queries all M users, and adds either SML-aggregated or majority-vote labels to the preference dataset.Source: Full Crowd-PrefRL Algorithm · p. 13
• During policy learning, environment transitions and predicted rewards populate buffer B, whose minibatches are used to optimize policy π.Source: Full Crowd-PrefRL Algorithm · p. 13

• Experiments use Walker-walk, Quadruped-walk, and Cheetah-run from DMControl; Walker-walk and Quadruped-walk were previously used in Lee et al. (2021a).Source: Experiment Setup Hyperparameters · p. 14
• Crowd-PrefPPO hyperparameters match Lee et al. (2021a), while MO-Hopper settings match those used for Hopper in stable-baselines3.Source: Experiment Setup Hyperparameters · p. 14
• Table 1 lists Crowd-PrefPPO hyperparameters for Walker-walk, Quadruped-walk, and Cheetah-run.Source: Table 1 · p. 14