Source-linked AI summary
Fine-tuning language models to find agreement among humans with diverse preferences
Michiel A. Bakker, Martin J. Chadwick, Hannah R. Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew M. Botvinick, Christopher Summerfield
TL;DR
Existing preference-tuning methods treat human preferences as homogeneous and static, despite social problems involving diverse values. This paper fine-tunes an LLM to generate and reward-rank group consensus statements, finding that many reduce divisiveness and sometimes achieve unanimous support, including on unseen topics.
Problem
Existing preference-tuning methods treat human preferences as homogeneous and static, although social coordination and group decision-making involve diverse values.
Method
The authors fine-tune an LLM to generate consensus candidates from group opinions, then train a reward model to score individual agreement and rerank candidates by expected social welfare.
Results
65.6% of SFT-Utilitarian candidates were less divisive than initial position statements, and 40.8% of divisive rounds produced a candidate with unanimous support.
Takeaways & Limitations
The approach opens a language-modeling direction for accommodating diverse preferences and helping groups produce agreement in natural language.
Takeaways & Limitations
UK-only participants limit cohort diversity and the generalisability of the findings, while risking demographic bias in the resulting consensus statements.
Abstract
from arXiv · showhide
Recent work in large language modeling (LLMs) has used fine-tuning to align outputs with the preferences of a prototypical user. This work assumes that human preferences are static and homogeneous across individuals, so that aligning to a a single "generic" user will confer more general alignment. Here, we embrace the heterogeneity of human preferences to consider a different challenge: how might a machine help people with diverse views find agreement? We fine-tune a 70 billion parameter LLM to generate statements that maximize the expected approval for a group of people with potentially diverse opinions. Human participants provide written opinions on thousands of questions touching on moral and political issues (e.g., "should we raise taxes on the rich?"), and rate the LLM's generated candidate consensus statements for agreement and quality. A reward model is then trained to predict individual preferences, enabling it to quantify and rank consensus statements in terms of their appeal to the overall group, defined according to different aggregation (social welfare) functions. The model produces consensus statements that are preferred by human users over those from prompted LLMs (>70%) and significantly outperforms a tight fine-tuned baseline that lacks the final ranking step. Further, our best model's consensus statements are preferred over the best human-generated opinions (>65%). We find that when we silently constructed consensus statements from only a subset of group members, those who were excluded were more likely to dissent, revealing the sensitivity of the consensus to individual contributions. These results highlight the potential to use LLMs to help groups of humans align their values with one another.
1 Introduction
Existing preference-alignment methods largely assume stable, homogeneous human values, but consensus formation requires accommodating legitimate disagreement. This paper studies whether LLMs can generate written opinions that maximize approval across diverse groups.
- Extant preference-tuning methods treat human preferences as homogeneous and static.
- Consensus is important for cooperation and democratic decision making but is difficult to achieve among people with divergent opinions.
- The study fine-tunes a 70 billion parameter Chinchilla model to generate consensus candidates from groups’ written opinions.
- The model is trained to produce group-based consensus candidates rather than adopt or persuade users toward a particular opinion.
2 Related Work
The paper connects preference-based language-model alignment with summarization, opinion summarization, and collective reasoning, while distinguishing consensus from fact-grounded summaries. Its approach uses human feedback and large-scale language modeling to support public deliberation.
- Language models have been fine-tuned from human preferences for tasks including summarization, translation, citation, and instruction following.
- Summarization: Unlike factual summaries, consensus statements address subjective disagreement rather than a single accepted viewpoint.
- Summarization: Consensus resembles opinion summarization but is grounded in the opinions of a specific group and evaluated with feedback from those same individuals.
- Collective Reasoning: The paper applies large-scale language modeling to consensus generation as a tool for public deliberation and collective reasoning.
3 Methods
The method collects group opinions on clustered debate questions, generates multiple consensus candidates, predicts individual agreement, and selects candidates using social welfare. Human ratings support iterative supervised fine-tuning and reward-model training.
- 3.1 Generating debate questions: The dataset contains 2,922 debate questions organized into 110 topic clusters after expanding 152 seed questions with a prompted 70 billion parameter LLM.
- 3.2 Data collection and environment design: Groups of four or five UK participants wrote opinions, rated consensus candidates, and provided baseline agreement ratings in a custom online application.
- 3.3 Group alignment: Predicted individual agreement scores are aggregated with cardinal social welfare functions, including Utilitarian, Rawlsian, and Bernoulli-Nash objectives.
- 3.4 Training: A supervised fine-tuned LLM generates 16 candidates, which a reward model reranks by expected social welfare before selection.
- 3.4 Training: Training iterates through human candidate ratings, quality-filtered supervised fine-tuning, and reward-model learning from agreement ratings.
- 3.3 Group alignment: A reward model predicts each participant’s agreement with a candidate conditional on that participant’s own opinion.
4 Results
The SFT-Utilitarian model is evaluated against prompted, fine-tuned, and human-generated alternatives on within- and out-of-distribution questions. It generally receives stronger human preference, including on divisive questions, while analyses test generalisation, robustness, and agreement outcomes.
- Evaluation design: The evaluation includes separate within- and out-of-distribution experiments, group-level mean and minimum agreement metrics, and median-based robustness checks.The out-of-distribution questions came from unseen topic-space clusters, while median analyses left the primary substantive conclusions unchanged.
- Baseline comparisons: The SFT-Utilitarian model samples 16 statements and selects the candidate maximizing predicted welfare under a Utilitarian max-mean aggregation function.SFT-Base omits aggregation-function selection and samples only one statement; few-shot and zero-shot prompted Chinchilla models provide additional baselines.
- Baseline comparisons: The SFT-Utilitarian model was preferred over SFT-Base in agreement ratings, while quality ratings increased across the training pipeline.The mixed-effects analysis estimated β = 0.12 for SFT-Utilitarian over SFT-Base (p = 0.011); supervised fine-tuning and reward modelling both affected quality ratings.
- Agreement outcomes: 65.6% of candidates were less divisive than the initial statements, and 40.8% of divisive rounds produced unanimous support.These results concern rounds where participants initially held different positions.
- Human-opinion comparisons: The model’s best-rated candidate was preferred to the best-rated human opinion 65% of the time, with win rates of 66% on more divisive and 63% on less divisive questions.The overall 95% confidence interval was [61%, 69%]; divisive-question intervals were [61%, 71%] and [57%, 69%].
- Out-of-distribution generalisation: 76% of mean-candidate comparisons and 59.5% of max-candidate comparisons favored the model over human opinions on out-of-distribution questions.The corresponding confidence intervals were [74%, 79%] and [56%, 63%], and performance was numerically similar within distribution.
5 Discussion
The discussion presents consensus generation as a way to accommodate diverse preferences while emphasizing unresolved limitations involving welfare aggregation, participant diversity, scale, misuse, factuality, and representation.
- The model is intended to generate statements that maximize agreement within a group of diverse human preferences.
- Social welfare functions: Utilitarian reranking improves both minimum and mean ratings over baselines, but alternative welfare functions show no meaningful rating differences.
- Data collection: UK-only recruitment limits participant diversity and generalisability, while potentially reflecting demographic biases in consensus statements.
- Scale: Groups contain only four or five people, constraining applications that require aggregating thousands of opinions.
- Misuse for persuasion: Although the model was not trained to persuade a particular political position, LLM-generated persuasion could become manipulative or coercive.
- Factuality and representation: Consensus statements may be misleading, factually false, or misrepresent minority views when presented as unanimous opinions.
B.1 Experimental protocol
Participants wrote opinions, reviewed generated consensus statements, and rated each statement for agreement and quality across repeated randomized presentations. Reliability filtering and harmful-content reporting supported data-quality control.
- Each experiment used four political-question rounds with opinion writing, candidate generation, and candidate rating phases.
- Participants rated consensus candidates on seven-point Likert scales for both agreement and quality, with their question and own opinion displayed as reminders.
- All candidates were shown twice in different random orders, enabling intra-rater reliability measurement for data-quality filtering.
- B.2.1 Intra-rater reliability: Linearly-weighted Cohen’s Kappa quantified rating consistency, and participants below κ = 0.6 were filtered as unreliable.
- B.2.2 Harm report: The harmful-content reporting function was used nine times across the data, and every report was explicitly described by the participant as a mistake.
C Model training and evaluation
The study trained consensus-generating Chinchilla models through prompting and supervised fine-tuning, using iterative human-rated data and welfare-oriented prompt construction. Training used filtered high-quality candidates, randomized opinion order, and evaluation-set perplexity.
- The pipeline fine-tuned generative and reward models after two consecutive data-collection rounds, progressing from prompted candidates to models trained on earlier data.
- C.1.1 Zero-shot and few-shot prompting: Few-shot prompts used three real examples containing three, four, and five opinions to promote diversity and generalization across input-group sizes.
- C.1.1 Zero-shot and few-shot prompting: Few-shot examples were filtered for mean quality above “Neutral,” ranked under the specified social welfare function, and sampled from the top 10 candidates.
- C.1.2 Supervised fine-tuning: Supervised fine-tuning used the zero-shot prompt and highly rated candidates from previous rounds as training targets.
- C.1.2 Supervised fine-tuning: Opinion order was randomized each training epoch to encourage order invariance and increase training diversity.
- C.1.2 Supervised fine-tuning: The first and second fine-tuning rounds used 1525 and 2355 new candidates, with 3880 total candidates used in the second round.
C.2 Reward model
The reward model predicted individual agreement with candidate statements and supported welfare-based reranking. It was trained from filtered pairwise human preferences and outperformed naive baselines, with a small gain after the second training round.
- The reward model predicted a scalar reward from the question, one opinion, and a consensus candidate, then used individual scores to select candidates maximizing expected group welfare.
- Training converted agreement ratings into pairwise preferences, removed unreliable-rater data and rating differences of one point or less, and cleaned the candidates.
- The first and second rounds produced 6639 and 3795 rating pairs, while the second reward-model iteration trained on 10434 pairs from both rounds.
- The reward model was warm-started from pretrained Chinchilla with an added linear reward-prediction layer and trained with standard cross-entropy loss.
- Both reward models outperformed naive baselines on within-distribution and out-of-distribution test questions, with an approximately 1% accuracy increase after the second training round.
D.1 Model size ablation experiment
The model-size ablation directly compared 70B and 1.4B Chinchilla-based models in a human evaluation of six consensus-generating systems.
- The ablation fine-tuned a 1.4B model and trained a 1.4B reward model using the same architecture and dataset as the larger Chinchilla system.
- Human evaluation with n = 224 participants compared consensus candidates generated by six models, including 70B and 1.4B systems.
D.3 Win-rate analysis on group median agreement scores
Median aggregation provides a complementary robustness check for group agreement scores, and the primary conclusions remain unchanged under this analysis.
- Median aggregation is used alongside the mean because Likert-rating distributions may be non-Gaussian, especially for divisive questions.The median is also more robust to outliers in small samples.
- The substantive conclusions of the primary analyses are unchanged when group agreement is aggregated by the median.
D.4 Regression analysis details
The appendix supplements win-rate evaluations with mixed-effects regression and tests whether aggregation choices affect model preferences. The out-of-distribution results favor SFT-Utilitarian over SFT-Base, while welfare-function sensitivity is not reliably detected.
- Regression analysis details: The regression models account for non-independent ratings clustered by participant and question, including random intercepts and model-variant slopes.
- Regression analysis details: β = 0.26; SE = 0.071; z = 3.6; p = 0.0002 for stronger agreement with SFT-Utilitarian than SFT-Base on out-of-distribution questions.
- Regression analysis details: β = 0.35; SE = 0.071; z = 4.9; p < 0.001 for preferring SFT-Base over the few-shot model on out-of-distribution questions.
- Regression analysis details: β = 0.17; SE = 0.089; z = 1.4; p = 0.16 indicates no preference for few-shot over zero-shot models on out-of-distribution questions.
- Social welfare functions: Control over the reranking aggregation function did not show the predicted effects on group welfare.The evaluation compared candidates generated under distinct social welfare functions, including Egalitarian.
D.7 Divisiveness of Candidate Consensus statements
The analysis measures whether generated consensus statements reduce divisiveness relative to initial position statements and whether they achieve unanimous consent on divisive questions. SFT-Utilitarian performs best among the compared models on both measures.
- Divisiveness measure: Candidate consensus divisiveness is the proportion of participants who agree with the statement, while initial-position divisiveness is an unsigned Group Internal Agreement measure.
- Less-divisive candidates: 65.6% [61.9, 69.3] of SFT-Utilitarian candidates were less divisive than the corresponding Position Statements.The corresponding proportions were 58.8% [54.9, 62.6] for SFT-Base, 53.1% [49.4, 57.2] for Few-shot, and 54.6% [50.8, 58.4] for Zero-shot.
- Unanimous consent: 40.8% [35.4, 46.2] of SFT-Utilitarian rounds achieved unanimous consent on divisive questions.The corresponding rates were 31.9% [26.8, 36.9] for SFT-Base, 31.6% [26.4, 36.6] for Few-shot, and 33.8% [28.7, 39.2] for Zero-shot.
- Supplementary materials: The appendix tables document prompt templates, question examples, Likert labels, model-size comparisons, participant examples, and baseline candidates.