PaperlayerView exact revision on arXiv

Source-linked AI summary

Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback

Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, William S. Zwicker

arXiv:2404.10271v2cs.LGcs.AIcs.CLcs.CYcs.GT

TL;DR

AI alignment must contend with potentially divergent human input, including who shapes principles and how preferences are combined. This paper applies social choice theory to these questions and argues it is well placed to help address them, while calling for substantial further research.

  • Problem

    RLHF and constitutional AI raise open questions about whose input should shape model behavior and how potentially diverse preferences should be aggregated.

    Source: §1 · pp. 1–2

  • Method

    The paper uses social choice theory to frame questions about selecting feedback providers, collecting input, and aggregating or using it in AI systems.

    Source: §1 · p. 2, §6 · p. 5

  • Results

    The paper argues that social choice is well placed to help address collective-choice problems in AI alignment, conceptually and pragmatically.

    Source: §10 · p. 9

  • Takeaways & Limitations

    Social choice can inform not only RLHF but also choices about AI objectives, training data, and which systems to build.

    Source: §10 · p. 9

    Addressing these questions requires complementary research, including work on practical decisions and associated legal and political considerations.

    Source: §10 · p. 9

Abstract

from arXiv · show

Foundation models such as GPT-4 are fine-tuned to avoid unsafe or otherwise problematic behavior, such as helping to commit crimes or producing racist text. One approach to fine-tuning, called reinforcement learning from human feedback, learns from humans' expressed preferences over multiple outputs. Another approach is constitutional AI, in which the input from humans is a list of high-level principles. But how do we deal with potentially diverging input from humans? How can we aggregate the input into consistent data about "collective" preferences or otherwise use it to make collective choices about model behavior? In this paper, we argue that the field of social choice is well positioned to address these questions, and we discuss ways forward for this agenda, drawing on discussions in a recent workshop on Social Choice for AI Ethics and Safety held in Berkeley, CA, USA in December 2023.

1. Introduction

RLHF and constitutional AI both raise questions about whose input shapes model behavior and how differing preferences should be combined. The paper argues that social choice theory can guide this work, while noting potential benefits and unresolved challenges.

1. Introduction

  • RLHF can rely on unrepresentative data, unrealistic decision models, and insufficient modeling of human diversity, while constitutional AI still requires decisions about who shapes its principles.The authors link limited modeling of diversity to possible political bias.Source: §1 · pp. 1–2
  • Social choice theory offers established methods for deciding whose preferences to integrate and how to aggregate them, while new work is needed for AI-specific problems.The paper presents this as a position and research agenda, not a completed solution.Source: §1 · p. 2
  • Principled inclusion of broader input may support fairness, more accurate feedback on truthfulness, and broader buy-in.The paper presents these as likely advantages, with epistemic-democracy literature cited for the accuracy claim.Source: §1 · p. 2
  • Naive aggregation can produce cyclical preferences or logically inconsistent conclusions, but social choice theory also offers tools to address these problems..Source: §1 · p. 2
  • Although computational social choice is established, little work has connected it to alignment of modern AI systems.Source: §1 · p. 2
  • The paper calls for further research to support responsible AI based on potentially diverging feedback from multiple stakeholders.Ad-hoc approaches may fail to represent stakeholders well or create conflict between groups.Source: §1 · p. 2

2. Background

The background introduces value alignment, human- and AI-feedback fine-tuning, and social choice theory. It outlines how these areas relate and the aggregation challenges they raise.

2.1. Value Alignment

  • Value alignment concerns whether increasingly capable AI systems act in accordance with human and societal values.Source: §2.1 · p. 2

2.2. Reinforcement Learning from Human Feedback

  • RLHF collects preferences over model outputs, fits a reward model to those judgments, and uses reinforcement learning to train a reward-maximizing policy.The reward-model approach assumes human preferences reflect a ground-truth reward function up to noise; diverse-source data can implicitly create intransparent or flawed aggregation.Source: §2.2 · p. 2

2.3. Alternate Preference-Based Fine-Tuning Objectives

  • DPO directly optimizes a loss on preference labels, while KTO learns a policy from unpaired preferences.The paper notes that DPO avoids online policy sampling and explicit reward-model training.Source: §2.3 · p. 3

2.4. Constitutional AI

  • Constitutional AI uses a human-written set of principles and a separate LLM to generate preference and instruction data for fine-tuning.For instruction data, the model revises an output against sampled principles; preference data uses principles as context for a feedback model.Source: §2.4 · p. 3

2.5. Social Choice

  • Arrow’s theorem shows that a preference aggregation rule meeting specified conditions can only be dictatorial; relaxing independence of irrelevant alternatives permits other methods.The paper notes that analogous results apply to social choice functions selecting a winner or a set of alternatives.Source: §2.5 · p. 3
  • Borda Count, Instant Runoff, and Ranked Pairs can aggregate the same individual rankings differently.The example includes a majority cycle, which Ranked Pairs resolves by dropping the preference with the smallest victory margin.Source: Figure 1 · p. 3, §2.5 · p. 3
  • Social choice studies aggregation of preferences and other inputs, and recent work has begun applying its methods to RLHF.The paper identifies aggregation of approvals, grades, judgments, and subjective probabilities among the field’s objects of study.Source: §2.5 · pp. 3–4

3. What Are the Collective Decision Problems and their Alternatives in this Context?

The section considers what AI-related alternatives people might evaluate and how collective choices could represent their preferences. It contrasts explicit response options with probabilistic and multi-winner approaches.

3. What Are the Collective Decision Problems and their Alternatives in this Context?

  • For AI alignment, the set of alternatives is difficult to specify, and treating all parameterizations of a network architecture as options would be conceptually intractable.The paper frames alternatives as the objects over which preferences are collected and choices are made.Source: §3 · p. 4
  • RLHF commonly asks evaluators to choose among a small set of responses, while allowing them to supply a preferred response offers another way to conceptualize alternatives.The latter approach does not resolve how preferences generalize beyond a single prompt.Source: §3 · p. 4
  • Collective choices could select probability distributions over responses or a small set of answers, rather than a single deterministic response.The paper suggests multi-winner selections could be combined into a response listing representative answers; probabilistic rules may take preferences over distributions or plain alternatives as input.Source: §3 · p. 4

4. Who Provides the Human Feedback?

The section asks whose feedback should count when not all affected stakeholders can be queried. It sketches representative selection, stakeholder choice of representatives, delegation, and existing recruitment practices.

4. Who Provides the Human Feedback?

  • When querying all affected stakeholders is infeasible, a representative subset could be selected to approximate the alignment that full-population feedback would produce.The paper points to citizens’ assemblies and stratified sampling as relevant approaches.Source: §4 · p. 4
  • Stakeholders could elect proportionally representative assemblies or delegate their feedback rights through liquid democracy.Source: §4 · p. 4
  • RLHF has used Mechanical Turk, Upwork, Scale AI, Lionbridge, and purpose-built platforms for evaluator recruitment, a component the authors say deserves deeper discussion informed by social choice.Source: §4 · p. 4

5. What Is the Format of Human Feedback?

Feedback format is a design choice: inputs should be natural to give, informative about preferences, and usable for alignment; reward models offer a way to normalize diverse inputs for aggregation.

5. What Is the Format of Human Feedback?

  • Preference elicitation studies how to query agents for preferences, while communication complexity studies the number of bits needed to communicate information.Both topics have also been studied in voting settings.Source: §5 · p. 4
  • Some stakeholders whose preferences are difficult to elicit, including small children and non-human animals, might need representation by dedicated others.Source: §5 · p. 4
  • Li et al. (2023) propose using LLMs for preference elicitation.Source: §5 · p. 4

5.1. Multiple Format Options

  • Useful human feedback should be natural to give, informative about preferences and values, and usable for alignment.The section notes that different input formats may lead to differently aligned systems and affect human behavior.Source: §5.1 · p. 5
  • Evaluators could choose formats ranging from individual responses and dialogue sessions to votes, rankings, ratings, or verbal feedback interpreted as preference data.They could also evaluate multiple behavioral attributes, such as helpfulness, humor, or toxicity.Source: §5.1 · p. 5

5.2. Processing Diverse and Informal Feedback

  • Reward models could convert evaluators’ diverse input into a common form that can be aggregated with other evaluators’ input.Source: §5.2 · p. 5
  • An interpretation model ϕ could map prompts, response sets, evaluator features, feedback, and a response to a numerical rating consistent with the evaluator’s verbal feedback.The model could be based on a pretrained LLM fine-tuned for this interpretation task, which can be viewed as meta-learning.Source: §5.2 · p. 5
  • A preference model ψ could learn from simulated ratings to rate responses directly, then support fine-tuning or real-time steering, provided evaluators, prompts, and response sets match its training distribution.Including evaluator features allows preferences to be aggregated transparently in a subsequent social-choice step; omitting them reduces ψ to the standard RLHF preference model.Source: §5.2 · p. 5

6. How Do We Incorporate Diverse Individual Feedback?

The section sketches social-choice approaches for incorporating diverse feedback: aggregate preferences during RLHF, or simulate collective decisions for supervised training or inference-time selection.

6. How Do We Incorporate Diverse Individual Feedback?

  • The paper sketches two approaches: add preference aggregation during training to create RLCHF, or add a simulated collective decision during training or real-time decisions.The approaches draw on social-choice methods to incorporate diverse feedback consistently.Source: §6 · p. 5

6.1. Proposal: Reinforcement Learning from Collective Human Feedback (RLCHF)

  • In rankings-based RLCHF, a jury’s individual rankings are combined by an ordinal social welfare function before the collective ranking trains the preference model.This aggregation can be inserted between collecting rankings and training the common preference model.Source: §6.1 · p. 6, Figure 2 · p. 6
  • In ratings-based RLCHF, evaluator-feature-conditioned preference models produce individual ratings that a cardinal social welfare function aggregates into rewards for fine-tuning.A representative sample of stakeholder feature vectors supplies the simulated individual ratings.Source: §6.1 · p. 6

6.2. Proposal: Simulated Collective Decisions

  • Supervised learning from simulated collective decisions replaces aggregated ratings with a social-choice function’s winning response, which becomes training data for the AI system.A multiwinner rule could instead select several responses and combine them into one response.Source: §6.2 · p. 6
  • An inference-time variant leaves the LLM pretrained, generates candidate responses and representative evaluator features for each prompt, and directly returns the simulated collective-choice winner.The choice rule could be multiwinner or probabilistic.Source: §6.2 · p. 6

7. Which Traditional Social-Choice-Theoretic Concepts Are Most Relevant?

The section argues that the relevance of social-choice concepts depends on the application and illustrates this with properties of response selection, strategic feedback, voter treatment, and principles.

7. Which Traditional Social-Choice-Theoretic Concepts Are Most Relevant?

  • Figure 3 presents evaluator features as additional reward-model inputs alongside aggregated ranks.Source: Figure 3 · p. 7
  • Social-choice concepts should be assessed for application-specific relevance; false-name-proofness matters for online voting but not public hand-raising.False-name-proofness concerns whether someone can benefit by participating multiple times under multiple accounts.Source: §7 · p. 7

7.1. Independence of Clones

  • Introducing a clone can change an election’s winner under plurality; this matters for chatbots because near-duplicate responses may be common, and Borda Count violates clone independence.In the example, splitting the Chinese restaurant into two options changes the winner to the Indian restaurant, despite 52% preferring Chinese food.Source: §7.1 · p. 7

7.2. Strategic Voting

  • Strategic feedback can distort reported preferences; in the rating example, reporting 0 instead of 3 moves the average from 5 to 4, while the median stays 6.The median is strategyproof when an evaluator’s sole goal is to move it closer to their true rating.Source: §7.2 · pp. 7–8
  • Figure 4 describes simulating responses from individual or cardinal reward models, broadening preference studies and opening future work on personalization.Source: Figure 4 · p. 8

7.3. Anonymity

  • Anonymity means swapping two voters’ ballots does not change the voting rule’s output, but AI development may justify weighting evaluators differently.Experience, evaluator ratings, or dependence between inputs may motivate departures from anonymity.Source: §7.3 · p. 8

7.4. Principles as Voters

  • Ethical principles could act as voters evaluating AI outputs; respondents could justify rankings by principles while the system learns principle-specific ratings and aggregates them.The principles themselves could also be outputs of a prior social-choice procedure involving human voters.Source: §7.4 · p. 8

8. How Should We Account for Behavioral Aspects and Human Cognitive Structures?

Preference elicitation often assumes consistent preferences and responses, but people may become indecisive when asked to make morally consequential choices. The section raises questions about whether and how to correct for such behavioral effects.

8. How Should We Account for Behavioral Aspects and Human Cognitive Structures?

  • Preference elicitation often assumes people have well-defined, consistent preferences and answer consistently apart from random noise.Source: §8 · p. 8
  • People may become indecisive when asked for input with moral implications, as studied in the context of kidney allocation.Source: §8 · p. 9
  • Correcting behavioral effects raises questions about whether underlying “true” values exist, whether correction removes valuable information, and whether it could be abused to remove inconvenient feedback.Source: §8 · p. 9

9. How Do We Navigate a Multiplicity of AIs?

The paper considers when disagreement might justify multiple AI systems and identifies both coalition-based alternatives and practical risks of fragmentation and conflict.

9. How Do We Navigate a Multiplicity of AIs?

  • Strong disagreement could justify multiple systems, with feedback groups formed before collection or formed afterward through representation mechanisms.The paper gives country-based systems as an example of ex ante grouping.Source: §9 · p. 9
  • People dissatisfied with an existing AI might pool resources to build their own system, a scenario considered in cooperative game theory (cf. Chalkiadakis et al. 2011).Cooperative game theory also bears on representation.Source: §9 · p. 9
  • Multiple systems may divide people unnecessarily, fail to fit existing social structures, or conflict when they interact.A government seeking one system for all citizens is an example where splitting may be infeasible.Source: §9 · p. 9
  • Cooperative AI research may help avert adverse interactions, but avoiding competing goals may be preferable.The paper also suggests designing systems to make conflict less likely.Source: §9 · p. 9

10. Conclusion

The conclusion argues that social choice can help address conflicting stakeholder input, while stressing that aggregation is complex and requires further, complementary research.

10. Conclusion

  • Social choice is well placed to help make consistent collective decisions from diverse stakeholder input on AI systems.The paper points to aggregating preferences, judgments, and other inputs, and notes computational social choice researchers are positioned to engage with AI alignment researchers.Source: §10 · p. 9
  • Aggregation requires choices about stakeholder selection, feedback collection and weighting, expertise, credibility, bias, transparency, and accountability.The paper says significant research is needed to understand both the possibilities and effects of applying social choice to these tasks.Source: §10 · p. 9
  • The questions require complementary research, including work on practical decisions and their legal and political considerations.The paper says these questions cannot be adequately addressed without complementary, not necessarily AI-specific, research.Source: §10 · p. 9
  • Although the paper focuses on RLHF, social choice also bears on choosing AI objectives, training data, and which systems to build.The conclusion calls for researchers to forge these connections amid rapid AI development.Source: §10 · p. 9

Impact Statement

The impact statement calls for collaboration and further research, noting both potential benefits of broader input and the risk of inconsistent collective judgments.

Impact Statement

  • Broader participation in AI ethics and safety research is urged, alongside systems that align with societal values and promote accountability and transparency.The paper expects its approach to yield fairer systems, fewer blind spots, and broader buy-in.Source: Impact Statement · p. 10
  • Majority aggregation can produce cyclical preferences or logically inconsistent judgments, while social choice studies ways to obtain consistent aggregations.The examples include majority preferences cycling among alternatives and majority judgments conflicting over whether an output is safe, helpful, and should be given.Source: Impact Statement · p. 10

Related papers

Loading 2404.10271v2…