PaperlayerView exact revision on arXiv

Source-linked AI summary

What are human values, and how do we align AI to them?

Oliver Klingefjord, Ryan Lowe, Joe Edelman

arXiv:2404.10636v2cs.CYcs.AIcs.CLcs.HCcs.LG

TL;DR

The paper asks how to align language models with human values when concrete proposals remain limited and operator intent may not suffice. It proposes six criteria and Moral Graph Elicitation, which gathers contextual values and reconciles them into a moral graph. In a study with 500 Americans, the authors report promising results across the criteria, while noting that user preference for a model fine-tuned on the graph remains unknown.

  • Problem

    Concrete proposals for eliciting human values and aligning language models to them remain limited, while operator intent alone may be insufficient.

    Source: §1 · pp. 1–2

  • Method

    Moral Graph Elicitation uses contextual values cards and participant comparisons of which value is wiser to build a moral graph, guided by six alignment-target criteria.

    Source: §1 · pp. 2–3

  • Results

    MGE showed promising results across all six criteria in a case study with a representative sample of 500 Americans.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Contextual comparisons in a moral graph let wiser values emerge from participants rather than requiring expertise to be specified in advance.

    Source: §1 · p. 3

    Whether users will prefer interacting with a model fine-tuned on the moral graph remains unknown.

    Source: §6.3 · p. 25

Abstract

from arXiv · show

There is an emerging consensus that we need to align AI systems with human values (Gabriel, 2020; Ji et al., 2024), but it remains unclear how to apply this to language models in practice. We split the problem of "aligning to human values" into three parts: first, eliciting values from people; second, reconciling those values into an alignment target for training ML models; and third, actually training the model. In this paper, we focus on the first two parts, and ask the question: what are "good" ways to synthesize diverse human inputs about values into a target for aligning language models? To answer this question, we first define a set of 6 criteria that we believe must be satisfied for an alignment target to shape model behavior in accordance with human values. We then propose a process for eliciting and reconciling values called Moral Graph Elicitation (MGE), which uses a large language model to interview participants about their values in particular contexts; our approach is inspired by the philosophy of values advanced by Taylor (1977), Chang (2004), and others. We trial MGE with a representative sample of 500 Americans, on 3 intentionally divisive prompts (e.g. advice about abortion). Our results demonstrate that MGE is promising for improving model alignment across all 6 criteria. For example, almost all participants (89.1%) felt well represented by the process, and (89%) thought the final moral graph was fair, even if their value wasn't voted as the wisest. Our process often results in "expert" values (e.g. values from women who have solicited abortion advice) rising to the top of the moral graph, without defining who is considered an expert in advance.

1 Introduction

The paper asks how to align language models with human values beyond operator intent, and focuses on eliciting values and reconciling them into an alignment target. It proposes six criteria for such targets and introduces Moral Graph Elicitation (MGE), which uses contextual values and comparisons to construct a moral graph.

1 Introduction

  • The authors argue that operator intent alone can cause harm, while concrete proposals for aligning language models with human values remain scarce.They frame the paper around eliciting values and reconciling them into a training target, leaving model training itself for future work.Source: §1 · pp. 1–2
  • A suitable alignment target should be legitimate, robust, fine-grained, generalizable, auditable, and scalable.These criteria concern endorsement by affected people, resistance to influence, behavioral guidance, transfer to new situations, interpretability, and improvement with more participants.Source: §1 · p. 3
  • The authors contrast MGE with approaches they say lack fine-grained guidance or legitimacy, and note that case-specific directions may require expertise to be specified in advance.They argue that MGE can allow wiser values to emerge through the process instead.Source: §1 · p. 3
  • Values cards represent what is important or meaningful to a person in a particular context, rather than relying on abstract value words alone.The authors ground this approach in Taylor (1977) and Chang (2004a).Source: §1 · p. 3
  • MGE elicits values through language-model interviews about specific prompts, then reconciles them into a moral graph for the relevant context.The graph records comparisons in which one values card is considered wiser than another; the authors propose it as an alignment target.Source: Figure 1 · p. 2, §1 · p. 3
  • MGE was trialed with a representative sample of 500 Americans, and the authors report promising results across all six alignment-target criteria.They say participants endorsed the resulting values cards and found the process personally clarifying, while emphasizing that the criteria admit further improvement.Source: §1 · p. 3

2 Background

Existing alignment approaches trade off simplicity, scale, legitimacy, auditability, or contextual detail, while broad claims about values are difficult to operationalize. The paper motivates a context-sensitive alignment target that preserves distinct values and their relations rather than averaging them into a single choice.

2.1 Existing approaches

  • Supervised fine-tuning is simple but depends on representative demonstrations and is not competitive by itself; RLHF scales but is hard to audit and vulnerable to manipulation.The authors also describe RLHF as having low legitimacy in current implementations.Source: §2.1 · p. 4
  • Constitutional approaches can reduce human-data needs, but their vague principles are difficult to apply and reconcile in context; moral-judgment datasets often use simplistic scenarios.Collective CAI can improve legitimacy, but its elicited principles remain high-level and vague.Source: §2.1 · pp. 4–5

2.2 What are values?

  • The paper frames a further challenge: alignment values should be understandable, clear enough to judge model behavior, and aggregatable or reconcilable.Comments surfaced as shared values can instead be policies, goals, vague aspirations, or hard-to-interpret statements.Source: §2.2 · p. 5
  • Taylor’s definition treats values as non-instrumental criteria used in choice, but the authors say it remains vague about reliably eliciting or reconciling them.The restaurant example distinguishes a value-laden criterion, such as intimacy, from the instrumental criterion that a restaurant is open.Source: §2.2 · p. 6

2.3 How do values fit together?

  • Values are contextual, and averaging or selecting popular, median, or bridging values can discard where values apply and cut off productive moral learning.The paper argues that values change as people encounter new contexts and better understand what matters to them.Source: §2.3 · p. 7
  • A shared-data-structure approach can represent context-rich, fine-grained solutions across many stakeholders, unlike voting, while making values and their relations explicit.The authors propose reconciling values through contextual applicability, improvement, or balancing relationships.Source: §2.3 · pp. 7–8

3 Desiderata for an alignment target

The authors set six desiderata for alignment targets: three for steering model behavior and three for political acceptability. They argue that an interpretable target should capture context-specific values while remaining generalizable, scalable, robust, legitimate, and auditable.

3 Desiderata for an alignment target

  • The six criteria balance steering model behavior with political workability: groups should be able to agree on, update, and protect the target from manipulation.The section organizes the criteria into three for steering behavior and three for political considerations.Source: §3 · p. 8

3.1 Steering model behavior

  • Targets should be fine-grained, generalizable, and scalable, reflecting context-specific values while extending to new situations and improving as participation grows.The authors contrast this with vague principles that can lead models to advocate diversity in inappropriate situations or refuse actions that only sound harmful.Source: §3.1 · pp. 8–9

3.2 Political considerations

  • An alignment target should be legitimate, robust, and auditable; this paper focuses on resistance to persuasion and ideological factionalism, not hacking or sophisticated bad actors.Legitimacy concerns public scrutiny, while auditability includes tracing which values shaped a model output.Source: §3.2 · pp. 9–10

4 Moral Graph Elicitation

Moral Graph Elicitation (MGE) turns context-specific values articulated by people into values cards, then reconciles them in a graph of judgments about which values are wiser. The process uses LLM interviews and story-based comparisons to elicit cards and graph edges.

4.1 Values cards

  • Values cards represent values as sets of constitutive attentional policies used together in a shared choice context, rather than as ambiguous abstract labels.Each card bundles coherent policies with a title and short summary; the authors argue this supports robust, fine-grained, legitimate, auditable, and generalizable alignment targets.Source: §4.1 · pp. 11–13

4.2 What is a moral graph?

  • A moral graph reconciles values by recording context-specific judgments that one value is wiser than another, rather than selecting values in isolation.The authors use PageRank so transitive comparisons can contribute to identifying winning values for each context.Source: §4.2 · pp. 13–14

4.3 Eliciting values cards with LLMs

  • MGE interviews participants with an LLM to identify attentional policies, check that they are constitutive rather than merely instrumental, and let users edit their resulting values cards.97% of case-study participants reached a values card, including participants who began with a slogan.Source: §4.3 · pp. 14–15

4.4 Building the moral graph

  • To elicit graph edges, participants judge whether fictional people became wiser after moving between values in GPT-4-generated stories.The stories are not a guarantee that participants agree with the intended meaning of “wiser,” and the authors note susceptibility to model bias.Source: §4.4 · pp. 15, 17
  • A representative sample of Americans constructed a coherent moral graph with only one small cycle.Source: §4.4 · p. 15
  • 70.3% of participants agreed they learned something about others’ values, and 75.2% gained respect for other participants.Participants saw stories explaining the moral reasoning behind proposed value upgrades when voting on graph edges.Source: §4.4 · p. 16

5 Case study

The case study found favorable legitimacy ratings, including fair placement of participants’ values, and reported evidence of robustness, fine-grainedness, generalizability, and scalability. The authors also describe how the process handles value conflicts and identify limits to the evidence.

5.1 Process description

  • The 500-person US case study produced a moral graph with 85 deduplicated values and 100 edges, in a process taking a median of 15 minutes.Unresolved conflicts were omitted about 4% of the time, leaving those parts of the graph pluralistic and allowing model discretion.Source: §5.1 · p. 17

5.2 Evidence of robustness

  • Participants agreed that their values expressed what they care about, with no correlation between that judgment and how ideological their initial responses seemed.The ideological ranking was based on a prompt, and the authors qualify the robustness claim as applying to well-meaning rather than malicious ideological participants.Source: §5.2 · pp. 17–18, Figure 6 · p. 18

5.3 Evidence of fine-grainedness

  • Context-tagged edges indicate when a value applies, while values-card attentional policies give more specific guidance about behavior than the authors find in CAI principles.The paper contrasts this with CCAI principles that may be vague or vapid.Source: §5.3 · p. 18

5.4 Evidence of generalizability

  • 38.2% of selected values came from participants’ chosen scenario, versus 38.3% from a different scenario, indicating cross-scenario applicability in this test.Participants selected wise values from a mix that included values articulated for their chosen scenario and two other scenarios.Source: §5.4 · p. 20

5.5 Evidence of scalability

  • Informed Autonomy, most commonly articulated by women familiar with the abortion topic, rose in PageRank rank as more participant edges were considered.The study used unique relevant life experience as a proxy for expertise; the authors say broader topics and fine-tuned models remain to be tested.Source: §5.5 · pp. 20–21, Figure 9 · p. 21

5.6 Evidence of legitimacy

  • 89% of participants agreed their value had a fair position in the graph, and MGE scored higher than CCAI on all relevant legitimacy questions.The authors note that their assessment did not cover all aspects of input legitimacy or the social effects of models aligned with the graph.Source: §5.6 · pp. 21–22, Figure 10 · p. 22

6 Discussion

The discussion presents moral graphs as an auditable, context-specific alignment target, sketches ways to train models with them, and identifies limits involving deployment, participants, and evidence. The authors emphasize that model training and broader applicability remain open questions.

6.1 Relevance to alignment research

  • The moral graph makes values and their contexts explicit, which the authors argue can aid auditing, tracing unwanted behavior, and resisting manipulation.They also suggest that context-bounded values may reduce the risk of over-optimizing a single value, while noting that over-optimization could still occur within a context.Source: §6.1 · pp. 23–24
  • The authors suggest the graph’s human-inspectable edges could support continued moral reasoning by stronger AI systems, though this is a prospective possibility.They connect this idea to participants’ ability to assess claimed gains in wisdom and to endorse values other participants ranked as wiser.Source: §6.1 · p. 24

6.2 How to train a model on a moral graph

  • Training options include rating completions against the highest-ranked context-specific value or training a reward model directly on wisdom-upgrade preference tuples.The authors say more work is needed to evaluate the viability of these approaches.Source: §6.2 · p. 24

6.3 Limitations

  • The case study used a representative sample of Americans, so performance in countries with very different cultures remains untested.The authors also do not yet know whether users will prefer interacting with a model fine-tuned on a moral graph.Source: §6.3 · pp. 24–25
  • MGE requires participant time and substantial inference compute, averaging 15 minutes and about 35k tokens per user.The authors note that this compute requirement is absent from non-LLM alternatives such as Pol.is.Source: §6.3 · p. 24
  • The process may not resolve cycles in the moral graph, and human-elicited values may not fit LLM contexts or situations beyond human experience.The authors also identify possible bias from GPT-4-generated value-transition stories and extra steps in creating fine-tuning data.Source: §6.3 · p. 25

A.1 Social choice implications

The appendix argues that moral graphs broaden the information available for social choice beyond people’s stated or revealed preferences. It proposes that information about values and their relations may help address impasses in social choice theory.

A.1 Social choice implications

  • The authors argue that ordinary social-choice inputs—revealed or stated population preferences—are inadequate for making good social choices.They invoke Sen’s discussion of broadening the informational basis of social choice.Source: §A.1 · p. 30
  • MGE gathers information about how people think values fit together, in addition to values and votes for values.This provides a broader informational basis than preferences alone.Source: §A.1 · p. 30
  • The authors’ view is that data about underlying values, value relationships, and value-context relationships could help overcome social-choice impasses.They state that fully exploring this possibility is beyond the paper’s scope.Source: §A.1 · p. 30

A.2 Note on ideology

The appendix distinguishes ideological stances from the values that may motivate them. It also notes that ideology can arise from non-value factors or values unrelated to the ideology’s stated rationale.

A.2 Note on ideology

  • An ideological stance can be distinct from the values that may have motivated its adoption.The authors give valuing the voices of disempowered people or efficiency as possible motives for taking a stance on organizational structure.Source: §A.2 · p. 30
  • Ideology may also be adopted through factors that are not values, such as peer pressure.A person’s value might instead concern something different from what the ideology presents, such as a sense of shared cause.Source: §A.2 · p. 30
  • The authors’ definition allows an ideological position to be distinguished from any underlying values, when such values exist.Source: §A.2 · p. 30

A.3 Note on CCAI control studies

The authors ran two pol.is control studies, each with a US-representative sample of 300 participants. One replicated a prior deliberation; the other focused discussion on the abortion scenario.

A.3 Note on CCAI control studies

  • Both pol.is control studies used 300 participants representative of the US, matching the case-study sample size and representativeness.Source: §A.3 · p. 30
  • One study replicated Ganguli et al. (2023), using their first 25 seed comments and the same pol.is question.Source: §A.3 · p. 30
  • The context-constrained study used the abortion scenario, with 25 initial MGE chat responses as seed comments and the first abortion-question chat message as its pol.is question.Source: §A.3 · p. 30

A.4 Elaboration on robustness to ideological rhethoric

The appendix elaborates how MGE’s values and comparisons can reduce ideological rhetoric in the abortion case, while describing deduplication and the interview process used to build its values cards.

A.4 Elaboration on robustness to ideological rhethoric

  • In the abortion example, the branch winner “Faith-anchored personal growth” was widely endorsed as wiser, and 89% of people asked endorsed the wiser value over their own.The authors argue this adds a buffer against ideological trends, socially pressured beliefs, and rhetoric.Source: §A.4 · p. 31

B.1 Chat and values card articulation

  • The prompts for chat and values-card articulation are omitted from the appendix text, but the authors say they are available in their open-source repository.Source: §B.1 · p. 31

B.2 Ideology

  • The ideology prompt rates responses from 1 to 5 based on slogans, strong normative judgments about others’ lives, and persuasive intent.Source: §B.2 · p. 31

B.3 Has had abortion

  • A separate prompt identifies whether a user explicitly said they had or considered having an abortion, requiring a yes-or-no answer.Source: §B.3 · pp. 31–32

C Deduplicating values

  • Deduplication matches custom cards to nearby canonical cards, asks users whether a match represents them, and adds unmatched cards to the canonical pool; 95% felt their deduplicated value represented them well.Cards can be coalesced when their attentional policies align and their authors feel fully represented by the other card.Source: C Deduplicating values · p. 32

D.1 Example value elicitation dialogue

  • The example dialogue elicited a participant’s attentional policies and formed a “Religious Adherence” card, which the assistant clarified was specific to that user’s interactions.For users with different beliefs, the assistant would seek to respect their own values.Source: §D.1 · p. 34

D.2 Example of wisdom upgrades

The parenting example distinguishes values that participants saw as incomparable from moves between values they understood as clarifying what mattered or shifting to another good.

D.2 Example of wisdom upgrades

  • Participants saw igniting curiosity and fostering a healthy family environment as important in their own ways, with no “better” option between them.Source: §D.2 · p. 35
  • Moving between instilling discipline and igniting curiosity was viewed as clarifying a shared fundamental good, while moving from curiosity to a healthy family environment shifted focus to a different good.Source: §D.2 · p. 35
  • Figure 14 contains the full attentional policies for inspiring discipline and igniting curiosity.Source: §D.2 · p. 35

D.3 Example of winning values

The winning-values figures show two highest-PageRank values for each of the abortion, parenting, and weapons scenarios.

D.3 Example of winning values

D.4 Example of story generation

Figure 14 illustrates story generation for a motivation context, from sampling relevant values to presenting a generated story alongside values cards.

D.4 Example of story generation

  • For motivation, the example samples Inspiring Discipline and Igniting Curiosity as relevant values.The values are selected using a prompt.Source: Figure 14 · p. 38
  • A prompt chain generates a transition story step by step from the sampled values.Source: Figure 14 · p. 38
  • Users see the final story together with the values cards.Source: Figure 14 · p. 38
  • The prompts are available in the linking.ts file in the project repository.Source: §D.4 · p. 38
Loading 2404.10636v2…