PaperlayerView exact revision on arXiv

Source-linked AI summary

Moral Foundations of Large Language Models

Marwa Abdulhai, Gregory Serapio-Garcia, Clément Crepy, Daria Valter, John Canny, Natasha Jaques

arXiv:2310.15337v1cs.AIcs.CLcs.CY

TL;DR

Internet-trained LLMs may reflect cultural and political biases, but how their moral judgments relate to human values and prompt context is not well understood. The paper measures LLM moral foundations against human populations, tests their stability and promptability, and examines downstream effects. It finds that foundations are relatively consistent across contexts, can be deliberately shifted, and are associated with changes in donation behavior.

  • Problem

    Because LLMs are trained on internet-scale data, their absorbed cultural and political biases and effects on downstream tasks remain poorly understood.

    Source: §1 · p. 1

  • Method

    The authors use the Moral Foundations Questionnaire to compare LLM scores with human studies, test variation across prompts, and assess deliberate prompting and donation behavior.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    39% less was donated under harm-prioritizing than loyalty-prioritizing prompts, while LLM moral-foundation tendencies were relatively consistent across contexts.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Prompting can shift GPT-3's moral-foundation scores and affect donation behavior, creating a risk of targeted advertising designed to appeal to people's moral sensibilities.

    Source: §5 · p. 8, §6 · p. 9

    Further evidence is needed to determine whether LLM questionnaire responses reflect a consistent set of moral values akin to humans.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Moral foundations theory (MFT) is a psychological assessment tool that decomposes human moral reasoning into five factors, including care/harm, liberty/oppression, and sanctity/degradation (Graham et al., 2009). People vary in the weight they place on these dimensions when making moral decisions, in part due to their cultural upbringing and political ideology. As large language models (LLMs) are trained on datasets collected from the internet, they may reflect the biases that are present in such corpora. This paper uses MFT as a lens to analyze whether popular LLMs have acquired a bias towards a particular set of moral values. We analyze known LLMs and find they exhibit particular moral foundations, and show how these relate to human moral foundations and political affiliations. We also measure the consistency of these biases, or whether they vary strongly depending on the context of how the model is prompted. Finally, we show that we can adversarially select prompts that encourage the moral to exhibit a particular set of moral foundations, and that this can affect the model's behavior on downstream tasks. These findings help illustrate the potential risks and unintended consequences of LLMs assuming a particular moral stance.

1 Introduction

The paper applies a psychological tool to examine whether internet-trained LLMs reflect cultural or political biases in their moral judgments. It compares model foundations with human populations, tests their stability across prompts, and examines whether prompting affects downstream behavior.

1 Introduction

  • The paper addresses limited understanding of how cultural and political biases absorbed from internet-scale training data carry into LLM behavior.The authors note that model complexity and opacity make these biases difficult to understand.Source: §1 · p. 1
  • Moral Foundations Theory decomposes human intuitive ethical judgments into dimensions including care/harm, fairness/cheating, loyalty/betrayal, authority/subversion, and sanctity/degradation.The paper notes that liberals often emphasize care and fairness, while conservatives place relatively equal weight across foundations.Source: §1 · p. 1
  • The authors use the 30-question Moral Foundations Questionnaire to score LLMs and compare them with human studies across societies.They also test whether foundation scores remain stable across conversational prompts.Source: §1 · p. 1
  • In a dialog-based charitable donation benchmark, models prompted to prioritize the harm foundation donated 39% less than those prompted to prioritize the loyalty foundation, showing moral-foundation weighting can affect downstream behavior.Source: §1 · p. 2

2 Related Works

The related work situates this study amid research on LLM biases and moral reasoning, and uses Moral Foundations Theory as a framework for examining model judgments.

2.1 Language Models

  • Research documents LLM biases and inconsistencies, while this paper examines whether models maintain a consistent moral framework across contexts.Unlike work focused on Delphi, it studies general-purpose GPT and PaLM, compares them with human populations, and tests prompting effects on downstream behavior.Source: §2.1 · p. 2

2.2 Moral Foundation Theory

  • Moral Foundations Theory decomposes human moral judgments into dimensions that capture much of the variation across people and cultures.Prior studies have used the framework to examine political views and positions on health care and climate change.Source: §2.2 · p. 2

3 Method

The methods use questionnaire responses and comparisons with human studies to assess LLM moral foundations, their variation across contexts, and the effects of deliberate prompting. The experiments also test whether prompted foundations correspond to different donation behavior.

3 Method

  • The study analyzes GPT-3 and PaLM to examine moral foundations that may reflect values encoded from training data.The PaLM model is a 62B-parameter quantized version; GPT-3 engines range from 2.7B to 175B parameters.Source: §3 · p. 3, §3.1 · p. 3

3.1 Applying Moral Foundation Questionnaire to LLMs

  • The researchers prompt each model with MFQ questions, repeat each question 50 times, and score majority-voted responses using the questionnaire key.They include possible rating labels and examples in prompts, varying the example rating to check for bias.Source: §3.1 · p. 3

3.2 Experimental Methodology

  • The first experiment compares default and politically prompted LLM scores with human populations, using absolute-error distances and t-SNE visualizations.The human studies include US-American and South Korean participants; the comparison also covers GPT-3 engines and PaLM.Source: §3.2 · pp. 3–4
  • To assess consistency without targeted moral manipulation, the researchers prompt models with 50 randomly sampled BookCorpus dialogues before administering the questionnaire.They measure the resulting foundation scores across prompts.Source: §3.2 · p. 4
  • For deliberate manipulation, the researchers select prompts that maximize each moral foundation relative to the others for GPT-3 DaVinci2.They also test whether political prompts can change questionnaire scores.Source: §3.2 · p. 4
  • The downstream experiment tests whether political or moral-foundation prompts change GPT-3's donation to Save the Children.Across 20 trials per prompt, the researchers estimate average donations from probabilities over five possible amounts.Source: §3.2 · pp. 4–5

4 Experiments

The experiments compare LLM moral-foundation profiles with human groups, test their stability and steerability under prompts, and examine whether prompted profiles correspond to different donation behavior. The reported results include default-model political similarities and a mismatch between conservative prompting and human-conservative Fairness scores.

Question 1: Similarity between LLMs and Human Moral Foundations

  • DaVinci2 had smaller differences from human profiles than Babbage and Curie, whereas DaVinci3 had greater distances; PaLM had the lowest absolute difference to any human model.The authors suggest larger or more expressive models may come closer to human values, while noting DaVinci3’s greater distance after human-feedback training.Source: §4 · p. 5, Table 1 · p. 6
  • Default LLM profiles were generally closest to conservative human groups, with DaVinci2 closest to anonymous conservatives, PaLM and Curie to conservative Koreans, and Babbage to conservative US-Americans.DaVinci3 was most similar to moderate Koreans; these comparisons are based on the reported Table 1 and Figure 1 analyses.Source: §4 · p. 5, Table 1 · p. 6, Figure 1 · p. 6
  • Without a political prompt, most models’ profiles were most similar to conservative humans, while DaVinci2’s default profile had the lowest absolute error against anonymous conservative participants.The authors suggest the anonymous participants may align more closely with DaVinci2’s training data.Source: §4 · p. 5, Table 1 · p. 6
  • DaVinci2’s unprompted profile weighted Fairness and Harm most and Authority least; a liberal prompt captured human liberals’ stronger Fairness and Harm preferences.The comparison uses MFQ scores from the anonymous online human study and DaVinci2.Source: §4 · p. 6, Figure 2 · p. 6
  • A conservative prompt made DaVinci2 less similar to human conservatives than its default profile, alongside downweighting Fairness despite human conservatives’ strong Fairness scores.The authors suggest this may reflect a caricature of conservatism absorbed from training data.Source: §4 · pp. 6–7

Question 2: Measuring consistency

  • Across random dialogue prompts, GPT-3 showed little score variance for Fairness and In-group but more for Harm and Authority; PaLM was more consistent across prompts.The consistency analysis used 50 random BookCorpus dialogues.Source: §4 · p. 7, Figure 3 · p. 7

Question 3: Changing moral reasoning of LLMs

  • Prompting could maximize particular moral-foundation scores, but Fairness-maximizing prompts also increased Harm, while Authority and Purity responses appeared correlated.The prompt “You believe that some people are more important than others” maximized Purity, while “You believe in traditional roles” maximized Authority.Source: §4 · p. 7

Question 4: Effect on downstream tasks

  • Donation amounts differed across moral-foundation prompts; Ingroup-, Purity-, and Fairness-prompts donated most, while conservative prompts donated least, with no donation in 7/10 runs.The authors identify differing behavior on the donation task as the salient finding, while noting the amounts are specific to this task.Source: Table 2 · p. 7, §4 · p. 8

5 Discussion

The authors report that LLMs show relatively consistent moral-foundation tendencies associated with political affiliations, and that prompted foundations can correspond to changes in donation behavior. They caution that the human comparisons are preliminary and the downstream test covers only charity donations.

5 Discussion

  • Popular LLMs may tend toward particular moral foundations and political affiliations across conversation contexts, and these tendencies can affect downstream behavior.The authors describe their findings as preliminary and say further investigation is warranted.Source: §5 · p. 8
  • Prompts that elicited particular moral foundations also produced significant differences in GPT-3’s donations to Save the Children, linking MFQ scores with behavior on another task.The authors say the scores are correlated with behavioral changes; this does not establish that MFQ scores cause them.Source: §5 · p. 8
  • GPT-3’s default MFQ responses most resembled a politically conservative human, while liberal or moderate prompts produced responses most similar to liberal or moderate humans.The authors also report that political prompts could reliably change MFQ scores.Source: §5 · p. 8
  • Prompting a model to adopt a moral stance could enable targeted political advertising designed to appeal to people’s moral sensibilities.The authors identify this as a potential ethical risk, not as a demonstrated deployment outcome.Source: §5 · p. 8

5.1 Limitations

  • The downstream analysis tested only charitable donations, so the authors call for research on how moral foundations influence a wider variety of tasks.They particularly point to tasks aligned with actual LLM use through interfaces such as the GPT API.Source: §5.1 · pp. 8–9
  • The authors characterize their comparison of LLM and human questionnaire responses as an initial attempt and say more evidence is needed to establish human-like consistency in LLM moral values.They also propose examining language differences and note that the human studies date from 2012–2016, potentially reflecting a different political climate.Source: §5.1 · p. 9
  • Safety fine-tuning with reinforcement learning can reduce measurement sensitivity because the model answers the questionnaire with high confidence, complicating comparison with human-study responses.The authors say such comparisons may require further investigation.Source: §5.1 · p. 9

6 Ethics Statement

The authors warn that consistent political bias in LLM responses could disadvantage opposing views, while deliberate prompting can also steer moral stance and downstream behavior. They identify targeted, manipulative political advertising as a potential risk, alongside the possibility of prompting more moderate responses.

6 Ethics Statement

  • Consistent political bias in LLM responses could promote unfair bias against opposing political views.The authors connect this concern to the ACL ethics principles of avoiding harm and acting fairly.Source: §6 · p. 9
  • Deliberately prompting more moderate political views may help produce downstream responses with neither conservative nor liberal political bias.The authors describe this as a potential use of prompting.Source: §6 · p. 9
  • Prompts can be used to elicit a particular moral stance, creating a potential risk of targeted political advertising that manipulates people through their moral sensibilities.The authors compare this possibility with targeted advertising based on personality profiles.Source: §6 · p. 9
  • The authors report that LLM moral bias extends beyond MFQ answers to behavior on a downstream charitable-donation task.They state that further research is needed to determine how far such effects extend to other tasks.Source: §6 · p. 9

7 Appendix

The appendix outlines the moral foundations and the questionnaire used to assess them. It describes the rating prompts, answer scales, and item-to-foundation scoring.

7.1 Moral foundations background

  • Moral Foundations Theory uses five dimensions—care/harm, fairness/cheating, loyalty/betrayal, authority/subversion, and sanctity/degradation—to describe foundations of intuitive ethical judgments.The appendix presents these dimensions as arising from evolutionary thinking about morality and cross-cultural research on virtues.Source: §7.1 · p. 12

7.2 Moral foundations questionnaire

  • The 30-item questionnaire asks respondents to rate each statement’s relevance to right and wrong on a 0–5 scale.The scale ranges from “not at all relevant” to “extremely relevant.”Source: §7.2 · p. 12
  • The questionnaire includes questions about emotional suffering, fairness, loyalty, authority, and purity, among other topics.It also includes two catch questions intended to identify people not responding seriously.Source: §7.2 · p. 12
  • Each foundation score is the mean of six item responses, using separate item groups for harm, fairness, ingroup, authority, and purity.The listed statements use an agreement scale from strongly disagree to strongly agree.Source: §7.2 · p. 13

7.3 Applying MFQ to LLMs

The prompt construction varies the example rating shown to the model, then aggregates responses to limit example-induced bias and compare model scores with human studies.

7.3 Applying MFQ to LLMs

  • The example prompt asks the model to rate how relevant a sentence is to determining right and wrong, illustrating the task with “The sky is blue.”The example labels that sentence “very relevant.”Source: §7.3 · p. 13
  • The researchers vary the example rating, collect six responses per question for each possible rating, and average results to reduce bias from the example rating.Source: §7.3 · p. 13
  • They use majority vote for each question because a mean could yield an answer absent from human response distributions, then compare five-foundation scores with human studies using absolute error.The passage says this score-level comparison is typical of comparisons between human populations.Source: §7.3 · p. 13

7.4 LLMs Details and Parameters

DaVinci2 can be prompted to better match human moral foundations associated with different political affiliations, while the other listed models do not show the same adaptation. The section also presents PaLM moral-foundation scores and specifies GPT-3 settings.

7.4 LLMs Details and Parameters

  • Across GPT-3 engines, temperature is 0 and max tokens is 64; all other settings use defaults.The same model-specific parameters are maintained across engines.Source: §7.4 · p. 13

7.5 Moral Foundations for PaLM

  • PaLM moral-foundation scores are presented and used to compute values in Tables 1 and 2.The supplied passage does not report the score values.Source: §7.5 · p. 13, Figure 4 · p. 13

7.6 Supplemental Analysis of LLM responses to MFQ

  • DaVinci2 prompted with a political affiliation moves closer to human moral-foundation scores, unlike Curie, Babbage, DaVinci3, and PaLM.Liberal prompting most closely matches Korean liberals, moderate prompting matches a moderate anonymous-online population, and conservative prompting matches conservative populations.Source: §7.6 · p. 14

7.7 Distribution of responses from LLMs

The section presents response distributions and reports standard deviations for the first ten questions for davinci-002 and davinci-003.

7.7 Distribution of responses from LLMs

  • Response distributions are shown for every model.The reported standard deviations cover the first 10 questions for davinci-002 and davinci-003.Source: §7.7 · p. 14

7.8 Maximizing Moral foundation Scores

The section describes selecting prompts that maximize each moral-foundation score and gives examples of prompts tried for the foundations.

7.8 Maximizing Moral foundation Scores

  • Prompts are selected to maximize the score for a particular moral foundation relative to the others.The displayed material identifies the maximized foundation for each prompt.Source: §7.8 · p. 14, Figure 5 · p. 14
  • The appendix also shows scores for other prompts the authors tried.The supplied passage does not give those scores.Source: §7.8 · p. 14
  • Example prompts include nurturing or harm-focused language, justice and reciprocity, loyalty, hierarchy and tradition, and purity or religiosity.The examples span care, fairness, loyalty, authority, and sanctity prompts.Source: §7.8 · p. 14

7.9 Donation Experiment

The donation experiment uses a standardized employee response listing donation options to limit variation in the model's donation choice. The section also provides conversation examples and visuals describing donation and moral-foundation comparisons.

7.9 Donation Experiment

  • A standard employee response listing donation options is used to limit variance in the model's donation amount.The model selects from the listed options.Source: §7.9 · p. 14, Figure 7 · p. 15
  • Example conversations include GPT-3 responses to political profiles and prompts maximizing moral-foundation scores.The passage identifies bold statements as responses from GPT-3.Source: §7.9 · p. 14
  • Table 3 describes absolute-error comparisons between prompted DaVinci2 moral foundations and human groups by political affiliation and society.The table caption identifies the comparison scope but provides no cell values here.Source: Table 3 · p. 15
  • The appendix visuals show donation effects for political grounding prompts and maximized authority and fairness prompts.The captions identify conservative, liberal, and moderate grounding prompts on GPT-3, plus authority and fairness prompts; they do not state outcomes.Source: Figure 8 · p. 15, Figure 9 · p. 15, Figure 10 · p. 15

Related papers

Loading 2310.15337v1…