PaperlayerView exact revision on arXiv

Source-linked AI summary

Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity

Gabriel Simmons

arXiv:2209.12106v2cs.CL

TL;DR

The paper asks whether LLMs reproduce moral biases associated with political groups, a question relevant to understanding how generated content may affect the human informational environment. It measures moral-foundation language in model outputs and finds identity-aligned mimicry, situational responsiveness, and a relationship between mimicry and model size.

  • Problem

    The paper examines whether LLMs reproduce political groups’ moral-foundation biases, amid concern that generated text may influence human opinions and the informational environment.

    Source: §1 · p. 1

  • Method

    The study measures five moral foundations in text from GPT-3, GPT-3.5, and OPT models prompted with scenarios and political identities.

    Source: §2 · pp. 2–3, §6 · p. 9

  • Results

    Models showed identity-aligned moral-foundation use and scenario responsiveness, differed from human consensus more than individual humans did, and showed size-related mimicry patterns that varied by model family.

    Source: §3 · pp. 6–7

  • Takeaways & Limitations

    The findings provide evidence that political identity-conditioned LLM outputs can reflect corresponding moral-foundation biases, while remaining distinguishable from human foundation use.

    Source: §3 · p. 6, §6 · p. 9

    Dictionary-based analysis is limited in its ability to detect nuanced moral expressions.

    Source: §4 · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated impressive capabilities in generating fluent text, as well as tendencies to reproduce undesirable social biases. This study investigates whether LLMs reproduce the moral biases associated with political groups in the United States, an instance of a broader capability herein termed moral mimicry. This hypothesis is explored in the GPT-3/3.5 and OPT families of Transformer-based LLMs. Using tools from Moral Foundations Theory, it is shown that these LLMs are indeed moral mimics. When prompted with a liberal or conservative political identity, the models generate text reflecting corresponding moral biases. This study also explores the relationship between moral mimicry and model size, and similarity between human and LLM moral word use.

1 Introduction

The introduction motivates examining whether LLMs can produce moral content aligned with political identities, given morality’s role in persuasion and polarization and the potential influence of LLM text on opinions. It previews evidence of situationally responsive moral vocabulary and political-identity mimicry, with mimicry related to model size.

1 Introduction

  • Morality matters to persuasion and polarization; moral argumentation can modulate willingness to compromise, and moral congruence affects argument effectiveness and perceived ethicality.The introduction also motivates understanding how LLM-generated content may influence human opinions and the informational environment.Source: §1 · p. 1
  • Moral Foundations Theory distinguishes five foundations; U.S. liberals primarily use individualizing foundations, while conservatives make more balanced appeals and use binding foundations more than liberals (Graham et al., 2009; Do˘gruyol et al., 2019; Frimer, 2020).The analysis uses the original five foundations available in the Moral Foundations Dictionaries, not the proposed sixth Liberty/Oppression foundation.Source: §1 · p. 1
  • The study addresses whether language models reproduce moral foundational biases associated with social groups such as political identities.The introduction notes prior work on moral biases in fine-tuned language models, other social biases, cultural values, and political-view repetition by dialogue agents, but says moral mimicry had not yet been examined.Source: §1 · pp. 1–2
  • The paper attributes moral argumentation or congruence to LLMs only as properties readers may perceive in outputs, not as evidence of communicative intent.The author notes that anthropomorphizing system behavior can distort perceptions of its underlying mechanisms.Source: §1 · p. 1
  • LLMs generated confirmatory liberal- and conservative-identity-aligned foundation use; mimicry increased with size in OPT and GPT-3/3.5 considered together, less so for GPT-3 alone.The study also reports that LLMs respond to salient moral attributes in scenarios, but their foundation use differs from human consensus more than individual humans’ use does.Source: §1 · p. 2
  • LLMs increased use of foundation words relevant to a scenario’s salient foundation, but differed from human consensus foundation use more than humans did.The study compares LLM outputs with human foundation use and reports both situational responsiveness and a gap from human consensus.Source: §1 · p. 2

2 Methods

The study prompts GPT-3, GPT-3.5, and OPT models with scenarios, political identities, and moral stances, then measures moral-foundation language using dictionaries. The results show identity-conditioned moral mimicry, differences from human consensus, and variation in mimicry across model families and sizes.

2 Methods

  • Prompts combine a scenario, political-identity phrase, moral stance, and one of five template styles; experiments draw scenarios from Moral Stories, ETHICS, and Social Chemistry 101.For the identity experiment, 2,000 Moral Stories and 1,000 ETHICS scenarios yield 12,000 prompts across liberal/conservative identities and moral/immoral stances, tested on text-davinci-001, text-davinci-002, text-davinci-003, and OPT-30B.Source: §2 · p. 2, §2.2 · p. 5

Measuring Moral Content

  • Three Moral Foundations Dictionaries estimate foundation content, with vice and virtue merged and general morality and sentiment scores excluded from the analysis.The dictionaries are MFDv1, MFDv2, and eMFD; foundation-specific scores are normalized by the total score across foundations.Source: Measuring Moral Content · pp. 3–4

2.2 Are LLMs Moral Mimics?

  • Across model families, political-identity prompts produce moral-foundation effects that mostly match the expected liberal-individualizing and conservative-binding pattern.The study calculates identity differences in foundation-expression probabilities for matched scenarios and stances, then averages over scenarios, stances, and dictionaries.Source: Measuring Moral Content · p. 4, §2.2 · p. 5

2.1 LLM vs. Human Moral Foundation Use

  • The study tests whether foundation vocabulary rises for scenarios labeled with that foundation and whether model-to-consensus differences are smaller than individual-human-to-consensus differences.For the human comparison, model outputs and human-written rationalizations are conditioned on each scenario’s normative moral acceptability.Source: §2.1 · p. 4
  • On Social Chemistry scenarios, text-davinci-002 increases use of all five foundation-associated vocabularies for matching ground-truth foundations, yet generally differs more from human consensus than individuals do.The foundation-specific comparison meets Criterion A, but the human comparison does not meet the stronger Criterion B.Source: Results · p. 4, Figure 2 · p. 5

2.3 Is Moral Mimicry Affected By Model Size?

  • For OPT, model size and MFH-SCORE correlate at r=0.69, while the GPT-3 relationship is weaker at r=0.23; combined GPT-3 and GPT-3.5 models correlate at r=0.84.The author notes limited statistical power for OPT; GPT-3.5 models show stronger mimicry than the 175B GPT-3 model, alongside differences in fine-tuning and pre-training.Source: §2.3 · pp. 5–6

2.2 Are LLMs Moral Mimics?

  • 11 of 60 identity-effect combinations oppose the Moral Foundations Hypothesis, and every opposite-direction effect is below 1 point absolute difference.Most effects align with the hypothesis: liberal prompting predicts more individualizing-foundation use and conservative prompting more binding-foundation use.Source: §2.2 · p. 5, Figure 3 · p. 6

3 Discussion

GPT-3, GPT-3.5, and OPT models tailored foundation-word use to political identity, while their alignment with human moral-word use and the role of model size varied.

3 Discussion

  • LLMs used more words from foundations salient to clear-consensus scenarios, yet differed from human consensus more than individual humans did.These findings correspond to the weaker Criterion A and stronger Criterion B, respectively.Source: §3 · p. 6
  • Across GPT-3, GPT-3.5, and OPT, conservative identity increased binding-foundation use and liberal identity increased individualizing-foundation use, matching the stated US political pattern.The stated pattern is that conservatives use foundations more evenly, while liberals rely mostly on Care/Harm and Fairness/Cheating.Source: §3 · p. 6
  • How dataset curation or other methods might influence moral-mimicry capabilities that emerge during training remains an open question.Source: §3 · p. 7
  • Moral mimicry generally rose with parameter count in OPT, less so in GPT-3; both 175B GPT-3.5 models exceeded 175B text-davinci-001, so size is not the only factor.The authors note that GPT-3.5 models received additional supervised fine-tuning and text-and-code pretraining, unlike the GPT-3 family.Source: §3 · p. 7

4 Limitations

The study’s conclusions are bounded by its dictionary-based measure, author-selected prompt variations, U.S.-based human comparison populations, and choice of models and moral framework. It examines apparent moral rationalizations and discourages normative use of its methods and results.

4 Limitations

  • Moral Foundations Dictionaries may miss nuanced moral expressions, and the study notes that machine-learning approaches and human evaluation could complement them.Prompt styles were also selected by the author; a more principled selection procedure might produce more diverse prompts.Source: §4 · p. 7
  • The human studies used for comparison drew on U.S. populations, while the political connotations of “liberal” and “conservative” differ across demographics.The study suggests examining additional demographic information and multilingual models in future work.Source: §4 · p. 7
  • The experiments tested GPT-3, GPT-3.5, and OPT, not BLOOM, which the authors could not test because of compute budget.GPT-3 and GPT-3.5 weights are unavailable for download, while hardware needs may hinder experimentation with open models.Source: §4 · p. 7
  • Moral Foundations Theory faces disputes about its parsimony, foundations, cross-cultural stability, and the Moral Foundations Questionnaire.The study chose MFT for its established text measures and links between political affiliation and moral biases; applying these methods to other moral theories is left for future work.Source: §4 · pp. 7–8
  • The study examines apparent moral rationalizations, not a system for normative judgment, and discourages normative use or interpretation of its methods and results.The authors frame the work as studying prompted model behavior amid concerns about accountability and the role of non-human systems in moral judgment.Source: §4 · p. 8

5 Related Work

Prior work studies LLM ethical judgments, scenario-based moral norms, normative alignment, and reproduced social biases, often without prompting a particular identity. This study instead conditions models on political identities and examines moral-foundation use as a response variable.

5 Related Work

  • Earlier work assessed normative ethical judgments, scenario-relevant moral norms, moral exceptions, ethical alignment, and reproduced societal biases in language models.Fraser, Kiritchenko, and Balkir (2022) found that Delphi responses to the Moral Foundations Questionnaire reflected biases of groups that produced the model and its training data.Source: §5 · p. 8
  • Unlike studies treating pre-trained models as the locus of cohesive biases, this work prompts opposed political identities and evaluates the moral-foundation fidelity of the resulting simulations.The identity-simulation framing draws on Argyle et al. (2023) and Aher et al. (2023).Source: §5 · p. 8
  • Compared with work using cultural-identity prompt language or fine-tuning GPT-2 on issue-specific stances, this study conditions on political identity and evaluates larger models without domain-specific fine-tuning.It investigates whether models mimic moral preferences.Source: §5 · pp. 8–9
  • Moral mimicry concerns apparent moral-foundation salience conditioned on identity labels, unlike sycophancy’s mirroring of political views in dialogue or algorithmic fidelity’s broader response distributions.Argyle et al. (2023) study partisan descriptors, voting patterns, and survey-response correlations as other response variables.Source: §5 · p. 9

6 Conclusion

The study concludes that prompted LLMs can reproduce moral-foundation biases associated with liberal and conservative identities. Their moral-word use also varies by situation, differs from human use, and may relate to model size.

6 Conclusion

  • LLMs conditioned on political identities reproduce moral-foundation biases associated with liberal and conservative groups.The study names this capability moral mimicry.Source: §6 · p. 9
  • LLMs modify moral-foundation use situationally, but their use is not indistinguishable from human use.Source: §6 · p. 9
  • The study reports that moral mimicry may relate to model size.Source: §6 · p. 9

A Appendix A: Additional Details Related to Experimental Methods

Table 1 identifies the models evaluated and provides information sources for GPT-3, GPT-3.5, OPT, and OPT-IML. Its caption also explains the FeedME and PPO labels and marks uncertain instruction fine-tuning.

A Appendix A: Additional Details Related to Experimental Methods

  • Table 1 covers models evaluated in the study, with information for GPT-3 and GPT-3.5 from OpenAI (2022), OPT from Zhang et al. (2022), and OPT-IML from Iyer et al. (2023).Source: Table 1 · p. 12
  • FeedME denotes supervised fine-tuning on human demonstrations and highly rated model samples.The caption defines the rating as 7/7 on an overall quality score.Source: Table 1 · p. 12
  • PPO denotes reinforcement learning with reward models trained from human comparisons, while “?” marks uncertain instruction fine-tuning based on documentation.Source: Table 1 · p. 12

A.2 Additional Details Related to Datasets Used in the Study

Moral Stories scenarios combine situation, intent, and a moral or immoral action, while ETHICS scenarios come from short commonsense examples balanced by normative acceptability.

A.2.1 Preprocessing Details for Moral Stories Dataset

  • Moral Stories scenarios concatenate a situation, intent, and moral or immoral action, excluding norms and consequences that could bias moral-foundation content.The study uses 2,000 scenarios: 1,000 moral and 1,000 immoral, randomly sampled.Source: §A.2.1 · p. 12
  • Both datasets provide moral and immoral scenarios, but only Moral Stories excludes scenario norms and consequences to avoid biasing generated moral-foundation content.ETHICS scenarios are labeled for binary ethical acceptability; Moral Stories scenarios are assembled from situation, intent, and action.Source: §A.2.1 · p. 12, §A.2.2 · p. 13

A.2.2 Preprocessing Details for ETHICS Dataset

  • ETHICS examples come from the commonsense train split, with 1,000 short scenarios randomly selected and evenly divided by ethical acceptability.The source split contains 6,661 short examples; its 7,249 longer Reddit examples were unused, primarily because of their higher use costs.Source: §A.2.2 · p. 13

A.2.3 Preprocessing Details for Social Chemistry Actions Dataset

The Social Chemistry Actions dataset is filtered to retain single-action situations with unanimous moral-foundation and acceptability annotations, yielding balanced examples for five foundations.

A.2.3 Preprocessing Details for Social Chemistry Actions Dataset

  • Social Chemistry 101 situations are filtered to those unanimously annotated as containing one action, simplifying interpretation of their moral-foundation annotations.RoT breakdowns parse judgments into actions and approval levels; “expected/OK” actions are discarded, while bad and very bad or good and very good are collapsed.Source: §A.2.3 · p. 13
  • Only morality-ethics Rules-of-Thumb are retained, excluding advice and etiquette categories that do not indicate moral reasoning.Actions must also have unanimous ratings for moral foundation and normative acceptability.Source: §A.2.3 · p. 13
  • The resulting dataset contains 1,300 actions: 130 normatively moral and 130 normatively immoral actions for each of five moral foundations.The actions carry binary normative-acceptability labels and moral-foundation labels.Source: §A.2.3 · p. 13

A.2.4 Preprocessing Details for Social Chemistry Situations Dataset

The Social Chemistry Situations corpus contains 170 multiply annotated scenarios, while prompt templates differ for longer situations and brief action descriptions.

A.2.4 Preprocessing Details for Social Chemistry Situations Dataset

  • Social Chemistry Situations uses 170 scenarios with at least four Rules-of-Thumb and four breakdowns per Rule-of-Thumb to estimate human foundation use and consensus judgment.The corpus is not foundationally balanced because balancing it would have further reduced its size.Source: §A.2.4 · p. 13
  • The situation corpus trades foundation balance for more multiply annotated scenarios; balancing foundations would have reduced the corpus further.The scenarios support the Criterion B comparison between human- and LLM-written moral reasoning.Source: §A.2.4 · p. 13

A.4 Additional Details Related to Prompt Construction

  • Longer scenarios use Table 2 templates, while brief Social Chemistry Actions fragments use Table 3 templates to ensure grammaticality.Five prompt-template styles were used to check sensitivity to phrasing.Source: §A.4 · pp. 13–14, Table 2 · p. 14, Table 3 · p. 14

B.1 Effect Size vs. Dataset

Across datasets, liberal-versus-conservative effect directions are consistent for Care/Harm and Authority/Subversion; across prompt styles, results vary more for other foundations.

B.1 Effect Size vs. Dataset

  • Care/Harm and Authority/Subversion effect sizes and directions are consistent across Moral Stories and ETHICS datasets.Figure 6 presents effects by dataset and dictionary.Source: Figure 6 · p. 14, §B.1 · p. 14

B.2 Effect Size vs. Prompt Template Style

  • Across five prompt styles, liberal identity consistently increases Care/Harm words and conservative identity increases Authority/Subversion words, regardless of dictionary.Liberal prompting increases Fairness/Cheating use in 13 of 15 style-dictionary combinations, but decreases it for styles 1 and 2 under MFDv2.Source: §B.2 · p. 14, Figure 7 · p. 15
  • Sanctity/Degradation directions vary by dictionary: MFDv2 shows lower liberal-prompt use across styles, whereas MFDv1 or eMFD show decreases in fewer styles.With eMFD, liberal prompting decreases Sanctity/Degradation use in four of five styles.Source: §B.2 · p. 15
  • Loyalty/Betrayal directions also depend on dictionary: MFDv1 shows higher scores under liberal prompting for every style, while eMFD shows decreased or equal use.Figure 8 contains randomly selected Moral Stories completions from OpenAI models of increasing size.Source: §B.2 · p. 15, Figure 8 · p. 16
Loading 2209.12106v2…