Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do

Theories of Morality

• The paper treats normativity as behavior conforming to expected societal norms and contracts, while noting that norms can be implicit and difficult to measure consistently [25].It also describes moral norms as partly explicit and partly implicitly established.Source: Theories of Morality · p. 3

Contextual Influence in Human Moral Judgements: A User Study

• Context significantly changed participants’ moral judgments of actions, showing that normativity depends on context rather than the verb-based action alone.The Wilcoxon signed-rank test reported T = 2278, Z = −7.114, p < 0.001, and r = 1.34.Source: Contextual Influence in Human Moral… · p. 5

Moral Norms Contained in Pre-trained Language Models

• The authors define the MoralDirection (MD) as a direction in a pre-trained language model’s embedding space, used to rate the normativity of textual phrases encoding actions and assess moral acceptability.They describe moral norms as expressions of individual or shared values [39].Source: Moral Norms Contained in Pre-trained… · p. 4, Identifying the Moral Direction of… · p. 5

Identifying the Moral Direction of Language Models

• The authors identify the moral direction by applying PCA to sentence embeddings of verb-based actions phrased as questions such as “Should I lie?”.They use multiple question templates and mean sentence embeddings; after identifying the direction, they can score arbitrary phrases.Source: Identifying the Moral Direction of… · p. 5
• PCA’s first component explained 25.64% of variance in the selected action vectors, yet the authors report that it captures essential information for rating phrase normativity.Other top components did not correlate well with information about normative and non-normative actions.Source: Identifying the Moral Direction of… · p. 6
• Using USE [41], the authors found no clear single moral direction but multiple ones (1-PC explained 12.11% of variance and 2-PC 7.86%); BERT had a more prominent moral direction.Source: Identifying the Moral Direction of… · p. 6
• BERT’s MoralDirection may reflect English-speaking cultures represented in its pre-training data, and its scores can exhibit gender bias and context-insensitive judgments.The authors note uncertainty around actions such as “having a gun” and “divorce my wife/husband”; BERT’s slight score difference for “marry my girlfriend” versus “boyfriend” motivates future study of social and demographic biases.Source: Identifying the Moral Direction of… · p. 6

BERT’s MoralDirection Strongly Correlates with Human Moral Norms

• BERT’s masked-language-model generations often use moral or evaluative terms, suggesting that it encodes human-like moral values.The authors queried “[ACTION] [CONTEXT] is a [MASK] behaviour”; generated terms included politeness, criminality, and good or bad behaviour.Source: BERT’s MoralDirection Strongly… · p. 6
• r=0.78 and r=0.79 were the significant strong correlations between BERT’s MoralDirection scores and human moral judgments in the regional and global studies.The global AMT study included 234 participants; the authors characterize the findings as empirical validity because human and model scores depend on participants’ backgrounds and training corpora.Source: BERT’s MoralDirection Strongly… · p. 7
• Randomly sampled verb sets yielded no or only moderate correlation with human scores, supporting the significance of the few-shot verb samples used to derive the direction.The random sets’ first principal component explained a mean variance of 14.73% (standard deviation 0.58).Source: BERT’s MoralDirection Strongly… · p. 7

Reducing Neural Toxic Degeneration in Language Models

• 84.16% accuracy on 966 test pairs showed that MoralDirection could select the preferable commonsense-morality statement without exposure to the training set.The pairs included statements whose morality changed with context or justifications.Source: Reducing Neural Toxic Degeneration in… · p. 9
• The approach uses BERT’s MoralDirection to score candidate continuations and filter non-normative options while retaining at least five tokens.It applies an additional filtering step after top-k and top-p filtering, ranking candidate next tokens by the score of the extended sequence.Source: Reducing Neural Toxic Degeneration in… · pp. 9–10
• The MD approach kept expected maximum toxicity at 0.67 through 500 generations, while the five baseline models exceeded 0.5 within 100 generations and 0.9 by 1,000.A higher threshold reduced expected maximum toxicity more strongly; t=−0.5 could still allow toxic generation.Source: Reducing Neural Toxic Degeneration in… · p. 10
• BERT’s MoralDirection outperformed existing methods on average maximum toxicity and the probability of toxic text in both unconditioned and conditioned generation.Other methods, including PPLM and DAPT, also significantly reduced the probability of toxic text.Source: Reducing Neural Toxic Degeneration in… · p. 10
• The testbed includes prompts that consistently cause all models and approaches to generate toxicity at least once, at toxicity 0.9.Source: Reducing Neural Toxic Degeneration in… · p. 10
• Toxic generations above 0.9 occurred for 2.62% of prompts with DAPT, 2.63% with PPLM, and 1.17% with MD (t=0), versus 9.82% with GPT-2 and 11.99% with GPT-3.The comparison measures prompts producing at least one toxic generation across multiple generations.Source: Table 3 · p. 11, Reducing Neural Toxic Degeneration in… · p. 11
• The proposed approach retrieves retained moral knowledge from a large-scale LM and can reduce toxic degeneration in other language models.Source: Reducing Neural Toxic Degeneration in… · p. 11

Conclusions

• The study reports that modern language models retain knowledge of deontological choices, moral norms, and values.The authors describe this as the first demonstration of human-like right-and-wrong biases in modern language models.Source: Conclusions · p. 11
• The quality of the norms and their scores depends on the language model and the data used to train it.Source: Conclusions · p. 11
• BERT’s mirrored norms agree well with human judgments, and MoralDirection can guide text generation toward normative output while preventing toxic degeneration.The approach is also described as independent of the generative language model’s representation and designed in a few-shot fashion.Source: Conclusions · p. 11
• The authors propose interactive exploration and adaptation of learned language representations, while noting that the approach also relies on an unsupervised-trained language model.Source: Conclusions · p. 11
• Future work could develop language models that reason about social norms [46], using explanatory interactive learning [47] [49] and integrating transformers with moral-reasoning calculi.Source: Conclusions · p. 11
• The authors also call for investigation across other languages and cultural spaces and of the universalization logic underlying models’ moral judgments.Source: Conclusions · p. 11

Details on participant recruitment and study procedure

• Researchers conducted two user studies—one in a controlled setting at Technical University Darmstadt and one using Amazon Mechanical Turk; of 282 AMT volunteers, 234 remained after response-quality exclusions, with Africa and Oceania underrepresented.Participants were excluded for answering control questions incorrectly or giving the same answer to most questions.Source: Methods · pp. 12–13

Statistical analysis of the user study

• Because the regional-study scores violated normality, the authors used Wilcoxon’s signed-rank test, which found the difference between ACI and AA significantly above zero (T = 2278, Z = −7.114, p<0.001).The normality check was Shapiro-Wilk (W = 0.729, p<0.001); the analysis used a 5% significance level.Source: Methods · p. 13

Generating (Moral) Values with LAMA

• LAMA [13] probes moral values by asking a pretrained model to predict the masked word in a sentence such as “Helping a thief is a [MASK] behaviour.”The framework returns the top-k candidate words for the masked position.Source: Methods · p. 14

Asking the Language Model for its Moral Score

• The question-answering system compares a question with two answer choices by cosine similarity; it combines several question-answer prompts into an overall averaged score and restricts inputs to question-answer pairs.Both approaches were evaluated on normative and non-normative actions.Source: Methods · p. 14

The MoralDirection of language models

• PCA over BERT sentence embeddings of 54 positive, negative, and neutral actions defines a moral direction that also reflects context-dependent judgments.The actions are placed in the same question templates used by the question-answering approach; examples include trusting humans versus strangers and killing time versus people.Source: Methods · p. 15

Testbed for evaluating the toxicity of text generations

• RealToxicityPrompts [15] evaluates GPT-2 on unconditional and prompted generation, with outputs scored by Perspective API and toxicity assessed using maximum-toxicity and toxicity-probability metrics.The testbed labels scores ≥0.5 as toxic and cautions that automated toxicity tools can be imperfect and biased.Source: Methods · p. 16

Guiding Generative Language Models using the MoralDirection

• For MoralDirection-guided generation, the authors use GPT-2 with top-k and top-p sampling, and BERT-large with SBERT pooling to obtain sentence embeddings.The described MoralDirection moral score is normalized.Source: Methods · p. 16

Generation method

• During generation, MoralDirection rates each phrase extended by candidate next tokens and removes candidates whose ratings fall below a predefined threshold.Filtering is applied only to candidates already selected by top-k and top-p sampling; top-k was increased relative to the GPT-2 setup [15] to allow more choices before filtering.Source: Methods · p. 16

GPT-3

• 86.48% accuracy versus a 53.98% random baseline showed GPT-3’s API responses aligned with binarized human-study scores on moral questions.With limited API access, the authors prompted “Should I kill?” and “Should I love?” and used the responses as few-shot examples; internal representations were unavailable for extracting a direction.Source: Methods · pp. 16–17

Differences between the MoralDirection approach and related methods

• Unlike data-based detoxification, MoralDirection is a decoding-time add-on that does not require adapting the generative model’s parameters.DAPT [23] requires access to model parameters and additional pre-training on non-toxic data; PPLM [24] uses a discriminator that must be trained for each generative model.Source: Methods · p. 17

Limitations

• The moral direction is not insulated from social biases in training data, and the global user study was limited by the crowdsourcing platform’s user base.The authors explicitly caution that the method does not establish that language models understand morality or provide a basis for actual advice.Source: Methods · p. 17
• The study mainly investigates BERT-large and leaves transfer of MoralDirection to other masked or autoregressive models as future work.The authors also suggest exploring hyperbolic probing and PCA for hyperbolic spaces because the Euclidean assumption has no strict evidential basis.Source: Methods · p. 17
• MoralDirection reduced toxic degeneration and outperformed DAPT and PPLM, though the BERT-based ratings can inherit biases from its training text.The authors also question whether the toxicity rating system itself is unbiased and note that BERT sometimes struggles with semantics such as negation.Source: Methods · pp. 17–18

Data availability

• User study data is available in the paper’s code repository.The passage provides the repository URL: https://github.com/ml-research/MoRT_NMI/tree/master/.Source: Data availability · p. 18
• Generated text using the presented approach is reported as available in Supplemental_Material/UserStudy.The passage’s URL is incomplete, ending at “https:”.Source: Data availability · p. 18

• The authors confirm compliance with all relevant ethical regulations.Source: Statement of ethical compliance · p. 18
• Informed consent was obtained from each participant before the user study began.Source: Statement of ethical compliance · p. 18

Corresponding author

• Correspondence is directed to Patrick Schramowski and Cigdem Turan.Source: Corresponding author · p. 19

A User Study Material

• The user-study material is available in the supplement’s repository, specifically in the StimuliMoral.txt file.The passages point to the UserStudy material directory.Source: A User Study Material · p. 23

B.1 Question Template

• To compute PCA, the authors prompted actions with the same question templates used in the question-answering approach to amplify transformation into the moral subspace.Table 4 presents the question/answer prompts introduced by.Source: §B.1 · p. 23, Table 4 · p. 23

B.2 Atomic Actions to compute PCA.

• The selected actions include positive social verbs such as “help,” “love,” and “thank,” alongside harmful verbs such as “harm,” “murder,” and “bully.”The list also includes other actions, including “eat,” “travel,” and “marry.”Source: §B.2 · p. 23
• For the random-set check, researchers sampled 64 verbs from the 1,000 most common English verbs, embedded them in “Should I <VERB> ?” templates, and sampled three times (seeds 0, 1, 2).They computed sentence embeddings, then PCA, and correlated PC1 with human scores. The selected-verb set had a significant strong correlation (r = 0.78***).Source: §B.2 · pp. 23–24
• Random verb sets correlated with human scores at rset1 = 0.64***, rset2 = 0.60***, and rset3 = −0.01, versus r = 0.78*** for the original selection.The paper describes the first two random-set correlations as moderate and the third as no correlation.Source: §B.2 · p. 24
• Random-set PCA variances were similar across PC1–PC5, but PC1 variance was much lower than the original selection’s 25.64%.Table 5 presents PCA variance explained for the random verb sets.Source: Table 5 · p. 24, §B.2 · p. 24
• Non-overlapping Do/Don’t verbs yielded top-PC correlation r=0.92*** with the original direction, and overlapping verbs yielded r=0.95***.The alternative verbs were selected in equal number to the original set.Source: §B.2 · p. 24

B.3 Atomic actions and additional context.

• Table 6 samples are used to create Figure 1, which shows generalisation of BERT’s MoralDirection.Source: §B.3 · p. 24

B.4 PCA Variance explained.

• 25.64%: BERT’s top principal component explains most embedding variance; PC2 and PC3 show no human-score correlation, with reported coefficients 0.03 and −0.01.Source: §B.4 · p. 24
• Using USE, the analysis found multiple possible moral directions rather than one clear direction.Source: §B.4 · p. 24

B.5 MoralDirection correlation with human moral norms.

• The human-norm comparison uses verb-based actions paired with varied contexts, which Table 6 describes as a test of generalization.Source: Table 6 · p. 25, §B.5 · p. 25
• The comparison includes USE [41], sentence-BERT [40], averaged GloVe [?], question-answering [26], and MoralDirection.Source: §B.5 · p. 24, Figure 6 · p. 26

B.5 Querying the MoralDirection with averaged question embeddings, questions and raw actions.

• The query comparison evaluates MoralDirection on study phrases, averaged question-template embeddings, and raw actions.Source: Figure 7 · p. 26, §B.5 · p. 26

B.5 MoralDirection correlation with human moral norms.

• r = 0.78: BERT combined with MoralDirection strongly correlates with human moral scores.Source: §B.5 · p. 26, Figure 6 · p. 26

D Toxic Degeneration Testbed - RealToxcitityPrompts

• The additional toxic-degeneration samples use prompts selected by Gehman et al. [15] and show GPT-2/3 and Moral Direction (MD) generations.Tables 8 and 9 include the lowest- and highest-toxicity generations from multiple generations.Source: D Toxic Degeneration Testbed -… · p. 26, Table 8 · p. 28, Table 9 · p. 29
• Table 8 and 9 also show MD generations under different threshold choices alongside GPT-2/3 outputs.The captions specify lowest- and highest-toxicity generations out of multiple generations.Source: D Toxic Degeneration Testbed -… · p. 26, Table 8 · p. 28, Table 9 · p. 29

C LAnguage Model Analysis (LAMA)

• Table 7 presents examples of moral norms and values retained by BERT, with the top five tokens generated for each using LAMA mask filling and log probabilities.The paper identifies it as the full table of questions from the user study.Source: C LAnguage Model Analysis (LAMA) · p. 26, Table 7 · p. 27

D Toxic Degeneration Testbed - RealToxcitityPrompts

• Table 10 compares MD with PPLM and DAPT on the same samples from [15], including lowest- and highest-toxicity generations.The passage notes that † marks results recomputed from data provided by [15].Source: D Toxic Degeneration Testbed -… · p. 26, Table 10 · p. 30

A User Study Material

• The user-study stimuli are provided in the supplemental UserStudy material, specifically the StimuliMoral.txt file.The passage locates the material in the paper’s linked repository.Source: A User Study Material · p. 31

B.1 Question Template.

• For PCA, the authors prompted actions using the same question templates as the question-answering approach to amplify transformation into the moral subspace.Table 1 is identified as containing question-and-answer prompts for that approach.Source: §B.1 · p. 31, Table 1 · p. 31

B.2 Atomic Actions to compute PCA.

• The selected PCA verbs include prosocial verbs such as “smile” and “help,” harmful verbs such as “harm” and “murder,” and others including “eat” and “travel.”Source: §B.2 · p. 31
• For the random-set test, the authors sampled 64 verbs from the 1,000 most common English verbs, embedded them in the same question template, and repeated sampling with three seeds.The template was “Should I <VERB> ?” and the seeds were 0, 1, and 2.Source: §B.2 · p. 31
• rset1 = 0.64∗∗∗, rset2 = 0.60∗∗∗, and rset3 = −0.01, versus r = 0.78∗∗∗ for the selected verbs.The first two random sets had moderate correlations with human scores, while the third had no correlation.Source: §B.2 · p. 32
• Random sets had similar PC1–PC5 variance patterns, but PC1 variance was much lower than the selected set’s 25.64%.Table 2 presents the variance explained using the random verb sets.Source: §B.2 · p. 32, Table 2 · p. 32
• r = 0.92∗∗∗ for non-overlapping alternative Do/Don’t verbs and r = 0.95∗∗∗ for overlapping verbs support robustness of the PCA direction.The alternative sets were selected to represent (non-)normativity and compared through their top principal components.Source: §B.2 · p. 32

B.3 Atomic actions and additional context.

• Samples in Table 3 were used to create Figure 1, which shows the generalization of BERT’s MoralDirection.Source: §B.3 · p. 32

B.4 PCA Variance explained.

• BERT’s top principal component explains 25.64% of embedding variance, while PC2 and PC3 show no correlation with human scores (r = 0.03 and r = −0.01).The authors conclude that the top component represents the moral direction.Source: §B.4 · p. 32
• Using USE, the authors found multiple moral directions rather than one clear direction.The passage points to Figure 1 for this result.Source: §B.4 · p. 32, Figure 1 · p. 33

B.5 MoralDirection correlation with human moral norms.

• The correlation study uses BERT-large as its main focus and compares it with other popular language models, including USE and averaged GloVe embeddings.The action set includes dos, don’ts, and neutral actions, with additional contexts used to test generalization.Source: §B.5 · p. 32, Table 3 · p. 33
• The correlation comparison covers GloVe, USE, BERT, question-answering, and MoralDirection embeddings or approaches.The figure caption says the r-value indicates correlation and asterisks indicate significance; the text reports the BERT–MoralDirection result separately.Source: §B.5 · pp. 32, 34, Figure 2 · p. 34
• BERT combined with MoralDirection correlates strongly with human scores (r = 0.78).The comparison also reports that the BERT-based sentence model underlying the question-answering system leads to a higher correlation.Source: §B.5 · p. 34, Figure 2 · p. 34

B.5 Querying the MoralDirection with averaged question embeddings, questions and raw actions.

• For the query-method evaluation, the authors compute the moral subspace from averaged BERT question embeddings using atomic actions, then compare querying with study phrases, averaged prompted actions, and raw actions.Figure 3 presents the three query strategies’ correlations with human scores, but the supplied passages do not provide their numerical results.Source: §B.5 · p. 34, Figure 3 · p. 34

• Table 4 contains all user-study questions used to extract BERT’s moral knowledge with LAMA.Source: C LAnguage Model Analysis (LAMA) · p. 34

D Toxic Degeneration Testbed - RealToxcitityPrompts

• The additional prompted-generation examples use samples selected by Gehman et al. [?].Tables 5 and 6 display GPT-3 and GPT-2 generations alongside MD generations.Source: D Toxic Degeneration Testbed -… · p. 34
• Tables 5–6 compare GPT-2/3 generations with MD generations across threshold choices, showing the lowest- and highest-toxicity outputs among multiple generations.The comparisons use the same samples provided in [?].Source: D Toxic Degeneration Testbed -… · p. 34, Table 5 · p. 36, Table 6 · p. 37
• Table 7 compares PPLM, DAPT, and MD approaches for preventing toxic degeneration, reporting lowest- and highest-toxicity generations across multiple outputs.The results are based on the same samples provided in [?]; † marks results recomputed from data provided in [?].Source: D Toxic Degeneration Testbed -… · p. 34, Table 7 · p. 38