PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do

Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A. Rothkopf, Kristian Kersting

arXiv:2103.11790v3cs.CLcs.CY

TL;DR

Large pre-trained language models may encode human-like judgments about right and wrong, but whether such moral norms surface in their representations is unclear. The paper extracts a moral direction from language-model embeddings and shows that it aligns with human judgments and can guide generation toward normative text while preventing toxic degeneration.

  • Problem

    The paper asks whether large pre-trained language models trained on human text reflect society’s moral norms and deontological judgments.

  • Method

    The paper extracts a MoralDirection from language-model embeddings and uses it as a few-shot, model-independent compass for normativity in text generation.

  • Results

    The extracted norms agree well with human judgments, and the MoralDirection guides models toward normative text while preventing toxic degeneration.

  • Takeaways & Limitations

    Language-model embeddings retain knowledge about deontological choices and moral norms that can be used to steer generation toward normative text.

  • Takeaways & Limitations

    The approach is shown to extract moral scores for atomic actions, but its generalisation to moral actions with additional context has not been investigated.

Abstract

from arXiv · show

Artificial writing is permeating our lives due to recent advances in large-scale, transformer-based language models (LMs) such as BERT, its variants, GPT-2/3, and others. Using them as pre-trained models and fine-tuning them for specific tasks, researchers have extended state of the art for many NLP tasks and shown that they capture not only linguistic knowledge but also retain general knowledge implicitly present in the data. Unfortunately, LMs trained on unfiltered text corpora suffer from degenerated and biased behaviour. While this is well established, we show that recent LMs also contain human-like biases of what is right and wrong to do, some form of ethical and moral norms of the society -- they bring a "moral direction" to surface. That is, we show that these norms can be captured geometrically by a direction, which can be computed, e.g., by a PCA, in the embedding space, reflecting well the agreement of phrases to social norms implicitly expressed in the training texts and providing a path for attenuating or even preventing toxic degeneration in LMs. Being able to rate the (non-)normativity of arbitrary phrases without explicitly training the LM for this task, we demonstrate the capabilities of the "moral direction" for guiding (even other) LMs towards producing normative text and showcase it on RealToxicityPrompts testbed, preventing the neural toxic degeneration in GPT-2.

Pre-trained Language Models, and the Sense of Right and Wrong

Pre-trained language models encode human-like moral norms from their training texts, which can be represented as a MoralDirection for scoring phrase normativity. BERT’s direction aligns with human judgments, ranks moral choices, and can guide generation away from toxic degeneration, while reflecting cultural and gender biases.

  • Context sensitivity: Context significantly changed moral judgments, as shown by Wilcoxon’s signed-rank test with T =2278, Z =−7.114, p<0.001, α=0.05, r=1.34.Moral norms were therefore not judged exclusively from the verb-based action.
  • MoralDirection: A MoralDirection in the embedding space provides a computable score for the normativity and moral bias of arbitrary phrases.The method assesses moral acceptability without explicitly extracting moral norms or training the language model for this task.
  • MoralDirection: 25.64% of vector variance was explained by the first principal component, whose direction carried the essential information for rating phrase normativity.Other top principal components did not correlate well with normative-action information.
  • Limitations: The encoded moral norms reflect human-like judgments but may primarily mirror English-speaking 21st-century cultures and exhibit gender bias.The authors note that some decisions are disputable without considering the overall behavioral context.
  • Validation: BERT’s MoralDirection significantly correlated with human moral scores, with r=0.78 resp. r=0.79.The direction aligned with user-study judgments, although positive and negative actions remained difficult to predict.
  • Guiding generation: BERT’s MoralDirection selected preferable moral choices with 84.16% accuracy without exposure to the train set.This capability supports guiding generative language models toward normative text and preventing toxic degeneration.
  • Guiding generation: The MoralDirection kept expected maximum toxicity lower than baseline approaches, reaching a maximum toxicity of 0.67.All five compared language models exceeded toxicity 0.5 within 100 generations, while DAPT exceeded 0.9 within 1,000 generations; thresholds t=0.0 and t = 0.5 reduced toxicity further.

Conclusions

The results show that modern language models retain human-like knowledge of deontological choices, moral norms, and values, with quality depending on the model and training data. MoralDirection aligns with human judgments and can guide generation toward normative text while reducing toxic degeneration, motivating interactive, explanatory, and neuro-symbolic extensions.

  • Conclusions: Modern language models retain knowledge of deontological choices, moral norms, and values, demonstrating human-like biases about right and wrong.The quality of this moral information depends on the language model and the data used to train it.
  • Conclusions: Using BERT, the study found that norms implicitly expressed in training texts agree well with human judgments.These mirrored norms provide a basis for assessing normativity without explicitly training the language model for the task.
  • Conclusions: MoralDirection acts as a compass for text-generation normativity, preventing toxic degeneration and guiding language models to generate normative text.The approach is few-shot and does not depend on the given language model’s representation.
  • Future work: Future work includes interactive systems for exploring and adapting learned language representations, including toxicity-related behavior.The authors also propose language models that reason about social norms and explanatory interactive learning that supports explanations and revision.
  • Future work: The authors propose integrating transformers with moral-reasoning calculi to develop a neuro-symbolic moral approach and investigating other languages and cultural spaces.They also identify the universalization logic underlying language models and its role in moral judgment as a research direction.

Methods

The methods combine embedding-space representations, human moral judgments, and masked-language-model probing to extract and evaluate moral norms. A novel sampling-time approach extends moral scoring beyond isolated question–answer pairs toward contextual actions and LM generation.

  • Embeddings: Word and sentence embeddings represent text as points in vector space, where proximity captures semantic similarity and supports mathematical vector arithmetic.The methods build on neural embedding approaches including Word2Vec and sentence extensions.
  • Moral probing: Masked-language-model probing used closed sentences such as “Helping a thief is a [MASK] behaviour.” to predict moral-value words for examined actions.This LAMA-based procedure defines moral-value analysis through masked-object prediction.
  • Moral scoring: Because prior scoring handled atomic actions and restricted inputs to question–answer pairs, the authors introduced a novel approach evaluated on normative and non-normative actions with additional context.The new approach addresses limitations in generalizing moral-action analysis beyond isolated question–answer inputs.
  • Generation control: The proposed decoding method is decoupled from the generative LM and plugged only into sampling, while still depending on that model’s learned representation.This design means the method is not directly affected by biases learned by the generative language model, although downstream MoralDirection remains affected by BERT’s encoded biases.

Data availability

The user study data and generated text from the presented approach are available through the project’s code repository and supplemental materials.

  • Data availability: User study data is available in the project’s code repository.Repository: https://github.com/ml-research/MoRT_NMI/tree/master/.
  • Data availability: Generated text using the presented approach is available in the supplemental materials.The materials are listed under Supplemental_Material/UserStudy.

Statement of ethical compliance

The authors state that they complied with relevant ethical regulations and obtained informed consent from every user-study participant before participation.

  • The authors confirm compliance with all relevant ethical regulations.
  • Informed consent was obtained from each participant before the user study began.

Author information

The paper lists Patrick Schramowski and Cigdem Turan as correspondence contacts.

  • Author information: Correspondence is directed to Patrick Schramowski and Cigdem Turan.

Supplementary Information for … B.1 Question Template.

The supplementary material identifies the paper and provides user-study materials, while describing question templates used to compute the PCA-based moral direction. These templates follow a prior question-answering approach and are intended to amplify transformation into the moral subspace.

  • Supplementary Information for: The supplementary information is associated with the paper “Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do.”
  • A User Study Material: User-study materials are available in the paper’s GitHub repository.
  • A User Study Material: The relevant user-study stimuli are contained in the StimuliMoral.txt file.
  • B The MoralDirection approach: The MoralDirection approach computes the PCA by prompting actions into shared question templates.
  • B.1 Question Template.: The question templates come from the question-answering approach introduced by.
  • B.1 Question Template.: The prompts are designed to amplify the transformation into the moral subspace.
  • B.1 Question Template.: Table 4 presents the question/answer prompts used in the question-answering approach.

B.2 Atomic Actions to compute PCA. · B.3 Atomic actions and additional context. · B.4 PCA Variance explained.

The paper constructs MoralDirection with selected atomic actions, tests its robustness against random and alternative Do/Don’t verb sets, and examines how context and embedding-model choice affect PCA-based moral structure. BERT yields a clear moral direction, whereas USE does not reveal a single one.

  • B.2 Atomic Actions to compute PCA.: MoralDirection is computed by embedding selected atomic actions and applying PCA to transform language-model embeddings into a moral subspace.The atomic actions include normative verbs such as “smile” and “help” alongside harmful or non-normative verbs such as “murder” and “assault.”
  • B.2 Atomic Actions to compute PCA.: The robustness test samples 64 random verbs from the 1000 most common English verbs, embeds them in the same question template, and repeats sampling with seeds [0, 1, 2].The random sets are compared by correlating their first principal component with human scores.
  • B.2 Atomic Actions to compute PCA.: rset1 = 0.64∗∗∗, rset2 = 0.60∗∗∗, and rset3 = −0.01, compared with r = 0.78∗∗∗ for the original verb selection.The first two random sets show moderate correlation, while the third shows no correlation.
  • B.2 Atomic Actions to compute PCA.: 25.64% is the variance explained by PC1 for the original verb selection, whereas random sets have similar PC1–PC5 variance patterns but much lower PC1 variance.The cited passage reports the original PC1 variance as 25.64%.
  • B.2 Atomic Actions to compute PCA.: r = 0.92∗∗∗ for a non-overlapping alternative Do/Don’t verb set and r = 0.95∗∗∗ for an overlapping set, supporting robustness of the identified direction.The alternative verbs come from sets of 100 Dos and 100 Don’ts provided by Jentzsch et al.
  • B.3 Atomic actions and additional context.: The samples in Table 6 are used to create Figure 1, which shows the generalisation of BERT’s MoralDirection.This connects the additional-context samples to the manuscript’s visualization.
  • B.4 PCA Variance explained.: 25.64% of BERT sentence-embedding variance is explained by PC1, while PC2 and PC3 show no correlation with human scores: rP C2 = 0.03 and rP C2 = −0.01.The authors therefore conclude that PC1 represents the moral direction m.
  • B.4 PCA Variance explained.: Using USE, the authors could not find a clear single moral direction, instead observing multiple directions.This contrasts with the clear single direction identified using BERT.

B.5 MoralDirection correlation with human moral norms. · B.5 Querying the MoralDirection with averaged question embeddings, questions and raw actions.

The study compares MoralDirection’s alignment with human moral norms across several language-model embeddings and finds the strongest correlation with BERT. It also evaluates three ways to query MoralDirection: averaged question embeddings, questions, and raw actions.

  • B.5 MoralDirection correlation with human moral norms.: The comparison includes Universal Sentence Encoder, SentenceBERT, and averaged GloVe embeddings, alongside BERT-based approaches.The models are evaluated against human scores for Dos and Don’ts actions.
  • B.5 MoralDirection correlation with human moral norms.: 61.32%, 71.22%, and 76.55% are the reported average textual-similarity scores for GloVe, USE, and SentenceBERT, respectively.These scores motivate comparing the models’ moral-score correlations.
  • B.5 MoralDirection correlation with human moral norms.: GloVe produces a weak correlation, making positive and negative actions difficult to predict from its moral scores.The results are statistically significant, but the plotted separation remains weak.
  • B.5 MoralDirection correlation with human moral norms.: USE yields a significant positive correlation, making moral-score distinctions more feasible, although complex actions remain insufficiently correlated with human judgments.The human scoring of more complex actions is still not strongly correlated to this moral score.
  • B.5 MoralDirection correlation with human moral norms.: r = 0.78 is achieved by combining BERT with the proposed MoralDirection approach, producing a strong correlation with human scores.Using BERT as the underlying model of the question-answering system leads to a higher correlation.
  • B.5 Querying the MoralDirection with averaged question embeddings, questions and raw actions.: The moral subspace is computed with sentence-BERT embeddings averaged over the listed atomic actions.The approach uses the sentence variant of BERT for constructing the moral subspace.
  • B.5 Querying the MoralDirection with averaged question embeddings, questions and raw actions.: Because MoralDirection accepts arbitrary phrases, the evaluation compares averaged question embeddings, complete questions, and raw actions as query forms.These three query strategies are assessed through correlation with human moral scores.

C LAnguage Model Analysis (LAMA) … B.1 Question Template.

The supplementary material documents BERT’s LAMA-based extraction of moral knowledge, additional GPT-2/3 toxic-degeneration examples, user-study materials, and the question templates used to compute the moral subspace.

  • C LAnguage Model Analysis (LAMA): The supplement provides the full user-study table for extracting moral knowledge from BERT with the LAMA framework.Table 7 reports examples of moral norms and values retained in language models, including BERT’s top five generated tokens for each prompt.
  • D Toxic Degeneration Testbed - RealToxcitityPrompts: Additional examples show GPT-2 and GPT-3 generations for samples selected by Gehman et al., including MoralDirection outputs under different threshold settings.The supplement also compares the MoralDirection, PPLM, and DAPT approaches, with † marking recomputed results based on data from Gehman et al.
  • D Toxic Degeneration Testbed - RealToxcitityPrompts: Tables 8–10 compare GPT-2/3 and toxicity-prevention approaches using the same samples from Gehman et al., reporting the lowest and highest toxicity generations across multiple generations.Table 10 specifically compares PPLM, DAPT, and MoralDirection, while Tables 8 and 9 compare GPT-2/3 with MoralDirection.
  • A User Study Material: The user-study materials are available in the project repository, specifically in the StimuliMoral.txt file.The provided supplement identifies the repository path for the study material and the stimuli file.
  • B The MoralDirection approach: To compute the PCA, the approach prompts actions using the same question templates as the question-answering method.This procedure is intended to amplify the transformation into the moral subspace.
  • B.1 Question Template.: Table 1 lists the question-and-answer prompts used by the question-answering approach.These templates are the basis for prompting actions when computing the PCA.

B.2 Atomic Actions to compute PCA. · B.3 Atomic actions and additional context.

The moral subspace is computed from selected Do and Don’t atomic actions, with validation showing that its alignment with human norm scores depends on the verb set. Alternative normativity verb sets nevertheless produce highly correlated principal components, and the resulting direction generalises in BERT.

  • B.2 Atomic Actions to compute PCA.: The transformation from the language model’s embedding space to the moral subspace is calculated from atomic actions.The selected actions include normative verbs such as “smile” and “help” and non-normative verbs such as “harm” and “murder.”
  • B.2 Atomic Actions to compute PCA.: The selection was tested using three random sets of 64 verbs sampled from the 1000 most common English verbs and embedded in the same question template.The random sampling used seeds = [0, 1, 2] before sentence embeddings and PCA were computed.
  • B.2 Atomic Actions to compute PCA.: 0.64***, 0.60***, and −0.01 were the Pearson correlations for random verb sets 1, 2, and 3, compared with r = 0.78*** for the original selection.The first two random sets showed moderate correlation, while the third showed no correlation.
  • B.2 Atomic Actions to compute PCA.: 25.64% was the PC1 variance for the original verb selection, while random sets had much lower PC1 variance despite similar PCA variance across PC1–PC5.The passage states that the PCA variance profiles for all random sets were very similar overall, but their PC1 variance was much lower than the original selection.
  • B.2 Atomic Actions to compute PCA.: Using a non-overlapping set of Dos and Don’ts from Jentzsch et al. with the same number of verbs produced a significant strong correlation of r = 0.92*** between the top PCs.This evaluates whether the identified direction is robust to a different normativity verb selection.
  • B.2 Atomic Actions to compute PCA.: Using an overlapping set of Dos and Don’ts produced a significant strong correlation of r = 0.95*** between the top PCs.The overlapping-set result further supports robustness of the identified direction.
  • B.3 Atomic actions and additional context.: The samples in Table 3 were used to create Figure 1, which shows the generalisation of BERT’s MoralDirection.This connects the atomic-action samples and additional context to the reported BERT generalisation figure.

B.4 PCA Variance explained.

For BERT sentence embeddings, the top principal component explains 25.64% of variance and aligns with human moral scores, supporting a single moral direction. USE does not reveal one clear moral direction, but rather multiple directions.

  • B.4 PCA Variance explained.: The study uses sentence-embedding variants of BERT and the Universal Sentence Encoder (USE) for the MoralDirection approach.These transformer-based language models generate sentence embeddings.
  • B.4 PCA Variance explained.: 25.64%: BERT’s top principal component explains the majority of variance in the sentence-embedding vectors.This concentration of variance motivates interpreting the component as a candidate moral direction.
  • B.4 PCA Variance explained.: USE does not yield a clear single moral direction, instead suggesting multiple moral directions.The result is referenced in relation to Figure 1.

B.5 MoralDirection correlation with human moral norms. · B.5 Querying the MoralDirection with averaged question embeddings, questions and raw actions. · C LAnguage Model Analysis (LAMA)

The MoralDirection extracted with BERT correlates strongly with human judgments of moral norms, while alternative embeddings show weaker or limited predictive alignment. The analysis also compares querying methods and reports a full BERT moral-knowledge table using LAMA.

  • B.5 MoralDirection correlation with human moral norms.: BERT is the primary masked-language model studied because it captures relational, factual, and commonsense knowledge accurately relative to competitors.The comparison includes the Universal Sentence Encoder, a BERT-based sentence-embedding model, and averaged GloVe embeddings.
  • B.5 MoralDirection correlation with human moral norms.: The comparison evaluates GloVe, USE, SentenceBERT, question-answering embeddings, and the proposed MoralDirection approach against human scores.The results are visualized through correlations between extracted language-model moral scores and human scores, with significance indicated in the plots.
  • B.5 MoralDirection correlation with human moral norms.: GloVe produces a weak correlation, making positive and negative actions difficult to predict from its moral scores.The correlation is reported using Pearson’s Correlation Coefficient.
  • B.5 MoralDirection correlation with human moral norms.: USE yields a significant positive correlation, but human judgments of more complex actions remain not strongly correlated with its moral score.Moral-score distinctions become more feasible with USE, although complex-action prediction remains limited.
  • B.5 MoralDirection correlation with human moral norms.: BERT combined with MoralDirection achieves a strong correlation with human moral scores, r = 0.78.This approach improves on the weaker correlation observed with GloVe and the significant positive correlation observed with USE.
  • B.5 Querying the MoralDirection with averaged question embeddings, questions and raw actions.: The MoralDirection correlation evaluation compares averaged question embeddings, questions, and raw actions as alternative query forms.The moral subspace is computed with sentence-variant BERT from averaged question embeddings over the listed atomic actions, while arbitrary phrases can be queried.
  • C LAnguage Model Analysis (LAMA): The LAMA analysis provides the full table of moral knowledge extracted from BERT for all questions in the conducted user study.The table is identified as Table 4 within the LAMA framework.

D Toxic Degeneration Testbed - RealToxcitityPrompts

This section presents additional prompted-generation examples from the RealToxicityPrompts samples and compares GPT-2/3 with moral-direction-based generation for preventing toxic degeneration. It also compares the moral direction with PPLM and DAPT, including different threshold settings.

  • D Toxic Degeneration Testbed - RealToxcitityPrompts: Additional prompted-generation examples use samples selected by Gehman et al.The examples are presented in the section’s supplementary generation analysis.
  • D Toxic Degeneration Testbed - RealToxcitityPrompts: Tables 5 and 6 compare GPT-2/3 generations with the moral-direction approach for preventing toxic degeneration.The tables report the lowest and highest toxicity generations across multiple generations for the same samples.
  • D Toxic Degeneration Testbed - RealToxcitityPrompts: Table 7 compares the moral-direction, PPLM, and DAPT approaches for preventing toxic degeneration.It reports the lowest and highest toxicity generations across multiple generations.
  • D Toxic Degeneration Testbed - RealToxcitityPrompts: The moral-direction generations are evaluated with different parameter choices for the threshold.The section states that the comparison includes the MD approach under different threshold settings.
Loading 2103.11790v3…