PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating the Moral Beliefs Encoded in LLMs

Nino Scherrer, Claudia Shi, Amir Feder, David M. Blei

arXiv:2307.14324v1cs.CLcs.AIcs.CYcs.LG

TL;DR

The paper addresses how to elicit and evaluate moral beliefs encoded in LLMs despite token-based outputs and sensitivity to question wording. It develops statistical measures and applies them to a survey of moral scenarios administered to 28 models. Most models align with commonsense in low-ambiguity cases and are uncertain in high-ambiguity cases, while some show wording-sensitive uncertainty or distinct preferences.

  • Problem

    The paper asks how to analyze LLM choices and encoded moral beliefs when outputs are token sequences and responses vary with question wording.

  • Method

    The paper introduces action likelihood, marginal action likelihood, entropy, QF-C, and QF-E, then applies them in the MoralChoice survey administered to 28 open- and closed-source LLMs.

  • Results

    Most models align with commonsense in low-ambiguity scenarios and express uncertainty in high-ambiguity scenarios, while some models show wording-sensitive uncertainty or clear preferences.

  • Takeaways & Limitations

    Evaluating LLM moral responses requires multiple prompt variations, and some models exhibit distinct preferences even when scenarios lack a clear answer.

  • Takeaways & Limitations

    The survey scenarios lack diversity, focusing on norm violations without fully representing different professions and contexts.

Abstract

from arXiv · show

This paper presents a case study on the design, administration, post-processing, and evaluation of surveys on large language models (LLMs). It comprises two components: (1) A statistical method for eliciting beliefs encoded in LLMs. We introduce statistical measures and evaluation metrics that quantify the probability of an LLM "making a choice", the associated uncertainty, and the consistency of that choice. (2) We apply this method to study what moral beliefs are encoded in different LLMs, especially in ambiguous cases where the right choice is not obvious. We design a large-scale survey comprising 680 high-ambiguity moral scenarios (e.g., "Should I tell a white lie?") and 687 low-ambiguity moral scenarios (e.g., "Should I stop for a pedestrian on the road?"). Each scenario includes a description, two possible actions, and auxiliary labels indicating violated rules (e.g., "do not kill"). We administer the survey to 28 open- and closed-source LLMs. We find that (a) in unambiguous scenarios, most models "choose" actions that align with commonsense. In ambiguous cases, most models express uncertainty. (b) Some models are uncertain about choosing the commonsense action because their responses are sensitive to the question-wording. (c) Some models reflect clear preferences in ambiguous scenarios. Specifically, closed-source models tend to agree with each other.

1 Introduction

The paper develops a survey-based statistical method for eliciting and evaluating moral beliefs encoded in LLMs, then applies it to hypothetical scenarios across different ambiguity levels. It finds that models generally reflect scenario ambiguity, while some show wording sensitivity or distinct preferences.

  • The study treats LLMs as survey respondents to examine moral preferences encoded in models through hypothetical decision scenarios.
  • The survey distinguishes 687 low-ambiguity scenarios, where one action is clearly preferred, from 680 high-ambiguity scenarios, where neither action is clearly preferred.
  • Action likelihood maps token-sequence probabilities into a distribution over semantic actions, while marginal action likelihood aggregates choices across randomly varied question forms.
  • Entropy measures quantify choice uncertainty, while QF-C and QF-E evaluate consistency and uncertainty under variations in question form.
  • The survey is administered to 28 open- and closed-source LLMs, whose responses generally align with commonsense in unambiguous scenarios and show uncertainty in ambiguous ones.
  • The paper contributes a statistical methodology, the MoralChoice dataset, and survey findings about moral beliefs encoded in 28 LLM respondents.

2 Defining and Estimating Beliefs encoded in LLMs

The paper defines action-based measures for eliciting LLM beliefs from token outputs, then extends them to question-form variation and uncertainty. It estimates these quantities by mapping sampled responses to actions and averaging across prompt templates.

  • Action likelihood: Action likelihood maps an LLM’s token-sequence probabilities into a probability distribution over possible actions.The mapping groups semantically equivalent sequences that express the same action preference.
  • Uncertainty: Action entropy measures uncertainty about an action preference rather than uncertainty about individual token outputs.The paper identifies this as a semantic entropy measure over actions.
  • Question-form variation: Marginal action likelihood averages action preferences across syntactically different but semantically equivalent question forms.The question-form distribution p(z) determines how these forms are weighted.
  • Question-form variation: Question-form consistency measures similarity among action likelihoods conditioned on different question forms, using the Generalized Jensen-Shannon divergence.Average question-form-specific action entropy complements this consistency measure by capturing uncertainty across forms.
  • Diagnosing uncertainty: Low consistency and low question-form-specific entropy indicate confident but wording-sensitive beliefs whose interpretation lacks robustness.High marginal entropy can instead reflect inconsistency, inherent ambiguity, or both.
  • Estimation: Monte Carlo estimation samples token sequences, maps them deterministically to actions, and averages estimates across prompt templates.This approach addresses exponentially large continuation spaces and APIs that do not expose token probabilities.

3 The MoralChoice Survey

The MoralChoice survey uses hypothetical scenarios to examine moral preferences encoded in LLMs while avoiding assumptions required by human self-report surveys. It constructs low- and high-ambiguity datasets, controls question-form bias, and evaluates 28 models.

  • Survey design: Hypothetical moral scenarios let the study assess encoded preferences without directly applying human questionnaires that assume respondent agency.The framework also scales to situations where users may seek advice from LLMs.
  • Scenario construction: Scenario generation is grounded in Gert’s common morality framework, with action pairs designed so at least one action violates a rule.Auxiliary rule-violation labels are assigned through majority votes from three annotators.
  • Survey respondents: The survey covers 28 LLM respondents, including 12 open-source and 16 closed-source models from seven companies.Model sizes range from flan-t5-small at 80m parameters to gpt-4 with an unknown parameter count.
  • Administration and processing: Three question styles and randomized action order are used to address sensitivity to prompt wording and answer-choice order.Responses are collected one question at a time with fixed headers and sampling procedures, while API model changes remain an acknowledged constraint.
  • Survey findings: Figure 3 shows most models concentrating probability on commonsense actions in low-ambiguity scenarios, while most are uncertain in high-ambiguity scenarios.Only a few models exhibit certainty in the high-ambiguity setting.

4 Results

Across moral scenarios, most LLMs align with commonsense when ambiguity is low but become uncertain when ambiguity is high. Some models are inconsistent across question forms, while others show stable preferences and agreement, particularly among certain closed-source models.

  • General Trends: Most models align with commonsense in low-ambiguity scenarios, whereas most are uncertain about the preferred action in high-ambiguity scenarios.The low-ambiguity pattern is reflected in high likelihood for the commonsense action; high-ambiguity scenarios generally produce high uncertainty.
  • Action Likelihood: At least 7 of 28 models strongly prefer the less favorable action in some low-ambiguity scenarios, especially those involving sports or games and rules such as not cheating or deceiving.A strong preference is defined as marginal action likelihood ≥0.75.
  • Action Likelihood: The smallest candidate models show elevated uncertainty in low-ambiguity scenarios, with some OpenAI, Google, and BigScience models standing out as outliers.The identified outliers share the characteristic of being the smallest models in their respective candidate sets.
  • Action Likelihood: In high-ambiguity scenarios, a subset of mostly large models has average marginal action entropy of 0.7, corresponding to approximately 80% to 20% decision splits.These models include several OpenAI and Anthropic systems and Google’s flan-t5-xl and text-bison-001; all except flan-t5-xl are API-accessible.
  • Model Agreement: Hierarchical clustering separates models into a closed-source commercial cluster and a mixed cluster, with sub-cluster A showing pairwise correlations ≥0.75 internally but at most 0.28 with the mixed cluster.Sub-cluster A contains GPT-4, three Anthropic models, and Google’s text-bison-001, indicating agreement in both response patterns and encoded preferences.

5 Discussion & Limitations

The discussion identifies prompt sensitivity and sequential context resets as limitations, alongside limited diversity in scenarios and question forms. It also notes that low-ambiguity responses generally align with commonsense while varying substantially across prompt formats.

  • Discussion: Most models align with commonsense in low-ambiguity scenarios, but prompt-format changes can greatly alter response distributions.The findings motivate using multiple prompt variations when evaluating or fine-tuning LLMs.
  • Limitations: Scenario diversity is limited because the survey focuses on norm violations and does not cover enough professions or contexts.The authors plan to incorporate professional conduct codes and identify underlying factors before creating distinct scenarios.
  • Limitations: The survey uses only English prompts and three hand-curated question templates, limiting question-form consistency evaluation.Future work proposes automatically generating semantic-preserving prompt perturbations.
  • Limitations: Survey questions are administered sequentially with a reset context window, which reduces ordering biases but differs from real-world multi-turn use.Future research will examine how asking multiple questions sequentially affects outcome analysis.

A.1 Dataset Overview

MoralChoice contains 1,767 scenarios divided into low- and high-ambiguity settings, constructed around Gert’s common morality rules through separate generation pipelines. The scenarios are manually authored or LLM-generated, curated, deduplicated, and augmented with auxiliary rule-violation labels.

  • Dataset Overview: 1,767 scenarios comprise 687 low-ambiguity and 680 high-ambiguity moral decision-making cases.Low-ambiguity pairs contain one favorable and one unfavorable action, whereas high-ambiguity pairs contain two potentially unfavorable actions.
  • Dataset Overview: Gert’s common morality framework provides ten rules, categorized under “Do not harm” and “Do not violate trust,” to guide scenario construction.The rules are used to cover a broad range of moral decision-making.
  • Construction: MoralChoice construction follows scenario generation, manual curation, and auxiliary-label acquisition.The generation stage separately creates low- and high-ambiguity context–action triples guided by Gert’s rules.
  • Auxiliary Labels: The dataset augments each scenario with auxiliary labels identifying violated Gert morality rules.These labels are acquired through external annotation and finalized by majority voting.
  • Scenario Generation: Low-ambiguity scenarios use zero-shot prompting with gpt-4, while high-ambiguity scenarios use stochastic few-shot prompting based on handwritten demonstrations.The high-ambiguity pipeline uses handwritten examples because zero-shot generation had limited quality and diversity.

A.3 Dataset Curation

Dataset curation combines manual validity checks, grammar and pronoun corrections, consequence-clause removal, and similarity filtering. Annotators then assess action preferences and rule violations under separate low- and high-ambiguity instructions.

  • Validity and Post-Processing: Manual review removes incoherent, irrelevant, incorrectly ambiguous, and jargon-dependent scenarios.The curation criteria enforce the requested ambiguity level for each dataset.
  • Validity and Post-Processing: Remaining scenarios receive grammar and pronoun corrections, while clauses explaining potential consequences are removed.The post-processing aims to standardize scenario wording without adding consequence information to actions.
  • Similarity Filtering: Low-ambiguity duplicates and overly similar cases are filtered using cosine similarity of word-count vectors with threshold 0.6.Similarity is computed separately on contexts conditioned by rule and on full scenario text.
  • Annotation: Low-ambiguity annotation first asks whether action 1 is clearly preferred, then evaluates whether each action causes harm or violates trust.Affirmative answers trigger questions about ten specific rule violations.
  • Annotation: Three annotators label each question, with approximately 100 annotators employed through Surge AI.The annotators were paid $15 per hour, and the total labeling cost was $4,600.

A.5 Dataset Statistics

The appendix reports MoralChoice scenario statistics in Table 11.

  • Dataset Statistics: MoralChoice dataset statistics are reported in Table 11.

B Evaluation Protocol

The evaluation presents scenarios through multiple question templates and answer-order variations, then maps model outputs to actions using an iterative matching pipeline.

  • Question design: Three question templates require different reasoning abilities, and each scenario is presented twice with the answer options switched.The templates are A/B, Repeat, and Compare; switching options addresses recency and common-token bias.
  • Question design: Question templates encode the scenario context and two possible actions within the evaluation prompts.
  • Question design: Instruction headers specify the system-level prompt content before the main question template.
  • Response processing: The semantic-mapping pipeline checks exact matches, expanded answer variations, and stemming matches in sequence.

C.1 Model Cards

The evaluation documents model cards, data-access timing, and download timing to characterize the evaluated systems and support reproducibility.

  • Model documentation: Table 14 records evaluated models’ architecture, pre-training, and fine-tuning information.
  • Model documentation: Model-size estimates are sourced from reported details and may have changed over time.
  • Training information: The model-card material describes supervised fine-tuning, human demonstrations, model samples rated by human labelers, and internet-derived pre-training data.
  • Reproducibility: API calls and model downloads were timestamped, with exact response timestamps recorded for acquired samples.
  • Reproducibility: Separate evaluation timeframes are reported for the low- and high-ambiguity MoralChoice datasets.
  • Reproducibility: Downloaded HuggingFace model weights were stored on the cluster and reloaded from that point.

D.1 Invalid Responses and Refusals

The analysis reports invalid answers and refusals, then examines confident rule violations, model agreement, and question-form sensitivity across ambiguity settings.

  • Refusals: Only text-bison-001 showed an approximately 1% refusal tendency in low-ambiguity queries.Its refusals appeared as empty answer strings, whereas other small models often repeated instruction text followed by random text.
  • Invalid answers: Invalid answer rates for most models remained below 1% in low-ambiguity scenarios, while smaller models had higher rates.Figure 6 marks the 1% threshold with a red line.
  • Refusals: Several smaller models, gpt-3.5-turbo, and text-bison-001 exhibited relatively high refusal rates in high-ambiguity scenarios.Refusals ranged from generic “I cannot” responses in smaller models to scenario-contextualized refusals in some larger models.
  • Rule violations: Only a small set of models strongly preferred harm-causing actions in some low-ambiguity scenarios, with more violations involving deception and cheating rules.A violation counts only when marginal action likelihood for the unfavorable action reaches 0.75.
  • Model comparison: In high-ambiguity scenarios, four models sometimes strongly agreed on the same action, while other scenarios showed strong disagreement between gpt-4 and three other models.Strong agreement and disagreement use a marginal action-likelihood threshold of at least 0.75.
  • Question-form sensitivity: Several models were sensitive to answer-option ordering, and most models showed higher uncertainty for Compare questions.Figure 9 reports consistency and uncertainty across question styles.
Loading 2307.14324v1…