PaperlayerView exact revision on arXiv

Source-linked AI summary

When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment

Zhijing Jin, Sydney Levine, Fernando Gonzalez, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, Bernhard Schölkopf

arXiv:2210.01478v3cs.CLcs.AIcs.CYcs.LG

TL;DR

Predicting human moral judgments is difficult when rules admit exceptions, especially in novel circumstances. The paper introduces MoralExceptQA and MORALCOT, a cognitive-science-inspired prompting strategy; MORALCOT improves on the Delphi++ baseline by 6.2% F1. The authors present the task as a direction for studying how LLMs align with human moral practice.

  • Problem

    AI systems must predict human moral judgments to cooperate safely, but moral decisions require flexible, multi-step reasoning in novel situations.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces MoralExceptQA, a challenge set of potentially permissible rule-breaking cases, and uses MORALCOT to prompt an LLM with cognitive-science-inspired moral reasoning steps.

    Source: §1 · p. 2, §4 · p. 7

  • Results

    6.2% F1 improvement over Delphi++ is reported for MORALCOT on MoralExceptQA.

    Source: §5.1 · p. 8

  • Takeaways & Limitations

    MoralExceptQA opens a direction for AI safety research on how LLMs align with human moral practice.

    Source: §6 · p. 10

    The dataset is small, so the authors did not set aside a development set for prompt tuning.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

AI systems are becoming increasingly intertwined with human life. In order to effectively collaborate with humans and ensure safety, AI systems need to be able to understand, interpret and predict human moral judgments and decisions. Human moral judgments are often guided by rules, but not always. A central challenge for AI safety is capturing the flexibility of the human moral mind -- the ability to determine when a rule should be broken, especially in novel or unusual situations. In this paper, we present a novel challenge set consisting of rule-breaking question answering (RBQA) of cases that involve potentially permissible rule-breaking -- inspired by recent moral psychology studies. Using a state-of-the-art large language model (LLM) as a basis, we propose a novel moral chain of thought (MORALCOT) prompting strategy that combines the strengths of LLMs with theories of moral reasoning developed in cognitive science to predict human moral judgments. MORALCOT outperforms seven existing LLMs by 6.2% F1, suggesting that modeling human reasoning might be necessary to capture the flexibility of the human moral mind. We also conduct a detailed error analysis to suggest directions for future work to improve AI safety using RBQA. Our data is open-sourced at https://huggingface.co/datasets/feradauto/MoralExceptQA and code at https://github.com/feradauto/MoralCoT

1 Introduction

Human moral judgment depends on flexibly applying rules, including in novel situations that challenge AI systems. The paper introduces MoralExceptQA and MORALCOT, reporting improved performance over existing language models.

1 Introduction

  • Human moral rules can be overridden, created, or abandoned as circumstances change, making judgments in novel situations difficult for AI systems.The paper frames this flexibility as central to human morality and relevant to cooperation as opportunities to help and harm change.Source: §1 · p. 2
  • Moral decisions often require challenging, multi-step, multi-aspect reasoning, leaving the extent to which LLMs replicate human moral flexibility uncertain.The introduction notes that even people can struggle to respond to morally charged scenarios.Source: §1 · p. 2
  • MoralExceptQA tests whether breaking a familiar moral rule is permissible in familiar and unfamiliar circumstances.Its cases come from moral psychology studies and use careful parameter manipulation to create circumstances unlikely to appear in LLM training data.Source: §1 · p. 2
  • MORALCOT combines LLMs with cognitive-science accounts of moral reasoning through a moral philosophy-inspired chain-of-thought prompting strategy.The strategy follows cognitive mechanisms of contractualist moral decision-making.Source: §1 · p. 2
  • 6.2% improvement over the best state-of-the-art LLM is reported for the proposed model.The paper also presents an error analysis of LLMs’ moral-flexibility performance and suggests directions for future work.Source: §1 · p. 2

2 Background

The background connects moral judgment to AI safety and to cognitive-science accounts of flexible rule use. It situates the paper’s challenge set and prompting strategy within those concerns.

2.1 Important Questions for AI Safety

  • Human values are complex to specify in AI objectives, while cognitive science describes systematic structure in value-guided judgments despite variation across people and cultures.The paper draws on recent cognitive-science work to motivate integrating models of value-driven decisions into AI systems.Source: §2.1 · p. 3
  • The paper frames literal obedience to misspecified rules as a potential risk and positions MoralExceptQA and MORALCOT as ways to study and improve LLM moral flexibility.It also connects the work to challenging-task identification and analysis of undesired model behavior.Source: §2.1 · p. 3

2.2 The Human Moral Mind Is Flexible

  • Moral psychology studies how people decide when to break, update, or create rules, rather than treating rules as invariably fixed.The discussion gives examples of exceptions involving line-cutting and property damage, while noting individual variation in judgments.Source: §2.2 · pp. 3–4
  • Existing MoralQA approaches commonly fine-tune on broad datasets, but the paper argues they may struggle with completely novel cases and seeks to model underlying moral-judgment structure.The cited examples include 130K ETHICS samples and 1.7M Commonsense Norm Bank samples; prior approaches are described as achieving about 70 to 85% test performance.Source: §2.2 · p. 4

3 MoralExceptQA Challenge Set

MoralExceptQA probes moral flexibility by varying cases governed by a small set of rules. It is designed as a structured challenge set, with case studies spanning different kinds of norms and potential violations.

3 MoralExceptQA Challenge Set

  • MoralExceptQA probes whether established rules may permissibly be broken, using three rules as case studies of broader categories of socio-moral norms.The dataset probes many possible violations of each norm, while the authors note that the three rules cover only a small fraction of rules relevant to human cooperation.Source: §3 · p. 4
  • The authors limit their conclusions about moral judgments to U.S. residents and identify cross-cultural variation as a subject for future study.The paper calls for investigating how judgments in these and other cases vary across cultures.Source: §3 · p. 4
  • Unlike broad MoralQA datasets, MoralExceptQA varies similar contexts governed by the same rule, testing sensitivity to subtle manipulations.The dataset is described as a high-quality challenge set built from carefully manipulated stimuli and many human responses, despite having relatively few vignettes.Source: §3 · pp. 5–6

3.1 Norm 1: No Cutting in Line

  • The no-cutting-in-line study varies the reason for cutting, delay to others, the cutter’s need, and whether cutting violates the line’s function.The study used scenarios in four locations and collected responses from 403 subjects.Source: §3.1 · p. 6

3.2 Norm 2: No Interfering with Someone Else’s Property

  • The property-rights study varies the offer made to Hank and the type and extent of requested property damage.The offer ranged from $100 to $1 million or a threat to kill Hank’s son; 360 subjects participated, with 60 judgments per condition.Source: §3.2 · p. 6

3.3 Norm 3: No Cannonballing into the Pool (Novel Rule)

  • The novel cannonballing rule study varies whether the prohibition’s function is violated and who else may be harmed or benefited.The rule concerns protecting nearby children’s art or avoiding distraction; 149 subjects judged 28 scenarios.Source: §3 · p. 4, §3.3 · p. 6

4 MORALCOT: A Cognitively-Inspired Model

MORALCOT uses a cognitive-science-inspired sequence of questions to elicit morally relevant reasoning before a final judgment. Its design focuses on rule functions and the costs and benefits of potential rule-breaking.

4 MORALCOT: A Cognitively-Inspired Model

  • The paper situates its approach as a pathway for combining cognitive-science insights with LLMs to improve MoralExceptQA performance.The section frames this as an exploration of one pathway, rather than a claim to exhaust possible approaches.Source: §4 · p. 7
  • Contractualist moral judgment considers the agreement of people affected by a decision, and rules’ underlying functions can determine whether an action is permissible.The paper’s example is that cutting a deli line to replace a wrong order may be permissible when the person’s claim to the resource was not treated equivalently.Source: §4 · p. 7
  • MORALCOT models a three-step reasoning process: identify a rule’s function, assess whether a breach is permitted given that function, and consider what else is at stake.The final consideration weighs utility gained and lost.Source: §4 · p. 7
  • MORALCOT prompts an InstructGPT model with a sequence of questions and prior answers, ending with an overall question about whether breaking the rule is acceptable.The final Yes/No answer is parsed as binary permissibility.Source: §4 · p. 7
  • The prompt primes morally relevant features through subquestions instead of asking directly for an overall judgment at the outset.The intended role of the subquestions is to let the model use those concepts in constructing its final decision.Source: §4 · p. 7

5 Experiments

MORALCOT improves performance on MoralExceptQA, while error analyses identify weaknesses in purpose reasoning, utility estimation, and scenario-sensitive judgments. Dataset size also constrained prompt tuning.

5.1 Main Results

  • Evaluation uses weighted F1, accuracy, conservativity, and human-distribution MAE and cross-entropy across established moral-QA baselines and language models.Conservativity measures the share of errors due to predicting “not permissible.”Source: §5.1 · p. 8
  • MORALCOT reaches 64.47% F1, improving 10.53% over InstructGPT and 6.2% over Delphi++; its conservativity score is 66.96%.Table 3 reports performance across multiple metrics and subsets.Source: §5.1 · p. 8, Table 3 · p. 9

5.2 Detailed Error Analysis

  • In InstructGPT’s subquestion analysis, Loss is easiest because it follows the literal rule, while Purpose—whether an action serves a rule’s underlying function—is hardest.The evaluation also considers Benefit as a subquestion.Source: §5.2 · pp. 8–9, Table 4 · p. 8
  • InstructGPT’s cost estimates have 0.711 log-MAE for general actions but 1.77 in property-violation cases involving multiple actors or sentimental value.The property-violation study elicited human estimates of compensation for property damage.Source: §5.2 · p. 9
  • Utility analysis misses mentions of 38.51% of different parties and incompletely describes 58.10% of parties; 79% of explanations quote the literal rule.The analysis also reports that factual misunderstandings occur in 7.43% of cases.Source: §5.2 · p. 9
  • Among explanations that refer to scenario-specific conditions, reasoning quality is 73%, and errors are often too dogmatic, such as prohibiting cannonballing when no art class is disturbed.The authors use InstructGPT as the case study for this error analysis.Source: §5.2 · pp. 8, 10
  • Textual-similarity and prediction-similarity correlation is 0.190 overall but 0.902 for bathroom-line cases.The analysis examines whether correlation changes across scenario keywords.Source: §5.2 · p. 10, Table 5 · p. 10

5.3 Discussions

  • Dataset size prevented holding out a dev set for prompt tuning; the authors suggest collecting more data while retaining MoralExceptQA’s structure.They also report a prompt-paraphrase sensitivity analysis in the Appendix.Source: §5.3 · p. 10

6 Conclusion

The conclusion presents MoralExceptQA as a challenge set for studying moral flexibility and reports improved LLM performance with MORALCOT.

6 Conclusion

  • The paper introduces MoralExceptQA to probe moral flexibility and reports improved LLM performance using MORALCOT, a prompting strategy inspired by multi-step human reasoning.The authors frame the task as a direction for future AI safety research on aligning LLMs with human moral practice.Source: §6 · p. 10

Checklist

The checklist records author responses about claims, reproducibility, research assets, and human-subject procedures, often referring readers to other sections or the appendix.

Checklist

  • Checklist responses about code and dataset licenses are inconsistent: one refers to Appendix A, another calls the assets proprietary, and another marks licensing not applicable..Source: Checklist · p. 15
  • The authors state that the paper describes its limitations and directs readers to Section 5.3 for limitations and potential negative societal impacts.They also mark that the main claims reflect the paper’s contributions and scope.Source: Checklist · p. 16
  • The checklist points readers to the appendix for code, data, and result-reproduction information, training details, error bars, and compute resources.It also refers to the appendix for asset licensing and human-subject materials, including consent and participant-risk information.Source: Checklist · p. 16

A Studies with Human Subjects: Data Collection Details

This appendix describes human-subject data collection for studies of rule-breaking involving line-cutting, interference with property, and a novel pool rule, using text or picture-supported scenarios to assess permissibility. In the pool-rule study, participants also answered questions about the violation, its consequences, need, and benefit; permissibility responses were converted to probabilities and averaged.

A.1 Norm 1: No Cutting in Line

  • The line-cutting research included text-only deli, bathroom, and airport prompts and picture-and-text classroom snack-line studies.The picture-supported work was divided into Snack Line Studies 1 and 2.Source: §A.1 · p. 17

A.1.1 Snack Line Study 1

  • Snack Line Study 1 retained 48 of 72 participants after excluding 24 for failed controls; participants judged whether children in varied classroom scenarios could go to the front of the snack line.Responses were coded as 1 for OK and 0 for Not OK, and the proportion of OK responses was computed for each question.Source: §A.1.1 · pp. 17–18

A.1.2 Snack Line Study 2

  • Snack Line Study 2 had 121 participants; 19 were excluded for incorrect control answers, and 54 answered permissibility questions about snack-line scenarios, with the study text also shown in pictures.Source: §A.1.2 · pp. 18–19

A.2 Norm 2: No Interfering with Someone Else’s Property

  • The property-interference research comprised separate permissibility-judgment and compensation-demand studies, both involving US-based MTurk workers and approved by MIT’s institutional review board.The appendix refers readers to Levine et al. (2018) for full experimental details and reports aggregate MTurk demographics rather than collecting further demographic data.Source: §A.2 · pp. 20–21

A.2.1 Permissibility Judgments

  • In the property-permissibility study, 360 participants were assigned across six offer conditions and each judged 10 randomized cases of damage to a neighbor’s property.Offer conditions ranged from $100 to $1,000,000, with a threat against Hank’s son as the sixth condition.Source: §A.2.1 · p. 21

A.2.2 Compensation Demands

  • The compensation-demand study asked 100 participants to state the minimum payment for allowing each listed form of property damage to be done to Hank’s property.The prompt specified that the stranger would pay Hank and that nothing else would happen to Hank or his property.Source: §A.2.2 · pp. 21–22

A.3 Norm 3: No Cannonballing into the Pool (Novel Rule)

  • The pool-rule study randomized 149 participants to noise or splash conditions, then presented 14 scenarios with four-level permissibility ratings and free-response justifications for a random subset.The scenarios varied whether the rule’s rationale applied, such as rescuing someone, avoiding harm, or removing the disturbance the rule addressed.Source: §A.3 · pp. 22–23
  • Subjects saw the scenarios again in random order, with the child cannonballing in each, including a scenario where she sought to retrieve a phone dropped in the pool.The example scenario describes the child cannonballing while trying to get the counselor’s phone out of the water.Source: §A.3 · p. 23
  • After each scenario, subjects rated likely distraction or art damage, whether the rule was broken, how necessary the cannonball was, and how much it helped someone else.Ratings used four ordered response options, from Definitely Yes to Definitely No or from A whole lot to Not at all.Source: §A.3 · p. 23
  • Subjects then answered demographic questions and could report confusion; permissibility responses were coded from 1 for Definitely OK to .25 for Definitely Not OK, then averaged by question.Maybe OK was coded .75 and Maybe Not OK .5.Source: §A.3 · pp. 23–24

B.1 Implementation Details

The appendix documents model settings and prompts, then describes the subquestions used to assess InstructGPT’s judgments across several scenario types.

B.1 Implementation Details

  • GPT experiments use davinci and davinci-text-002 with temperature zero, while four masked language models use top_k 15; Delphi’s positive and neutral classes are merged as permissible.The four LM baselines take approximately one hour to run; OpenAI API usage cost $600, and standard metrics use sklearn’s classification_report. Few-shot prompting was excluded because results varied with example randomness and the challenge set was small.Source: §B.1 · p. 24

B.2 Prompt Templates

  • Table 6 lists prompts for LLMs, and the authors provide an example prompt online.The passage does not specify the table’s layout or individual prompt contents.Source: §B.2 · p. 24, Table 6 · p. 25

B.3 Paraphrases of the Prompt

  • The prompt paraphrases vary the wording and capitalization of the yes-or-no question about whether breaking the rule is acceptable.Some versions request a one-word response; others refer to the preceding discussion or answers.Source: §B.3 · p. 25

B.4 Prompts for “Quality of Sub-question Answers”

  • Prompts for subquestion-answer quality elicit judgments about harms, benefits, line-waiting effects, and property-damage prices.Examples ask about effects on people in line, whether a child has already waited, and the amount needed to permit property damage.Source: §B.4 · pp. 25–26

B.5 Prompts of Multiple Parties & Explanations

  • The multiple-parties prompt asks who is positively or negatively affected and how much worse or better off they become; a separate prompt requests an explanation for the acceptability judgment..Source: §B.5 · p. 26

C.1 Experimental Details: Subquestions

  • Subquestion evaluation covers Loss, Benefit, and Purpose; weighted F1 and accuracy include only the line-cutting and cannonballing subsets, weighted by sample count.Property-damage questions are simplified to monetary analysis; human responses are compared with model outputs across scenario-specific question variants.Source: §C.1 · p. 27, Table 7 · p. 27

C.2 Experimental Details: Understanding Utility

This section checks InstructGPT’s understanding of monetary values by comparing its cost estimates with human responses and reporting log-MAE across two action-set sizes.

C.2 Experimental Details: Understanding Utility

  • InstructGPT’s log-MAE was 0.711 on 50 actions and 0.673 on the extended set of 251 actions.The authors report these as results from the first and extended versions of the action set, respectively.Source: §C.2 · p. 27
  • The 251-action set was assembled to assess whether InstructGPT understands monetary values for actions, a necessary intermediate step in the property-violation case study.The authors describe the 50-action set as the dataset’s first version.Source: §C.2 · p. 27
  • For the monetary-value assessment in the property-violation study, Table 8 reports average costs suggested by human responses and InstructGPT for selected examples from the 251-action set.Source: §C.2 · p. 27

C.3 Experimental Details: Checking the Explanations

The explanation checks illustrate factual misunderstanding, omitted utility-relevant parties or conditions, and reasoning that treats any rule violation as unacceptable; Table 8 compares human and GPT-3 cost predictions.

C.3 Experimental Details: Checking the Explanations

  • Table 8 compares costs in USD predicted by GPT-3 with human responses.The supplied table text does not state particular cost values.Source: Table 8 · p. 28
  • In the mural example, InstructGPT gives a factually inconsistent explanation by saying the neighbor’s daughter painted without her father’s permission, although the scenario says she did so without the neighbor’s permission.The response also recommends asking the neighbor for permission before painting over the mural.Source: §C.3 · pp. 27–28
  • In the classroom line example, InstructGPT overlooks the waiting cost to other children, the girl’s benefit, the teacher’s benefit, and conditions such as quick action or appreciation.Humans judged the action permissible 79% of the time, while InstructGPT answered “No.”Source: §C.3 · p. 28
  • In the camp example, InstructGPT rejects cannonballing even though the art is already ruined, repeatedly citing the rule instead of its purpose.Humans judged the action permissible 70% of the time.Source: §C.3 · p. 28

C.4 Experimental Details: Dependence on the Literal Text

The section describes a scenario-keyword analysis comparing textual similarity with model-prediction similarity among samples sharing keywords, with low-frequency keywords excluded from the main paper.

C.4 Experimental Details: Dependence on the Literal Text

  • The main paper excludes scenario keywords with fewer than six samples from its analysis.The full list in Table 9 includes keywords beyond those retained in the main paper.Source: §C.4 · p. 28, Table 9 · p. 29
  • For each 0.1 decile, the main paper keeps the keyword with the largest number of samples.This selection rule is described alongside the minimum-sample filter.Source: §C.4 · p. 28
  • Table 9 reports correlation scores between textual similarity and model-prediction similarity for pairs of samples sharing scenario keywords.The table presents the broader keyword list; the main paper applies additional filtering.Source: §C.4 · p. 28, Table 9 · p. 29
Loading 2210.01478v3…