Source-linked AI summary
Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences
Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, Yejin Choi
TL;DR
Moral Stories addresses limited goal-oriented social reasoning that jointly considers actors’ motivations, action outcomes, and adherence to moral norms. It introduces a structured narrative dataset and expert-model decoding strategies, which improve constrained generation quality according to human evaluation and support anticipating outcomes for socially optimal decisions and discovering unspoken moral principles.
Problem
Prior work did not jointly ground social behavior in concrete situations while considering actors’ motivations, action outcomes, and conformity to moral norms.
Method
The paper introduces the 12k-narrative Moral Stories dataset and Chain-of-Experts decoding algorithms that combine expert models for goal-oriented moral reasoning.
Results
Models identify moral actions and plausible consequences effectively, while expert-model decoding improves generative constraint satisfaction according to human evaluation.
Takeaways & Limitations
Anticipating plausible action outcomes can support socially optimal decisions and discovery of unspoken moral principles governing social interactions.
Takeaways & Limitations
The dataset is noticeably skewed toward White, educated US residents, limiting representation of less-represented groups’ moral narratives.
Abstract
from arXiv · showhide
In social settings, much of human behavior is governed by unspoken rules of conduct. For artificial systems to be fully integrated into social environments, adherence to such norms is a central prerequisite. We investigate whether contemporary NLG models can function as behavioral priors for systems deployed in social settings by generating action hypotheses that achieve predefined goals under moral constraints. Moreover, we examine if models can anticipate likely consequences of (im)moral actions, or explain why certain actions are preferable by generating relevant norms. For this purpose, we introduce 'Moral Stories', a crowd-sourced dataset of structured, branching narratives for the study of grounded, goal-oriented social reasoning. Finally, we propose decoding strategies that effectively combine multiple expert models to significantly improve the quality of generated actions, consequences, and norms compared to strong baselines, e.g. though abductive reasoning.
1 Introduction
The paper studies whether NLG models can generate goal-achieving actions that respect or violate moral norms in concrete social situations and anticipate their consequences. It introduces Moral Stories and Chain-of-Experts decoding to support and improve this reasoning.
- Motivation: Social navigation requires both achieving intended outcomes and following unspoken norms, because socially objectionable actions can produce negative consequences.For example, rudely ordering food may offend staff and result in worse service.
- Approach: The work grounds moral reasoning in concrete situations by generating actions that fulfill goals while observing or violating norms.This treats NLG models as proxies for agents navigating social spaces.
- Novelty: Moral Stories unifies prior work on intentions and consequences (Rashkin et al., 2018; Hwang et al., 2020) with norm alignment (Forbes et al., 2020) by modeling goals, context, and moral constraints together.The dataset focuses on descriptive morality: subjective judgments guided by an implicit code of conduct (Gert and Gert, 2002).
- Contributions: The paper presents Moral Stories, a structured corpus of 12k short narratives, evaluates baseline models on classification and generation tasks, and introduces Chain-of-Experts decoding.The decoding algorithms sequentially combine expert models to improve generation quality.
2 The Moral Stories Dataset
Moral Stories is a crowd-sourced dataset of seven-sentence narratives that represent a shared social context alongside divergent moral and immoral action–consequence paths. Stories are elicited from semantically diverse norms and quality-controlled through worker qualification, feedback, disqualification, and validation.
- Story structure: Each seven-sentence story contains a norm, situation, intention, moral action and consequence, and immoral action and consequence.The norm is a generally observed rule; the situation introduces participants, the intention states a reasonable goal, and each action fulfills that goal either while observing or violating the norm.
- Story structure: The context segment combines separately with moral or immoral path segments to form two self-contained, morally divergent sub-stories.The moral path contains the moral action and consequence, while the immoral path contains their analogues.
- Data collection: Crowd-workers on Amazon Mechanical Turk write realistic stories from semantically diverse norms drawn from SOCIAL-CHEM-101’s Morality/Ethics and Social Norms categories, excluding controversial or value-neutral entries.Each worker chooses one of three norm prompts, with instructions requiring rational intentions, plausible intention-satisfying actions, and direct consequences.
- Quality control: Dataset quality is supported by qualification rounds, periodic submission reviews with feedback, disqualification for repeated substandard work, and a subsequent validation round.These procedures were used to enforce formatting and narrative quality during and after collection.
3 Grounded Classification
Grounded classification experiments show that increasing contextual information improves recognition of morally contrasting actions and plausible consequences. However, consequence-based inputs can enable near-perfect performance through sentiment shortcuts, while lexical bias and minimal-pair splits expose shortcut reliance.
- Experimental setup: The experiments use RoBERTa (Liu et al., 2019)3 with grid-searched hyperparameters across classification tasks that vary the amount of grounding information.The grounded settings combine story components such as norms, situations, intentions, actions, and consequences.
- Action classification: Increasing grounding improves action-classification accuracy across all test sets, with norms enabling links between actions and behavioral rules and contextual grounding supporting moral reasoning without shortcuts.Classifying actions in isolation becomes difficult after lexical biases are controlled, whereas adding norms and context improves performance.
- Action classification: 98-99% accuracy remains when consequences are the sole grounding source, because moral actions are associated with positive and immoral actions with negative consequences.Adding consequences to norms and context produces near-perfect performance by allowing prediction from consequence sentiment.
- Generalization and split strategies: Performance differences across test sets show that classifiers exploit lexical correlations, but these shortcuts become less important with richer grounding; lexical-bias and minimal-pair sets are similarly challenging.The split strategies target generalization to novel norms, surface-level lexical correlations, and nuanced moral reasoning.
- Consequence classification: Contextual grounding improves consequence classification, enabling models to distinguish plausible consequences of morally contrasting actions.Negative samples pair consequences with actions of opposing moral orientation within the same story.
4 Grounded Generation
Grounded generation evaluates whether language models can produce norm-compliant goal-directed actions, plausible consequences, and explanatory norms. Task-specific expert-model decoding improves constraint satisfaction and consequence plausibility, while future information helps guide actions and norms.
- Action generation: Conditioning action generation on consequences has little automatic-metric effect, but human judges prefer these actions because future information improves goal and norm satisfaction.The consequence signal may bias models toward goal-directed and morally oriented actions.
- Consequence generation: Grounded prediction produces more plausible consequences but does so unreliably, while models exploit rich contextual information and show inferior performance on immoral targets.These results support a moral positivity bias in pretrained language models.
- Norm generation: Contextual grounding does not improve norm relevance, but adding consequences increases relevance despite generating norms that are consistently less diverse than story prompts.Consequences may help by referencing parts of action descriptions and directing the model toward relevant norms.
- Decoding strategies: Expert-model decoding strategies improve constraint satisfaction across generated outputs by ranking candidates and conditioning generators on other experts’ predictions.The proposed chains of fine-tuned expert models address failures to satisfy explicit and implicit generation constraints.
- Consequence generation: Both consequence-generation decoding strategies improve plausibility and narrow the moral–immoral gap, although immoral consequences remain less plausible than moral ones.The simpler ranking strategy is reported as more successful than the alternative refinement strategy.
- Norm generation: Synthetic consequences make generated norms as relevant as norms based on reference consequences, showing that anticipated outcomes can guide norm generation when outcomes are unknown.The method predicts and ranks consequences for both actions before conditioning norm generation on the best predictions.
5 Related Work
The study builds on computational research into social dynamics and moral reasoning, drawing on annotated norms and prior ethical-judgment evaluations. Its CoE algorithms relate to NLG rescoring and deliberation networks.
- 5 Related Work: The study is motivated by computational research on social dynamics and efforts to assess whether NLU/NLG models reason about moral and ethical principles (Rashkin et al., 2018; Sap et al., 2019a,b, 2020; Frazier et al., 2020).
- 5 Related Work: Moral Stories relies on richly annotated norms from SOCIAL-CHEM-101 (Forbes et al., 2020) and follows initial evaluations of NLU ethical judgments on long-form, unstructured texts (Lourie et al., 2020; Hendr).
- 5 Related Work: The CoE algorithms relate to NLG rescoring methods and deliberation networks developed to improve machine translation quality (Holtzman et al., 2018; Goldfarb-Tarrant et al., 2020).CoE refinement uses a secondary expert model to refine initial hypotheses.
6 Conclusion and Future Work
The study finds that classifiers can identify moral actions and plausible consequences accurately from rich grounding, while generative models often violate task-specific constraints. It introduces expert-model-based decoding algorithms and outlines extensions toward complex scenarios, automated norm discovery, and narrative or dialogue generation.
- Classifiers identify moral actions and plausible consequences with high accuracy by leveraging rich grounding information.
- Generative models frequently fail to satisfy task-specific constraints, including norm relevance and plausibility; expert-model-based decoding algorithms are introduced to address this issue.
- Future work may study more complex moral scenarios, automate norm discovery for non-Western norms and customs, and integrate these methods into narrative and dialogue generation.
7 Ethical Considerations
The dataset was curated with worker welfare and narrative quality in mind, but it is predominantly shaped by mainstream US English and White, educated US contributors. Its stories focus on everyday social interactions and reflect the June–September 2020 collection period.
- Worker compensation: Workers were compensated so median pay exceeded $15/hour, including qualification and evaluation rounds.Rewards were adjusted using monitored median HIT completion times.
- Curation rationale: Accepted narratives had to be coherent, internally cohesive, instruction-compliant, non-offensive, unbiased, and grounded in everyday scenarios.
- Speaker demographics: 98.5% of contributors were US-based, and the dataset skews toward White, educated residents despite variation across age, gender, and economic backgrounds.Future collection should target less-represented groups.
G. RECORDING QUALITY: N/A
The dataset was created from norm-based writing prompts and has a largely U.S.-based, white worker population. The authors caution that it is for scientific machine-reasoning research, while immoral examples provide negative training signals but may also enable malicious-agent training.
- I. Provenance Appendix: The dataset’s crowd-workers were predominantly white (89%) and U.S.-based (94%), with roughly balanced gender, age, and economic-class distributions comparable to SOCIAL-CHEM-101 contributors (Forbes et al., 2020).Workers received norms extracted from SOCIAL-CHEM-101 as prompts to encourage thematic narrative variety.
- H. Other: The authors state that the work is strictly scientific machine-reasoning research, not guidance for human interactions.They explicitly caution that the dataset should not be treated as advice for human behavior.
- H. Other: Immoral actions and consequences may help adversaries train malicious agents, but they also provide negative training signals for cooperative agents to minimize harm.Future dataset users should specify how moral and immoral examples are used and for what purpose.
- H. Other: The authors aimed to reduce offensive or biased language in the dataset by giving workers guidance.The supplied passage begins describing this mitigation but does not provide its full details.
A Moral Stories: Supplementary Details
Supplementary analyses characterize Moral Stories’ component lengths and thematic coverage, while documenting the crowd-worker instructions used for collection. The dataset centers on interpersonal relationships across domestic, commercial, and educational settings, with norms and intentions shorter than other components.
- Component Lengths: Norms and intentions are substantially shorter than other story components, while morally divergent actions and consequences have comparable lengths, reducing likely sequence-length artifacts in classification.The comparable action and consequence lengths make exploiting sequence length for classification performance gains unlikely; the shorter norms and intentions are attributed to limited semantic content.
- Dataset Topics: LDA analysis shows that Moral Stories centers on interpersonal relationships across domestic life, commerce, and education.The analysis lists words for ten latent topics in Table 13.
- Collection Instructions: Crowd-workers received extensive HIT instructions, qualification-round practice, annotated positive and negative examples, and feedback to resolve uncertainties.These materials supported familiarization with the task and the required narrative format during story collection.
B Classification: Supplementary Details
This section reports supplementary classification details, including shared training settings, development-set performance, and the sizes of classification data subsets.
- Development Results: Development-set performance is reported for action and consequence classification, with models generally performing better than on test data because development splits are less challenging and contain exploitable training-data shortcuts.Results appear in Tables 8 and 9; split properties are reported in Table 1.
- Dataset Sizes: Table 10 lists the numbers of samples in each classification data subset across splitting methods and task settings.The table covers all subsets considered in the classification experiments.
C Generation: Supplementary Details
The supplementary material documents generation-model hyperparameters, dataset subset sizes, input formats, additional results, qualitative examples, and the expert-model components used by CoE strategies.
- Training and evaluation details: Fine-tuning hyperparameters and generation-data subset sizes are reported in Tables 11 and 12, while training duration varies by task and architecture because of early stopping.Tables 15–17 also report automatic quality-estimation metrics for the second- and third-best models across generation tasks and settings.
- Additional results: Additional test results for action, consequence, and norm generation are provided in Tables 15, 16, and 17, respectively.These tables extend the reported generation-task evaluations across settings.
- Input formats: Table 18 specifies the input formats for different generation settings, including decoder initialization for BART and T5.Special separator tokens formatted as <|TOKEN|> are added to each model vocabulary before fine-tuning with randomly initialized embeddings.
- Qualitative examples: Figures 4, 5, and 6 provide examples of generated actions, consequences, and norms produced by the methods discussed in the main text.The figures distinguish relevant or plausible predictions from irrelevant, implausible, or nonsensical ones.
- CoE components: Table 19 summarizes the component expert models used by the proposed CoE decoding strategies.The supplementary material therefore documents the model composition underlying CoE decoding.