PaperlayerView exact revision on arXiv

Source-linked AI summary

Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes

Nicholas Lourie, Ronan Le Bras, Yejin Choi

arXiv:2008.09094v2cs.CL

TL;DR

The paper asks how AI systems can model the ethical norms of different communities, especially when judgments are complex and divided. It introduces SCRUPLES, the WHO’S IN THE WRONG? task, and methods for learning from judgment distributions and estimating attainable performance. Complex anecdotes remain highly challenging, while simpler situations are more promising, and the proposed likelihoods distinguish intrinsic from model uncertainty.

  • Problem

    AI systems need to understand communities’ ethical norms, but pragmatic interpretation and disagreement in real-world moral judgments remain challenging.

  • Method

    The paper releases SCRUPLES and DILEMMAS, formalizes WHO’S IN THE WRONG?, models judgment distributions with alternative likelihoods, and introduces a best-performance estimator.

  • Results

    Complex SCRUPLES anecdotes are extremely challenging for existing methods, while simpler DILEMMAS situations are more promising; Dirichlet-multinomial layers improve current techniques by using all annotations.

  • Takeaways & Limitations

    Ethical norms are often controversial rather than clean-cut, so systems should represent community judgment distributions and distinguish norms’ controversiality from model uncertainty.

  • Takeaways & Limitations

    Norms from one community may be inappropriate for another, and the code, models, and data should not be deployed without careful evaluation.

Abstract

from arXiv · show

As AI systems become an increasing part of people's everyday lives, it becomes ever more important that they understand people's ethical norms. Motivated by descriptive ethics, a field of study that focuses on people's descriptive judgments rather than theoretical prescriptions on morality, we investigate a novel, data-driven approach to machine ethics. We introduce Scruples, the first large-scale dataset with 625,000 ethical judgments over 32,000 real-life anecdotes. Each anecdote recounts a complex ethical situation, often posing moral dilemmas, paired with a distribution of judgments contributed by the community members. Our dataset presents a major challenge to state-of-the-art neural language models, leaving significant room for improvement. However, when presented with simplified moral situations, the results are considerably more promising, suggesting that neural models can effectively learn simpler ethical building blocks. A key take-away of our empirical analysis is that norms are not always clean-cut; many situations are naturally divisive. We present a new method to estimate the best possible performance on such tasks with inherently diverse label distributions, and explore likelihood functions that separate intrinsic from model uncertainty.

1 Introduction

The paper frames ethical understanding as a pragmatic and community-dependent challenge, introducing SCRUPLES and related resources to model both ethical judgments and their divisiveness. It finds that complex narratives challenge current methods, while modeling annotation distributions and uncertainty addresses variability in moral judgments.

  • Motivation: Pragmatic understanding remains difficult because ethical interpretation requires social, cultural, and moral inferences beyond literal text.Readers must infer unstated implications shaped by social, cultural, and moral conventions.
  • Motivation: Machine ethics requires anticipating the norms of different communities, so the paper studies people’s descriptive ethical judgments rather than prescribing morality.This focus distinguishes descriptive ethics from prescriptive ethics.
  • Resources and task: The WHO’S IN THE WRONG? task predicts which person in an anecdote a community considers to be violating ethical norms, while representing the distribution of judgments.The task is motivated by the observation that judgments on complex real-life scenarios are often divisive.
  • Modeling ambiguity: The paper introduces a general estimator of the best possible metric score and evaluates likelihoods that separate intrinsic uncertainty from model uncertainty.The approach treats diversity in human judgments as a property to model rather than annotation noise.
  • Resources and task: SCRUPLES contains more than 32,000 real-life ethical anecdotes and 625,000 judgments extracted from Reddit, while DILEMMAS provides 10,000 crowdsourced normative action judgments.The resources support a challenge set for complex situations and a simpler resource for ethical actions.
  • Empirical challenge: Existing methods find SCRUPLES extremely challenging, whereas results on simpler DILEMMAS suggest that much of the difficulty lies in understanding complex narratives rather than basic ethical judgments.This contrast motivates separating narrative understanding from ethical judgment prediction.

2 Datasets

SCRUPLES combines real-life anecdotes with community ethical judgments, while DILEMMAS provides paired actions for simpler normative comparisons. The datasets preserve both judgment distributions and extracted story attributes for studying ethical prediction.

  • ANECDOTES: Each anecdote includes a title, body text, label scores, type, and an extracted action when available.Types distinguish historical events from hypothetical ones, while the label is the highest-scoring class.
  • ANECDOTES: Label scores count community judgments, so normalized scores estimate the probability that a community member holds each opinion.The dataset therefore retains more information than a single majority label.
  • ANECDOTES: SCRUPLES contains 32,766 anecdotes totaling 13.5 million tokens and 626,714 ethical judgments, with a median of eight annotations per story.Most stories have enough annotations to provide insight into judgment distributions.
  • ANECDOTES: SCRUPLES is constructed from Reddit anecdotes and comments, using rules and regular expressions to extract story fields and ethical labels.Manual evaluation reported filtering F1 scores of 97% for comments and 99% for posts, with label extraction averaging 92% across five classes.
  • DILEMMAS: DILEMMAS contains paired actions from the anecdotes and asks crowd workers which action is less ethical.Pairs were randomly matched within each split, and each pair was labeled by five Mechanical Turk annotators after worker vetting with MACE.

3 Methodology

The methodology addresses distributional ethical judgments by estimating an oracle-based performance target and separating intrinsic label controversy from model uncertainty. It recommends distribution-aware likelihoods, metrics, calibration, and targets for ambiguous tasks.

  • Human judgments can be divisive, so ethical classification must model label distributions rather than assume clean-cut categories.
  • Target Performance: BEST estimates the oracle classifier’s performance from gold annotations when direct measurement of distributional human performance is difficult.The oracle represents average human annotation behavior rather than the label chosen by any particular annotator.
  • Target Performance: BEST is validated through simulations of ANECDOTES, three-annotator settings, and a mixed Dirichlet prior, with relative estimation error reported in Table 5.
  • Metrics: Distributional predictions should be evaluated with total variation distance or cross-entropy against soft gold labels, with probabilities calibrated via temperature scaling.
  • Target Performance: The methodology recommends targeting BEST performance, especially for ambiguous tasks where human performance can be misleadingly low.
  • Modeling: Dirichlet-multinomial layers model distributions over class probabilities, separating controversiality in labels from model uncertainty while generalizing softmax layers.Unlike softmax, the approach can use annotation counts instead of majority-vote labels.

4 Experiments

The experiments test whether SCRUPLES can be solved through dataset artifacts, lexical cues, and pretrained language models. RoBERTa with a Dirichlet likelihood performs best on both datasets, while complex anecdotes remain harder than simpler dilemmas.

  • Experimental setup: The experiments test feature-agnostic, stylistic, lexical, and deep baselines to identify dataset biases and assess ethical understanding.The evaluation includes label-only and stylistic controls, n-gram classifiers, and BERT or RoBERTa models with and without a Dirichlet-multinomial likelihood.
  • Experimental setup: The models are calibrated with temperature scaling so cross-entropy comparisons use calibrated development and test scores.Calibration is applied because machine-learning models are often poorly calibrated out of the box.
  • Results: RoBERTa with a Dirichlet likelihood outperforms all other models on both ANECDOTES and DILEMMAS under cross-entropy.The Dirichlet likelihood uses all annotations and can separate question controversiality from model uncertainty.
  • Results: Label-only and stylistic baselines perform poorly on both datasets, well below human and BEST performance.These controls therefore do not explain the task results through simple label distributions or stylistic artifacts.
  • Results: Shallow models perform poorly on ANECDOTES, while a bag-of-n-grams logistic ranker learns some aspects of DILEMMAS.The authors attribute the difference to richer narrative structure in ANECDOTES and the greater role of lexical knowledge in short DILEMMAS.

5 Analysis

The analysis examines whether using full annotation distributions improves training and explores the lexical structure of DILEMMAS. Full-annotation likelihoods generally outperform majority-vote training, while DILEMMAS contain many ordinary verbs and everyday social topics.

  • 5.1 Comparing Different Likelihoods: Soft, Counts, and Dirichlet all represent response variability, but Counts and Dirichlet weight heavily annotated ANECDOTES more strongly.For DILEMMAS, fixed annotations make the likelihoods equivalent; for ANECDOTES, annotation counts vary substantially across examples.
  • 5.1 Comparing Different Likelihoods: Likelihoods using all annotations consistently outperform majority-vote training in cross-entropy, except Counts on ANECDOTES.Dirichlet and Soft perform comparably, while Counts can perform poorly because uneven annotation counts overweight some examples.
  • 5.2 Lexical Analysis: DILEMMAS are short, single-act situations with relevant context, unlike ANECDOTES, which contain many actors and diverse conditions.This structural contrast motivates separate lexical analysis of the simpler DILEMMAS resource.
  • 5.2 Lexical Analysis: The DILEMMAS training set contains 1520 unique root verbs, with “wanting” at 14%, “telling” at 7%, and “being” at 5%.Permutation testing with Holm-Bonferroni correction identifies verbs associated with more or less ethical choices.
  • 5.2 Lexical Analysis: Many significantly associated verbs do not evoke clear normative meanings, and “wanting” leans more ethical less than 60% of the time.The analysis therefore finds that lexical associations are not uniformly strong moral indicators.
  • 5.2 Lexical Analysis: Topic analysis shows that interpersonal relationships feature heavily in DILEMMAS, alongside retail and work interactions.The topics are learned with five-topic Latent Dirichlet Allocation and represented by their top words.

6 Related Work

Related work spans AI safety, machine ethics, top-down and bottom-up moral reasoning, and narrower NLP studies of ethical behavior. SCRUPLES differs by predicting communities’ ethical reactions to real-life scenarios.

  • Prior work: AI safety and machine ethics research has long examined the need for machines to understand or implement ethical behavior.The field remains fragmented and interdisciplinary, with few existing systems understanding human values.
  • Prior work: Machine-ethics approaches commonly divide into top-down systems that specify ethical behavior and bottom-up systems that learn from interactions or examples.Top-down work often uses symbolic methods, while bottom-up work commonly uses supervised or reinforcement learning.
  • NLP research: NLP research has often focused on narrower domains such as hate speech, fairness, and bias, although some work addresses general ethical understanding.Other approaches use moral foundations theory, normative representations, or curated ethical-response corpora.
  • SCRUPLES: SCRUPLES emphasizes predicting the ethical reactions of communities in response to real-life scenarios.This distinguishes its data-driven task from prior work described in the related literature.

7 Conclusion

The paper introduces WHO’S IN THE WRONG? and SCRUPLES to model ethical judgments in both simple dilemmas and complex anecdotes. Its Dirichlet-multinomial approach uses all annotations and separates model uncertainty from norms’ controversiality.

  • Conclusion: SCRUPLES combines simple ethical dilemmas with complex anecdotes to study whether models can reproduce basic and challenging ethical judgments.The paper presents the dataset together with the WHO’S IN THE WRONG? task.
  • Conclusion: Dirichlet-multinomial layers improve current techniques by fully using annotations instead of relying only on majority votes.They also separate model uncertainty from the controversiality of norms.
  • Conclusion: The paper introduces BEST, a general estimator for the best score attainable on a classification dataset under a given metric.The authors present BEST as a target for models on tasks with diverse label distributions.
  • Conclusion: Normative understanding remains an important unsolved problem in NLP and AI, motivating future work with the paper’s datasets, models, and methods.The conclusion frames these contributions as a starting point for subsequent research.

Ethics Statement

The paper estimates oracle performance for tasks with diverse ethical judgments and emphasizes that deployment must account for community-specific norms and careful evaluation.

  • Community norms may be inappropriate for other communities, so models and datasets require careful evaluation before deployment.
  • BEST estimates oracle performance from multiple gold annotations when each example’s true label distribution is unknown.The method fits a prior, estimates posterior label distributions, and averages repeated samples to estimate oracle performance.
  • The procedure assumes independent annotations, roughly Dirichlet-distributed true label distributions, and annotation counts independent of labels.

B Dataset Construction

The dataset uses AITA subreddit posts as community-generated accounts of norm violations and judgments, with responses collected through the forum interface.

  • The AITA subreddit provides stories and comments indicating whether community members view the poster as in the wrong.
  • Users can view existing comments before responding, but comments appear only after the response form.

B.2 Extraction

The appendix describes extraction and baseline construction for the anecdotes, including post filtering, action and text recovery, label extraction, and models ranging from stylistic features to Dirichlet likelihoods.

  • Extraction: The pipeline filters subreddit tags by mapping AITA to HISTORICAL and WIBTA to HYPOTHETICAL while discarding META posts.
  • Extraction: 94.4% of stories have successfully extracted actions produced by rules-based normalizers, filters, and regular expressions.
  • Extraction: Original post text is used to prevent leakage from later edits, and posts lacking preserved original text are discarded.
  • Extraction: Labels are extracted from top-level comments using regular expressions matching site initialisms and equivalent textual expressions.
  • Baselines: Baselines test whether label distributions, superficial style, shallow lexical features, or deep neural models can solve SCRUPLES.
  • Baselines: The Dirichlet likelihood uses all annotations rather than only the majority label and separates question controversiality from model uncertainty.

C.5 Alternative Likelihoods

The paper compares likelihoods that use annotation distributions rather than majority votes, including soft labels and label counts, and relates these alternatives to annotation structure.

  • Alternative likelihoods: Soft Labels trains with cross-entropy against the average annotation label instead of a hard majority-vote label.
  • Alternative likelihoods: Label Counts treats each annotation as its own example, equivalently using unnormalized label counts.

D The Role of Lexical Knowledge

The appendix analyzes lexical associations between root verbs and ethical labels, and uses topic modeling to characterize DILEMMAS.

  • The first analysis measured associations between root verbs and whether actions were judged less ethical.It used a likelihood ratio comparing word probabilities under less-ethical and more-ethical judgments.
  • The second analysis extracted topics describing the DILEMMAS.Table 14 reports the top 25 words for each of five topics learned through LDA.
  • Statistically significant verb associations were selected with a two-tailed permutation test and Holm-Bonferroni correction at the 0.05 level.The permutation distribution used 100,000 Monte Carlo samples, although some p-values were computed as zero.

E Latent Trait Analysis

The latent trait analysis tests whether binary ethical labels capture a smaller set of underlying concerns and evaluates how well such traits explain annotator responses. The linear model left substantial variation unexplained, while random pairings in DILEMMAS limited interpretability.

  • Motivation: Binary labels simplify ethical judgments, but conflicting concerns and reasonable disagreement motivate modeling richer latent structure.The analysis examines whether binary right-versus-wrong labels provide a sufficiently precise representation of complex norms.
  • Method: Latent trait analysis models categorical-response dependence with logistic regression on a Gaussian latent variable, fitted by maximum likelihood after marginalizing Z.The exploratory analysis was motivated by moral foundations theory and tested whether judgments reflect a small number of underlying concerns.
  • Evaluation: Deviance measures goodness-of-fit against a saturated model, while explained deviance compares the candidate model with the fully independent null model.The saturated model represents the best possible fit, assigning observed frequencies to response vectors.
  • Results: The high unexplained deviation across models with varying trait counts suggests annotator disagreement is not well explained by the linear model.The experiment densely annotated 20 development-set questions with 1000 additional annotators.
  • Limitations: Randomly pairing actions in DILEMMAS makes comparisons difficult because actions may come from unrelated contexts and neither may be clearly worse.The authors suggest future annotations capture more nuanced information about actions or how annotators reach conclusions.
Loading 2008.09094v2…