PaperlayerView exact revision on arXiv

Source-linked AI summary

The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems

Caleb Ziems, Jane A. Yu, Yi-Chia Wang, Alon Halevy, Diyi Yang

arXiv:2204.03021v1cs.CL

TL;DR

Chatbots can express problematic viewpoints, while moral judgments vary and may conflict, complicating efforts to assess their integrity. MIC captures moral assumptions in prompt-reply exchanges through annotated Rules of Thumb and their attributes. Models generate RoTs for unseen interactions, but still struggle in some scenarios.

  • Problem

    Dialogue systems can express insensitive or incoherent viewpoints, and moral judgments vary or conflict, while existing research lacks a systematic resource for examining those judgments.

    Source: §1 · pp. 1–2

  • Method

    MIC collects prompt-reply pairs annotated with Rules of Thumb and attributes including alignment, consensus, violation severity, and moral foundations.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Models generalize to generating RoTs for unseen interactions, with the best T-5 model achieving ROUGE-L 53 and closely approximating or matching human ratings, though models still struggle in some scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MIC offers an interpretable resource for understanding chatbot moral assumptions and benchmarking integrity while representing multiple moral viewpoints.

    Source: §1 · p. 2, §7 · pp. 8–9

    The judgments reflect a limited population: English-speaking U.S. workers in the 21st century, with prompts drawn from Reddit, which is skewed toward younger or middle-aged males.

    Source: §7 · p. 9

Abstract

from arXiv · show

Conversational agents have come increasingly closer to human competence in open-domain dialogue settings; however, such models can reflect insensitive, hurtful, or entirely incoherent viewpoints that erode a user's trust in the moral integrity of the system. Moral deviations are difficult to mitigate because moral judgments are not universal, and there may be multiple competing judgments that apply to a situation simultaneously. In this work, we introduce a new resource, not to authoritatively resolve moral ambiguities, but instead to facilitate systematic understanding of the intuitions, values and moral judgments reflected in the utterances of dialogue systems. The Moral Integrity Corpus, MIC, is such a resource, which captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs). Each RoT reflects a particular moral conviction that can explain why a chatbot's reply may appear acceptable or problematic. We further organize RoTs with a set of 9 moral and social attributes and benchmark performance for attribute classification. Most importantly, we show that current neural language models can automatically generate new RoTs that reasonably describe previously unseen interactions, but they still struggle with certain scenarios. Our findings suggest that MIC will be a useful resource for understanding and language models' implicit moral assumptions and flexibly benchmarking the integrity of conversational agents. To download the data, see https://github.com/GT-SALT/mic

1 Introduction

Chatbots’ perceived integrity matters to user trust, yet neural dialogue systems can exhibit problematic behavior and existing mitigation approaches lack gold-standard judgments and a systematic way to represent moral variation. MIC provides prompt-reply annotations with Rules of Thumb and attributes, and experiments show models can generate RoTs for unseen interactions while still struggling in some scenarios.

1 Introduction

  • Integrity matters because user trust depends partly on whether agents display competence and align with users’ commonsense reasoning about social and moral values.Neural chatbots can inherit problematic behaviors from large pre-trained language models.Source: §1 · pp. 1–2
  • Existing approaches address toxicity or reward good replies, but dialogue context complicates mitigation and the field lacks gold-standard judgments and a framework for representing differences in moral reasoning.A reply can violate a norm through its framing relative to a prompt, not just through isolated wording.Source: §1 · p. 2
  • MIC contains 38k unique query-response pairs, 99k distinct Rules of Thumb, and 114k sets of structured annotations for open-domain dialogue.RoTs offer interpretable explanations for why chatbot behavior may appear acceptable or problematic.Source: §1 · p. 2
  • Dataset construction filters chatbot responses for reasonable and interesting normative content and uses an annotation scheme that measures ideological variation in moral judgments.The authors report that over 90% of the data reflects reasonable and interesting normative content.Source: §1 · p. 2
  • T5 generates RoTs for previously unseen interactions, achieving ROUGE-L 53 and closely approximating or matching human ratings of well-formedness, relevance, and fluency.The experiments also find that neural models struggle in certain scenarios.Source: §1 · p. 2

2 Related Work

Prior moral datasets and models often address narrative situations or classification, while MIC focuses on minimal chatbot exchanges and descriptive explanations of their implicit moral assumptions. Its interpretable RoTs are also presented as resources for several approaches to chatbot safety.

2 Related Work

  • MIC’s interpretable RoTs can support safety classifiers, alternative responses, controlled-generation steering, and reinforcement-learning penalty models.The authors frame the resource as accommodating different moral viewpoints.Source: §2 · p. 3
  • MIC differs from narrative-based moral resources by examining open-ended prompts and social chatbot replies in minimal chit-chat exchanges.The cited narrative collections include SOCIAL-CHEM-101, MORAL STORIES, and ETHICS.Source: §2 · p. 3
  • Unlike Delphi’s universal judgments about hypothetical narratives, MIC uses RoTs to descriptively explain moral assumptions implicit in foundation models rather than issue authoritative judgments.The authors also identify dialogue as a setting with distinct challenges that prior work had overlooked.Source: §2 · p. 3

3 Moral Annotation Framework

The framework treats a prompt and chatbot reply as a minimal unit for observing moral assumptions. It represents judgments with Rules of Thumb that can explain a reply in more than one, potentially conflicting, way and annotates their attributes.

3 Moral Annotation Framework

  • The framework focuses on a prompt and chatbot response as a minimal dialogue unit for systematically observing chatbot moral assumptions.The authors choose this unit because dialogue trajectories may be long and complex.Source: §3 · p. 3

3.1 Rules of Thumb (RoTs)

  • A Rule of Thumb is a general judgment about right and wrong that pairs a judgment with an action and remains understandable outside its original context.It should explain basic good or bad behavior while retaining enough detail to function as a general rule.Source: §3.1 · pp. 3–4
  • Multiple conflicting RoTs can apply to one prompt-reply pair, and the framework uses them to describe what may be morally right or wrong about the answer.In the marijuana example, the reply can be evaluated through rules about harmful substances, recreational drugs, lawbreaking, and medical treatment.Source: §3.1 · p. 4
  • RoT attributes record reply alignment, perceived global agreement, violation severity, and moral foundations, using six foundations: care, fairness, liberty, loyalty, authority, and sanctity.Annotators can characterize a reply as agreeing with, disagreeing with, or neither aligning with a RoT.Source: §3.1 · p. 4

4 The MORAL INTEGRITY CORPUS

MIC collects and filters open-domain prompt-reply pairs, then has trained workers write RoTs and annotate their attributes. Its collection records worker perspectives, while attribute agreement is fair to moderate and varies with the measurement task.

4 The MORAL INTEGRITY CORPUS

  • The corpus benchmarks chatbot integrity on natural and adversarial prompts using trained MTurk workers and prompt-reply tuples.Workers were trained with definitions, examples, a qualifying test, feedback, and additional quality checks.Source: §4 · p. 4, §4.2 · pp. 4–5

4.1 Collecting Prompt-Reply Pairs

  • After filtering for morally dense, understandable, specific, and relevant replies, the corpus retained 30,880 BlenderBot, 11,521 DialoGPT, and 51,141 GPT-Neo prompt-reply pairs.The process began with 217,700 prompts selected from AskReddit using moral-foundation and subjectivity vocabulary, then generated responses from three chatbots.Source: §4.1 · p. 4

4.2 Annotating RoTs

  • Three annotators independently wrote one RoT for each prompt-reply pair, then rated the RoT’s alignment, global consensus, violation severity, and moral foundations.Workers also supplied a revised answer that was neutral or aligned with their RoT, intended to support alternatives to immoral content.Source: §4.2 · p. 5
  • The worker pool was primarily liberal, with conservative-leaning workers comprising 25% of workers and authoring 22% of annotations.Workers reported political leaning and completed a moral questionnaire; liberal-leaning workers emphasized Care and Fairness more than other foundations, while conservative-leaning workers valued them more evenly.Source: §4.2 · p. 5
  • Agreement across four RoT attributes was fair to moderate, with intraclass correlations from 0.42 to 0.72 among 186 raters.Consensus and severity had lower Krippendorff’s alpha, which the authors expect because raters may calibrate the five-point scales differently.Source: §4.2 · p. 6

5 Models

The models section describes RoT generation from prompt–reply pairs and separate classification of RoT attributes. It also specifies the shared evaluation split and classification strategies.

5 Models

  • The pipeline fine-tunes language models to generate RoTs from questions and replies, then trains separate transformer classifiers to predict RoT attributes using a shared 80-10-10 split.Prompt-reply pairs are kept within a single split.Source: §5 · p. 6

5.1 RoT Generation

  • GPT-2, BART, and T5 model RoT generation; GPT-2 is trained over the full [q; a; r] sequence, unlike the encoder-decoder models.The models maximize a language-modeling objective over tokenized RoTs.Source: §5.1 · p. 6
  • Generation is also compared with random-RoT and SBERT retrieval baselines, while decoding tests greedy search, beam search, and nucleus sampling.For beam search and nucleus sampling, three hypotheses are generated and the highest-scoring one is selected.Source: §5.1 · p. 6

5.2 RoT Attribute Classification

  • Attribute classifiers use RoT alone except for alignment, which takes the RoT and prompt-reply text and predicts agree, disagree, or neither.Severity and consensus use ordinal regression; moral foundations use multi-label classification.Source: §5.2 · p. 7

6 Results

Generative models approach human evaluation levels but still produce many irrelevant RoTs. The results also highlight dialogue-specific challenges and moderate-to-high attribute-classification agreement.

6.1 RoT Generation Results

  • GPT-2 and T-5 beam outputs reached relevance 4.03, fluency 4.67 vs 4.55 gold, and well-formedness 0.88 vs 0.83; best T-5 still scored relevance <2 nearly 28%.The models' well-formedness and fluency averages exceeded the gold-standard averages, while the relevance failure rate shows the task remains unsolved.Source: §6.1 · pp. 7–8

6.2 Unique Challenges in MIC Á

  • GPT-2 trained on SOCIAL-CHEM-101 did not outperform RoT-generation baselines trained on MIC, indicating differences and challenges in transferring to this chit-chat corpus.The transfer evaluation uses the GPT-2 setup and parameters from Forbes et al. (2020).Source: §6.2 · p. 8
  • MIC includes nuanced, multifaceted viewpoints, spontaneous cooperative-principle breaks, pragmatic violations, and adversarial prompts that elicit otherwise unexpected moral viewpoints.An answer may be morally wrong in its dialogue context even when it is not immoral in isolation.Source: §6.2 · p. 8

6.3 Attribute Classification Results

  • ALBERT showed moderate-to-high correlations for severity and consensus, moderate alignment reliability, and weaker moral-foundation classification excluding minority Sanctity labels.The reported values are r = 0.59 and r = 44 for severity and consensus, F1 = 76.0 for alignment, and F1 = 40.8 for six-way moral foundations.Source: §6.3 · p. 8

7 Discussion and Conclusion

MIC provides a large corpus for examining chatbot moral assumptions and finds that models can generate RoTs for unseen interactions at near-human quality, though irrelevant outputs remain common. Its judgments reflect a geographically and demographically bounded collection.

7 Discussion and Conclusion

  • MIC contains 38k chatbot replies to human-authored prompts, annotated with 99k RoTs; across 114k total prompt-reply pairs, it has 15k duplicate RoTs.The paper describes the task as rich and challenging.Source: §7 · p. 8
  • The best models generate RoTs for unseen interactions at near-human quality, but still produce irrelevant RoTs nearly 28% of the time.The reported near-human comparison concerns the quality, fluency, and relevance of generated RoTs.Source: §7 · pp. 8–9
  • The corpus is limited to English-speaking workers in the 21st-century United States and Reddit prompts skewed toward younger or middle-aged males.The paper notes that moral judgments can shift across cultures, regions, and time.Source: §7 · p. 9

Ethics

The paper frames MIC as a collection of subjective moral viewpoints, not a universal ethical system or source of moral advice. Its ethics section also specifies deployment boundaries and describes supporting data-generation and classification procedures.

Ethics

  • MIC treats RoTs as subjective, discrete intuitions rather than a cohesive, universally binding ethical system.The paper models subjectivity through Global Consensus and Violation Severity and notes that judgments vary across cultures and time.Source: Ethics · p. 9
  • The resource is intended for research, and Moral Transformer judgments are explanations of how some people may interpret chatbot utterances, not moral advice.Users must complete a Data Use Agreement linked in the project repository.Source: Ethics · p. 10
  • The annotation study received internal review board approval and warned workers that examples could be disturbing, profane, or racist.Workers were told they could skip a task or take a break if uncomfortable.Source: Ethics · p. 10

A.1 Co-opting GPT-Neo as a Chatbot

  • GPT-Neo is a general-purpose autoregressive model co-opted as a chatbot with a conversation prompt and generated replies ending at a line break, speaker change, or end token.The setup fills the two speaker placeholders with randomly selected names from the 2018 top-names list and replaces the question placeholder with the prompt.Source: §A.1 · p. 16

A.2 RoT Attribute Classification

  • Violation Severity and Global Consensus classifiers were tuned by optimizing mean squared error.The passage does not specify further details of this tuning procedure.Source: §A.2 · p. 16

B Chatbot Response Filtering

The filtering process selects prompt-reply pairs that are both sufficient and moral, using classifier judgments trained from worker labels. Its labeling rules require unanimous agreement for sufficiency but allow one worker to identify moral content.

B Chatbot Response Filtering

  • A response is sufficient only if it is understandable, specific, and relevant, while moral content means someone could judge an expressed opinion, idea, or behavior as right or wrong.The filtering step targets a high density of both sufficient and moral content.Source: B Chatbot Response Filtering · p. 16
  • ALBERT-base-v2 sentence-pair classifiers were trained on binary sufficiency and moral-content labels assigned to sampled prompt-reply pairs.For each chatbot, two MTurk workers judged 1,000 randomly sampled pairs.Source: B Chatbot Response Filtering · p. 16
  • Sufficiency labels required both workers to agree, whereas a single worker's judgment was enough to label a response as containing moral content.The latter rule reflects the study's interest in whether anyone might identify moral content, rather than whether annotators reached consensus.Source: B Chatbot Response Filtering · p. 16
  • The selected model was tuned on a 60-20-20 split and evaluated on the test split; pairs scored above 0.5 for both labels were retained.The table caption identifies test-set precision, recall, and F1 as the reported classifier measures, while the authors call performance reasonably sufficient for filtering.Source: B Chatbot Response Filtering · p. 16, Table 4 · p. 16

C Moral Foundations

The paper adopts six moral foundations and supplies worker instructions for creating and labeling RoTs. Those instructions combine a moral judgment and action with a general rule that remains understandable outside its originating context.

C Moral Foundations

  • The framework uses Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, Sanctity/Degradation, and Liberty/Oppression, with working definitions expressed in terms of motives.The authors did not include non-standard foundations because an early exploration found them unrepresented in the data.Source: C Moral Foundations · pp. 16–17

D.1 RoT Instructions

  • A RoT should express a basic moral or social rule, contain a judgment and an action, and remain understandable out of context.The instructions distinguish moral judgments from bare opinions or encyclopedia facts.Source: §D.1 · p. 17
  • The rule should be specific enough to connect to its situation without relying on vague wording or details unique to that exact scenario.The example guidance contrasts a contextually grounded general rule with rules that are too vague, too context-specific, or overly detailed.Source: §D.1 · p. 17
  • The acceptable example turns a chatbot's response about laughing at parents' death into a judgment, rather than merely repeating the scenario.The instructions contrast it with overly broad, unclear, or excessively scenario-specific alternatives.Source: §D.1 · p. 17

D.2 Moral Foundations Instructions

  • Workers classify each RoT using one or more of the six moral-foundation dimensions and their supplied definitions and examples.The annotation-instructions section introduces the guidance given to workers.Source: §D.2 · p. 17, D Annotation Instructions · p. 17

E.1 Qualification Test

Annotation quality was controlled through a qualifying test, automated submission checks, and staged human review. Workers received feedback during a small staging round before entering the larger main stage, where RoT quality guided inspection.

E.1 Qualification Test

  • To access staging, workers had to be located in the United States and answer at least six of seven questions correctly about Moral Foundations and RoT criteria.Test examples included matching RoTs to Moral Foundation labels and identifying an RoT that followed the response instructions.Source: §E.1 · p. 18

E.2 Automatic Quality Checks (Scripting)

  • Scripts checked grammar and spelling, minimum three-word counts for responses and RoTs, response–RoT distinctness, and repeated RoT bigrams.The maximum frequency of any RoT bigram had to be below three.Source: §E.2 · p. 19

E.3 Manual Quality Control

  • During staging, an author inspected every submission, gave feedback on errors, and granted main-stage access after four consecutive correct HITs.The staging batch contained approximately 100 HITs, compared with approximately 1,000 HITs in the main stage.Source: §E.3 · p. 19
  • In the main stage, inspectors reviewed RoTs as an indicator of overall annotation quality, rejecting poor work and blocking workers after repeat rejections.The authors report that strong RoTs generally accompanied reasonable attribute annotations; inspectors ignored other fields during this stage.Source: §E.3 · p. 19

Related papers

Loading 2204.03021v1…