PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning AI With Shared Human Values

Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, Jacob Steinhardt

arXiv:2008.02275v6cs.CYcs.AIcs.CLcs.LG

TL;DR

Measuring whether AI systems grasp human values remains difficult, despite the importance of moral behavior. The paper introduces ETHICS, a benchmark of diverse ethical scenarios, and finds current models’ ethical-judgment prediction low but promising.

  • Problem

    Researchers lacked a way to measure AI systems’ grasp of general human values, despite wanting AI to behave morally.

    Source: §1 · p. 1

  • Method

    The paper introduces ETHICS, a dataset of diverse contextualized scenarios spanning justice, deontology, virtue ethics, utilitarianism, and commonsense moral judgments.

    Source: §1 · p. 2

  • Results

    Pretrained and fine-tuned language models show low but promising performance on ETHICS, indicating that progress on this problem is feasible today.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ETHICS provides a way to measure ethical knowledge in models, but not to load that knowledge into them.

    Source: §1 · p. 2

    The dataset would benefit from representing more value systems and communities, since annotator results suggest disagreement about moral ground truth across cultures.

    Source: §4 · p. 9

Abstract

from arXiv · show

We show how to assess a language model's knowledge of basic concepts of morality. We introduce the ETHICS dataset, a new benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality. Models predict widespread moral judgments about diverse text scenarios. This requires connecting physical and social world knowledge to value judgements, a capability that may enable us to steer chatbot outputs or eventually regularize open-ended reinforcement learning agents. With the ETHICS dataset, we find that current language models have a promising but incomplete ability to predict basic human ethical judgements. Our work shows that progress can be made on machine ethics today, and it provides a steppingstone toward AI that is aligned with human values.

1 INTRODUCTION

ETHICS addresses the lack of a way to measure broad moral understanding by benchmarking models on diverse open-world ethical scenarios. Models show low but promising performance.

Motivation and related work

Normative ethics framing

  • Fairness, safety constraints, prosocial imitation, and preference learning form proto-ethical facets of justice, deontology, virtue ethics, and utilitarianism.These are small pieces of broader normative theories rather than comprehensive accounts of moral behavior.Source: §1 · pp. 1–2

Dataset contribution

  • ETHICS covers contextualized natural-language scenarios across justice, deontology, virtue ethics, utilitarianism, and commonsense moral judgments.The dataset is designed to assess basic ethical knowledge and common human values across diverse open-world scenarios.Source: §1 · p. 2

Open-world scenario design

  • Pretrained models must connect basic world facts to human values, since contextualized scenarios make the same phenomena pleasant or harmful depending on circumstances.For example, fire’s heat varies with distance, while coughing can expose others to illness.Source: §1 · p. 2

Results and potential applications

  • Pretrained NLP models fine-tuned on ETHICS show low but promising performance; its over 130,000 examples measure, rather than load, ethical knowledge.The benchmark may support future work on selecting good actions in open-world or reinforcement-learning settings and steering chatbot text.Source: §1 · p. 2

2 THE ETHICS DATASET

ETHICS uses natural-language scenarios to assess moral judgments across justice, deontology, virtue ethics, utilitarianism, and commonsense morality. Its diverse situations require connecting facts about the world to their ethical consequences.

2 THE ETHICS DATASET

  • Examples are designed to be clear-cut, relabeled to exclude low-agreement cases, and written by qualified workers; collection focuses on English speakers in the US, Canada, and Great Britain.The authors describe broader language and country coverage as future work.Source: §2 · p. 3
  • More than 130,000 examples span five ethical perspectives, reflecting distinct morally relevant factors and the authors’ aim to draw on enduring ethical theories rather than a single theory.The perspectives are justice, deontology, virtue ethics, utilitarianism, and commonsense moral intuitions.Source: §2 · pp. 2–3, Table 1 · p. 3

2.1 JUSTICE

  • Justice tasks test whether explanations for unequal treatment and claims about what someone deserves, merits, or is entitled to are reasonable; the dataset has approximately 27K examples.Both tasks use sets of reasonable and unreasonable alternatives, with small edit distances for desert claims.Source: §2.1 · p. 4

2.2 VIRTUE ETHICS

  • The virtue ethics task asks models whether a trait is exemplified by a scenario, using counterfactually augmented scenarios and five trait choices per scenario.The dataset contains almost 40K scenario-trait pairs.Source: §2.2 · pp. 4–5

2.3 DEONTOLOGY

  • Deontology tasks test whether a purported exemption from a request or a responsibility implied by a role is reasonable; the dataset includes around 25K examples.The request task involves conflicts of duties or constraints, while the roles task probes the boundaries of role-based responsibilities.Source: §2.3 · p. 5

2.4 UTILITARIANISM

  • Utilitarianism models output a scalar for each scenario, and evaluation checks whether its induced pleasantness ordering matches human preferences, using about 23K scenario pairs.Workers rank scenarios from the usual perspective of a typical person from the US, and sets with substantial disagreement are excluded.Source: §2.4 · pp. 5–6

2.5 COMMONSENSE MORALITY

  • Commonsense morality tests normative judgments about whether a first-person character clearly should not have acted; its scenarios include 10K short and 11K detailed examples.The detailed scenarios are multi-paragraph and require weighing multiple morally salient details; some can exceed usual token limits.Source: §2.5 · p. 6

3 EXPERIMENTS

Experiments on ETHICS find that language models show some ability to predict moral judgments but remain challenged by adversarially filtered examples. The analyses also reveal limitations in utility estimates and in detecting ethical disagreement.

Training and Metrics

  • The experiments fine-tuned BERT-base, BERT-large, RoBERTa-large, and ALBERT-xxlarge on ETHICS, evaluated GPT-3 few-shot, and used a GloVe word-averaging baseline.All tasks used 0/1-loss; accuracy required all related examples to be correct for Justice, Deontology, and Virtue Ethics, while Commonsense Morality used classification accuracy.Source: §3 · p. 7

Results

  • Average performance remained below 50% on Hard Test, where models performed substantially worse than on normal Test; larger fine-tuned models trained on more data performed better overall.The word-averaging baseline performed worst; few-shot GPT-3 was competitive with fine-tuned Transformers on Hard Test but worse on normal Test.Source: Table 2 · p. 7, §3 · p. 7

Utility Function Analysis

  • RoBERTa-large’s learned utility function partially separates pleasant from unpleasant scenarios but exhibits framing sensitivity and scope insensitivity.For example, it assigned different utilities to “My cup is half full” and “My cup is half empty,” and rated everyone on Earth dying above a severe car accident.Source: §3 · pp. 7–8, Figure 6 · p. 8
  • For the Trolley Problem, the model assigned greater utility to pulling the lever to save three lives at the cost of one than to not pulling it.The respective utility values were −4.6 and −7.9.Source: §3 · p. 8

Moral Uncertainty and Disagreement Detection

  • BERT-large, RoBERTa-large, and ALBERT-xxlarge were poor at distinguishing contentious from clear-cut scenarios, with AUROCs of 58%, 69%, and 56%, respectively.The contentious Commonsense Morality set contains about 1K long scenarios on which users were split over the verdict; AUROC uses 50% as random-chance performance.Source: §3 · p. 8

4 DISCUSSION AND FUTURE WORK

The discussion frames ETHICS as a first step toward learning shared human values, while identifying limits in value coverage and implementation. It proposes expanding ethical evaluation and applying value models in broader settings.

Value Learning

  • The (self-)supervised deep-learning approach targets broad, transferable real-world preferences rather than task-specific trajectories or product preferences, potentially guarding against reward hacking and undesirable instrumental goals.The authors contrast this approach with inverse reinforcement learning (Christiano et al., 2017) and recommender systems.Source: §4 · p. 8

Law

  • Rules alone may not adequately constrain AI failure modes, so systems should understand legal concepts and ethical factors to make decisions within the law.Legal permission to act does not by itself mean an action is ethically appropriate.Source: §4 · p. 9

Fairness

  • The justice task evaluates impartiality in everyday contexts using human annotations, rather than imposing mathematical fairness definitions that may be arbitrary, incompatible, or inappropriate.The authors describe this as the first work they know of to use human judgments to evaluate fairness; impartiality is associated with justice theories.Source: §4 · p. 9

Deciding and Implementing Values

  • ETHICS captures pluralistic value systems but needs broader community representation and further work to implement diverse values; Indian annotators scored 93.9% on the Commonsense Morality Test set.Combining deontology with decision theory may require philosophers and technical researchers to collaborate.Source: §4 · p. 9

Future Work

  • Future work should extend ethical models beyond text and test them in multimodal, sequential, specialized, and open-ended chatbot settings.Proposed directions include assessing explanations for partial or unfair statements, steering chatbots away from repugnant outputs (Krause et al., 2020), and testing explainability, adversarial robustness, and distribution shift (Goodfellow et al., 2014; Hendrycks and Dietterich, 2019). The authors also propose testing legal knowledge, cross-cultural entitlements, and predictions of emotions, intent, and well-being (Robeyns, 2017, p. 118; Parfit, 1987, p. 493; Haidt et al., 2003).Source: §4 · p. 9

A CLEANING DETAILS · A.1 CONSENSUS

The dataset examples were relabeled by separate MTurkers and retained only under task-specific agreement rules. The cleaning procedure varied across Justice, Deontology, Commonsense Morality, Virtue Ethics, and Utilitarianism.

A.1 CONSENSUS

  • The relabeling was conducted by MTurkers separate from those who initially collected the examples.Source: §A.1 · p. 14
  • For Justice, Deontology, and Commonsense Morality, examples were retained when at least 4 of 5 relabelers agreed.Source: §A.1 · p. 14
  • For Virtue Ethics, three MTurkers labeled candidate traits, and a trait was kept only when all three agreed.They considered 10 candidates per scenario: one true trait, one from a contrast example, and eight random traits used to form a set of five.Source: §A.1 · p. 14
  • For Utilitarianism, seven MTurkers relabeled rankings for adjacent scenario pairs, and a set was kept when a majority agreed with every adjacent comparison.Source: §A.1 · p. 14
  • The order of rankings shown to Utilitarianism relabelers was randomized to mitigate bias.Source: §A.1 · p. 14
  • The paper reports that Table 1 gives the exact number of examples remaining for each task after cleaning.Source: §A.1 · p. 14

A.2 LONG COMMONSENSE MORALITY

Long Commonsense Morality examples came from AITA, with vote-weighted judgments used to select balanced, high-agreement scenarios and a separate ambiguous subset.

A.2 LONG COMMONSENSE MORALITY

  • Examples came from the AITA subreddit, with highly sexual or politicized cases and examples edited from Test or Test Hard excluded.The exclusions were intended to avoid giveaway information from edited examples.Source: §A.2 · p. 14
  • For each comment with a clear judgment, its upvotes contributed to that judgment’s count; negative total counts were rounded up to zero.Source: §A.2 · p. 14
  • The main set retained scenarios with at least 95% agreement, then subsampled them to balance labels; the ambiguous subset required 50% ± 10% agreement.Source: §A.2 · p. 14

A.3 ADVERSARIAL FILTRATION

Adversarial filtration removes potentially spurious cues by selecting the hardest examples for the test set, using an ensemble of two language models to rank candidates.

A.3 ADVERSARIAL FILTRATION

  • The filtration method trains distil-BERT and distil-RoBERTa on 80% development splits and evaluates them on 20% test splits.The process is repeated five times with different splits, and each candidate’s losses are averaged across both models.Source: §A.3 · p. 14
  • Candidates with the highest average loss become test examples; for set-based tasks, losses are averaged over each set before ranking, followed by consensus cleaning.A sample of the remaining examples or sets is used to form the normal Test set.Source: §A.3 · p. 14

A.4 CONTRAST EXAMPLES

The section describes contrast examples that pair similar scenarios with different labels. Similarity is enforced using task-specific textual constraints.

A.4 CONTRAST EXAMPLES

  • Examples with different labels are collected simultaneously as “counterfactual augmentations” (Kaushik et al., 2020) or “contrast set” examples, while keeping scenarios similar.Source: §A.4 · p. 14
  • For Utilitarianism, some scenario pairs share the same first sentence; for Commonsense Morality, Desert, and Virtue Ethics, adjacent scenarios have a small Damerau-Levenshtein distance.Source: §A.4 · p. 14

B EXPERIMENTS

The experiments examine model performance across ethical tasks, describe fine-tuning and few-shot setups, and analyze utility outputs and moral-classification errors.

Hyperparameters

  • Fine-tuned binary classifiers use task-specific grid searches, while the word-averaging baseline averages fasttext and 100-dimensional GloVe vectors and maps them through an affine classifier or ranker.Grid searches test learning rates of 1×10−5 or 3×10−5, batch sizes of 8 or 16, and 2 or 4 epochs; models report best exact match on Test and adversarially filtered Hard Test sets. Inputs are capped at 64 tokens, except Commonsense Morality at 512; the word-averaging model uses Adam with default learning rate 10−3.Source: B EXPERIMENTS · pp. 14–15

Utility Analysis

  • RoBERTa-large utility values partially separate pleasant from unpleasant states, but rankings are mixed and are learned outputs, not ground truth.The utility function has no reject option for meaningless inputs such as “jdkfjlsd.”Source: Figure 8 · p. 15, Figure 7

Error Analysis

  • BERT-base handles basic wrongdoing but morality judgments shift with framing, wording, and even a character-name change, motivating transparency work.The authors describe these inconsistencies as unexplained and call for work on model transparency.Source: B EXPERIMENTS · p. 15

GPT-3 Sizes

  • 2.7-billion-parameter Ada/Small through 175-billion-parameter Davinci/X-Large show slowly increasing ETHICS average performance with model size.The authors access GPT-3 through the OpenAI API and compare its Small, Medium, Large, and X-Large models.Source: Figure 9 · p. 16, B EXPERIMENTS · p. 16

GPT-3 Prompts

  • GPT-3 few-shot prompts use 32 examples for Justice and Deontology, Virtue Ethics, and short Commonsense Morality, but eight for long Commonsense Morality.For Utilitarianism, GPT-3 rates each scenario from 1 to 10 using an arbitrary, underspecified rating function rather than comparing scenarios directly.Source: B EXPERIMENTS · p. 16

C CROSS-CULTURAL AGREEMENT

India-based annotators’ majority-vote labels agreed with the U.S. dataset’s Commonsense Morality test labels at a 93.9% rate. The authors suggest many disagreements reflect idioms or annotator misunderstandings and call for annotations from more countries and groups.

C CROSS-CULTURAL AGREEMENT

  • India-based annotators’ labels agreed with the U.S. dataset’s Commonsense Morality test labels at 93.9%.Ten MTurkers from India relabeled each short example, with majority vote and random tie-breaking; the authors suspect many disagreements reflect idioms or annotator misunderstandings and propose collecting annotations from more countries and groups.Source: C CROSS-CULTURAL AGREEMENT · pp. 16–17

D DATASHEETS · D.1 MOTIVATION

The ETHICS dataset was created to evaluate models’ understanding of basic shared human values. The authors provide a datasheet following Gebru et al. (2018).

D DATASHEETS

D.1 MOTIVATION

  • The dataset’s stated purpose is to evaluate how well models understand basic shared human values.The passage refers readers to the main body for more detail.Source: §D.1 · p. 17
  • The datasheet asks whether ETHICS was created for a specific task or to fill a specific gap.Source: §D.1 · p. 17
  • The authors say the dataset creators and their institutional affiliation are described in the main document.Source: §D.1 · p. 17
  • The authors refer readers to the main document for information about dataset funding and any associated grant.Source: §D.1 · p. 17
  • The datasheet states that there are no other comments.Source: §D.1 · p. 17

D.2 COMPOSITION … G QUALIFICATION FORMS

The documentation describes ETHICS as a large collection of everyday text scenarios, mostly crowdsourced, with task-specific labels and splits. It also notes limits on the dataset’s scope and representativeness, and provides examples, collection forms, and qualification materials.

D.2 COMPOSITION

  • ETHICS contains more than 130K raw-text everyday scenarios across tasks, with labels except for ambiguous long Commonsense Morality examples.Each task has Development, Test, and Hard Test sets; the Test set is adversarially filtered, while dev sets are too small to load ethical knowledge.Source: §D.2 · p. 17

D.3 COLLECTION PROCESS

  • Crowdsourcing supplied every subtask except long Commonsense Morality, which was scraped from Reddit; MTurk labels were verified through crowdsourcing.Examples were collected in Spring 2020, and the study received IRB approval; Reddit posters were not notified because their examples were public, and no impact analysis was conducted.Source: §D.3 · pp. 18–19

D.4 PREPROCESSING/CLEANING/LABELING

  • Preprocessing and cleaning are described in Appendix A, but the raw data was not saved and the preprocessing software was not available.The dataset was filtered and cleaned to retain high-quality examples with unambiguous labels.Source: §D.2 · p. 17, §D.4 · pp. 19–20

D.5 USES

  • Most examples come from Western countries and crowdsourcing or the Internet, so sample-selection biases may remain despite efforts to collect mostly unambiguous examples.ETHICS is intended for everyday ethical understanding, not moral dilemmas or scenarios with substantial disagreement.Source: §D.5 · p. 20

D.6 DISTRIBUTION; D.7 MAINTENANCE

  • The dataset was to be publicly distributed, with no update plans; its documentation gives details such as the MTurk forms to support related dataset creation.Hosting and contact information are referred to the main document.Source: §D.6 · p. 20, §D.7 · pp. 20–21

E LONG COMMONSENSE MORALITY EXAMPLES

G QUALIFICATION FORMS

Related papers

Loading 2008.02275v6…