Source-linked AI summary
Aligning AI With Shared Human Values
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, Jacob Steinhardt
TL;DR
The paper addresses the lack of a way to measure whether language models understand general human values. It introduces ETHICS, a benchmark of contextualized moral scenarios spanning multiple ethical theories and commonsense morality, and finds that current models show low but promising performance. The dataset provides a basis for measuring progress toward machine ethics, while its coverage remains limited across cultures and value systems.
Problem
Existing work lacks a way to measure a system’s grasp of general human values across diverse open-world situations.
Method
The authors create the ETHICS dataset and fine-tune pretrained language models to predict moral judgments about contextualized scenarios covering justice, deontology, virtue ethics, utilitarianism, and commonsense morality.
Results
Current pretrained and fine-tuned language models achieve low but promising performance on ETHICS, while models are poor at distinguishing controversial from uncontroversial scenarios.
Takeaways & Limitations
ETHICS makes progress on machine ethics measurable and may support future chatbot steering or regularization of open-world reinforcement-learning agents.
Takeaways & Limitations
The dataset does not capture all value systems or communities, and cultural disagreement about commonsense judgments remains.
Abstract
from arXiv · showhide
We show how to assess a language model's knowledge of basic concepts of morality. We introduce the ETHICS dataset, a new benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality. Models predict widespread moral judgments about diverse text scenarios. This requires connecting physical and social world knowledge to value judgements, a capability that may enable us to steer chatbot outputs or eventually regularize open-ended reinforcement learning agents. With the ETHICS dataset, we find that current language models have a promising but incomplete ability to predict basic human ethical judgements. Our work shows that progress can be made on machine ethics today, and it provides a steppingstone toward AI that is aligned with human values.
1 INTRODUCTION
The paper introduces ETHICS to measure models’ understanding of broad human values in diverse open-world scenarios. It connects existing proto-ethical methods with normative theories and finds low but promising performance.
- Motivation: Existing approaches address narrow facets of ethics, while no established method measures a system’s grasp of general human values.Fairness, safety, prosocial behavior, and utility resemble pieces of justice, deontology, virtue ethics, and utilitarianism, respectively.
- Contribution: ETHICS is a dataset for assessing basic ethical knowledge across justice, deontology, virtue ethics, utilitarianism, and commonsense moral judgments.The benchmark uses diverse contextualized natural-language scenarios.
- Novelty: Open-world scenarios require models to connect physical and social facts with their ethical consequences.Examples include how distance changes fire’s pleasantness or pain and why being coughed on is undesirable because it may cause illness.
- Results and implications: The dataset contains over 130,000 examples and measures ethical knowledge rather than loading it into models.The authors suggest the learned representations may later support action selection in reinforcement learning or steer chatbot text.
2 THE ETHICS DATASET
ETHICS evaluates moral judgment through contextualized scenarios spanning multiple normative theories and commonsense morality. Its subtasks test whether models recognize relevant factors such as impartiality, desert, character, duties, and well-being.
- Dataset design: ETHICS uses natural-language scenarios involving interpersonal relationships, everyday events, and thousands of objects to test ethical consequences.Models must distinguish, for example, taking a penny from taking cash in a wallet.
- Normative coverage: The benchmark covers justice, deontology, virtue ethics, utilitarianism, and commonsense moral intuitions.These emphasize impartiality and desert, rules and obligations, character traits, well-being, and combinations of morally salient factors.
- Dataset design: More than 130,000 examples capture everyday intuitions about individual normative factors and all-things-considered commonsense scenarios.Justice examples focus on impartiality and desert, whereas commonsense examples combine many normative factors.
- Justice: Justice subtasks classify whether explanations of unequal treatment or claims about desert, merit, and entitlement are reasonable.Workers create contrast sets with reasonable and unreasonable claims, and models perform binary classification.
- Virtue Ethics: Virtue Ethics presents scenarios and candidate traits, asking models whether each trait is exemplified by the character.The dataset includes almost 40K scenario-trait pairs created through counterfactual augmentation.
- Deontology: Deontology tests whether purported exemptions and role-based responsibilities are reasonable under rules and constraints.The Roles component contains around 25K examples across the two deontological subtasks.
3 EXPERIMENTS
The experiments evaluate ETHICS with pretrained language models across normal and adversarially filtered scenarios, finding low but promising performance and partial utility-based moral separation.
- Training: Fine-tuned Transformer models and few-shot GPT-3 are evaluated across ETHICS tasks, alongside a GloVe word-averaging baseline.The experiments include BERT-base, BERT-large, RoBERTa-large, ALBERT-xxlarge, GPT-3, and a simple baseline.
- Results: Models perform substantially worse on the adversarially filtered Hard Test set than on the normal Test set.Table 2 reports Test and Hard Test percentages, with Hard Test created through adversarial filtration.
- Results: Average performance remains below 50% on Hard Test, although larger models trained on more data generally perform better overall.RoBERTa-large improves over BERT-large, and BERT-large improves over BERT-base; GPT-3 is competitive on Hard Test but worse on normal Test.
- Utility Function Analysis: RoBERTa-large assigns higher utility to some pleasant scenarios and lower utility to some unpleasant scenarios, but these values are not ground truth.Examples include an award scenario at 8.8, an admired sculpture at 6.9, and rewiring electricity at -15.1.
- Utility Function Analysis: In the Trolley Problem, the model prefers pulling the lever, assigning utility −4.6 versus −7.9 for not pulling it.This preference matches the utilitarian choice described in the experiment.
- Moral Uncertainty and Disagreement Detection: Models are poor at detecting moral disagreement from predictive uncertainty, with AUROC values of 58% for BERT-large, 69% for RoBERTa-large, and 56% for ALBERT-xxlarge.The contentious dataset contains about 1K Commonsense Morality examples, and 50% AUROC represents random chance.
4 DISCUSSION AND FUTURE WORK
The discussion places ETHICS within broader approaches to value alignment while emphasizing that shared human values are culturally incomplete and difficult to implement. Future work should expand the represented value systems and communities.
- Value Learning: Value learning approaches address risks including reward hacking and unintended instrumental goals when systems pursue incompletely specified objectives.The paper contrasts formal bottom-up value learning with empirical approaches such as inverse reinforcement learning and recommender systems.
- Law: Rule-based laws and regulations may be insufficient for reliably constraining diverse AI failure modes and negative externalities.The discussion notes that a laundry list of rules may be intractable for systems operating across varied situations.
- Fairness: The justice task tests impartiality in everyday contexts rather than reducing justice to simple mathematical fairness constraints.Earlier fairness criteria are described as mutually incompatible, inappropriate in some situations, and potentially unintuitive to annotators.
- Deciding and Implementing Values: ETHICS does not represent all value systems or communities: Indian annotators reached 93.9% accuracy on Commonsense Morality, indicating possible cross-cultural disagreement about ground truth.The authors call for more stakeholders and more diverse and individualized values.
- Future Work: Future work could add legal knowledge, cross-cultural entitlements, multidimensional well-being, emotional responses, and additional aspects of justice.These directions are presented as extensions of the dataset’s coverage of ethical concepts.
A.1 CONSENSUS
The consensus procedure uses independent relabeling and retains examples or traits only when annotators meet task-specific agreement thresholds.
- Consensus: Examples were independently relabeled by separate MTurk workers after initial collection.This creates a second annotation pass for consensus filtering.
- Consensus: Justice, Deontology, and Commonsense Morality examples were retained when at least 4 of 5 MTurkers agreed.The threshold is applied separately to each example.
- Consensus: Virtue Ethics traits were retained only when all 3 MTurkers agreed, while Utilitarianism rankings were relabeled by 7 MTurkers.The annotation requirements differ by task.
- Consensus: The cleaned number of examples for each task is reported in Table 1.The passage identifies Table 1 as the source of post-cleaning counts.
A.2 LONG COMMONSENSE MORALITY
Long Commonsense Morality examples were collected from AITA and filtered to reduce sexual, politicized, edited, or otherwise potentially misleading content before judgments were counted.
- Long Commonsense Morality: Examples came from the AITA subreddit, with highly sexual or politicized items removed.Edited examples were also excluded from the Test and Test Hard sets to avoid giveaway information.
- Long Commonsense Morality: Judgment counts were based on comment upvotes for clear verdicts about whether the poster was in the wrong.The procedure added upvotes to the corresponding judgment count and addressed rare negative totals by rounding.
A.3 ADVERSARIAL FILTRATION
The dataset uses adversarial filtration to remove easy test examples and reduce reliance on spurious cues. It repeatedly evaluates candidate examples with two models, selecting the hardest examples for the filtered test set.
- Adversarial filtration removes easy examples from the test set to reduce spurious cues.
- A distil-BERT and distil-RoBERTa ensemble evaluates candidate examples after repeated development/test splits.The procedure uses five different splits and averages losses across both models.
- Examples with the highest average loss are selected as the hardest test examples.For tasks evaluated over sets, losses are averaged over each set before ranking.
- Counterfactual and contrast-set examples pair similar scenarios with different labels.These constructions help enforce similarity while changing the target judgment.
B EXPERIMENTS
The experiments fine-tune and prompt language models across ETHICS tasks, while also examining utility rankings, model size, and error patterns. Results include slowly improving performance with scale but sensitivity to framing and extraneous details.
- Models are fine-tuned for binary classification on Justice, Duty, Virtue Ethics, and Commonsense Morality.The reported search varies learning rate, batch size, and epochs for each architecture.
- RoBERTa-large partially separates pleasant from unpleasant open-world states, but its utility rankings mix reasonable and unreasonable assignments.The utility values reflect the model’s learned utility function rather than ground-truth values.
- BERT-base correctly classifies some clear moral examples but changes its judgments with framing, wording, and superfluous details.For example, closely related statements receive different wrongness probabilities.
- Model size improves average ETHICS performance, although the increase is slow.
- GPT-3 is evaluated with few-shot prompts, including arbitrary 1-to-10 ratings for independently processed Utilitarianism scenarios.The numerical ratings are not ground truth values; the model mimics a rating system.
C CROSS-CULTURAL AGREEMENT
The cross-cultural relabeling study found high agreement between Indian annotators and the final U.S. labels, while identifying idioms and misunderstandings as possible sources of disagreement.
- 93.9% agreement was observed between the Indian relabeling and the final U.S. dataset labels.
- Some annotation differences may reflect cultural differences, but the authors suspect many arose from idioms and annotator misunderstandings.
- The authors propose collecting annotations from more countries and groups in future work.
D DATASHEETS
The datasheet section documents the ETHICS dataset’s motivation, provenance questions, funding, and distribution status. It states that the dataset evaluates models’ understanding of basic shared human values and will be publicly distributed.
- The ETHICS dataset was created to evaluate how well models understand basic shared human values.
- The datasheet directs readers to the main document for information about who created the dataset.
- The datasheet directs readers to the main document for funding and grant information.
- The datasheet records no additional comments.
D.2 COMPOSITION
The ETHICS dataset contains text scenarios across multiple moral-evaluation tasks, with labels, contrast relationships, and recommended splits. It is crowdsourced and partly Internet-derived, with documented collection, preprocessing, and privacy characteristics.
- D.2 COMPOSITION: The dataset contains raw-text scenarios representing several task formats, with more than 130K examples overall.Each task has its own scenario format, and the dataset provides labels for every scenario except ambiguous long Commonsense Morality examples.
- D.2 COMPOSITION: Examples may encode relationships through shared traits or low-edit-distance contrast sets.These relationships are explicitly indicated for Virtue Ethics examples and other contrastive scenario sets.
- D.2 COMPOSITION: Each task includes Development, Test, and Hard Test sets, with adversarial filtering used to remove spurious cues from the Test set.Before adversarial filtering, the Test set can be used for hyperparameter selection and preliminary accuracy estimation.
- D.2 COMPOSITION: The dataset is fixed and self-contained despite partially relying on Internet-scraped data, and it received IRB approval.Crowdsourced examples came directly from MTurkers, whereas long Commonsense Morality examples came directly from Reddit.
- D.2 COMPOSITION: Because long Commonsense Morality examples were publicly posted online, users who posted corresponding examples may be identifiable.The dataset documentation reports no confidential or sensitive data, but notes this possible indirect identification risk.
- D.2 COMPOSITION: Data collection used Amazon Mechanical Turk for crowdsourcing and PRAW for Reddit scraping, with crowdsourced verification of labels for crowdsourced scenarios.Most data was collected through or contracted via Mechanical Turk; examples were collected in Spring 2020, while long Commonsense Morality examples covered subreddit posts available through collection.