Source-linked AI summary
Learning Norms from Stories: A Prior for Value Aligned Agents
Spencer Frazier, Md Sultan Al Nahian, Mark Riedl, Brent Harrison
TL;DR
Value-aligned agents must pursue goals beneficial to humans, yet values are difficult to specify and traditional imitation or preference learning depends on behavioral observations. This paper learns a normative prior from the naturally occurring Goofus & Gallant comic, trains classifiers on its behavior descriptions, and evaluates transfer to unrelated tasks. Transformer models classify the comic’s behavior accurately and transfer without new normative labels, while a small amount of target-task supervision brings performance near the in-domain level.
Problem
Value alignment is difficult because values are hard to specify directly, while traditional approaches infer them from human behavior through imitation or preference learning.
Source: Introduction · p. 1Method
The paper learns a value-aligned prior from Goofus & Gallant, where Gallant and Goofus provide normative and non-normative behavior descriptions for classifier training.
Source: Introduction · p. 1, Goofus & Gallant · pp. 2–3Results
Transformer models transfer adequately to unrelated behavior-classification tasks without normative labels, and small amounts of new-task data bring accuracy near Goofus & Gallant performance.
Source: Introduction · p. 2, Discussion · p. 6, Conclusions · p. 6Takeaways & Limitations
Naturally occurring stories can provide a value-aligned prior that complements imitation learning and supports behavior classification beyond the story corpus.
Source: Introduction · pp. 1–2, Conclusions · p. 6The dataset primarily models Western, specifically American, values, and the models do not currently use context absent from the classified sentence.
Source: Discussion · p. 6
Abstract
from arXiv · showhide
Value alignment is a property of an intelligent agent indicating that it can only pursue goals and activities that are beneficial to humans. Traditional approaches to value alignment use imitation learning or preference learning to infer the values of humans by observing their behavior. We introduce a complementary technique in which a value aligned prior is learned from naturally occurring stories which encode societal norms. Training data is sourced from the childrens educational comic strip, Goofus and Gallant. In this work, we train multiple machine learning models to classify natural language descriptions of situations found in the comic strip as normative or non normative by identifying if they align with the main characters behavior. We also report the models performance when transferring to two unrelated tasks with little to no additional training on the new task.
Introduction
Value alignment is difficult to specify and learn from demonstrations alone, so the paper proposes learning a normative prior from stories, using Goofus & Gallant to classify behavior and transfer that knowledge to new tasks.
Motivation
- Value alignment aims to keep agents’ goals and activities beneficial to humans, but directly specifying values is difficult in open-world settings.Imitation learning also struggles with generalization, demonstration cost, and examples involving negation or inaction.Source: Introduction · p. 1
- Stories provide examples of normative and non-normative behavior that can complement imitation learning with a prior over action quality.The paper defines normativity as conformity to expected societal norms and non-normativity as deviation from them.Source: Introduction · p. 1
Goofus & Gallant
- Goofus & Gallant supplies labeled behavior descriptions because Gallant acts properly while Goofus acts improperly in common everyday scenarios.The comic is presented as a children’s story corpus whose character behavior provides normative and non-normative labels.Source: Introduction · pp. 1–2, Figure 1 · p. 2
Contributions
- The learned prior classifies Goofus & Gallant sentences accurately and transfers adequately to unrelated tasks without new normative labels.With a small amount of labeled data in the new task, performance becomes nearly as strong as on Goofus & Gallant.Source: Introduction · p. 2
Scope
- The dataset models Western, specifically American, values rather than a culturally universal notion of normativity.The authors note that analogous datasets could support alignment to other cultures and societies.Source: Introduction · p. 2
Related Work
Prior value-alignment work learns values from demonstrations, preferences, imitation, or stories, while this paper uses naturally occurring stories to learn a prior that complements policy-learning approaches.
Value-alignment approaches
- Demonstration, preference, imitation, and inverse-reinforcement-learning methods infer latent values from human input but can require costly training signals.The related work motivates strong priors when the relevant signal is sparse or expensive to acquire.Source: Related Work · p. 2
Learning from Stories
- Learning from Stories replaces demonstrations with natural-language stories so agents can extract reward signals for more human-like action sequences.Earlier work showed that agents could learn to avoid non-normative behavior, but its stories were crowdsourced and therefore expensive.Source: Related Work · p. 2
Positioning
- This paper differs from Learning from Stories by learning a value-aligned prior rather than directly learning a value-aligned policy.The prior is intended to complement imitation learning and other demonstration-based approaches by biasing agents toward certain actions.Source: Related Work · p. 2
Preference learning
- Preference-trained language models show that human preference data can shape generation, although sentiment is not equivalent to values.The cited GPT-2 work provides a related example of preference-based language-model training.Source: Related Work · p. 2
Datasets
The paper assembles a labeled Goofus & Gallant corpus and two separately annotated datasets to evaluate transfer of normative-behavior classifiers. These datasets provide everyday, narrative, and science-fiction situations with consensus-based normative labels.
Goofus & Gallant
- The corpus couples everyday situations with explicit contrasting examples, making character behavior a source of normative and non-normative labels.Gallant illustrates proper navigation of a situation, while Goofus illustrates improper behavior that violates social conventions or norms.Source: Goofus & Gallant · pp. 2–3
- Goofus & Gallant provides 1,387 text-only sentences labeled positive for Gallant’s socially acceptable actions and negative for Goofus’s unacceptable actions.The corpus uses recent comics from 1995–2017, replaces character names with pronouns, and splits the data evenly into training and test sets.Source: Goofus & Gallant · p. 3
Plotto Dataset
- Plotto contributes 900 cleaned narrative plot points after removing overtly racist or misogynistic events from its original 1,462 entries.The source offers thousands of predetermined narrative events with character slots and a primary actor.Source: Plotto Dataset · p. 3
- 555 Plotto phrases received consensus normative labels from five classifications per plot point, with items showing more than one dissenting classification discarded.Workers judged whether each behavior would be surprising or unsurprising given its context.Source: Plotto Dataset · p. 3
Science Fiction Summaries Dataset
- The science-fiction summaries dataset contains 800 manually extracted story events, including split sentences when one sentence encoded multiple behaviors.Character names were replaced with pronouns before annotation.Source: Science Fiction Summaries Dataset · p. 3
- After context-sensitive consensus annotation, the science-fiction corpus contains 445 annotated sentences, excluding events with any dissenting vote.Annotators judged normativity within each fictional universe to avoid imposing real-world standards on science-fiction settings.Source: Science Fiction Summaries Dataset · p. 3
Methods
The paper trains binary classifiers on Goofus & Gallant descriptions to identify normative versus non-normative behavior, then evaluates transfer to Plotto and science-fiction narratives. Fine-tuned transformers achieve the strongest classification and transfer performance, while small amounts of task-specific labeling further improve transfer.
Methods
- The study trains binary classifiers that map a single story sentence to normative or non-normative behavior using Bi-LSTM, DPCNN, BERT, and XLNet.BERT-GG and XLNet-GG are fine-tuned on Goofus & Gallant, while base transformer models receive no corpus training.Source: Models · p. 4, Experimental Setup · p. 4
Experimental Setup
- Goofus & Gallant evaluation compares models on a held-out test set using accuracy, precision, recall, F1-score, and Matthews correlation coefficient, alongside human judgments.The human study used 20 participants whose tags were compared with the cartoons’ ground-truth labels.Source: Experimental Setup · p. 4, Experiment 1: Goofus & Gallant… · p. 4
Experiment 1: Goofus & Gallant Classification
- BERT-GG achieves the best results across all reported metrics, improving accuracy by 21.33% over DPCNN.The comparison is reported for Goofus & Gallant classification in Table 2.Source: Experiment 1: Goofus & Gallant… · p. 4, Table 2 · p. 5
- Transformer fine-tuning substantially improves classification over base BERT and XLNet, while contextualized embeddings help represent actions whose meanings depend on situation.BERT-Base and XLNet-Base achieve 61.4% and 60.6% accuracy, respectively, before fine-tuning.Source: Experiment 1: Goofus & Gallant… · pp. 4–5
Experiment 2: Transfer
- The transfer experiment tests models trained on Goofus & Gallant on Plotto and science-fiction story domains using zero-shot and fine-tuned protocols.Zero-shot transfer uses no additional training; fine-tuned transfer adds training on labeled examples from the target task.Source: Experiment 2: Transfer · p. 5, G&G to Plotto Transfer · p. 5
G&G to Plotto Transfer
- On Plotto, BERT-GG outperforms the other models in zero-shot accuracy and MCC, indicating that Goofus & Gallant training transfers to unseen plot events.The models had never seen Plotto plot events before zero-shot evaluation.Source: G&G to Plotto Transfer · p. 5
G&G to Sci-Fi Transfer
- Fine-tuning on Plotto significantly increases performance, with transformer models still drastically outperforming the non-transformer methods.On science-fiction summaries, transformers also perform well zero-shot, though worse than on Plotto, and even small amounts of task-specific normative labeling produce a dramatic increase.Source: G&G to Plotto Transfer · p. 5, G&G to Sci-Fi Transfer · p. 6, Table 4 · p. 6
Discussion
Transformer models trained on Goofus & Gallant transfer normative-behavior classification to unrelated corpora without new normative labels, while limited labels nearly restore in-domain performance. The approach remains bounded by culturally specific norms, sentence-only context, and the possibility of misuse.
Discussion
- High accuracy on unrelated event-description corpora shows transformer models can transfer without normative labels from the new tasks.With a small number of transfer-task labels, accuracy increases to nearly the level achieved on Goofus & Gallant.Source: Discussion · p. 6
- The models reflect Western, specifically American, cultural norms encoded in the comics rather than a universal value system.The authors state that models could be retrained if labeled datasets representing other value systems become available.Source: Discussion · p. 6
- Swapping positive and negative labels could produce an anti-value-aligned model that biases language generators, reinforcement-learning agents, or robots toward harmful behavior.The authors note that additional demonstrations of normative behavior may remediate such bias before a final policy converges.Source: Discussion · p. 6
- Because appropriateness can depend on preceding events, sentence-only classification does not capture context needed for sequential tasks.This limitation is especially relevant to reinforcement-learning agents learning sequential rather than episodic tasks.Source: Discussion · p. 6
Conclusions
The paper uses naturally occurring stories to learn a value-aligned prior that classifies normative behavior and transfers to unrelated tasks. Transformer models perform especially well, and the learned prior can complement imitation, preference, and related value-alignment methods.
Conclusions
- Stories can provide a strong, robust prior for value alignment because their characters embody normative and non-normative behavior.The Goofus & Gallant corpus supplies naturally occurring examples with ground-truth labels for socially normative and non-normative behavior.Source: Conclusions · p. 6
- Machine-learning models trained on Goofus & Gallant accurately classify behavior, with transformer-based language models achieving excellent performance.Source: Conclusions · p. 6
- These models transfer to unrelated event-description tasks without ground-truth labels, complementing learning by demonstration, preference learning, and other imitation-learning techniques.Source: Conclusions · p. 6