Source-linked AI summary
Reducing Non-Normative Text Generation from Language Models
Xiangyu Peng, Siyan Li, Spencer Frazier, Mark Riedl
TL;DR
Language models trained on diverse corpora can generate non-normative descriptions of behavior. This paper fine-tunes GPT-2 using a normative classifier as a reinforcement-learning reward signal. The technique reduced non-normative generation across datasets, while its normative-classifier results are bounded by the cultural scope of the classifier’s training source.
Problem
Language models trained on mixed corpora can generate non-normative descriptions, and the paper asks whether a normative classifier can reduce that output.
Source: §1 · pp. 1–2Method
The authors fine-tune GPT-2 with a sentence-level normative classifier that supplies reward and punishment values for generated text.
Source: §3 · p. 4Results
27–61%: classifier-reward fine-tuning reduced generated non-normative text across datasets.
Source: §5 · p. 7Takeaways & Limitations
The results support policy-gradient fine-tuning as an approach to reducing non-normative generation, described by the authors as teaching a language model to censor itself.
Source: §5 · p. 8, §6 · p. 8The normative-classifier results reflect Western, particularly American, mainstream ideals, and the authors cannot conclude they would hold with classifiers trained on other sources.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Large-scale, transformer-based language models such as GPT-2 are pretrained on diverse corpora scraped from the internet. Consequently, they are prone to generating non-normative text (i.e. in violation of social norms). We introduce a technique for fine-tuning GPT-2, using a policy gradient reinforcement learning technique and a normative text classifier to produce reward and punishment values. We evaluate our technique on five data sets using automated and human participant experiments. The normative text classifier is 81-90% accurate when compared to gold-standard human judgments of normative and non-normative generated text. Our normative fine-tuning technique is able to reduce non-normative text by 27-61%, depending on the data set.
1 Introduction
The introduction previews a classifier-reward fine-tuning approach to reducing non-normative language generation and reports reductions across datasets. It frames the work around language models trained on mixed internet text and asks whether a normative classifier can steer their output.
1 Introduction
- Language models trained on mixed internet and book corpora can generate non-normative descriptions, including harmful behaviors and biased or racist text.The training data mix material from groups with different norms and intentionally non-normative speech.Source: §1 · p. 1
- The paper asks whether Frazier et al.’s normative classifier, proposed as a value-aligned prior, can reduce language models’ generation of non-normative behavior descriptions.The authors distinguish this question from the prior work’s speculation that the classifier could bias agent output toward more normative behavior.Source: §1 · p. 2
- Because large normative exemplar corpora may be unavailable, the authors use reinforcement learning with the normative classifier as a non-differentiable reward function.Reward is back-propagated according to the generated text’s degree of non-normativity.Source: §1 · p. 2
- 27–61%: classifier-reward fine-tuning reduced generated non-normative text across datasets, with experiments also testing negative-sentiment and toxic-language classifiers.The evaluation includes automated and human-participant experiments.Source: §1 · p. 2
2 Background and Related Work
The background reviews normative-prior classifiers, language-model fine-tuning approaches, and datasets used to test the proposed technique. It distinguishes sentence-level normative classification from sentiment or toxicity signals and motivates evaluation across varied text domains.
2.1 Value Alignment and Normative Priors
- Frazier et al. (2019) trained a binary classifier on Goofus & Gallant comics to distinguish normative from non-normative text and reported zero-shot and few-shot transfer, but did not directly test its effect on language-model behavior.The comics provide naturally labeled examples of behavior framed as proper or deviant within the society they represent.Source: §2.1 · p. 2
2.2 Language Model Training & Fine-Tuning
- Unlike preference-based reward learning, the proposed work uses a normative classifier as a prior for fine-tuning GPT-2; PPLM instead uses fixed prefixes and word-by-word classification.The authors argue their approach assesses normativity over entire sentences, including cases where context matters.Source: §2.2 · p. 3
2.3 Datasets
- The story datasets span science-fiction summaries, Plotto plot points, and ROCstories, covering fictional actions and more mundane events.The science-fiction corpus contains 2,276 stories and presents transfer to novel entities; ROCstories contains 52,666 five-sentence everyday-life stories.Source: §2.3 · p. 3
- IMDb, Yelp, and Amazon reviews support sentiment-classifier experiments, while toxicity classifiers test whether the technique works beyond the normative classifier.Sentiment is used as a surrogate despite a stated distinction between sentiment and normativity; toxicity is described as a subset of non-normativity.Source: §2.3 · p. 3
3 Normative Fine-Tuning
Normative fine-tuning augments GPT-2’s token-level cross-entropy objective with a sentence-level punishment derived from a classifier. The punishment is scaled relative to word loss, then back-propagated through the model’s output logits.
3 Normative Fine-Tuning
- The method adds a classifier-derived sentence punishment to GPT-2’s cross-entropy loss, since normativity is assessed over a whole sentence.The classifier supplies one numeric reward value per generated sentence.Source: §3 · p. 4
- The punishment scales with average word loss; its strength is controlled by ρ, while β decreases with training iterations to reduce the back-propagation step size.A normative classification yields near-zero punishment, whereas a non-normative classification yields higher sentence loss.Source: §3 · p. 4
- A generated continuation is classified as normative or non-normative, and the resulting punishment is combined with word loss to form sentence loss.The sentence loss is averaged into token-level loss and applied to the continuation’s logits before back-propagation into GPT-2.Source: §3 · p. 4
- The same input sentences are reused during fine-tuning to limit drift from the original dataset, while generated outputs and classifier rewards can change across iterations.The pipeline back-propagates loss through GPT-2’s output logits.Source: §3 · pp. 4–5, Figure 1 · p. 4
4 Experiments
The experiments validate the normative classifier on generated continuations, test reward-based fine-tuning, and assess whether other classifiers can be used with the technique.
4.1 Experiment 1: Replication of the Normative Classifier
- Classifier accuracy decreased on generated continuations but remained on par with accuracy on test-corpus sentences and Frazier et al. (2019), indicating the shift during generation was likely inconsequential.Source: §4.1 · p. 5
4.2 Experiment 2: Decreasing Non-Normative Generation
- Experiment 2 adapted 117M GPT-2 to ROCstories, sci-fi, and Plotto, then applied reward-based fine-tuning and evaluated generated text with classifier and human labels.Because the datasets were small and GPT-2 easily overfit, the experiments fine-tuned only one of its 12 attention heads; at least three crowd workers labeled each sentence.Source: §4.2 · pp. 5–6
- Across ROCstories, sci-fi, and Plotto, classifier- and human-rated non-normative continuations decreased after reward fine-tuning.For ROCstories, classifier rates fell from 64% to 26% (59% decrease) and human labels from 58% to 22% (61% decrease); for sci-fi, classifier rates fell from 35% to 15% (55% decrease) and human labels from 26% to 18% (31% decrease). For Plotto, classifier rates fell from 81% to 59% (27% decrease) and human labels from 72% to 53% (27% decrease).Source: Table 2 · p. 6, §4.2 · p. 6
- Plotto models generated more non-normative continuations and were harder to shift than sci-fi models, while fine-tuning left perplexity steady.The sci-fi models had the lowest non-normative generation frequency, which the method could reduce further.Source: §4.2 · p. 6
4.3 Experiment 3: Other Classifiers
- Using sentiment and toxicity classifiers also reduced negative-sentiment and toxic continuations, supporting the technique's operation with classifiers beyond the normative classifier.For GPT-senti-ext, negative sentiment fell from 76% to 42% (45% reduction), and for GPT-senti-bal from 36% to 29% (17% reduction); toxic language fell from 57% to 30% (46% reduction) for GPT-toxic-ext and from 33% to 28% (13% reduction) for GPT-toxic-bal.Source: §4.3 · p. 7
- Sentiment and normativity labels matched for 53.08% of sampled sentences, while 46.92% had conflicting labels.Source: §4.3 · p. 7
5 Discussion
The discussion reports that reward-based fine-tuning reduces non-normative generation, but outcomes vary by dataset and interpretation is bounded by the classifier’s cultural and contextual scope.
5 Discussion
- Approximately 27–61% reduction in non-normative generation, depending on the dataset.The classifier-based reward nudges GPT-2 toward more normative and neutral descriptions.Source: §5 · p. 7
- Datasets with more non-normative content are more resistant to reduction, and those with little non-normativity initially are naturally more resistant.Source: §5 · p. 8
- Policy-gradient fine-tuning is presented as an alternative or complement to debiasing training data, roughly teaching the language model to censor itself.Source: §5 · p. 8
- Because the classifier was trained on Goofus & Gallant comics, the results reflect Western, particularly American, mainstream norms; transfer from other norm sources is uncertain.Source: §5 · p. 8
- The sentence-level classifier lacks context, so it may miss sentences that appear normative alone but are non-normative in context.Source: §5 · p. 8
- Fine-tuning on Plotto, ROCstories, and sci-fi can yield neutral as well as normative sentences; exclusively normative output would require a ternary classifier and adjusted rewards.Classifier training also requires labeled examples.Source: §5 · p. 8
- The reward rule can be trivially modified to punish normative text, a misuse possibility the discussion explicitly acknowledges.Source: §5 · p. 8
6 Conclusions
The conclusion reports fewer non-normative behavior descriptions from large transformer-based models after data-efficient policy-gradient reinforcement learning. It also notes that the normative classifier reflects Western/American values, while the technique generalizes beyond that classifier.
6 Conclusions
- Large-scale transformer-based language models can generate fewer descriptions of non-normative behavior with data-efficient, policy-gradient reinforcement learning.Source: §6 · p. 8
- The normative classifier is rooted in Western/American norms, while small normative-example datasets can support few-shot classifiers applied to new domains.Source: §6 · pp. 8–9
- Replication with sentiment and toxic classifiers indicates the technique is not specific to one classifier.Source: §6 · p. 9