PaperlayerView exact revision on arXiv

Source-linked AI summary

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, Noah A. Smith

arXiv:2009.11462v2cs.CL

TL;DR

Pretrained language models can generate toxic language even from innocuous prompts, creating a challenge for safe deployment. The paper introduces RealToxicityPrompts, evaluates detoxification methods, and analyzes pretraining corpora, finding persistent toxic degeneration and toxic or unreliable web content.

  • Problem

    Pretrained language models can generate toxic language, but the risk of toxic degeneration and the effectiveness of mitigation methods require systematic evaluation.

  • Method

    The paper releases a 100K-prompt toxicity testbed, evaluates controllable generation methods, and analyzes toxicity and provenance issues in two web text corpora.

  • Results

    Pretrained models generate toxicity from non-toxic prompts; data- or compute-intensive steering is more effective than simpler methods, but no tested method is failsafe.

  • Takeaways & Limitations

    The findings highlight the difficulty of avoiding toxicity in generation and the need to reconsider content used for language-model pretraining.

  • Takeaways & Limitations

    The study uses an imperfect toxicity measure, covers five language models, and provides only lower-bound corpus-toxicity estimates because of missing metadata and incomplete data coverage.

Abstract

from arXiv · show

Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration. We create and release RealToxicityPrompts, a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widely-used toxicity classifier. Using RealToxicityPrompts, we find that pretrained LMs can degenerate into toxic text even from seemingly innocuous prompts. We empirically assess several controllable generation methods, and find that while data- or compute-intensive methods (e.g., adaptive pretraining on non-toxic data) are more effective at steering away from toxicity than simpler solutions (e.g., banning "bad" words), no current method is failsafe against neural toxic degeneration. To pinpoint the potential cause of such persistent toxic degeneration, we analyze two web text corpora used to pretrain several LMs (including GPT-2; Radford et. al, 2019), and find a significant amount of offensive, factually unreliable, and otherwise toxic content. Our work provides a test bed for evaluating toxic generations by LMs and stresses the need for better data selection processes for pretraining.

1 Introduction

The paper introduces a testbed for measuring toxic degeneration and shows that pretrained language models can generate toxic text from seemingly non-toxic prompts. It evaluates mitigation methods and examines toxic content in pretraining corpora.

  • Pretrained language models can degenerate into toxicity even without explicitly toxic prompts, hindering safe deployment.
  • REALTOXICITYPROMPTS contains 100K naturally occurring sentence-prefix prompts paired with toxicity scores from the PERSPECTIVE API.
  • The study evaluates controllable generation methods for steering language models away from toxic content.
  • More computationally or data-intensive methods are more successful than simpler controls, but even the best steering methods can still generate highly toxic content.
  • Large-scale analyses of GPT-2’s training corpus and its open-source replica find non-negligible toxic, harmful, abusive, unreliable, and provenance-related content.
  • The findings highlight the difficulty of avoiding toxicity and the need to reconsider content used for language-model pretraining.

2 Operationalizing Toxicity

The study operationalizes toxicity with an automated classifier score, while recognizing that the detector and its annotations have important biases and limitations.

  • The study relies on PERSPECTIVE API, an automated tool for detecting toxic language and hate speech in large text corpora.
  • PERSPECTIVE API defines TOXICITY as the probability that a comment is rude, disrespectful, unreasonable, or likely to make people leave a discussion.
  • The toxicity score is the calibrated output of a convolutional neural network trained on proprietary comment data, with an AUC of 0.97.
  • Prompts with TOXICITY ≥0.5 are labeled toxic, while lower-scoring prompts are labeled non-toxic.
  • The detector can exhibit minority-related bias, low annotation agreement, and overreliance on lexical cues such as swearwords and slurs.

3 Out-of-the-Box Generation Toxicity

The paper measures unprompted toxicity in five pretrained autoregressive language models and finds that all can produce highly toxic outputs across repeated generations.

  • The investigation covers five autoregressive Transformer language models: GPT-1, GPT-2, GPT-3, CTRL, and CTRL-WIKI.
  • The models differ in pretraining data and scale, including GPT-2’s 40GB of English web text and GPT-3’s 175B parameters.
  • The study generates up to 20 tokens using nucleus sampling with p = 0.9 unless otherwise noted.
  • The analysis generates 10K spans per model and bootstrap-estimates expected maximum toxicity across repeated samples.
  • All five models exceed toxicity 0.5 within 100 generations, and most exceed maximum toxicity 0.9 within 1K generations.
  • GPT-2 reaches an expected maximum toxicity of 0.65 with 100 unprompted generations, while CTRL-WIKI has significantly lower expected maximum toxicity than the other models.
  • The results suggest that models acquire toxicity from pretraining data, motivating further analysis of those corpora.

4 REALTOXICITYPROMPTS

REALTOXICITYPROMPTS is a 100K-prompt testbed for measuring toxic language generation under naturally occurring contexts. Across five language models, even non-toxic prompts produced toxic continuations at substantial rates.

  • Prompt Creation and Selection: REALTOXICITYPROMPTS contains 100K naturally occurring prompts and continuations scored for toxicity from English web text.Prompts were sampled across four equal-width toxicity ranges and split from their source sentences into prompt–continuation pairs.
  • Prompt Creation and Selection: 22K prompts have TOXICITY ≥0.5, while prompt and continuation toxicity are slightly anti-correlated (r = –0.08, p ≤ 0.001).The dataset’s prompts average 11.7 ± 4.2 tokens, and toxicity is usually confined to one half of a sentence.
  • Prompted Toxicity in Neural Models: The evaluation measures expected maximum toxicity and the probability of generating TOXICITY ≥0.5 at least once across 25 generations.These metrics capture worst-case toxicity and generation frequency, respectively.
  • Prompted Toxicity in Neural Models: All five models reached toxicity probabilities near or above 0.5 for non-toxic prompts.Thus, innocuous contexts still elicited toxic generations, including from CTRL-WIKI.

5 Detoxifying Generations

The paper evaluates data-based and decoding-based methods for steering GPT-2 away from toxicity. All methods reduce toxic behavior, but steering does not completely prevent toxic generations, with DAPT (Non-Toxic), vocabulary shifting, and PPLM performing best among the tested approaches.

  • Detoxification Methods: The study compares further pretraining methods with decoding strategies that alter generation without changing model parameters.GPT-2 is evaluated using REALTOXICITYPROMPTS, with 25 generations per prompt for most methods.
  • Data-Based Detoxification: DAPT continues pretraining on non-toxic or toxic subsets of approximately 150K OWTC documents, while ATCON conditions training and generation on toxicity attribute tokens.The training corpora are disjoint from the prompt data.
  • Decoding-Based Detoxification: Decoding-based methods include vocabulary shifting, word filtering, and PPLM, which respectively boost non-toxic token likelihoods, block listed words, or modify hidden representations using discriminator gradients.Vocabulary shifting uses a toxicity representation for every GPT-2 vocabulary token and sets β = 3.
  • Effect of Controllable Solutions on Generation Toxicity: All proposed techniques reduce toxic behavior, but steering does not completely solve neural toxic degeneration.DAPT (Non-Toxic), vocabulary shifting, and PPLM yield the lowest generation toxicity; PPLM uses 10K prompts because of computational budget.
  • Effect of Controllable Solutions on Generation Toxicity: DAPT (Non-Toxic) is among the most effective methods despite its simplicity, highlighting the importance of pretraining data in neural toxic degeneration.The comparison includes DAPT (Toxic) as a reference for controlling language-model behavior.

6 Analyzing Toxicity in Web Text

The paper analyzes toxicity and provenance in OPENAI-WT and OWTC, finding toxic content alongside links to unreliable news sites and banned or quarantined Reddit communities. These findings characterize problematic material in corpora used for language-model pretraining.

  • Corpus Analysis: The analysis measures toxicity and provenance in GPT-2’s OPENAI-WT corpus and its open-source replica, OWTC.Provenance analysis tracks documents from unreliable news sites and quarantined or banned subreddits.
  • Corpus Construction: OWTC contains about 8M English documents totaling 38 GB, scraped from outbound URLs in Reddit submissions with karma scores of at least 3.The corpus includes domain and subreddit metadata for further analysis.
  • Corpus Overlap: The corpora overlap by about 29%, with at least 2.3M OPENAI-WT documents also appearing in OWTC.The overlap was estimated using large-scale similarity search with locality-sensitive hashing.
  • Toxicity in Web Text: 2.1% of OWTC and 4.3% of OPENAI-WT documents have TOXICITY ≥0.5.Despite OPENAI-WT’s blocklist of offensive subreddits and words, its toxicity rate is twice OWTC’s.
  • Sources of Toxic Content in Web Text: News reliability correlates negatively with toxic-document proportion (Spearman ρ = –0.35), and at least 12% (272K) of rated overlapping documents come from low- or mixed-reliability sites.Low-reliability news sites are less prevalent but contain more toxic documents than higher-reliability sites.
  • Sources of Toxic Content in Web Text: At least 3% (212K) of OWTC documents come from banned or quarantined subreddits, whose shared documents contain substantially more toxicity than those from standard subreddits.Among overlapping documents, at least 63K were shared on such subreddits; GPT-2’s training data includes at least 40K from /r/The Donald and 4K from /r/WhiteRights.

7 Discussion and Recommendations

The paper finds that toxicity remains difficult to eliminate from language-model generation and traces this challenge to both steering limitations and toxic pretraining data. It recommends greater transparency, scrutiny of data curation, and improved toxicity evaluation.

  • Toxicity is prevalent in both neural language generation and the web text corpora used for pretraining.
  • Effectiveness of “Forgetting” Toxicity: Adaptive pretraining lowers unprompted toxic generation, but prompted generations can still remain toxic.
  • Decoding with a Purpose: PPLM is among the most effective tested decoding methods for avoiding toxicity with toxic prompts, although no steering method fully protects models.
  • Choice of Pretraining Data: Pretraining-data analysis is a crucial first step toward understanding toxic, biased, or degenerate model behavior, motivating public release of collection metadata.
  • Choice of Pretraining Data: Reddit-popularity curation can bias whose language and perspectives enter pretraining, creating representational harm.
  • Toxicity-detector bias limits conclusions about detoxification effectiveness, while the study is also limited to five models and provides lower-bound corpus estimates.

8 Related Work

Related work establishes that pretrained models acquire social biases and that prompts can trigger toxic generations. This paper extends that line of inquiry with naturally occurring prompts and broader toxicity evaluation across controlled and uncontrolled models.

  • Prior work shows that large pretrained sentence encoders acquire toxicity and social biases, while autoregressive models also produce incoherent, bland, and repetitive generations.
  • Unlike nonsensical universal adversarial triggers, this work releases naturally occurring web-text prompts that trigger toxicity in GPT-2 and other language models.
  • Compared with templated identity prompts, REALTOXICITYPROMPTS uses 100K naturally occurring prompts, including prompts without identity mentions, and studies broad toxicity rather than sentiment and regard.
  • The study also draws on research into conversational derailment and controllable language-model generation.

9 Conclusion

The paper concludes by introducing a 100K-prompt testbed, evaluating toxic generation and detoxification methods, analyzing toxicity in pretraining corpora, and recommending improved pretraining-data practices.

  • REALTOXICITYPROMPTS is a testbed of 100K prompts for evaluating toxic degeneration in pretrained language models.
  • The framework quantifies toxicity across multiple pretrained models and measures how effectively methods detoxify their generations.
  • The paper analyzes toxicity in two large web-text corpora, including GPT-2’s pretraining corpus, to investigate potential causes of toxic generations.
  • The dataset uses naturally occurring English web-text prompts paired with toxicity scores from the Perspective API.
  • Prompts are stratified across four equally sized toxicity intervals, producing 100,000 sentences with toxic and non-toxic examples.
  • The experiments use pretrained models including GPT-1, GPT-2, CTRL, and their documented training corpora and implementations.

B.2 Detoxification Data

The detoxification experiments compare data-based, token-based, vocabulary-based, filtering, and PPLM strategies for steering generation away from toxicity. Evaluations also examine language-model quality, model size, and social bias patterns in generated text.

  • Detoxification Data: 150K-document non-toxic, toxic, and randomly sampled corpora are created from OWTC, with documents selected from the bottom or top 2 toxicity percentiles for the extreme sets.The corpora are disjoint from REALTOXICITYPROMPTS and split into training and evaluation sets.
  • Detoxification Procedure: ATCON continues GPT-2 pretraining with attribute tokens and prepends <|nontoxic|> during generation to steer outputs away from toxicity.The attribute tokens are added to the vocabulary before continued pretraining.
  • Detoxification Procedure: VOCAB-SHIFT re-weights vocabulary logits using a learned projection from binary toxicity labels to vocabulary-sized adjustments.GPT-2 parameters are frozen while the projection layer is trained on balanced toxic and non-toxic data.
  • Detoxification Procedure: WORD FILTER sets logits to −∞ for tokens that would complete banned words, while PPLM uses a released Jigsaw toxicity classifier for detoxification.Generation generally uses temperature 1 and nucleus sampling with p = 0.9, except for PPLM.
  • Evaluation: All evaluated models achieve GPT-2-like perplexity on OPENAI-WT, suggesting observed toxicity reductions do not weaken language-model quality.Perplexity is measured on an unreleased test set and its non-toxic subset.
  • Further Analyses: GPT-2-small and GPT-2-medium show similar generation toxicity, while profanity and identity mentions in prompts predict corresponding mentions in GPT-2 generations.Prompted profanity correlates with generated profanity at r = 0.169, and minority identity mentions are more likely than majority mentions to recur in generations.
  • Further Analyses: PERSPECTIVE API toxicity scores correlate with likelihood of African American English in prompts and generations, indicating a documented bias concern for the detector.Correlations are r = 0.16 for prompts and r = 0.21 for naturally occurring continuations, both with p < 0.001.

D.2 Further Analyses of OPENWEBTEXT CORPUS and OPENAI-WT

The analyses characterize toxic content in OPENWEBTEXT CORPUS and compare it with OPENAI-WT using domain, subreddit, and overlap analyses. They also report correlations between prompt toxicity and profanity or identity mentions.

  • URLs: Most popular OPENWEBTEXT domains are predominantly news sites, with separate analyses of domains in its toxic subset.
  • Toxicity correlates: Prompt toxicity correlates with profanity (r = 0.43) and, more weakly, minority and majority identity mentions (r = 0.10 and r = 0.04).
  • Subreddits: Toxic documents are analyzed by their source subreddits, including the most common subreddits overall and among toxic documents.
  • Subreddits: Banned or quarantined subreddits have higher proportions of toxic documents than all subreddits in OPENWEBTEXT, using a toxicity threshold of 0.5 across any PERSPECTIVE label.
  • Corpus overlap: Near-duplicate documents between OPENWEBTEXT CORPUS and OPENAI-WT are identified with 5-shingles, LSH candidate generation, and exact Jaccard filtering at 0.9.

D.3 BooksCorpus

The BooksCorpus analysis evaluates toxicity in a random sentence sample and presents additional generation examples. It reports substantial toxicity in the corpus and illustrates generations across models, prompts, and steering methods.

  • Corpus toxicity: 9.1% of the sampled BooksCorpus sentences are considered toxic by the PERSPECTIVE API.
  • Corpus toxicity: The toxicity distribution for 100K random BooksCorpus sentences uses a 0.5 threshold, with estimated toxic-document percentages shown for each corpus.
  • Generation examples: Additional tables provide examples of toxic documents from BooksCorpus and unprompted toxic generations from GPT-1, GPT-2, GPT-3, CTRL, and CTRL-WIKI.
  • Generation examples: The paper compares lowest- and highest-toxicity generations conditioned on four innocuous prompts, alongside examples from different steering models and GPT-2.
Loading 2009.11462v2…