PaperlayerView exact revision on arXiv

Source-linked AI summary

ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, Ece Kamar

arXiv:2203.09509v4cs.CL

TL;DR

Toxicity detectors can mistake minority-group mentions for toxicity while missing implicit hate. ToxiGen addresses this gap with controlled machine generation and adversarial decoding, and fine-tuning on it improves performance on human-written toxicity datasets.

  • Problem

    Toxicity detectors may rely on spurious minority-identity correlations and struggle to detect implicit toxicity, which lacks explicit profanity or slurs.

  • Method

    The paper uses demonstration-based prompting and ALICE classifier-in-the-loop decoding to generate balanced toxic and benign statements about 13 minority groups.

  • Results

    +7–19% performance improvements were obtained on three human-written implicit toxic datasets after fine-tuning classifiers on TOXIGEN.

  • Takeaways & Limitations

    TOXIGEN provides a large, balanced, mostly implicit resource for improving toxicity classifiers and testing their robustness.

  • Takeaways & Limitations

    Toxicity is inherently subjective, motivating more nuanced labeling systems beyond binary detection.

Abstract

from arXiv · show

Toxic language detection systems often falsely flag text that contains minority group mentions as toxic, as those groups are often the targets of online hate. Such over-reliance on spurious correlations also causes systems to struggle with detecting implicitly toxic language. To help mitigate these issues, we create ToxiGen, a new large-scale and machine-generated dataset of 274k toxic and benign statements about 13 minority groups. We develop a demonstration-based prompting framework and an adversarial classifier-in-the-loop decoding method to generate subtly toxic and benign text with a massive pretrained language model. Controlling machine generation in this way allows ToxiGen to cover implicitly toxic text at a larger scale, and about more demographic groups, than previous resources of human-written text. We conduct a human evaluation on a challenging subset of ToxiGen and find that annotators struggle to distinguish machine-generated text from human-written language. We also find that 94.5% of toxic examples are labeled as hate speech by human annotators. Using three publicly-available datasets, we show that finetuning a toxicity classifier on our data improves its performance on human-written data substantially. We also demonstrate that ToxiGen can be used to fight machine-generated toxicity as finetuning improves the classifier significantly on our evaluation subset. Our code and data can be found at https://github.com/microsoft/ToxiGen.

1 Introduction

ToxiGen addresses biased toxicity detection by generating balanced, largely implicit statements about minority groups and using adversarial decoding to challenge classifiers. Human evaluation and downstream experiments support the dataset’s quality and usefulness for improving toxicity detection.

  • Dataset motivation and construction: ToxiGen contains 274,186 toxic and benign statements about minority groups, generated with GPT-3, demonstration-based prompting, and pretrained toxicity classifiers.The dataset targets both subtle toxicity and benign statements that mention minority groups.
  • Dataset motivation and construction: Machine generation enables equal toxic and benign coverage for demographic groups while producing implicit toxicity that is difficult to detect or collect at scale.This approach is presented as an alternative to web scraping, which can encode spurious identity-toxicity correlations.
  • Dataset characteristics: 98.2% of TOXIGEN statements are implicit and contain no explicit profanity, slurs, or swearwords.The dataset is designed to represent toxicity without explicit lexical markers.
  • Adversarial generation: ALICE uses classifier-in-the-loop constrained decoding to steer a language model toward higher or lower toxicity during beam search.Experiments with five public toxicity classifiers produced sentences that fooled the classifiers.
  • Evaluation and impact: 90% of 792 machine-generated sentences could be mistaken for human-written text by annotators.The test statements were rated by three annotators each after filtering training examples for similarity.
  • Evaluation and impact: +7–19% performance improvements were observed when classifiers fine-tuned on TOXIGEN were evaluated on three human-written implicit toxic datasets.The datasets were ImplicitHateCorpus, SocialBiasFrames, and DynaHate.
  • Evaluation and impact: The authors publicly release TOXIGEN, its code, two pretrained models, and human evaluations.These releases support reuse of the dataset and trained models.

2 Implicit Hate Against Minority Groups

Implicit toxicity toward minority groups is difficult to detect because it lacks explicit markers and is often underrepresented in collected data. ToxiGen is designed to address these gaps through controlled, balanced generation.

  • Detection challenges: Implicit toxicity can involve stereotyping and microaggressions, lack profanity or swearwords, and remain difficult to detect or collect at scale.Such language may also have positive sentiment despite being harmful.
  • Why implicit toxicity matters: Implicitly toxic language can psychologically damage targeted groups and reinforce stereotypical or hateful perceptions.The passage links these effects to research on psychological harm and media influence.
  • Dataset bias: Over 93% of Jewish-mentioning examples in one dataset were toxic, encouraging models to exploit spurious minority-toxicity correlations.These correlations can replace analysis of the deeper semantics of text.
  • ToxiGen’s response: TOXIGEN targets implicit toxicity while balancing toxic and benign statements to address gaps in prior datasets.Existing datasets contain substantial explicit toxicity and imbalances in minority-mentioning posts.

3 Creating TOXIGEN

TOXIGEN combines demonstration-based prompting with ALICE adversarial decoding to generate over 274,000 balanced toxic and benign statements about 13 identity groups, largely using implicit language.

  • 3 Creating TOXIGEN: TOXIGEN generates toxic and benign sentences mentioning minority groups, primarily without profanity or slurs.The dataset contains equal numbers of toxic and benign sentences for 13 identity groups.
  • 3.1 Prompt Engineering: Demonstration-based prompting uses curated benign and implicitly toxic examples, expanded through repeated human-in-the-loop prompt refinement.The process creates 26 prompt sets: benign and toxic prompts for each of 13 target groups.
  • 3.2 ALICE: Attacking Toxicity Classifiers with Adversarial Decoding: ALICE applies classifier-in-the-loop constrained beam search to steer generated text toward higher or lower toxicity probabilities.It combines language-model and classifier scores while preserving coherence through weighted decoding.
  • 3.2 ALICE: Attacking Toxicity Classifiers with Adversarial Decoding: In its two adversarial setups, ALICE targets false negatives by maximizing benign-class probability for toxic prompts and false positives by maximizing toxic-class probability for benign prompts.The first setup can also detoxify model outputs when classifier steering succeeds.
  • 3.4 TOXIGEN Statistics: The final dataset varies substantially in generation length, is almost entirely implicit, and includes an evaluation test set validated by human annotators.ALICE-generated data contributes a challenging adversarial subset designed to attack the target toxicity classifier.

4 Human Validation of TOXIGEN

Human validation evaluates whether TOXIGEN statements appear human-written, represent the intended toxicity and identity groups, and challenge toxicity classifiers. Results show high human-likeness, substantial toxicity-label alignment, and stronger adversarial behavior from ALICE decoding.

  • 4.2 Constructing TOXIGEN-HUMANVAL: Annotator agreement on toxicity was moderate, with Fleiss’ κ=0.46 and Krippendorff’s α=0.64.All three annotators agreed in 55.17% of cases, while at least two agreed in 93.4%.
  • 4.2 Constructing TOXIGEN-HUMANVAL: 90.5% of machine-generated examples were judged human-written by a majority of annotators, while 94.56% of toxic examples were classified as hate speech.Toxic examples were mislabeled as human-written at 92.9%, compared with 90.2% for non-toxic examples.
  • 4.2 Constructing TOXIGEN-HUMANVAL: 30.2% of machine-generated examples were rated harmful and 4% ambiguous, indicating that the evaluation set contained distinguishable toxic and benign material.All identity groups were represented, although prompted and referenced groups sometimes differed because generations could conflate or mention multiple identities.
  • 4.2 Constructing TOXIGEN-HUMANVAL: Toxicity ratings were similar when annotators assumed the speaker was human versus AI.The most common framing tactic was moral judgment, which questions the morality of an identity group.
  • 4.3 Comparing Generation Methods: 26.4% of ALICE-decoded sentences fooled HateBERT versus 16.8% of top-k sentences, while toxic-prompt ALICE outputs averaged 2.97 toxicity versus 3.75 for top-k.The toxicity difference was statistically significant with p < 0.001, indicating harder and more ambiguous ALICE examples.

5 Improving Toxicity Classifiers

The authors fine-tune HateBERT and ToxDectRoBERTa on ToxiGen and evaluate them across three human-written datasets. Fine-tuning improves performance across all datasets and also helps detect machine-generated toxicity.

  • 5 Improving Toxicity Classifiers: The authors fine-tune HateBERT and ToxDectRoBERTa on ToxiGen, using prompt labels as proxies for true toxicity labels.They compare the fine-tuned models with out-of-the-box versions on three publicly available human-written datasets.
  • 5 Improving Toxicity Classifiers: Fine-tuning HateBERT and ToxDectRoBERTa on ToxiGen improves performance across all evaluated datasets.The improvement is reported on three human-written toxicity datasets.
  • 5 Improving Toxicity Classifiers: Fine-tuned HateBERT performs strongly on TOXIGEN-HUMANVAL, demonstrating that ToxiGen can help guard against machine-generated toxicity.This result extends the reported benefit beyond human-written evaluation datasets.

6 Conclusions

The paper releases TOXIGEN, a large, balanced, implicit toxic-language dataset covering 13 minority groups. It also proposes ALICE and reports improvements in toxicity classifiers, while human evaluation finds the generated text difficult to distinguish from human-written text.

  • 6 Conclusions: TOXIGEN contains over 274k balanced benign and toxic sentences covering 13 minority groups.The dataset is machine-generated and designed to represent each group with balanced benign and toxic samples.
  • 6 Conclusions: ALICE is an adversarial decoding scheme that attacks toxicity classifiers and generates sentences designed to challenge them.The authors report effectiveness across publicly available toxicity detection systems.
  • 6 Conclusions: Fine-tuning pre-trained hate classifiers on TOXIGEN improves performance on three popular human-generated toxicity datasets.
  • 6 Conclusions: 90.5% of machine-generated examples in the human study were judged to be human-written.The study evaluated a subset of TOXIGEN and found that annotators struggled to distinguish machine-generated from human-written text.

7 Societal and Ethical Considerations

The paper discusses both the defensive value and misuse risks of generating toxic language with AI. It also emphasizes that toxicity detection remains subjective and that TOXIGEN covers only a limited, potentially noisy slice of problematic language.

  • 7 Societal and Ethical Considerations: The methods can be misused to generate machine-generated hate speech, although the authors intend them to support resources for combating harm toward minority groups.
  • 7 Societal and Ethical Considerations: ALICE can pass existing content filters, so similar approaches should be used to improve filters and prevent large-scale attacks on sensitive platforms.The authors report this behavior for five publicly available systems.
  • 7 Societal and Ethical Considerations: Toxicity is inherently subjective, making more nuanced labeling systems important beyond binary detection tasks.
  • 7 Societal and Ethical Considerations: Effective classifiers for machine biases are needed to address the scale of online harm and biased targeting of minority groups.
  • 7 Societal and Ethical Considerations: TOXIGEN does not capture the full complexity of problematic language, which is context-dependent, dynamic, and variable in severity.The authors call for multidisciplinary work involving human experience.
  • 7 Societal and Ethical Considerations: The dataset covers implicit toxicity for 13 identified minority groups and can naturally be noisy because of its large scale.The authors identify additional target groups and combinations as directions for future work.

Supplementary Materials

The supplementary materials describe the prompting procedure, model generality, and human-validation workforce. They also document worker demographics and safeguards used before annotation.

  • Supplementary Materials: For each minority group, five example sentences are joined into a prompt, and generation stops at the next newline.An appended hyphen asks the language model to produce a matching new sentence.
  • Supplementary Materials: The generation methods are not limited to GPT-3 and can in principle use other realistic text-generation models.The paper names GPT-Neo, GPT-J, and Turing-NLG as examples.
  • Supplementary Materials: Human validation used 156 Mechanical Turk workers with prior experience annotating toxic language.Fifty-one workers participated in data annotation.
  • Supplementary Materials: The reported worker demographics include race and ethnicity, gender, and age distributions collected through an optional survey.
  • Supplementary Materials: Annotators received a strong warning and signed consent was required before any text was shown.

C How does perplexity change across groups?

ALICE-generated sentences have substantially lower perplexity than top-k generations across minority groups, although average perplexity varies by subgroup and method.

  • C How does perplexity change across groups?: ALICE-generated sentences have significantly lower perplexity than top-k generations across all minority groups.After removing sentences with perplexity over 500, ALICE text reached up to 5 times lower perplexity than regular decoding.
  • C How does perplexity change across groups?: Average perplexity varies substantially between minority subgroups, with greater variation for top-k-generated text.Black-mentioning text is most likely across methods, while the least-likely subgroup differs by decoding method.
  • C How does perplexity change across groups?: The analysis drops roughly 10% of the dataset because GPT-2 assigns those sentences perplexity above 500.Perplexity is measured with a pretrained GPT-2 model from Hugging Face.
  • C How does perplexity change across groups?: Human validation separately measures whether generations mention the targeted identity group, enabling comparison of group-reference rates across decoding methods.Table 6 reports the proportions of generated sentences mentioning targeted groups with and without ALICE.

E Analysis of Large-Scale Human Validation

Large-scale human validation compares toxicity, adversariality, and target-group reference rates across randomly sampled ALICE and top-k generations, while highlighting annotation subjectivity.

  • E Analysis of Large-Scale Human Validation: 8,960 randomly sampled training examples received human labels using the same annotation framework and MTurk worker pool.The sample was split 50.9% top-k and 49.1% ALICE, without enforcing identical prompts across methods.
  • E Analysis of Large-Scale Human Validation: 66.86% of ALICE generations from toxic prompts were toxic, versus 57.91% for top-k generations.For non-toxic prompts, 93.21% of ALICE generations were non-toxic versus 90.01% for top-k.
  • E Analysis of Large-Scale Human Validation: 58.97% of toxic ALICE examples fooled HateBERT, compared with 26.88% of toxic top-k examples.Among non-toxic examples, ALICE fooled HateBERT 15.51% of the time versus 11.35% for top-k.
  • E Analysis of Large-Scale Human Validation: At least one annotator identified the exact target group in 78.3% of ALICE examples, compared with 70.4% of top-k examples.The full group-reference breakdown is reported in Figure 8 and Table 7 describes the evaluation subsets.
  • E Analysis of Large-Scale Human Validation: Annotation subjectivity can produce errors when annotators do not recognize group-specific context, as illustrated by the MGTOW example.Annotators labeled the example non-toxic, likely because they did not recognize MGTOW as a misogynistic group.
  • E Analysis of Large-Scale Human Validation: Demonstration-based prompting uses example sentences to encourage group-mentioning generations, with 26,000 prompts released alongside the dataset.A trailing hyphen encourages continuation in the style of the preceding examples.

G Releasing a Pretrained Model and its Propagated Labels

The authors release a RoBERTa classifier fine-tuned on human-annotated TOXIGEN examples, together with predictions that function as propagated labels for the full dataset.

  • G Releasing a Pretrained Model and its Propagated Labels: The released RoBERTa classifier is fine-tuned on 8,960 human-annotated TOXIGEN examples, starting from Zhou et al. (2021) weights.The model is intended as an entry point for community engagement with the dataset and code.
  • G Releasing a Pretrained Model and its Propagated Labels: The authors run the pretrained RoBERTa model over the full TOXIGEN dataset and release its predictions with the data.These predictions may help correct some mislabeled examples.
  • G Releasing a Pretrained Model and its Propagated Labels: The dataset records prompts, generated text, decoding method, prompt toxicity label, target group, and RoBERTa prediction probability.The prompt label is binary, with 1 indicating toxic and 0 indicating benign, but is slightly noisy according to human annotators.

I Further comparing toxicity classifiers

The classifier comparison evaluates toxicity judgments and annotation dimensions on machine-generated examples, including whether texts are human- or machine-generated and whether they refer to targeted groups.

  • I Further comparing toxicity classifiers: Without fine-tuning, classifiers perform worse on ALICE-generated data, indicating that ALICE produces examples more confusing to each model.Fine-tuning on each TOXIGEN-VAL subset yields strong performance on its corresponding evaluation set.
  • I Further comparing toxicity classifiers: The evaluation asks annotators whether text would be harmful if written by an AI system or by a person.This paired framing directly compares perceived harm across authorship settings.
  • I Further comparing toxicity classifiers: The evaluation includes examples where implicit stereotyping targets Black people, such as claims about being bred for sports and entertainment.These examples illustrate the type of subtle toxicity the annotation framework examines.
  • I Further comparing toxicity classifiers: Annotators also judge human-versus-machine origin, harmfulness, positive stereotyping, intentional harm, and sexual content.The questionnaire includes examples such as “all asians are good at math” and offensive or rude humor.
  • I Further comparing toxicity classifiers: Target-group analysis records whether text refers to the given group, other groups, and the manner of reference.The framework separately asks how the text refers to the targeted individual or group.
Loading 2203.09509v4…