PaperlayerView exact revision on arXiv

Source-linked AI summary

ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, Ece Kamar

arXiv:2203.09509v4cs.CL

TL;DR

Toxicity detectors can over-rely on minority identity mentions, risking missed subtle hate and false flags on benign text. ToxiGen combines a large-scale dataset of toxic and benign statements with ALICE, an adversarial decoding method; fine-tuning on ToxiGen improves performance by 7–19% across three human-written implicit-toxicity datasets.

  • Problem

    Toxicity detectors may over-rely on minority identity mentions, risking underdetection of subtle hate and over-detection of benign statements.

    Source: §1 · p. 1

  • Method

    TOXIGEN uses language-model generation to create toxic and benign statements, while ALICE steers decoding with a toxicity classifier to produce adversarial examples.

    Source: §1 · pp. 1–2, §6 · p. 9

  • Results

    +7–19% performance improvement from fine-tuning existing classifiers on TOXIGEN across three human-written implicit-toxicity datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ALICE offers a way to evaluate toxicity-classifier robustness by generating sentences that fool publicly available systems.

    Source: §1 · p. 2, §6 · p. 9

    Because toxicity is inherently subjective, the authors argue that responsible systems require more nuanced labeling than binary detection.

    Source: §7 · p. 9

Abstract

from arXiv · show

Toxic language detection systems often falsely flag text that contains minority group mentions as toxic, as those groups are often the targets of online hate. Such over-reliance on spurious correlations also causes systems to struggle with detecting implicitly toxic language. To help mitigate these issues, we create ToxiGen, a new large-scale and machine-generated dataset of 274k toxic and benign statements about 13 minority groups. We develop a demonstration-based prompting framework and an adversarial classifier-in-the-loop decoding method to generate subtly toxic and benign text with a massive pretrained language model. Controlling machine generation in this way allows ToxiGen to cover implicitly toxic text at a larger scale, and about more demographic groups, than previous resources of human-written text. We conduct a human evaluation on a challenging subset of ToxiGen and find that annotators struggle to distinguish machine-generated text from human-written language. We also find that 94.5% of toxic examples are labeled as hate speech by human annotators. Using three publicly-available datasets, we show that finetuning a toxicity classifier on our data improves its performance on human-written data substantially. We also demonstrate that ToxiGen can be used to fight machine-generated toxicity as finetuning improves the classifier significantly on our evaluation subset. Our code and data can be found at https://github.com/microsoft/ToxiGen.

1 Introduction

ToxiGen addresses toxicity detectors’ over-reliance on minority identity mentions with a large, machine-generated dataset designed to include balanced toxic and benign statements and implicit toxicity. The introduction reports improved performance after fine-tuning on ToxiGen and describes adversarial generation and human evaluation.

Introduction

  • ToxiGen contains 274,186 machine-generated toxic and benign statements, including over 135,000 of each about 13 minority identity groups.The dataset was created using demonstration-based prompting and pretrained toxicity classifiers.Source: §1 · p. 1
  • 98.2% of ToxiGen statements are implicit, containing no explicit profanity, slurs, or swearwords.Machine generation and careful prompting also allow equal numbers of toxic and benign statements for each demographic group, including groups often overlooked in corpora.Source: §1 · pp. 1–2
  • ALICE steers generation during beam search by pitting a toxicity classifier against a text generator to produce sentences that fool classifiers.It can steer toxic prompts toward less toxic outputs or neutral prompts toward more toxic outputs; experiments used five publicly available toxicity classifiers.Source: §1 · p. 2
  • 90% of a sample of 792 machine-generated sentences could be mistaken for human-written text in human evaluation.The evaluation also found a wide variety of specific references to minority groups mentioned in the prompts.Source: §1 · p. 2
  • +7–19% performance improvement was reported after fine-tuning existing classifiers on ToxiGen across three human-written implicit toxic datasets.The datasets were ImplicitHateCorpus, SocialBiasFrames, and DynaHate.Source: §1 · p. 2
  • The authors publicly release the ToxiGen code and dataset, along with two models pretrained on ToxiGen and human evaluations.Source: §1 · p. 2

2 Implicit Hate Against Minority Groups

Implicit toxicity about minority groups is difficult to detect and collect, while imbalanced data can teach systems to treat minority mentions as toxic. TOXIGEN addresses these gaps with large-scale generation that controls implicitness and balances toxic and benign statements.

2 Implicit Hate Against Minority Groups

  • Implicit toxicity lacks profanity, can have positive sentiment, is harder to detect or collect at scale, and can harm marginalized groups by reinforcing stereotypes.The passage characterizes such language as psychologically damaging and capable of reinforcing stereotypical or hateful perceptions.Source: §2 · pp. 2–3
  • Imbalanced toxicity labels around minority mentions teach models spurious shortcuts, leading classifiers to block minority-referencing language and risk erasure.Large language models can also produce stereotypical, biased, or toxic content when prompted with minority mentions.Source: §2 · p. 3
  • TOXIGEN aims to balance toxic and benign implicit statements at scale, addressing prior scraped datasets’ explicit-content and minority-mention imbalance through controllable language-model generation.Generation lets the authors control which minority groups are mentioned and the statements’ implicitness.Source: §2 · p. 3

3 Creating TOXIGEN

TOXIGEN uses demonstration-based prompting and classifier-in-the-loop decoding to generate a large, balanced collection of implicitly toxic and benign statements about minority groups. ALICE makes some generations adversarial to toxicity classifiers, while the dataset statistics indicate that almost all statements are implicit.

3 Creating TOXIGEN

  • Over 274,000 statements—equal numbers of toxic and benign examples—were generated for 13 identity groups.The dataset includes generations produced both with and without ALICE.Source: §3 · p. 3

3.1 Prompt Engineering

  • TOXIGEN prompts a language model for group-naming toxic and benign sentences that are mainly implicit and avoid profanity or slurs.Demonstrations encourage the model to produce a similar but distinct sentence; ten neutral Muslim examples often elicit another neutral sentence.Source: §3.1 · pp. 3–4, §3.1.1 · p. 4

3.1.1 Demonstration-based prompting

  • Demonstration-based prompting uses example statements to elicit similar but distinct statements from a language model.The authors use GPT-3, but state that the methods could work with any human-like text generator.Source: §3.1.1 · p. 4

3.1.2 Collecting demonstrations

  • For all 13 target groups, authors iteratively expanded benign and toxic examples through a human-in-the-loop process, creating 26 prompt sets—one benign and one toxic per group—and collecting 20–50 demonstrations per group.Benign examples came from blog posts and news articles; implicitly toxic examples came from hate forums and Reddit.Source: §3.1.2 · pp. 4–5

3.2 ALICE: Attacking Toxicity Classifiers with Adversarial Decoding

  • ALICE adapts constrained beam search (Anderson et al., 2017; Hokamp and Liu, 2017; Holtzman et al., 2018) to generate classifier-adversarial text while preserving language-model coherence.Toxic prompts target false negatives by maximizing the classifier’s benign-class probability, while benign prompts target false positives by maximizing its toxic-class probability.Source: §3.2 · p. 5

3.3 Decoding Details

  • ALICE decoding pairs GPT-3 with HateBERT’s OffensEval classifier, using λL = λC = 0.5, a 30-token maximum, beam size 10, and temperature 0.9.Because of GPT-3 API limits, decoding restricts the vocabulary to the top 100 tokens and resamples allowed tokens with top-k.Source: §3.3 · pp. 5–6

3.4 TOXIGEN Statistics

  • Generation lengths vary substantially, and almost all final-dataset statements are implicit; ALICE generations contribute a challenging adversarial subset.The released data also includes a test set validated by human annotators.Source: §3.4 · p. 6

4 Human Validation of TOXIGEN

The human validation evaluates whether ToxiGen statements resemble human writing, convey the intended toxicity and group references, and challenge toxicity classifiers. Annotators found the generated text difficult to distinguish from human writing, while ALICE-generated examples fooled HateBERT more often than top-k samples.

4.1 Human Validation Design

  • The validation asked annotators to identify human versus AI authorship, rate harm under each assumption, judge harmful intent and positive stereotypes, and explain how statements frame groups.Harm ratings used a 1–5 scale; additional questions covered lewdness and whether a statement claimed fact or opinion.Source: §4.1 · p. 6

4.2 Constructing TOXIGEN-HUMANVAL

  • The human-validated test set contains 792 statements, each rated by three annotators; annotator agreement on toxicity was Fleiss’ κ=0.46 and Krippendorff’s α=0.64.All three annotators agreed in 55.17% of cases, and at least two agreed in 93.4%.Source: §4.2 · pp. 6–7
  • 90.5% of machine-generated examples were judged human-written by a majority of annotators; this held for 92.9% of toxic and 90.2% of non-toxic examples.Most toxic examples were also labeled hate speech (94.56%).Source: §4.2 · p. 7, Figure 4 · p. 7
  • 30.2% of machine-generated examples were rated harmful and 4% ambiguous; all dataset identity groups appeared in the study, though prompts and generated text sometimes referenced different groups.The authors suggest GPT-3 may conflate identities or mention multiple groups.Source: §4.2 · p. 8
  • Annotators’ toxicity ratings did not differ significantly by whether they believed the speaker was human or AI, and moral judgment was the most common framing tactic.Moral judgment involves questioning an identity group’s morality.Source: Figure 5 · p. 7, §4.2 · p. 8

4.3 Comparing Generation Methods

  • ALICE fooled HateBERT more often than top-k (26.4% vs. 16.8%) and lowered human-annotated toxicity for toxic prompts (2.97 vs. 3.75, p < 0.001), yielding harder, more ambiguous examples.The comparison used 125 prompts—62 toxic and 63 non-toxic—with one ALICE and one top-k statement per prompt.Source: §4.3 · p. 8

5 Improving Toxicity Classifiers

Fine-tuning HateBERT and ToxDectRoBERTa on TOXIGEN improves performance across the tested datasets, and fine-tuned HateBERT performs strongly on the machine-generated TOXIGEN-HUMANVAL evaluation set.

5 Improving Toxicity Classifiers

  • The study fine-tunes both classifiers on TOXIGEN’s training set, using prompt labels as proxies for true toxicity labels, then compares them with out-of-the-box models.Evaluation covers three publicly available human-written datasets and TOXIGEN-HUMANVAL.Source: §5 · p. 8
  • To ablate decoding-method contributions, the researchers split TOXIGEN into equal numbers of ALICE-generated and top-k-generated examples.Source: §5 · p. 8
  • Fine-tuning HateBERT and ToxDectRoBERTa on TOXIGEN improves performance across all tested datasets, while fine-tuned HateBERT performs strongly on TOXIGEN-HUMANVAL.The human-written evaluation datasets are IMPLICITHATECORPUS, the SOCIALBIASFRAMES test set, and DYNAHATE.Source: §5 · pp. 8–9

6 Conclusions

TOXIGEN is a large, balanced dataset of implicit toxic language about 13 minority groups, and fine-tuning pre-trained hate classifiers on it improved performance on three human-generated toxicity datasets. The paper also presents ALICE for evaluating and attacking toxicity classifiers and reports that annotators often mistook generated examples for human-written text.

  • Fine-tuning pre-trained hate classifiers on TOXIGEN improved performance on three popular human-generated toxicity datasets.Source: §6 · p. 9
  • TOXIGEN contains over 274k sentences, balanced toxic and benign samples for each of 13 minority groups.The authors describe it as larger and more diverse than previous datasets.Source: §6 · p. 9
  • ALICE is an adversarial decoding scheme that evaluates toxicity-classifier robustness and generates attack sentences; it was effective on publicly available systems.Source: §6 · p. 9
  • In a human study, 90.5% of machine-generated examples were thought to be human-written, indicating annotators struggled to distinguish them from human text.Source: §6 · p. 9

7 Societal and Ethical Considerations

The authors acknowledge that their toxicity-generating methods could be misused to spread machine-generated hate, while arguing that the work can help the community counter harm. They emphasize that the dataset and detection systems have important limits and should be developed with human experience, interdisciplinary research, and informed policy.

Risks in dataset release

  • The generation methods could be exploited to spread machine-generated hate, but the authors see the work as an opportunity for the community to push back against harm toward minority groups.They exclude identity dimensions historically associated with agents of oppression and note that the dataset does not capture the full complexity of human experiences.Source: §7 · p. 9

ALICE

  • The ALICE method uses an adversarial game between two AI systems to pass existing content filters, as demonstrated on 5 publicly available systems.The authors urge leveraging this and similar approaches to improve filters and prevent large-scale attacks against sensitive platforms.Source: §7 · p. 9

Improving Toxicity Detection

Relationship to Policy

  • Policy and regulation can oversee content moderation and toxicity detection, where errors may disproportionately affect minority groups.The authors envision pairing tools like theirs with human expertise and informed policy and regulation to develop scalable, reliable solutions (Wischmeyer and Rademacher, 2020; Reich et al., 2021).Source: §7 · p. 9

Responsible AI Considerations

  • The dataset covers implicit hate speech for 13 identified minority groups and may be noisy; future work could include more target groups and combinations of groups.The authors also caution that annotations and models cannot capture all problematic language, which is context-dependent, dynamic, and variable in severity.Source: §7 · pp. 9–10

Supplementary Materials … B.2 Annotation Interface

The supplementary materials describe how ToxiGen sentences were generated and outline the language-model options and human-validation procedures, including worker selection and annotation safeguards.

A Generation Details

  • Each generated sentence used five randomly sampled examples for its minority group as hyphen-prefixed prompts, with generation stopped at the next newline.A fresh random set of five examples was used for every sentence.Source: A Generation Details · p. 14

A.1 Language Model Selection

  • Although this work uses GPT-3, the generation methods could also use GPT-Neo, GPT-J, or Turing-NLG.Source: §A.1 · p. 14

B.1 Selecting MTurk Workers

  • The human validation recruited 156 MTurk workers with prior toxic-language annotation experience; 51 participated in data annotation.The cited prior work is Sap et al. (2020).Source: §B.1 · p. 14
  • Among surveyed workers, 56.9% identified as White, 9.8% as Black, 3.9% as Hispanic, 3.9% as Asian, and 5.9% as Other.The demographics survey was optional and administered after the annotation task.Source: §B.1 · p. 14
  • Most workers were ages 25–45 (58.8%); 45.1% identified as female, 37.3% male, and 2% non-binary, while political identities and other demographics varied.Political identities included 25.5% left-leaning, 23.5% very left-leaning, 13.7% moderate, 17.6% right-leaning, and 3.9% very right-leaning; 5.9% identified as LGBTQ+ and 2% as Pacific Islander.Source: §B.1 · p. 14

B.2 Annotation Interface

  • The annotation interface was shown in Figure 6, and workers received a strong warning and had to sign consent before viewing any text.Source: §B.2 · p. 14
  • Some workers did not answer the optional demographic questions; the paper notes that the remaining workers chose not to respond.This qualification applies to the reported worker demographics.Source: §B.2 · p. 14

C How does perplexity change across groups? · D Does generated text actually mention the targeted groups?

ALICE decoding produces lower-perplexity text than top-k across minority groups, while both methods usually generate text that mentions the targeted group. The analyses also show variation by subgroup and decoding method.

C How does perplexity change across groups?

D Does generated text actually mention the targeted groups?

E Analysis of Large-Scale Human Validation · F Example Prompt

Large-scale human validation finds that ALICE-generated examples more often fool HateBERT than top-k examples, while the example prompt section illustrates demonstration-based generation of group-mentioning text.

E Analysis of Large-Scale Human Validation

  • The validation labeled 8,960 randomly sampled training examples, split 50.9% top-k and 49.1% ALICE, without enforcing matched prompts across methods.The same-prompt comparison had been conducted in Section 4.3.Source: E Analysis of Large-Scale Human… · p. 15
  • Average toxicity scores were similar for ALICE and top-k given a prompt label, with ALICE slightly higher for toxic-labeled prompts and lower for non-toxic prompts.Source: E Analysis of Large-Scale Human… · p. 15
  • 58.97% of toxic ALICE-generated examples fool HateBERT, compared to 26.88% of toxic top-k generated examples.ALICE-generated non-toxic examples also fooled HateBERT more often: 15.51% versus 11.35% for top-k generations.Source: E Analysis of Large-Scale Human… · p. 15
  • 66.86% of ALICE texts from toxic-labeled prompts were actually toxic, versus 57.91% for top-k; 93.21% from non-toxic prompts were actually non-toxic, versus 90.01% for top-k.Source: E Analysis of Large-Scale Human… · p. 15
  • 78.3% of ALICE-generated examples had an annotator identify a direct or indirect reference to the exact target group, compared with 70.4% of top-k examples.Source: E Analysis of Large-Scale Human… · p. 15
  • Annotator subjectivity can affect toxicity judgments; annotators labeled a positive MGTOW example non-toxic, likely because they did not recognize MGTOW as misogynistic.Source: E Analysis of Large-Scale Human… · p. 15

F Example Prompt

  • The illustrated demonstration-based prompt encourages group-mentioning output, using neutral Native American examples that span cultural, policy, and historical statements.Source: F Example Prompt · pp. 15–16
  • A trailing hyphen cues another sentence in the growing list, and all 26,000 generated prompts are released with the dataset.Source: F Example Prompt · p. 16

G Releasing a Pretrained Model and its Propagated Labels · H Dataset Description

The authors release a RoBERTa classifier trained on human-annotated ToxiGen data and its predictions across the full dataset. ToxiGen is released as a dataframe with generation metadata, labels, group information, and model scores.

G Releasing a Pretrained Model and its Propagated Labels

H Dataset Description

  • ToxiGen is released as a dataframe containing each generation’s prompt and generated text.Source: H Dataset Description · p. 16
  • The generation_method field records whether ALICE was used; its values are ALICE or top-k, with top-k indicating ALICE was not used.Source: H Dataset Description · p. 16
  • The prompt_label field marks prompts as toxic (1) or benign (0), and is slightly noisy but largely accurate according to human annotators.Source: H Dataset Description · p. 16
  • The dataframe records the target group and each instance’s RoBERTa prediction probability, which can be used as propagated labels.Source: H Dataset Description · p. 16

I Further comparing toxicity classifiers

Finetuned classifiers performed strongly on their respective TOXIGEN-VAL subsets, while unfinetuned models performed worse on ALICE-generated data, which the authors interpret as evidence that ALICE generations are more confusing to each model.

I Further comparing toxicity classifiers

  • Finetuning on each TOXIGEN-VAL subset yielded strong performance on its respective evaluation set, while unfinetuned models performed worse on ALICE-generated data.The authors interpret the unfinetuned models’ weaker performance as evidence that ALICE generates data that are more confusing to each model.Source: I Further comparing toxicity classifiers · p. 16

Related papers

Loading 2203.09509v4…