PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Hate Speech with GPT-3

Ke-Li Chiu, Annie Collins, Rohan Alexander

arXiv:2103.12407v4cs.CL

TL;DR

Large language models can generate hateful text, and their training data may reproduce bias and abusive language; this motivates testing whether they can identify such content. The paper evaluates GPT-3 with zero-, one-, and few-shot prompts, finding mixed-category few-shot learning most effective, with average accuracy around 83% for racist and 85% for sexist extracts.

  • Problem

    The paper asks whether language models that can generate harmful language can also identify it, against a backdrop of biased training data and harmful outputs.

    Source: §1 · pp. 1–2

  • Method

    The authors test GPT-3 on sexist and racist text using zero-, one-, and few-shot classification prompts, including mixed-category examples.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Mixed-category few-shot learning performs best, with average accuracy around 83% for racist extracts and 85% for sexist extracts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that, with appropriate settings such as instructions and curated examples, large language models can identify sexist and racist text at specificity similar to other methods.

    Source: §5 · p. 18

    Because sexist and racist language changes quickly online, models may miss newly evolved usage unless prompts provide considerable context.

    Source: §5 · p. 18

Abstract

from arXiv · show

Sophisticated language models such as OpenAI's GPT-3 can generate hateful text that targets marginalized groups. Given this capacity, we are interested in whether large language models can be used to identify hate speech and classify text as sexist or racist. We use GPT-3 to identify sexist and racist text passages with zero-, one-, and few-shot learning. We find that with zero- and one-shot learning, GPT-3 can identify sexist or racist text with an average accuracy between 55 per cent and 67 per cent, depending on the category of text and type of learning. With few-shot learning, the model's accuracy can be as high as 85 per cent. Large language models have a role to play in hate speech detection, and with further development they could eventually be used to counter hate speech.

1 Introduction

Large language models can generate hateful text, while their internet-based training data may reproduce demographic bias and overgeneralized harmful language. The paper tests whether GPT-3 can also identify sexist and racist text, finding its strongest results with mixed-category few-shot learning.

1 Introduction

  • Large language models can produce hateful text targeting marginalized groups, and their training data may contain biases and abusive language.The paper notes that such outputs can cause unpleasant or triggering interactions even when models lack intent.Source: §1 · pp. 1–2
  • Internet-based training data raises concerns about exclusion, over-generalization, and exposure, including disproportionate representation of white, male, and young perspectives.Generalized language in training data may lead models to reproduce overstated harmful statements.Source: §1 · p. 2
  • The paper asks whether large language models that can produce harmful language can also identify it, motivated by this technology’s dual-use potential.The authors suggest detection could potentially counter human-produced abusive language and enable models to self-police.Source: §1 · p. 2
  • Mixed-category few-shot learning performed best, correctly classifying around 83% of racist and 85% of sexist extracts on average.The corresponding F1 scores were 79% for racist extracts and 77% for sexist extracts.Source: §1 · p. 2

2 Background

The background explains how language models predict text and situates GPT-3 among Transformer models, then reviews hate-speech definitions, detection challenges, and prior datasets and classifiers.

2.1 Language models, Transformers and GPT-3

  • Language models assign probabilities to token sequences and predict the next token; Transformer architecture underlies models including BERT, GPT-2, and GPT-3.GPT-3 can perform classification and supports few-shot learning from a small number of examples rather than fine-tuning.Source: §2.1 · p. 3
  • GPT-3 has also generated hateful content, including radicalized text targeting Islamic populations and false claims about the September 11 attacks.The background cites work documenting such outputs and misconceptions propagated by language models.Source: §2.1 · p. 3

2.2 Hate speech detection

  • Hate speech lacks a commonly held definition, and its detection is complicated by context and the difficulty of distinguishing hate speech from offensive language.Prior work includes datasets and classifiers, while studies have also found racial bias in annotated Twitter data.Source: §2.2 · p. 4
  • The cited Twitter policy prohibits specified threats and promotion of violence based on protected characteristics.The policy lists characteristics including race, gender, sexual orientation, age, and disability.Source: §2.2 · p. 4

3 Methods

The study evaluates GPT-3 classification of racist, sexist, and neutral text across zero-, one-, and few-shot prompts. It uses a balanced ETHOS sample and compares single-category with mixed-category few-shot setups.

3 Methods

  • The experiments compare zero-, one-, and few-shot classification of racist and sexist text using GPT-3’s Davinci engine at temperature 0.3.Mixed-category prompts classify each passage as racist, sexist, or neither; single-category prompts assess one category at a time.Source: §3 · pp. 4–5

3.1 Dataset

  • The study selects 60 sexist, 60 racist, and 120 neutral statements from ETHOS, using hate-speech labels with at least 50% annotator agreement for the targeted categories.The dataset began with 998 binary-labeled statements, and the authors retained 76 race-based and 84 gender-based candidates meeting the agreement threshold before sampling.Source: §3.1 · p. 5

3.2 Zero-shot learning

  • Zero-shot prompts provide no examples and ask for a yes-or-no category judgment, with the same 120 comments classified 100 times.The model’s answers rely on its training rather than a researcher-trained model, which the authors say they could not afford to create and run.Source: §3.2 · p. 6

3.3 One-shot learning

  • One-shot prompts provide one example from the selected category before asking for a yes-or-no judgment on each of 120 comments, repeated 100 times.The same example is used for each unique comment across its repeated classifications.Source: §3.3 · pp. 6–7

3.4 Few-shot learning

  • Single-category few-shot prompts use two examples from the selected category and one non-hate example, while mixed-category prompts use one example per label.Mixed-category prompts classify as racist, sexist, or neither; ten example sets were used, producing 2,400 responses across 243 unique comments.Source: §3.4 · pp. 7–8

4 Results

The results compare GPT-3’s hate-speech classifications across zero-, one-, and few-shot settings using accuracy, precision, recall, and F1. Mixed-category few-shot learning performed best overall, while instruction effects varied by category and by whether all three labels were counted.

4.1 Zero-shot learning

  • 56 per cent was the average zero-shot accuracy for identifying hate speech, while its average F1 score was 70 per cent.Racist-comment accuracy was 58 per cent and sexist-comment accuracy 55 per cent; their F1 scores were 58 per cent and 63 per cent, respectively.Source: Table 1 · p. 9, §4.1 · p. 9

4.2 One-shot learning

  • 55 per cent was the one-shot average accuracy, close to zero-shot’s 56 per cent, but one-shot’s F1 score was 55 per cent versus zero-shot’s 70 per cent.One-shot accuracy and F1 were similar for racist and sexist comments.Source: Table 2 · p. 10, §4.2 · p. 10

4.3 Few-shot learning – single category

  • 67 per cent was the single-category few-shot average accuracy, above one-shot’s 55 per cent and zero-shot’s 56 per cent; its average F1 score was 62 per cent.The model performed slightly better on sexist than racist comments in this setting.Source: §4.3 · pp. 10–11, Table 3 · p. 11

4.4 Few-shot learning – mixed category

  • Example Set 10 achieved 91 per cent accuracy and 87 per cent F1 for racist comments, while example sets differed enough to suggest example content affected inference.Example Set 8 had the lowest racist-text F1 score at 69 per cent; Example Set 9 had the lowest sexist-text F1 score at 69 per cent.Source: §4.4 · p. 11
  • Mixed-category few-shot learning performed distinctly better overall than zero-shot, one-shot, and single-category few-shot learning for both racist and sexist text.Across example sets, racist-text accuracy was approximately 87 per cent; results also varied with example content.Source: §4.4 · p. 11, Table 4 · p. 12

4.5 Few-shot learning – mixed category with instruction

  • Instruction constrained mixed-category answers to five unique outputs apart from three omitted responses, while unrestricted prompting generated many out-of-scope answers.The instructed outputs included “racist”, “sexist”, “neither”, “both”, and “racist and sexist”.Source: §4.5 · p. 13
  • With instruction, Example Set 10 performed best for sexist text at 86 per cent accuracy and 78 per cent F1; Example Set 7 led for racist text at 88 per cent accuracy and 82 per cent F1.Overall racist-text performance was similar with and without instruction, while sexist-text identification appeared slightly better without instruction.Source: §4.5 · p. 13, Table 7 · p. 15
  • 71 per cent of all-category classifications matched with instruction, versus 65 per cent without instruction across mixed-category few-shot example sets.The instructed approach almost always performed better on this all-label match measure.Source: §4.5 · p. 13, Figure 1 · p. 16

5 Discussion

GPT-3 performed best at identifying racist and sexist comments with mixed-category few-shot learning without instruction, while performance varied across settings and remained vulnerable to misclassification. The discussion highlights errors involving category confusion, identity terms, and misspellings, alongside the limits of a small dataset and rapidly changing language.

5 Discussion

  • 67% average accuracy and 62% average F1 were achieved in single-category few-shot learning, compared with 56% accuracy and 70% F1 zero-shot and 55% accuracy and 55% F1 one-shot.The zero-shot, one-shot, and single-category few-shot settings were described as likely not ideal for hate speech detection, with average accuracy between 50% and 70%.Source: §5 · p. 16
  • Mixed-category few-shot learning without instruction was the most effective setting, while instruction-assisted classification had 79% F1 and 50% recall for sexist speech.That recall means nearly half of sexist comments were wrongly classified.Source: §5 · p. 16
  • Adding instructions increased overall classification matches in almost every example set, but metric gains varied by category and could coexist with low recall.The authors suggest curated examples may improve accuracy.Source: §5 · p. 17
  • With instruction, GPT-3 sometimes classified a sexist comment about gender-based violence as racist, possibly because it was required to choose one category.The authors identify allowing multiple hate-speech categories as one reason mixed-category few-shot learning performed better without instruction.Source: §5 · p. 17
  • Identity terms may contribute to false positives when GPT-3 overlooks the context in which those terms are used.The authors note that language changes quickly and that words becoming sexist or racist after training may go undetected without sufficient prompt context; they also identify the relatively small dataset as a weakness.Source: §5 · pp. 17–18
  • Misspellings or censorship changed the classification of six of 61 tested comments, all racist; two comments became classified as racist only after misspellings were added.A separate example of a racist comment classified as neither involved the misspelling “phuck.”Source: §5 · pp. 17–18

A Example sets for the mixed-category, few-shot learning, experiments

The appendix includes a table of the ten example sets used in the mixed-category few-shot experiments.

A Example sets for the mixed-category, few-shot learning, experiments

  • Table 8 presents the ten example sets used in the mixed-category few-shot learning experiments.Source: Table 8 · p. 19

B.1 Zero-shot

The appendix provides zero-shot classification tables covering sexist statements, racist statements, and hate speech.

B.1 Zero-shot

  • Table 9 covers zero-shot classification of racist statements.Source: Table 9 · p. 20
  • Table 10 covers zero-shot classification of sexist statements.Source: Table 10 · p. 20
  • Table 11 covers zero-shot classification of hate speech.Source: Table 11 · p. 20

B.2 One-shot

The appendix provides one-shot classification tables covering racist statements, sexist statements, and hate speech.

B.2 One-shot

B.3 Few-shot single category

Three tables present GPT-3 few-shot classification for racist statements, sexist statements, and hate speech, each using a single category.

B.3 Few-shot single category

  • A table presents GPT-3 few-shot classification of racist statements using a single category.Source: Table 15 · p. 22
  • A table presents GPT-3 few-shot classification of sexist statements using a single category.Source: Table 16 · p. 22
  • A table presents GPT-3 few-shot classification of hate speech using a single category.Source: Table 17 · p. 22

B.4 Few-shot mixed category, without instruction

The tables present mixed-category few-shot GPT-3 classification for racist and sexist statements, including conditions with and without instruction.

B.4 Few-shot mixed category, without instruction

  • A table presents mixed-category few-shot GPT-3 classification of racist statements without instruction.Source: Table 18 · p. 23
  • A table presents mixed-category few-shot GPT-3 classification of sexist statements.Source: Table 19 · p. 24

B.5 Few-shot mixed category, with instruction

  • A table presents mixed-category few-shot GPT-3 classification of racist statements with instruction.Source: Table 20 · p. 25

C Effect of misspellings

This section includes a mixed-category few-shot classification table for sexist statements with instruction and a table of profanity-extraction strings and text edits used to test misspelling and censorship.

C Effect of misspellings

  • A table presents mixed-category few-shot GPT-3 classification of sexist statements with instruction.Source: Table 21 · p. 26
  • A table lists strings used to extract comments with profanity and edits introduced to test misspelling and censorship in hate-speech classification.Source: Table 22 · p. 26
Loading 2103.12407v4…