PaperlayerView exact revision on arXiv

Source-linked AI summary

HateCheck: Functional Tests for Hate Speech Detection Models

Paul Röttger, Bertram Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, Janet B. Pierrehumbert

arXiv:2012.15606v2cs.CL

TL;DR

HateCheck addresses the limited diagnostic value and potential misleadingness of held-out hate-speech evaluation. It introduces 29 stakeholder- and research-motivated functional tests with validated targeted cases, finding critical weaknesses across transformer and commercial models. The authors position it as a complement to, not a substitute for, real-world held-out evaluation.

  • Problem

    Held-out metrics reveal limited information about specific weaknesses and can overstate generalisable performance because hate-speech datasets contain systematic gaps and biases.

  • Method

    HateCheck specifies 29 functionalities, constructs targeted hateful and non-hateful contrast cases, and validates their gold-standard labels through structured annotation.

  • Results

    HateCheck reveals critical weaknesses in all tested models, including keyword sensitivity, poor handling of reclaimed slurs and contrasts, and target-group bias in transformer models.

  • Takeaways & Limitations

    HateCheck provides targeted diagnostic insights for model development while complementing held-out tests of real-world hate speech.

  • Takeaways & Limitations

    HateCheck primarily reveals weaknesses rather than generalisable strengths, and its coverage is limited, including for slurs and identity terms.

Abstract

from arXiv · show

Detecting online hate is a difficult task that even state-of-the-art models struggle with. Typically, hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score. However, this approach makes it difficult to identify specific model weak points. It also risks overestimating generalisable model performance due to increasingly well-evidenced systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, we introduce HateCheck, a suite of functional tests for hate speech detection models. We specify 29 model functionalities motivated by a review of previous research and a series of interviews with civil society stakeholders. We craft test cases for each functionality and validate their quality through a structured annotation process. To illustrate HateCheck's utility, we test near-state-of-the-art transformer models as well as two popular commercial models, revealing critical model weaknesses.

1 Introduction

HateCheck addresses limitations of held-out evaluation by providing targeted functional tests for diagnosing hate speech detection models. Applied to transformer and commercial systems, it reveals systematic weaknesses involving keywords, contrasts, and target-group coverage.

  • Motivation: Held-out performance metrics provide limited insight into specific model weaknesses and may overstate generalisable quality because hate-speech datasets contain systematic gaps and biases.Models can learn simple decision rules from dataset artifacts, show lexical sensitivity, and generalise poorly across datasets.
  • Approach: HateCheck is a functional-testing suite that evaluates model outputs on targeted test cases designed to diagnose particular capabilities.Functional testing assesses specified model behaviours using sets of targeted cases.
  • Approach: The suite covers 29 functionalities selected through a literature review and interviews with civil society stakeholders.Each functionality receives a separate test, including hateful expressions and non-hateful contrasts.
  • Approach: The authors hand-craft test cases with clear gold-standard labels and validate their quality through structured annotation.The tests include 18 hateful-content cases and 11 non-hateful contrasts.
  • Findings: Evaluation of two near-state-of-the-art BERT models and two commercial systems shows critical weaknesses across all tested models.The models are overly sensitive to keywords and misclassify negated hate, counter speech, and other non-hateful contrasts; BERT models also vary across target groups.

2 HATECHECK

HateCheck operationalises hate-speech detection as binary classification and constructs 29 functional tests with targeted, contrastive English-language cases. Its design combines research and stakeholder input with controlled coverage, annotation, and secondary labels.

  • Operationalisation: A model functionality is its ability to provide the specified hateful or non-hateful classification for cases in a corresponding functional test.The framework separates functionalities such as detecting hateful profanity from recognising non-hateful profanity.
  • Selection: The initial functionality list draws on previous research and interviews with 21 employees from 16 NGOs working on online hate.Interviews addressed challenges, moderation errors, and desired improvements using a semi-structured grounded-theory approach.
  • Selection: HateCheck restricts testing to individual English-language text documents and excludes modalities, languages, and contexts requiring information beyond individual documents.These boundaries reflect practical constraints and the focus of most existing hate-speech detection models.
  • Functional Tests: The suite includes 29 tests grouped into 11 classes: 18 tests cover distinct hateful expressions and 11 provide contrastive non-hate cases.Non-hateful cases share linguistic features with hateful expressions and are built as direct contrasts with minimal changes.
  • Functional Tests: Tests cover derogatory and threatening hate, slurs, profanity, pronouns, negation, phrasing variants, spelling variations, and non-hateful contrasts including reclaimed slurs and counter speech.The authors use diverse vocabulary and syntax within functionalities to reduce similarity-driven instability.
  • Test Cases: The dataset contains 3,901 short cases generated from 460 templates or crafted individually, with 2,659 hateful and 1,242 non-hateful examples.Templates cover seven protected groups and support balanced targeting across groups.
  • Test Cases: Each case has a primary hateful or non-hateful label and up to two secondary labels identifying the protected group and, for hateful cases, group-versus-individual targeting.The secondary labels add structured information beyond the binary classification.

3 Testing Models with HATECHECK

HATECHECK evaluates English-language hate speech models with targeted black-box tests spanning 29 functionalities, exposing weaknesses across labels, functional contrasts, slurs, and target groups.

  • Model setup: HATECHECK is a black-box test suite applicable to models with different architectures, training datasets, and limited public information, including commercial models.It contains 29 functionalities and 3,728 test cases, with accuracy reported for each functional test.
  • Model setup: The evaluation includes BERT models fine-tuned on Davidson and Founta data, plus Google Perspective and Two Hat’s SiftNinja.B-D and B-F achieve held-out macro F1 scores of 70.8 and 70.3, respectively.
  • Results: All models show deficits across hateful and non-hateful cases, with B-D, B-F, and Perspective relatively stronger on hateful cases but often misclassifying non-hateful cases.SiftNinja is strongly biased toward predicting non-hateful, making it accurate on non-hateful cases but inaccurate on most hateful cases.
  • Results: B-D and B-F fall below 50% accuracy on 8 and 4 non-hate functional tests, especially reclaimed slurs, negated hate, and counter speech.They also struggle with spelling variations, including added spaces and leet spellings.
  • Results: Perspective exceeds 95% accuracy on 11 of 18 hateful functional tests but performs worse than B-D and B-F on reclaimed slurs, negated hate, and counter speech.SiftNinja is near-perfect on non-hateful tests but misclassifies most hateful cases because of its non-hateful prediction bias.
  • Discussion: Across models, specific slurs trigger overly sensitive keyword-based behavior, while negation and counter speech are frequently misclassified as hateful.B-D and B-F also classify hate targeting some protected groups, such as women, less accurately than equivalent cases targeting others.
  • Limitations: HATECHECK identifies weaknesses but provides only indirect evidence about whether they arise from biased training data or architectural limits.The differing functional-test and target-coverage performance of B-D and B-F despite sharing an architecture highlights training-data composition as one possible factor.
  • Discussion: Targeted data augmentation is proposed as a possible improvement when poor performance stems from biased training data, especially for negated hate, reclaimed slurs, and counter speech.The authors caution that additional data may introduce unforeseen biases.

4 Limitations

HATECHECK provides targeted diagnostic insights, but its results have bounded interpretive and coverage limits. Positive results do not establish generalisable model strengths, and the suite excludes several contexts and language phenomena.

  • Interpretive limits: Good performance reveals absence of a particular weakness, not necessarily a generalisable model strength.Claims about model quality should therefore not be overextended from positive HATECHECK results.
  • Interpretive limits: HATECHECK complements rather than substitutes for held-out evaluation on real-world hate speech.
  • Coverage limits: The suite tests separate English-language text documents, excluding external context, non-text modalities, and other languages.
  • Coverage limits: Its tests cover distinct expressions of hate and non-hate, but not more complex compound statements such as combinations of slurs and profanity.
  • Coverage limits: HATECHECK is static, so it does not test functionalities related to language change.Dynamic adversarial benchmarks are suggested as one possible future direction.
  • Coverage limits: The suite could be expanded to additional protected groups, intersectional characteristics, broader slurs, platform-specific language, and group-specific hate such as misogynistic tropes.

5 Related Work

HATECHECK extends targeted and contrastive evaluation traditions in NLP to hate speech detection. It combines these precedents with task-specific motivation, construction, and validation of functional tests.

  • Targeted diagnostics: Targeted diagnostic datasets have been used across NLP tasks but have seen limited use for hate speech detection.
  • Contrastive evaluation: HATECHECK builds on minimally contrastive pairs and applies contrastive cases to hate speech model evaluation.
  • Contrastive evaluation: Its non-hateful contrast cases follow approaches proposed for augmenting datasets and enabling more meaningful evaluation of generalisable models.
  • Novelty: HATECHECK is presented as the first application of this contrastive approach to hate speech detection.
  • Relation to CHECKLIST: Unlike CHECKLIST’s general NLP emphasis, HATECHECK stresses motivating functional-test selection and constructing and validating hate-speech-specific test cases.

6 Conclusion

The paper introduces HATECHECK as a validated suite of targeted contrasts for diagnosing hate speech detection models. Tests across transformer and commercial systems reveal keyword sensitivity and target-coverage bias, while supporting more focused model development.

  • Contribution: HATECHECK uses interviews with civil society stakeholders and a literature review to motivate its functional tests.
  • Contribution: Each functional test contains targeted cases with clear gold labels validated through structured annotation.
  • Findings: HATECHECK revealed critical weaknesses across near-state-of-the-art transformer and commercial hate speech detection models.
  • Findings: Models appeared overly sensitive to keywords and phrases, performing poorly on reclaimed slurs, counter speech, and negated hate.
  • Findings: Transformer models exhibited strong biases in target coverage.
  • Implications: The authors hope HATECHECK’s targeted diagnostic insights will clarify model limitations and aid development of better models.

Impact Statement

HATECHECK is intended as an evaluative tool providing structured diagnostic insights into hate speech model functionalities. The paper also identifies misuse risks and documents ethical safeguards for its interviews and synthetic test cases.

  • Ethics: Interviewees gave explicit consent, responses were anonymised, participant demographics and affiliations were withheld, and ethics-board approval was obtained.
  • Ethics: The authors crafted HATECHECK test cases as synthetic data, presenting no intellectual-property-rights risk.
  • Annotator compensation: Ten annotators were paid £16 per hour, with training time and meetings also compensated.
  • Intended use: HATECHECK’s intended use is evaluating hate speech detection models through structured, targeted diagnostic insights.
  • Intended use: The authors also discuss data augmentation as a possible alternative use aimed at aiding better model development.
  • Potential misuse: Overextending claims about model functionalities from HATECHECK performance is identified as misuse.

A Data Statement

HATECHECK documents how its test cases were generated and validates their binary gold-standard labels through structured annotation.

  • The data statement records the generation and provenance of HATECHECK test cases.
  • HATECHECK contains 3,901 short English-language documents created manually and with simple templates for group identifiers and slurs.Each document corresponds to one functional test and a hateful or non-hateful gold-standard label.
  • Ten annotators validated the gold-standard labels, with five annotators assigned to each document.

B. LANGUAGE VARIETY

HATECHECK is limited to English-language text, reflecting the field’s prevailing focus and the expertise of its authors and annotators, while its manually crafted cases are author-produced.

  • HATECHECK covers only English-language text documents.The authors identify limited language variety as a limitation and prioritize expansion to other languages for future research.
  • The English-language focus reflects the predominance of English in hate-speech detection research and the expertise of the authors and annotators.
  • All test cases were hand-crafted, so their speakers are the authors, with most cases primarily generated by the lead author and reviewed by two co-authors.
  • The ten annotators worked for two weeks and had varied genders, ages, education levels, language backgrounds, nationalities, and ethnic backgrounds.
  • Annotators regularly used social media, had encountered online abuse, and had prior annotation experience with at least one other hate-speech dataset.
  • All HATECHECK test cases were created between 23 November and 13 December 2020.

C Hateful Slurs in HATECHECK

HATECHECK selects representative slurs for seven protected groups and includes reclaimed-slur cases reviewed by members of relevant communities.

  • For each of seven protected groups, the authors searched Hatebase and generally selected the three most frequently logged group-targeting slurs.
  • When the third-most logged slur was substantially less frequent than the second, only the top two slurs were selected.
  • For immigration status, the authors selected two slurs because that category was not represented on Hatebase.
  • Reclaimed-slur tests focus on terms reclaimed by Black, gay, and women’s communities, reflecting concerns raised by interview participants.
  • Self-identifying members of relevant groups reviewed reclaimed-slur cases for valid and realistic usage, which held true for all such cases.

D.1 Davidson et al. (2017) Data

The Davidson and Founta datasets provide large, crowd-annotated Twitter benchmarks that are converted to HATECHECK’s binary format and used to fine-tune BERT models.

  • Sampling: Davidson et al. sampled 25,000 tweets from tweets posted by 33,458 users identified through Hatebase-derived keywords.
  • Annotation: Davidson et al. used CrowdFlower workers to label tweets as hateful, offensive, or neither, with most tweets annotated by three workers.They report a 92% CrowdFlower score for inter-annotator agreement.
  • Data: The Davidson dataset contains 1,430 hateful tweets and 23,353 non-hateful tweets after offensive and neither labels are collapsed.
  • Sampling: Founta et al. began with 32 million randomly collected tweets and augmented a random subset using negative sentiment and offensive-word sampling.
  • Annotation: Founta et al. labeled tweets as hateful, abusive, spam, or normal using five crowd workers per tweet.Four of five annotators agreed on a label for 55.9% of tweets.
  • Data: The Founta data contain 4,965 hateful tweets and 95,031 non-hateful tweets after abusive, spam, and normal labels are collapsed.

F Comparison to SOTA Results

The authors fine-tune unweighted multiclass B-D and B-F models for comparability with prior work, finding that BERT-base performs competitively on both datasets. Reported results also show that external SOTA work did not evaluate these datasets.

  • Method: Unweighted multiclass B-D and B-F models were fine-tuned using the same parameters described in Appendix E.This preserves comparability with prior work using the original multiclass label format.
  • Founta et al. (2018): 81.7 micro F1 was achieved by the unweighted multiclass BERT-base model on Founta et al. (2018) data, versus 79.1 for DeepHate.The comparison concerns multiclass Founta et al. (2018) data.
  • Related work: HABERTOR achieved SOTA on several other hate speech datasets, but Tran et al. (2020) did not evaluate it or BERT-base on Davidson or Founta data.Their work reports that BERT-base performs consistently near their SOTA on those other datasets.
Loading 2012.15606v2…