Source-linked AI summary
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, Douwe Kiela
TL;DR
Online hate detection models remain inaccurate, non-robust and difficult to generalise, motivating more challenging training data. The paper iteratively generates adversarial datasets with annotators and models, and reports better later-round performance, including 95% HATECHECK accuracy in Round 4.
Problem
Existing hate-detection systems lack accuracy, robustness and generalisability, producing errors that limit downstream moderation and hate measurement.
Method
The paper uses four rounds of human-and-model-in-the-loop data generation, including adversarial content and contrast-set perturbations, to retrain hate-detection models.
Results
Later-round models perform better on test sets, are harder for annotators to trick, and improve HATECHECK accuracy from 60% in Round 1 to 95% in Round 4.
Takeaways & Limitations
Dynamic dataset generation yields large, finely annotated datasets with challenging content and models that perform better and more robustly, including on out-of-domain functional tests.
Takeaways & Limitations
The approach requires substantial infrastructure, resources and domain expertise, while time-constrained annotation and perturbation work can add pressure and training needs.
Abstract
from arXiv · showhide
We present a human-and-model-in-the-loop process for dynamically generating datasets and training better performing and more robust hate detection models. We provide a new dataset of ~40,000 entries, generated and labelled by trained annotators over four rounds of dynamic data creation. It includes ~15,000 challenging perturbations and each hateful entry has fine-grained labels for the type and target of hate. Hateful entries make up 54% of the dataset, which is substantially higher than comparable datasets. We show that model performance is substantially improved using this approach. Models trained on later rounds of data collection perform better on test sets and are harder for annotators to trick. They also perform better on HateCheck, a suite of functional tests for online hate detection. We provide the code, dataset and annotation guidelines for other researchers to use. Accepted at ACL 2021.
1 Introduction
The paper addresses difficult and potentially unsafe online hate detection by iteratively combining annotator challenges with model retraining. Across four rounds, this process produces challenging, finely labelled data and improves model performance.
- Online hate detection remains difficult because state-of-the-art models raise concerns about performance, robustness, generalisability and fairness.
- The process uses four rounds in which annotators try to trick the current model, then newly generated examples train the next model.The approach could, in principle, continue indefinitely.
- Rounds 2–4 combine original content with contrast-set perturbations that minimally change text to flip its label.Rounds 3 and 4 additionally target specific hate types and draw on real-world hate sites to increase adversarial realism and variety.
- 60% to 95%: HATECHECK accuracy improves from Round 1 to Round 4 as later-round models become more accurate and harder for annotators to trick.Later-round test sets are more adversarial, while models trained on later rounds achieve higher accuracy.
- The resulting dataset contains 40,000 entries with 54% hate and fine-grained annotations for label, hate type and target.
2 Background
Existing hate-detection systems suffer from accuracy, robustness and generalisability problems, partly because static datasets encourage failures on challenging content. Dynamic benchmarking and contrast sets provide adaptive alternatives, but their combination remained under-explored for hate detection.
- Existing-dataset systems produce false positives and false negatives that can make them unsuitable for content moderation or measuring online hate.
- False positives can arise from lexical similarity and overrepresentation of identity terms in toxic training data, encouraging models to overfit spurious cues.
- Simple obfuscations, target substitutions, implicit language and complex forms such as sarcasm can provoke false negatives.Leetspeak is one example of obfuscation that replaces letters with similar-looking numbers or symbols.
- Dynamic dataset creation repeatedly asks annotators to produce adversarial samples that challenge baseline models, enabling more targeted data collection.
- Contrast sets use minimal label-changing modifications to reduce overfitting on spurious correlations, and prior work reports improved robustness after contrast-set fine-tuning.
- No prior hate-detection work had combined dynamic benchmarking and contrast sets within one system, according to the paper.
3 Dataset labels
The paper uses a hierarchical labelling scheme that combines binary hate classification with secondary labels for hate type and target. It balances granularity, conceptual distinction and annotation simplicity while covering diverse identities and intersectional targets.
- The taxonomy assigns every entry either ‘Hate’ or ‘Not Hate’, while hateful entries receive secondary labels for hate type and target.The scheme addresses limitations of binary labelling while acknowledging that established taxonomies and standards remain limited.
- Hate is defined as abusive speech targeting specific group characteristics, including ethnic origin, religion, gender or sexual orientation.
- The hate-type taxonomy includes derogation, animosity, threatening language, support for hateful entities and dehumanization.
- Annotators focused on 29 identities, such as women, Black people, Muslims, Jewish people and gay people, plus some intersectional variations.
- Men, white people and heterosexuals were considered out of scope for Hate in the annotation scheme.
4 Annotation
Annotation used an open-source platform that supports dynamic dataset creation and model benchmarking, with expert oversight and guidelines updated after each round.
- An open-source web platform supported human-and-model-in-the-loop dataset creation and model benchmarking for NLP tasks.
- Two online-hate experts oversaw annotation, while guidelines were revised after each round as annotators needed more detail.The process also followed guidance for protecting and monitoring annotator well-being.
5 Dataset formation
The dataset was built over four rounds, combining model-guided annotation, adversarial perturbations, validation, and progressively more realistic and varied hate content. Each round contributed roughly 10,000 entries while later rounds added fine-grained labels and challenging examples.
- 5 Dataset formation: 41,255 entries were generated over four rounds, each involving approximately 10,000 entries, then split into 80% training, 10% development, and 10% test data.Approximately half of each test set came from annotators absent from the training and development sets.
- 5 Dataset formation: Every round placed a target model in the loop, with each model trained on data collected in previous rounds using a consistent RoBERTa classification architecture.The target model supplied the system that annotators attempted to challenge in the next round.
- 5.1 Round 1 (R1): 11,157 entries were collected in Round 1, including 7,197 Hate entries (65%) and 3,960 Not Hate entries (35%).Round 1 did not record the type and target of hate.
- 5.2 Round 2 (R2): Round 2 collected 9,996 entries using adversarial pivots, offline perturbations, and validation by additional annotators.Perturbations aimed to make minimal changes while preserving realism, label criteria, diversity, and the requested label flip.
- 5.2 Round 2 (R2): Round 2 validation identified template statements, unrealistic examples, and annotator biases, leading to a template ban, revised realism guidance, and held-out-annotator evaluation.Performance dropped substantially when annotators’ content was held out rather than stratified across splits.
- 5.3 Round 3 (R3): Round 3 collected 9,950 entries, half original and half perturbed, while drawing on real-world hateful content and specific hate types.Its hateful entries were distributed across Derogation, Dehumanization, Threats, Support for hateful entities, and Animosity.
- 5.3 Round 3 (R3): Krippendorff’s alpha was 0.55 for Round 3 original entries, indicating moderate agreement, with greater disagreement associated with Animosity labels.This agreement was lower than in Round 2 but comparable with another cited hate-speech dataset.
- 5.4 Round 4 (R4): Round 4 collected 10,152 entries by targeting specific identities and seeking both challenging hateful and non-hateful content from relevant online communities.The round was balanced between 5,076 Hate and 5,076 Not Hate entries and included original and perturbed content.
6 Model performance
Performance improves as models are trained on progressively harder, adversarial data: later-round models are harder to trick, perform better across test sets, and score higher on HATECHECK.
- Model error rate: 54.7% of entries tricked M1, compared with 27.7% for the final model M4.M1 was trained on public hate-speech datasets, whereas M4 was trained after successive dynamic rounds.
- Test set performance: Annotator-generated content becomes harder to classify in later rounds, lowering all models’ performance on later-round test sets.The paper attributes this increasing difficulty to annotators becoming more experienced and adversarial.
- Test set performance: M4 achieves macro F1 of 75.97 on R4 test data, versus 74.83 for M3 and 60.87 for M2.M4 is also the best-performing model on R1, R2, and R4 data.
- Test set performance: Cumulative training generally outperforms training on a single round; on R4 data, M4 reaches 75.97 macro F1 versus 69.6 for M(R4 only).Models trained on only one round can perform well on some rounds but substantially worse on others.
- HATECHECK: HATECHECK accuracy rises from 60% for M1 to 95% for M4, exceeding the best prior reported model’s 77% overall accuracy.M4 is consistent across classes, with 95% on Hate and 93% on Not Hate.
- HATECHECK: HATECHECK provides negative predictive evidence: it identifies particular model weaknesses but does not necessarily establish generalizable strengths.The suite contains functional tests covering hateful expressions and non-hateful contrasts.
7 Discussion
Dynamic dataset generation addresses the difficulty of building hate-detection systems that are accurate, robust, and generalisable across targets, types, and domains. Iterative model feedback produces challenging, varied data and improves later-round model performance, while requiring substantial infrastructure and expertise.
- Later-round models show higher test-set accuracy and better HATECHECK performance, indicating improved performance and robustness.The paper reports that target-model performance improves over four rounds, with later models performing better on HATECHECK.
- Dynamic dataset generation supports iterative correction of design flaws while data collection and model training proceed.Annotator guidelines and strategies were refined during the process, and perturbations were introduced to increase content difficulty.
- Real-time target-model feedback enables annotators to create increasingly challenging content designed to expose model weaknesses.Discussion sessions showed that annotators adjusted their entries across rounds to find better ways to trick the model.
- The dataset is approximately 54% hate, covers many targets, and includes challenging content such as slurs and identity referents.These properties are intended to provide varied examples for model learning while meeting machine-learning requirements.
- The approach requires substantial infrastructure, domain expertise, and careful annotation training, while time pressure and perturbations may introduce biases.The project depended on a low-latency model-serving interface and a cross-disciplinary team, and dynamic validation creates additional annotation pressures.
8 Conclusion
The paper concludes that human-and-model-in-the-loop collection produced large, expert-annotated hate datasets with fine-grained labels and difficult perturbations. Models trained on these datasets performed better on hate detection, including out-of-domain functional tests, while future work will broaden annotators, models, and evaluation settings.
- The process collected four rounds of large, high-quality datasets using expert annotators, fine-grained hate labels, and difficulty-increasing perturbations.The labels cover hate type and target.
- Models trained on dynamically generated datasets performed better at hate detection, including on out-of-domain functional test suites.
- Future work will expand annotator diversity, evaluate models beyond RoBERTa, and investigate alternative training and evaluation combinations.The authors also identify multi-label classification of hate type and target as a future direction.
Impact Statement & Ethical Considerations
The paper describes synthetic, expert-created data and intended uses for more accurate, robust, and fine-grained hate classification. It also identifies misuse risks and operational challenges, while illustrating the varied linguistic phenomena included in the annotation framework.
- Ethical considerations: The dataset entries were synthetically created or substantially adjusted from real-world inspiration, rather than copied directly from online sites.
- Intended use: The dataset and models are intended to support more accurate and robust hate detection and classification, including multiclass analysis of hate types and targets.
- Potential misuse: The dataset and models could be misused to train generative hate-speech systems or investigate ways to evade content moderation.The paper states that these outcomes are possible in principle but considers them unlikely and weighs the overall risks against the research benefits.
- Challenging content: The annotation framework includes difficult cases involving convoluted syntax, positive sentiment, rhetorical questions, misspellings, rare synonyms, and long phrases.
- Non-hate cases: It also includes non-hate cases such as counter-speech, negation, polysemy, non-hateful profanity, and negativity toward non-identity targets.
D Model, Training, and Evaluation Details
The models used a RoBERTa-base sequence-classification architecture and were trained with a largely standardized optimization setup. Target-model training varied by round, while non-target models used multiple seeds and early stopping.
- Model: The model architecture was RoBERTa-base with a sequence-classification head and approximately 125 million parameters.
- Training: Training used a learning rate of 2e-5 with Hugging Face's default optimizer, and each model trained for approximately one day using 8 GPUs.
- Target-model training: The R2 target used three epochs and selected five multiples of Round 1 training data as the best configuration among the searched options.
- Evaluation details: Non-target models used five random seeds and early stopping based on development-set loss over four training epochs.
E Data statement
The paper documents the dataset’s provenance and construction, including synthetic data generation, English-only scope, and the possibility of applying the system to other languages.
- Documentation: A data statement documents the final dataset’s creation process and provenance.
- Dataset construction: The dataset was created as a 40,000-entry synthetic corpus over four rounds using a model-in-the-loop interface and trained annotators.Data was generated rather than sampled.
- Language scope: All content is in English because of the available annotation team, resources, and project leaders’ expertise.The authors state that the system could, in principle, be applied to other languages.
C. SPEAKER DEMOGRAPHICS
Because the dataset is synthetic, its speakers are the annotators; the paper reports annotator characteristics and the identities prioritized during collection.
- Speaker demographics: The synthetic dataset uses annotators as its speakers.
- Annotator demographics: Twenty annotators received onboarding, training, expert feedback, and well-being support while contributing over the four-round project.Guidelines were updated after each round, and annotators could consult experts in real time.
- Annotator demographics: Annotators included multiple genders, age groups, education levels, nationalities, language backgrounds, and ethnic identities.Most were 18–29, female, native English speakers, and ethnically white, according to the reported demographics.
- Target identities: Data collection focused on identities spanning disability, gender, immigration status, race or ethnicity, religion, sexual orientation, national origin, and class.