Source-linked AI summary
Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech
Margherita Fanton, Helena Bonaldi, Serra Sinem Tekiroglu, Marco Guerini
TL;DR
Existing counter-narrative datasets and collection strategies do not reliably provide both high quality and high quantity, while manual intervention cannot match the scale of online hate. The paper iteratively refines a generative language model using expert-reviewed outputs; across loops, efficiency rises while novelty and diversity can decline, motivating dynamic adaptation.
Problem
Existing hate-speech/counter-narrative collection strategies tend to deliver quality or quantity rather than both, despite the data needs of neural generation and the impracticality of manual intervention at the scale of online hate.
Source: §1 · p. 1Method
Results
Efficiency increases across loops while novelty and diversity decrease; dynamic loop adaptation can improve overall dataset quality without significantly hindering efficiency.
Source: §9 · p. 9Takeaways & Limitations
The resulting methodology is reported to scale and support diverse, novel, and numerous examples, yielding a 5,000-pair English dataset spanning multiple hate targets.
Source: §1 · p. 2The authors argue that hate-countering generation should remain human-supervised rather than fully autonomous, with a human moderator making the final decision.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Undermining the impact of hateful content with informed and non-aggressive responses, called counter narratives, has emerged as a possible solution for having healthier online communities. Thus, some NLP studies have started addressing the task of counter narrative generation. Although such studies have made an effort to build hate speech / counter narrative (HS/CN) datasets for neural generation, they fall short in reaching either high-quality and/or high-quantity. In this paper, we propose a novel human-in-the-loop data collection methodology in which a generative language model is refined iteratively by using its own data from the previous loops to generate new training samples that experts review and/or post-edit. Our experiments comprised several loops including dynamic variations. Results show that the methodology is scalable and facilitates diverse, novel, and cost-effective data collection. To our knowledge, the resulting dataset is the only expert-based multi-target HS/CN dataset available to the community.
1 Introduction
Counter-narratives are non-aggressive responses to hate speech, but the scale of online hate makes manual intervention impractical and existing data-collection strategies struggle to provide both quality and quantity. The paper proposes an iterative, expert-reviewed language-model process and reports scalable collection of a multi-target dataset.
1 Introduction
- Counter-narratives are non-aggressive textual responses that use credible evidence and factual arguments to confront hate speech while respecting human rights.The paper presents them as an alternative to moderation actions such as content removal, account suspension, or shadowbanning.Source: §1 · p. 1
- Because the vast amount of online hate makes effective manual intervention impossible, NLP research has pursued semi- or fully automated counter-narrative generation.Prior collection strategies and datasets address the data needs of neural generation, but the paper characterizes existing strategies as granting either quality or quantity, not both.Source: §1 · p. 1
- The proposed approach reuses expert-validated, post-edited outputs to update a generative language model, aiming to reduce expert intervention while collecting high-quality counter-narratives for multiple targets.It builds on an author-reviewer framework in which reviewers filter and may post-edit generated hate-speech/counter-narrative pairs.Source: §1 · p. 1
- The experiments produced a 5,000-pair English HS/CN dataset covering multiple hate targets, and the authors describe it as the first semi-automatically constructed, expert-based multi-target dataset of its kind.The reported targets include race, religion, country of origin, sexual orientation, disability, and gender.Source: §1 · p. 2
- Across loops, the framework is reported to scale while improving collection efficiency and supporting dataset diversity, novelty, and quantity.The first session reduced reviewer post-editing effort or discarded examples; dynamic author configurations in the second session addressed unwanted behaviors or flaws.Source: §1 · p. 2
2 Related Work
Related work spans datasets for hate detection, methods for collecting counter-narratives, and hybrid human–model data collection. The paper positions its approach within efforts to combine human review with model-generated examples.
2 Related Work
- The related-work review organizes hate-countering research around detection datasets, countering datasets, and hybrid collection approaches.Source: §2 · p. 2
- Hate-detection datasets mainly use social-media or forum content; other resources add rationales or descriptions of implicit bias.Examples include datasets from Twitter [Waseem and Hovy, 2016; Waseem, 2016; Ross et al., 2017], Facebook, WhatsApp, and forums [de Gibert et al., 2018], as well as rationale annotations [Mathew et al., 2020] and SBIC [Sap et al., 2020].Source: §2 · p. 2
- Counter-narrative collection has used crawling, non-expert crowdsourcing, and NGO-expert nichesourcing.The passage notes only a few works focus on data collection for counter-narrative generation; the cited approaches are Mathew et al. (2018), Qian et al. (2019), and Chung et al. (2019).Source: §2 · p. 2
- Hybrid human-in-the-loop collection has been used for adversarial examples and offensive-language detection, with prior work reporting reduced model error or more efficient annotator work across rounds.Tekiroğlu et al. (2020) use a language model trained on seed hate-speech/counter-narrative pairs to generate pairs for human validation and post-editing.Source: §2 · p. 2
3 Methodology
The methodology iteratively combines GPT-2 generation with human validation and post-editing, returning reviewed examples to training. It begins with an expert-built seed dataset and compares successive collection sessions and author configurations.
3 Methodology
- At each loop, GPT-2 generates hate-speech/counter-narrative candidates for reviewers to filter and possibly post-edit; reviewed examples are added to the training data before the author is fine-tuned from scratch again.The author-reviewer pipeline is repeated over multiple data-collection loops.Source: §3 · p. 2, Figure 1 · p. 3
3.1 Seed dataset
- The V1 seed dataset contains 880 hate-speech/counter-narrative pairs collected through nichesourcing from 20 experts at two NGOs.Experts responded to prototypical hate texts for nine targets and wrote their own pairs; collection lasted roughly one month.Source: §3.1 · p. 3
3.2 Sessions
- The first session iterated four times from V1, collecting accepted or post-edited examples and reaching V5 with 3,000 pairs; the second ran four alternative configurations in parallel from V5, reaching 5,000 examples.Each second-session configuration collected 500 pairs; the first session used prior versions as training data and acquired 500 examples per loop, except V2, which contained 620.Source: §3.2 · p. 3
3.3 Author Models
- The author models are GPT-2 variants that differ in fine-tuning or conditioning; generation uses nucleus sampling with p = 0.9, and standard configurations are conditioned on the hate-speech start tag.The models were trained for three epochs with a batch size of 1,024 tokens and a learning rate of 2e-5.Source: §3.3 · p. 3
3.4 Reviewers
- Three reviewers filtered and post-edited generated pairs after two weeks of training; they could accept, amend, or discard pairs and were asked to verify factual claims and label accepted pairs by hate target.The collection campaign lasted 18 weeks, and an adapted mitigation procedure included workload limits, regular breaks, and weekly check-ins.Source: §3.4 · pp. 3–4
- Table 1 presents a GPT-2-generated hate-speech/counter-narrative example alongside its post-edited version and hate-target annotation.Source: Table 1 · p. 4
4 Metrics
The paper tracks target balance, reviewer acceptance and effort, novelty, repetition, and vocabulary expansion across collection loops. These measures capture dataset composition, review efficiency, and changes in language content.
4 Metrics
- Imbalance Degree measures how far hate-target category frequencies diverge from a perfectly balanced distribution; balanced multi-target datasets could support more representative generation models.Source: §4 · p. 4
- Acceptance Rate is the percentage of scrutinized pairs reviewers accept, either unchanged or after post-editing, and estimates the framework’s ability to produce reasonable-quality material.Source: §4 · p. 4
- HTER measures reviewers’ average number of edits over accepted pairs, with 0.4 used as an upper-bound threshold for easily post-editable pairs.The measure is adapted from sentence-level translation post-editing effort.Source: §4 · p. 4
- Novelty compares each loop’s examples with previously collected training data using Jaccard similarity, while Repetition Rate measures non-singleton n-gram types as an indicator of language diversity.Repetition Rate is independent of corpus size, allowing direct comparison across dataset versions.Source: §4 · p. 4
- Vocabulary Expansion tracks new tokens contributed by the author and reviewers at each loop, including tokens that appear first for one target but were already present for another.Source: §4 · p. 4
5 Session One
Across V2–V5, iterative retraining increased pair acceptance and reduced post-editing effort, while repetition, novelty, and target-distribution measures showed worsening patterns.
5 Session One
- For V2–V5, GPT-2 was fine-tuned on all earlier versions, and candidate pairs were generated by conditioning on the <|startofhs|> tag.Evaluation used micro averages; macro averages were also calculated to isolate target-class imbalance, with separate HS and CN micro averages.Source: §5 · p. 5
- Accepted modified pairs rose from 35.8 in V2 to 50.1 in V5, and untouched pairs from 1.5 in V2 to 10.9 in V5; HTER fell below 0.4 after V3.The authors conclude that HITL improved collection efficiency compared with a static approach represented by V2.Source: §5 · p. 5, Figure 2 · p. 5, Figure 3 · p. 5
- Repetition Rate increased and novelty declined across versions, while target-distribution ID rose from 2.2 in V1 to 4.5 in V5.The authors interpret the cumulative novelty trend as indicating a possible saturation point for obtaining novel material.Source: §5 · p. 5, Figure 3 · p. 5, Figure 2 · p. 5
- Untouched pairs averaged 30.7 tokens versus 37.3 tokens for other accepted pairs before post-editing, and were somewhat stereotypical, adding little novelty.Source: §5 · pp. 5–6
6 Session Two
Session Two tested four parallel V6 author configurations to address problems observed in Session One. The configurations improved target balance and produced more accepted modified pairs, while quality outcomes varied across configurations.
6 Session Two
- V6,SBF conditioned GPT-2V5 on offensive statements from SBIC, while V6,LAB encoded the hate target in a starting token.V6,ARG used argument-pair fine-tuning on Kialo before fine-tuning on V1–V5 data; V6,MIX combined the three approaches.Source: §6 · p. 6
- To balance data across seven hate targets, all configurations incorporated target labels explicitly or implicitly, and reviewers received only the first five pairs from each generated chunk.Source: §6 · p. 6
- V6 configurations raised the average Imbalance Degree balance score to 2.3 from 4.5 for V5, and V6,MIX led modified-pair acceptance at 66.15% while V6,ARG led untouched-pair acceptance at 6.37%.All V6 configurations had slightly higher acceptance rates than V5; the authors report a strong decrease in untouched pairs and increase in modified pairs.Source: §6 · pp. 6–7, Figure 5 · p. 7
- Overall HTER increased for all V6 approaches, but HTER for modified pairs differed less between V5 and V6, and every V6 configuration remained below the 0.4 acceptability threshold.V6,ARG had the lowest HTER for modified and untouched pairs; V6,LAB had higher HTER, which the authors relate to its label-only conditioning and fewer untouched pairs.Source: §6 · p. 7, Figure 6 · p. 7
- All V6 configurations increased novelty over V5 and expected V6 except V6,ARG; CN-only novelty was 0.741–0.745 versus 0.737 for V5, with V6,MIX highest at 0.76.The authors attribute V6,ARG's exception possibly to conditioning on HSs from V1–V5.Source: §6 · p. 7, Figure 7 · p. 7
- All V6 configurations had Repetition Rates below V5 and expected V6, with V6,ARG lowest at 5.474 and V6,LAB highest among V6 configurations.The authors describe the overall Session Two pair quality in novelty and Repetition Rate as much higher than before, while V6 produced more sub-optimal but fewer perfect pairs.Source: §6 · pp. 7–8
7 Vocabulary analysis.
Across loops V2–V5, GPT-2 added new vocabulary at a higher average rate than reviewers did during post-editing, and its contribution did not reach saturation.
7 Vocabulary analysis
- Across V2–V5, GPT-2 injected a higher average percentage of new words into the dataset than reviewers added during post-editing, and the trend lines did not converge.The authors say GPT-2's ongoing addition of new material indicates it had not reached a saturation point; reviewer contributions were limited by instructions to make minimal edits.Source: §7 · p. 8
- GPT-2's generation increasingly used words associated with the same target, while words first seen in other targets showed that cross-fertilization persisted across loops.The authors report that V6 versions increased both the injection of novel words and GPT-2's cross-fertilization ability.Source: §7 · p. 8
8 Qualitative Analysis
The qualitative analysis discusses when argumentation can support counter narratives, examples of novel arguments, and generation for unseen hate targets.
8 Qualitative Analysis
- Using a counter argument as a counter narrative is an assumption, since logical validity alone does not ensure an acceptable CN, which also presupposes values.The authors note that argumentation across varied topics is not enough to provide a meaningful CN in response to hate speech.Source: §8 · p. 8
- Manual analysis and annotator discussions found CNs with genuinely novel arguments absent from training data, including a response using data on suicidal attempts among homosexual youth.Source: §8 · p. 8
- GPT-2 generated HS/CN pairs for unseen, including intersectional, targets, but limited commonsense knowledge could yield out-of-scope results such as addressing muggleborns.Source: §8 · p. 9
9 Conclusions
The paper presents an author-reviewer human-in-the-loop methodology in which experts validate data used to iteratively refine a language model. Its experiments find that iteration increases collection efficiency while reducing novelty and diversity, and that dynamic loop adaptations can improve overall dataset quality without significantly hindering efficiency.
9 Conclusions
- The author-reviewer methodology iteratively refines a language model using data from previous loops that experts have validated.Source: §9 · p. 9
- As loops iterate, data-collection efficiency increases while dataset novelty and diversity decrease; additional dynamic loop adaptations improve overall quality without significantly hindering efficiency.Source: §9 · p. 9
A.1 Vocabulary expansion algorithm
The appendix defines vocabulary expansion by classifying words by version and target, then reports changes in quality metrics and target distributions across collection loops. Session Two largely follows the pair-level findings, with a reported exception for V6,ARG hate speeches.
A.1 Vocabulary expansion algorithm
- Vocabulary expansion classifies words by version and target, then calculates macro-average percentages across word categories for each target and version.Novelty is defined relative to the collective vocabulary of previous versions; the algorithm distinguishes words from post-edited and generated pairs.Source: §A.1 · p. 12
A.2 Additional material for Session One
- CN HTER decreases clearly; HS HTER is less consistent, with higher V3/V4 scores, while HS repetition rises faster and exceeds CN repetition.From V3 onward, HS HTER declines, reaching below CN-only HTER in V5.Source: §A.2 · p. 12, Figure 9 · p. 12
- HS novelty scores are lower than CN scores and decline faster; the passage attributes this to more frequent prototypical HSs.Source: §A.2 · p. 12, Figure 10 · p. 12
- MUSLIMS covers a significant share in every loop and over half of V5 pairs; JEWS, MIGRANTS, and DISABLED decline while other targets remain stable.Source: §A.2 · p. 12, Figure 11 · p. 13, Table 3 · p. 13
A.3 Additional material for Session Two
- Session Two mostly follows the HS/CN-pair conclusions; V6,ARG HSs are an exception, with cumulative-novelty effects attributed to conditioning GPT-2 on training-set hate speeches.Those HSs also have lower HTER and higher repetition rate.Source: §A.3 · p. 13, Figure 13 · p. 13, Figure 12 · p. 13
A.4 Tables
- The appendix tables report HS/CN-pair results, separate HS and CN results, and examples of pairs before and after post-editing.Source: §A.4 · p. 13, Table 5 · p. 14, Table 6 · p. 14