Source-linked AI summary
A Benchmark Dataset for Learning to Intervene in Online Hate Speech
Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, William Yang Wang
TL;DR
Online hate-speech research has emphasized detection while often ignoring conversational context and responses that may discourage future hate speech. This paper introduces a generative intervention task with fully labeled Gab and Reddit conversation datasets, and its human evaluations show substantial room to improve automated responses.
Problem
Prior hate-speech datasets and methods largely focus on isolated-post detection, leaving conversational context and intervention responses insufficiently addressed.
Method
The paper introduces a generative hate speech intervention task and two fully labeled conversation datasets from Gab and Reddit with crowd-sourced intervention responses.
Results
Human-written responses were selected as the most effective and diverse option 70% or more of the time on both datasets.
Takeaways & Limitations
The evaluated Seq2Seq, VAE, and RL models leave substantial opportunity for improving automated hate-speech intervention responses.
Takeaways & Limitations
The experiments use only post text, although user information and Reddit submission titles are also available.
Abstract
from arXiv · showhide
Countering online hate speech is a critical yet challenging task, but one which can be aided by the use of Natural Language Processing (NLP) techniques. Previous research has primarily focused on the development of NLP methods to automatically and effectively detect online hate speech while disregarding further action needed to calm and discourage individuals from using hate speech in the future. In addition, most existing hate speech datasets treat each post as an isolated instance, ignoring the conversational context. In this paper, we propose a novel task of generative hate speech intervention, where the goal is to automatically generate responses to intervene during online conversations that contain hate speech. As a part of this work, we introduce two fully-labeled large-scale hate speech intervention datasets collected from Gab and Reddit. These datasets provide conversation segments, hate speech labels, as well as intervention responses written by Mechanical Turk Workers. In this paper, we also analyze the datasets to understand the common intervention strategies and explore the performance of common automatic response generation methods on these new datasets to provide a benchmark for future research.
1 Introduction
The paper addresses online hate speech by proposing generative intervention rather than relying only on detection and blocking. It introduces conversational, human-intervention datasets from Gab and Reddit for developing and evaluating automated responses.
- Motivation: Existing hate-speech research mainly detects or blocks harmful content, while this paper targets responses that intervene during hateful conversations.The authors motivate intervention as a way to encourage more speech while addressing harmful language.
- Task: The generative hate speech intervention task automatically generates responses intended to counter hate speech in online conversations.The task extends beyond identifying hateful posts to producing intervention responses.
- Datasets: The datasets contain 5K Reddit conversations and 12k Gab conversations, retaining conversational context and adding human-written intervention responses.The paper identifies context and intervention responses as important for training generative mitigation models.
- Datasets: The two datasets are fully labeled for hate or non-hate speech and can also support hate-speech detection experiments.Mechanical Turk workers manually labeled posts, and classifier performance is reported separately.
- Contribution: The datasets focus on Gab and Reddit, platforms that the authors identify as underrepresented in prior hate-speech research.This complements the paper’s emphasis on conversational rather than isolated-post data.
2 Related Work
Prior hate-speech datasets largely center on Twitter and classify isolated posts or users. The paper distinguishes its datasets by focusing on Reddit and Gab conversations and by including intervention responses.
- Prior datasets: Earlier Twitter datasets differ in sampling and labeling design, including hate-related keyword selection, suspicious-user selection, random sampling, and fine-grained categories.These approaches produce datasets with different coverage and potential selection or class-balance issues.
- Prior datasets: One prior multi-source dataset combines Twitter, Reddit, and The Guardian, containing 20,432 comments with 4,136 labeled as harassment.Its labels concern harassment rather than the conversational intervention task introduced here.
- Comparison: Table 1 compares the paper’s datasets with previous hate-speech datasets using conversation and intervention dimensions.The caption defines Conv. as conversation and Interv. as intervention.
- Dataset sources: Most publicly available hate-speech datasets are collected from Twitter, while hate speech on Reddit and Gab is less widely studied.Other sources mentioned include Instagram, Yahoo!, and Ask.fm.
- Research focus: Previous datasets primarily support hate-speech classification or detection from a single post or user, omitting conversational context and intervention methods.The paper frames intervention as a distinct need beyond isolated-instance detection.
3 Dataset Collection
The authors collect conversational data from targeted Reddit communities and Gab, filter and annotate the conversations, and solicit short intervention responses from multiple Mechanical Turk workers.
- Collection: Reddit data are retrieved from ten selected subreddits by collecting the top 200 hottest submissions and focusing on conversations containing hate keywords.The selection strategy is designed to obtain high-quality conversational data likely to include hate speech.
- Data filtering: Conversations longer than 20 comments are filtered out before crowd-sourcing to reduce worker burden.Each assignment contains five conversations.
- Crowd-sourcing: Workers identify hateful posts and, when hate speech exists, write an intervention response limited to 140 characters.Conversations judged free of hate speech receive “n/a” for both questions.
- Crowd-sourcing: Workers receive a hate-speech definition and instructions discouraging hateful or overly general intervention responses.For Reddit, workers also see the submission title and content for additional context.
- Data quality: Each conversation is assigned to three workers, with hate labels aggregated by majority and requiring at least two of three workers to mark a comment as hateful.The response aggregation follows the aggregated hate-speech result.
4 Dataset Analysis
The datasets differ substantially in size, conversation structure, label balance, and keyword distributions, while collected interventions reveal several recurring response strategies.
- Dataset statistics: 5,020 Reddit conversations contain 22,324 comments, including 5,257 hateful and 17,067 non-hateful comments.A total of 10,243 intervention responses were collected, averaging 2.66 per hateful conversation.
- Dataset statistics: 11,825 Gab conversations contain 33,776 posts, including 14,614 hateful and 19,162 non-hateful posts.The dataset contains 31,487 originally collected intervention responses, averaging 2.82 per hateful conversation.
- Dataset comparison: 94.5% of Gab conversations contain hate speech, compared with 76.6% of Reddit conversations.The datasets also differ in average conversation length, post or comment length, and label balance.
- Dataset comparison: The Gab and Reddit datasets share popular hate keywords but have substantially different keyword distributions and other source characteristics.Reddit conversations and comments are longer, while its hate and non-hate labels are more imbalanced.
- Intervention strategies: Common intervention strategies include identifying hate keywords, categorizing hate speech, using a positive tone before a warning, and suggesting proper actions.The positive-tone strategy combines affirmation with a transition into an explanation or warning.
5 Generative Intervention
The paper frames generative hate speech intervention as generating responses that mitigate hate speech during conversations. It benchmarks established response-generation methods and adapts reinforcement learning to this single-response setting using a semantic-coherence reward.
- Generative hate speech intervention generates responses intended to mitigate hate speech during an online conversation.
- Seq2Seq and VAE serve as baseline methods for assessing the initial feasibility of automatically generating intervention responses.
- The task differs from dialog generation because it has no multiple utterance turns, so the agent selects response tokens from the input posts and previously generated tokens.
- Rewards for ease of answering and information flow do not apply, while semantic coherence remains applicable to generative intervention.
- The reinforcement-learning reward combines log p(r|c) and log pback(c|r), weighted by λ1 and λ2 after response generation.p(r|c) models generating the response from the conversation, while pback(c|r) models generating the conversation from the response using another generation network.
6 Experiments
The experiments benchmark hate-speech detection and generative intervention on separate Gab and Reddit datasets, using automatic and human evaluation. Results show strong dataset-dependent performance differences and a mismatch between automatic metrics and human judgments of intervention quality.
- Experimental setup: The experiments evaluate four binary hate-speech classifiers and three generative intervention models on separate Gab and Reddit datasets.Detection uses LR, SVM, CNN, and RNN; generation uses Seq2Seq, VAE, and RL.
- Experimental setup: 20% of each dataset is randomly held out for testing, with natural class distributions retained for detection evaluation.Detection is evaluated using F-1, PR AUC, and ROC AUC.
- Dataset comparison: Gab consistently outperforms Reddit for both classification and generation, plausibly because it is larger, more balanced, and less noisy.Reddit posts and conversations are longer, while its training set is smaller.
- Generative results: The three generative models perform similarly on automatic metrics, while Seq2Seq benefits from filtered-conversation inputs and VAE does not.Filtered conversations contain only posts labeled as hate speech; this setting was necessary for evaluating RL.
- Generative results: Reinforcement learning does not improve automatic scores, but human evaluation suggests its responses may be more effective and diverse, especially on Gab.On Reddit, RL and Seq2Seq responses have similar effectiveness and diversity.
- Human evaluation: Human-written responses are selected as most effective and diverse at least 70% of the time, exposing substantial room for automated improvement.The authors argue that universal responses can score well automatically while ignoring conversation context and topic.
7 Conclusion
The paper proposes generative hate speech intervention and introduces two fully labeled datasets from Reddit and Gab with crowd-sourced intervention responses.
- The authors propose generative hate speech intervention and two fully labeled datasets collected from Reddit and Gab with crowd-sourced intervention responses.