Source-linked AI summary
Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, Michael Wojatzki
TL;DR
Social media disseminate hateful messages, creating a need to assess annotation reliability for hate-speech detection. The paper conducts a comprehensive study across many annotators, finding that definitions can influence judgments without consistently improving reliability, which remains low overall.
Problem
Social media are used to disseminate hateful messages, motivating further investigation of annotation reliability for hate-speech detection.
Method
The paper investigates reliability through a comprehensive study across many annotators and compares agreement between groups across three annotation questions.
Results
Reliability was very low, ranging from α = .18 to .29, and did not consistently increase when participants were shown a definition.
Takeaways & Limitations
Hate speech is a vague concept requiring better definitions and guidelines, and detection may be treated as regression rather than binary classification.
Takeaways & Limitations
The study identifies the need to find characteristics that make users consider content hateful for future hate-speech detection models.
Abstract
from arXiv · showhide
Some users of social media are spreading racist, sexist, and otherwise hateful content. For the purpose of training a hate speech detection system, the reliability of the annotations is crucial, but there is no universally agreed-upon definition. We collected potentially hateful messages and asked two groups of internet users to determine whether they were hate speech or not, whether they should be banned or not and to rate their degree of offensiveness. One of the groups was shown a definition prior to completing the survey. We aimed to assess whether hate speech can be annotated reliably, and the extent to which existing definitions are in accordance with subjective ratings. Our results indicate that showing users a definition caused them to partially align their own opinion with the definition but did not improve reliability, which was very low overall. We conclude that the presence of hate speech should perhaps not be considered a binary yes-or-no decision, and raters need more detailed instructions for the annotation.
1 Introduction
The paper examines whether hate-speech annotations can reliably support automatic detection, motivated by widespread hateful content and ambiguity in human judgments. It compiles a corpus and estimates annotation reliability across different questions.
- Automatic detection could scan large volumes of internet text for hateful content and facilitate large-scale study of its diffusion.
- Reliable human annotations are essential because they determine what classifiers learn and bound expected detection performance.
- The authors conduct a comprehensive study with many annotators after preliminary agreement among six annotators was very low.
- The paper presents a first German hate-speech corpus focused on the European refugee crisis.
- The paper provides an estimate of hate-speech annotation reliability.
- The study investigates how annotation reliability changes with the exact question asked.
2 Hate Speech
Hate speech has expanded alongside social media and carries potential real-world consequences, but governments, organizations, and the public lack consensus about its definition. Existing guidelines and legal definitions leave room for interpretation or apply only within particular jurisdictions.
- Existing researcher guidelines leave room for personal interpretation because judgments about offensiveness can differ.
- Legal definitions are tied to specific jurisdictions and therefore do not capture every form of offensive or hateful speech.
- Social media expanded hate speech’s reach because users can post content visible worldwide with little more than an internet connection.
- Hate speech can precede or encourage hate crimes, become a health issue, and induce negative feelings toward its targets.
- The refugee crisis increased interest in controlling social-media hate speech while exposing limited consensus about what hate speech is.
3 Compiling A Hate Speech Corpus
Because no suitable German refugee-crisis hate-speech corpus existed, the authors compiled one from filtered Twitter data and began measuring its annotation characteristics. Preliminary expert agreement was low.
- The authors compiled their own corpus because no German corpus addressed their needs, especially the recent European refugee crisis.
- 13 766 tweets were gathered using 10 potentially insulting or offensive hashtags, then filtered for textual original tweets and duplicates.
- 541 tweets were split into six parts, with each part annotated by two of six rotated annotators for hate-speech presence.
- Offensiveness was rated on a 6-point Likert scale, which was also used in the later study.
- Krippendorff’s α = .38 among researchers familiar with the definitions indicated low agreement.
4 Methods
The study uses two online surveys to compare annotation judgments with and without the Twitter definition. Participants classified tweets as hate speech, judged whether they should be banned, and rated offensiveness.
- Two between-subjects online surveys with 56 participants examined non-experts’ agreement about hate speech in varied social-media content.
- One survey displayed the Twitter definition before and above every tweet, whereas the other supplied no definition.
- Each participant evaluated 20 randomly ordered tweets using hate-speech, banning, and 6-point offensiveness questions.
- Participants could explain or propose a hate-speech definition and provided sociodemographic information, including political orientation and migration background.
5 Preliminary Results and Discussion
The surveys produced 1120 annotations, and the definition influenced participants’ banning judgments but did not consistently improve reliability. Agreement remained very low across questions and groups.
- Study setup: 1120 annotations came from 56 participants rating 20 tweets each.Table 1 reports summary statistics for the resulting annotations.
- Analysis: The analysis compared the two survey groups on hate-speech judgments, banning judgments, and mean offensiveness ratings using WMW tests.WMW tests were chosen because preliminary Shapiro-Wilk tests indicated that some data were not normally distributed.
- Definition effects: Participants shown the definition were more likely to recommend banning tweets, and they gave different hate-speech and banning answers in only 3.6% of instances.This pattern suggests alignment between personal opinions and the supplied definition in that group.
- Reliability measure: Krippendorff’s α measured agreement rather than association and supports comparisons across any number of coders and nominal or interval data.The measure was selected because human coders should be interchangeable in content analysis.
- Reliability results: Agreement was very low, ranging from α = .18 to .29 across groups and questions.Krippendorff recommends α = .80, or α = .66 where some uncertainty is unproblematic; reliability did not consistently increase when participants saw a definition.
6 Conclusion and Future Work
The paper reports low agreement in its hate-speech corpus and treats hatefulness as more nuanced than a binary label. It proposes clearer coding criteria and considers richer annotation targets for future detection systems.
- Conclusion: The paper describes a hate-speech corpus and reports low agreement among users identifying hateful messages.These findings imply that hate speech is a vague concept requiring better definitions and guidelines for reliable annotation.
- Annotation implications: Researchers may collect multiple labels per tweet and average them, but this does not make the original data more reliable.More labels provide a more detailed picture of objective or intersubjective hatefulness.
- Future work: Future work will identify characteristics that make users consider content hateful for automatic detection and for studying why users disseminate hateful content.The proposed direction connects user judgments with models of hateful content and its dissemination.