Source-linked AI summary
Dealing with Annotator Disagreement in Hate Speech Classification
Somaiyeh Dehghan, Mehmet Umut Sen, Berrin Yanikoglu
TL;DR
Subjective interpretations make hate-speech labels especially prone to disagreement on subtle or borderline content, while common approaches either discard such samples or force a single label. This paper evaluates aggregation strategies and perceived-strength supervision for Turkish tweets. On the Silver test set, all-data Max aggregation performs best on average across the three classification tasks, and strength scores offer a practical annotation alternative.
Problem
Subjective interpretations of hate speech produce frequent disagreement, while discarding non-consensus samples or forcing a single gold label fails to preserve those varied judgments.
Source: §1 · pp. 1–2Method
The paper evaluates consensus and majority subsets, ordinal label aggregation, and perceived-strength scores for Turkish hate-speech classification.
Source: §1 · p. 2, §4.2 · p. 9, §5.3 · p. 14Results
On the Silver test set, all-data Max aggregation achieves the strongest average performance across three classification granularities, followed closely by clear-majority-only training.
Source: §1 · p. 2, §6 · p. 15Takeaways & Limitations
Perceived-strength annotations can be collected more quickly than fine-grained categories and used as coarse supervision or an initial stage before detailed categorization.
Source: §6 · p. 15The ordinal assumption is motivated by category-strength distributions even though a strict severity order among categories is not accepted in the literature.
Source: §4.2 · p. 9
Abstract
from arXiv · showhide
Hate speech detection is a crucial task, especially on social media where harmful content can spread quickly. Collecting social media content (tweets etc.) to train machine learning models is easy, but detecting and categorizing hate speech can be difficult due to the inherently subjective nature. This subjectivity leads to frequent disagreement among annotators, particularly for subtle or borderline content. Traditional approaches either discard non-consensus samples or force a ''gold standard'' through expert adjudication, ignoring valuable information about uncertainty and diverse human perspectives. We examine the largely overlooked problem of annotator disagreement in hate speech classification and evaluate a range of aggregation methods, including majority voting, ordinal strategies (minimum, maximum, and mean), and analyze their impact across binary, 4-class, and 6-class classification tasks. In addition, we leverage annotators' perceived hate speech strength scores to explore regression-based and hybrid modeling approaches. Among others, we show that filtering non-consensus samples results in over-optimistic results and that the perceived strength provides a complementary signal that enhance classification performance. Finally, we establish new state-of-the-art results for hate speech detection in Turkish tweets, and demonstrate that annotator disagreement, when properly modeled, is a valuable resource for building more robust and reliable systems.
Disclaimer:
The authors note that examples of offensive language, hate speech, and profanity are included because of the study’s subject, not to express their opinions.
Disclaimer:
- Examples of offensive language, hate speech, and profanity reflect the study’s subject matter, not the authors’ opinions.The authors state that the work aims to aid detection and prevention of harmful content and violence against minorities.Source: Disclaimer · p. 1
1 Introduction
Subjective, personal, cultural, and contextual interpretations make annotator disagreement common in hate speech classification, especially for subtle or implicit content. This work evaluates ways to preserve and aggregate disagreement in Turkish social-media data, reporting stronger results on the Silver test set when all training data are used with Max aggregation.
1 Introduction
- Hate speech judgments reflect personal, cultural, and contextual perspectives, with frequent disagreement on subtle or implicit discrimination.Annotators more readily agree on explicit threats or slurs than on borderline examples.Source: §1 · pp. 1–2
- Discarding disagreement removes challenging borderline cases, while forcing one gold label overlooks subjective variation; distribution modeling may be poorly suited to datasets with few annotators.The introduction identifies majority voting, adjudication, and distribution modeling as existing approaches with these limitations.Source: §1 · p. 2
- The paper evaluates aggregation strategies, including ordinal minimum, maximum, and average assignments, and explores perceived hate-speech strength as an alternative or complement to detailed categories.It also reports state-of-the-art Turkish-tweet detection and classification results using fine-tuned BERT models.Source: §1 · p. 2
- On the larger Silver test set, all-data Max aggregation has the strongest average performance across three classification granularities, followed closely by clear-majority-only training.The authors report that consensus-only training performs strongly on the Gold test set but significantly worse on Silver.Source: §1 · p. 2
2 Related Work
Prior work documents subjective disagreement in hate-speech annotation and explores soft labels, perspective-aware methods, context, and annotator characteristics. This paper focuses on strategies for combining potentially multiple labels.
2 Related Work
- Prior research identifies annotation subjectivity and disagreement as challenges that can affect dataset quality and model performance.It also notes ambiguities such as covert intent and hate discourse conveyed in images.Source: §2 · p. 2
- Soft-label and disagreement-aware approaches model annotation variation, while other work explores hierarchical, individual, group-based, and perspective-specific labels.Studies also report gains from perspective-specific models, but Casola et al. (2023) found mixed hate-speech results amid greater annotator variability.Source: §2 · pp. 3–4
- Studies report that annotator perspectives and annotation paradigms correspond to different agreement and test-set performance patterns.Reported examples include optimistic-versus-pessimistic annotator performance differences, low agreement in descriptive annotation, and disagreements that may reflect valid interpretations.Source: §2 · pp. 3–5
- Context can improve annotation consistency, especially for replies, while demographic representation and annotator characteristics can affect judgments.Prior work also reports that workers may need original posts, replies, or images to make accurate judgments.Source: §2 · pp. 3–4
- The paper builds on this work by focusing on strategies for combining annotations that may contain multiple labels.The authors identify a gap in thorough examination of annotation processes and their challenges.Source: §2 · pp. 2, 5
3 Dataset
The dataset contains Turkish tweets across five topics and uses both prescriptive category labels and descriptive perceived-strength scores. Its annotation guidelines allow multiple labels when one category cannot capture a tweet, while such cases form a minority of annotations.
3 Dataset
- The dataset contains 11,021 Turkish samples across five topics, including immigration, the Israel-Palestine conflict, anti-Greek sentiment, religion/race/ethnicity, and LGBTQ+.The Israel-Palestine topic includes tweets posted before October 7, 2023.Source: §3 · p. 5
3.1 Annotation Process
- Three annotators independently labeled randomly assigned batches of 50 tweets using prescriptive guidelines and Label Studio.The guidelines define six categories, from no hate speech to threats of harm.Source: §3.1 · p. 6
- The guidelines permit multiple category labels when a tweet spans categories or no single category fits, and about 16% of annotations are multi-labeled.The guidelines were iteratively refined to address ambiguities, including covert hate speech and multiple targets.Source: §3.1 · p. 6
- Alongside category labels, annotators rated perceived hate-speech strength on a 0–10 scale without additional guidelines to capture their natural perceptions.The annotators chose the finer-grained 0–10 scale over a 0–5 alternative.Source: §3.1 · pp. 6–7
4 Problem Formulation and Methodology
Section 4 evaluates ways to handle annotator disagreement, then uses perceived strength to motivate ordinal aggregation, consolidates categories, and describes BERT-based classification and strength prediction.
4.1 Handling Annotator Disagreement
- Consensus filtering can inflate performance metrics, while reconciliation is resource-intensive and majority voting still forces subjective judgments into one label.The authors argue that discarding non-consensus cases removes challenging examples and ignores subjective interpretations.Source: §4.1 · p. 7
4.1.1 Majority Voting for Multi-Label Annotations
- In multi-label annotation, each annotator may select multiple classes; majority voting returns the class set with the highest vote count, including ties, as Figure 1 illustrates.The figure contrasts full agreement, clear majority, and no clear majority.Source: §4.1.1 · pp. 7–8, Figure 1 · p. 8
4.1.2 Strategies When Majority Vote Is Not Clear
- For unclear majorities, the paper evaluates minimum, maximum, and mean aggregation under a coarse severity ordering; weighted voting helped in some cases but hurt others.Random tie-breaking and discarding tied examples are also described as alternatives; weighted-voting experiments are excluded from reported results.Source: §4.1.2 · pp. 8–9
4.2 Analyzing Hate Speech Category vs Strength
- Annotators assign a single 0–10 strength score per tweet, and scores span the full range for every class except class 0, revealing substantial variability.A tweet with two category labels contributes its strength score to both category distributions, shown for the six-class setting in Figure 2.Source: §4.2 · p. 9, Figure 2 · p. 9
- Class 1 has many strength scores of 0, reflecting tension between guideline-based discriminatory labels and annotators’ personal severity judgments.Whether discriminatory speech should be classified as hate speech was one of the most debated points among annotators.Source: §4.2 · p. 9
- Despite no universally accepted strict severity order, category means increase from class 0 to 5; mapped labels correlate with strength at Spearman’s 0.8174, Pearson’s 0.7147, and Kendall’s tau 0.6784.These observations motivate treating the categories as ordinal for aggregation.Source: §4.2 · p. 9
4.3 Category Consolidation
- For four classes, categories 2–3 and 4–5 are grouped before majority counts; for two classes, all hateful categories 1–5 merge while class 0 remains separate.The four-class grouping is motivated by similar strength means and yields more separated class means, shown in Figure 3.Source: §4.3 · p. 10, Figure 3 · p. 10
4.4 Model Architecture and Training
- BERTurk is fine-tuned with a prediction layer; classification uses cross-entropy, while strength regression predicts mean 0–10 scores with mean squared error.Training uses 15 epochs, a 5% validation split, learning rate 5 × 10−6, batch size 16, and weight decay 0.01.Source: §4.4 · pp. 10–11
5 Experimental Results
The experiments compare disagreement-handling strategies across 2-, 4-, and 6-class tasks and Gold and Silver test sets. Max aggregation has the strongest average Silver-set performance across tasks, while results also show substantial differences between consensus-only and broader evaluation and training data.
5.1 Classification Experiments
- E1 uses unanimous single-category samples, E2 adds clear-majority samples, E3 uses all 8,820 samples, and E4 averages Min- and Max-trained classifiers.E1 retains only 2,580/2,606/2,786 samples for 6-, 4-, and 2-class tasks, respectively.Source: §5.1 · p. 11
5.2 Observations on Classification Results
- On Silver, Max aggregation averaged 72.59% F1 and 80.37% accuracy across tasks, versus Experiment 2’s 72.14% F1 and 80.63% accuracy.No aggregation strategy was consistently optimal at every classification granularity.Source: Table 1 · p. 12, §5.2 · p. 12
- Silver-set F1 and accuracy were over 20 percentage points lower than Gold results in E1 or E2; consensus-only training ranked best on Gold for 6- and 4-class tasks.It ranked second on the 2-class Gold task.Source: Table 1 · p. 12, §5.2 · p. 12
- For 2-class classification, all-data Min achieved 90.56% M-F1 and 93.94% accuracy on Gold, while E2–E4 consistently outperformed E1 on Silver.The findings indicate that fully agreed training samples alone were not always sufficient for the broader Silver setting.Source: Table 1 · p. 12, §5.2 · p. 13
- The E4 ensemble maintained 90.56% M-F1 on 2-class classification but did not surpass E3 strategies on 4- or 6-class tasks.It combines a Min-trained lenient classifier with a Max-trained sensitive classifier.Source: §5.2 · p. 13
5.3 Regression Model Using Hate Speech Strength
- Strength regression uses mean annotated scores on a 0–10 scale; consensus-only testing yielded RMSE 1.64 and 89.80% accuracy, while all-data testing dropped accuracy by over 10 points.The regression scores were also thresholded to evaluate binary classification.Source: §5.3 · pp. 13–14
- On the unfiltered test set, all-data strength regression achieved 79.42% accuracy versus 78.91% when trained on agreements; E3 with Max aggregation achieved 89.43%.The authors describe perceived strength as a proxy for detailed annotation guidelines.Source: §5.3 · p. 14
5.4 Comparison to Literature
- On an extended version of the same test set, the model achieved 83.63% macro F1 for 2-class classification and outperformed the compared literature results, which were not directly comparable.The comparison uses the paper’s best Silver-set result from E3 with Max aggregation.Source: §5.4 · pp. 14–15, Table 4 · p. 14
6 Conclusion and Future Work
The conclusion finds that consensus-only training performs worse on the larger Silver set, while Max aggregation performs best across the three classification tasks. It also presents perceived-strength labels as a practical alternative for large-scale collection and identifies annotator backgrounds as future work.
6 Conclusion and Future Work
- The work evaluates alternatives for aggregating labels in multi-annotator and multi-label annotation settings.Source: §6 · p. 15
- E1's strong Gold-set baseline performed significantly worse on Silver, while E3's Max aggregation performed best averaged across the three tasks, followed closely by E2.E1 trains exclusively on fully agreed-upon annotations; E2 uses consensus and clear-majority samples.Source: §6 · p. 15
- 79.42% versus 83.63% for 2-class classification: regression using perceived-strength labels scored lower than categorical labels but was presented as practical for large-scale collection.The authors suggest collecting these scores as coarse supervision during pretraining or before detailed categorization.Source: §6 · p. 15
- Future work could examine how annotators' gender, demographics, ethnicity, education, outlook, and biases affect hate-speech annotation quality.Source: §6 · p. 15