Source-linked AI summary
Overview of the HASOC track at FIRE 2020: Hate Speech and Offensive Content Identification in Indo-European Languages
Thomas Mandla, Sandip Modha, Gautam Kishore Shahi, Amit Kumar Jaiswal, Durgesh Nandini, Daksh Patel, Prasenjit Majumder, Johannes Schäfer
TL;DR
Social-media growth has increased the need for multilingual hate-speech detection beyond the predominantly English research focus. HASOC provides a shared evaluation platform for English, German, and Hindi, using two classification tasks and reporting challenging results across languages.
Problem
The spread of hate speech across widely used social media creates a need for multilingual detection research beyond the strong focus on English.
Method
HASOC develops and evaluates two hate-speech classification tasks across English, German, and Hindi using Twitter-archive data and a changed sampling approach.
Results
The fine-grained task remained difficult, with best F1 scores of 0.29 for German, 0.33 for Hindi, and 0.26 for English; overall results were below 0.55.
Takeaways & Limitations
Transformer models and variants such as ALBERT achieved the best results, while small differences across languages suggest potential for pre-trained deep learning models in lower-resource languages.
Takeaways & Limitations
Keyword-focused sampling can bias datasets, while the new sampling process remained subjective and produced a large portion of profane content.
Abstract
from arXiv · showhide
With the growth of social media, the spread of hate speech is also increasing rapidly. Social media are widely used in many countries. Also Hate Speech is spreading in these countries. This brings a need for multilingual Hate Speech detection algorithms. Much research in this area is dedicated to English at the moment. The HASOC track intends to provide a platform to develop and optimize Hate Speech detection algorithms for Hindi, German and English. The dataset is collected from a Twitter archive and pre-classified by a machine learning system. HASOC has two sub-task for all three languages: task A is a binary classification problem (Hate and Not Offensive) while task B is a fine-grained classification problem for three classes (HATE) Hate speech, OFFENSIVE and PROFANITY. Overall, 252 runs were submitted by 40 teams. The performance of the best classification algorithms for task A are F1 measures of 0.51, 0.53 and 0.52 for English, Hindi, and German, respectively. For task B, the best classification algorithms achieved F1 measures of 0.26, 0.33 and 0.29 for English, Hindi, and German, respectively. This article presents the tasks and the data development as well as the results. The best performing algorithms were mainly variants of the transformer architecture BERT. However, also other systems were applied with good success
1. Introduction: Hate Speech and Its Identification
HASOC addresses the growing need to identify hate speech in large, multilingual social-media collections. Its second edition keeps the same tasks and languages as HASOC 2019 while changing the data-creation approach for comparison.
- Social-media platforms contain increasing amounts of problematic content, prompting research and policy efforts around hate-speech identification.
- Open resources are important for assessing current technology and the remaining challenges of hate-speech identification.
- Multilingual datasets and benchmarks have been developed for languages including Greek, Portuguese, Danish, Mexican Spanish, and Turkish.
- HASOC 2020 changes its data-creation approach while retaining the two tasks and three languages from HASOC 2019 for comparability.
2. HASOC Task Description
HASOC 2020 evaluates hate-speech and offensive-language identification in English, German, and Hindi through coarse binary and fine-grained classification tasks. The fine-grained task separates hateful, offensive, and profane content, with examples provided from the dataset.
- HASOC offers both tasks in English, German, and Hindi, including Hindi and German as comparatively lower-resource languages and English as a comparison testbed.
- Sub-task A: Sub-task A is binary classification of tweets into Hate and Offensive (HOF) or Non-Hate and Offensive (NOT).
- Sub-task A: NOT denotes posts without hate speech, profane content, or offensive content, whereas HOF denotes posts containing hate, offensive, or profane content.
- Sub-task B: Sub-task B further classifies hate-speech and offensive posts from sub-task A into three fine-grained categories.
- Sub-task B: HATE covers group-directed hateful content, OFFN covers degrading, dehumanizing, insulting, or threatening content toward individuals, and PRFN covers profanity without hate or offensive content.
- Sub-task B: Examples from all classes in the final dataset are presented in Table 1.
3. Dataset Description
HASOC 2020 developed multilingual hate-speech datasets for English, German, and Hindi using archive-based sampling designed to reduce keyword-selection bias. Tweets were manually annotated through repeated judgments, reliability-based conflict resolution, and language-specific procedures.
- Sampling rationale: Keyword-focused sampling can bias hate-speech datasets, while fully random sampling may produce too few hate-speech examples for practical annotation.Prior benchmark systems trained on biased datasets substantially underperformed on datasets sampled with different keywords.
- Sampling rationale: HASOC 2020 aimed to minimize training-data bias by using a sampling process that relied on less input than keyword-driven collection.The dataset was developed from Twitter archives rather than only handcrafted hate-speech terms.
- Archive construction: Tweets were extracted from May 2019 archives for English, German, and Hindi, with language metadata and scripts used for filtering.German used the full May archive, while English and Hindi used the 1st, 10th, and 19th of May.
- Candidate selection: Potentially hateful tweets were identified with TF-IDF SVM classifiers trained on language-relevant prior corpora, then supplemented with randomly selected non-hateful tweets.The English and Hindi procedures used prior OLID, HASOC, and TRAC resources; German used GERMeval and HASOC data.
- Annotation and reliability: Each tweet received two annotations, with conflicts assigned to a third annotator when possible and unresolved cases handled using reliability-based judgments or re-annotation.For Hindi, conflicted data was re-annotated; for English and German, the more reliable annotator’s vote was used in two-annotator conflicts.
- Annotation and reliability: The annotation process remained highly subjective, and the new sampling method produced a large portion of profane content.Annotators discussed guidelines and borderline cases, but some disagreements remained unresolved.
4. Participation and Evaluation
HASOC 2020 received submissions from 40 teams across multilingual hate-speech tasks, with evaluation conducted through Codalab. Teams used a macro F1 measure to rank classifications with multiple, unevenly sized classes.
- Participation: 40 teams submitted results after 116 participants registered for the multilingual HASOC tasks.Teams could participate in either of the two sub-tasks for English, German, and Hindi.
- Task structure: The tasks covered English, German, and Hindi, with two sub-tasks offered for each language.
- Evaluation: Macro F1 was used to rank submissions because each language task contained multiple classes with non-uniform sample counts.The measure was selected as an item-weighted evaluation measure.
- Evaluation: Participants accessed training data online, monitored development-set performance, and could observe their relative leaderboard position.The submission process was handled on Codalab.
- Participation: More than 252 experiments were submitted overall.
5. Results for Tasks
HASOC 2020 results show that hate-speech identification and fine-grained offensive-content classification remained difficult across Hindi, German, and English. The strongest systems used varied architectures, with transformer models prominent but not universally best.
- Overall results: No F1 score exceeded 0.55, and HASOC 2020 test-set results were lower than those for HASOC 2019.The results indicate that both identifying and further classifying offensive language remained challenging.
- Hindi: Hindi Task A achieved an F1 score slightly above 0.53, while the ten best submissions received very similar scores.The best Hindi Task A system used a one-layer BiLSTM with fastText embeddings; another high-ranked system used TF-IDF and SVM.
- Hindi: Hindi Task B reached only a score above 0.33, with the first system substantially outperforming the following submissions and ranks 2–10 scoring similarly.The best-ranked system fine-tuned BERT, while other high-ranked systems used BiLSTM, fastText, BERT, GPT-2, and CNN-based approaches.
- German: German results were similar to Hindi, with a low best F1 score and closely grouped submissions across both tasks.Fine-tuned BERT, DistilBERT, and RoBERTa achieved the best German Task A performance, while Task B results were also very close.
- English: English Task A achieved a best performance of 0.52, while performance differences among the top 30 teams were extremely small.The best English Task A result used an LSTM with GloVe embeddings, although many submissions used BERT or RoBERTa.
- English: English Task B achieved an F1 value of 0.26, with the top three systems using BERT or related variants.The best result used fine-tuned BERT; other systems used ALBERT, BERT, and DPCNN.
6. Discussion and Interpretation
Across the three languages, the strongest teams were closely grouped despite using heterogeneous approaches. Fine-grained classification was especially challenging, with best Task B scores of 0.26, 0.33, and 0.29 for English, Hindi, and German.
- Cross-language interpretation: Top teams were closely grouped, so no particular technology showed a clear advantage despite varied approaches.Deep learning models were common, especially transformer architectures and BERT variants such as ALBERT.
- Fine-grained classification: Task B best systems reached 0.29 for German, 0.33 for Hindi, and 0.26 for English, making fine-grained classification a major challenge.
7. Conclusion and Outlook
HASOC 2020 confirms that hate-speech identification remains difficult, with lower performance than HASOC 2019 and only small differences across languages. The paper points toward multimodal evaluation, misinformation analysis, evaluation reliability, failure analysis, and response strategies as future directions.
- Conclusion: HASOC 2020 test-set performance was considerably lower than HASOC 2019, likely because a different sampling method produced more profane content.The paper states that this sampling method was closer to realistic platform procedures.
- Conclusion: Pre-trained transformer models and variants such as ALBERT achieved the best results, while performance differences across the three languages were small.The authors interpret this as indicating potential for pre-trained deep learning models in languages with fewer traditional resources.
- Outlook: Future evaluations may need to supplement language analysis with visual-material analysis because offensive intent can depend on both text and images.The outlook also identifies misinformation, evaluation reliability, failure analysis, and responses such as deletion, labeling, or counter-speech as research areas.