Source-linked AI summary
Multilingual and Multi-Aspect Hate Speech Analysis
Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, Dit-Yan Yeung
TL;DR
Hate-speech research has largely focused on monolingual, binary classification despite multilingual exposure and the complexity of hateful language. The paper builds a trilingual, multi-aspect dataset and evaluates multilingual multitask learning, finding that it generally helps lower-frequency target-group labels while performance varies across tasks and languages.
Problem
Hate-speech analysis lacks multilingual, multi-aspect evidence, while binary labels do not capture directness, hostility type, targets, groups, or annotator reactions.
Method
The authors annotate around 13,000 English, French, and Arabic tweets across five tasks and compare multilingual and monolingual multitask and single-task models.
Results
Multilingual multitask learning helps tasks with fewer annotated examples per label, especially target-group classification, while performance decreases in most other multilingual multitask cases.
Takeaways & Limitations
The dataset supports studying cross-language and cross-task relationships in hate-speech detection and classification.
Takeaways & Limitations
Remaining code-switched and dialectally mixed tweets add noise, and future work is needed to exploit corpus comparability and multilinguality more fully.
Abstract
from arXiv · showhide
Current research on hate speech analysis is typically oriented towards monolingual and single classification tasks. In this paper, we present a new multilingual multi-aspect hate speech analysis dataset and use it to test the current state-of-the-art multilingual multitask learning approaches. We evaluate our dataset in various classification settings, then we discuss how to leverage our annotations in order to improve hate speech detection and classification in general.
1 Introduction
Existing hate-speech research is constrained by insufficient filtering, an English-centric and monolingual focus, and binary labels that miss important dimensions of hateful content. The paper introduces a multilingual, multi-aspect dataset and unified learning framework to address these gaps.
- Motivation: Online hate speech remains widespread despite platform policies, motivating improved detection across social media text.Reported online experiences include offensive name-calling, threats, harassment, and frequent assaults against women on Twitter.
- Research gap: Existing work centers on English and usually studies monolingual corpora, while multilingual tasks often retain separate classification settings.The paper highlights limited comparison of hate speech across languages and limited examination of cross-language correlations.
- Research gap: Binary classification misses directness, hostility type, discriminated attribute, target group, and annotator sentiment.The annotation schema also captures fear out of ignorance and reactions to hate speech.
- Contribution: The dataset annotates around 13,000 potentially derogatory tweets in English, French, and Arabic across five prediction tasks.Amazon Mechanical Turk provides the annotations, and the paper releases the dataset and code.
- Contribution: A unified multilingual multitask model aligns the three languages and compares multilingual and monolingual multitask and single-task settings.The approach uses an adapted multitask learning algorithm and Babylon cross-lingual embeddings.
2 Related Work
Prior hate-speech datasets face definitional ambiguity, sparse and noisy resources, target-detection bias, and limited multilingual comparability. This paper responds with a trilingual corpus covering multiple targets, hostility types, and annotator reactions.
- Existing challenges: Slurs do not reliably identify hate speech, because they can occur in non-aggressive speech while subtle metaphors or sarcasm can convey offensiveness.This ambiguity contributes to the lack of a human-annotated vocabulary explicitly revealing hate speech.
- Existing resources: Existing hate-speech corpora are sparse and noisy, and fine-grained annotation schemes remain uncommon.Prior datasets differ substantially in labels and sizes.
- Target annotation: Target detection can be biased toward identity words, causing false predictions; this work labels six target attributes across 16 named groups.The annotation covers attributes such as origin, gender, and religious affiliation, with groups including refugees and immigrants.
- Multilingual modeling: Multilingual embeddings are used to address multilinguality and code switching in hate-speech data.Related corpora include code-mixed Hindi-English and English data containing Hindi tokens.
- Contribution: The dataset is the first trilingual corpus of English, French, and Arabic tweets spanning varied targets and hostility types.The authors also describe it as the first work to examine annotators’ reactions to hate-speech comments.
3 Dataset
The paper constructs a multilingual, multi-aspect hate-speech dataset from English, French, and Arabic tweets, using separate prediction tasks for directness, hostility, targets, groups, and annotator sentiment. It documents annotation procedures, label definitions, language-specific distributions, and inter-annotator agreement.
- Annotation process: Each tweet was annotated by three annotators using language-aligned guidelines and labelsets for the three languages.Annotators were native speakers selected for reputation scores above 0.90, and additional fields supported label refinement.
- Task design: The dataset treats each annotated aspect as a separate classification task, enabling independent evaluation and analysis of task correlations.The aspects cover explicitness, hostility type, target attribute, target group, and annotator reaction.
- Data quality: Average Krippendorff inter-annotator agreement was 0.153 for English, 0.244 for French, and 0.202 for Arabic.The authors describe these values as comparable to existing complex annotations given the task complexity and number of labels.
- Label definitions: Hostility type is multilabel, with abusive, hateful, offensive, disrespectful, fearful, and normal categories; offensive is the most frequent label across languages.The categories reflect danger, hate intensity and tone, fear-based expression, and normal content.
- Label definitions: The sentiment task uses shock, sadness, disgust, anger, fear, confusion, and indifference from the negative and neutral scales of the hourglass model.The authors exclude self-conscious or moral emotions such as shame and guilt.
4 Experiments
The experiments evaluate five hate-speech classification tasks across single-task, multilingual, and multitask settings using traditional and deep-learning baselines. Multilingual models generally help target-group classification, while multitask multilingual learning yields limited overall improvement and can hurt directness classification.
- The study evaluates directness, hostility type, discriminating target attribute, target group, and annotator sentiment as five classification tasks.
- The comparison includes BOW logistic regression, one-layer biLSTM models, Sluice networks, and STSL, STML, and MTML configurations with multilingual embeddings.Babylon embeddings are used because they appear to outperform MUSE on this data.
- All methods use an 8:1:1 train-dev-test split, with micro-F1 and macro-F1 reported on the test set.Development data tunes thresholds for binary classification problems in multilabel settings.
- STSL performs best on directness classification because directness has two labels, whereas imbalance, multiclass, and multilabel annotations harm multitask performance.
- Deep-learning models achieve high English macro-F1 on the direct class, while STSL is comparable to or better than BOW classifiers on many other tasks.Macro-F1 averages F1 across individual labels.
- MTML provides little overall improvement, but pairwise training improves hostility-type F1 by 0.03, 0.05, and 0.01 in English, French, and Arabic, respectively, and Arabic target-group macro-F1 by 0.25.The authors attribute limited MTML gains potentially to class imbalance, multilabel tasks, and differences between task types; they propose exploiting task correlations with logic rules.
5 Conclusion
The paper presents a multilingual hate-speech dataset and finds that deep learning generally outperforms BOW models, while multilingual multitask learning helps labels with less annotated data.
- The dataset contains English, French, and Arabic hate-speech tweets annotated for multilingual and multitask learning.
- Deep learning models perform better than traditional BOW-based models in most multilabel classification tasks.
- Multilingual multitask learning helps tasks whose labels have less associated annotated data.
- Better-tuned multilingual and multitask models are expected to outperform existing state-of-the-art embeddings and algorithms on this data.The authors also identify transfer learning, multimodal information, additional unlabeled data, label transformation, and label information sharing as future directions.