Source-linked AI summary
Deep Learning Models for Multilingual Hate Speech Detection
Sai Saketh Aluru, Binny Mathew, Punyajoy Saha, Animesh Mukherjee
TL;DR
Hate speech detection research is limited by the predominance of English datasets, motivating broader multilingual evaluation. The paper analyzes deep learning models across 16 datasets in 9 languages under low- and high-resource monolingual and multilingual settings. LASER with logistic regression is strongest in low-resource settings, while BERT-based models perform better with more data; zero-shot LASER + LR reaches an F-score of 0.6567 on Portuguese versus 0.6941 with full Portuguese training data.
Problem
Most hate speech detection datasets are English-only, while non-English datasets are relatively small, limiting multilingual coverage.
Method
The paper evaluates multilingual hate speech classifiers across 16 datasets in 9 languages using LASER, MUSE, BERT, and mBERT-based approaches.
Results
LASER + LR performs best in low-resource settings, BERT-based models perform better with more data, and zero-shot LASER + LR reaches an F-score of 0.6567 for Portuguese versus 0.6941 with full Portuguese training data.
Takeaways & Limitations
The resulting catalogue identifies effective models for languages according to available training data and is intended as a resource for future multilingual hate speech research.
Takeaways & Limitations
Error interpretations rely on translations, which may themselves contain errors.
Abstract
from arXiv · showhide
Hate speech detection is a challenging problem with most of the datasets available in only one language: English. In this paper, we conduct a large scale analysis of multilingual hate speech in 9 languages from 16 different sources. We observe that in low resource setting, simple models such as LASER embedding with logistic regression performs the best, while in high resource setting BERT based models perform better. In case of zero-shot classification, languages such as Italian and Portuguese achieve good results. Our proposed framework could be used as an efficient solution for low-resource languages. These models could also act as good baselines for future multilingual hate speech detection tasks. We have made our code and experimental settings public for other researchers at https://github.com/punyajoy/DE-LIMIT.
1 Introduction
Hate speech is a growing social concern, while detection research remains concentrated in English and limited in other languages. This paper addresses that gap through a large-scale analysis of multilingual hate speech detection across 16 datasets in 9 languages.
- Hate speech has been associated with severe incidents, outgroup prejudice, minority exclusion, and discriminatory public-resource distribution.
- Most hate speech detection datasets are available only in English, while non-English datasets are relatively small.
- The paper analyzes deep learning models on 16 datasets from 9 languages in monolingual and multilingual settings.
- In monolingual experiments, LASER embedding with logistic regression performs best in low-resource settings, whereas BERT-based models perform better with more data.
- The multilingual analysis finds that training with other languages is especially effective when little or no target-language training data is available.
2 Related Works
Hate speech detection research has progressed from simple lexical methods toward deep learning and graph embeddings, while multilingual research remains comparatively new.
- Early hate speech classifiers used dictionary lookup and bag-of-words methods.
- Larger datasets encouraged the use of deep learning and graph embedding techniques for hate speech detection.
- Multilingual hate speech datasets exist for languages including Arabic, French, Indonesian, Italian, Polish, Portuguese, and Spanish.
- Few studies had used these multilingual datasets to build multilingual hate speech classifiers.
3 Dataset description
The study assembles publicly available hate speech datasets across 9 languages and standardizes them for binary hate-speech-versus-normal classification. Dataset sources vary by language, platform, annotation scheme, and original label structure.
- The researchers identified 16 publicly available sources across 9 languages.
- Because source datasets mix offensive, abusive, profanity, insult, and related categories, the study retains only hate speech and normal labels.
- The English collection contains six publicly available datasets, reflecting the greater availability of English hate speech data.
- The datasets include Twitter, Facebook, and forum data, with examples including Stormfront, refugee-crisis tweets, and hate speech targeting immigrants and women.
- Several source datasets use multiclass or multilabel annotations, which the study filters to hate-speech and normal examples.
4 Experiments
The experiments compare multilingual embeddings and neural classifiers under consistent data splits and hyperparameter selection. The evaluated approaches include MUSE with CNN-GRU, translation with BERT, LASER with logistic regression, and mBERT.
- Each language uses a stratified 70%/10%/20% train/validation/test split, with macro F1-score as the performance metric.
- LASER produces sentence embeddings using max-pooling over a shared-vocabulary BiLSTM encoder, while MUSE provides multilingual word embeddings.
- BERT uses transformer encoder layers with self-attention to compute weighted token representations.
- The experiments evaluate MUSE + CNN-GRU, Translation + BERT, LASER + LR, and mBERT.
- Only 0.17% of datapoints contain more than 128 tokenized tokens, supporting the models’ 128-token input choice.
- Validation performance selects hyperparameters from learning rates of 2e−5, 3e−5, and 5e−5 and training durations of 1 to 5 epochs.
5 Results
Across monolingual and multilingual experiments, model effectiveness depends on target-language training data and resource level. LASER + LR is strongest with limited data, while BERT-based models generally lead when more data is available, with some strong zero-shot results.
- LASER + LR performs best in monolingual low-resource settings across all languages, while BERT-based models perform better with sufficient training data.At 256 datapoints, Translation + BERT and mBERT already approach LASER + LR for several languages.
- Translation + BERT is competitive for German, Polish, Portuguese, and Spanish, whereas MUSE + CNN-GRU performs worst in almost all monolingual cases.The authors describe Translation + BERT as an excellent compromise, while noting that improved translations could further improve it.
- Increasing training data generally improves classifier performance in both monolingual and multilingual settings.The multilingual analysis particularly observes this pattern for low-resource target languages including English, Indonesian, Italian, Polish, and Portuguese.
- In zero-shot evaluation, mBERT outperforms LASER + LR for Arabic, German, and French, while LASER + LR leads in the remaining six languages.For Portuguese, zero-shot LASER + LR reaches an F-score of 0.6567 versus 0.6941 with full Portuguese training data.
- The proposed model catalogue recommends LASER + LR for low-resource settings and BERT-based models for high-resource settings.The catalogue is presented as a reference for selecting models according to data availability in multilingual hate speech detection.
6 Discussion and Error Analysis
The discussion compares how LASER + LR and mBERT interpret hate speech and examines their characteristic errors. LASER + LR emphasizes hateful keywords, whereas mBERT uses broader context but can miss implicit derogatory meanings.
- Interpretability: LASER + LR focuses more on hateful keywords, while mBERT searches for contextual information around them.The comparison uses LIME-based word importance and examples from the two models.
- Interpretability: LASER + LR may be useful in highly toxic environments, whereas mBERT may help more on less toxic platforms where context matters.The paper contrasts GAB with Twitter as examples of these settings.
- Error Analysis: The error analysis relies on translated interpretations, and the translations themselves may contain errors.This caveat limits confidence in interpreting the analyzed mistakes.
- Error Analysis: The analysis categorizes errors as annotation dilemmas, confounding factors, hidden context, and over-dependence on abusive words.These categories distinguish annotation issues, irrelevant textual cues, missed context, and keyword-driven mistakes.
- Error Analysis: mBERT can misclassify posts containing abusive terms used non-hatefully and miss derogatory contextual meanings such as “parasite” toward refugees.The reported examples include abusive terms used in a prohibition and an implicit derogatory reference.
- Error Analysis: LASER + LR can rely on confounding tokens such as “USER,” miss implicit hatred toward LGBT people, and overpredict hate speech from words like “retarded.”These examples illustrate both irrelevant-feature dependence and insufficient contextual interpretation.
7 Conclusion
The paper presents a large-scale multilingual hate speech analysis across multiple datasets and languages, comparing models under varied resource and language settings. It concludes that LASER + LR is more effective in low-resource conditions, while BERT models are more effective in high-resource conditions, and offers a catalogue for future research.
- Conclusion: The study analyzes multilingual hate speech classifiers using 16 datasets from 9 languages across low- and high-resource, monolingual and multilingual settings.The experiments cover a variety of languages and conditions.
- Conclusion: LASER + LR is more effective in low-resource settings, whereas BERT models are more effective in high-resource settings.This is the paper’s overall model-performance conclusion.
- Conclusion: The paper suggests a catalogue intended to benefit future research in multilingual hate speech detection.The catalogue is presented as a resource for subsequent work.