Source-linked AI summary
Hate Speech detection in the Bengali language: A dataset and its baseline evaluation
Nauros Romim, Mosahed Ahmed, Hriteshwar Talukder, Md Saiful Islam
TL;DR
Bengali social-media hate-speech detection faces a shortage of datasets, alongside the challenge of diverse social-media language. This paper builds a categorized comment dataset and benchmarks models; SVM achieves the best reported accuracy and F-1 score.
Problem
Method
The paper builds a Bengali comment dataset across seven categories and benchmarks deep-learning models with Word2Vec, FastText, and BengFastText embeddings.
Source: §1 · p. 2, §4.2 · p. 8Results
SVM achieved the overall best result, with 87.5% accuracy and a 0.911 F-1 score.
Source: §4.5 · p. 10Takeaways & Limitations
The 30,000-comment dataset spans seven categories, while the experiments report overfitting and bias toward non-hate speech when training data are highly imbalanced.
Source: §5 · p. 11Existing Bengali embeddings draw on newspaper and blog text unlike social-media language, which includes misspellings, grammatical errors, meme language, and emoji.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Social media sites such as YouTube and Facebook have become an integral part of everyone's life and in the last few years, hate speech in the social media comment section has increased rapidly. Detection of hate speech on social media websites faces a variety of challenges including small imbalanced data sets, the findings of an appropriate model and also the choice of feature analysis method. further more, this problem is more severe for the Bengali speaking community due to the lack of gold standard labelled datasets. This paper presents a new dataset of 30,000 user comments tagged by crowd sourcing and varified by experts. All the comments are collected from YouTube and Facebook comment section and classified into seven categories: sports, entertainment, religion, politics, crime, celebrity and TikTok & meme. A total of 50 annotators annotated each comment three times and the majority vote was taken as the final annotation. Nevertheless, we have conducted base line experiments and several deep learning models along with extensive pre-trained Bengali word embedding such as Word2Vec, FastText and BengFastText on this dataset to facilitate future research opportunities. The experiment illustrated that although all deep learning models performed well, SVM achieved the best result with 87.5% accuracy. Our core contribution is to make this benchmark dataset available and accessible to facilitate further research in the field of in the field of Bengali hate speech detection.
1 Introduction
Social media hate speech is increasing, and its varied, nontraditional language makes automatic detection difficult. Bengali research faces an additional shortage of datasets and tools; this paper introduces a 30,000-comment dataset and evaluates baseline models.
1 Introduction
- Social media expression has accompanied a continuous rise in hate speech and offensive language.The paper also notes that social-media language differs from print and has diverse linguistic features, complicating automatic detection.Source: §1 · pp. 1–2
- Bengali hate-speech detection has few resources, while a shortage of datasets and tools complicates machine-learning development for this low-resource language.The paper links this resource gap to rising online abuse, especially abuse directed at females in Bangladesh.Source: §1 · p. 2
- The paper introduces 30,000 Bengali comments from YouTube and Facebook, spanning seven categories, and benchmarks deep learning models and Bengali word embeddings.The categories are sports, entertainment, crime, religion, politics, celebrity, and TikTok and meme.Source: §1 · p. 2
2 Literature review
Prior hate-speech research includes deep learning, semantic features, and transfer learning, but Bengali social media remains under-resourced. Earlier Bengali datasets were limited in size or usable labels, motivating this paper’s larger, more diverse dataset.
2 Literature review
- Related work spans deep learning,, semantic-feature extraction, fastText, annotator-bias studies, and transfer or multilingual models,.The cited examples include Arabic hate-speech comparisons and multilingual hate-speech identification using mBERT and XLM-RoBERTa.Source: §2 · p. 2
- Bengali hate-speech datasets were scarce: many contained about 5,000 examples, while a 10,000-example corpus had five classes and only its toxicity-labeled sentences were usable in experiments,,,.Another dataset contained 2,665 entries translated from an English hate-speech dataset.Source: §2 · pp. 2–3
- The paper presents what it describes as the first Bengali social-media hate-speech dataset combining large size with comments from seven categories.The authors frame this as addressing the lack of datasets that are both large and diverse.Source: §2 · p. 3
3 Dataset
The dataset gathers Bengali social-media comments across seven categories, applies context-sensitive hate-speech rules and repeated annotation, and reports category and text-length patterns. Its extraction choices and label distribution define the scope of the resulting resource.
3.1 Data extraction
- Comments were gathered from YouTube and a Facebook public page; Facebook’s Graph API restriction made YouTube the primary source.YouTube videos covered controversial topics from 2017–20, were selected with at least 700,000 views, and supplied comments across five categories; meme and TikTok searches supplied the seventh category.Source: §3.1 · p. 3
- The dataset contains 30,000 mostly Bengali comments, retaining mixed emoji, numbers, punctuation, and English but excluding wholly non-Bengali text and symbol-only comments.Comments were labeled by category and keyword, with keywords identifying the controversial event within a category.Source: §3.1 · pp. 3–4
3.2 Dataset annotation
- The annotation rules define hate speech around dehumanization of a person or community, including direct attacks and comments that support such attacks.Slang alone does not qualify, and comments requiring outside context were treated as not hate speech; examples distinguish dehumanizing language from context-dependent attacks.Source: §3.2 · p. 4
- Authors’ manual checks of 300 randomly sampled comments per category found the majority-vote annotations 91.05% correct.Fifty undergraduate annotators labeled every comment three times, with majority decisions used as final labels.Source: §3.2 · p. 5
3.3 Dataset statistics
- 30,000 comments include 10,000 hate-speech comments, and non-hate-speech comments dominate every category, especially celebrity and politics.The authors attribute the low celebrity count partly to attacks requiring outside context and the low politics count to fewer direct attacks.Source: §3.3 · p. 5
- Average hate-speech comments are shorter than non-hate-speech comments, while meme comments are shortest overall and celebrity comments longest.The authors connect the celebrity category’s longer average length to users posting opinions and analysis on an individual public figure’s page.Source: §3.3 · p. 6
4 Experiment
The experiment evaluates Bengali hate-speech models using a cleaned train/test split and several word-embedding and model combinations. SVM achieved the highest reported result, while the error analysis highlights the challenge of interpreting words in context.
4.1 Preprocess
- Baseline preprocessing removed emoji, punctuation, numbers, non-Bengali letters, and symbols, then split 30,000 comments into 24,000 training and 6,000 test comments.Source: §4.1 · p. 6
4.2 Word embedding
- Word2Vec and FastText embeddings were trained on the 30,000-comment dataset at 300 dimensions, while BengFastText was pretrained and lacked YouTube training data.Word2Vec used CBoW and FastText used skip-gram.Source: §4.2 · p. 8
4.3 Models
- The evaluated models included linear-kernel SVM, LSTM, and Bi-LSTM; the recurrent models used dropout rate 0.2 and the Adam optimizer.LSTM used 100 layers and Bi-LSTM used 64 layers.Source: §4.3 · p. 8
4.4 Experimental setting
- Models were trained on 80% of the dataset and tested on 20%, using five epochs and batch size 64; each embedding was paired with each deep-learning algorithm.Evaluation measured accuracy and F-1 score.Source: §4.4 · pp. 8–9
4.5 Result
- FastText had higher accuracy but lower F-1 than Word2Vec, while Word2Vec was more overfitted; FastText learns from word substrings.The authors suggest substring learning may explain FastText’s performance advantage.Source: §4.5 · p. 10
- 87.5% accuracy and 0.911 F-1 score made SVM the overall best model in the comparison.Source: Table 5 · p. 10, §4.5 · p. 10
4.6 Error analysis
- An example error was a non-hateful sentence predicted as hate because aggressive words were used without dehumanizing anyone, illustrating the difficulty of contextual interpretation in Bengali.Source: §4.6 · pp. 10–11
5 Conclusion
The conclusion presents a 30,000-comment Bengali dataset and reports that imbalanced training can leave hate speech poorly predicted despite high overall accuracy. It identifies social-media language and emoji as resource gaps and points to more advanced models and social-media embeddings as possible help.
5 Conclusion
- Imbalanced training can overfit models toward non-hate speech, so high overall accuracy may coexist with poor hate-speech prediction.The dataset contains 30,000 comments, including 10,000 hate-speech comments, across seven categories; hate-speech comments tend to be shorter in length and word count.Source: §5 · p. 11
- Bengali social-media text has spelling variants, grammatical errors, cryptic meme language, and emoji that differ from newspaper and blog text.The authors note that the same word can appear with multiple spellings, while emoji can convey an emotion on their own.Source: §5 · p. 11
- The authors report that no dataset or pretrained model currently classifies the sentiment of emoji used in social media.Source: §5 · p. 11
- Standard deep-learning models are insufficient for accurate detection; the authors suggest mBERT, XLM-RoBERTa, and embeddings trained on extensive, diverse social-media comments may help.The proposed models use large multilingual data and attention mechanisms.Source: §5 · p. 11