PaperlayerView exact revision on arXiv

Source-linked AI summary

A systematic review of Hate Speech automatic detection using Natural Language Processing

Md Saroar Jahan, Mourad Oussalah

arXiv:2106.00742v1cs.CL

TL;DR

Hate-speech detection remains challenging as social media enables rapid, anonymous dissemination of harmful content. This paper systematically reviews automatic textual hate-speech detection research, finding that BERT-based models ranked top among compared deep-learning architectures while dataset limitations hinder cross-study comparison.

  • Problem

    Automatic hate-speech detection remains an important challenge because social media facilitates the rapid and anonymous spread of aggressive and harmful content.

  • Method

    The paper conducts a PRISMA-guided systematic review of hate-speech detection literature, analyzing terminology, datasets, processing pipelines, methods, and deep-learning architectures.

  • Results

    BERT-based models ranked top among compared deep-learning models, while FastText, Word2Vec, and GloVe showed close performance and ELMO performed slightly better than them.

  • Takeaways & Limitations

    The review identifies dataset availability, size, labeling, and agreement problems that make results harder to compare across hate-speech studies.

  • Takeaways & Limitations

    Hate-speech research remains constrained by scarce non-English NLP resources, despite work covering 21 languages.

Abstract

from arXiv · show

With the multiplication of social media platforms, which offer anonymity, easy access and online community formation, and online debate, the issue of hate speech detection and tracking becomes a growing challenge to society, individual, policy-makers and researchers. Despite efforts for leveraging automatic techniques for automatic detection and monitoring, their performances are still far from satisfactory, which constantly calls for future research on the issue. This paper provides a systematic review of literature in this field, with a focus on natural language processing and deep learning technologies, highlighting the terminology, processing pipeline, core methods employed, with a focal point on deep learning architecture. From a methodological perspective, we adopt PRISMA guideline of systematic review of the last 10 years literature from ACM Digital Library and Google Scholar. In the sequel, existing surveys, limitations, and future research directions are extensively discussed.

1 Introduction

The paper addresses the growing challenge of automatic hate-speech detection in social media and reviews the field through a systematic NLP and deep-learning perspective.

  • The paper focuses on automatic textual hate-speech detection using machine learning and deep learning.
  • The review examines hate-speech branches, deep-learning technology, and dataset conditions through three research questions.It asks about conceptual scopes, practical deep-learning status, and the state of available datasets.
  • A PRISMA-based systematic approach categorizes existing technologies, literature, and dataset characteristics.The review critically analyzes theoretical aspects and practical resources, including datasets, methods, and projects.
  • The review is organized around definitions, prior reviews, collection methodology, literature findings, resources, discussion, and future directions.

2 Background

Hate speech is a complex, relationship-dependent phenomenon whose categories overlap, while automatic detection requires staged data preparation and feature extraction.

  • 2.1 What is hate speech?: Hate speech targets people or groups and commonly involves religion, gender, race, ethnicity, discrimination, or violence.Definitions also include criteria such as curses, disability, property damage, age, and serious disease.
  • 2.2 Other Related Concepts: Related concepts include cyberbullying, racism, sexism, abusive language, radicalization, and religious hate speech.
  • 2.2 Other Related Concepts: The category hierarchy treats hate speech as a parent concept from which cyberbullying, abusive language, sexism, racism, and radicalization derive.
  • 2.3 Relationship of HS Concepts and Example: Text classification examples show that insults, targets, negation, and target type do not individually determine hate-speech categories.For example, negation can block a category, while an insulting statement about asylum seekers may be hate speech but not cyberbullying.
  • 2.4 Generic Pipeline of Automatic HS Detection: The generic detection pipeline begins with dataset collection and preparation, then converts text into structured features using methods such as TF-IDF, semantic, lexical, topic, sentiment, BOW, and word embeddings.Preparation may include tokenization, stopword removal, spelling correction, noise removal, stemming, and lemmatization.

3 List of Acronyms

This section introduces an acronym list for the machine-learning, deep-learning, and feature-set terminology used throughout the review.

  • The paper lists acronyms and their complete forms to clarify terminology used across its machine-learning, deep-learning, and feature-set discussions.
  • The supplied PRISMA figure concerns selection of previous review and survey documents rather than acronym definitions.

4 Related work

Prior reviews of automatic hate-speech detection were limited in number and coverage, motivating an updated PRISMA-based review that tracks methods, datasets, performance, languages, and resources.

  • 4 Related work: Only seven eligible review papers were identified from 2100 records, and just two used a systematic-review approach.The screening excluded non-review, duplicate, insufficiently detailed, and unrelated articles.
  • 4 Related work: Earlier reviews differed in scope, covering topics such as feature engineering, datasets, Twitter cyberbullying, multilingual work, or narrow textual collections.Several reviews were narrative, focused on specific domains, or examined limited datasets.
  • 4 Related work: The Fortuna and Nunes review was systematic and relevant but required updating because its literature search ended in late 2017.It also reported that inconsistent dataset standards could bias performance comparisons.
  • 4 Related work: This study extends prior work with PRISMA, Google Scholar and ACM searches, systematic tracking of methods, evaluation performance, datasets, publication trends, multilingual collections, and open-source resources.The authors also identify challenges and an open research agenda.

5 Systematic literature review methodology for collecting hate speech documents

The review collects hate-speech literature through a PRISMA-based search of ACM Digital Library and Google Scholar, using broad category, model, and language keywords before staged screening.

  • 5 Systematic literature review methodology for collecting hate speech documents: The study adopted the PRISMA framework to structure keyword selection, source searches, filtering, and inclusion decisions.The methodology explicitly highlights the search and screening process.
  • 5.1 Keyword selection: The search criteria covered hate speech, sexism, racism, cyberbullying, abusive language, and offensive language, alongside model terms such as CNN, LSTM, RNN, and BERT.The authors also added 20 widely spoken languages to retrieve multilingual studies.
  • 5 Systematic literature review methodology for collecting hate speech documents: The search used ACM Digital Library and Google Scholar to collect computer-science and engineering records and support automated duplication and trend analysis.Metadata were exported for further analysis, including titles, abstracts, authors, venues, publication years, and full-text links.
  • 5 Systematic literature review methodology for collecting hate speech documents: 44,030 documents were initially collected, 33,670 remained after automatic duplicate removal and initial scrutiny, and 1,329 underwent full-text review.Two independent reviewers manually screened the remaining papers and resolved disagreements by consensus.
  • 5 Systematic literature review methodology for collecting hate speech documents: The final review included 463 articles, of which 96 used deep-learning methods.These counts define the corpus used for the systematic analysis.

6 Systematic review results

The review maps 463 hate-speech detection studies, showing rapid growth in deep-learning research, concentration in English and social-media data, and diverse modeling practices. Across studies, deep learning often outperformed traditional classifiers, but comparisons among architectures and embeddings remained inconsistent or limited.

  • 6.1 Number of publications per year: 463 papers were identified from 2000–2021, including 96 deep-learning studies published from 2017–2021 as deep-learning research grew rapidly.The collection stopped on March 18, 2021, explaining the relatively small 2021 count.
  • 6.3.1 Statistical trends of results: English datasets accounted for 51% of studies, while Arabic represented 13%, Turkish 6%, Danish 5%, and several other languages each represented smaller shares.Chinese was comparatively underrepresented despite being the world’s second-largest speaking language, which the review associates with its absence from cited multilingual competitions.
  • 6.3.1 Statistical trends of results: Twitter supplied 47% of datasets, followed by Facebook at 12%, YouTube at 9%, and Wikipedia at 5%.The review links this concentration to social media’s accessibility, rapid dissemination, and societal impact.
  • 6.3.1 Statistical trends of results: Supervised methods dominated with 73% of records, while the review found no substantial evidence that supervised, semi-supervised, or unsupervised approaches consistently performed best.The reported suitability of each category depends on data context, including training-sample availability and quality.
  • 6.3.1 Statistical trends of results: SVM covered 29% of records, deep-learning models about 22%, and LR and NB 20% and 14%, respectively; TF-IDF covered 29% of features while word embeddings covered 33%.BERT was the most prevalent deep-learning model at 33% of deep-learning records, followed by LSTM at 20% and CNN at 12%.
  • 6.3.2 Overview of Deep-learning records: Deep-learning models outperformed NB, LR, RF, and SVM in most reviewed studies, with one comparison reporting CNN or LSTM gains of 13–20%, while architecture comparisons remained mixed.CNN, LSTM, BiLSTM, and GRU results varied by study; combined models sometimes outperformed single models, and embedding comparisons were sparse or showed little difference.

7 Resources for hate speech detection

The review identifies datasets and open-source resources for hate speech detection, while highlighting substantial variation in languages, platforms, annotation schemes, dataset sizes, class ratios, and resource maintenance.

  • Datasets language and platform: 69 datasets span 21 languages, with English dominant and Twitter the most common collection platform.English accounts for 26 datasets, while Twitter supplies 45% of the datasets.
  • Annotation classes: Annotation schemes range from binary labels to fixed multi-class and multi-level schemes covering categories, severity, and target groups.The reviewed datasets include diverse labels such as hate, offensiveness, race, gender, sexism, misogyny, toxicity, and target group.
  • Dataset size and class ratio: 55% of datasets are very small, and none is balanced, creating challenges for acquiring large-scale labeled data.41% contain 0–5k posts and 14% contain 5–10k sentences.
  • Dataset size and class ratio: 37% of datasets contain less than 20% offensive content, whereas 63% contain more than 20%.The review treats the latter proportion as more suitable for training purposes.
  • Dataset citations and quality: Dataset citation counts do not necessarily indicate dataset quality, which also depends on annotation criteria, label definitions, homogeneity, size, and class ratios.The review reports insufficient experimental evidence for comparing these quality indicators across hate-speech domains.

8 Research challenges and opportunities

The review identifies challenges in dataset quality, annotation, model comparison, practical evaluation, language change, and multilingual resource coverage, and points to corresponding research opportunities.

  • Language and system barriers: Rapidly changing language, evasive users, images containing hate text, abbreviations, and code-mixing complicate continued automatic detection.These shifts create a need for continuity in hate-speech dataset development.
  • Dataset standards: No commonly accepted ideal dataset exists because datasets differ in labels, size, hate-content proportion, and annotation practices.The review notes 47 annotation labels across 69 datasets and widespread small datasets with little hate content.
  • Label definitions: Clear label definitions are needed to distinguish hate speech from other offensive language while supporting fine-grained, multi-label, or hierarchical categories.The review highlights ambiguity in some racism and sexism labels.
  • Annotation quality: Annotation quality is constrained by loose grammatical and cross-sentence boundaries, while only about two-thirds of datasets report agreement, guidelines, definitions, and examples.The review recommends continuously updating datasets as knowledge changes.
  • Model comparison: Comparative studies across preprocessing, feature engineering, training, and evaluation remain scarce despite 24 identified deep-learning hybridization schemes.The review calls for comprehensive comparisons of alternative approaches.
  • Multilingual research: English dominates available resources, while other languages remain comparatively underdeveloped despite work spanning 21 languages.The review identifies an opportunity to develop stronger NLP tools for non-English languages.

9 Conclusion

This systematic review traces hate-speech detection from traditional machine-learning methods toward deep-learning architectures and analyzes the datasets supporting the field. Its findings identify BERT as a leading architecture while exposing persistent dataset and resource constraints.

  • Scope and method: The review analyzed 463 articles selected under PRISMA from Google Scholar and the ACM Digital Library, covering literature published between 2000 and 2021.It examines terminology, methods, deep-learning technology, datasets, and open-source resources.
  • Deep-learning findings: BERT-based models were reported to outperform ELMO, CNN, and RNN models and ranked highest among architectures compared in several shared-task evaluations.The review also reports CNN+LSTM and CNN+GRU outperforming single LSTM and CNN applications.
  • Dataset constraints: The review found that retrieving older Twitter datasets is often incomplete because tweets may be removed, limiting data availability and cross-study comparison.This issue slows research progress and makes comparing results more difficult.
Loading 2106.00742v1…