Source-linked AI summary
Detection of Hate Speech using BERT and Hate Speech Word Embedding with Deep Model
Hind Saleh, Areej Alhothali, Kawthar Moria
TL;DR
Online hate speech detection is difficult because hateful communication uses abbreviations, intentional misspellings, and coded words. This paper evaluates domain-specific embeddings with a BiLSTM and BERT for binary hate-speech classification, finding improvements for both approaches.
Problem
Online hate speech threatens communities, while abbreviations, intentional misspellings, and coded words make detection challenging.
Method
The paper develops unsupervised hate-speech word embeddings for a BiLSTM model and evaluates BERT for binary hate-speech classification.
Results
The BiLSTM with domain-specific embeddings improves performance by 5 to 6 points over available datasets' state-of-the-art techniques, while BERT improves by about 2 points over the domain-specific embedding model.
Takeaways & Limitations
BERT provides an appropriate feature-extraction and classification procedure, while domain-specific embeddings better capture hate terms, abbreviations, and intentional misspellings.
Takeaways & Limitations
The study detects English text only and treats hate speech, including offensive language, as a binary hate-or-not-hate classification task.
Abstract
from arXiv · showhide
The enormous amount of data being generated on the web and social media has increased the demand for detecting online hate speech. Detecting hate speech will reduce their negative impact and influence on others. A lot of effort in the Natural Language Processing (NLP) domain aimed to detect hate speech in general or detect specific hate speech such as religion, race, gender, or sexual orientation. Hate communities tend to use abbreviations, intentional spelling mistakes, and coded words in their communication to evade detection, adding more challenges to hate speech detection tasks. Thus, word representation will play an increasingly pivotal role in detecting hate speech. This paper investigates the feasibility of leveraging domain-specific word embedding in Bidirectional LSTM based deep model to automatically detect/classify hate speech. Furthermore, we investigate the use of the transfer learning language model (BERT) on hate speech problem as a binary classification task. The experiments showed that domainspecific word embedding with the Bidirectional LSTM based deep model achieved a 93% f1-score while BERT achieved up to 96% f1-score on a combined balanced dataset from available hate speech datasets.
1 Introduction
Online hate speech is widespread, harmful, and difficult to detect because communities use coded language, abbreviations, and intentional misspellings. This study compares domain-specific embeddings with BiLSTM and BERT for binary hate-speech classification.
- Hate speech detection matters because hateful content can instill hatred and provoke criminal activity or violence.
- Coded words, abbreviations, and intentional spelling mistakes make hate speech difficult to detect automatically.Users may employ ordinary words to express hate implicitly and evade detection.
- The paper evaluates domain-specific embeddings with a BiLSTM and BERT as approaches to binary hate-speech classification.
- The study develops an unsupervised domain-specific embedding model for terminology, acronyms, and purposefully misspelled hate words.
- Domain-agnostic embeddings performed about 1% better than domain-specific embeddings despite being trained on much larger corpora.
2 Background
The background reviews word embeddings, BiLSTM sequence modeling, and BERT contextual representations as foundations for hate-speech detection. It distinguishes static context-free embeddings from bidirectional contextual processing.
- 2.1 Hate Speech Detection: Prior hate-speech research has addressed general detection and specific categories including anti-religious, jihadist, sexist, and racist hate speech.
- 2.2 Word Embedding: Word embeddings represent words as dense numerical vectors derived from context and usable for estimating word similarity.
- 2.2.1 Word2Vec: Word2Vec learns vectors from surrounding context using CBOW or Skip-Gram training objectives.
- 2.2.2 GloVe: GloVe learns distributional vectors from word co-occurrence statistics using a count-based process.Its matrix construction requires substantial time and space.
- 2.3 Bidirectional Long Short-Term Memory (BiLSTM): BiLSTM processes sequences in both directions and combines the resulting representations to capture surrounding context.It uses two LSTMs, one left-to-right and one right-to-left.
- 2.4 BERT Pre-trained Language Model: BERT creates bidirectional contextual word representations, whereas Word2Vec and GloVe provide static context-free representations.Word2Vec and GloVe map a word to one embedding vector even when its meaning varies by context.
3 Review of Literature
The literature covers embedding-based and deep-learning approaches to hate-speech detection, followed by this study’s two methodological approaches. The study preserves hate-related misspellings during preprocessing and compares domain-specific embeddings with BERT.
- Related Work: BERT had produced strong results in earlier NLP benchmarks and was also applied to offensive-language detection with an 83.88% f1-score.
- Methodology: The study uses domain-specific embeddings with a BiLSTM classifier and fine-tuned BERT for hate-speech classification.BERT encodes input text and adds a neural-network classification layer.
- Approach 1: The first approach uses domain-specific embeddings to represent hate terms and abbreviations and a BiLSTM to preserve dependencies from both directions.
- Data Collection: 1,048,563 sentences were collected from existing hate-speech datasets and additional Twitter data for embedding extraction.Twitter data included hate keywords, explicitly hateful accounts, and hate hashtags.
- Preprocessing: Domain-specific preprocessing retains intentional misspellings and avoids stemming so hate-related code words remain represented.The data are lowercased, negations are normalized, and stop words and punctuation are removed.
5 Scope of the study
The study targets English hate speech across categories and treats detection as a binary distinction between hate and non-hate content.
- The study covers English hate speech across race, sex, gender, and other types, including offensive language, as two classes: hate and non-hate.
6 Experiment and Results
The experiments evaluate hate-speech detection on three available datasets using embedding-based BiLSTM and BERT sequence classification. The deep model outperforms the LR classifier, while BERT achieves the strongest overall performance among the evaluated approaches.
- 6.1 Datasets: The study tests both approaches on three available datasets and combines dataset labels into binary hate and non-hate classes for comparison.The combined dataset includes collapsed labels such as racism and sexism as hate and neither as non-hate.
- 6 Experiment and Results: Performance is reported with weighted precision, recall, AUC, and f1-score, alongside class-specific f1-scores and confusion matrices.These measures address class imbalance and distinguish false positives, false negatives, true positives, and true negatives.
- 6.2 HSW2V and BiLSTM based Deep Model: The BiLSTM deep model surpasses the LR classifier when evaluated with embedding features.The comparison includes LR from Gupta and Waseem (2017) and the proposed BiLSTM-based deep model.
- 6.2 HSW2V and BiLSTM based Deep Model: HSW2V outperforms Google Word2Vec and GloVe using the same BiLSTM deep model despite being trained on a much smaller corpus.The HSW2V corpus contains 1M entries, whereas the other domain-agnostic models use at least 2B.
- 6.3 BERT Language Model: BERT overcomes the embedding models with the proposed deep model in overall performance.BERT was implemented and fine-tuned for sequence classification, and its testing results are summarized in Table 4.
7 Discussion
The discussion compares domain-specific hate-speech embeddings with domain-agnostic embeddings, BERT, and classifiers, while examining how models handle misspellings and interpret predictions. HSW2V captures hate-domain vocabulary that BERT may miss, whereas BERT achieves stronger overall performance.
- Embedding and classifier comparisons: HSW2V was competitive with domain-agnostic embeddings despite training on a much smaller corpus, and the BiLSTM model improved performance over logistic regression by at least 5 percent.The comparison used the same BiLSTM classifier for embeddings and contrasted it with a prior logistic-regression experiment.
- Model interpretation: The confusion-matrix comparison evaluates HSW2V with BiLSTM against BERT across datasets using true positives, true negatives, false positives, and false negatives.The supplied discussion identifies the matrix as a graphical comparison of the two models’ performance but does not provide cell values.
- Embedding and classifier comparisons: BERT’s overall performance surpassed the proposed embedding-based deep model, while BERT Large produced similar results to BERT Base on larger datasets.The paper notes that BERT Large’s computational requirements limit its practical use despite its performance.
- Embedding and classifier comparisons: Domain-specific embeddings retrieve intentionally misspelled and commonly used hate terms that BERT fails to retrieve from its base vocabulary.The discussion attributes this difference to the hate-domain content represented in HSW2V versus BERT’s books-and-Wikipedia training data.
- Model interpretation: LIME explanations identified words contributing to both correct hate and non-hate predictions, with darker colors indicating greater contribution.For a true-positive example, “n*gga” contributed to the hate prediction; for a true-negative example, “announcement” contributed toward the non-hate prediction.
9 Conclusion and Future Work
The paper concludes that BERT provides an appropriate procedure for hate speech detection, while domain-specific embeddings better capture hate terms, abbreviations, and intentional misspellings; future work may extend detection to multiclass hate speech.
- BERT provides an appropriate feature extraction and classification procedure for hate speech detection.
- Domain-specific word embeddings better detect hate terms, abbreviations, and intentionally misspelled meanings than BERT.
- The study identifies multiclass hate speech detection as a direction for future work.