Source-linked AI summary
A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media
Marzieh Mozafari, Reza Farahbakhsh, Noel Crespi
TL;DR
Online hate-speech detection is challenged by limited labeled data and annotation biases. This paper applies BERT transfer learning with new fine-tuning strategies to Twitter hate-speech datasets, achieving strong comparative performance and identifying some dataset biases.
Problem
Online hate-speech detection lacks sufficient labeled data and is affected by existing annotation and collection biases.
Method
The paper fine-tunes the pretrained BERT language model using new strategies that examine representations from different transformer layers on labeled hate-speech datasets.
Results
The BERT fine-tuning approach outperforms previous work in precision, recall, and F1-score across the evaluated datasets and strategies.
Takeaways & Limitations
The model can capture some biases in dataset collection or annotation, providing a potential clue for future hate-speech dataset debiasing.
Takeaways & Limitations
The model misclassifies some implicit abuse and struggles with contextual semantics associated with words such as “daughters,” “women,” and “burka.”
Abstract
from arXiv · showhide
Generated hateful and toxic content by a portion of users in social media is a rising phenomenon that motivated researchers to dedicate substantial efforts to the challenging direction of hateful content identification. We not only need an efficient automatic hate speech detection model based on advanced machine learning and natural language processing, but also a sufficiently large amount of annotated data to train a model. The lack of a sufficient amount of labelled hate speech data, along with the existing biases, has been the main issue in this domain of research. To address these needs, in this study we introduce a novel transfer learning approach based on an existing pre-trained language model called BERT (Bidirectional Encoder Representations from Transformers). More specifically, we investigate the ability of BERT at capturing hateful context within social media content by using new fine-tuning methods based on transfer learning. To evaluate our proposed approach, we use two publicly available datasets that have been annotated for racism, sexism, hate, or offensive content on Twitter. The results show that our solution obtains considerable performance on these datasets in terms of precision and recall in comparison to existing approaches. Consequently, our model can capture some biases in data annotation and collection process and can potentially lead us to a more accurate model.
1 Introduction
Online hate speech is difficult to detect reliably because it can harm users and communities, while available labeled data remain limited and biased. The paper addresses this gap with BERT transfer learning and new fine-tuning strategies.
- Online platforms have increasingly enabled abusive language and hate-based activities, with online hate speech linked to serious social harm.
- Hate speech detection is difficult because both false positives and false negatives can seriously affect users and online-community safety.
- Traditional supervised approaches depend on carefully designed surface, sentiment, lexical, linguistic, knowledge-based, user, or platform features.
- The paper proposes BERT-based transfer learning with new supervised fine-tuning strategies for low-resource hate speech detection.
- Fine-tuning all BERT transformer layers to leverage syntactic and contextual information outperforms previous work on precision, recall, and F1-score.
2 Previous Works
Previous hate-speech systems rely on engineered textual or metadata features and increasingly use neural models, while research also identifies substantial annotation and sampling biases.
- Traditional hate-speech classifiers commonly rely on surface features such as bag-of-words and word- or character-level n-grams.
- Neural approaches have used paragraph embeddings, FastText, CNN, LSTM, and varied word or character embeddings for classification.
- Multi-task and unified models address differing annotation schemes, geographic influences, and multiple abusive-language categories using text and metadata.
- Prior studies report systematic racial and dataset-composition biases, including stronger effects in datasets containing more implicit abuse.
3 Methodology
The methodology analyzes BERT representations for hate-speech classification and compares four fine-tuning strategies that use different encoder outputs. BERT is fine-tuned on labeled datasets to adapt general pretrained representations to social-media content.
- BERT model: BERT is a bidirectional transformer encoder pretrained on English Wikipedia and BookCorpus, with BERTbase and BERTlarge variants.
- BERT model: BERTbase has 12 transformer layers, 12 self-attention heads, 110 million parameters, and 768-dimensional embeddings.
- BERT model: For classification, BERT processes token sequences up to length 512 and uses the final-layer [CLS] embedding as the whole-sequence representation.
- Fine-Tuning Strategies: The study implements four fine-tuning approaches because lower and higher BERT layers can capture different levels of general and task-specific information.
- Fine-Tuning Strategies: The baseline strategy feeds only the final [CLS] output into a classifier, while another adds two 768-unit hidden layers with Leaky ReLU activations.
- Fine-Tuning Strategies: Other strategies feed transformer outputs into a Bi-LSTM or concatenate outputs from all encoders for CNN-based classification.
4 Experiments and Results
The study evaluates BERT-based fine-tuning for hate-speech detection on Waseem and Davidson Twitter datasets, using stratified splits and comparisons with prior baselines. The best approach achieves higher weighted F1-scores than the baselines, while error analysis identifies confusion involving implicit abuse and dataset biases.
- Datasets: The experiments use Waseem and Davidson Twitter datasets, with classes covering racism, sexism, hate, offensive content, or neither.The Waseem datasets were merged, while Davidson tweets were sampled and crowdsourced-labeled.
- Implementation: The implementation normalizes social-media text and trains BERT with WordPiece tokenization, a maximum sequence length of 64, batch size 32, and three epochs.Training uses Adam with a 2e-5 learning rate and dropout probability 0.1.
- Experimental setup: The data are split into 80% training, 10% validation, and 10% test sets using stratified sampling by class.The authors retain the original class imbalance rather than applying oversampling or undersampling.
- Results: 81% and 91% weighted-average F1-scores are obtained by BERTbase with a linear classifier on the Waseem and Davidson datasets, respectively, exceeding previous baselines.The evaluation compares precision, recall, and weighted-average F1-score across fine-tuning strategies and literature baselines.
- Error analysis: The BERTbase + CNN model separates Waseem sexism from racism effectively, but errors mainly involve confusing hateful categories with neither.For Davidson, hate content is misclassified as offensive in 63% of cases, while annotation and collection biases also contribute to errors.
5 Conclusion
The study addresses hate-speech detection errors and limited generalization with BERT transfer learning and new fine-tuning strategies. Its approach outperforms previous work while revealing biases in dataset collection and annotation.
- Conflating hateful content with offensive or harmless language can cause automatic detection tools to flag user-generated content incorrectly.
- The proposed approach uses BERT transfer learning and new fine-tuning strategies to improve hate-speech detection performance and generalization to new datasets.
- The model outperforms previous work by using syntactical and contextual information from different BERT transformer-encoder layers with CNN-based fine-tuning.
- The results indicate that the model can detect biases in dataset collection and annotation processes.
- Identifying dataset biases may help future studies investigate contextual information from BERT layers alongside features associated with different bias types.