Source-linked AI summary
Deep Learning for Hate Speech Detection in Tweets
Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, Vasudeva Varma
TL;DR
Twitter hate-speech detection requires classifying tweets as racist, sexist, or neither despite complex language, and matters for analyzing and filtering online content. The paper tests deep learning architectures that learn semantic, task-specific embeddings, finding that they outperform state-of-the-art char/word n-gram methods by approximately 18 F1 points.
Problem
Twitter hate-speech detection classifies tweets as racist, sexist, or neither, but complex language makes automated classification challenging and useful for analysis and filtering.
Method
The paper compares multiple classifiers using baseline text representations and task-specific embeddings learned with FastText, CNNs, and LSTMs.
Results
Approximately 18 F1 points: deep learning methods outperform state-of-the-art char/word n-gram methods on 16K annotated tweets.
Takeaways & Limitations
Neural-network-learned embeddings combined with gradient boosted decision trees achieve the best reported accuracy values.
Abstract
from arXiv · showhide
Hate speech detection on Twitter is critical for applications like controversial event extraction, building AI chatterbots, content recommendation, and sentiment analysis. We define this task as being able to classify a tweet as racist, sexist or neither. The complexity of the natural language constructs makes this task very challenging. We perform extensive experiments with multiple deep learning architectures to learn semantic word embeddings to handle this complexity. Our experiments on a benchmark dataset of 16K annotated tweets show that such deep learning methods outperform state-of-the-art char/word n-gram methods by ~18 F1 points.
1. INTRODUCTION
The paper frames Twitter hate-speech detection as a difficult, socially useful classification problem and investigates deep learning methods as an alternative to earlier feature-based approaches.
- Twitter hate-speech detection can support sentiment analysis, content recommendation, and AI chatterbot training while discouraging wrongful activities.
- The task classifies tweets as racist, sexist, or neither, but varied forms of hatred, targets, and expressions make it challenging.
- Earlier work primarily used manual feature extraction or representation learning followed by a linear classifier.
- The paper is presented as the first study to experiment with deep learning architectures for hate-speech detection.
- The experiments compare task-specific embeddings from FastText, CNNs, and LSTMs with char n-gram, TF-IDF, and BoWV baselines across several classifiers.
- The reported methods outperform state-of-the-art methods by approximately 18 F1 points.
2. PROPOSED APPROACH
The approach compares conventional text representations with neural architectures that learn task-specific tweet embeddings, then uses those embeddings with classifiers.
- Baseline Methods: The paper evaluates char n-grams, TF-IDF, and averaged GloVe word embeddings as baseline representations.
- Proposed Methods: CNNs, LSTMs, and FastText are used to learn representations for hate-speech detection.
- Proposed Methods: The networks are fine-tuned with labeled data and learn word embeddings tuned toward racist, sexist, or neither labels.
- Proposed Methods: Learned embeddings are also evaluated as features for SVM and GBDT classifiers.
3. EXPERIMENTS
Experiments use 16K annotated tweets and compare baseline and neural methods under cross-validation. Neural approaches outperform baselines, with learned embeddings plus GBDTs producing the strongest results and task-specific semantic effects.
- Dataset and Experimental Settings: The dataset contains 16K annotated tweets: 3,383 sexist, 1,972 racist, and the remainder neither sexist nor racist.
- Dataset and Experimental Settings: Evaluation uses 10-fold cross-validation with weighted macro precision, recall, and F1-scores.
- Results and Analysis: Table 1 compares baseline methods, neural-network-only methods, and GBDTs using averaged neural embeddings as features.
- Results and Analysis: Proposed neural methods significantly outperform baselines; CNN exceeds LSTM, LSTM exceeds FastText, and embedding features with GBDTs outperform neural-network-only methods.
- Results and Analysis: The best method is LSTM + Random Embedding + GBDT, while combining CNN, LSTM, or FastText embeddings as GBDT features does not improve results.
- Results and Analysis: DNN-learned embeddings reveal racist or sexist associations for selected words that are not visible in the original GloVe embeddings.
4. CONCLUSIONS
The paper concludes that deep neural architectures significantly outperform existing hate-speech detection methods, with neural embeddings combined with GBDTs achieving the best accuracy values.
- Deep neural network architectures significantly outperform existing methods for hate-speech detection.
- Embeddings learned by deep neural networks combined with gradient boosted decision trees produce the best accuracy values.