Source-linked AI summary
Challenges in Discriminating Profanity from Hate Speech
Shervin Malmasi, Marcos Zampieri
TL;DR
The paper addresses the underexamined problem of distinguishing general profanity from hate speech in social media. It evaluates supervised classifiers using lexical and clustering-based features on a three-class dataset, achieving 80% accuracy while finding the distinction difficult and annotation variability consequential.
Problem
Binary hate-speech systems often conflate targeted hate speech with general profanity, while separating non-antagonistic profanity from targeted insults remains underaddressed.
Method
The study uses supervised classification with n-grams, skip-grams, Brown-cluster word representations, single classifiers, ensembles, and stacked generalization.
Results
80% accuracy was achieved for the three-class classification task, while analysis found hate speech and profanity difficult to discriminate.
Takeaways & Limitations
Surface features may be insufficient for high-accuracy discrimination, motivating features that capture deeper textual understanding.
Takeaways & Limitations
Gold labels vary because annotators subjectively disagree about offensiveness, indicating a need for better annotation.
Abstract
from arXiv · showhide
In this study we approach the problem of distinguishing general profanity from hate speech in social media, something which has not been widely considered. Using a new dataset annotated specifically for this task, we employ supervised classification along with a set of features that includes n-grams, skip-grams and clustering-based word representations. We apply approaches based on single classifiers as well as more advanced ensemble classifiers and stacked generalization, achieving the best result of 80% accuracy for this 3-class classification task. Analysis of the results reveals that discriminating hate speech and profanity is not a simple task, which may require features that capture a deeper understanding of the text not always possible with surface n-grams. The variability of gold labels in the annotated data, due to differences in the subjective adjudications of the annotators, is also an issue. Other directions for future work are discussed.
1 Introduction
Existing social-media classification often conflates targeted hate speech with general profanity, although offensive language also appears in non-antagonistic communication. This study addresses that gap by evaluating classification methods for separating general profanity from targeted hate speech.
- Hate speech denigrates groups, whereas bullying intimidates individuals; both commonly contain profanity, which classifiers often exploit.
- Binary classification studies commonly distinguish target content from acceptable text, leaving systems potentially untested on non-antagonistic profanity.
- Filtering every item containing offensive language is undesirable because profanity is prevalent in informal online conversation.
- The study evaluates standard classifiers for distinguishing general profanity from targeted hate-speech attacks.
- The research introduces a language resource, conducts feature- and method-based classification experiments, and assesses performance and future issues.
2 Background
Prior hate-speech research overwhelmingly uses binary classification and often conflates profanity with hateful content. This study extends the task to three-way discrimination and investigates classifier ensembles while examining annotation quality.
- Earlier studies used sentiment analysis, topic modeling, user features, unigrams, embeddings, and temporal or syntactic features for bullying or hate-speech detection.
- Reported binary-classification results included a best F-score of 0.77 for cyber hate and 0.79 for abusive versus non-abusive language.
- Dataset construction and annotation are laborious, with disagreement arising because hate speech lacks consistently reliable definitions and guidelines.
- Most prior systems classify hate speech or abusive language against socially acceptable text rather than separately modeling non-antagonistic profanity.
- Classifier ensembles are presented as an innovative alternative to the single-classifier approaches common in previous work.
3 Data
The study uses a 14,509-tweet English dataset labeled into hate speech, offensive language without hate speech, and no offensive content. Each message received judgments from at least three annotators.
- The dataset contains 14,509 English tweets annotated specifically for three classes: HATE, OFFENSIVE, and OK.
- At least three annotators judged each short message and assigned an adjudicated label.
- The HATE class contains hate speech, OFFENSIVE contains offensive language without hate speech, and OK contains no offensive content.
- The texts are tweets limited to a maximum of 140 characters.
4 Features
The feature set combines surface n-grams and skip-grams with word representations based on Brown clustering. Cluster bitstrings and prefixes provide hierarchical lexical features that capture syntactic and semantic relationships.
- Basic surface features include character n-grams of lengths 2–8 and word n-grams of lengths 1–3.
- Skip-word bigrams with skips of 1, 2, and 3 approximate longer-distance dependencies that ordinary bigrams may miss.
- The study uses word-representation features based on Brown clustering rather than distributional or embedding representations.
- Brown Clustering: The clustering resources derive from 56 million English tweets containing 837 million tokens and 1,000 hierarchical clusters over 217 thousand words.
- Brown Clustering: Brown clusters organize words hierarchically in a binary tree, with related words sharing syntactic and semantic properties.
- Brown Cluster Feature Representation: Each word’s cluster bitstring and prefixes of lengths 2, 4, 6, through 16 are used as features representing ancestor-node supersets.
5 Methodology
The methodology combines linear SVM classifiers with diverse feature spaces and ensemble fusion methods, including parallel and stacked architectures. Fusion rules operate on class labels, confidence probabilities, medians, or ranked preferences.
- Texts are lowercased, with mentions, URLs, and emoji removed during preprocessing.
- Linear SVMs perform multi-class classification for both single-model and ensemble experiments.For ensembles, each SVM is trained on a different feature type to maximize diversity.
- Ensembles independently run classifiers and aggregate their outputs through late fusion or stacked meta-learning.Parallel architectures fuse predictions, while stacked architectures feed first-level outputs into a meta-classifier.
- Ensemble elements can vary by algorithm, parameters, feature type, preprocessing, scaling, or training-data sampling.Bagging uses bootstrap samples, whereas boosting changes instance weights toward misclassified examples.
- Fusion-method choice depends on whether classifiers output class labels or continuous confidence probabilities, and no single method dominates universally.Performance is influenced by the task, training data, ensemble size, base classifiers, and output diversity.
- Voting selects the most frequent class, while mean probability fusion selects the label with the highest summed or average support.Probability fusion also credits support for the true label when it is not the classifier’s top prediction.
- Median fusion uses a robust central-tendency estimate, while Borda fusion ranks labels and assigns points by preference order.Borda count can select a label that lacks the majority of first-preference votes.
5.4 Evaluation
The study evaluates classification accuracy with stratified 10-fold cross-validation and compares predictions with majority and oracle baselines.
- Accuracy is reported under stratified 10-fold cross-validation, preserving approximately equal class proportions across folds.
- Results are compared against a majority baseline and an oracle that is correct whenever at least one classifier predicts the instance’s true label.The oracle estimates the potential upper limit given the classifiers and features.
6 Single Classifier Experiments
Single-classifier experiments compare feature spaces under cross-validation and examine learning with increasing training data. Character 4-grams perform best, while combining all features adds dimensionality without significant gains.
- The first experiment trains one classifier per feature space and one classifier using all feature classes together.
- Character 4-grams achieve the best performance among the evaluated feature spaces.
- Word unigrams perform similarly well, whereas word bigrams, trigrams, and skip-grams show degraded performance.Word-representation n-grams do not perform significantly better than the other features.
- 5.5 million features result from combining all feature classes, but the combined classifier does not significantly outperform individual classifiers.
- Accuracy increases continuously with more training data, while fold-to-fold standard deviation decreases.Accuracy grows more slowly after 15k training sentences and does not plateau at the maximum training size.
7 Ensemble Classifier Experiments
The ensemble experiment combines 16 feature-specific classifiers using several fusion strategies. Mean probability fusion performs best among the ensembles but does not exceed the best single classifier.
- The ensemble contains 16 individual classifiers because simply combining all features did not improve performance.
- Each ensemble base classifier is trained on a different feature space.
- Mean probability fusion achieves the best ensemble performance, although it remains no better than the best single classifier.
8 Meta-classification Experiment
The study applies meta-classification to combine outputs from 16 classifiers for three-class discrimination. Both meta-classifiers outperform base classifiers, with an RBF SVM reaching 79.8% accuracy, while HATE remains difficult to classify.
- Meta-classification: Meta-learning combines classifier outputs by training a second-level learner rather than using rule-based aggregation.Stacked generalization maps base-classifier outputs to true class labels.
- Meta-classification: The meta-classifier receives 48 features formed from three continuous class outputs supplied by each of 16 classifiers.Each classifier contributes one value per class.
- Results: 79.8% accuracy was achieved by the RBF kernel SVM meta-classifier, outperforming the base classifiers by a substantial margin.The experiment compares two meta-classifiers for combining individual classifiers.
- Results: The best result improved almost 2% over the best single classifier but remained roughly 12% below oracle performance.These comparisons are reported for the RBF kernel SVM meta-classifier.
- Results: The HATE class was highly confused with OFFENSIVE, whereas OK was classified most correctly.The normalized confusion matrix accounts for imbalanced class distributions.
9 Feature Analysis
Feature analysis examines the most informative SVM-weighted unigrams and bigrams across the three classes. Profanity appears in both HATE and OFFENSIVE, while grammatical-word bigrams help identify OK posts.
- Feature-ranking method: SVM feature weights were used to rank the most informative unigrams and bigrams for each class.The analysis focuses on linguistically relevant patterns in the highest-ranked features.
- HATE and OFFENSIVE: Profane and obscene words were highly informative for both HATE and OFFENSIVE classes.The shared prominence of profanity limits its ability to distinguish these classes alone.
- HATE and OFFENSIVE: HATE posts more often targeted specific ethnic or social groups using terms such as nigger(s), jew(s), queer(s), and faggot(s).The analysis also distinguishes the ranked use of nigger(s) and nigga(s) across HATE and OFFENSIVE.
- OK: OK posts were discriminated by frequent bigrams containing grammatical words, including “and should” and “of those”.Twelve highly informative OK-class bigrams are listed in Table 7.
10 Discussion
The discussion concludes that hate speech and profanity are difficult to distinguish with standard surface features. It highlights deeper semantic processing, improved annotation, and further feature and model development as directions for future work.
- Discussion: Distinguishing hate speech from profanity remains difficult, and standard surface features may not provide high accuracy.The study identifies this difficulty as a key finding.
- Future directions: Dependency parsing and semantic parsing are proposed as ways to provide deeper linguistic or meaning-based representations.These features are suggested for improving performance in the task.
- Annotation: Manual review found many apparent classifier errors were instead incorrect gold labels, indicating a need for better annotation.The authors suggest more annotators, explicit instructions, and possible dataset augmentation.
- Semantic difficulty: The example “Girls like you need to be 6 feet under” was labeled OK although the authors judged it to be HATE.The case illustrates hate speech expressed without profanity or colorful language.
- Feature limitations: Hierarchical word clusters produced no significant performance boost because their 1,000 clusters were not fine-grained enough for useful semantic distinctions.The authors leave larger cluster inventories, such as 3,000 or more, for future work.
- Model limitations: The meta-classifier hyperparameters were not tuned, so tuning could potentially improve performance.This is identified as an additional limitation of the experiments.
- Alternative formulation: Verification through novelty or outlier detection is proposed as an alternative to binary or multi-class classification.A one-class SVM is suggested as one possible implementation.