Source-linked AI summary
Contextualizing Hate Speech Classifiers with Post-hoc Explanation
Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, Xiang Ren
TL;DR
Hate-speech classifiers can over-associate group identifiers with hate speech when imbalanced data limits their ability to learn contextual usage. The paper uses SOC post-hoc explanations to detect this bias and regularizes identifier importance so models attend more to surrounding context. The approach reduces false positives on out-of-domain data while maintaining or improving in-domain performance, though it does not support reasoning across sentences.
Problem
Hate-speech classifiers struggle to distinguish group identifiers used offensively from the same identifiers used in non-hateful contexts.
Method
The paper extracts SOC post-hoc explanations from fine-tuned BERT classifiers and regularizes identifier importance during training to increase sensitivity to surrounding context.
Results
Regularization greatly reduces false positives on out-of-domain news articles while maintaining or improving in-domain out-of-sample classification performance.
Takeaways & Limitations
Regularizing SOC explanations makes hate-speech classifiers more context-sensitive and less reliant on high-frequency identifier words in imbalanced training sets.
Takeaways & Limitations
The method does not yet support the cross-sentence background reasoning required by some complex hate-speech examples.
Abstract
from arXiv · showhide
Hate speech classifiers trained on imbalanced datasets struggle to determine if group identifiers like "gay" or "black" are used in offensive or prejudiced ways. Such biases manifest in false positives when these identifiers are present, due to models' inability to learn the contexts which constitute a hateful usage of identifiers. We extract SOC post-hoc explanations from fine-tuned BERT classifiers to efficiently detect bias towards identity terms. Then, we propose a novel regularization technique based on these explanations that encourages models to learn from the context of group identifiers in addition to the identifiers themselves. Our approach improved over baselines in limiting false positives on out-of-domain data while maintaining or improving in-domain performance. Project page: https://inklab.usc.edu/contextualize-hate-speech/.
1 Introduction
Hate speech classifiers can mistake group identifiers for hateful language because imbalanced data and limited contextual learning produce false positives. The paper uses SOC explanations to identify this bias and regularizes models to attend more to surrounding context.
- Motivation: Imbalanced hate-speech data makes neural classifiers struggle to generalize to real-world applications.The paper attributes this difficulty to the relative sparsity of hate speech and biased sampling strategies.
- Problem: Group identifiers such as “Muslim”, “gay”, and “black” become problematic when classifiers treat their presence as hateful without the right context.A fine-tuned BERT model incorrectly classified one document containing “black” as hate speech, although only the other document used it in a hateful context.
- Approach: SOC post-hoc explanations expose models’ bias toward group identifiers while providing context-independent estimates of feature importance.The method is applied to fine-tuned BERT classifiers and the Gab Hate Corpus.
- Approach: Explanation-based regularization increases model sensitivity to context surrounding group identifiers during training.The method regularizes identifier importance rather than removing identifiers entirely.
- Results: Regularization reduces attention to group identifiers and increases the importance of generalizable hate-speech features such as dehumanizing and insulting language.These changes accompany improved handling of identifier-containing examples.
- Results: On an out-of-domain news test set, regularization greatly reduces false positives while maintaining or improving in-domain out-of-sample performance.The news articles contain group identifiers but are heuristically treated as non-hate speech.
2 Related Work
Prior work addresses hate-speech detection, dataset bias, social-group bias, and post-hoc explanations. This paper extends those directions by regularizing explanations of identifier terms to manipulate how models represent their surrounding context.
- Hate-speech detection: Earlier hate-speech studies formulate detection as distinguishing offensive words used in abusive contexts from the same words used non-abusively.Later work broadens the typology and trains models on annotated or heuristically labeled data.
- Hate-speech detection: Highly skewed language distributions in hate-speech datasets remain a challenge for trained classifiers.The related work links this issue to the distributions present in the underlying datasets.
- Bias in classification: Bias research has measured and mitigated classifiers’ unwanted sensitivity to social-group tokens and documented how annotator biases can be magnified during training.Related studies address toxicity, gender, and racial bias.
- Explanations: Post-hoc explanation methods reveal word-level or phrase-level input importance, enabling interpretation of fine-tuned language models.The paper focuses on explanation approaches that can be used with Transformer-based hate-speech classifiers.
- Paper’s position: This paper directly regularizes explanations of identifier terms to manipulate models’ representations of the surrounding context.The approach complements prior bias-reduction work and is designed for efficient use with fine-tuned language models.
3 Data
The experiments use two publicly available corpora designed to capture rhetorical aspects of hate speech rather than only slurs or explicit offensive language. They cover Gab and Stormfront domains with related annotation goals.
- Corpus selection: The selected corpora emphasize rhetorical hate speech rather than merely slurs and explicitly offensive language.This design targets the contextual and rhetorical aspects of the task.
- Gab Hate Corpus: The Gab Hate Corpus contains 27,655 randomly sampled posts from Gab and is annotated for “hate-based rhetoric”.Its construct draws on hate-speech criminal codes outside the United States and research on prejudice and dehumanization.
- Stormfront: Stormfront posts were sampled and annotated at the sentence level using an annotation guide similar to the one used for the Gab Hate Corpus.Both datasets concern domains associated with elevated rates of hate speech.
- Splits and labels: Stormfront uses random 80/20 train-test splits with “hate” as the positive binary label.For the Gab test set, the authors draw a stratified sample by target population and combine human degradation with calls for violence into one hate label.
- Splits and labels: GHCtrain contains 24,353 posts, including 2,027 labeled as hate, while GHCtest contains 1,586 posts.The test set is constructed through random stratified sampling with respect to the target-population tag.
4 Analyzing Group Identifier Bias
The paper measures identifier bias using linear-model features and post-hoc BERT explanations, then examines how identifier removal affects hate-speech and false-positive performance. SOC produces hierarchical importance explanations for phrases and words.
- Analysis design: The analysis combines top features from a linear model, post-hoc BERT explanations, and an adversarial New York Times dataset.The NYT articles are used to examine how identifier-related tendencies affect predictions.
- Analysis design: The classifiers analyzed are bag-of-words logistic regression and fine-tuned BERT models.BERT uses the final-layer CLS representation for two-way hate versus non-hate classification.
- Identifier analysis: The identifier set contains twenty-five identity words collected from the highest-weighted features of bag-of-words logistic regression.Examples include “homosexual”, “muslim”, and “black”.
- SOC explanations: SOC generates hierarchical explanations by estimating phrase importance while eliminating compositional effects between a phrase and its surrounding context.The method is intended to capture compositions in which words become offensive only with particular groups.
- SOC explanations: SOC aggregates word-level explanations and clusters them hierarchically to summarize model importance patterns.The authors compute average word importance on GHCtest and report the top 20 words.
- Prediction bias: Removing identifiers lowers false positives on the NYT set but also harms hate-speech detection performance.The experiment removes random subsets of 0 to 25 words, with each subset size repeated five times.
5 Contextualizing Hate Speech Models
The paper addresses models’ over-sensitivity to group identifiers by regularizing their explanation importance, encouraging attention to surrounding context while retaining identifier information.
- Hate speech models are over-sensitive to group identifiers and unable to learn from the surrounding context during training.
- The proposed approach regularizes models to assign no explained importance to identifier terms, alongside the classification objective.
- Unlike word removal, explanation regularization preserves contextual information related to group identifiers because SOC eliminates compositional effects.
- The regularization objective penalizes squared SOC importance for group identifiers, with α controlling regularization strength.
- The paper also experiments with input occlusion explanations, which measure prediction changes when words or phrases are masked.
6 Regularization Experiments
Experiments compare regularized and word-removal models across in-domain hate-speech classification and NYT evaluation. Regularization improves identifier-context modeling while reducing identifier importance and correcting a false positive.
- Regularization improves in-domain hate-speech classification and false-positive reduction on the NYT corpus for GHC and Stormfront.
- SOC regularization may outperform word removal because it does not prohibit models from using contextual information related to group identifiers.
- Regularization decreases identity-term importance while increasing importance for hate-related terms such as “poisoned” and “blamed.”
- Table 1 reports precision, recall, and F1 on GHCtest and Stormfront, plus NYT accuracy, across BERT, word removal, OC, and SOC models.
- Table 2 compares the top 20 mean SOC-weighted words before and after regularization, including changes in importance rank.
- Hierarchically clustered SOC explanations visualize regularization’s effect in a corrected false-positive example.
7 Conclusion & Future Work
The paper presents SOC explanation regularization as a way to make hate speech classifiers more context-sensitive and less reliant on high-frequency identity terms. The authors report improved internal representations while noting that complex rhetorical reasoning remains unresolved.
- Regularizing SOC explanations makes hate speech classifiers more context-sensitive and less reliant on high-frequency words in imbalanced training sets.
- The approach integrates directly into Transformer-based models without data augmentation, offering a technique for directing internal representations of target phenomena.
- Future Work: Validation across other language models, hate speech datasets, and identity-term sets remains future work.
- Limitations: Regularization improves sensitivity to compositional hate speech, but the models do not yet reason across sentences requiring background information.
- Explanation algorithms provide a window into complex predictive models, while regularization can improve models’ internal representations of target phenomena.
A.1 Full List of Curated Group Identifiers
The appendix lists curated group identifiers used in the experiments. The GHC set contains 25 identifiers selected from top TF-IDF BOW features, while the Stormfront set contains 10 identifiers.
- Table 3 lists 25 group identifiers selected from top-weighted words in a TF-IDF BOW linear classifier on the GHC.
- Table 4 lists 10 group identifiers selected for the Stormfront dataset.
A.2 Visualizations of Effect of Regularization
The appendix visualizes explanation behavior and reports cross-domain evaluation and computational costs. The method uses hierarchical SOC explanations, compares several BERT variants, and adds moderate training overhead relative to baseline BERT.
- Visualizations: Figure 4 shows hierarchical explanations for an NYT test instance where false-positive predictions are corrected.
- Visualizations: Figure 5 shows a Gab instance where both models predict correctly, but only the regularized model predicts for correct reasons.
- Cross-Domain Evaluation: Cross-domain evaluation compares BERT, BERT+WR, BERT+OC, and BERT+SOC using mean and standard deviation across 10 runs.
- Computational Efficiency: BERT+SOC requires 4 times the original BERT training time per epoch, compared with 2 times for BERT+OC.
- Computational Efficiency: All methods finish training at around the third epoch, as reported for per-epoch training-time comparisons on the Gab corpus.