Source-linked AI summary
One-step and Two-step Classification for Abusive Language Detection on Twitter
Ji Ho Park, Pascale Fung
TL;DR
Abusive-language detection on social media is important but difficult because large-scale moderation exceeds human capacity and annotation is subjective and context-dependent. The paper compares two-step detection and type classification with one-step multi-class classification for sexist and racist language, finding that two-step classification can boost simpler models such as logistic regression.
Problem
Abusive-language detection matters for social media, but large volumes exceed human moderation capacity and subjective, context-limited annotations are difficult for non-experts.
Method
The paper compares two-step classification with one-step multi-class classification and evaluates CNN models using character, word, or combined features on concatenated English Twitter datasets.
Results
Two logistic regression classifiers perform about as well as one-step HybridCNN and outperform one-step logistic regression by more than 10 F1 points.
Takeaways & Limitations
The two-step approach shows potential for boosting simpler models and combining classifiers according to their performance on different datasets.
Takeaways & Limitations
Future work proposes training the two-step classifiers on separate datasets to build a more robust and detailed detector.
Abstract
from arXiv · showhide
Automatic abusive language detection is a difficult but important task for online social media. Our research explores a two-step approach of performing classification on abusive language and then classifying into specific types and compares it with one-step approach of doing one multi-class classification for detecting sexist and racist languages. With a public English Twitter corpus of 20 thousand tweets in the type of sexism and racism, our approach shows a promising performance of 0.827 F-measure by using HybridCNN in one-step and 0.824 F-measure by using logistic regression in two-steps.
1 Introduction
The paper frames abusive-language detection as an important, difficult moderation problem and proposes comparing one-step and two-step classification, alongside CNN models using character- and word-level inputs.
- Online platforms struggle to moderate the volume of abusive posts using only human resources.
- Earlier work applied machine-learning classifiers, embeddings, regression, and deeper models to abusive-language detection.
- Abusive-language annotation is difficult because judgments are subjective and tweets often lack context.
- The study compares two-step detection followed by type classification with one-step multi-class classification of sexist and racist language.
- The study also evaluates CNN models using character-level, word-level, or combined inputs across different dataset segmentations.
2 Methodology
The methodology compares CharCNN, WordCNN, and HybridCNN architectures that differ in their character- and word-level inputs. HybridCNN combines both feature types to address word-only limitations and accommodate misspellings and novel vocabulary.
- The study implements CharCNN, WordCNN, and HybridCNN, using character, word, or combined input features.
- CNN layers apply multiple filter sizes and max-pooling to extract salient features from inputs.
- CharCNN converts each input character into a one-hot encoding drawn from 70 character categories.
- The CharCNN uses a shallow architecture with dropout because the dataset is relatively small and overfitting is a concern.
- WordCNN uses fixed 300-dimensional pretrained word2vec embeddings and keeps them non-trainable for the small dataset.
- 2.3 HybridCNN: HybridCNN combines character and word channels, with outputs concatenated after 1-max-pooling before softmax classification.
3 Experiments
The experiments use English Twitter datasets labeled for sexist and racist language to compare one-step multi-class classification with a two-step abusive-language pipeline. Performance is evaluated with weighted F1-based measures against several classifiers and tuned validation settings.
- 3.1 Datasets: The study combines two English Twitter datasets containing sexist and racist comments, then creates datasets for one-step and two-step classification.
- 3.2 Training and Evaluation: The one-step experiment detects none, sexist, and racist language in a single classification task.
- 3.2 Training and Evaluation: The two-step experiment first detects abusive language and then classifies it as sexist or racist.
- 3.2 Training and Evaluation: The evaluation uses weighted averaged F1 scores together with precision and recall to account for label imbalance.
- 3.2 Training and Evaluation: Baselines include character n-gram logistic regression, SVM, and FastText classifiers using average bag-of-words representations.
- 3.2 Training and Evaluation: Hyperparameters are selected using a validation set, including specified convolutional filters, feature-map sizes, pooling, and L2 regularization.
4 Result and Discussions
The experiments compare one-step multi-class classification with two-step binary classification for abusive, sexist, and racist language. HybridCNN performs best among one-step models, while two-step logistic regression is competitive with one-step HybridCNN and exceeds one-step logistic regression by more than 10 F1 points.
- One-step classification: HybridCNN performs best among the one-step multi-class methods.The authors attribute its improvement over WordCNN to the additional character input channel.
- One-step classification: Baseline methods have high averaged F1 but low racism and sexism scores because of low recall.
- Two-step classification: Two-step classification with two binary classifiers produces results comparable to one-step classification.
- Two-step classification: More than 10 F1 points separate two-step logistic regression from one-step logistic regression, while two-step logistic regression performs about as well as one-step HybridCNN.
- Two-step classification: HybridCNN performs best for abusive language detection, followed by WordCNN and logistic regression.
- Two-step classification: The second classifier shows significant performance in predicting a specific type given that the language is abusive.
5 Conclusion and Future work
The paper concludes that two-step classification can combine different classifiers and improve simpler models such as logistic regression. Future work proposes training the two steps on separate datasets to support a more robust and detailed detector.
- Conclusion: The two-step approach combines an abusive-language classifier with a classifier for specific sexist and racist types.
- Conclusion: The proposed combination can pair convolutional neural networks and logistic regression according to their performance on different datasets.
- Conclusion: Two-step classification shows potential because specifically labeled abusive-language datasets are harder to acquire than datasets simply flagged as abusive.
- Future work: Future work would train the two classifiers on separate datasets, using a larger abusive-language dataset and a smaller specifically labeled dataset.
sentence classification . In Proceedings of EMNLP.,
This passage set consists of references to prior work on text classification, abusive-language detection, hate-speech annotation, word representations, and convolutional networks.
- Related work: The bibliography cites research on word and phrase distributed representations and character-level convolutional networks for text classification.
- Related work: The references include prior work on abusive-language and hate-speech detection using natural language processing.
- Related work: The cited literature includes work on annotation reliability and annotator influence in hate-speech detection.