PaperlayerView exact revision on arXiv

Source-linked AI summary

One-step and Two-step Classification for Abusive Language Detection on Twitter

Ji Ho Park, Pascale Fung

arXiv:1706.01206v1cs.CL

TL;DR

Abusive-language detection on social media is important but difficult because large-scale moderation exceeds human capacity and annotation is subjective and context-dependent. The paper compares two-step detection and type classification with one-step multi-class classification for sexist and racist language, finding that two-step classification can boost simpler models such as logistic regression.

  • Problem

    Abusive-language detection matters for social media, but large volumes exceed human moderation capacity and subjective, context-limited annotations are difficult for non-experts.

    Source: §1 · p. 1

  • Method

    The paper compares two-step classification with one-step multi-class classification and evaluates CNN models using character, word, or combined features on concatenated English Twitter datasets.

    Source: §1 · p. 1, §2 · p. 1, §2.3 · p. 2

  • Results

    Two logistic regression classifiers perform about as well as one-step HybridCNN and outperform one-step logistic regression by more than 10 F1 points.

    Source: §4.2 · p. 3

  • Takeaways & Limitations

    The two-step approach shows potential for boosting simpler models and combining classifiers according to their performance on different datasets.

    Source: §5 · p. 4

    Future work proposes training the two-step classifiers on separate datasets to build a more robust and detailed detector.

    Source: §5 · p. 4

Abstract

from arXiv · show

Automatic abusive language detection is a difficult but important task for online social media. Our research explores a two-step approach of performing classification on abusive language and then classifying into specific types and compares it with one-step approach of doing one multi-class classification for detecting sexist and racist languages. With a public English Twitter corpus of 20 thousand tweets in the type of sexism and racism, our approach shows a promising performance of 0.827 F-measure by using HybridCNN in one-step and 0.824 F-measure by using logistic regression in two-steps.

1 Introduction

The paper frames abusive-language detection as an important, difficult moderation problem and proposes comparing one-step and two-step classification, alongside CNN models using character- and word-level inputs.

  • Online platforms struggle to moderate the volume of abusive posts using only human resources.Source: §1 · p. 1
  • Earlier work applied machine-learning classifiers, embeddings, regression, and deeper models to abusive-language detection.Source: §1 · p. 1
  • Abusive-language annotation is difficult because judgments are subjective and tweets often lack context.Source: §1 · p. 1
  • The study compares two-step detection followed by type classification with one-step multi-class classification of sexist and racist language.Source: §1 · p. 1
  • The study also evaluates CNN models using character-level, word-level, or combined inputs across different dataset segmentations.Source: §1 · p. 1

2 Methodology

The methodology compares CharCNN, WordCNN, and HybridCNN architectures that differ in their character- and word-level inputs. HybridCNN combines both feature types to address word-only limitations and accommodate misspellings and novel vocabulary.

  • The study implements CharCNN, WordCNN, and HybridCNN, using character, word, or combined input features.Source: §2 · p. 1
  • CNN layers apply multiple filter sizes and max-pooling to extract salient features from inputs.Source: §2 · pp. 1–2
  • CharCNN converts each input character into a one-hot encoding drawn from 70 character categories.Source: §2.1 · p. 2
  • The CharCNN uses a shallow architecture with dropout because the dataset is relatively small and overfitting is a concern.Source: §2.1 · p. 2
  • WordCNN uses fixed 300-dimensional pretrained word2vec embeddings and keeps them non-trainable for the small dataset.Source: §2.2 · p. 2

2.3 HybridCNN

  • HybridCNN combines character and word channels, with outputs concatenated after 1-max-pooling before softmax classification.Source: §2.3 · p. 2

3 Experiments

The experiments use English Twitter datasets labeled for sexist and racist language to compare one-step multi-class classification with a two-step abusive-language pipeline. Performance is evaluated with weighted F1-based measures against several classifiers and tuned validation settings.

3.1 Datasets

  • The study combines two English Twitter datasets containing sexist and racist comments, then creates datasets for one-step and two-step classification.Source: §3.1 · p. 2

3.2 Training and Evaluation

  • The one-step experiment detects none, sexist, and racist language in a single classification task.Source: §3.2 · p. 2, §3.1 · p. 2
  • The two-step experiment first detects abusive language and then classifies it as sexist or racist.Source: §3.1 · p. 2, §3.2 · p. 3
  • The evaluation uses weighted averaged F1 scores together with precision and recall to account for label imbalance.Source: §3.2 · p. 3
  • Baselines include character n-gram logistic regression, SVM, and FastText classifiers using average bag-of-words representations.Source: §3.2 · p. 3
  • Hyperparameters are selected using a validation set, including specified convolutional filters, feature-map sizes, pooling, and L2 regularization.Source: §3.3 · p. 3

4 Result and Discussions

The experiments compare one-step multi-class classification with two-step binary classification for abusive, sexist, and racist language. HybridCNN performs best among one-step models, while two-step logistic regression is competitive with one-step HybridCNN and exceeds one-step logistic regression by more than 10 F1 points.

One-step classification

  • HybridCNN performs best among the one-step multi-class methods.The authors attribute its improvement over WordCNN to the additional character input channel.Source: §4.1 · p. 3, Table 2 · p. 4
  • Baseline methods have high averaged F1 but low racism and sexism scores because of low recall.Source: §4.1 · p. 3

Two-step classification

  • Two-step classification with two binary classifiers produces results comparable to one-step classification.Source: §4.2 · p. 3, Table 2 · p. 4
  • More than 10 F1 points separate two-step logistic regression from one-step logistic regression, while two-step logistic regression performs about as well as one-step HybridCNN.Source: §4.2 · pp. 3–4
  • HybridCNN performs best for abusive language detection, followed by WordCNN and logistic regression.Source: §4.2 · p. 3
  • The second classifier shows significant performance in predicting a specific type given that the language is abusive.Source: §4.2 · p. 3

5 Conclusion and Future work

The paper concludes that two-step classification can combine different classifiers and improve simpler models such as logistic regression. Future work proposes training the two steps on separate datasets to support a more robust and detailed detector.

Conclusion

  • The two-step approach combines an abusive-language classifier with a classifier for specific sexist and racist types.Source: §5 · p. 4
  • The proposed combination can pair convolutional neural networks and logistic regression according to their performance on different datasets.Source: §5 · p. 4
  • Two-step classification shows potential because specifically labeled abusive-language datasets are harder to acquire than datasets simply flagged as abusive.Source: §5 · p. 4

Future work

  • Future work would train the two classifiers on separate datasets, using a larger abusive-language dataset and a smaller specifically labeled dataset.Source: §5 · p. 4

sentence classification . In Proceedings of EMNLP.,

This passage set consists of references to prior work on text classification, abusive-language detection, hate-speech annotation, word representations, and convolutional networks.

Related work

Related papers

Loading 1706.01206v1…