Source-linked AI summary
A Unified Deep Learning Architecture for Abuse Detection
Antigoni-Maria Founta, Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Athena Vakali, Ilias Leontiadis
TL;DR
Abuse detection on social media is difficult because abusive behavior has multiple, interacting forms and cannot be reliably inferred from post text alone. The paper develops a unified deep-learning architecture that combines text with user and post metadata across tasks. It reports improved performance across diverse abuse datasets and states that the approach can be used without task-specific architectural tuning.
Problem
Abusive behavior on social media has multiple interacting forms, while text-only detection is insufficient for reliably identifying generic abuse.
Method
A unified deep-learning architecture combines automatically extracted text patterns with available user, network, account, and post metadata.
Results
The architecture outperforms state-of-the-art methods across datasets covering different abusive behaviors.
Takeaways & Limitations
One architecture can detect diverse abusive behaviors across four datasets without tuning or reconfiguring for each task.
Takeaways & Limitations
The experiments retain the original highly imbalanced Offensive dataset, and task-specific parameter tuning might improve performance further.
Abstract
from arXiv · showhide
Hate speech, offensive language, sexism, racism and other types of abusive behavior have become a common phenomenon in many online social media platforms. In recent years, such diverse abusive behaviors have been manifesting with increased frequency and levels of intensity. This is due to the openness and willingness of popular media platforms, such as Twitter and Facebook, to host content of sensitive or controversial topics. However, these platforms have not adequately addressed the problem of online abusive behavior, and their responsiveness to the effective detection and blocking of such inappropriate behavior remains limited. In the present paper, we study this complex problem by following a more holistic approach, which considers the various aspects of abusive behavior. To make the approach tangible, we focus on Twitter data and analyze user and textual properties from different angles of abusive posting behavior. We propose a deep learning architecture, which utilizes a wide variety of available metadata, and combines it with automatically-extracted hidden patterns within the text of the tweets, to detect multiple abusive behavioral norms which are highly inter-related. We apply this unified architecture in a seamless, transparent fashion to detect different types of abusive behavior (hate speech, sexism vs. racism, bullying, sarcasm, etc.) without the need for any tuning of the model architecture for each task. We test the proposed approach with multiple datasets addressing different and multiple abusive behaviors on Twitter. Our results demonstrate that it largely outperforms the state-of-art methods (between 21 and 45\% improvement in AUC, depending on the dataset).
Introduction
Online abuse is multifaceted, harmful, and difficult to detect with content alone. The paper proposes a unified architecture combining text and metadata across abuse types and datasets.
- Abusive behavior spans hate speech, offensive, sexist and racist language, aggression, cyberbullying, harassment, and trolling.
- The proposed architecture combines users’ behavior, social network, popularity, account settings, post metadata, and text representations.
- Alternating training between heterogeneous inputs helps prevent dominant features from controlling learning for each task.
- The unified architecture works across four abuse datasets without tuning or reconfiguring for each task.
- The methodology is presented as adaptable to toxic-behavior detection in online gaming without further tuning.
Background and Related Work
Prior work addresses individual abuse categories with traditional, deep-learning, textual, or task-specific methods. This paper positions its contribution as a unified architecture for diverse abusive behaviors and reports improvements across them.
- Research has separately studied hate speech, cyberbullying, abusive behavior, offensive language, racism, and sexism across platforms and datasets.
- Most previous methods use traditional classifiers or task-specific deep-learning architectures with textual and engineered features.
- The paper introduces a unified neural architecture that processes text and numerical metadata for multiple abuse forms.
- Interleaved training is introduced for the multi-input network and is described as a first application of this architecture to text classification.
- The proposed architecture improves over state-of-the-art methods across all types of abusive behavior.
Deep Learning Architectures
The paper presents a unified model that processes tweet text and metadata through separate paths before combining their learned representations. It uses interleaved training to prevent one input path from dominating and supports multiple abuse-detection tasks without task-specific reconfiguration.
- Unified Architecture: The unified architecture separately processes raw text and domain-specific metadata before combining them into one classifier.The design aims to support many abusive-content detection tasks with one architecture.
- Individual Classifiers: The text path uses a word-level recurrent network with pre-trained word embeddings, while the metadata path processes numerical user, tweet, and related features.The text classifier uses RNNs and GloVe embeddings; the metadata classifier uses dense layers after numerical preprocessing.
- Combined Classifier: The two paths can be fused by concatenating learned intermediate representations and adding a classification layer over the merged data.The architecture combines the paths before their outputs are reduced to class probabilities.
- Unified Architecture: Combining heterogeneous text and metadata avoids explicit text-feature construction and uses pre-trained word embeddings from larger datasets.The approach is intended to capture information unavailable from text alone.
- Training the Combined Network: Naive joint training is suboptimal because the paths converge at different rates and can produce unpredictable interactions through shared backpropagation.The paper also describes transfer learning, which pre-trains and freezes both paths before training only the final layer.
- Training the Combined Network: Interleaved training updates only one path per mini-batch while data flow through the whole network, alternating paths across batches and epochs.The authors state that this approach achieves the best results across all four datasets.
Dataset
The study evaluates its architecture on Twitter datasets covering cyberbullying, cyber-aggression, racism, sexism, hate speech, offensive language, and sarcasm. Dataset metadata availability varies, and some experiments use only word vectors or omit network features because of data and computation constraints.
- Dataset Overview: Table 1 summarizes tweet and user counts, class distributions, and the availability of word-vector, tweet-, user-, and network-based metadata.These feature categories are abbreviated WV, TF, UF, and NF.
- Datasets: The experiments use datasets spanning cyberbullying and cyber-aggression, racism and sexism, hateful and offensive language, and sarcasm.The datasets differ in labels, collection procedures, and available metadata.
- Cyberbullying Dataset: The cyberbullying dataset contains 9,484 tweets grouped into 1,303 user batches and labeled as bullying, aggressive, spam, or normal.The labels distinguish repeated bullying behavior from aggression involving at least one harmful post.
- Racism and Sexism Dataset: The racism-and-sexism dataset contains 16,000 tweets, including 2,000 racist and 3,000 sexist examples, manually labeled using criteria involving attacks, hate speech, violence, or slurs.The experiments use word vectors, tweet metadata, and user metadata, but not network metadata.
- Hate and Offensive Language Dataset: The hateful/offensive-language dataset provides 25,000 crowdsourced labels from a larger collection, with 77% offensive and 6% hateful tweets.Only tweet text was available for evaluation, so the study used word vectors without additional metadata.
- Sarcasm Dataset: The sarcasm dataset uses tweets collected through the hashtags #sarcasm and #not, after filtering out non-English tweets, retweets, short tweets, mentions, and URLs.The experiments retain the original highly imbalanced dataset and compare against imbalanced baseline results.
Evaluation
Across four datasets, the unified classifier consistently outperforms baselines and state-of-the-art methods, with performance strongest when text and metadata are combined. Interleaved training further improves the multi-input model by avoiding harmful simultaneous updates between paths.
- Training methodology: 0.96 AUC is reached on cyberbullying with alternate training, compared with 0.94 AUC for naive whole-network training.Transfer learning provides a smaller improvement, while alternate training avoids interactions caused by simultaneous updates to both paths.
- Classification performance: Across all datasets and abusive behaviors, the Interleaved classifier outperforms both baseline and state-of-the-art methods.The authors attribute this to combining text with user and network information and using pretrained word embeddings.
- Classification performance: 0.93 AUC and 0.89 accuracy are obtained on cyberbullying with single attribute sets, but the interleaved model reaches 0.96 AUC and 0.92 accuracy.The comparison with prior work is not direct because the prior results are user-level whereas these results are tweet-level.
- Classification performance: 0.98 AUC versus 0.7 for existing methodology is achieved on the sarcasm dataset when metadata complements text.The authors report that user, network, sentiment, and tweet-level metadata reveal information that text alone does not carry sufficiently.
- Metadata importance: 0.91 AUC is obtained from raw text alone, compared with 0.8 for user- and tweet-based metadata and 0.64 for network-level metadata.Individual metadata classes perform weakest, while combining attributes improves performance.
Behavior in Online Gaming
The same architecture was adapted to online-game toxicity detection using chat logs and domain-specific metadata. With only an attention layer added for long text, it exceeded prior reported performance on the general pardon-versus-punish task.
- Dataset: The gaming dataset contains chat logs, domain-specific player and match metadata, and crowdsourced pardon-or-punish labels.Features include player performance, other players’ performance, game outcome, report counts, and common report type.
- Dataset: 2,500 words per match versus 30 words per tweet makes the gaming text substantially longer than the Twitter data.The gaming metadata is also domain-specific and lacks a direct analogue in the Twitter datasets.
- Results: 0.93 accuracy and 0.89 AUC are achieved on the general gaming classification task with no architectural changes beyond an attention layer.The attention layer was enabled to handle the longer text.
- Results: 0.89 AUC exceeds the prior model’s 0.80 AUC on the general pardon-versus-punish task and even its 0.88 AUC on overwhelming pardons.The prior work reported 0.75 AUC for overwhelming punish decisions.
Summary
The paper presents one unified deep-learning architecture that combines text and metadata to detect multiple abusive behaviors. It uses interleaved training for multi-input paths and is designed to extend to other data types and domains.
- Unified architecture: The exact same deep-learning architecture handles all four abusive-behavior datasets without task-specific reconfiguration.Parameter finetuning could improve performance further, but the shared methodology surpasses state-of-the-art results for each behavior.
- All inputs help: Text, user, network, and tweet attributes are seamlessly combined in one classification model without particular tuning.The paper evaluates how each attribute contributes to each task.
- Training methodology: Interleaved training alternates updates between the text and metadata paths to improve multi-input performance.The paper compares this paradigm with ensemble, feature-transfer, and concurrent-training strategies.
- Flexible to other data: Additional input paths can be concatenated, including image features, text information, and image metadata in other applications.The paper gives image classification as an example of extending the architecture beyond text and metadata.