Source-linked AI summary
Mean Birds: Detecting Aggression and Bullying on Twitter
Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Athena Vakali
TL;DR
Cyberbullying and cyberaggression are widespread on social media, while effective Twitter detection tools remain scarce. The paper develops a scalable methodology using user, text, and network attributes with crowdsourced labels, and finds that it can distinguish behavioral classes with over 90% AUC.
Problem
Effective tools for detecting harmful online behavior on Twitter are scarce, partly because aggression is ambiguous and can appear in superficial comments or criticisms.
Method
The paper extracts 30 user-, text-, and network-based features, uses crowdsourced user labels, and applies machine-learning classification to Twitter data.
Results
0.907 weighted AUC, 89.9% precision, and 91.7% recall were achieved, while network-based features were the most effective for aggressive-user classification.
Takeaways & Limitations
Bully users were less popular and participated in fewer communities, whereas aggressive users were more negative and could be distinguished from normal users with high accuracy.
Takeaways & Limitations
The hate-related dataset also contains non-aggressive or non-hateful users, which can skew metrics such as popularity.
Abstract
from arXiv · showhide
In recent years, bullying and aggression against users on social media have grown significantly, causing serious consequences to victims of all demographics. In particular, cyberbullying affects more than half of young social media users worldwide, and has also led to teenage suicides, prompted by prolonged and/or coordinated digital harassment. Nonetheless, tools and technologies for understanding and mitigating it are scarce and mostly ineffective. In this paper, we present a principled and scalable approach to detect bullying and aggressive behavior on Twitter. We propose a robust methodology for extracting text, user, and network-based attributes, studying the properties of cyberbullies and aggressors, and what features distinguish them from regular users. We find that bully users post less, participate in fewer online communities, and are less popular than normal users, while aggressors are quite popular and tend to include more negativity in their posts. We evaluate our methodology using a corpus of 1.6M tweets posted over 3 months, and show that machine learning classification algorithms can accurately detect users exhibiting bullying and aggressive behavior, achieving over 90% AUC.
1 INTRODUCTION
Cyberbullying and cyberaggression are widespread, harmful behaviors that are difficult to detect on Twitter because tweets provide limited, noisy, and context-dependent evidence. The paper proposes a scalable, feature-rich detection methodology and reports strong classification performance.
- Cyberbullying typically involves repeated hostile behavior, whereas cyberaggression involves intentional electronic harm perceived as offensive or unwanted.
- About 50% of young social media users reported being bullied online in 2014.
- Twitter detection is limited by noisy short texts, sparse context, and the need to interpret behavior across users and tweets.
- The methodology extracts 30 user-, text-, and network-based features and uses crowdsourced labels to study and classify aggressive behavior.
- 0.907 weighted AUC, 89.9% precision, and 91.7% recall were achieved for detecting bullying and aggressive users.
2 RELATED WORK
Prior work detects abusive content using textual, sentiment, user, and structural features across several platforms. This paper extends that line of work with a scalable Twitter methodology emphasizing network-related attributes.
- Other work detected offensive content on platforms including Instagram, YouTube, 4chan, Yahoo Finance, and Yahoo Answers using textual and structural features.
- Related research addressed cyberbullying on YouTube and Ask.fm through manual annotation, bag-of-words classification, and fine-grained role or behavior analysis.
- Earlier studies used regression, Naive Bayes, Support Vector Machines, and decision trees with linguistic, lexical, sentiment, and user-based features.
- The paper analyzes Twitter users’ tweets individually and in groups while extracting text, user, and network-based attributes.
- The methodology advances prior work by examining Twitter-specific network attributes and comparing detections with observed account suspension and deletion.
3 METHODOLOGY
The methodology transforms Twitter data into session-level labeled user representations, extracts user, text, and network features, and applies machine-learning classification. Its design supports parallel and online processing at large scale.
- The pipeline comprises data collection, preprocessing, sessionization, ground-truth building, feature extraction, user modeling, and classification.
- Tweets are grouped into time-based sessions because individual tweets often lack enough context to identify aggression or bullying.
- Ground truth is built from crowdsourced human labels assigned to sets of tweets from each user.
- Features represent user profiles, tweet text, and network structure, including attributes such as followers, tweets, and hashtags.
- Classification can use probabilistic classifiers, decision trees, ensembles such as Random Forests, or neural networks.
- Several processing steps can be parallelized over N data subsets and cores, with periodic annotation enabling model updates on new data.
4 DATASET & GROUND TRUTH
The evaluation combines random and hate-related Twitter samples, cleans and sessionizes user activity, and uses crowdsourced batches to construct ground truth. The labeling process captures user-level behavior but the hate-related sample includes non-aggressive activity that can skew metrics.
- Data Collection: The dataset contains 1M random tweets and 650k hate-related tweets collected through Twitter’s Streaming API between June and August 2016.
- Data Collection: The hate-related set was collected using 309 bullying- and hate-related hashtags, including hashtags identified from a 1% sample containing #GamerGate.
- Data Collection: Hate-related users showed more followers than baseline users, although this difference may reflect the dataset’s composition.
- Data Collection: The hate-related dataset includes users who are not aggressive or hateful, potentially skewing measures such as popularity.
- Sessionization: Users tweeting fewer than five times in three months were removed, and sessions used an 8-hour maximum inter-arrival threshold.
- Crowdsourced Labeling: Each user was labeled from tweet batches by five workers using majority vote; 4.5% were bullies, 3.4% aggressors, 31.8% spammers, and 60.3% normal.
5 FEATURE EXTRACTION
The paper extracts user-, text-, and network-based features and tests distributional differences across behavioral classes. These features include activity, language, sentiment, network position, and community structure.
- User-based features: User-based features include tweet counts, account age, subscribed lists, verification, and default profile-image usage.
- User-based features: Normal users subscribe to more lists than bullying, spam, and aggressive users, with medians of 74, 24, 57, and 40, respectively.The reported maximum list counts are 4,327, 428, 3,723, and 1,075, respectively.
- Text-based features: Text features measure hashtags, uppercase text, emoticons, URLs, word embeddings, sentiment, hate speech, and curse words.Word2Vec represents each input text by averaging its words’ 300-dimensional vectors, while SentiStrength estimates positive and negative sentiment on a [-4, 4] scale.
- Text-based features: Normal users post fewer URLs than bully, spam, and aggressive users, whose median URL values are 1, 1, and 0.9 versus 0.6 for normal users.The maximum URL values are 1.17 for bully users, 2 for aggressive users, 2.38 for spam users, and 1.38 for normal users.
- Text-based features: Aggressive and bully users use more hashtags, while their sentiment distributions differ statistically from normal users.The paper reports Kolmogorov-Smirnov testing with statistical significance defined as p < 0.05.
- Network-based features: Network features capture friends, followers, reciprocity, power difference, hubs, authorities, eigenvectors, clustering coefficient, and communities.Bullies have fewer friends and followers, while bullies and spammers are less prone to create clusters than aggressive and normal users.
6 MODELING AGGRESSORS & BULLIES
The paper evaluates Random Forest models for distinguishing bully, aggressive, spam, and normal users, including experiments after spam removal and data balancing. Performance improves substantially after spam removal, while user- and network-based features contribute most to classification.
- Modeling procedure: Random Forest is selected as the reported classifier because it provides the best results with respect to training time and performance among tested algorithms.The experiments use repeated 10-fold cross-validation in WEKA without balancing the data in the initial evaluation.
- Experimental setups: The study tests four-class and three-class classification setups for distinguishing offensive user types from normal users.The three-class setup excludes spam after preprocessing.
- Classification results: 73.45% accuracy is achieved in four-class classification, with 71.6% average precision and 73.32% average recall.The model detects 43.2% of bully cases and 11.8% of aggressive cases by recall.
- Classification results: 91.08% accuracy and 0.907 AUC are achieved after spam removal, with 89.9% average precision and 91.7% average recall.Bully precision increases by 14.4% and recall by 17.7%, while aggressor precision and recall remain nearly unchanged.
- Feature evaluation: User- and network-based features are the strongest contributors across both classification setups.These features describe user activity and connectivity in the network.
- Balanced data: 91.25% accuracy is obtained after balancing the three-class training data, while bully and aggressive case detection reaches 66.7% and 40%, respectively.Only the training set is resampled; the test set is not.
7 DISCUSSION & CONCLUSION
The paper presents a scalable Twitter system for classifying cyberaggression and cyberbullying using user, text, and network features. It finds that behavior patterns differ across user classes, network features are especially effective, and the classifier distinguishes harmful users with over 91% accuracy.
- Discussion: The system automatically classifies cyberaggression and cyberbullying on Twitter using a scalable methodology.The approach addresses the scarcity of effective tools for detecting harmful online behavior.
- User behavior: Bullies are less popular and participate in fewer communities, while aggressive users post in short bursts and use more hashtags and URLs.Aggressive users also tend to express more negative sentiment than bullies or normal users.
- User behavior: Aggressive users are more difficult to characterize and identify than bullies because they sometimes resemble normal or spam users.Their behavior varies across these user classes.
- Performance: Over 91% accuracy is achieved for distinguishing normal, aggressive, and cyberbullying users with a Random Forest model.The methodology for data analysis, labeling, and classification scales to millions of tweets.
- Feature importance: Network-based features are the most effective for classifying aggressive behavior, whereas text-based features contribute less except for tweet characteristics, hashtags, URLs, and sentiment.Half of the top 12 features by discriminatory power are network-based.
- Observed account statuses: Aggressors are suspended more often than bullies or normal users, while bullies tend to delete their accounts rather than being suspended.The paper compares predictions with account statuses observed in the wild.