Source-linked AI summary
Antisocial Behavior in Online Discussion Communities
Justin Cheng, Cristian Danescu-Niculescu-Mizil, Jure Leskovec
TL;DR
Antisocial behavior can harm online communities, yet large-scale evidence about its development and detection is limited. The paper analyzes banned and non-banned users across three communities longitudinally, finding worsening behavior, declining tolerance, distinct trajectories, and over 80% AUC for early prediction of future bans.
Problem
Large-scale quantitative understanding of online antisocial behavior and its evolution remains limited, despite its harm to users and communities.
Method
The paper retrospectively analyzes complete longitudinal activity from three large communities, comparing Future-Banned Users with Never-Banned Users and modeling antisocial behavior.
Results
Future-Banned Users write worse posts over time, become less tolerated, form distinct behavioral trajectories, and can be predicted with over 80% AUC from posting behavior.
Takeaways & Limitations
Antisocial users can be identified early, potentially reducing the manual labor required to manage antisocial behavior.
Takeaways & Limitations
The analysis is restricted to permanently banned users and three communities, so other user trajectories and community settings remain unexamined.
Abstract
from arXiv · showhide
User contributions in the form of posts, comments, and votes are essential to the success of online communities. However, allowing user participation also invites undesirable behavior such as trolling. In this paper, we characterize antisocial behavior in three large online discussion communities by analyzing users who were banned from these communities. We find that such users tend to concentrate their efforts in a small number of threads, are more likely to post irrelevantly, and are more successful at garnering responses from other users. Studying the evolution of these users from the moment they join a community up to when they get banned, we find that not only do they write worse than other users over time, but they also become increasingly less tolerated by the community. Further, we discover that antisocial behavior is exacerbated when community feedback is overly harsh. Our analysis also reveals distinct groups of users with different levels of antisocial behavior that can change over time. We use these insights to identify antisocial users early on, a task of high practical importance to community maintainers.
Introduction
The paper uses large-scale longitudinal analysis to characterize antisocial behavior, its evolution, community responses, user types, and early prediction of future bans.
- Antisocial behavior can negatively affect users and communities, including through trolling, flaming, bullying, and harassment.
- Quantitative evidence remains limited because prior research often relies on qualitative studies of small user samples in specific communities.
- The study analyzes three large discussion communities using retrospective longitudinal data to examine behavioral development, community reactions, and early identification.
- Future-Banned Users write less similarly to others, produce harder-to-understand posts, and use less positive or more conflict-provoking language than Never-Banned Users.
- FBUs concentrate posts in individual threads and receive more replies, potentially drawing others into fruitless discussions.
- Over time, FBUs’ post quality worsens, community tolerance declines, deletion rates increase, user types diverge, and future bans can be predicted from early posting behavior.A classifier achieves over 80% AUC after observing 5 to 10 posts.
- The methods can identify antisocial users early and potentially reduce the manual labor required to manage antisocial behavior.
Related Work
Prior work has largely characterized online antisocial behavior qualitatively, while this paper emphasizes large-scale quantitative analysis and early detection across communities.
- Earlier literature discusses trolling, flaming, and griefing through varied definitions emphasizing disruptive or upsetting online behavior.
- Research has commonly used detailed case studies of a small number of manually identified trolls to examine types, motivations, and responses.
- The present work differs by analyzing antisocial behavior quantitatively across three large online communities and developing tools for early troll detection.
- Related detection studies use language, reputation, metadata, relevance, insults, and other features to identify undesirable behavior in different settings.
Data Preparation
The study constructs a longitudinal dataset of banned and non-banned users, operationalizes antisocial behavior through moderator bans, and measures post quality with human-informed text modeling.
- The dataset covers CNN.com, Breitbart.com, and IGN.com, with timestamped activity from March 2012 to August 2013.
- The analysis treats moderator bans as relatively strong user-level indicators of antisocial behavior, while acknowledging that they do not identify every antisocial user.
- Future-Banned Users are banned users with at least five posts who joined after March 2012; multiply banned users and link spammers are excluded.
- The retained banned-user sets contain 10,476 CNN users, 660 IGN users, and 736 Breitbart users with complete activity traces.
- Matching controls comparisons between FBUs and NBUs for total posting activity and posts per day.
- Human ratings of sampled posts support a logistic-regression bigram classifier that labels posts as appropriate above 3, achieving 0.70 AUC under ten-fold cross-validation.
- Figure 1 compares FBUs and NBUs on within-thread similarity, readability, and positive emotion; higher readability index values indicate harder-to-read posts.
Understanding Antisocial Behavior
Future-Banned Users (FBUs) differ from Never-Banned Users in language, readability, discussion activity, and behavior over time. Their posts become less appropriate, while community tolerance also declines, and harsh deletion may worsen later behavior.
- Writing behavior: FBUs write less similarly to other users, appear less readable, and use less positive language and more profanity.Their higher Automated Readability Index indicates lower readability than NBUs.
- Discussion activity: FBUs receive more replies, contribute more posts per thread, and concentrate activity in fewer threads than average users.Their discussion-generation pattern varies by community: they reply more on Breitbart and IGN but start more discussions on CNN.
- Explanatory analyses: The study tests whether declining quality and increasing community bias explain rising deletion rates, and examines whether excessive early deletion worsens subsequent writing.The latter analysis compares matched users with similarly rated initial posts but different early deletion histories.
- Evolution over time: FBUs’ post quality decreases over time, with larger within-user changes than for NBUs.The first-half versus second-half comparison finds larger effect sizes for FBUs (d>0.15) than NBUs (d<0.02).
- Community tolerance: Posts made later in an FBU’s life are more likely to be deleted even when matched to earlier posts with similar predicted quality.Across communities, the effect is significant for FBUs (W>14705, p<10^-4, r>0.21) but not NBUs (p>0.39, r<0.04).
Types of Antisocial Users
FBUs comprise distinct groups with different deletion patterns and behavioral trajectories. Their deletion rates and discussion behavior can also change across the stages of a user’s community life.
- User types: FBUs’ deleted-post proportions are bimodal, separating users into roughly equal high- and low-deletion populations.Hi-FBUs have proportions above 0.5, while Lo-FBUs are below 0.5.
- User types: Hi-FBUs use less accommodating language, receive more replies, and write more posts per thread than Lo-FBUs.On CNN and Breitbart, Hi-FBUs also swear more; overall, they write about half as many posts over a shorter period.
- Deletion trajectories: Hi-FBUs begin with high deletion rates that remain high, whereas Lo-FBUs start lower and rise significantly during the second half of their lives.Lo-FBUs also become less textually similar to other posts and shift toward posting more frequently in fewer threads.
- Deletion trajectories: A two-phase model fits deletion-rate trends separately across the first and second halves of each user’s life using slopes m1 and m2.Users are represented as points (m1, m2) to compare aggregate behavior and changes over time.
- Behavioral change: FBUs are more likely than NBUs to worsen continuously, while NBUs are more likely to improve; some initially high-deletion users improve over time.Across quadrants, increasing text similarity is associated with lower deletion rates, and decreasing similarity with higher rates.
- Behavioral change: Banned status is an incomplete proxy for antisocial behavior because some NBUs improve, remain uncaught, or later become model community members.Many NBUs fall into the improving pattern, while some FBUs improve yet are still banned.
Identifying Antisocial Users
The paper develops early classifiers for future bans using behavioral signals observed in users’ initial posts, while examining how prediction changes with observation timing and user type. These models achieve strong within-community and cross-community performance, but prediction varies substantially across antisocial-user groups.
- Motivation and feature design: Users who are later banned tend to live for long periods before banning, motivating early identification from their initial activity.On CNN, Future-Banned Users average 264 posts over 42 days, including 124 deleted posts before banning.
- Motivation and feature design: The classifier combines post content, activity, community response, and moderator signals observed during a user’s first ten posts.Features include readability, affective language, thread concentration, votes, and deletion-rate statistics.
- Prediction performance: 0.82 mean AUC is achieved for predicting subsequent bans, compared with 0.69 for a bigram bag-of-words baseline and 0.73 using deleted-post proportion alone.Removing moderator features reduces mean AUC to 0.79.
- Prediction performance: 0.80 AUC is achieved using only a user’s first five posts, with performance generally peaking near ten observed posts.Prediction becomes harder when examined posts are increasingly distant from the eventual deletion or banning time.
- Prediction performance: 0.99 mean recall identifies high-deletion-rate users, versus 0.41 for low-deletion-rate users.Reweighting improves low-deletion-rate recall to 0.57 while reducing high-deletion-rate recall to 0.97; a low-deletion-rate-specific classifier reaches 0.79 mean AUC and 0.83 recall.
- Cross-community generalization: 0.74 mean AUC is obtained across communities, and a Breitbart-trained classifier reaches 0.76 on CNN and 0.74 on IGN versus 0.78 on Breitbart.Bag-of-words classifiers generalize less well, with mean AUC 0.58.
Discussion & Conclusion
The paper concludes that antisocial behavior can be characterized and identified early, but its operational definition and interventions have important boundaries. The authors call for richer behavioral analyses while cautioning that harsh enforcement and imperfect classification may worsen outcomes or harm innocent users.
- Contributions and scope: The study characterizes antisocial users, tracks worsening behavior and declining community tolerance, proposes a deletion-rate typology, and introduces early identification.Its analyses use users who are eventually banned as the operational signal of undesirability.
- Limitations and future work: Permanent banning captures varied overt antisocial behavior, but finer-grained labels could reveal a broader range of user behavior.The authors also note that covert behaviors such as deceptive argument instigation may differ from overt inflammatory behavior.
- Limitations and future work: Improved post-content and thread-relation analysis could reveal how antisocial users steer discussions off topic.Initial evidence suggests replies to Future-Banned Users are less similar to preceding thread posts than replies to Never-Banned Users.
- Limitations and future work: Future work should develop richer taxonomies, model user trajectories more flexibly, study temporarily banned users, and test communities beyond those examined.The paper specifically mentions small special-interest communities as an additional setting.
- Intervention considerations: 0.80 average classifier precision still leaves one in five identified users misclassified, making human review or opportunities for redemption relevant alternatives to automatic bans.The authors warn that extreme action against minor infractions can exacerbate antisocial behavior.