Source-linked AI summary
Ex Machina: Personal Attacks Seen at Scale
Ellery Wulczyn, Nithum Thain, Lucas Dixon
TL;DR
Personal attacks damage online discourse, yet their prevalence and impact are difficult to study at platform scale. The paper combines crowdsourcing with machine learning and evaluates the classifier against human annotator ensembles. Applied to English Wikipedia, the method produces large labeled corpora and shows that attacks are not primarily attributable to a few malicious or anonymous users.
Problem
The prevalence and impact of personal attacks are difficult to study at scale because platforms contain vastly more comments than can be affordably human-labeled.
Method
The paper crowdsources labels for a sample, trains a text classifier, and evaluates it by comparing its predictions with aggregated human annotator judgments.
Results
The classifier approximates 3 crowd-workers and supports analysis of Wikipedia’s 63M comments, where anonymous comments are 6 times more likely to be attacks but contribute less than half of them.
Takeaways & Limitations
Personal attacks are distributed across user types rather than concentrated among a few malicious or anonymous users, and fewer than a fifth trigger policy action.
Takeaways & Limitations
The crowdsourced labels may reflect the interpretations and unintended biases of a relatively small annotator pool rather than the Wikipedia community’s understanding.
Abstract
from arXiv · showhide
The damage personal attacks cause to online discourse motivates many platforms to try to curb the phenomenon. However, understanding the prevalence and impact of personal attacks in online platforms at scale remains surprisingly difficult. The contribution of this paper is to develop and illustrate a method that combines crowdsourcing and machine learning to analyze personal attacks at scale. We show an evaluation method for a classifier in terms of the aggregated number of crowd-workers it can approximate. We apply our methodology to English Wikipedia, generating a corpus of over 100k high quality human-labeled comments and 63M machine-labeled ones from a classifier that is as good as the aggregate of 3 crowd-workers, as measured by the area under the ROC curve and Spearman correlation. Using this corpus of machine-labeled scores, our methodology allows us to explore some of the open questions about the nature of online personal attacks. This reveals that the majority of personal attacks on Wikipedia are not the result of a few malicious users, nor primarily the consequence of allowing anonymous contributions from unregistered users.
1. INTRODUCTION
Online abuse is widespread, but studying personal attacks at Wikipedia’s scale is costly and difficult. The paper combines crowdsourcing and machine learning to analyze attacks across the full comment corpus and investigate their distribution.
- 73% of adult internet users have seen online harassment, while 40% have personally experienced it.Wikipedia’s policy prohibits personal attacks and allows attacks to be removed and their authors blocked.
- 63M English Wikipedia talk-page comments make human annotation prohibitively expensive and time consuming.Typical annotation efforts label thousands of comments, whereas platforms often contain many orders of magnitude more.
- The paper’s primary contribution is a crowdsourcing-and-machine-learning methodology for quantitative, large-scale, longitudinal analysis of online comments.It applies the methodology to personal attacks on Wikipedia and compares alternative features and labeling methods.
- The classifier approximates the aggregated judgments of 3 crowd-workers when generating labels for the full comment corpus.The paper evaluates the machine model against annotator ensembles and releases human and machine annotations plus the trained classifier.
- Calibrated classifier scores support subgroup analyses of personal attacks that the smaller crowdsourced sample alone could not answer.The analyses examine anonymity, users’ contribution quantity, concentration among users, moderator actions, and attack timing.
2. RELATED WORK
Related work distinguishes personal attacks from other forms of toxic online behavior and documents their effects, while emphasizing limited data and ongoing detection efforts. Existing studies use varied definitions, datasets, and machine-learning approaches.
- Definitions, Prevalence and Impact: Personal attacks are one focus within a broader literature covering hate speech, online harassment, and cyberbullying.The paper notes that negative online behavior lacks a clear, common definition.
- Definitions, Prevalence and Impact: Online harassment includes varied forms such as offensive names, purposeful embarrassment, stalking, sexual harassment, physical threats, and sustained harassment.The Wikimedia Foundation uses a different taxonomy of harassment forms.
- Definitions, Prevalence and Impact: 54% of people who experienced online harassment reported decreased participation in the project where it occurred.Related work also connects hate speech and cyberbullying with suppressing expression, physical violence, and suicide.
- Automated Detection: Machine-learning detection research has applied support vector machines and semantic or contextual features to harassment and abusive-language datasets.The cited work builds on approaches from sentiment analysis and spam detection.
- Data Sets: Public datasets for toxic-behavior detection remain scarce and limited to a small set of open corpora.Researchers have often created hand-coded datasets using crowd-sourced or in-house annotators.
3. CROWDSOURCING
The study builds a Wikipedia discussion corpus, elicits attack judgments through Crowdflower, and combines representative and attack-enriched sampling. Quality controls and repeated judgments support reliable labels for classifier training.
- Crowdsourcing Process: The crowdsourcing process comprises corpus generation, question selection, subset sampling, and a strategy for eliciting reliable labels.
- Corpus Construction: The corpus contains 63M comments from English Wikipedia user-talk and article-talk discussions dating from 2004–2015.
- Human Judgments: Annotators judged whether comments contained personal attacks, with additional questions about targets and quoted attacks excluded from this study.The final question wording was selected after testing variants for inter-annotator agreement on 1,000 comments.
- Sampling: Random sampling estimates overall prevalence, while comments surrounding blocks for personal attacks enrich the training data with more attacks.The random dataset has approximately 1% attacks, compared with approximately 17% in the blocked dataset.
- Label Quality: Each comment received 10 labels, with majority annotation defining an attack in the labeled-data summary.The study used Crowdflower and applied quality filtering and annotation cleaning after collection.
- Label Quality: Krippendorff’s alpha was used to evaluate inter-annotator agreement for overlapping but disparate comment assignments.
4. MODEL BUILDING
The paper builds personal-attack classifiers from crowdsourced labels, compares architectures and label representations, and evaluates whether machine predictions approximate aggregated human judgments. Character n-grams and empirical-distribution labels consistently improve performance, while the best model matches an ensemble of three annotators.
- Model design: The classifiers use comment-text features only, making their strengths and weaknesses inspectable through generated text examples and allowing application beyond Wikipedia.The models exclude authors’ past behavior and discussion context.
- Model design: The study compares logistic regression and multilayer perceptrons with word- or character-level n-grams and one-hot or empirical-distribution labels.These are the three major design dimensions, yielding eight modeling strategies.
- Label representation: Empirical-distribution labels encode the fraction of annotators judging a comment to be an attack, preserving distinctions that majority-class labels discard.For example, seven of ten attack judgments can produce a label of [0.3, 0.7] rather than [0,1].
- Evaluation: Character n-grams outperform word n-grams across model and label types, possibly because they are more robust to spelling variations common in online discussions.The authors specifically connect this robustness to expletives often used in personal attacks.
- Evaluation: Empirical-distribution labels consistently boost performance metrics over one-hot labels, including AUC for predicting the majority annotator judgment.They also provide supervision for estimating the fraction of annotators who consider a comment an attack.
- Human baseline comparison: The model outperforms an annotator ensemble of size np = 3 on both AUC and Spearman correlation, making full-history scoring equivalent to labeling each comment by three annotators on these metrics.The human-baseline comparison fixes the truth-group size at 10 and varies the prediction-group size.
5. ANALYSIS
The analysis calibrates and validates a classifier threshold before using machine labels to examine personal attacks across Wikipedia comments. It then shows how attacks vary by anonymity, user activity, toxicity, timing, and moderation.
- The study analyzes 2015 Wikipedia discussions using machine-labeled comments after excluding administrative and bot-generated comments.
- 5.1 Choosing a Threshold: The equal-error threshold balances precision and recall so classifier-labeled attack fractions match human-annotated fractions in aggregate.
- 5.1 Choosing a Threshold: The estimated attack rate from thresholded model labels falls within the 95% confidence interval of the crowd-labeled rate on the test set.
- 5.2 Understanding Attacks: Anonymous comments are six times more likely to be attacks than registered users’ comments, yet account for less than half of all attacks.
- 5.2 Understanding Attacks: Almost half of attacks come from users with fewer than 5 comments, while users with over 100 comments account for 30% of attacks.
- 5.2 Understanding Attacks: Almost 80% of attacks come from over 9,000 users with fewer than 5 attacking comments, while 34 users with toxicity above 20 account for almost 9%.
- 5.2 Understanding Attacks: 11.3% of attacks are followed by either a warning or block within 7 days, while precision-normalized estimates reach 17.9% for expected true attacks.
- 5.2 Understanding Attacks: For n = 1, the neighboring attack fraction is twenty-two times higher around attacking than non-attacking comments, indicating temporal clustering.
6. DISCUSSION & CONCLUSION
The paper demonstrates a scalable methodology for studying personal attacks and applies it to reveal distributional patterns, moderation gaps, and temporal clustering in Wikipedia discussions. It also identifies methodological challenges and broader research uses for the resulting datasets.
- 100k+ human-labeled and 63M machine-labeled comments support large-scale analysis of Wikipedia personal attacks.The classifier approximates the aggregate of 3 crowd-workers, enabling analysis beyond the smaller crowdsourced sample.
- Approximately 30% of attacks come from registered users with over 100 contributions, while less than half come from users with little prior participation.
- Anonymous comments are 6 times more likely to be attacks but contribute less than half of all attacks.
- Less than a fifth of personal attacks trigger action for violating Wikipedia’s policy, and attacks cluster in time.The authors suggest that early moderator intervention could therefore have a disproportionately beneficial impact.
- The methodology is limited by a relatively small, poorly characterized annotator pool and possible unintended bias in crowdsourced data.Annotators’ interpretations may differ from those of the Wikipedia community.
- The human- and machine-labeled corpora support further machine-learning, statistical, and large-scale longitudinal research across other comment properties and discussion corpora.
A. WIKIPEDIA COMMENT CORPUS
The corpus is constructed from English Wikipedia talk-page revision histories rather than snapshots, using edit diffs to recover added discussion content. The process defines comments from MediaWiki markup and filters administrative material, while excluding revisions removed from the public record.
- The corpus focuses on user-talk and article-talk namespaces among Wikipedia’s 35 talk namespaces.
- Revision-history diffs capture content added across edits, avoiding the incompleteness of snapshots that omit quickly removed attacks and read user-talk comments.
- Comments introduced on revisions removed from Wikipedia’s public record are absent from the corpus.Personal attacks are a valid reason for removing revisions.
- A talk-page comment is defined as the MediaWiki markup added during an edit, with a clean plaintext version also computed.
- 20%-50% of talk-page comments, depending on namespace, were administrative, so bot and template-based messages were filtered out.The study targets comments made by human users in discussion contexts.