Source-linked AI summary
Deep Learning for User Comment Moderation
John Pavlopoulos, Prodromos Malakasiotis, Ion Androutsopoulos
TL;DR
Abusive comments strain moderation, while fully automatic decisions can be difficult and moderator-assisted moderation had not been studied in previous work. The paper evaluates deep-learning models on Greek news-portal and English Wikipedia comments, including an RNN with classification-specific attention. The RNN outperforms the prior system across both datasets and moderation scenarios, and attention improves its overall results.
Problem
Abusive comments can damage news portals, and moderator workloads can delay non-abusive comments; previous work had not considered moderator-assisted moderation.
Source: §1 · p. 1Method
The paper evaluates an RNN with classification-specific attention, alongside CNN and word-list baselines, on Gazzetta and Wikipedia comments in automatic and semi-automatic moderation.
Source: §1 · pp. 1–2, §6 · p. 9Results
Takeaways & Limitations
The attention mechanism identifies comment words that most affect the accept-or-reject decision by examining them in context.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Experimenting with a new dataset of 1.6M user comments from a Greek news portal and existing datasets of English Wikipedia comments, we show that an RNN outperforms the previous state of the art in moderation. A deep, classification-specific attention mechanism improves further the overall performance of the RNN. We also compare against a CNN and a word-list baseline, considering both fully automatic and semi-automatic moderation.
1 Introduction
The paper applies deep learning to abusive-comment moderation, introducing a Greek news-portal dataset and considering both automatic and moderator-assisted review. It reports that an RNN with classification-specific attention outperforms the previous state of the art.
1 Introduction
- Semi-automatic moderation sends uncertain comments to a moderator while the system accepts or rejects the others, addressing a setting not considered in previous work.The paper discusses tuning systems according to moderator availability and workload.Source: §1 · p. 1
- On Gazzetta and Wikipedia comments in both moderation scenarios, the RNN outperforms the previous state of the art, and classification-specific attention further improves overall RNN performance.The previous system used logistic regression or a multi-layer perceptron.Source: §1 · p. 2
- The attention mechanism examines each word in its comment context to identify words that most affect the accept-or-reject classification decision.Source: §1 · p. 2
- The paper releases 1.6M moderated comments from a Greek sports portal and provides word embeddings pre-trained on 5.2M comments from that portal.The new dataset is publicly available.Source: §1 · pp. 1–2
2 Datasets
The paper uses Greek news-portal comments and English Wikipedia comments to study moderation across different datasets and labeling settings. Gazzetta supplies the new portal dataset, while the Wikipedia datasets cover personal attacks and toxicity.
2.1 Gazzetta dataset
- The Gazzetta dataset includes about 1.45M training comments and separate development and test sets, including a small test set re-moderated to reflect normal conditions.The training comments span Jan. 1, 2015 to Oct. 6, 2016; 100K of them form G-TRAIN-S. The 60,900 later comments were split into G-DEV (29,700), G-TEST-L (29,700), and G-TEST-S (1,500); G-TEST-S-R has a 22% reject ratio versus approximately 30% in the other subsets.Source: §2.1 · p. 2
- Five annotators re-moderated G-TEST-S-R, with Krippendorff’s alpha of 0.4762 and mean pairwise agreement of 81.33%.The passage attributes moderate chance-corrected agreement scores partly to the 22% reject and 78% accept class balance.Source: §2.1 · p. 2
2.2 Wikipedia datasets
- The Wikipedia datasets contain crowdsourced personal-attack labels and toxicity labels, with binary or probabilistic gold labels and task-specific train, development, and test splits.The attacks set contains about 115K comments; the toxicity set contains 159,686. Rejected comments are 12% in each attacks split and 10% in each toxicity split, ratios described as artificial.Source: §2.2 · pp. 2–3
- Wikipedia comments are longer than Gazzetta comments, and the paper excludes the aggression dataset because its labels closely correlate with the personal-attacks labels.Median lengths are 38 and 39 tokens for attacks and toxicity, versus 25 for Gazzetta; the two Wikipedia label sets have Spearman correlation 0.8992 and Pearson correlation 0.9718.Source: §2.2 · p. 3
3 Methods
The methods compare recurrent, convolutional, and word-list classifiers with DETOX, then tune thresholds for fully automatic and semi-automatic moderation. The RNN attention variants use different ways of weighting or representing comment words.
3 Methods
- The experiments compare an RNN, its attention variants, a CNN, DETOX, and a word-list baseline.The methods are evaluated for comment moderation.Source: §3 · p. 3
3.1 DETOX
- DETOX represents comments with word or character n-grams and uses logistic regression or an MLP; tuning selected character n-grams with logistic regression.Probabilistic labels were selected when available for Wikipedia datasets, while Gazzetta training data allowed only binary labels.Source: §3.1 · p. 3
3.2 RNN-based methods
- The RNN encodes comment tokens with GRU cells, and its attention variant classifies from a weighted sum of hidden states rather than only the final state.Attention weights are produced by a multilayer network and emphasize hidden states at positions with evidence for accepting or rejecting the comment; tuning selected four layers.Source: §3.2 · pp. 3–4
- Variants test attention on embeddings rather than contextual hidden states, equal attention, and representations that remove the RNN, attention, or both.In da-RNN, embeddings receive attention scores while hidden states remain the pooled representation; eq-RNN averages hidden states, while da-CENT and eq-CENT use embedding averages.Source: §3.2 · pp. 4–5
3.3 CNN
- The CNN applies narrow convolutions to word n-grams and max-pools kernel outputs; Gazzetta uses n-grams of sizes 1–5, yielding 1,500 features.For Wikipedia, n-gram sizes 1–4 use 300 kernels per size.Source: §3.3 · p. 5
3.4 LIST baseline
- LIST records training-set word rejection precision and assigns each comment the maximum precision among its words.Words must occur in more than 10 comments for some training sets or more than 100 for G-TRAIN-L to enter the list.Source: §3.4 · p. 5
3.5 Tuning thresholds
- For semi-automatic moderation, coverage sets the share handled automatically, while tuned acceptance and rejection thresholds leave the remainder for moderators.Threshold pairs are selected on development data to maximize Fβ; β = 2 emphasizes acceptance precision because wrongly accepting abusive comments is a greater concern than wrongly rejecting non-abusive ones.Source: §3.5 · pp. 5–6
4 Experimental results
Across the reported comparisons, RNNs outperform CNN and DETOX, while attention improves the RNN on Gazzetta but not consistently on Wikipedia. Results also show how coverage and training-set size affect moderation performance.
4 Experimental results
- Increasing Gazzetta training data from G-TRAIN-S to G-TRAIN-L significantly improves every method; DETOX results are omitted because its implementation could not handle G-TRAIN-L's size.Figure 4 also reports results for the larger training set with dashed lines.Source: §4 · p. 6
- RNN outperforms CNN and DETOX; a-RNN beats RNN on Gazzetta, but not always on Wikipedia.The tables report Gazzetta and Wikipedia results separately.Source: Table 2 · p. 6, §4 · p. 6, Table 3 · p. 7
- At 50% coverage, a-RNN obtains Paccept, Preject ≥0.85 on Gazzetta; at 100%, it obtains Paccept ≥0.82 and Preject ≥0.56.F2 threshold tuning emphasizes avoiding wrongly accepted comments, at the expense of wrongly rejected comments.Source: §4 · pp. 6–7
- On W-ATT-TEST and W-TOX-TEST, a-RNN obtains Paccept, Preject ≥0.94 for all coverages.Figure 4 reports F2 scores for varying coverage.Source: §4 · pp. 6–7
5 Related work
Related work spans abusive-content classifiers, large-scale moderation datasets, and methods for assessing conversation quality. Some adjacent datasets exclude abusive comments or target different labels and tasks.
5 Related work
- Some conversation-quality datasets are unsuitable for abusive-comment moderation: YNACC filtered abusive comments, while the insult dataset misses abuse not directly aimed at discussion participants.Related work also describes thread-quality methods using comment embeddings, CRFs, and thread-level features.Source: §5 · pp. 7, 9
- Large moderated-comment studies used DOC2VEC with logistic regression or combined n-gram, embedding, dependency, and hand-crafted features; the cited Yahoo datasets were not available.Nobata et al. found character n-grams strongest alone and all features best overall; the paper contrasts its own use of no hand-crafted features or parsers.Source: §5 · pp. 7–8
- Wikipedia moderation work introduced DETOX, which was later used to label 63M comments as a proxy for human annotators; the paper proposes its better-performing methods as possible replacements.The cited prior study examined personal attacks and moderation-related questions.Source: §5 · p. 8
- Prior abusive-content work covered nastiness, sarcasm, profanity, harassment, hate speech, cyberbullying, and insults using varied classifiers and features.The described approaches include bootstrapping, n-grams, topical features, sentiment, and context.Source: §5 · pp. 8–9
6 Conclusions
The paper evaluates RNN-based moderation on a new Greek dataset and English Wikipedia comments, finding gains over prior and alternative methods. It also considers both automatic and moderator-assisted settings and outlines directions for further work.
6 Conclusions
- A GRU RNN on word embeddings outperforms the previous state of the art, a vanilla CNN, and an automatically constructed word-list baseline on the evaluated datasets.The new Greek sports-news dataset contains 1.6M moderated comments and is publicly available.Source: §6 · p. 9
- Deep classification-specific attention further improves the RNN overall and also improves a simpler method that averages word embeddings.The paper evaluates fully automatic and semi-automatic moderation, including threshold tuning and evaluation measures for both.Source: §6 · p. 9
- Planned extensions include user and thread information, character-level RNNs or CNNs, and using a-RNN attention scores to highlight suspicious words or phrases for moderators.Examples include past rejection ratios, thread depth, and revisiting-user counts.Source: §6 · p. 9