Source-linked AI summary
Toxicity Detection: Does Context Really Matter?
John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, Ion Androutsopoulos
TL;DR
Toxicity datasets often omit conversational context, leaving unclear whether context changes human judgments or improves automated detection. The authors study Wikipedia discussions using datasets with parent comments and discussion titles, finding that context changes a small but significant subset of labels but does not improve classifier performance. They conclude that larger corpora annotated in context are needed, while noting that their context definition and random sampling limit the scope of the findings.
Problem
Most toxicity datasets omit post context and context-aware annotations, leaving the effects of context on human judgments and classifier performance insufficiently addressed.
Method
The authors create Wikipedia Talk Page datasets and compare annotations and toxicity classifiers with versus without the previous comment and discussion title.
Results
Context changes toxicity labels for 5.2% of CAT-SMALL posts, but the authors find no evidence that context improves classifier performance across tested classifiers and mechanisms.
Takeaways & Limitations
The findings support studying toxicity with context while prioritizing larger datasets whose comments are annotated in context.
Takeaways & Limitations
The study considers only the previous comment and discussion title and uses randomly sampled comments, so broader contexts or topic-specific conversations may yield different effects.
Abstract
from arXiv · showhide
Moderation is crucial to promoting healthy on-line discussions. Although several `toxicity' detection datasets and models have been published, most of them ignore the context of the posts, implicitly assuming that comments maybe judged independently. We investigate this assumption by focusing on two questions: (a) does context affect the human judgement, and (b) does conditioning on context improve performance of toxicity detection systems? We experiment with Wikipedia conversations, limiting the notion of context to the previous post in the thread and the discussion title. We find that context can both amplify or mitigate the perceived toxicity of posts. Moreover, a small but significant subset of manually labeled posts (5% in one of our experiments) end up having the opposite toxicity labels if the annotators are not provided with context. Surprisingly, we also find no evidence that context actually improves the performance of toxicity classifiers, having tried a range of classifiers and mechanisms to make them context aware. This points to the need for larger datasets of comments annotated in context. We make our code and data publicly available.
1 Introduction
Existing toxicity datasets generally omit conversational context, motivating this study of whether context changes human judgments and improves toxicity classifiers. The authors create context-aware Wikipedia datasets and find that context changes a small but significant subset of labels, while not improving classifier performance.
- Motivation: Most toxicity datasets omit preceding comments and context-aware annotations, so systems commonly judge comments independently.A comment’s meaning and perceived toxicity may depend on its parent comment.
- Research questions: RQ1 asks how often context changes human perceptions of toxicity and whether it amplifies or mitigates those perceptions.
- Research questions: RQ2 asks whether context improves toxicity-classifier performance and how classifiers can be made context-aware.
- Contributions: 5.2% of CAT-SMALL posts changed perceived toxicity labels when annotators were given the parent comment and discussion title.Context amplified toxicity for approximately 3.6% of comments and mitigated it for approximately 1.6%.
- Contributions: 20k comments were collected for classifier experiments, split into 10k annotated out of context and 10k annotated in context.The subsets trained context-aware and context-unaware classifiers evaluated on held-out comments annotated in context.
- Contributions: Across tested classifiers and context mechanisms, the authors found no evidence that context improves toxicity-classification performance.The paper suggests this may relate to the small number of context-sensitive comments.
2 Related Work
Prior toxicity datasets and studies often lack usable conversational context or context-aware annotations. This paper positions its datasets as addressing a gap in which systems cannot reliably exploit linked parent comments during training.
- Terminology: Toxicity is used as an umbrella term covering related phenomena such as offensive, abusive, and hateful language.Prior work also distinguishes these phenomena by directness and target.
- Existing datasets: No existing English dataset was identified as providing both context and context-aware annotations.This limits direct study of how linked conversational context affects toxicity labels and model training.
- Existing datasets: Most existing datasets do not provide annotators with preceding comments, discussion titles, or discussion topics during labeling.Gao and Huang (2017) is identified as an exception, but its dataset has limited size and incomplete reconstruction information.
- Existing datasets: Only two known datasets included context-aware labels, and their context was limited to news-article titles.Other shortcomings included unavailable raw text, missing links between targets and parents, or single-annotator decisions.
- Context in NLP: Earlier work commonly used preceding text in language modeling and text classification, including encoded sentence histories and concatenated conversational utterances.Reported gains in prior conversational-response work reached up to 11% BLEU.
3 Experiments
The experiments test whether context changes human toxicity judgments and improves toxicity classifiers, using Wikipedia comments annotated with the parent comment and discussion title. Context affected perceived toxicity, but context-aware classifiers showed no consistent or large performance gains.
- RQ1: CAT-SMALL compared toxicity judgments from annotators given the parent comment and discussion title with judgments made without context.The dataset contained 250 randomly selected Wikipedia Talk Page comments, annotated by two groups.
- RQ1: 2 percentage points: CAT-SMALL’s toxicity ratio increased from 4.4% without context to 6.4% with context.The increase was statistically significant, although aggregate ratios may hide comments moving in opposite directions.
- RQ1: 5.2%: 13 of 250 CAT-SMALL comments received different toxicity labels across the no-context and context groups.Toxicity decreased for 4 comments (1.6%) and increased for 9 comments (3.6%).
- RQ1: CAT-SMALL cannot accurately estimate the frequency of context-sensitive posts because its sample is small.The authors therefore call for larger annotated datasets to estimate how often context amplifies or mitigates toxicity.
- RQ2: CAT-LARGE used 20k comments: 10k annotated without context and 10k other comments annotated with context, each by three workers.The context condition included the parent comment and discussion title; comments were selected from Wikipedia Talk Pages.
- RQ2: No consistent or large improvement appeared when classifiers incorporated context through concatenation or context-aware BILSTM and BERT mechanisms.This held across simple and more powerful classifiers and across models trained with context-derived or no-context gold labels.
4 Conclusions and Future Work
The study finds that context significantly affects toxicity annotation for a narrow subset of comments, but does not improve classifier performance in the collected data. It used two context-aware datasets and identifies limited context and random sampling as boundaries for future work.
- 5.2% of comments in the first dataset showed a statistically significant effect of context on toxicity annotation.
- The datasets support investigation of context effects on both human annotation and automated toxicity detection.
- Context did not improve toxicity-classifier performance across simple and more powerful classifiers or multiple context mechanisms.
- The study considered only the previous comment and discussion title as context, leaving broader thread context for future work.
- Random comment sampling may understate context effects in topic-specific, sarcastic, or abuse-targeted conversations.