PaperlayerView exact revision on arXiv

Source-linked AI summary

Watch Your Language: Investigating Content Moderation with Large Language Models

Deepak Kumar, Yousef AbuHashem, Zakir Durumeric

arXiv:2309.14517v2cs.HCcs.AIcs.CLcs.CRcs.SI

TL;DR

This paper studies LLM performance in content moderation, where recent interest has outpaced research evidence. It evaluates LLMs on community-rule moderation and toxicity detection, finding useful performance but also substantial variation and only marginal toxicity gains from newer, larger models.

  • Problem

    Despite enthusiasm for LLM-powered content moderation, research evidence on how LLMs perform in moderation settings remains limited.

    Source: Introduction · p. 1

  • Method

    The authors evaluate commodity LLMs on rule-based moderation across 95 Reddit subcommunities and on toxicity detection using a balanced comment dataset.

    Source: Introduction · p. 1

  • Results

    LLMs outperform commercial toxicity classifiers, while toxicity-detection performance improves only marginally across newer models; GPT-3.5 has median accuracy of 64% and precision of 83% for community moderation.

    Source: Introduction · pp. 1–2

  • Takeaways & Limitations

    The findings caution that increasing model size does not necessarily improve moderation performance and point to continued research on challenging toxicity cases.

    Source: Discussion · p. 10

    The study focuses on text-based comment moderation on discussion-oriented platforms, primarily evaluates balanced datasets, and finds tested LLMs too expensive for immediate use in most moderation contexts.

    Source: Discussion · p. 10

Abstract

from arXiv · show

Large language models (LLMs) have exploded in popularity due to their ability to perform a wide array of natural language tasks. Text-based content moderation is one LLM use case that has received recent enthusiasm, however, there is little research investigating how LLMs perform in content moderation settings. In this work, we evaluate a suite of commodity LLMs on two common content moderation tasks: rule-based community moderation and toxic content detection. For rule-based community moderation, we instantiate 95 subcommunity specific LLMs by prompting GPT-3.5 with rules from 95 Reddit subcommunities. We find that GPT-3.5 is effective at rule-based moderation for many communities, achieving a median accuracy of 64% and a median precision of 83%. For toxicity detection, we evaluate a suite of commodity LLMs (GPT-3, GPT-3.5, GPT-4, Gemini Pro, LLAMA 2) and show that LLMs significantly outperform currently widespread toxicity classifiers. However, recent increases in model size add only marginal benefit to toxicity detection, suggesting a potential performance plateau for LLMs on toxicity detection tasks. We conclude by outlining avenues for future work in studying LLMs and content moderation.

Introduction

The paper examines whether LLMs can support content moderation, a use case attracting attention despite limited research on their performance. Across community-rule moderation and toxicity detection, it reports promising results alongside important qualifications.

Introduction

  • The study evaluates rule-based community moderation and toxic content detection using test harnesses that compare commodity LLMs with available ground truth.For community moderation, it prompts 95 GPT-3.5 models with Reddit subcommunity guidelines; for toxicity detection, it compares GPT-3, GPT-3.5, GPT-4, Gemini Pro, and LLAMA 2 on a balanced dataset.Source: Introduction · p. 1
  • GPT-3.5 reaches median accuracy of 64% and median precision of 83% for rule-based moderation across subreddits.Performance approaches human-moderator levels in some communities, while the paper also reports substantial differences when underlying models change over time.Source: Introduction · pp. 1–2
  • LLMs outperform commercial toxicity classifiers, but larger and newer models bring only marginal gains on toxicity detection.GPT-3.5 has the most balanced performance among tested models, with Accuracy of 0.73 and F1 of 0.75; models can miss implicit toxicity or lack needed world context.Source: Introduction · p. 2
  • The authors frame the findings as an early step and say LLMs still require human review and careful guardrails before widespread adoption.They describe the results as useful to researchers and practitioners studying how to incorporate LLMs into moderation pipelines.Source: Introduction · p. 2

Background and Related Work

Prior work examines AI-assisted moderation workflows and transparency, as well as systems that help moderators identify candidate content. Research on toxicity detection includes classifiers, older LLMs, and newer prompted models.

AI-Assisted Content Moderation

  • Existing moderation research covers AI-assisted workflows, transparency, and systems that surface potential moderation candidates for human moderators.The paper builds on this work by studying LLM enforcement of written community guidelines and explanations of moderation decisions.Source: AI-Assisted Content Moderation · p. 2
  • Prior studies report that transparency through explanations can yield prosocial outcomes, while scale challenges can limit widespread adoption of explanations.The cited system Crossmod surfaced candidates based on outcomes in other subcommunities and showed significant alignment with human-moderator decisions.Source: AI-Assisted Content Moderation · p. 2

Detecting Toxic Content

  • Toxicity-detection research includes Perspective API, improved classifiers, and studies using older LLMs or GPT-3.5 for hate-speech detection.Examples include a DeBERTa classifier that marginally outperforms Perspective API and work prompting local LLMs for toxicity-related tasks.Source: Detecting Toxic Content · p. 2

Rule-Based Content Moderation

The paper evaluates subreddit-specific LLM moderation by comparing decisions against human-moderator labels, then examines aggregate performance and prompt sensitivity across communities.

Rule-Based Content Moderation

Collecting ground-truth moderation decisions

  • The ground-truth dataset uses 500 moderated and 500 unmoderated comments from each of 95 subreddits, sampled from a 2016–2017 dataset.Five original subreddits were no longer available, so the study covers the remaining 95.Source: Collecting ground-truth moderation… · pp. 2–3

Extracting Subreddit metadata

  • The authors extract subreddit descriptions and comment-applicable rules from Reddit’s APIs to construct the guideline prompts.Rules that do not apply to comments are filtered out before the rules are numbered for prompting.Source: Extracting Subreddit metadata · p. 3

Prompt design

  • Each zero-shot subreddit prompt includes the community description and rules, then requests a moderation decision, violated rules, rating, and explanation in JSON.The returned fields support consistent downstream analysis.Source: Prompt design · p. 3

Evaluation strategy

  • The main evaluation compares subreddit-specific GPT-3.5 decisions with human decisions on binary performance metrics; the study uses temperature 0 for repeatability.GPT-3.5 was selected to balance at-scale measurement of 100,000 queries with cost.Source: Evaluation strategy · p. 3

Aggregate Performance

  • GPT-3.5 achieves median accuracy of 63.7%, median precision of 83%, and median recall of 39.8% across subreddits, with performance ranging from below chance to high accuracy.It reaches 81.5% accuracy in r/movies but only 38.5% in r/askscience; expert-oriented communities may require world context that the setup lacks. Figure 1 summarizes accuracy, precision, and recall distributions across 95 subreddits.Source: Aggregate Performance · p. 3, Figure 1 · p. 3

Prompt Sensitivity

  • Overall performance is near identical across the three tested prompt strategies, though Reddiquette and chain-of-thought prompting generally shift the precision–recall tradeoff.For r/askscience, either added context or chain-of-thought increased precision from 0.87 to 0.95 without changing recall; Table 1 summarizes the prompt-strategy comparison.Source: Prompt Sensitivity · p. 4, Table 1 · p. 4

Errors in Rule-Based Moderation

In real-time moderation, subreddit-specific GPT-3.5 models performed worse on average than in the balanced task, while model updates shifted the precision–recall balance. The analyses also show variation in errors and rule expression across communities and rule types.

Errors in Rule-Based Moderation

  • False negatives made up 86.9% of GPT-3.5’s errors overall, but in r/NSFW GIF, false positives were 68% and false negatives 31%.The authors attribute the community-specific reversal partly to missing context or flagging sexual content that the community permits.Source: Errors in Rule-Based Moderation · p. 4

Stability of LLM Performance Over Time

  • Across GPT-3.5 releases, median F1 fell from 0.68 to 0.53 and recall from 0.63 to 0.40, while precision rose from 0.75 to 0.83.The comparison used the March 1 and June 13, 2023 models with prompts and analyzed comments held constant; the authors say moderators should consider model stability when deploying these tools.Source: Stability of LLM Performance Over Time · p. 4, Table 2 · p. 4

LLMs and Rule Enforcement

  • GPT-3.5 expressed 467 of 562 rules (83.1%), and expressed restrictive rules more often (87%) than prescriptive (77%) or format-related rules (71%).The differences between restrictive rules and each of the other two types were statistically significant (p < 0.01).Source: Table 3 · p. 5, LLMs and Rule Enforcement · p. 5
  • False-positive error rates were similar across restrictive rules (18.5%), prescriptive rules (15.9%), and formatting rules (16.9%).The authors report little evidence that rule type affects error rates.Source: LLMs and Rule Enforcement · p. 5

Moderating Subreddits in Realtime

  • Average ROC AUC fell from 0.67 to 0.6 in real-time moderation; recall fell from 43% to 29%, and precision from 83% to 6.2%.Accuracy rose from 70% to 90%, largely because unmoderated comments created significant class imbalance; the authors caution that real-time Reddit moderation decisions can be noisy.Source: Moderating Subreddits in Realtime · p. 5

Case Study: r/worldnews

A qualitative analysis of GPT-3.5’s r/worldnews errors found both cases where raters favored the model and failures linked to missed context. Adding parent comments corrected some real-time errors, while Gemini Pro performed similarly on the tested subreddits.

Case Study: r/worldnews

  • For r/worldnews moderation, raters agreed with GPT-3.5’s decisions in 48% of false positives and 16% of false negatives; 49% of false negatives involved missed personal or identity attacks.Most errors were attributed to missing context from the conversation, post, or shared world knowledge.Source: Table 4 · p. 6, Case Study: r/worldnews · p. 6

Does Conversational Context Reduce Errors?

  • Adding a parent comment corrected 35% of real-time errors, with more false positives corrected (40%) than false negatives (6%).The study used 4,131 parent–child comment pairs drawn from a corpus of 8,859 real-time errors.Source: Does Conversational Context Reduce… · p. 6

Case Study: Comparison to Gemini Pro

Toxicity Detection

The study evaluates commodity LLMs for toxicity detection against commercial classifiers, using comment ratings and prompts to test aggregate performance, thresholding, and errors. The LLMs generally outperform Perspective, but results also show prompt sensitivity, persistent error patterns, and limits in the Llama 2 case study.

Dataset

  • The 107K-comment dataset was sampled into 5,000 toxic and 5,000 nontoxic examples, with toxicity defined by the median of five raters’ scores.The prompts supplied Google Jigsaw’s toxicity definition and asked models to rate comments from 1–10 with a natural-language explanation.Source: Dataset · p. 7

Aggregate Performance

  • LLMs beat Perspective on nearly all metrics: GPT models reached 0.71–0.73 accuracy versus Perspective TOXICITY’s maximum 0.66, while LLM F1 was 0.72–0.75 versus 0.64.GPT-3.5 and GPT-4 achieved the highest accuracy and F1 scores, 0.75; GPT-3’s F1 was 0.72.Source: Aggregate Performance · p. 7, Table 6 · p. 8

Prompt Sensitivity

  • Without a toxicity definition, GPT-3.5 precision rose to 0.85 from 0.69, but recall fell from 0.84 to 0.12; chain-of-thought results matched the base prompt.The prompt comparison used a 1,000-comment subsample.Source: Prompt Sensitivity · p. 7, Table 7 · p. 8

Thresholding LLMs for Toxicity Detection

  • 0.77–0.81 ROC AUC for GPT-3.5, GPT-4, and Gemini Pro exceeded Perspective’s 0.76; at the highest threshold, GPT-4 achieved 0.94 precision and 0.25 recall.At that threshold, Perspective achieved 0.96 precision and 0.02 recall; GPT-3.5 and Gemini Pro achieved F1 of 0.49–0.53.Source: Figure 2 · p. 8, Thresholding LLMs for Toxicity Detection · p. 8

Do LLM Toxicity Ratings Align with Human Raters?

  • GPT-3.5 and GPT-4 toxicity ratings correlated strongly with human ratings (r = 0.60 and r = 0.61), compared with Gemini Pro’s weaker correlation (r = 0.5).The comparison correlated model ratings from 1–10 with the median of five human ratings from 1–5.Source: Do LLM Toxicity Ratings Align with… · pp. 8–9

Where Do LLMs Make Mistakes?

  • GPT-4 false positives often came from misreading offensive language or identifying harassment targets incorrectly, while false negatives included implicit toxicity, sarcasm, and opinions.The reported false-positive categories were 34% for poor language or stereotypes and 47% for misidentified targets; false-negative categories included 27% without direct threats or insults, 23% sarcasm or humor, and 22% opinions.Source: Where Do LLMs Make Mistakes? · p. 9, Table 9 · p. 9

Case Study: Comparison to Llama 2

  • Only 32% of Llama 2 responses were valid and parseable JSON; among valid responses, its maximum F1 was 0.68, with 63% accuracy and 0.59 precision.The 13B model can run locally, but the authors found its results significantly worse than those of the other evaluated LLMs.Source: Case Study: Comparison to Llama 2 · p. 10

Discussion

The discussion identifies performance gaps and practical constraints for LLM moderation, alongside directions for improving context, monitoring model changes, and supporting moderators.

Increasing transparency in LLM decisions

  • Model performance can change rapidly, leaving the causes of moderation-decision changes unclear and motivating continuous monitoring benchmarks.The authors advise caution to maintain reliability and stability when deploying these tools.Source: Discussion · p. 10

Improving rule-based moderation with behavior and context

  • CrossMod reached 86% accuracy versus GPT-3.5’s 64% on the same dataset, with the authors contrasting learned removal patterns against forward rule-based reasoning.They suggest combining rule-based reasoning with few-shot behavioral examples as a possible improvement.Source: Discussion · p. 10
  • Adding conversational or world context corrected 35% of errors with little prompt addition, motivating tests of post details and multimodal context.Source: Discussion · p. 10

Performance saturation in toxicity detection

  • Toxicity performance improved only marginally across much larger LLMs, cautioning against assuming that model scaling will improve moderation results.The authors identify implicit hate and identity language as challenges and suggest specialized pipelines or human review for gray-area comments.Source: Discussion · p. 10

Future experiments in content moderation

  • LLMs may help moderators find potentially violative content, although the evaluation found that 48% of false positives seemed to violate community rules.The authors propose collaborative studies of LLMs within existing moderation workflows.Source: Discussion · p. 10

Limitations of our work

  • The evaluation centers on text-based comment moderation and balanced datasets, so results may not extend to multimodal or real-world settings; tested LLMs are also costly.The authors call for research balancing performance with cost.Source: Discussion · p. 10

Conclusion

The authors present a tempered but optimistic case for LLMs in content moderation and position this work as groundwork for future research.

  • The authors frame their results as a tempered but optimistic view of introducing LLMs into content moderation.Source: Conclusion · p. 11
  • The work is offered as groundwork for researchers and practitioners and as a pathway for further study of LLMs in content moderation.Source: Conclusion · p. 11
  • The authors plan to release code, prompts, and data at publication to enable further research on automated content moderation with LLMs.Source: Conclusion · p. 11

Rule Type Classifier Details

The authors replicate rule-type classifiers using logistic regression and report comparable results to the prior paper.

Rule Type Classifier Details

  • Three logistic regression classifiers identify restrictive, prescriptive, and format-related rules, the rule types selected because they were abundant in the training data.Source: Rule Type Classifier Details · p. 12
  • The classifiers use unigram, bigram, and trigram features, with 20% of rules held out for testing using random seed 42.Source: Rule Type Classifier Details · p. 12
  • The restrictive, prescriptive, and format models achieved F1 scores of 0.93, 0.89, and 0.79, respectively, comparable to the original paper.Source: Rule Type Classifier Details · p. 12

Conversational Context Prompt

The prompt presents a two-comment tree and asks for a structured moderation response about the second comment.

Conversational Context Prompt

Cost and Performance of LLMs for Moderation

The paper compares model costs and overall performance, including experiment time, while cautioning that time estimates are difficult to reproduce.

Cost and Performance of LLMs for Moderation

Related papers

Loading 2309.14517v2…