Designing Toxic Content Classification for a Diversity of Perspectives

• Users may be harmed by content that does not violate platform policy, while lived experience, cultural perspective, political views, and access to context shape toxicity judgments.Prior work had identified groups at greater risk of online harassment, but had not investigated how diverse users interpret toxicity.Source: §1 · p. 1
• A survey of 17,280 participants asked them to rate 20 random comments from Twitter, Reddit, and 4chan, alongside demographic and harassment-experience questions.The study then examined aggregate ratings, factors associated with divergent ratings, and classifier tuning.Source: §1 · p. 1
• Participants disagreed about 85% of comments, including whether they were comfortable seeing them online and which toxic-content subcategory applied.In aggregate, 53% were labeled not toxic, 39% slightly or moderately toxic, and 8% very or extremely toxic.Source: §1 · p. 1
• Personal harassment experience, at-risk group membership, and attitudes toward filtering online discourse all correlated with rating comments as toxic.Holding other factors constant, identifying as LGBTQ+ increased the odds of rating a comment toxic 1.64 times; regularly witnessing others targeted was associated with odds decreasing by 0.78 times.Source: §1 · pp. 1–2
• 86% improvement in accuracy per individual and 22% per demographic cohort were achieved through personalized tuning of existing classifiers.The study evaluates Jigsaw’s Perspective API and Instagram’s comment nudge.Source: §1 · p. 2
• The authors shared their findings with Jigsaw and released their labeled dataset to support reproducibility and further research.Source: §1 · p. 2

2.1 What is toxic content?

• The paper uses toxic content as an umbrella for identity-based attacks, bullying, trolling, threats, and sexual harassment, focusing on text while noting attacks may also use images and video.It distinguishes these examples from broader abuse that can include stalking, doxxing, sextortion, and intimate partner violence.Source: §2.1 · p. 2
• Prior U.S. studies report different harassment risks across demographic groups, while detailed global demographic studies are not yet available.The paper argues that understanding how people perceive toxic content is necessary for detection that works for at-risk groups.Source: §2.1 · p. 2

2.2 Detecting toxic content

• Blocklists target exact abusive terms or known users but may not generalize; machine-learning classifiers use crowdsourced labels from varied sources, and domain language or concept drift can degrade performance.Prior studies also examined cross-community transfer and annotator disagreement; deployed examples include Perspective API and Instagram’s comment nudge.Source: §2.2 · pp. 2–3

2.3 Other intervention strategies

• Community tribunals, bystander reporting, and tools for family or friends can use community and context, but may not scale and do not remove moderators’ emotional burden.Source: §2.3 · p. 3

2.4 Differentiation from prior work

• Prior work studied training-data bias and classifier manipulation, but had not shown whether harassment experiences relate to differences in online toxicity perceptions.Compared with a college-campus hate-speech study of fewer than 500 participants, this work covers more participants, toxic-content categories, and online discussion, and tests personalized filters.Source: §2.4 · p. 3

3.1 Survey instrument

• The pilot found “toxic” elicited the broadest interpretation, including judgments about constructiveness, relevance, and threats, so the researchers adopted it for the survey.The pilot compared “hateful,” “offensive,” and “toxic” terminology.Source: §3.1 · p. 3, Table 1 · p. 3
• Five participants rated each comment: in the pilot, 78% of comments converged to their final rating distribution after five ratings, with only marginal gains from additional ratings.Convergence meant the running average fell within 10% of the global average toxicity score.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 Sourcing potentially toxic content

• Comments were drawn from Twitter, Reddit, and 4chan using Perspective API toxicity-score strata, with the 0.5–0.8 ranges oversampled because participants agreed least there.The final sample included at least 5% of comments from each API stride and at least 16,000 comments per platform.Source: Table 2 · p. 4, §3.2 · pp. 4–5

3.3 Recruitment and validation

• The validated dataset contains 17,280 participants and 107,620 rated comments; the US-based MTurk sample included historically at-risk groups but did not perfectly match Census demographics.Participants passed an attention check and an open-ended response threshold; demographic questions could be declined.Source: §3.3 · p. 5, Table 3 · p. 5

3.4 Ethical considerations

• Participants received an exposure warning, could decline the task, and saw a small number of potentially toxic comments after the sampling strategy excluded the most egregious content.The study received institutional IRB approval.Source: §3.1 · p. 3, §3.4 · p. 5

4.2 Subcategories of toxic comments

• Insults were identified in 67% of comments, profanity in 52%, and identity attacks in 51%; identity attacks appeared in 85% of “Extremely toxic” comments versus 57% rated “Slightly toxic” or lower.These figures describe behaviors raters identified, not their prevalence across the sampled sites; threats and sexual harassment were more often rated “Extremely toxic” than other listed categories.Source: §4.2 · p. 6, Table 4 · p. 6

4.4 Filtering and removal recommendations

• 42% of participants said toxic content was frequently or very frequently a problem, while 51% said it was rarely or occasionally a problem.47% assigned responsibility for addressing toxic content to the sender, compared with 27% who assigned it to the hosting platform.Source: §4.4 · pp. 7–8

5.1 Modeling participant decision making

• The analysis models toxicity ratings as quasi-Binomial outcomes using demographic traits, harassment experiences, and beliefs as predictors.Ratings of “Moderately toxic” or higher count as toxic; the model holds other traits constant when reporting odds.Source: §5.1 · p. 8

5.2 Influence of personal experiences

• Being a target of toxic content raised the odds of rating new content toxic 1.483 times, while witnessing it lowered the odds to 0.780 times.The authors suggest empathy may relate to the first association and desensitization to the second.Source: §5.2 · p. 8
• Participants answered the harassment-experience questions after labeling, so their responses may have been colored by the comments they rated.Source: §5.2 · p. 8

5.3 Influence of demographics

• Participants ages 18–34 had 1.227–1.238 times the odds of rating comments toxic compared with participants ages 35–44.The authors also report higher harassment experience among younger participants, which may shape sensitivity.Source: §5.3 · p. 9
• Identifying as LGBTQ+ raised the odds of rating comments toxic 1.644 times; LGBTQ+ participants also had higher odds across all toxic-content subcategories.The largest subcategory increase was for threats, at 1.865 times.Source: §5.3 · p. 9
• Religion being “Very important” raised the odds of rating comments toxic 1.840 times, while being a parent raised them 1.330 times.Religious participants had higher odds across subcategories; parents also more often flagged sexual harassment and profanity.Source: §5.3 · p. 9
• Racial or ethnic minority status raised the odds of rating comments toxic 1.126 times; advanced degrees raised them 1.365 times versus high school education.Identifying as an independent lowered the odds to 0.901 times compared with liberal participants.Source: §5.3 · p. 9

5.4 Influence of technology beliefs

• Participants who saw toxic content as “Very frequently” a problem had 1.125 times the odds of rating content toxic; a “Very positive” view of technology corresponded to 1.142 times.The authors suggest these participants may have a lower toxicity threshold or feel more obligation to address toxic content.Source: §5.4 · p. 9

6.1 Perspective API

• At the highest Perspective API stride, participant agreement was only 45% in Table 6; the API’s accuracy was 51% in the reported analysis.The authors note that the stratified dataset omits most benign content, making whole-dataset standard performance metrics misleading.Source: §6.1 · pp. 9–10, Table 6 · p. 10
• Perspective API overflagging often involved profanity participants did not find toxic (82%), while identity attacks made up 39% of comments it missed.The authors also found sensitivity to minor text changes, such as “nofuckingbody” scoring 0.21 versus 0.93 with spaces.Source: §6.1 · p. 10
• Personalized Perspective API thresholds raised average accuracy to 0.68, an 86% increase over the one-size-fits-all classifier; 71.5% of participants improved.The median personalized threshold was 0.61; 21.6% of participants were best served by a threshold of 0.0.Source: §6.1 · p. 11, Figure 5 · p. 11
• Demographic threshold tuning improved accuracy by 0–20.6%, less than personalized tuning; age-specific thresholds produced the best cohort-level gain.The authors conclude that cohort models would need to account for multiple factors.Source: §6.1 · p. 11, Table 7 · p. 11

6.2 Instagram nudges

• Instagram’s classifier triggered on 27% of sampled toxic comments and on none of the sampled benign comments.Identity-based attacks were 47% of the toxic comments that triggered it; the authors found no unifying theme explaining missed toxic comments.Source: §6.2 · p. 12

Best practices for crowdsourced labeling

• Five ratings per comment balanced crowdsourcing cost and accuracy, but aggregate labels may not generalize across demographic cohorts.Mechanical Turk participants best understood “toxic” among the tested terms; the authors recommend validating results on other crowdsourcing platforms.Source: §7 · p. 12

Towards personalized definitions of toxicity

• Per-user tuning increased decision accuracy by 86%, while single-trait demographic models improved accuracy by 0–20.6%.The authors say further performance gains will be challenging without specific user feedback and examples; community filters using multiple traits are suggested when personalization is unavailable.Source: §7 · p. 12

Measuring toxicity using existing classifiers

• The authors caution against off-the-shelf classifier use without post-processing or calibration: participants disagreed with Perspective’s verdict in 50% of cases at thresholds of 0.9 or higher on measured sites.Source: §7 · p. 12

Online Context

• The study omitted comment context, which may make it hard for participants to distinguish toxicity from sarcasm or joking.The authors note that in-situ context, such as whether content targets an acquaintance, may change responses.Source: §7 · p. 12

8 Conclusion

• The study surveyed 17,280 participants across the United States about their perspectives on online toxic content.Source: §8 · p. 12
• Toxicity ratings varied with demographic background, personal harassment experiences, and attitudes toward technology and toxic content.Source: §8 · p. 12
• Personalized thresholds for existing classifiers improved toxic-detection accuracy by 86% on average, and the authors released their labeled dataset for future work.Source: §8 · p. 12

Labeling exercise

• Participants reviewed 20 comments randomly drawn from online platforms and rated each comment from extremely to not at all toxic.They also indicated why a comment was toxic, whether they wanted to see it, and whether it should be allowed online.Source: Appendix—Survey instrument · pp. 15–16

Post-exercise questions

• Participants reported prior exposure to similar comments, personal targeting, and their views on who is responsible for addressing toxic comments.The instrument also asked what “toxic comment” means to them and whether they would like to add anything about toxic comments.Source: Appendix—Survey instrument · pp. 15–16

Demographic questions

• The survey collected demographics including age, race, gender, transgender identity, sexual orientation, education, religion, political orientation, and parenthood.It also asked about participants’ use of different site types and their views on technology’s impact and toxicity as a problem.Source: Appendix—Survey instrument · p. 16

Table 8

• Table 8 shows that Perspective API over-emphasizes profanity in comments participants rarely found toxic.It covers comments with high API scores and low participant ground-truth scores.Source: Table 8 · p. 18, Appendix—Divergent Comments · p. 18

Table 9

• Table 9 highlights nuanced identity-based and political attacks that participants deemed toxic despite low Perspective API scores.The table covers low API scores and high participant ground-truth scores.Source: Table 9 · p. 18, Appendix—Divergent Comments · p. 18

Model outputs

• For each model, the appendix reports treatment and reference groups, coefficient, standard error, z-score, p-value, and odds ratio.The treatment group is binary or categorical depending on the variable.Source: Appendix—Regression Tables · p. 19

Toxicity Model

• Table 10 reports a logistic regression model predicting whether a participant flags a random comment as toxic.The appendix describes the broader model as quasi-binomial.Source: Table 10 · p. 19, Appendix—Regression Tables · p. 19

Sublabel Models

• Tables 11–13 model whether participants flag random toxic comments as insults, identity attacks, or threats.Each table presents a logistic regression model for its respective sublabel.Source: Table 11 · p. 20, Table 12 · p. 21, Table 13 · p. 22
• Tables 14–15 model whether participants flag random toxic comments as profanity or sexual harassment.Each table presents a logistic regression model for its respective sublabel.Source: Table 14 · p. 23, Table 15 · p. 24