Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation

1 INTRODUCTION

• Toxicity classifiers have been shown to score some non-toxic language from minority communities as more toxic than equivalent language from non-minority communities.Examples include African American English and tweets by drag queens.Source: §1 · p. 1
• Raters outside a community may miss harms that are understood through shared lived experience, while community members may sometimes perceive comments as non-toxic when outsiders perceive them as toxic.Source: §1 · p. 1
• The paper compares ratings from African American and LGBTQ raters with those from a pool that does not identify with either group, focusing on comments representing those identities.Source: §1 · p. 2

2 RELATED WORK

• Related work includes studies of comment-author identity and community norms, while this paper focuses on annotator identity [19] [20] [34].Source: §2 · p. 2
• The paper extends work on slur annotation by studying multiple toxicity-related facets in Civil Comments with hundreds of selected identity-group raters rather than 20 mixed-identity Reddit raters.Source: §2 · p. 2
• Prior studies grouped annotators by imagined racial perspective or activist expertise; this study instead distinguishes lived identities, which the authors say reveals nuanced differences between identity groups.Source: §2 · p. 2
• Expert-annotated data has also shown bias, and the authors use crowdworkers’ identities and lived experiences as a proxy for expertise rather than recruiting trained experts [11].Source: §2 · p. 2
• Earlier work found that White and non-White annotators rated tweets differently, especially on topics such as police brutality, antiracist politics, and empowering history.Source: §2 · p. 3
• Prior research has examined annotator demographics through clustering, identity-based pools, expert pools, mixed-rater designs, classifier evaluation, author demographics, and model robustness.Source: §2 · p. 3
• The authors contribute an annotated Civil Comments dataset for toxicity, insult, threat, identity attack, and profanity, alongside analysis of African American and LGBTQ American raters’ annotations.Source: §2 · p. 3
• The study reports statistically significant annotation differences between specialized rater pools, while noting that the differences have nuances.Source: §2 · p. 3

3.1 Research Questions

• The research questions ask whether African American-identified and LGBTQ-identified raters annotate comments related to their communities differently from raters who do not identify with those groups.Source: §3.1 · p. 3

3.2 Specialized and Control Rater Pools

• Specialized pools comprise self-identified African American and LGBTQ American raters, while the control pool excludes people identifying with either group so the pools remain disjoint.Source: §3.2 · pp. 3–4

3.3 Designing Rater Pools: Crowd Contributor and Identity Considerations

• Participants were U.S.-based English speakers aged 18–35, screened on toxicity-related questions with a 75% accuracy threshold and assigned to pools by self-described identity.Source: §3.3 · p. 4

3.4 Ethical Considerations when Working with Rater Identity

• The study used pseudonymized annotations, minimum-wage assurance, limits on toxic-comment exposure, optional peer support, and disclosure that identity determined pool assignment.Source: §3.4 · p. 4

3.5 Dataset

• Researchers sampled 25,500 Civil Comments, including 8,500 identity-neutral comments and 8,500 for each identity group, and released 382,500 annotations.Identity-related comments were sampled using dataset labels and identity-mention models; the strategy could include both community discussions and comments about identities.Source: §3.5 · pp. 4–5

3.6 Task Design

• All three pools rated the same 25,500 comments, with five ratings per annotator and 15 ratings per comment; toxicity and related categories were rated on Likert scales.Source: §3.6 · p. 5

3.7.1 Descriptive Statistics.

• Descriptive statistics define mean differences between specialized pools and control, and classify identity comments by agreement or high disagreement.High agreement means a mean difference of 0; high disagreement means a difference of at least 1.Source: §3.7.1 · p. 5

3.7.2 Regression Analysis.

• Regression measures compare specialized pools’ proportional odds of higher toxicity, identity-attack, insult, profanity, and threat ratings.Source: §3.7.2 · p. 5

4.1.1 Rater disagreement.

• Differences between specialized and control-pool ratings ran in both directions, with no overall tendency toward higher or lower toxicity ratings across labels.The distributions of mean score differences had similar negative and positive sides.Source: §4.1.1 · pp. 5–6
• Using an absolute mean-difference threshold of at least 1, toxicity disagreement affected >12% of comments in both African American and LGBTQ rater pools, versus <1% for threat and profanity.Source: §4.1.1 · p. 6
• Across African American, LGBTQ, and control rater pools, high-disagreement comments had a higher percentage of toxicity (mean score < 0) than low-disagreement comments, while control pools overall judged more comments toxic than specialized pools.Source: §4.1.1 · p. 6
• Identity mentions appeared in 71% of comments with high disagreement, compared with 50% and 49% of comments with no disagreement.This pattern supports the authors’ hypothesis that specialized pools matter for understanding comments referencing identity groups.Source: §4.1.2 · p. 6

4.2 Regression Analysis

• Ordinal logistic regression modeled each label using rater pool as a predictor, with comment ID and rater ID as covariates.The analysis used cumulative odds models with proportional odds; the authors report that assumptions were met, although most cells were sparse with zero frequencies.Source: §4.2 · p. 6

4.2.1 Toxicity Odds Ratio.

• Toxicity disagreement exceeded 12% for both specialized pools, while threat and profanity disagreement was below 1%; the figure shows differences in both directions.The percentages are descriptive, not statistical significance tests.Source: Figure 1 · p. 7, Table 1 · p. 8
• High-disagreement comments were more often toxic by every pool’s ratings, but control pools found more such comments toxic overall than specialized pools.The reported overall percentages were 7.2% for control versus 4.5% for African American raters, and 6.6% versus 5.3% for LGBTQ raters; these differences were not presented as statistically significant.Source: Table 2 · p. 8, Table 3 · p. 8
• Control ratings were significantly lower than LGBTQ ratings across all five labels; versus African American ratings, control ratings were lower for identity attacks, profanity, and threats, higher for insults, and not significantly different for toxicity.The authors characterize the differences as relatively small.Source: §4.2.5 · p. 9

4.2.2 Identity Attack Odds Ratio.

• Control pools disagreed more with LGBTQ than African American raters about identity attacks, and under significant disagreement control pools rated more comments toxic than either specialized pool.Source: §4.2.5 · p. 9

4.3 Preliminary Content Analysis

• The analysis sampled the 100 comments with the largest mean differences, categorized them by broad themes, and found many were political.Comments with multiple themes were assigned to each relevant domain.Source: §4.3 · p. 9

4.3.1 Mean Differences between Specialized Rater Pools and Control

• Example comments were selected from the top 100 high-difference cases, and their fraction of toxic ratings served as the model-training score.The authors also compared these examples with Perspective API scores.Source: §4.3.1 · p. 9
• Using only control-pool ratings could yield false negatives when specialized raters judge a comment more toxic, or false positives when they judge it less toxic.Source: §4.3.1 · p. 9
• Perspective API scores were closer to control-pool scores for most comments, suggesting the models could benefit from annotations by specialized pools.Source: §4.3.1 · p. 9

4.4 Models

• The study trained three BERT models on separate rater-pool data and compared their AUC with Perspective on synthetic, HateCheck, and Civil Comments test sets.The study motivated model evaluation because observed annotation differences alone do not show how models trained on those data perform.Source: §4.4 · pp. 9–10
• For Civil Comments, the researchers selected identity-tagged comments where LGBTQ- versus control-trained scores or African American- versus control-trained scores differed most.The selected examples were intended to examine high model-score differences on comments mentioning the corresponding identity.Source: §4.4 · p. 10
• Specialized-pool models improved BPSN bias scores for several identity terms over the control-trained model, with gains differing by pool and term.The LGBTQ model improved scores for “queer,” “homosexual,” and “muslim,” and slightly for “gay”; the African American model improved “muslim” but declined for “gay” and “queer.”Source: §4.4 · p. 10
• Perspective usually outperformed the study’s classifiers, but the LGBTQ-trained model was sometimes competitive despite using much less training data.The comparison covered AUC scores for classifiers trained on control, LGBTQ, and African American annotations versus Perspective.Source: §4.4 · p. 10, Table 7 · p. 11

5 DISCUSSION

• Raters’ lived experience may inform toxicity judgments, including interpretation of African American English by people who identify with that community.The discussion notes that raters without that identity or linguistic training may lack nuanced understanding of African American English.Source: §5 · p. 11

5.1 Specialized Rater Pools for Inclusive ML Models

• Specialized rater pools are formed around a dimension such as African American ethnicity or LGBTQ identity, shifting who helps decide what counts as toxic.The discussion questions whether affected communities should help determine which models are considered effective, since existing benchmark annotations did not use specialized pools.Source: §5.1 · pp. 11–12

5.2 When to Use Specialized Rater Pools?

• For identity-related comments, the authors recommend specialized pools to check for opinion differences that could contribute to model bias if left unaddressed.Comments with high disagreement between control and specialized pools had higher percentages of identity mentions than comments with low disagreement.Source: §5.2 · p. 13

5.3 Specialized Rater Pools beyond Ethnicity and Sexual Orientation

• Identity affected annotations across toxicity, identity attacks, insults, threats, and profanity; the authors call for participatory research with different rater pools.They encourage researchers to consider and test additional kinds of specialized pools.Source: §5.3 · p. 13
• Specialized pools do not resolve all bias, since members of marginalized communities may still show bias toward their own communities.The authors characterize consulting communities as a good-faith effort toward reducing model bias, not a complete solution.Source: §5.3 · p. 13

5.4 Specialized Rater Pools Dataset and Future Directions

• The released dataset contains 382,500 annotations of 25,500 Civil Comments from LGBTQ, African American, and neither-identity pools.The authors invite deeper content analysis; preliminary findings identify identity attacks, threats, and profanity as differentiating categories, and the dataset can support comparisons of simulated groups with identity groups.Source: §5.4 · p. 13
• The dataset is intended to support replication and development of specialized pools across additional identities.The authors also suggest analyzing models trained on the released annotations and using the data to compare group-simulation methods against identity groups.Source: §5.4 · p. 13

6 LIMITATIONS

• The study included African American and LGBTQ American pools, so whether findings hold for other identity groups remains to be explored.The authors specifically call for future research with additional groups.Source: §6 · p. 13
• The study focused on binary identities and leaves open how different pools disagree on intersectional identities.The authors call for future work beyond binary identities and further research into pool disagreement.Source: §6 · p. 14
• Future work could use community-specific discussions rather than discussions about communities and apply deeper qualitative methods to explain rating differences.Source: §6 · p. 14

7 CONCLUSION

• Specialized-pool raters differed from control raters in several cases, and models trained on their smaller datasets can outperform models trained on larger datasets labeled by randomly assigned annotators.Source: §7 · p. 14

• The appendix includes the rating template (Figure 2) and examples for the rating instructions (Figure 3).Source: A RATER TEMPLATE · p. 15, Figure 2 · p. 15

• The appendix lists bias metrics for control, LGBTQ, and African-American pool models in Figures 4, 5, and 6, respectively.Source: B MODEL BIAS METRICS · p. 16
• The control model has lower BPSN scores for “gay,” “queer,” “homosexual,” and “muslim.”Source: Figure 4 · p. 16
• The LGBTQ model improves BPSN performance for “queer,” “homosexual,” and “muslim,” and slightly for “gay,” compared with the control model.Source: Figure 5 · p. 17
• The African-American model improves BPSN performance for “muslim” but drops for “gay” and “queer” compared with the control model.Source: Figure 6 · p. 17