PaperlayerView exact revision on arXiv

Source-linked AI summary

When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks

Eve Fleisig, Rediet Abebe, Dan Klein

arXiv:2305.06626v5cs.CLcs.AI

TL;DR

Majority-vote labels can miss real differences in how annotators judge potentially harmful text, especially when the targeted group is a small part of the annotator pool. The paper predicts individual ratings and target groups, then uses them to estimate target-group members’ opinions. It reports gains over a baseline across individual ratings, annotator variance, and target-group aggregate ratings.

  • Problem

    For subjective hate-speech judgments, annotator disagreement may reflect real differences in opinion, making it important to assess whether text is offensive to its targeted group.

    Source: §1 · pp. 1–2

  • Method

    The model predicts individual annotator ratings and text target groups, then combines them to predict ratings from target-group members.

    Source: §1 · p. 2

  • Results

    22% higher performance for individual-rating prediction, 33% for annotator-variance prediction, and 22% for target-group aggregate-rating prediction versus a baseline that does not model individual annotators.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Predicted target-group disagreement can flag examples as potentially mislabeled or for human moderation, while annotator-rating variance can estimate model uncertainty.

    Source: §5 · pp. 8–9

    The work uses English text primarily representing U.S. varieties and annotators from the United States and Canada; generalization to other groups and countries remains unclear.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Though majority vote among annotators is typically used for ground truth labels in natural language processing, annotator disagreement in tasks such as hate speech detection may reflect differences in opinion across groups, not noise. Thus, a crucial problem in hate speech detection is determining whether a statement is offensive to the demographic group that it targets, when that group may constitute a small fraction of the annotator pool. We construct a model that predicts individual annotator ratings on potentially offensive text and combines this information with the predicted target group of the text to model the opinions of target group members. We show gains across a range of metrics, including raising performance over the baseline by 22% at predicting individual annotators' ratings and by 33% at predicting variance among annotators, which provides a metric for model uncertainty downstream. We find that annotator ratings can be predicted using their demographic information and opinions on online content, without the need to track identifying annotator IDs that link each annotator to their ratings. We also find that use of non-invasive survey questions on annotators' online experiences helps to maximize privacy and minimize unnecessary collection of demographic information when predicting annotators' opinions.

1 Introduction

For subjective tasks, disagreement can reflect real differences in whether people find a text harmful, rather than annotation noise. The paper models individual opinions and target groups to identify cases where the targeted group may disagree with the majority.

  • In subjective tasks, a text may genuinely be offensive to some people and not others, so disagreement need not be noise.Source: §1 · p. 1
  • Majority-vote aggregation can obscure differences in opinion among annotators.Source: Figure 1 · p. 1
  • Target-group members may differ from the annotator majority because of relevant lived experience or greater familiarity with a statement’s context.Source: §1 · pp. 1–2
  • The model predicts individual annotators’ offensiveness ratings and text target groups, then combines them to predict target-group members’ ratings.Source: §1 · p. 2
  • 22% higher performance for individual-rating prediction, 33% for annotator-variance prediction, and 22% for target-group aggregate-rating prediction versus a baseline that does not model individual annotators.Source: §1 · p. 2

2 Motivation and Related Work

The section motivates treating annotator disagreement as informative and modeling the opinions of groups targeted by potentially harmful text. It also frames prediction without individually linking annotator identities as a way to limit collection of identifiable or extensive demographic information.

  • Annotator judgments can vary with experience, political views, race, and familiarity with language and community context.Source: §2 · p. 2
  • Underrepresented Black annotators’ opinions may be obscured when aggregated labels align more with White annotators’ opinions.Source: §2 · p. 2
  • Target-group members are typically well-suited to judge whether potentially harmful statements affect their group, given their stake and likely relevant lived experience.Source: §2 · p. 2
  • The approach predicts target groups and their members’ ratings without human-in-the-loop supervision, providing evaluators an independent source of information about possible majority-label errors.Source: §2 · p. 3
  • The method uses annotator demographics and online-preference survey responses to predict ratings without linking responses to individual annotators.Source: §2 · p. 3
  • Survey information about online preferences can help predict annotator ratings while reducing reliance on extensive, potentially invasive demographic questions.Source: §2 · p. 3

3 Approach

The approach combines a module that predicts individual annotator ratings with one that predicts the demographic groups potentially harmed by text. At test time, it predicts target-group members’ ratings using annotator information and the predicted target groups.

3 Approach

  • The two parallel modules predict individual annotator ratings from annotator and text information, and predict text target groups; at test time, target-group predictions select whose ratings to predict.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Individual Rating Prediction Module

  • The rating module uses survey responses and demographic information alongside the text, represented as template strings, to predict each annotator’s rating.Source: §3.1 · pp. 3–4
  • Individual ratings are trained as separate regression targets on a 0-to-4 offensiveness scale, and predicted rating variance is evaluated as an uncertainty measure.Source: §3.1 · p. 4

3.2 Target Group Prediction Module

  • The two modules are trained separately because no available dataset then combined sufficiently extensive annotator demographics, individual ratings, and target-group labels.Source: §3.2 · p. 4
  • The target-group module generates free-text group labels, standardizes variants, then matches them to annotator demographic templates to identify target-group annotators.Source: §3.2 · p. 5

4 Results

The results evaluate individual-rating prediction, target-group prediction, and the combined task of predicting target-group members’ ratings, while also testing whether less annotator information can retain accuracy. The authors compare demographic and survey inputs and assess performance across groups and prediction targets.

4.1 Individual Rating Prediction Module

  • Combining demographic and survey information improves individual-rating prediction by 22% and variance prediction by 33% over baseline; aggregate-rating prediction improves by 16%.Either demographic or survey information alone improves individual-rating prediction by 10%.Source: §4.1 · pp. 5–6

4.2 Target Group Prediction Module

  • On test-set examples with target groups identified in the text, the model achieves word movers’ distance 0.370, exact-match accuracy 58%, and partial-match accuracy 81%.Source: §4.2 · p. 6

4.3 Full Model Performance

  • Among annotators who were target-group members, the combined model achieved average-rating MAE 0.59 (22% improvement over baseline), individual-rating MAE 0.73 (18%), target offense error 0.8 (17%), and variance MAE 1.22 (28%).Variance was evaluated among target-group members when two or more of them labeled an item.Source: §4.3 · p. 7
  • Error varied across groups: target-text MAE was under 0.35 for the best-predicted target groups and over 1.52 for the worst, while best-predicted annotator groups had MAE under .66.For text targeting up to four groups, individual-rating MAE changed by no more than 0.02.Source: §4.3 · p. 7

4.4 Ablations

  • Online-toxicity opinions were the most useful isolated feature, followed by race; race, toxicity opinion, and social-media use nearly matched full demographics (individual rating MAE=0.79).The authors suggest targeted information collection to reduce privacy risks, while noting that proxy variables may skew data collection and need further study.Source: §4.4 · pp. 7–8, Table 3 · p. 8

5 Conclusion

The paper models individual annotator ratings and target-group opinions to identify disagreement with majority labels. Its findings support using annotator information to estimate uncertainty and guide moderation while limiting unnecessary collection of identifying data.

5 Conclusion

  • The model raises performance over the baseline by 22% at predicting individual annotators’ ratings, 33% at predicting variance among annotators, and 22% at predicting target group members’ aggregate ratings.The predicted variance provides an estimate of model confidence, while disagreement within a target group may call for attentive care.Source: §5 · pp. 8–9
  • Annotator survey responses and demographic features allow ratings to be modeled without tracking identifying annotator IDs.A model using annotators’ race, opinions on online toxic content, and social media usage performs comparably to one using full demographic information.Source: §5 · p. 8
  • Predictions of target-group ratings can flag examples where the group disagrees with the majority as potentially mislabeled or for human content moderation.Predicted variance among annotators can also help identify cases where a model is uncertain.Source: §5 · pp. 8–9
  • Disagreement among target-group members may warrant particularly attentive care when those members are experienced stakeholders.Opinions within a demographic group are not monolithic.Source: §5 · p. 9

Limitations

The study is limited to English text primarily representing varieties used in the U.S., with annotators from the U.S. and Canada. Its generalizability and handling of multiple-group and intersectional harms remain open questions.

Limitations

  • The study uses English text primarily representing varieties used in the U.S., with annotators from the U.S. and Canada.Kumar et al. (2021) annotators were from the United States; Social Bias Frames annotators were from the U.S. and Canada.Source: Limitations · p. 9
  • How well the framework generalizes to groups facing severe harms in the U.S. or harms in other countries remains unclear.The passage notes that in some contexts members of a persecuted group may be unsafe identifying themselves to annotate data.Source: Limitations · p. 9
  • Statements involving potential harms to multiple groups may be harder to analyze, while disagreements across multiple and intersectional groups are identified as future work.Source: Limitations · p. 9

Ethical Considerations

The paper cautions that demographic predictions can flatten diversity within groups and cannot replace representative data collection. It also notes privacy risks from detailed annotator information and the limits of coarse-grained demographic questionnaires.

Ethical Considerations

  • Predicting opinions from demographic groups risks imputing a single opinion to a group rather than representing its internal diversity.The authors say individual annotator modeling should aid, not replace, diverse and participatory data collection.Source: Ethical Considerations · p. 9
  • The model may help identify examples needing more data from a community or decisions delegated to community stakeholders rather than automation.Source: Ethical Considerations · p. 9
  • Detailed annotator information may make annotators re-identifiable, raising privacy concerns.Source: Ethical Considerations · p. 9
  • Coarse-grained questionnaires may erase specific or intersectional subgroup experiences and expertise shared across groups.The authors call for further research into effective and inclusive data collection.Source: Ethical Considerations · p. 9

A Input Formatting

The appendix illustrates survey and demographic information as third-person descriptions of a reader. The examples include online experiences and preferences as well as personal and demographic characteristics.

A Input Formatting

  • The survey example describes social-media, news, video-site, and web-forum use, along with exposure to toxic comments and views about technology and toxicity.Source: A Input Formatting · p. 10
  • The demographic example includes age, race, gender, education, political affiliation, parenthood, and the importance of religion.Source: A Input Formatting · pp. 10–11
  • The demographic example also states sexual orientation and cisgender identity.Source: A Input Formatting · p. 11

B Further Model and Dataset Details

The section documents dataset splits, model scale, and training compute. It reports splits for the Kumar et al. and Social Bias Frames datasets and the model’s recommended hyperparameters.

C Further Model and Dataset Details

C Results: Details

The section reports target-group prediction error patterns, demographic breakdowns of rating errors, and context about annotator counts. Tables 4–6 cover validation results and best- and worst-performing groups.

C Results: Details

  • Annotators labeled 20 examples on average (standard deviation 9); the final ablation category level averaged 31 annotators per bucket (standard deviation 142).Source: C Results: Details · p. 11
  • 33% of missed target groups were race-based, followed by religion (23%); false positives most often involved race (25%) or gender (18%).Source: C Results: Details · p. 11
  • Table 4 reports validation-set mean absolute error for the rating prediction module across different input features.Source: C Results: Details · p. 11, Table 4 · p. 12
  • Table 5 lists the five annotator demographic groups with the highest and lowest individual mean absolute error.Source: Table 5 · p. 12
  • Table 6 lists the five predicted target demographic groups with the highest and lowest individual mean absolute error, excluding non-mappable words.Source: Table 6 · p. 12

Related papers

Loading 2305.06626v5…