PaperlayerView exact revision on arXiv

Source-linked AI summary

Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

Aida Mostafazadeh Davani, Mark Díaz, Vinodkumar Prabhakaran

arXiv:2110.05719v1cs.CLcs.CY

TL;DR

Subjective annotator disagreements can reflect meaningful perspectives that majority aggregation removes, motivating models that retain annotator-level judgments. The paper uses shared multi-task representations to predict individual labels, finding comparable or better performance across seven tasks and uncertainty estimates more aligned with disagreement. The approach is most constrained by the need for annotator-level data and manageable annotator-pool sizes.

  • Problem

    Subjective tasks may lack a single right answer, while annotator values and experiences can produce systematic disagreements that aggregation discards.

  • Method

    The paper models each annotator’s judgment as a separate task with shared learned representations, preserving annotator perspectives through prediction.

  • Results

    Across seven subjective tasks, the multi-task approach matches or outperforms majority-label baselines and achieves 63.2 F1-score for annotator modeling in GHC.

  • Takeaways & Limitations

    Predicted annotator disagreement provides uncertainty estimates that better correlate with annotation disagreement and can help identify cases for non-prediction or manual review.

  • Takeaways & Limitations

    The approach becomes computationally expensive with large annotator pools and requires annotator-level labels with enough observations per annotator.

Abstract

from arXiv · show

Majority voting and averaging are common approaches employed to resolve annotator disagreements and derive single ground truth labels from multiple annotations. However, annotators may systematically disagree with one another, often reflecting their individual biases and values, especially in the case of subjective tasks such as detecting affect, aggression, and hate speech. Annotator disagreements may capture important nuances in such tasks that are often ignored while aggregating annotations to a single ground truth. In order to address this, we investigate the efficacy of multi-annotator models. In particular, our multi-task based approach treats predicting each annotators' judgements as separate subtasks, while sharing a common learned representation of the task. We show that this approach yields same or better performance than aggregating labels in the data prior to training across seven different binary classification tasks. Our approach also provides a way to estimate uncertainty in predictions, which we demonstrate better correlate with annotation disagreements than traditional methods. Being able to model uncertainty is especially useful in deployment scenarios where knowing when not to make a prediction is important.

1 Introduction

Subjective annotations can encode systematic differences in values and perspectives, making a single majority-vote ground truth potentially inconsistent and lossy. The paper therefore models annotators separately while sharing task representations, preserving disagreement and enabling uncertainty estimates.

  • Multiple annotations are commonly aggregated by majority voting, averaging, or expert adjudication into a single label for supervised training.
  • Subjective tasks may lack one right answer, so enforcing a single ground truth sacrifices nuances in annotators’ assessments and disagreements.
  • Annotators’ socio-demographic factors, moral values, and lived experiences can systematically influence judgments in stance, sentiment, abuse, and hate-speech tasks.
  • Majority voting can assign inconsistent labels to similar instances when different annotator groups disagree about the same phrase.
  • The proposed multi-annotator architectures preserve each annotator’s internal consistency and systematic disagreements through prediction.
  • Across seven subjective tasks, the multi-task strategy matches or outperforms baselines and yields uncertainty estimates that better correlate with annotator disagreements.

2 Literature Review

Subjectivity and disagreement are established concerns in NLP, especially for abuse and emotion detection. Prior work includes aggregation alternatives, disagreement-aware training, and uncertainty estimation, while this paper models annotator perspectives directly with shared representations.

  • Subjective NLP tasks include sentiment, affect, emotion, and hate-speech detection, where a single real ground truth may be impossible or unnecessary.
  • 2.1 Detecting Online Abuse: Online-abuse research spans typologies, annotated datasets, efficient detectors, and subtler phenomena such as condescension and microaggressions.
  • 2.1 Detecting Online Abuse: Abuse-detection models can reflect and propagate societal biases, including biases involving identity terms, named entities, disabilities, and African American Vernacular English.
  • 2.2 Detecting Emotions: Emotion perception is affected by contextual factors including time, speaker, mood, personality, and culture, motivating distributional representations beyond aggregated labels.
  • Disagreement may arise from random variation, annotator or data features, label clarity, text ambiguity, and differing subjectivities or value systems.
  • Prior approaches estimate true labels, select uncertain items, weight disagreement in losses, or use multi-task modeling rather than resolving disagreement during data preparation.
  • 2.4 Prediction Uncertainty: Prediction uncertainty can support interpretation and human-in-the-loop moderation, while single-label probabilities may overlook factors contributing to uncertainty.

3 Methodology

The study defines annotations as an incomplete instance-by-annotator matrix and compares ensemble, multi-label, and multi-task architectures with a majority-vote baseline. Models predict annotator outputs separately and aggregate them at test time.

  • The dataset comprises text instances, annotators, and a binary annotation matrix with missing entries because annotators label only subsets of instances.
  • All three multi-annotator architectures use BERT-base representations, although the methods can also be implemented with other pretrained language models.
  • The baseline trains a single-task classifier to predict each instance’s aggregated majority label.
  • Ensemble: The ensemble approach independently fine-tunes one single-task classifier for each annotator, which may increase training time substantially.
  • During prediction, models do not know which annotators originally labeled each instance, so original majority voting is unavailable.
  • Multi-label Approach: The multi-label model predicts |A| annotator labels from one input representation using a shared fully connected layer and sigmoid outputs.
  • Multi-task Approach: The multi-task model treats annotators as separate classification tasks with shared encoder layers and separate output layers, then aggregates outputs to predict the majority label.

4.1 Data

Experiments use two datasets containing per-annotator labels and anonymous annotator IDs: Gab Hate Corpus for hate speech and a GoEmotions subset covering six Ekman emotions.

  • The experiments use Gab Hate Corpus and GoEmotions, both of which provide per-annotator labels and anonymous IDs for modeling annotators separately.
  • Gab Hate Corpus: Gab Hate Corpus contains 27,665 social-media posts, each annotated for hate speech by at least three annotators from a pool of 18.
  • GoEmotions: The GoEmotions subset contains Reddit posts labeled for six Ekman emotions: anger, disgust, fear, joy, sadness, and surprise.

4.2 Experimental Setup

The experiments use HuggingFace Transformers with Adam, task-specific learning rates, and cross-validation or early stopping depending on the dataset.

  • Models were implemented with the transformers v3.1 library from HuggingFace.
  • Training used the Adam optimizer.
  • GHC experiments used five iterations of stratified five-fold cross-validation, while GoEmotions experiments used early stopping.

4.3 Results on GHC

On GHC, the multi-task model outperformed majority-vote training while modeling annotator-level predictions and uncertainty. Multi-annotator uncertainty estimates correlated better with disagreement, with little added training cost for multi-task and multi-label models.

  • Prediction Results: F1=59.49 for the multi-task model versus F1=57.32 for the baseline on majority-vote prediction.
  • Prediction Results: The baseline had higher standard deviations for precision, recall, and F1 than the other three methods.
  • Prediction Results: F1=63.20 was the highest score for individual annotator-label prediction among the evaluated multi-annotator approaches.
  • Modeling Uncertainty: Multi-annotator uncertainty estimates correlated significantly better with annotation disagreement than Softmax and MC dropout.
  • Modeling Uncertainty: The multi-task and multi-label uncertainty estimates had correlation 0.86, while multi-task and MC dropout had correlation 0.53.
  • Computation Time: Multi-label and multi-task models added no significant training time compared with the baseline, unlike the ensemble approach.

4.4 Results on GoEmotions

On GoEmotions, the multi-task model’s performance depended on annotation coverage: it struggled on the full dataset but generally surpassed the baseline when annotators had enough training instances. It also estimated uncertainty more consistently with annotator disagreement, with little additional training cost.

  • Full dataset: The multi-task model outperformed the baseline on joy and sadness but underperformed it on the other four emotions in the full dataset.The F1-score ranges largely overlapped, while multi-task variability was higher than in GHC.
  • Data limitations: 29 annotators had fewer than 1000 training annotations, including six with fewer than 100.The resulting scarcity of positive examples made some annotator-specific tasks difficult or impossible to learn.
  • Filtered subset: With annotations from 53 annotators having more than 1000 training instances, the multi-task model outperformed the baseline on four of six emotions.It performed better on anger, fear, sadness, and surprise, while scoring slightly lower on disgust and joy.
  • Filtered subset: The filtered subset produced lower multi-task standard deviations than the baseline, suggesting greater robustness.The comparison used the same six emotion tasks and retained overlapping F1-score ranges.
  • Uncertainty and efficiency: Across six emotions and both dataset versions, multi-task uncertainty correlated better with annotator disagreement than Softmax and MC dropout, except for full-dataset joy.Training took 6.1 minutes per epoch versus 5.2 minutes for the baseline.

5 Analysis

The analysis examines where the multi-task model differs from the baseline and whether its uncertainty identifies incorrect predictions. Most mismatches involved baseline false positives, while uncertainty was especially high for false positives and low for true negatives.

  • Error Analysis: Among 27,665 GHC instances, the multi-task and baseline models disagreed on 1,945 labels.Table 4 compares their predictions with majority labels and annotator or prediction-head percentages.
  • Error Analysis: 57.94% of mismatches were non-hateful majority-label instances correctly classified by the multi-task model but incorrectly classified by the baseline.These baseline false positives often contained slurs or social-group tokens.
  • Error Analysis: 12.19% of mismatches involved hateful majority-label instances predicted non-hateful by the multi-task model and hateful by the baseline.Many such cases had social-group mentions and near-even prediction-head splits, indicating model uncertainty.
  • Uncertainty Analysis: Incorrect predictions had mean uncertainty 0.170, compared with 0.049 for correct predictions.Most errors were highly uncertain, although a small significant group of errors was made confidently.
  • Uncertainty Analysis: True negatives were almost always certain with median uncertainty 0.040, whereas false positives were almost always uncertain with median uncertainty 0.199.True positives and false negatives showed bimodal uncertainty distributions.

6 Discussion

The discussion positions multi-annotator modeling as an alternative to collapsing subjective judgments into a single ground truth. It preserves annotator perspectives, estimates disagreement-linked uncertainty, and faces data, computation, and privacy constraints.

  • Results: Across seven subjective tasks, the multi-task approach matches or outperforms majority-vote baselines, including 63.2 microaveraged F1 for annotator modeling in GHC.The reported performance covers both individual annotator prediction and aggregated predictions.
  • Practical implications: Multiple predictions provide uncertainty estimates that better match annotator disagreement and can support abstention or routing examples to manual moderation.The authors describe this as a practical deployment signal for potentially contentious instances.
  • Approach: Multi-annotator modeling predicts individual annotator labels while sharing representation layers, preserving consistency and systematic differences across perspectives.Each annotator receives a separate prediction head, while shared layers learn from the full dataset.
  • Practical implications: Separate annotator heads can support predictions conditioned on different value systems and represent diverse affective states in subjective tasks.The paper discusses applications to international media platforms and affect categories or dimensions.
  • Limitations: The approach is computationally viable with small annotator pools but may become expensive for datasets containing thousands of annotators.The experiments used 18 annotators for GHC and 82 for GoEmotions; clustering could reduce the number of heads.
  • Limitations: Multi-annotator modeling requires annotator-level labels and enough annotations per annotator, which many existing datasets do not provide.The authors note that sparse per-annotator data can make individual modeling difficult and suggest anonymized release of labels when feasible.

7 Conclusion

The conclusion presents separate annotator classifier heads as an alternative to majority-vote prediction for subjective classification. Across hate-speech and emotion tasks, the method models annotators effectively, matches majority-vote performance, and yields disagreement-sensitive uncertainty estimates.

  • Conclusion: The approach uses a different classifier head for each annotator instead of predicting the dataset’s aggregated majority vote.It is evaluated on hate-speech detection and six emotion-detection tasks.
  • Conclusion: The method improves annotator modeling, matches majority-vote prediction performance, and produces uncertainty more correlated with annotation disagreement than traditional methods.The conclusion also identifies bias mitigation and active learning as future application areas.

8 Ethical Considerations

The paper frames individual-annotator modeling as a way to preserve minority perspectives in subjective NLP tasks. It also warns that optimizing majority-label performance or exposing personal preferences can undermine representation and privacy.

  • Scope: The approach is intended for subjective tasks where a single true answer fails to capture nuances in affect, abusive language, and hate-speech annotations.The paper emphasizes that individual judgments should remain represented during supervised training.
  • Risks: Weighting annotator labels to improve majority prediction may further marginalize under-represented perspectives, especially heads that disagree with the majority.The authors caution that preserving minority perspectives depends on how individual predictions are used.
  • Privacy: Annotator-pool size matters for protecting privacy and anonymity because the model represents personal subjective preferences and biases.This concern is especially acute for sensitive tasks such as hate-speech annotation.
Loading 2110.05719v1…