Source-linked AI summary
On Releasing Annotator-Level Labels and Information in Datasets
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, Mark Díaz
TL;DR
NLP dataset aggregation can flatten subjective judgments into labels that do not uniformly represent annotators or social groups. The paper analyzes majority voting across eight binary tasks and finds uneven individual and racial-group agreement, motivating releases that preserve annotator-level labels and responsibly collected demographics.
Problem
Aggregation of subjective annotations may assume a single correct label while obscuring individual and socio-demographic perspectives.
Method
The paper analyzes majority-label agreement across eight binary tasks in three datasets and compares agreement across selected demographic groups.
Results
Majority votes often under-represent annotators; disgust had κ < 0.3 for 27 of 82 annotators, and White annotators averaged 0.42 agreement versus 0.37 for Black annotators.
Takeaways & Limitations
Dataset releases should preserve annotator-level labels and, when responsibly viable, socio-demographic information so users can account for systematic disagreement.
Takeaways & Limitations
The demographic analysis excludes social groups with fewer than 50 annotators, and sentiment labels are binarized from a five-point scale for this analysis.
Abstract
from arXiv · showhide
A common practice in building NLP datasets, especially using crowd-sourced annotations, involves obtaining multiple annotator judgements on the same data instances, which are then flattened to produce a single "ground truth" label or score, through majority voting, averaging, or adjudication. While these approaches may be appropriate in certain annotation tasks, such aggregations overlook the socially constructed nature of human perceptions that annotations for relatively more subjective tasks are meant to capture. In particular, systematic disagreements between annotators owing to their socio-cultural backgrounds and/or lived experiences are often obfuscated through such aggregations. In this paper, we empirically demonstrate that label aggregation may introduce representational biases of individual and group perspectives. Based on this finding, we propose a set of recommendations for increased utility and transparency of datasets for downstream use cases.
1 Introduction
NLP datasets often aggregate multiple judgments into one ground-truth label, but subjective tasks have no single right answer and aggregation can obscure annotator differences. The paper analyzes whether majority labels represent individual and socio-demographic perspectives uniformly and recommends releasing annotator-level information.
- Subjective tasks such as sentiment and offensiveness detection may have no single right answer, so aggregation can sacrifice disagreement-related nuances.
- Annotators’ socio-demographic factors, moral values, and lived experiences influence interpretations of language in subjective tasks.
- The analysis covers eight tasks across three datasets to test whether majority labels uniformly capture annotator perspectives and socio-demographic groups.
- For many tasks, majority votes poorly reflect some annotators, with agreement scores between 0 and 0.4 for a significant proportion of annotators.
- The authors recommend releasing annotator-level labels and, when responsibly viable, socio-demographic information to support research on systematic disagreement.
2 Background
NLP datasets commonly flatten multiple judgments into a single label or score, although this practice assumes a correct label and can hide valuable subjective perspectives. More information-rich releases preserve label distributions, annotator-level labels, and sometimes annotator demographics.
- Aggregation strategies produce a single label or score for supervised learning, often obfuscating important annotation nuances.
- Single-label releases rely on the assumption that a majority or expert label is the correct one, overlooking the value of individual perceptions in subjective tasks.
- Label distributions reveal disagreement more than a single label but still provide less information than individual annotator records.
- Anonymous annotator-level labels enable users to study systematic individual differences, but do not reveal annotators’ socio-cultural backgrounds.
3 Impacts of Aggregation
The paper evaluates majority-label aggregation across eight binary tasks in hate-speech, sentiment, and emotion datasets. It finds that majority votes often fail to represent many annotators uniformly and that agreement can differ across racial groups.
- The study analyzes eight binary classification tasks across hate-speech, sentiment, and emotion datasets.
- The emotion analysis uses six standard Ekman emotions, while sentiment labels are mapped from five ordinal values to a positive-versus-negative distinction.
- 3.1 Q1: Do Aggregated Labels Represent Individual Annotators Uniformly?: Majority labels are defined as labels supported by at least half of annotators, and Cohen’s Kappa measures each annotator’s agreement with those labels.
- 3.1 Q1: Do Aggregated Labels Represent Individual Annotators Uniformly?: For disgust, 27 of 82 annotators had very low agreement with the majority vote, while sentiment showed similarly low agreement for around one-third of 450+ annotators.
- 3.2 Q2: Do Aggregated Labels Represent All Social Groups Uniformly?: The demographic analysis compares agreement by race, gender, and political affiliation in the sentiment dataset.
- 3.2 Q2: Do Aggregated Labels Represent All Social Groups Uniformly?: White annotators averaged 0.42 agreement versus 0.37 for Black annotators, a significant racial difference, while political and gender differences were not significant.
4 Utility of Annotator-level Labels
Prior work shows that retaining annotator-level information can support models that represent disagreement, annotator clusters, uncertainty, and individual perspectives. These approaches can improve predictive performance or model uncertainty without significant losses in accuracy or efficiency.
- Annotator disagreement has been incorporated into training loss functions to improve predictive performance.
- Multi-task approaches use annotator disagreements in machine translation and related modeling tasks.
- Annotator clusters can be detected from agreement patterns in hate-speech annotation.
- A multi-annotator architecture models annotators’ perspectives separately and supports uncertainty modeling without significant loss of accuracy or efficiency.
5 Discussion and Conclusion
The paper argues that preserving annotator perspectives is necessary for modeling subjective phenomena and recommends releasing annotator-level labels, socio-demographic information when responsible, and documentation of annotation processes.
- Annotator perspectives should remain separate because they reflect socially embedded experiences and knowledge relevant to subjective phenomena.
- Dataset developers should release annotator-level labels, preferably anonymously, while leaving downstream users free to choose how to use or aggregate them.
- Socio-demographic information can help assess whether datasets and trained models equitably represent perspectives from different social groups.
- Documentation should cover annotator recruitment, selection, assignment, and efforts to ensure representational diversity.